FINAL EVALUATION OF THE USAID GUINEA HEALTH SERVICE DELIVERY (HSD) ACTIVITY: INTEGRATED HEALTH SERVICE DELIVERY IN THE POST-EBOLA CONTEXT USAID’S HEALTH EVALUATION AND APPLIED RESEARCH DEVELOPMENT (HEARD) PROJECT December 2020 Health Evaluation and Applied Research Development (HEARD) is funded by the United States Agency for International Development (USAID) under cooperative agreement number AID-OAA-A-17-00002. The project team includes prime recipient University Research Co., LLC (URC) and sub-recipient organizations. ACKNOWLEDGEMENTS The evaluation team is deeply appreciative of the Guinean women and men, clients, health service providers, managers, policymakers, and implementing partners who gave their time and insights to enrich this evaluation through their participation. The evaluation would not have been possible without the expertise and diligence of our in-country implementation partners, Maferinyah Training and Research Center in Rural Health, under the leadership of Alexandre Delamou and assisted by Foromo Timothee Beauvogui. The team extends special thanks to the USAID Mission in Guinea for guiding this evaluation, particularly May Mwaka, Mamadou Lamarana Tambah Bah, Izetta Simmons, Souro Kamano, and Pierre Belamou; Jhpiego’s Health Service Delivery (HSD) project team led by Dr. Yolande Hyjazi and M & E Director Jacqueline Aribot; and the Strategy Reference Group Members. Recommended Citation Alison El Ayadi, Adriane Martin Hilber, Alexandre Delamou, Laura Buback, Samantha Ski. Final Evaluation of the USAID Guinea Health Service Delivery Activity: Integrated Health Service Delivery in the Post-Ebola Context. Cover photo Centre de Sante Kassapo, Boké (Credit: Marie Boré) FINAL EVALUATION OF THE USAID GUINEA HEALTH SERVICE DELIVERY (HSD) ACTIVITY: INTEGRATED HEALTH SERVICE DELIVERY IN THE POST-EBOLA CONTEXT DECEMBER 8, 2020 This report was produced for review by the United States Agency for International Development (USAID). It was prepared by University Research Co., LLC (URC) The University of California, San Francisco’s Global Institute for Health Sciences (UCSF), and the City University of New York’s Graduate School of Public Health and Health Policy (CUNY) and was authored by Alison El Ayadi, Adriane Martin Hilber, Alexandre Delamou, Laura Buback, and Samantha Ski. This report is made possible by the support of the American people through USAID. The contents of this report are the sole responsibility of the URC, UCSF, and CUNY School of Public Health and do not necessarily reflect the views of USAID or the United States Government. USAID’S HEALTH EVALUATION AND APPLIED RESEARCH DEVELOPMENT (HEARD) PROJECT This final performance evaluation of the Health Service Delivery (HSD) Activity reviewed the first 4 years of implementation to understand project effectiveness in increasing quality of care, increasing demand for services and strengthening health systems; to identify lessons learned and missed opportunities; and to identify factors affecting post-investment sustainability. Evaluation questions focused on quality of care, community-focused interventions, health systems strengthening, fistula prevention and care, gender￾based violence, and leveraging of USAID and other development partner activities to advance RMNCH+ results in Guinea. Primary mixed-methods data were collected from key stakeholders and a subset of health facilities. HSD activities contributed to substantial progress in strengthening the system and actors to provide the integrated care package through provider training and supervision, environmental and materials support, and advocacy efforts. Continuing challenges included training needs and managerial support, human resource deployment and retention, lack of essential medicines and commodities, appropriate space for service provision, and inadequate equipment maintenance. Systems strengthening achievements contributed to increased community member utilization, especially for ANC, family planning, and child health; however, important disparities and equity gaps remain. Ministry of Health (MOH) governance, leadership, and ownership of quality improvement at the facility level remained insufficient to sustain HSD-supported improvements. HSD activities contributed to building fistula surgical capacity and coverage; however, the investment was small compared to the need and sustainability remains a significant concern. Similarly, gender-based violence￾related health and legal services and prevention activities were made available in targeted areas, though greater investment and stronger governance is needed. While HSD activities built on prior USAID achievements and concurrent programming, flexibility in USAID programming, and strong government leadership and coordination is needed. The Standards￾Based Management and Recognition program was an important driver of quality improvements in family planning, emergency obstetric care, and infection prevention and control, and SBM-R recognition represented high-quality services to health staff and communities. Evaluation recommendations include: target client engagement throughout the continuum of care and institutionalize functional referral systems (1.1), augment support for integrated critical services for vulnerable populations (1.2); support the Government of Guinea to implement the Community Health Policy and integrate community health services (2.1); strengthen community engagement through supporting the community action cycle, community groups and health posts (2.2); strengthen social accountability for removing financial, gender and cultural barriers for vulnerable groups through addressing service fees and supporting community health mutuals (2.3); require accountability, leadership and ownership from implementing partners for sustainability through results-based accountability measures (3.1); align SBM-R with existing governmental quality improvement measures and advocate for an institutionalized quality improvement approach (3.2); continue supporting the DHIS2 health information system to achieve full implementation to facilitate evidence-based decision making (3.3); scale up support for pre-service midwifery and nursing education and skills labs (3.4); strengthen facility-based equipment, infrastructure, and supply management (3.5); leverage USAID health sector investments for improved collaboration, communication and impact between its projects/activities and others in the health development field (4.1); incorporate flexibility in project design, and adapt implementation and outcome measures to changing needs (4.2); design projects collaboratively to increase ownership and devolve responsibility to government partners (4.3); consider demonstrating the full effectiveness of investment activities by prioritizing depth of investment (4.4). ABSTRACT ii Abstract ......................................................................................................................................ii Acronyms.................................................................................................................................iv Executive Summary..........................................................................................................1 Evaluation Purpose..........................................................................................................3 Evaluation Questions.....................................................................................................4 Context and Project Background.......................................................................6 Guinea RMNCH Context.....................................................................................6 Health Systems Context......................................................................................8 USAID-Guinea Country Development Cooperation Strategy...............................................................................................................................9 Activity Background..............................................................................................10 Evaluation methods and limitations...............................................................13 Overview of methods...........................................................................................13 Primary data collection......................................................................................13 Sampling..................................................................................................................14 Data Collection..................................................................................................14 Data Analysis.......................................................................................................14 Ethics review................................................................................................................16 Limitations......................................................................................................................16 Findings ...................................................................................................................................18 Evaluation Question 1 – Quality: To what extent has quality of health care services improved as a result of the HSD activity?...........................................................................18 Evaluation Question 2 – Community focused interventions: To what extent did HSD contribute to improved access and use of the essential integrated care package of health services at targeted facilities?........22 Evaluation Question 3 – Health system strengthening: To what extent has HSD strengthened the health system to deliver an integrated package of RMNCH+ health care services across the continuum of care?...............................34 Evaluation Question 4 – Fistula Care: To what extent has local technical and managerial capacity been sufficiently built to support fistula prevention and care in Guinea............................................................................................................46 Evaluation Question 5 – Gender-Based Violence: How well have the limited financial resources been strategically directed to address GBV issues under the HSD project interventions? What could have been done differently to enhance achievement of results?.......................................................................................................................49 Evaluation Question 6 – Future perspectives: complementarity between usaid activities ...................................52 Key Considerations & Recommendations .............................................55 1. Availability of Integrated Care and Quality of Care...................................................................................................55 2. The Last Mile: Reaching the Most Vulnerable at the Community Level..............................................56 3. Health Systems Strengthening: Building on Successes...............................................................................57 4. USAID and Program Oversight: Programming for Sustainability.........................................................59 References............................................................................................................................62 Annexes ..................................................................................................................................65 Annex I: Evaluation Statement of Work..............................................65 Annex II: Evaluation Matrix..............................................................................74 Annex III: Data Collection Instruments............................................... 77 Annex IV: Sources of Information...........................................................112 Annex V: Strategy Review Group............................................................114 Annex VI: Disclosure of any Conflicts of Interest......................131 Annex VII: Data Tables.............................................................................................132 Annex VIII: SBM-R CASE STUDY...........................................................149 CONTENTS iv ANC Antenatal care CH Child health CHW/ RECO Community-based health workers: Community health worker/Community relay CI Confidence interval cIMNCI Community-level integrated management of newborn and childhood illness CMC Community medical center (Centre médico-communal) COSAH Comité de santé et d’hygiène/Health and hygiene committee CS Centre de santé (Health center) DHIS2 District Health Information Software 2 DHS Demographic and Health Survey DPS Direction Préfectoral de la Santé (Prefectural Health Directorate) DRS Direction Régionale de la Santé (Regional Health Directorate) EH EngenderHealth EICP Essential and integrated care package EmONC Emergency obstetric and neonatal care EVD Ebola virus disease FGD Focus group discussion FP Family planning GBV Gender-based violence ACRONYMS GHSC-PSM Global Health Supply Chain–Procurement Supply Management (also known as PSM) GOG Government of Guinea HMIS Health management information system (SNIS en français) HN Hôpital national (National hospital) HP Hôpital préfectoral (Prefectural hospital) HR Hôpital régional (Regional hospital) HSD Health service delivery iHRIS Human resources information system IMNCI Integrated management of newborn and childhood illnesses IPC Infection prevention and control IR Intermediate result KII Key informant interview KMC Kangaroo mother care LAM Lactational amenorrhea LARC Long-acting reversible contraception mCPR Modern contraceptive prevalence rates MH Maternal health MLDA Long-acting reversible contraception (MLDA en français) MNH Maternal and neonatal health MNCH Maternal, neonatal and child health MNCH-FP Maternal, neonatal and child health, and family planning MoH Ministry of Health NGO Non-governmental organizations OF Obstetric fistula PAC Postabortion care PPC Postpartum care PPFP Postpartum family planning PE Peer educator PS Poste de santé (Health post) PSE Pre-service education PY Project Year RH Reproductive health RH N&P Reproductive health norms and procedures RMNCH+ Reproductive, Maternal, Newborn, and Child Health SC Save the Children SBCC Social and behavior change communication SBM-R Standards-based management and recognition TFR Total fertility rate UNFPA United Nations Population Fund USAID U.S. Agency for International Development WHO World Health Organization Integrated Health Service Delivery in the Post-Ebola Context 1 BACKGROUND The USAID-Guinea Health Service Delivery (HSD) Activity is a 5-year Cooperative Agreement (2015–2020) implemented by Jhpiego, EngenderHealth, and Save the Children. Building on prior USAID-supported achievements, HSD seeks to expand access to and availability of integrated health services to improve the quality of reproductive, maternal, newborn and child health services, with a sub-focus on obstetric fistula, gender-based violence, and cross-cutting infection prevention and control. Efforts ultimately targeted 278 health facilities and 1,035 surrounding communities in 32 prefectures, throughout Boké, Conakry, Kindia, Mamou, Faranah, Kankan and Labé regions, covering 85% of Guinea’s population. SUMMARY OF FINDINGS ➤ Significant improvements in quality of care, including increased access to family planning services, improved quality of antenatal and delivery care, and expansion of fistula prevention and care. ➤ Greater access and use of the essential integrated care package throughout the continuum of care. ➤ Health systems and workforce strengthening through training, increased data availability, and the provision of equipment, drugs, and commodities. ➤ Capacity-building for gender-based violence prevention through facility-based screening and community-based outreach. ➤ Leveraging prior and concurrent USAID and partner-supported activities to develop effective synergies, particularly during the Ebola outbreak. EXECUTIVE SUMMARY EVALUATION APPROACH The HEARD Project evaluation team reviewed HSD performance in the first 4 years of implementation, utilizing the project’s Implementation Science approach. In September 2019, the evaluation team traveled to Guinea for a scoping visit. Meetings with partners and stakeholders provided insight that defined final evaluation objectives. Data collection and analysis was conducted in partnership with The Maferinyah Center for Training and Research in Rural Health, a local institute with extensive evaluation experience. Six questions, addressing quality of care, community-focused interventions, health systems strengthening, fistula prevention and care, gender-based violence, and future perspectives, guided the evaluation. Evaluation employed a mixed-methods approach, analyzing data from a variety of sources, including 196 in-depth interviews and focus groups, 139 surveys, and 70 facility/community data validations from 10 hospitals, 26 health centers, and a range of regional and national stakeholders. Additionally, a case study on the standards-based recognition and management process focused on 6 facilities. Results from all data sources were compared in a multi-stage analysis and triangulated for validation of key findings. An external Strategic Reference Group, consisting of 5 individuals with strong contextual knowledge, further reviewed and validated results, and translated findings into actionable recommendations. FINDINGS Quality Health systems and workforce strengthening. The number of facilities providing the full integrated package increased from 0 to 227, representing 82% of the 278 HSD-targeted facilities. Further, 99% of assessed service providers’ and community health workers’ performance met national standards. 2 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Expanded access and improved quality of family planning services. Across 278 HSD-supported facilities, family planning service availability increased from 51% in FY16 to 98% in FY19. A total of 999 providers and 731 community health workers underwent HSD￾supported training in family planning. Improved quality of antenatal and delivery care. Seventy-four percent of an estimated 275,000 pregnant women in project areas achieved the recommended 4 antenatal care visits in FY19. Most facilities offered antenatal care services within a clean and welcoming room (83%). From FY17 through FY19, facility delivery increased from 42% to 45%. Access and Use of the Essential Integrated Care Package Increased utilization throughout the continuum of care. The nationally-representative Guinea Demographic and Health Survey reports that between 2012 and 2018, demand for modern methods of family planning, health facility delivery, and diarrhea treatment increased by 16%, 14%, and 22%, respectively. However, demand for other services, including child vaccinations, decreased. Strong technical support for policy documents and training materials. HSD supported the development of the Norms & Procedures of Reproductive Health for the essential and integrated care package, guiding health service delivery and training at all levels of care. Health System Strengthening Strengthened governance for health system improvement. HSD strengthened the potential of national and local governance for health system improvement through training and managerial support, as well as increasing the availability and use of data in decision making. Provision of equipment, drugs, and commodities. To ensure staff had the conditions needed to perform to standard, HSD provided equipment, drugs, and other commodities broadly, and conducted infrastructural improvements at 10 hospitals and 2 health centers, This built community trust and supported service utilization. Fistula Prevention and Care Increased fistula surgical capacity and prevention. HSD expanded upon existing fistula repair services at three hospitals, while integrating new fistula repair services into three others. HSD contributions supported 760 repairs from FY2016 to FY2019, and trained 232 individuals on fistula surgery, screening, and care. HSD also supported community and facility-based prevention activities. Reintegration activities were conducted among more than 234 women. Gender-based Violence (GBV) Health systems strengthening for GBV. Though the HSD activity committed limited financial resources to GBV, it was effective in strengthening the health system’s capacity to address GBV through facility￾based screening and community-based prevention, and through legal case management. Future Perspectives Effective collaboration with concurrent programming. HSD activities were designed to complement other USAID and partner-supported activities in the country. These synergies were very effective, particularly during the Ebola outbreak. Strong government leadership and coordination across donors and programs should guide future work. Continuing challenges include human resource deployment and retention, lack of essential medicines and commodities, appropriate space for service provision, and inadequate equipment maintenance. RECOMMENDATIONS 1. Continue to expand availability of quality, integrated health services. — Target client engagement throughout the care continuum, and institutionalize functional referral systems to strengthen quality of care improvements. — Augment or continue support for integrated critical services for the most vulnerable, including adolescents and youth, women in need of fistula surgery, and survivors of gender-based violence. Integrated Health Service Delivery in the Post-Ebola Context 3 2. Develop community-level interventions to reach the most vulnerable. — Support the Government of Guinea to implement the Community Health Policy and integrate community health services. — Strengthen community engagement through support for the Community Action Cycle, community groups, and local health posts. — Address financial, gender and cultural barriers for the most vulnerable by removing service fees for critical services and supporting community health mutuals. 3. Strengthen health system in governance, quality improvement, and sustainability. — Align standards-based management and recognition with government’s existing quality improvement measures. Advocate for a system￾wide alignment within Ministry of Health supported facilities to ensure alignment of quality improvement processes. — Strengthen facility-based management of equipment, infrastructure, and supplies. 4. Leverage USAID investments for greater impact and sustainability. — USAID should leverage its investments for improved collaboration and communication between its own projects and others in the health development field. — Design projects collaboratively to increase ownership and devolve responsibility to government partners. — Integrate flexibility in project design, implementation and outcome measures to adapt to changing context and needs. EVALUATION PURPOSE The United States Agency for International Development’s (USAID’s) Health Evaluation and Applied Research Development (HEARD) Project has conducted this final and forward-facing evaluation of USAID/ Guinea’s Health Service Delivery Activity. The Purpose of this assignment was to conduct a final performance evaluation of the USAID/Guinea Health Service Delivery (HSD) Activity. The evaluation team engaged with stakeholders and partners to review HSD performance in the first 4 years of implementation with the aim of understanding project effectiveness against the results framework. The evaluation identifies and documents successful approaches practices; lessons learned and missed opportunities; and factors affecting post-investment sustainability of service delivery processes and outcomes. The evaluation findings will be used to help USAID and its partners to understand the successes and challenges of the activity to determine implications for future USAID support to RMNCH programming in Guinea (and elsewhere), including through the negotiation of the next USAID/Guinea Country Development Cooperation Strategy. This end of activity performance evaluation will inform subsequent USAID maternal child health and family planning intervention programming to support greater country self-reliance. 4 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity EVALUATION QUESTIONS Evaluation Question 3: Health system strengthening: To what extent have HSD activities strengthened the health system to deliver an integrated package of RMNCH+ health care services across the continuum of care? 3.1 To what extent did the HSD Activity strengthen governance by building leadership and commitment of local partners, including government, at different levels (e.g., Ministry of Health, Governorates; district health management teams) civil society, communities, and private sector midwifery and nursing schools? 3.2 To what extent has the HSD Activity contributed to the availability of equipment, commodities and drugs for the delivery of (integrated) prevention, care and treatment services? 3.3 To what extent has the HSD Activity improved the quality of health services data and its use for decision-making at all levels (e.g., service delivery point, managerial, provider, and data collection capacity)? 3.4 What was the added value of supporting pre￾service training through the Faculty of Medicine and midwifery schools? 3.5 To what extent have HSD initiatives demonstrated potential for post-investment sustainability? Evaluation Question 4: Fistula prevention and care: To what extent has local technical and managerial capacity been sufficiently built to support fistula prevention and care in Guinea? — To what extent has HSD contributed to building fistula surgical capacity and coverage? — How effectively have community-based prevention activities been supported? — How effective have reintegration activities been in terms of supporting women to reenter their communities? Evaluation Question 1: Quality of Care: To what extent has quality of health services improved as a result of the HSD Activity? 1.1 To what extent have HSD interventions contributed to increased availability of the complete package of integrated reproductive, maternal, newborn, child, and adolescent health (RMNCH+) care across targeted facilities (including family planning, obstetric fistula, and gender-based violence)? 1.2 To what extent has HSD strengthened provider capacity to deliver quality care to national performance standards? Evaluation Question 2: Community focused interventions: To what extent did HSD contribute to improved access and use of the essential integrated care package of health services at targeted facilities? 2.1 To what extent has HSD contributed to the delivery of the integrated community health package including referral linkages? To what degree did implementation of the integrated community health package and referral increase use of services? 2.2 To what extent have HSD interventions increased access and use of RMNCH+ services through training, capacity, coordination, and policy change? 2.3 To what extent has HSD’s social and behavior change communication approach improved the targeting and scale of those reached with health promotion activities? — Has the approach contributed to engaging and empowering community members and clients to access and use health care services? — To what extent did community members shift from recipients to advocates/promoters of health? 2.4 To what extent have community interventions worked to mitigate gender barriers and improve equity in access to care? Integrated Health Service Delivery in the Post-Ebola Context 5 Evaluation Question 5: Gender-based violence: How well have the limited financial resources been strategically directed to address gender-based violence issues under the HSD project activities? — What could have been done differently to enhance achievement of results? — To what extent have gender-based violence circles helped to mitigate the effect of gender-based violence in the community? — Have there been community targeted prevention efforts linked (or operating in parallel) to HSD activities? — Is referral from the community or the health facility to health and legal support services in place? Is it effective? How does it work? Evaluation Question 6. Future perspectives: How well did HSD complement and leverage efforts of other USAID and development partner activities to advance RMNCH+ results in Guinea? 6.1 To what extent did HSD complement and reinforce other USAID programs in the health and governance (e.g., synergies with USAID programs internally)? 6.2 To what extent do USAID health programs partner with the same health system levels and partners to facilitate synergies across partnerships? 6.3 To what extent do HSD and other USAID programs add value to other RMNCH+ initiatives in Guinea (i.e., external to USAID)? 6.4 What were missed opportunities for better synergy and advancement of results? 6 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity CONTEXT AND PROJECT BACKGROUND GUINEA RMNCH CONTEXT Achievements The United States is the largest bilateral donor of official development assistance to Guinea, providing approximately $46.4 million in investments in fiscal year 2018, $35.7 million of which was allocated to health and population activities.6 Investments by the United States made to Guinea over the years have contributed to strengthening democratic governance and successful transition to a democratic state with strong and transparent institutions, as well as supporting Guinea’s role in reducing fragility in the region. Investment in the health sector has been a key pillar of U.S. assistance, which has focused on increasing the availability of quality health services and improving food security, with a particular focus on improving delivery of health services, strengthening health systems, and encouraging communities to access health services. Guinea’s population of 12.8 million is young: 41% are under age 15 and 19% between age 15-24.1 Life expectancy at birth has increased considerably for Guineans over the past several decades, from 36.7 in 1980 through 62.1 in 2018.1 Guinea’s total fertility rate has remained relatively stable over time until recently; from 1983 to 2005, total fertility was constant at 5.8 and 5.7, respectively, and evidenced small reductions to 5.1 in 2012 and 4.8 in 2018.5 Adolescent childbearing, between ages 15-19, has declined over the past two decades from 37% in 1999 to 26% in 2018.5 Modern contraceptive prevalence among women in unions remain low at 11% in 2018,5 but use has doubled over the past five years. The most common contraceptive method among women in unions was lactational amenorrhea (3.7%), followed by injectables (1.9%), implants, and the oral contraceptive pill (1.8%). Among sexually active women not in union, 50.7% reported using a modern method and 1.6% reported using a traditional method; the modern contraceptive methods most commonly reported included the male condom (15.9%), implant (11.6%), and the injectable (9.7%).5 As modern method use has increased, the proportion of women using traditional methods of family planning has decreased to 0.3%.5 Total unmet need for family planning is estimated at 27.6% among women in union, of which 20.6% is for spacing and 7% is for limiting. The majority of women seek some antenatal care with a skilled provider (81%); however, significantly fewer women achieve the four recommended prenatal care visits (35%). Just over half of births from the period 2013-2018 were assisted by a skilled attendant, 53% in a health facility, an increase from 43% from the reference period for the prior Demographic and Health Survey (DHS) representing 2007-2012. A study conducted in two urban and peri-urban districts in Guinea identified mistreatment during facility-based childbirth as common, including physical and verbal abuse, neglect, and abandonment.7,8 Obstetric fistula from prolonged obstructed labor is twice as prevalent in Guinea, with lifetime prevalence of 6.0 (95% CI 3.9-7.4) per 1,000 of reproductive age compared to 3.0 (95% CI 1.3-5.5) per 1,000 women of reproductive age in sub-Saharan Africa overall.3,9 In the most recent DHS, 4% of women of reproductive age reported ever experiencing fistula￾related symptoms. From 1996–2005, maternal mortality was estimated at 980 deaths per 100,000 live births. Maternal mortality declined by about 26% to 724 per 100,000 live births from 2005–2012, and further to 679 per 100,000 live births in 2015.3,4 Estimates of maternal mortality following the Ebola outbreak are not yet available; however, death rates were anticipated to increase by 38% due to the impact of Ebola on healthcare personnel alone.10 Rates of maternal morbidity are not available, but are likely to be common. In the most recent DHS, 4% of women of reproductive age reported ever experiencing fistula-related symptoms.11 In 2010, the Integrated Health Service Delivery in the Post-Ebola Context 7 Government of Guinea introduced a free obstetric care policy in all public health facilities to include antenatal care and institutional delivery services, including cesarean section. Emergency transportation to district￾level hospitals was added to this policy in 2011. This policy introduction was found to result in a significant decrease in unmet obstetric needs between 2008-2012 in the health district of Kissidougou.12 Infant mortality experienced a decline from 91 per 1,000 live births in 2005 to 67 in 2012, although no subsequent reduction occurred through 2018 at 66 per 1,000 live births. Currently, only 18.7% of infants 12-23 months of age have received all recommended vaccines, with 23.9% having met the recommended vaccination schedule;5 22.3% of infants have not received any vaccinations at all.5 Health worker shortage has been cited as a significant impediment to providing adequate care. In 2014 the Guinean Ministry of Health reported a total of 1,417 general doctors, 1,505 nurses, 500 midwives, and 4284 allied health workers.13 Health care worker density per 1,000 population in 2015 was 0.1 physicians and 0.1 nurses and midwives, far below the 2.5 per 1,000 international benchmark and the 4.45 for universal health coverage.13,14 However, this varies substantially by district; for example, over half of all Guinean health care workers (55%) are located in the country’s capital, Conakry, despite the fact that only 13% of the Guinean population resides in this region. Additional challenges include an aging health care workforce, with 65% of health care workers aged 45 or above.13 Recent nationally-representative data on the prevalence of GBV are not available for Guinea; however, studies among sub-populations have identified high rates. For example, a 2014 study among family planning clients identified 92% had experienced intimate partner violence in their lifetime, including psychological (79%), sexual (68%), and physical (48%).15 However, nationally￾representative data on social norms around GBV indicate changing trends in Guinea. Between 2012 and 2018, the percentage of women who think that for at least one of the reasons cited, it is justified that a man beat his wife has decreased significantly, from 92% to 67%.12 Among men, this percentage decreased from 66% to 55%. These perspectives vary regionally, with the proportion of women for whom it is justified that a man has the right to beat his wife is higher in the Faranah and Kankan regions (81% in both cases) followed by Boké (79%) and Kindia (75%). The percentage is lowest in Labé region (53%). Many of Guinea’s efforts over time in the RMNCH+ sector have been supported by USAID investments. In September 2019, an evaluation team from USAID’s HEARD Project traveled to Conakry, Guinea for a scoping visit meant to inform the Final Evaluation of the USAID/Guinea Health Service Delivery Activity. Learnings from multiple stakeholder meetings provided historical context for how USAID has significantly supported health systems strengthening in the country through its investments, which substantially increased during the Ebola virus disease epidemic and post￾Ebola period. The portfolio of USAID investments and programs are presented in the Annex (Table 11). These investments have helped Guinea to address poor performance in the country’s health sector. Programs have prioritized RMNCH+ components, though the activities have also sought to build capacity across the health care system for infectious diseases, health financing, health governance, human resources for health, and facility infrastructure/construction. New Challenges There are several challenges impacting the demand for RMNCH+ health services in Guinea, perhaps none as pronounced as the legacy of the 2014-2016 Ebola Virus Disease epidemic in the country. This public health emergency led to a decline in health service utilization due to fear and low trust of the health system based on population perceptions of the system’s performance during the epidemic. Guinea was one of the primary countries affected by the 2014-2016 West African Ebola Virus Disease epidemic, recording a total of 3,351 laboratory confirmed cases and a case fatality rate of 62%.15 Approximately 6% of the cases occurred among health care workers.16 The first case of Ebola Virus Disease in Guinea occurred in 8 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity late December 2013, with the World Health Organization declaring the situation a Public Health Emergency of International Concern on August 8, 2014. The scope of the outbreak was largely attributed to the circulation of Ebola Virus Disease into crowded urban areas, cross-border mobilizations, and conflicts between key infection control practices and prevailing cultural and traditional practices.17 Most health districts reported Ebola Virus Disease cases (26 of 34 country-wide), with the greatest number of cases in Conakry and Macenta districts.17 The Forest region (Beyla, Guéckédou, Lola, Macenta, Nzérékoré, and Yomou districts) accounted for 44% of all Ebola Virus Disease cases despite representing just 22% of the country’s population.18,19 Guinea was declared Ebola-free in June 2016. In addition to the direct impact on reducing the number of health care workers, the Ebola Virus Disease epidemic exerted significant indirect impacts on health care availability and demand in Guinea for maternal and child health through halting preventive and emergency obstetric care and immunization programs;20-22 provider desertion from already understaffed health facilities due to fears of Ebola infection, particularly given a lack of adequate personal protective equipment and inadequate training;23,24 and reduced care-seeking behaviors for both emergency and preventive health care services due to fear of contracting Ebola at the health care facility.25,26 Broad community mistrust of health systems resulted due to amplified cultural differences between biomedical needs to stem the Ebola Virus Disease epidemic and the social context.26,27 Research studies have documented significant reductions in all-cause patient visits, fever cases, and malaria cases among children under the age of five years in Guinean health facilities during the Ebola Virus Disease epidemic compared to one year previously. For example, all-cause visits dropped by 31% at hospitals and up to 40% at health care centers, and fever cases reduced by 15% concurrent with a 24-30% reduction in antimalarial treatment. This contributed to thousands of preventable deaths associated with lack of treatment.28 Similar reductions to institutional deliveries and in￾hospital adult admissions were observed during the Ebola Virus Disease epidemic compared to the prior year,29–33 and Human Immunodeficiency Virus (HIV) services declined, including prevention of mother to child transmission services.34 In the year following the Ebola Virus Disease epidemic, antenatal care utilization, institutional delivery, and childhood vaccination rates did not show signs of recovery in the most heavily impacted areas.33 The challenge of improving the quality of RMNCH+ services is further complicated by the fact that Guinea has one of the youngest populations in the world. As previously mentioned, 60% of Guinea’s population is under the age of 25, and the median age is 19.1 This “youth bulge” stems from a high total fertility rate of 4.8 combined with declining infant and child mortality. A fast-growing population with an increasing number of individuals entering into their reproductive years will require concurrently increasing health systems investments in order to meet the RMNCH+ needs of the population. Finally, while the extent of disruption is currently uncertain, the current COVID-19 pandemic is likely to exacerbate many of the issues described above, requiring further innovative strategies to implement quality health care while reducing risk to health care personnel and patients and in ensuring that demand for health care does not wane. Close attention should be paid to how the pandemic plays out in health care supply and demand in Guinea, with support for continuity of care. HEALTH SYSTEMS CONTEXT The Guinean public health system is comprised of three national hospitals, 26 district hospitals, eight communal health centers, and 413 rural and urban health centers. The private sector includes 11 polyclinics and 33 medical and surgical clinics. RMNCH+ services are situated within the larger public health system that has faced many system-level challenges which were exacerbated by the Ebola Virus Disease epidemic. From a health systems perspective, provision of RMNCH+ services requires expanding access and availability of end-to- Integrated Health Service Delivery in the Post-Ebola Context 9 end services, as well as improving quality of care. Constraints to the health system include inadequate infrastructure, human resources, health financing, and health information systems. Specifically for RMNCH+, there are also non-standardized client flow mechanisms and management of RMNCH+ services.35 These challenges are largely tied to historically poor governance at all levels of the government and health system. Beyond the central level, the health system has a decentralized pyramid setup, cascading down to the regional and district level of local management. While this system assumes largely autonomous primary health care centers and hospitals, local￾level management capacity and motivation is weak. Likewise, health facilities may generate inconsistent revenue at the local level. Furthermore, the Government of Guinea allocates and disburses insufficient funds to the health sector, greatly underspending on the needs of the growing population. This history of poor governance is also rooted in the recurring authoritarian dictators since Guinea’s Independence from France in 1958. Current President Alpha Condé was the first democratically elected head of state in 2010, however, power remains concentrated with the president and civil society, citizen engagement has been historically weak, and government mistrust prevails. Thus, the theoretically decentralized health system is undermined by authoritarian political power and lack of resources.35 The Primary Health Care Initiative, however, was relaunched in 2010 following the 2008 Ouagadougou Declaration on Primary Health Care and Health Systems in Africa. Community mobilization efforts have led to increased community and citizen engagement in health services and community health and hygiene issues. Policy requires each health facility to have Community Health Workers (CHW/RECO) and Community Committees for Health and Hygiene (COSAH).13 Civil Society Organizations (CSOs) are also playing an increasing role in the relationship between citizens, communities, and their government. This increasing civil society may also impact community engagement in local health issues. USAID-GUINEA COUNTRY DEVELOPMENT COOPERATION STRATEGY The USAID Guinea Country Development Cooperation Strategy 2015-2020 (CDCS) outlines a broad plan about how USAID will support the Government of Guinea in carrying out its stated commitment to broad￾based political and economic reforms.35 This includes improved quality and utilization of health services to advance the quality of life of Guineans. The vision of the USAID/Guinea CDCS is: Citizens Drive Improvements to Guinean Quality of Life, which reflects the aspiration for a shift from the old authoritarian paradigm to a “Citizen to the Center” model in politics and development. With a renewed strategic vision, the Goal Statement “More Participatory Governance for a Healthier Guinea” was developed to reflect the principle that the old authoritarian paradigm needs to give way to a new model that elevates the citizen to the center of development. USAID/Guinea posits that measurable improvements in the quality of life for Guineans, most significantly reflected in improved health services, will be best advanced during the strategy period by continuing to focus on the most pervasive development challenges the country currently faces: poor governance and the resulting ineffective delivery of public goods and services. The results framework to reach the specified goal includes two development objectives (DO) each with respective intermediate results (IR) and sub-IR as outlined below: n DO 1: Utilization of Quality Health Services Increased — IR 1.1: Delivery of Quality Health Services Improved — IR 1.2: Healthy Behaviors & Demand for Quality Health Services Increased — IR 1.3: Health Systems Strengthened n DO 2: Democratic Governance and Economic Processes Strengthened — IR 2.1: Transparent, Competitive, Accountable Governance Strengthened — IR 2.2: Conflict Prevention and Mitigation Promoted in the Forest Region 10 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity — IR 2.3: Agricultural Education and Rural Micro￾Enterprises Strengthened DO 1 positions health as central to U.S. development assistance to Guinea, specifically addresses the need to improve quality of and demand for essential health services, health behaviors, and to strengthen health systems overall, in order to improve health outcomes among the population in Guinea. DO 1 is the foundation for the USAID/Guinea Health Service Delivery Activity and its results framework, which describes the role of improved RMNCH+ services in improving health outcomes by expanding access to and availability of the essential integrated health package (Figure 1). ACTIVITY BACKGROUND The USAID/Guinea Health Service Delivery (HSD) Activity is a five-year Cooperative Agreement (2015– 2020) with a total budget of US$28.8 million,* which was awarded in December 2015 to Jhpiego and its partners EngenderHealth and Save the Children. HSD aims to increase the availability and quality of an essential integrated care package for RMNCH+ across the health system. It sought to mitigate the effects of the Ebola epidemic, restore the community’s confidence in the health system and its use, and improve the health of Guinea’s population. Priorities of HSD include expanding access to and availability of integrated health services to improve the quality of RMNCH+ services. In addition to the primary focus on high-quality delivery of the essential integrated care package, HSD also includes a particular sub-focus on RMNCH+ areas such as obstetric fistula treatment, screening and treatment for precancerous cervical lesions, kangaroo mother care, cross-cutting elements such as strengthened biomedical waste disposal and infection prevention and control practices, and the integration of screening and care for gender￾based violence in the geographic areas of focus. To achieve its objectives, the project ultimately targeted 278 health facilities and 1,035 surrounding communities in 32 prefectures, throughout seven of eight regions in the country (Boké, Conakry, Kindia, Mamou, Faranah, Kankan and Labé; Figure 2). The program supported approximately 85% of the Guinean population. The eighth region, N’zérékoré, was not included within the HSD Activity due to existing donor-funded programming with similar objectives in that region. HSD consortium members were already working in Conakry, Faranah and Kankan regions under the Maternal and Child Survival Project and Fistula Care projects. The Activity added four new regions: Boké, Kindia, Labé, and Mamou. * An additional US$4.4 million was added in 2019 to the original budget ceiling of US$24.4 million, bringing the total award amount to US$28.8 million. The additional $4.4 million was to support mobile outreach services and additional facility infrastructure improvements. This evaluation does not examine the activities included within the supplemental funding, but focuses on the objectives within the original US$24.4 million. Figure 1. A Closer Look at DO1 from the USAID/Guinea CDCS Results Framework IR 1.1: Delivery of Quality Health Services Improved IR 1.2: Health Behaviors and Demand for Quality Health Services Improved IR 1.3: Health Systems Strengthened Sub-IR 1.1.1: Availability of Integrated Quality Maternal, Neonatal and Child Health Services Increased Sub-IR 1.1.2: Referral Linkages Strengthened Sub-IR 1.1.3: Availability of Prevention, Care and Treatment Services Increased Sub-IR 1.1.4: Access to Health Services Improved Integrated Health Service Delivery in the Post-Ebola Context 11 Boké* Gaoual* Koundara* Mali* Beyla Conakry* Coyah* Kérouané Kankan Ko Mandiana uroussa Siguiri Yomou Nzérékoré Lola Macenta Guéckédou Kissidougou Faranah Dinguiraye* Tougué* Koubia* Labé* Lélouma* Pita Télimélé Boa* Kindia Mamou Dabola Dubréka* Forécariah* Dalaba Fria* Regional HQ in Bold HSD Prefectures with previous RMNCH support under MCHIP/MCSP (Jhpiego, Save) HSD Prefectures with IPC support (Jhpiego) Fistula repair sites (EngenderHealth) New HSD Prefectures PMI/USAID Supported Prefectures Objectives and Implementation Approach HSD’s three intermediate results (IRs), shown in the HSD Results Framework (Figure 3) are to achieve improvements in the quality of care (IR1), demand for services (IR2) and strengthened health systems (IR3). HSD intervenes on multiple levels from training of providers and managers to building capacity of health facility directors and Ministry of Health staff to better manage facilities and the health care system to deliver quality integrated care. A key HSD indicator is for at least 80% of targeted facilities to offer the full range of essential integrated care package services consistent with their level in the healthcare system. HSD also works at the community level to engage with women and their families to proactively seek RMNCH+ care and provide services and timely referral through community health workers. HSD has, in addition, worked to increase * Figure 2. Guinea—Project Supported Prefectures access to screening, care, and referral for legal and support services for GBV survivors as well as prevention and care of obstetric fistula to fill service gaps and increase access to these neglected services. HSD has also implemented a quality improvement approach— the standards-based management and recognition (SBM-R) program—to incentivize hospital and health center managers to meet quality standards within their facilities. This recognition program has shown significant promise as a means to motivate managers and staff to engage in quality improvements in their facilities and to provide better care. This approach and the other intervention strategies of HSD presuppose a number of assumptions and hypotheses about how quality improvement efforts and community engagement can lead to sustained system improvements and changes in health seeking behaviors. 12 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity The full results framework is presented in Figure 3, including the six ‘guiding principles and cross-cutting considerations’: RMNCH+ integration, coordination, innovation, sustainability and capacity building, governance and accountability, and gender equity. Of the three IRs in the results framework, HSD was intended to focus primarily on IR1 (efforts to improve the delivery of quality health services) according to the Request for Applications (RFA), which guided applicants to only propose activities under IR2 and IR3 ‘that would naturally be a part of their health service delivery work’. • Promote and strengthen integration of RMNCH services • Strengthen the linkages along the home-to￾hospital continuum • Implement quality of care interventions • Strengthen integration of clinical and outreach services • Promote family and community-level care • Strengthen client/ community involvement • Improve clinical governance • Strengthen health care provider and system capacity • Improve data collection and use for decision making Increased availability of services, referral linkages, quality of services and access to services (Sub IR1.1.1-1.1.4) Improved coordination, quality, targeting and scale of SBCC and health promotion activities (Sub IR1.2.1–1.2.3) Strengthened policy, planning, governance and HRH; improved availability of commodities, drugs and data for decision making (Sub IR1.3.1–1.3.4) 1.1: Delivery of Quality Health Services Improved Illustrative Indicator: percent of facilities offering the essential integrated care package 1.2: Healthy Behaviors and Demand for Quality Health Services Improved Illustrative Indicator: percent of facilities with increased caseload over baseline 1.3: Health Systems Strengthened Illustrative Indicator: percent of facilities maintaining 80% of clinical and management performance standards • Utilization of quality health services increased (DO1) • Linkages across the continuum of care improved • Reduce missed opportunities • Quality of care improved • Patient satisfaction increased • Reduced maternal morbidity and mortality • Reduced newborn and child morbidity and mortality • Increased couple years of protection (FP) MNCH/FP Integration Coordination Innovation Sustainability & Capacity Building Governance & Accountability Gender Equity HSDP Interventions Sub-Intermediate Result Intermediate Result Outcome Impact Guiding Principles and Cross Cutting Considerations Figure 3. HSD Results Framework The RFA indicated IR2 and IR3 should only be allocated 10–15% of effort and not more than 20% of funds. The lower levels of effort and funding allocated in the RFA for the social and behavior change communication and health systems strengthening activities that would be required to achieve IR2 and IR3 were based on USAID Guinea’s intentions that HSD would contribute only a small portion of these efforts and would be collaborating with other USAID investments to accomplish the desired results. Over the course of the HSD Activity, such investments in the social and behavior change Integrated Health Service Delivery in the Post-Ebola Context 13 EVALUATION METHODS AND LIMITATIONS OVERVIEW OF METHODS The evaluation employed a mixed-methods approach that included both primary and secondary data from a variety of sources. The team reviewed project reporting, implementation, and monitoring data provided by the implementing partner, collected primary mixed-methods data, and accessed existing national demographic and health systems data. We also conducted a review of national policy documents and the literature to contextualize and inform the evaluation findings. In addition, a case study was completed on the quality improvement process implemented by HSD, which used a realist evaluation approach for the analysis. Results from all data sources were compared in a multi-stage analysis process and triangulated for validation of key findings. Recommendations and lessons learned were drawn from the conclusions of the analysis of findings. Further review and validation of the results were provided by an external Strategic Reference Group. Primary data collection activities sought to supplement the quantitative data already reported by the implementing partner, focusing heavily on qualitative inquiry to provide a more nuanced and contextualized understanding of progress, challenges and lessons learned from the HSD activities. These activities were conducted at facilities selected purposively to represent the range of experience and geography, in some local communities, and with regional and national key stakeholders. PRIMARY DATA COLLECTION Primary qualitative data collection activities included in-depth interviews and focus group discussions. Primary quantitative data collection activities included facility checklists, provider surveys, and a data validation activity between facility-based data and health cards. Data collection instruments are presented in Annex III: Data Collection Instruments. Qualitative ▶ In-depth interviews: MOH staff, national-level partners/USAID, hospital department heads, providers,* regional and district health directors, key partners/managers, vocational school directors ▶ Focus Group Discussions: Non facility-based community members (women and men separately), women treated for genital fistula, GBV community groups, and COSAH representatives. Quantitative ▶ Facility checklist ▶ Provider survey* ▶ Individual health card data and facility-based data validation * Quantitative surveys were administered among all providers. Among select providers, an additional qualitative interview was implemented; COSAH (comité de santé et d’hygiène/health and hygiene committee), GBV (gender-based violence) MOH (Ministry of Health). communication space have included the Health Communication Capacity Collaborative (2012–17) and Breakthrough Action (US$2.5 million; 2018–20); and in the HSS space, Ebola funding for the Maternal and Child Survival Program health systems strengthening activities (US$2.75 million; 2016–17), and the Comprehensive Health Systems Strengthening Activity for Guinea (US$7 million; 2016–18). 14 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity SAMPLING Main evaluation sample: 10 hospitals and 26 health centers were purposively selected for inclusion in the primary data collection sample from among the 278 facilities within the seven regions where HSD activities were focused. Sampling criteria by type of facility is as follows: Ministry of Health hospitals • Regional representation • SBM-R performance (five high performing, and five low performing) • Three in highly populous districts, two in less populous districts • Variability in timing of entry into SBM-R program • Fistula repair services offered (n=3) Health centers • Regional representation • Activity performance indicators (DHIS2 and SBM-R), length of time participating in SBM-R (as relevant) location (urban and rural), and service volume • Facilities that have benefited from infrastructure improvements and receipt of equipment; Facilities with active community components (Community Mobilization Team, CHW/RECO) versus those without or with fewer such elements (a range of ‘intervention intensity’) SBM-R case study sample: Target: six facilities (four hospitals, two health centers), variability in SBM-R performance over time. Data Sources: key informant in-depth interviews, focus group discussions, evaluation of monitoring and service data, and document review. Analysis: Realist program theory specifies which mechanisms will generate the outcomes and which features of the context will affect whether or not those mechanisms operate. The context-mechanism-outcome configuration is used as the main structure for realist analysis. DATA COLLECTION Data collection and analysis was conducted in partnership with The Maferinyah Center for Training and Research in Rural Health (CNFRSR Maferinyah), a public research and training institute with extensive experience in implementation of research and evaluation methodologies, including quantitative and qualitative data collection. The team of data collectors was fluent in English, French, and local languages of each project region (Boké, Conakry, Kindia, Mamou, Faranah, Kankan and Labé). Most field data collection was completed in French and local languages (e.g., Fula, Malinké, Susu, Kissi, Kepelle, and Toma, per the geography of our selected facilities), although certain key stakeholder interviews were conducted in English. Field teams were composed of supervisors and data collectors who were trained, mentored, and overseen by the HEARD evaluation team. CNFRSR Maferinyah also supported logistics, translation, and transcription of data for the project. Data collection instruments are available in the Appendix. A total of 139 surveys, 70 data validations, and 196 in-depth interviews or focus group discussions were conducted (Table 1). Challenges prevented the field team from achieving all planned participant targets (see limitations below); however at least two staff were interviewed from each facility. DATA ANALYSIS Overview The analysis strategy for the mixed-methods evaluation included standard quantitative assessment such as comparison of pre/post status, assessment of trends and iterative qualitative thematic analysis. The evaluation was informed by a realist evaluation approach which seeks to understand the underlying generative mechanisms that explain intervention impacts and the influence of context.36 This approach was directly applied in the SBM-R case study. Quantitative data were described using means and standard deviations (SD) and medians and interquartile ranges (IQR) for continuous variables, and proportions Integrated Health Service Delivery in the Post-Ebola Context 15 Boké Conakry Faranah Kankan Kindia Labé Mamou National Total Done Planned Qualitative Provider KII 7 3 16 11 12 8 8 65 82 Facility Manager KII 9 5 0 4 7 3 3 31 36 CHW/RECO KII 2 2 4 3 3 4 4 22 22 COSAH KII/FGD 1 1 2 1 2 1 1 9 7 FGD Women 1 1 2 2 2 2 2 12 14 FGD Men 1 0 2 1 2 1 1 8 7 KII Fistula Survivors 2 0 3 0 0 3 0 8 9 KII/FGD GBV Group 1 1 0 0 0 1 0 3 6 KII Decision Makers/ partners 3 3 3 3 3 3 3 12 33 35 KII Professional School Managers 0 2 0 0 2 1 0 5 6 Subtotal 27 18 32 25 33 27 22 12 196 224 Quantitative Facility Checklist 6 5 3 9 6 3 3 35 36 Provider Survey 20 19 10 23 12 10 10 104 105 Data Validation 8 13 16 12 13 0 8 70 80 Subtotal 34 37 29 35 31 13 21 209 221 Table 1. HSD Guinea Evaluation Primary Data Collection Activities, by Respondent Type and Region. Notes: CHW/RECO (Community health worker), COSAH (Comité de santé et d’hygiène/Health and hygiene committee) FGD (Focus group discussion), GBV (gender-based violence) KII (key informant interview) for categorical variables. Trends in SBM-R scores across the timeline of the HSD Activity were estimated by category (i.e., infection prevention and control, emergency obstetric care, and family planning) using mixed effects linear regression modelling, to accommodate differential timing of participation in the SBM-R process and varied timing and number of measurements across facilities. Results describe the quarterly trend in SBM-R scores and are interpreted as the average quarterly change in SBM-R scores across participating facilities. Analysis of Demographic and Health Surveys data followed standard indicator definitions. Analysis of quantitative data was conducted using Stata v. 16 (College Station, Tx). Triangulation For each evaluation sub-question, the evaluation team reviewed, compared and contrasted the evidence from all available data sources to triangulate findings. Summary sub-question findings informed broader findings across evaluation questions and key recommendations through a tiered, multi-level analysis of the data by source, region, and evaluation question. Findings are summarized by question and sub-question, with illustrative quotes from respondents where possible. 16 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Strategy Reference Group A Strategy Reference Group (SRG) consisting of five individuals with strong contextual knowledge and who were previous end users of similar project evaluations was convened to critically review the findings compiled and analyzed by the evaluation team and to debate, prioritize, and strengthen the recommendations that emerged from the findings. SRG members provided written ratings and feedback on the report and preliminary recommendations. They participated in a group discussion to refine the interpretation of our findings and their translation into recommendations that are actionable and informative for USAID and which speak to USAID’s comparative advantage, strategic priorities, and contextual realities. ETHICS REVIEW The evaluation protocol which outlined the risks, benefits, and measures for privacy and confidentiality was approved by the Guinean National Committee for Health Research Ethics in November 2019. Verbal informed consent was obtained from respondents using an information sheet. Trained interviewers reviewed the information sheet with each participant, each of whom had an opportunity to ask questions before providing their verbal consent. Permission was granted for audio recording of key informant interviews and focus group discussions (FGDs). All interviews and FGDs were conducted in private locations where discussion could not be overheard. Participants were not asked to reveal their names during the discussions. Privacy was protected in several other ways: 1) no subject is identified in any report or publication; 2) all study materials and data collection forms are identified by type of health facility and role of the participant only; 3) notes from interviews are kept in locked offices and/or in a locked file cabinet or password protected computer; 4) data was analyzed collectively and individual participant data remains anonymous. 5) Password protection is assured and data is stored securely on password-protected computers and networks encrypted and maintained by the evaluation team; the key linking the health care delivery location to the interview will be destroyed at the end of study. LIMITATIONS We acknowledge the following limitations may have impacted the findings, interpretation of findings, and recommendations resulting from this evaluation: ▶ Time period: The data collection period was short, due to the need for findings to inform the current activity and strategy design, as well as external factors, including political demonstrations in Guinea, delayed national institutional review board clearance, and the end-of-year holiday period. Not all key stakeholders were available within the time period given for the evaluation, and not all data requested were able to be accessed within this timeframe. This may limit robustness of findings. This was mitigated as much as possible through increasing the number of data collection teams, reconfiguring the data collection and analysis procedures, conducting follow-up interviews, and reviewing secondary documents. ▶ Data quality: The evaluation considers data collected by HSD, government data from the DHIS2, the health management information system, and survey data as available. The quality of these data sources may pose a potential limitation given the ongoing challenges of timely, accurate, and reliable data. Our evaluation design sought to obtain multiple perspectives on each question to triangulate results to reach our assessment findings. Such an approach assists in uncovering truths where data do not align. Additionally, significant delays were faced in obtaining certain secondary and program data (e.g., DHIS2, facility-level program data, SBM-R assessments by domain) which reduced the possible depth of analysis. Furthermore, the evaluation did not independently assess health worker capacity and performance. ▶ Bias: Each type of data employed in this evaluation has particular strengths and limitations. Key informant interviews have limitations of the tenure of positions held. Frequent personnel transition may have impacted institutional memory within USAID, HSD, and other USAID implementing partners, which would be reflected in the findings of this report. Integrated Health Service Delivery in the Post-Ebola Context 17 ▶ Desirability bias: Despite the informed consent process and information given to participants before interviews and focus groups, the evaluation team often faced the challenge in field work of receiving the response that respondents perceived to be desired by the team, perhaps for fear of losing benefits received by the program, given that it was made clear that the activity is an evaluation. All data collectors were trained to probe beyond this bias; however, it remains a potential bias nevertheless, particularly among providers and health facility managers. ▶ SBM-R case study sites: The selected SBM-R sites for case study are few, therefore the findings from the participating sites may not be generalizable to all health facilities in Guinea, posing a limitation for external validity of findings around SBM-R country￾wide. Subdomain data for the facilities included in the case study were incomplete and received very late in the evaluation timeline, which limited the depth of analysis possible. Of the two non-SBM-R sites, one facility was not responsive to participating in the evaluation, thus we were unable to include their experience within our case study findings. ▶ Recruitment: Given the rapid data collection timeline and delay of institutional review board approval, the evaluation team was on a prompt schedule moving from site to site, and therefore encountered some challenges recruiting adequate numbers of participants from certain selected groups (e.g., women who have undergone genital fistula repair, participants in gender-based violence groups/ circles). Where this occurred, we sought to obtain data from these stakeholders through in-depth interviews instead. It was particularly challenging reaching informants in Conakry, as well as at the district, region, and director level. Given the short evaluation timeline, if informants were out of office, the data collection teams were not able to return for follow-up, although some were accessed by phone. ▶ Attribution: While efforts were made to seek attribution of changes specifically due to the HSD Activity, there were ultimately many overlapping interventions and partners working in the Guinea health and development sector during this period. We parse apart the different partner activities to the extent possible; however, specific attribution was not provided by some interviewees. ▶ HSD components: The HSD Activity comprised many components, some of which were small in scale investments or short duration. As this evaluation was oriented to the larger comprehensive focus, some components are not touched on in this report, such as the cervical cancer training intervention. 18 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity FINDINGS EVALUATION QUESTION 1 – QUALITY: To what extent has quality of health care services improved as a result of the HSD activity? SERVICE QUALITY 1.1. HSD activities contributed to developing a common national definition of an integrated RMNCH+ package that conformed to international standards through reviewing and revising multiple key national documents guiding the health sector in collaboration with the Guinean Ministry of Health, including the Essential Integrated Care Package, the Norms and Procedures for Reproductive Health, the National Road Map for Mortality Reduction, the National Strategic Plan for Obstetric Fistula, the adaptation to the FIGO Competency-based Fistula Surgery Training Curriculum, and the Monitoring and Evaluation Strategic Plan of the National Health and Development Program. Guiding principles are an essential building block of promoting consistent standards for quality care by all stakeholders, to establish service expectations, and to inform financial and technical support. HSD activities also included contributions to the development and validation of certain related training materials, job aids, and management tools. 1.2. HSD made substantial progress in strengthening the system and local actors to provide the full package of integrated services (Figure 4) through support for competency-based training and mentoring of health staff, equipping health facilities, and facilitating the development and implementation of clinical and managerial performance standards and protocols needed to improve the quality of care. Over FY16-FY19, the number of facilities providing the full integrated package increased from 0 to 227, with some facilities building on achievements of prior USAID-funded MCHIP and MCSP activities, and some facilities newly engaged within the HSD timeframe. The HSD Activity funding structure was not equal across years, with higher levels of funding provided during the first years of the project; project activities and inputs followed this pattern. HSD reported that 99% of service providers’ and community health workers’ performance assessed met national standards, and 77% of faculty and preceptors achieved 80% of clinical training performance standards in FY19. Key stakeholder perceptions of HSD efforts were largely extremely positive. Providers valued HSD’s supervision and peer evaluation approaches, the combined approach of training and equipment provision, and facility-based integration of changes by those who were trained. Providers surveyed largely felt that HSD had important impacts on care practices, with the majority agreeing ’absolutely’ that HSD trainings have ‘improved management of RMNCH+ services, approaches, and organization’ (94%), ‘workflow and provider time management’ (94%), and ‘provider skills for responding to client need’ (89%) (Table 14). Providers and facility managers identified large gains in emergency obstetric care, family planning and infection prevention and control, with less impact in management tools and nutrition. However, not all providers interviewed had undergone HSD trainings despite their facilities having been the focus of HSD activities, and others perceived that improvements in quality of care could not be attributable to the HSD Activity alone (see Q6). The need for continued training was identified by topic area (e.g., emergency obstetric care, infection prevention and control, support for community-based family planning service provision.) and in part, related to significant staff turnover. “The HSD project has put in place the [provider] training and community sensitization sessions that now attract the community to the facility. [This is because community members know] that family planning services are available and especially because [they no longer have] misconceptions regarding the consequences of family planning.” (KII Provider, Boké) Integrated Health Service Delivery in the Post-Ebola Context 19 “The project has supported health services in various areas including rehabilitating and equipping facilities, improving supply of drugs and commodities, staff training, formative supervision, and data collection and analysis. The project has rehabilitated and equipped the operating theatre; it has provided us with equipment (respirator, delivery kits, operating tables for maternity and surgery, delivery beds, hospital beds, vacuum extractors); it has trained health workers in emergency obstetric care, family planning, use of the partogram, and has improved the quality of cesarean sections.” (KII Provider, Kindia) “HSD helps us to work well, to fight against [stock] ruptures and to create trust between providers and clients.” (KII Provider, Kindia) HSD reports indicate that the number of targeted facilities offering the complete package of integrated care has increased from 0 to 227 over the life of the project, representing 82% of the 278 HSD-targeted facilities. Providers and facility managers also generally report availability of the integrated care package, with 89% of providers surveyed reporting that integrated services are offered every day at their facility. However, despite these important reported achievements, further support is needed to ensure that the full integrated care package is consistently available at health facilities. For example, data validation identified low continuity of perinatal care, with few women (11.4%) achieving three key components: the recommended number of antenatal care visits, facility delivery and postpartum care; women’s self-report of perinatal care continuum achievement from the 2018 DHS ranged from 15.2% to 44.6% across region (Table 16). Data also revealed important disparities in perinatal care receipt by sociodemographic characteristics (Table 16), emphasizing the need for focus on equity. Other inconsistencies in the provision of essential services were noted (see 1.3, 1.4, 1.5 for detail), including the low availability of fistula and GBV services (see Evaluation questions 4 & 5 below). Challenges to rolling out the complete integrated package in areas targeted by HSD included lack of managerial tools (e.g., forms and registers), human resource deployment and retention, lack of essential medicines and commodities, lack of adequate space for consultation, and inadequate equipment functioning and maintenance, including laboratories. These challenges are described in more detail in Q3. “These services are not available because the health center is limited in equipment and providers.” (KII Provider, Kankan) 1.3 Both access and quality in family planning services increased during the course of HSD activities. However, some persisting challenges affecting the provision of quality care were evident, including contraceptive availability and lack of private and welcoming counseling space. Across HSD-supported facilities, family planning service availability increased from 51% in FY16 to 98% in FY19, and a total of 999 providers and 731 community health workers underwent HSD-supported training in family planning.37 Quality improvement efforts are resulting in tangible change, with the composite family planning assessment score in SBM-R facilities increasing an average of 0.9e points (95% CI 0.06-1.0, p<0.001) per quarter across the project timeline at the time o. At selected facility observations (Table 12), most had family planning counselors designated and available (89%) and good availability of a range of methods (implant 98%, condom 90%, injectable 90%, combined oral contraceptive pill 90%, and intrauterine device 81%; 250 200 150 100 50 0 Fiscal Year FY16 FY17 FY18 FY19 59 118 79 215 227 177 225 Number Target Achieved Figure 4. Facilities providing the full package of integrated services, FY16–FY19. Source: Monitoring data, Jhpiego 20 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Figure 5). Most of selected facilities reported offering all modern methods expected at their level (77%; 76% health centers and 80% of hospitals); however, some methods were reported to be currently out of stock including IUD (n=3), implant (n=2), and the progestin￾only contraceptive pill (n=1). Facilities were found to maintain information, education and communication materials for family planning counseling (100%), and most had job aids to assist with counseling (94%). However, in observed facilities, the family planning counseling space was observed to be clean and welcoming in only 54% of facilities, and private and confidential in only 60% (Table 12). Indeed, lack of space for family planning counseling was noted as a challenge by some providers and facility managers in interviews. Finally, interviewees frequently cited contraceptive stock outs as a limiting factor to providing contraceptive products, and several facility managers shared their frustration with the pharmaceutical request and fulfillment process resulting in delayed stock replenishment from central levels, particularly for busier facilities. HSD implementing partners acknowledged this challenge and advocated for improved processes to health leadership, however HSD was not designed to intervene directly in supply chain issues (refer to section 3.2). Figure 5. Proportion of Providers Surveyed Indicating Contraceptive Method Available at Their Facility, Dec 2019 Proportion of Providers Implant Condom Injectable OCP, combined IUD OCP, progestin only LAM Sterilization 0 20 40 60 80 100 1.4. Quality of antenatal care was considered to have improved, although not all essential components are consistently available. Project monitoring data report 74% of pregnant women in project areas achieved the recommended 4 antenatal care visits (including one consultation at 9 months of pregnancy), and HSD implementing partners acknowledge that additional efforts are needed for improvement.37 Most of the selected facilities surveyed offered antenatal care services (89%) within a clean and welcoming room (83%); however, this room was private and confidential in only two-thirds of facilities (66%) (Table 12). All facilities had job aids for antenatal care use, but breastfeeding support was available at 69% of the facilities Providers surveyed reported the following at all antenatal care visits: blood pressure (81%), weight (88%) folic acid supplementation (83%), iron supplementation (78%) (Table 14). Less than one￾third reported urinalysis at each antenatal care visit (36%). Glucose screening was also less commonly available (57%). “I have noticed changes in the promotion of health care use through community awareness campaigns. The distribution of medicines, the number of women attending antenatal care, and child vaccination have increased.” – (Female community member, Boké) 1.5. Quality of delivery care has improved, although opportunities for further improvement was evident, particularly by increasing utilization of facility delivery, partogram use, and postpartum care. From FY17 through FY19, facility delivery increased from 42% to 45%.37 HSD’s dual strategy of strengthening health systems while conducting community educational outreach was identified as an important facilitator of increased utilization of facility delivery within previously low-functioning health centers following the implementation of community dialogues and open houses. Increased provider capacity was facilitated through training in emergency obstetric care of 1,175 individuals over the four years of HSD activities assessed in this report. Quality improvement interventions such as the SBM-R Integrated Health Service Delivery in the Post-Ebola Context 21 care, and infection prevention and control domain scores across time; however, maintenance of the improvements require significant ongoing commitment from facility staff and managers. Achieving thresholds for recognition motivated staff to maintain progress and work collectively towards achieving new stars through following-up on SBM-R requirements put forward. Teamwork and respect for service provision norms and standards was understood as essential to achieve stars. Pride, determination, and a common vision were needed to be successful. Getting a star was considered a big achievement, and was contributed to through strong leadership, team work, staff cohesion, and community support. However, cultivating further support from staff, facility manager and important district and regional Ministry of Health managers will be needed to expand and sustain the success of the SBM-R process given quality improvement is a continuous process that must be maintained by the facilities themselves with the support of Regional Health Directorate (DRS) and Prefectural Health Directorate (DPS) supervision and mentoring. Further discussion of the role of the SBM-R process in improving quality of care and demand for services is elaborated within Q2 and Q3 and the SBM-R Case Study. “The HSD project has contributed to improving the quality of patient care and providing formative supervision. When we launch orders immediately, we receive them thanks to the support of the project, which itself provides us with the equipment we need.” (KII Facility Manager, Boké) “Through the use of the SBM-R tool, the quality of care and the achievement of performance standards has improved.” (KII Facility Manager, Boké) process documented improvements in quality of care provided with emergency obstetric care quality score increasing significantly over time by 1.5 percentage points (95% CI 1.2-1.7, p<0.001) per quarter on average across the project timeline. HSD monitoring data also evidence increases from FY17-FY19 in partograph use (48%-61%) and uterotonic administration (92%-99%) within included facilities. These data are consistent with selected provider survey results (Table 14), where most reported implementing selected best practices for delivery and early postpartum care. For example, 94% reported immediate oxytocin administration following delivery, and 91% immediate mother to child skin-to￾skin contact, although there was some variability in the proportion of births in the prior week with immediate skin-to-skin contact (mean 61.2, SD 36.8). Similarly, most providers reported supporting breastfeeding (89%), including helping women with positioning and placement. Postpartum family planning counseling was reported to always occur by 83% of selected providers surveyed. Within the data validation procedure, 72% of selected women delivering in the prior year had evidence of partogram used. HSD integrated principles of respectful maternity care into emergency obstetric and neonatal care standards starting in 2017. Finally, the implementation of kangaroo mother care for premature or low birth weight infants within 9 HSD-targeted facilities in conjunction with antenatal corticosteroid provision was felt to be a success of the program by key stakeholders and scalable nationwide at low cost. However, the referral system (Section 2.2) was considered to be a weakness, and posed challenges to emergency care access. 1.6. As described within each of the major maternal health topics above and discussed further within the dedicated case study, the SBM-R quality improvement process was an important driver of improvements in care quality across the integrated care services, with the star system widely recognized as representative of high-quality services to health systems staff and the community. Facilities participating in the SBM-R process largely improved family planning, emergency obstetric 22 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity EVALUATION QUESTION 2 – COMMUNITY FOCUSED INTERVENTIONS: To what extent did HSD contribute to improved access and use of the essential integrated care package of health services at targeted facilities? 2.1 Demand for quality services: Utilization Utilization has increased throughout the continuum of care, especially for antenatal care, family planning and child health. Facility delivery and postpartum care have shown slower uptake, and important disparities and equity gaps remain. While the comparison of maternal and child health indicators from the 2012 and 2018 Demographic and Health Surveys is complicated by the impact of the Ebola virus disease outbreak, overall improvements in demand are seen through increases in key maternal and child health indicators such as: family planning use, antenatal care, skilled attendance at delivery, postnatal care, and child health (Table 2; please see Table 12 and Table 13 for more detail). However, antenatal care still remains the most attended with over 80% receiving at least one antenatal care visit. Family planning and postnatal care 2012 2018 % change Demand for family planning satisfied by modern methods 15.8 32.1 +16.3 Recommended number of ANC visits (4+) 56.6 35.5 -21.1 Antenatal care from a skilled provider 85.4 81.7 -3.7 Place of delivery: Health facility 41.0 54.5 +13.5 Assistance during delivery from a skilled provider 41.8 57.5 +15.7 Postnatal visit for women in first 2 days after birth 36.7 48.6 +11.9 Received all 8 basic vaccinations 36.5 23.9 -12.6 Treatment of diarrhea: Either ORS or RHF 36.2 58.3 +22.1 Table 2. Demand for Health Care Services, Guinea Demographic and Health Survey, 2012–2018 Notes: ANC (antenatal care), ORS (oral rehydration salts), RHF (recommended home fluids). are attended by under 50% of women, though just over half delivered in a facility. HSD program monitoring data shows more recent increases in service utilization, as a result of specific activities involving the COSAH and community dialogues in participating facilities, as shown in Table 3. Large increases were seen in the overall number of deliveries, family planning, and antenatal care visits per reporting period at health centers in Kankan, Kindia, Mamou, and Faranah (Table 4). Many respondents perceived that there was an increased demand for health services in the post￾Ebola period, attributing it to the HSD Activity (Table 5). Providers and community members noted the cycle of better services resulted in increased trust and demand. As HSD helped improve provider capacity through training, better-equipped and supplied facilities, and increased quality and cleanliness of facilities, community member trust in the health system increased. Increases in utilization were also attributed to CHW/RECO activity, increased awareness, and better-quality services cycle of improving demand. However, some providers still noted the challenges of mistrust in health providers and facilities since Ebola still impeded utilization. As shown previously in finding 1.2, increased use of individual services does not necessarily translate into Integrated Health Service Delivery in the Post-Ebola Context 23 Table 3. Health Center attendance rates in 5 sites receiving GAC/COSAH supervision, July – September 2018 vs July – September 2019 Health Center July-Sept 2018 July-Sept 2019 % Increase CS Banfèlè 793 2,185 176 CS Doko 7,935 10,108 27 CS Kantoumanina 2,149 2,767 29 CS de Kossankoro 591 748 27 CS Sibiribaro 1,213 1,690 39 CS de Sogolon 1,086 1,591 47 Total 13,767 19,089 39 Notes: ANC (antenatal care), ORS (oral rehydration salts), RHF (recommended home fluids). Table 4. Health care seeking at health centers before and after the implementation of the community dialogues and open houses per region, October 2018 – September 2019 Region and number of health centers Assisted deliveries Family planning consultations Prenatal consultations Before After %Diff. Before After %Diff. Before After % Diff. Kankan (n=8) 1,016 2,712 167 1,068 9,230 764 1,302 1,575 21 Kindia (n=6) 108 136 26 N/A N/A N/A 971 1,182 22 Mamou (n=3) 60 112 87 187 236 26 148 292 97 Boké (n=7) 175 130 -26 218 139 -36 179 182 2 Faranah (n=8) 309 384 24 612 824 35 2,506 2,875 15 Total 1,668 3,474 108 2,085 10,429 400 5,106 6,106 20 Source: Jhpiego Annual Report PY4 increased uptake along the continuum of care. (Data Validation Findings) As shown in DHS and MICS survey data, attending any ANC visits is much more common than four ANC visits, and the rest of the recommended intrapartum care contact points (See Table 12 and Table 13 in Annex). Within the existing community health work, the emphasis for CHW/RECO referral of pregnant women to ANC is prominent, but there is a lack of tools and processes for ensuring follow up, as described by one COSAH member below. He explains that even though facilities have outreach teams that recruit women for ANC, women often “disappear”, discontinuing after registering, and do not always achieve the recommended four antenatal care visits or facility delivery: “In the antenatal care service, you’ll see women who register and follow the treatment up to a certain point. Then they disappear; [the facility has] outreach teams and the doors are open to everyone, but attendance is lacking.” (COSAH, Mamou) This finding is corroborated by data from the most recent Guinea Demographic and Health Survey (Table 12) which reveal incomplete antenatal care attendance rates. Challenges still remain for access and utilization of services, such as geographic proximity of facilities, / poor roads, lack of transportation, finances, lack of information, cultural and religious beliefs, including 24 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity rumors and need for male permission, especially for family planning adoption (Figure 6; Table 12). (Provider Survey, KII Providers/CHW/RECO, FGD men/women) The intermediate results of improved demand and access are further examined through the specific strategies and interventions implemented by HSD, namely community-based health services, referrals, social and behavior change communication increasing community awareness, and addressing financial barriers. 2.2 Community-based health services HSD provided strong technical support for policy documents and training materials for quality, integrated, scaled community health activities, yet coordination and financing for operationalization of the policy remains insufficient. HSD’s original approach included ensuring that two community health workers (CHW/RECO) per village, one male and one female, were competently providing the community￾based components of the essential integrated care package in the regions covered by HSD. HSD aimed to reach at least 2,400 villages, estimated on the basis of approximately eight villages per rural facility and five neighborhoods per urban facility in the regions covered. HSD provided technical support to the Ministry of Health and the Community Health Division through standardized training materials, policies, and protocols. It developed the skills of a pool of trainers, and ensured supplies and commodities necessary for delivery of community interventions. The theory of change aimed to improve quality, integration, scale, targeting, and coordination. As the scope of HSD was limited, it did not aim to fund CHW/RECO at national scale which was well beyond the scope of the activity. Guinea includes nearly 16,000 villages and the revised Community Health Policy requires 1,320 CHW and 17,000 RECO. HSD’s support to the development of the national Norms & Procedures of Reproductive Health document for the essential integrated care package was instrumental to guide how integrated services should be delivered throughout the continuum of care, including community￾based care components. The Norms & Procedures of Reproductive Health included a strong training and job aid package for cascading integrated care down to the community and health post levels. While the majority of Provider Perceptions of Utilization Community Perceptions of Utilization n “Before the target of the structure was [about 56] women per month (for ANC visits) but now it is 70 to 100 per month” (Provider, Mamou) n “[Due to the educational activities conducted by CHW/RECO, women are] coming to the facilities more frequently for MCH services including FP, delivery, and cesarean sections, and maternal deaths have reduced. For example, 14 maternal deaths in the first half of 2019 compared to only six in the second half of 2019.” (Provider, Faranah) n “Each day, the consultation can reach up to 100 children at a time...I have to hide to go home.” (Provider, Kindia) n “Thanks to the project, through awareness raising, the fear that had been driving women not to use health services since the days of Ebola has disappeared. Trust between health providers and the community has been restored. The COSAHs take some of the credit for this improvement.” (COSAH Kindia) n “Many women did not attend the center [previously] due to the lack of qualified staff, but currently they attend the center a little more, especially on Mondays. Women prefer to deliver with traditional birth attendants and they prefer to send children to a private clinic or to Kankan or Siguiri Hospital.” (FGD Men, Kankan) n “Service availability has played a big role in use. Since the advent of the Ebola epidemic, the use of services has fallen sharply, but with sector-by-sector awareness at the community level, more and more women are now coming to the health structures in search of care.” (CHW/RECO, Conakry) n “For young people, information goes hand in hand with awareness and since the project’s arrival, young girls have been using contraceptive services. Before, young people were ashamed, but now there is a good attendance.” (FGD Women, Kindia) Table 5. Provider and Community Perceptions of Health Care Utilization Integrated Health Service Delivery in the Post-Ebola Context 25 100 90 80 70 60 50 40 30 20 10 0 Do not require spouse consent for FP Proportion of Facilities Clients rights posted No User Fees for RMNCH+ on the wall Patient Access Characteristics CS/CMC Hospital Total Figure 6. Factors in the access and utilization of health care facilities HSD trainings targeted health center and hospital staff, over PY2-3, 376 health post staff were trained to reach individuals at the community level. Even though HSD did not fund CHW/RECO at scale, qualitative results demonstrate community-based health services are functioning in some cases. “The community no longer needs to come to the hospital for simple cases such as simple cough, diarrhea, short-acting family planning.” (COSAH, Kindia) “Community services are of good quality because the CHW/RECO support the community by sometimes giving free medicines to the sick who have no money.” (Women, Kindia), Qualitative data, however, revealed that CHW/RECO are delivering inconsistent prevention, treatment, and referral services, varying by region, and often reflecting the respective donor priorities and trainings. For example, community members in Kankan were aware of vaccination days as the sole community￾level intervention; however, other evidence showed that CHW/RECO are delivering other community-level services as recommended in the Norms & Procedures of Reproductive Health, though they often lacked the supplies. Additionally, child health, antenatal care, and family planning activities often are delivered vertically rather than in an integrated approach, as CHW/RECO have historically received support from differing donors to carry out targeted activities, such as vaccination days and distribution of vitamin A and mosquito nets. Further complicating the CHW/RECO landscape, the uneven roll out of the new CHW/RECO strategy whereby CHW/ RECO are paid for services that they previously provided for free has discouraged CHW/RECOs from working in zones where the supporting donor partner did not provide salary payments. “At the Labé regional hospital, community promotion activities have been reserved for the Projet d’Amélioration des Services de Santé Primaires project. The HSD project only intervened in awareness activities in hair salons by peer educators.” (KII Provider, Labé) “We need to recruit the very social people for these tasks of responsibility. They’re not doing their roles; I don’t know CHW/RECO in our community. The obstacle is related to the lack of financial means.” (FGD men, Boké) 26 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Jhpiego and Save the Children also provided facilitative support for technical workshops to input into the new Community Health Policy adopted by the Ministry of Health in 2018, beginning implementation in 2019. The Community Health Policy specifies guidance for coverage and intensity of CHW/RECO per population requires 1,320 CHW/RECO and 17,000 RECO for achieving national scale. In addition, CHW/RECO must have undergone specific training, and receive a small monthly payment. At the regional level, HSD coordinated with partners to identify gaps in coverage of CHW/RECO. CHW/RECO s expanded the reach of health promotion to promote use of services; though it is difficult to distinguish activities supported by HSD or other community-level partners as they all use the same CHW/RECO. For example, in Labé and Faranah, the World Bank/AFD-funded community health project funds all CHW/RECO and was appreciated by providers. The continuity of support was also questioned, as noted below, the support was “periodic” rather than ongoing, and some community members were not aware of CHW/RECO in their community. While the Norms & Procedures of Reproductive Health and Community Health Policy were key policy guidance accomplishments, the infrastructure and human resources necessary to deliver the integrated community health package remain a barrier. It was beyond the original scope of HSD to fully address this; nevertheless, findings reveal the situation at large. Sustainability for financing is questionable given its resource intensity. Respondents were clear that success of the new community health policy requires the government to strengthen the capacities of local collectives (L040 law on code of collectivities) through the Agence Nationale de Financement des Collectivités (ANAFIC), which receives 15% of national mining revenues. It was stated that ANAFIC has agreed that the communes will include budget lines from their 2020 Action Plan for paying the salaries of RECO. Without this decentralized support, it is unclear how the new Strategy can be implemented. Nevertheless, this new policy is the guiding model for community level service delivery and required HSD to shift their original approach. In the first years of the program, community facilitators from local non￾governmental organizations worked with the CHW/ RECO to support timely reporting, refresher skills for delivering health talks, and following protocols. Each year from 2017-2019, HSD supported over 1,000 CHW/ RECO, and conducted training for 602 community-level participants. However, with introduction of the new policy, it was not guaranteed that any of the prior CHW/ RECO would be selected to continue as government CHW/RECO due to the new educational requirements. While HSD could not ensure the people they trained continued to be engaged, they continued at the policy level to apply lessons learned from their small-scale experience. Therefore, HSD community level support shifted to supervision, monitoring, and coordination of partner support for CHW/RECO. 2.3 Referrals: strengthened integration of clinical and outreach services HSD aimed to strengthen referral linkages by ensuring coordination of care from the community to the hospital level, as well as improving transportation systems and monitoring and evaluation of the referral system. Table 6 shows project design of referral interventions at each level, with key accomplishments and existing gaps. HSD provided key contributions to some components of the referral system, but the scope of the referral system challenges was unable to be overcome with the level of investment. Currently the referral system in Guinea remains largely dysfunctional. While the N&P/RH were validated in year one, the normative documents for the referral system were not validated nationally until June 2019, due to transitions within the Ministry of Health and competing priorities. At the time of this evaluation, these normative referral documents have not yet been disseminated systematically across Guinea (Document review). While the contribution of HSD to these documents was strong, because their dissemination is incomplete, the referral system is still Integrated Health Service Delivery in the Post-Ebola Context 27 Level Plan Accomplishments Gaps Prefectural Support teams to manage/maintain referral systems; plan for ambulance use/ maintenance HSD contributed to normative documents validated in June 2019; HSD supported and monitored provider phone usage. Some evidence suggested referrals were taking place but functionality was unclear. Many regional/district stakeholders report the system is in place and some training has been done, but challenges remain (e.g. in ambulance use and maintenance) and enforcement is lacking Facility Develop/adapt tools/job aids; train providers Facilities report tools in place: 83% of facilities surveyed (n=35) have a register in use at antenatal care point for tracking referrals of complicated cases to a higher level; Many CHW/RECO have been trained on referral and tools are in place. Sometimes health posts refer directly to hospitals and skip the health center level. Community Train CHW/RECO to refer; enhance local planning Referral registers and cards are largely in place; referrals were primarily for ANC or severe child illness; Mamou has functional motorcycle referrals according to young women’s focus group Respondents and families were not referred in most cases; transportation was a primary challenge; CHW/RECO note reaching more remote households is a challenge COSAH Support local means for transportation Some COSAH spoke of examples of securing transportation for referrals Not standardized across all community groups; most community members still mentioned transportation challenges M&E Mapping of deaths/ near-misses; ongoing identification of weaknesses/M&E Monitoring of telephone usage Lack of comprehensive M&E system for all levels of referral system Table 6. HSD Referral Improvement Plan, Accomplishments, and Gaps, by Level not functioning well varying by district, facility, and health domain. HSD’s efforts in this domain focused on providing telephone network and monitoring the telephone usage of providers to follow frequency of referrals. While some tools and processes are in place, especially for severe child illnesses through community integrated management of newborn and child illness (c-IMNCI) interventions, overall the tools and processes are not harmonized or monitored, and challenges exist from community to hospital. As shown in Table 6, gaps exist largely in transportation and vehicle availability and ongoing monitoring and evaluation of the system and expansion of referral activities to these areas is likely to improve the system. This is also due in part to the multiple levels of referral possible and the costs associated. While it is perceived that CHW/RECOs are doing well at referring clients to facilities for ANC or child illness, the linkages and follow up for complicated illnesses and pregnancies are lacking at various levels. Some respondents did note some successes of the referral system: “The referral process of women to these services is carried out through referral cards and this process is efficient through the use of motorcycle taxis.” (FGD Young women, Mamou) “Referrals are always made with a community health worker and a family member who accompanies plus a referral form that allows the patient’s state of health to be understood once the patient is referred to an appropriate service. The RECOs play a major role in the reference system through a reference sheet that they have at their disposal. They also accompany the women to the health center for their care.” (Provider, Kindia) 28 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity However, challenges to referral were commonly cited by community members and community health workers: “We cannot tie a woman in labor behind the motorbike to bring her to the health center like other patients; we need a vehicle.” (FGD Women, Kindia) “The community health workers do not make referrals nor do they direct women to health facilities. It is our neighbors who send us to the hospital.” (FGD Women, Conakry) A regional-level stakeholder also had insight into the dysfunctional referral system gap as a key challenge to improving utilization of the health system and retention in care: “Well, we have a lot of problems at [the referral] level; … even if the RECO gives the referral to clients to go to a health center, sometimes, there is a loss. If the person is not well oriented, then he goes directly to the [traditional] healers or someone who sells drugs in the village and he will self-medicate. Even if the health center providers receive the referrals, they do not make a counter referral to the peripheral structure that sent the referral.” (KII Regional) The “counter referral” gap sheds light on the lack of communication from the various levels of health structures. Patients may only be sent up to the next level and records are not shared back to the home structure for continuity of care. 2.4 Social and behavioral change communication HSD’s social and behavior change communication approach was based on increasing community awareness about the need to access care, through strengthening community health groups, local NGOs, and community animators to complement the work of CHW/RECO. While the HC3 project had the main responsibility of implementing social and behavior change communication activities in Guinea,† HSD’s small-scale efforts working with community groups and community animators had mixed results and required frequent supervision from HSD staff to maintain the small-scale engagement with the community. While the project design aimed to mobilize beneficiaries to be promoters for their own health, this has not been widespread in project targeted areas except where community members are engaged in health committees or COSAHs. COSAH and/or community action groups exist at many HSD-supported facilities, expanding from 18 facilities in PY2 to 47 functional at the time of evaluation in November 2019 (PY5). The number of community mobilization teams increased from 18 in PY3 to 24 in November 2019 (PY5). Save the Children provided technical assistance for facilitating a training of trainers employing a “community action cycle” approach to district health officials which was intended to cascade down to community groups. The community action group approach fosters a community-lead process through which those most affected by and interested organize, explore, set priorities, plan and act collectively for improved health. HSD worked with the health and hygiene committees (COSAH), community action groups, and community mobilization teams to encourage the community action cycle approach for community mobilization. As shown in 2.1, supervision of the facilities with active community action groups and community dialogues showed increased facility attendance in the respective facilities. Respondents also spoke to the impact of COSAH. Through COSAH engagement, some community groups got involved in supporting their local facility to try and achieve SBM-R stars for quality improvement. In Kindia, for example, the community group supported quality improvement by helping to clean the compound of the facility each week which contributed to the facility receiving a star. † USAID funded demand creation and social and behavior change communication (SBCC) activities separately through the HC3 project. HSD planned a supporting technical assistance role in demand creation, and per the HSD RFA, HSD could allocate a maximum of 20% of its budget to IR2 and IR3 activities. HSD partnership with HC3 was to “ensure coordination, particularly in geographic scope, and best use of resources”. HSD did not have the resources to fund community-level and SBCC activities at scale. Integrated Health Service Delivery in the Post-Ebola Context 29 “It is the neighborhood chief who gave us this and that for the renovation of this health center and even the painting you see was done thanks to the contribution of a community member.” (FGD CHW/RECO Kankan) “The last time there was an outbreak of measles somewhere, it was [a COSAH member] who tipped us off.” (Facility Manager, Conakry) “That’s what we members of the hygiene committee are here for. We make sure that information is passed on whenever necessary through the RECOs and town criers, in the mosques and during market days. Women are well-informed through sensitization, word of mouth information through the media, and this information has engaged them even more about their health care needs since the intervention of the HSD project.” (COSAH, Kindia) Provider survey results demonstrate (Figure 7; Table 14) that providers at the health center level (n=46; hospitals excluded) primarily understand the role of COSAH to be awareness raising, as well as promoting communication between community and providers. However, the role of the COSAH in the new Community Health Policy has evolved to serve more as an extension of the health system than a catalyst for community representation or engagement. The role of COSAH, according to the Community Health Policy, is to provide supervision for community health activities within the commune-level government structure. These duties RMNCH+ Awareness raising/education activities for community Promote communication between community and providers Support evaluation of services by community Develop and implement plans for health promotion Other 0 10 20 30 40 50 60 70 80 90 100 Percent Figure 7. Provider Perceptions of Role of COSAH (n=46) include monthly supervision and evaluation of CHW/ RECO activities/interventions, ensuring recruitment of CHW/RECO and verifying they receive payment, as well as ensuring resource mobilization for community health activities. Likewise, some respondents felt the COSAHs are too close to the health system and cannot be not truly representative of community members. Yet it appears most providers have not yet been sensitized to the new role of COSAH. As additional social and behavioral change communication actors, HSD also engaged local non-governmental organizations in project years one through three to assist in the recruitment and management of CHW/RECO to conduct health talks and social mobilization, as well as strengthen capacity of the non-governmental organizations for grant management. In project year one, the CHW/ RECO conducted 337 health talks reaching 2,821 people within seven districts. Health talks can play an important role in social and behavioral change communication, and there were few examples from community members of engaging religious leaders and women’s groups as essential to the success in sensitization and behavior change: “Family planning before was a taboo subject that they didn’t dare to mention. I myself have attended several regional meetings where the leaders of the Islamic league have participated. Since then, awareness has increased and now you can talk about family planning.” (FGD men, Labé) 30 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity “There was awareness-raising of “serious” women’s groups. They were sensitized and asked to sensitize their relatives and close friends in turn. That’s how everything changed…now people say that Ratoma has changed. After that, the health workers also were made aware of how to address the patients, how to keep the center clean… This project has really supported us. The community has started coming to the health center in large numbers because they see the realities with their own eyes.” (FGD COSAH, Conakry) Most community members remain beneficiaries of community interventions and are not health promoters, unless they are themselves CHW/RECO or engaged as leaders in group social and behavior change communication activities. By project year three, the local non-governmental organization-led efforts were discontinued by HSD. This discontinuation was due to various factors: need to follow the new Community Health Policy; some weaknesses in management and reporting by the non￾governmental organizations; and the funding pattern of HSD, where project years four and five had less funds allocated as Ebola-related funds expired. Finally, HSD’s primary urban peer education social and behavior change communication approach leveraged hair salon employees and tailors as peer educators and family planning vendors to complement CHW/RECO (Table 7). This was a novel approach for reaching those who may not ordinarily come across CHW/RECO or frequent health services. While many health talks were conducted in this environment, the approach faced challenges to its success. Successful education and distribution of family planning in this context requires heavy ongoing supervision and monitoring that HSD was not consistently able to provide beyond training. This was especially noted in Conakry, where referrals to health facilities were not always done correctly by the hair salon employees. (KII Regional) While this approach was successful in helping to expand services to new family planning users, the incomplete linkage to health facilities for continuum of care or long-term methods was inconsistent and varying by region. 2.5 Addressing financial barriers to care HSD aimed to address financial barriers through supporting development of community health financing strategies such as community insurance Quarter Hair salons Tailoring shops Peer educators supervised Peer educators trained Educational talks Clients reached Women reached 1 15 9 40 75 147 1,101 1,033 2 38 15 92 231 247 1,704 1,425 3 38 13 119 255 227 2,728 2,082 4 38 16 138 266 211 2,046 1,703 Total 389 827 832 7,579 6,243 Quarter New FP users Continuing FP users Referrals 1 330 448 55 2 484 466 83 3 459 731 203 4 2296 503 161 Total 3,569 2,148 502 Table 7. Outputs and reach of hair salon and tailor shops, October 2018 – September 2019 Notes: FP (family planning). Integrated Health Service Delivery in the Post-Ebola Context 31 and mutual schemes. Despite some efforts, cost of RMNCH+ services remains a barrier to utilization, especially for the most vulnerable and those in the lowest socio-economic strata, in addition to other barriers discussed in 2.6. The 2010 user fee exemption policy from the Ministry of Health dictates that antenatal care and institutional delivery including cesarean section are free; however, family planning is subsidized and there are also often other costs for medicines or lab tests that deter women from attending facilities. 37% of facilities observed reported charging user fees for RMNCH+ services, and this was even higher among hospitals (50% reported user fees as shown in Figure 6). “If you go to the hospital without money, you die. You can spend up to 100,000 GNF on counseling and medication…we wanted it to be free.” (FGD Young Women, Conakry) 63% of 35 facilities surveyed had clients’ rights posted on the walls, which was even higher among hospitals (80%). As shown above left, these were written in a small font in French, and focus on privacy, confidentiality, security and choice, but do not include free services. The poster most commonly found was printed by the USAID-supported Maternal and Child Survival program, also led by Jhpiego, between 2014 and 2016. In order to avoid duplication, materials SUCCESS: HEALTH MUTUALS ➤ Health mutuals are community-based schemes to bring together community resources to help community members in need of funds for health services. ➤ HSD identified 195 health mutuals, with 23% functioning. Regional meetings were carried out to share guidance to strengthen the groups that support community members in need of healthcare funds. Tools for membership and meeting minutes were also provided to some mutuals for reinvigoration. HSD also provided assistance to the French non-governmental organization Solidarité Internationale, and the main partner that supports health mutuals, in developing performance standards. ➤ Interviews and focus groups with COSAH, CHW/RECO, and community members revealed that all regions had community will and acceptance to participate in such schemes to address financial barriers, including community and religious leaders support for the schemes. This momentum shows promise for expansion and scalability. Left; A clients’ rights poster displayed at a health center Above and right; Pricing and services signs inside and outside health clinics 32 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity produced during the Maternal and Child Survival program appropriate to HSD’s work continued to be distributed. Another poster found at many sites was the public listing of contraceptive prices by method and listing of free antenatal care services. While this helps to increase transparency and raise awareness, it does not tackle the barrier of related services requiring payment or target illiterate populations. 2.6 Increasing community awareness about need to access care The above strategies were for increasing community awareness about the need to access care to increase demand and utilization. Despite the accomplishments noted above, community interventions did not have a strong equity focus and barriers remain for women, youth, and the most vulnerable to access RMNCH+ services. In addition to financial barriers, another key barrier for the most vulnerable is lack of information about health services and illiteracy. Awareness raising/sensitization activities by CHW/ RECO and radio messages addressed this barrier in some ways. However, findings revealed demand challenges remain, such as distrust of providers from the EVD period: “So far we remember what people were doing during Ebola, that’s why some of us are still afraid.” (Woman, Kindia) Barriers are often tied to poverty, unemployment, and an inability to pay for services or transportation. (FGDs men/women, FGD COSAH) “Young people and adolescents face difficulties such as unemployment and poverty. Youth employment and assistance from friends are the measures at their level.” (FGD Woman, Mamou) A key barrier for family planning is spousal consent and 20% of facilities require spousal consent for family planning (Figure 9). This corroborates findings from the DHS 2018, where 30% of women reported permission as a barrier to accessing health care. “My husband is Muslim and doesn’t even like to be told about family planning... to plan for himself is to kill the children that will be born. For young people, information goes hand-in-hand with awareness and since the project’s arrival, young girls have Provider Perspectives Community Perspectives “When a young girl is seen around health facilities, she is considered to be sexually active. The barriers faced by young people in the locality are fear and shame of being seen in planning services and fear of being discriminated against. To this end, sensitization is carried out during market days to address these barriers.” (Provider, Labé) “Young men prefer to be seen by same-sex health workers, whereas the majority of health workers are female, and young veiled women also prefer female health workers, all for reasons of modesty. Out of 24 health workers there are only 3 men.” (Facility Manager, Conakry) “As for young people, there are sometimes difficulties, such as the lack of communication about sexual health to young people and the lack of special services for adolescents because young people are often ashamed to be with their parents in the institution.” (Facility Manager, Kankan) “As for young people, they sometimes face misinformation and early marriage. The measure taken in this regard is sensitization.” (Provider, Kankan) “The only obstacle our young people encounter is the lack of understanding between them and the doctors.” (FGD Men, Faranah) “Young people and adolescents face difficulties such as unemployment and poverty. Youth employment and assistance from friends are the measures [that can be taken to address barriers] at their level”. (Woman, Mamou) “The specific barriers that women in our community face in accessing health care are poverty and lack of transportation. The specific measures taken at the community or institutional level are community contributions, special assistance from relatives at the community level, and lower prices for life-saving drugs. (Woman, Mamou) Table 8. Provider and Community Perspectives on Youth Barriers to Health Care Access Integrated Health Service Delivery in the Post-Ebola Context 33 50 45 40 35 30 25 20 15 10 5 0 Hours of services Proportion of Providers Financial obstacles No obstacles Figure 8. Adolescent Barriers to Health Care Cost of services Not comfortable with sex of provider Waiting time Authorization of spouse/father Decision making power 100 90 80 70 60 50 40 30 20 10 0 Proportion of Providers Unaware of available services Religious/ cultural barriers Language barriers Discrimination/ perceived discrimination Long wait time for treatment/ service Transportation challenges Cost/financial barriers Figure 9. Overall Barriers to Health Care been using contraceptive services. Before, young people were ashamed, but now there is a good attendance.” (Woman, Kindia) Youth face specific social and cultural barriers related to shame and stigma for family planning that have not been addressed or specifically targeted through youth￾friendly messages or health services. Qualitative findings revealed various perspectives on addressing remaining challenges for youth. Youth are sometimes ashamed to be seen at health services if not married, and it was also noted that both male and female youth prefer same￾sex providers, as there is often lack of understanding or relatability between youth and providers. While the design of HSD intended to focus on the specific needs of youth as a cross-cutting element, the activities lacked specificity. One ongoing task was to work with the Ministry of Health and partners to integrate the Adolescents and Youth Health Strategy into HSD activities, but little progress was made because of HSD did not want to duplicate the efforts of UNFPA and key partners, as UNFPA is the lead partner in this domain. In 2018, Guinea was identified as a priority country for investment of UN-managed Muskoka Funds for adolescent health. HSD attempted to harness this momentum and identified the NGO Santé Plus to develop a youth-focused project in Conakry. Given 34 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity the intersectoral nature of this endeavor between the Ministry of Health, the Ministry of Youth, and various partners, progress has been slow due to the political nature of bringing together multiple ministries, and lack of time and funds prevented HSD from engaging further. HSD did recommend Santé Plus for this effort, but this has been a missed opportunity since the 2018 international focus on investing in Guinea. Yet, 84% of providers surveyed reported their facility is working on addressing barriers for youth. An important consideration in access to care is the role of health posts. However, it must be noted that health posts were excluded from the scope of the HSD Activity design which focused on the hospital and health center level, with the exception of some trainings for CHW/RECO at health posts. Nevertheless, the evaluation team visited some health posts that were present in selected communities, after hearing from community members that health posts are mal-equipped. Health posts are located in geographic areas farther from health centers, so can serve to bridge the access gap for the more remote and vulnerable populations. Yet qualitative data and health post visits revealed gaps were identified at this level: “To improve the quality of community health services it would be necessary to equip the health post, supply medicines and increase the number of staff.” (Focus Group, Men, Kindia) training, managerial guidance, quality improvement processes (e.g. SBM-R), close follow up and support. HSD increased engagement of Ministry of Health managers in the monitoring of performance and quality of care at facilities. HSD staff supported greater participation and leadership of national, regional and prefectural management teams in regular monthly supervisions, periodic analysis of service data, availability of data collection tools, training of health workers and coordination meetings demonstrated appropriate supervisory mechanisms to sustain quality improvement measures at the facility level. In Kindia, health providers and managers also reported the project improved the quality of services and their working environment in the health facilities (KII Providers, KII Facility Managers). Specifically, health providers and managers noted that HSD contributions included the provision of equipment (e.g. delivery kits, delivery tables and contraceptives), renovation of the maternity and pediatric wards, improvement of the quality of care (through training on performance standards), improvement of facility indicators, availability of medicines, community awareness, and FP counselling as facilitative of quality of care improvements and their working conditions. Ministry of Health capacity was built at the decentralized level to do planning, monitoring, and supervision, yet more is needed to improve leadership at national, Département Provincial de Santé (DPS), Département Regional de Santé (DRS, and district hospitals and heads of health centers for arbitration and coordination. For some partners, there was the perception that planning has improved significantly. “They do trimester and annual planning at hospital and district levels with support from Stop Palu… to create a culture of planning; now all the directors at all levels do the strategic planning together.” (KII National) Outstanding individual examples of leadership was reported: “National [Ministry of Health managers] participate in improving the quality of services through EVALUATION QUESTION 3 – HEALTH SYSTEM STRENGTHENING: To what extent has HSD strengthened the health system to deliver an integrated package of RMNCH+ Health care services across the continuum of care? STRENGTHENED HEALTH SYSTEM GOVERNANCE 3.1.1. The HSD Activity strengthened the potential of national and local governance for health system improvement including quality of care through Integrated Health Service Delivery in the Post-Ebola Context 35 supervision and monitoring, but also through the advocacy they do at the international level. They also participate in improving the quality of services by fighting against drug stock-outs, training, and ensuring post-training follow-up. SBM-R has created a partnership with local elected officials through the COSAHs.” (KII Provider, Kindia) “After Ebola, the mayor has always been involved in the technical meetings of the hospital, which are contentious, and which in turn sensitize the community about the new decisions.” (KII Provider, Kindia) Facility leadership largely determined the success of the quality improvement efforts. In some locations, individual leaders created the momentum and commitment and drove the change. Dedicated staff and managers that put in a lot of effort could motivate the whole staff. Some facilities, however, demonstrated less leadership and commitment. HSD activities focused on Ministry of Health priorities for the health sector. Despite capacity building, mentoring and joint supervisory missions, leadership and commitment by Ministry of Health managers, particularly at the national and regional levels, were insufficient to sustain the activities without HSD engagement. This was particularly true in Conakry where referral hospitals struggled with delays in referral, high cost of medicines, and challenges managing biomedical waste (e.g., lack of an incinerator or the means of transporting waste to the place of incineration). It should be noted that the large referral hospital Ignace Deen was added to the HSD Activity post-hoc after Donka Hospital began renovations. Despite its position and size, Ignace Deen was ill￾equipped to take over nearly all tertiary care activities, making participation in HSD particularly challenging. 3.1.2. Missed opportunity for partnership: Despite the apparent need to increase support at the sub￾national level for the delivery of quality health services, there was little mention of potential partnerships with the nongovernmental and private sectors as facilitators of health system improvements. While the RFA called for such partnership, HSD did not fully engage, possibly due in part to limited funding. NGOs participated in HSD-directed community sensitization activities but was not implicated in health facility-level quality improvements such as provider training on updated protocols. One exception was EngenderHealth as technical lead on obstetric fistula treatment and reintegration activities. Engaging with the private sector to conduct activities was regarded as too expensive despite the large presence of private sector providers in the capital city. 3.1.3. Missed opportunity for sustainability: National, regional, and prefectural ownership and leadership to sustain HSD-facilitated improvements was not evident. HSD did not manage to sufficiently implicate government partners at all levels to sustain achievements of HSD activities. Key interventions such as maintenance of donated equipment, participation in oversight, and supervision were generally delegated to lower-level staff, without accountability measures detailed and followed up. Generally, there was a lack of delineation of expectations of government partners to ensure promises made towards sustainability are upheld. However, many health providers and managers, were strongly engaged in some facilities. Respondents were mixed on the sustainability of HSD-supported activities beyond the life of the project. Some felt sufficient sustainability has been built-in to project activities through the structures and processes implemented, such as the monitoring and supervisory systems and regular district meetings, whereas others were less confident regarding continuity. While HSD has put in place the necessary tools, checklists, and managerial protocols, including the SBM-R system, to sustain quality improvements, policy-level changes supported by budget allocations are not yet in place. In addition, respondents were concerned that HSD investments may be discarded with staff changes, or if external incentives are not provided. Many respondents were confident that sustainability at the facility and community levels is assured through “continuous collaboration between the actors, 36 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity sensitization of the community, and regular formative supervision at all levels as is currently done by the HSD teams” (KII Facility Manager, Kindia). The dynamic created by HSD results from close facility supervision, follow-up, and monitoring of supported activities. Some stakeholders admitted that given USAID’s long-term engagement in the health sector, sustainability would be simply assured by the next project, thus divesting the government of their role in sustaining the gains made. “The service quality control team [of HSD] is always on our doorstep [so] whether you like it or not, you’re going to work well.” (KII Facility manager, Kindia) “HSD project creates a dynamic and a commitment of local partners to sustain the project interventions after the end of the project through the reinforcement of the training it has given and the supervision of what has been put in place. The HSD project has contributed in many areas: it has provided many materials for the maternity health (SONU, FP and PCI), training of many health workers in different services, and the implementation of performance standards.” (KII Facility Manager, Mamou) 3.1.4. Lack of leadership is a fundamental problem in the health system. Respondents expressed low confidence in the leadership by higher managerial echelons at district, regional, and national levels due to over-commitments limiting their management availability. Staff deployment and retention also contributes to the problem of low health system ownership of results (see section 1.1). Lack of follow up and oversight of staff norms, equipment maintenance, and regulations by facility and district￾level managers (if not incentivized by HSD) further deteriorates confidence. There was a wide consensus that continuity of HSD-supported improvements will not be possible until there is greater accountability by the Ministry of Health for their respective areas of responsibility (i.e. planning, quality control, etc.) with consequential oversight. One manager suggested to strengthen leadership and reassure partners it is better to “empower facility staff to take care of materials because they own them, including managing inputs and collecting and analyzing data.” (KII Facility Manager, Boké) The full effect of the managerial shortcomings that created delays due to inaction were largely compensated by HSD. When gaps emerged, HSD staff simply filled them or facilitated (through incentives) their completion through local Ministry of Health and government partners to sustain the interventions. Some respondents felt that Ministry of Health leadership focuses on policies and guidelines rather than prioritizing planning, monitoring, and facilitative supervision of health services, which may not be the most productive use of their time. Some stakeholders believed that HSD’s filling of leadership gaps decreased the likelihood that the government managers would step in and play their role successfully. This was evident in conversations with national-level managers who did not seem to feel responsible for ensuring use of data for decision making related to stock-outs at district level as this was considered the responsibility of local managers, despite all stock requisitions being filled at the national level. Some of the challenges could have been overcome through greater alignment and donor coordination by the Bureau Strategique de Developpement that was not taking place at central level. At region and district levels, some DRS try to coordinate inputs such as training with some success. Yet facility managers and NGO partners state that they have no control over who gets trained: they ask the DRS and DPS to select the right people for trainings, with the same persons selected for trainings. Those selected attend so many trainings that they have no time to implement and reinforce the programs. As a result, the proliferation of knowledge across the health system or even within one facility is often focused on few individuals which limits institutional change. The need to improve coordination and timing has been raised by USAID project implementing partners at donor meetings but no changes have occurred in how those trained get selected. Integrated Health Service Delivery in the Post-Ebola Context 37 3.2.1. HSD’s provision of equipment and infrastructure improvements in several facilities contributed to the delivery and quality of integrated health services, building community trust and supporting utilization of services. Critical to the HSD approach was the provision of equipment needed to perform interventions to standard. According to providers, “the project donated equipment such as an anesthetic machine, vacuum extractor, incinerators, autoclaves, oxygen extractors, delivery boxes and medicines” (KII Providers, Conakry). Facility managers and providers recognize HSD project assistance in terms of “equipment, management tools, small materials and various kits (delivery, caesarean section)” (KII Facility Manager, Boké). Improvement in quality achieved due to increased availability of equipment and supplies is widely appreciated by staff and community members and has been a critical adjunct to other quality improvement efforts including training to HSD’s successes in rebuilding communities’ trust in facilities. Facility staff and communities took pride in the increased quality that they were able to provide which led to improved service utilization. “HSD has improved the availability of equipment and drugs through significant contributions that facilitate access to care, access to drugs for comprehensive management of fistula patients, an example being the management of fistula. These various aids have helped to attract patients to the center and increase revenues.” (FGD COSAH, Labé) Providers’ job satisfaction increased due to improvements in their working environment at the facility level due to HSD contributions. According to facility managers, the HSD project has positively changed the working environment in the facilities covered by its activities. All the support helped the beneficiary facilities increase their intake capacity and the supply of quality maternal and child health services. “Thanks to the project, our midwives are happy and work better because they have the training and all the equipment for their services.” (KII Facility Manager, Kindia) 3.2.2. Missed Opportunity for Sustainability: The Ministry of Health did not put in resources or systems to maintain equipment or ensure the rehabilitation done by HSD was maintained. Despite HSD support for the purchase of essential equipment and commodities for the provision of integrated quality care, there did not appear to be an elaborated maintenance component within the project to support facilities to secure and maintain the equipment, nor to hold facility managers and the DPS accountable for the contributions received. While a memorandum of understanding was signed upon equipment delivery confirming facility receipt and responsibility for maintenance, HSD staff and many stakeholders, including providers, were very concerned that the equipment provided was being used for personal use, had been stolen, or was in disrepair shortly after it was received. Through the SBM-R process, for example, broken or dysfunctional equipment cost facilities points that limited their capacity to earn a star. Despite these concerns, HSD noted maintenance of equipment and infrastructure improvements were the responsibility SUCCESS: INTEGRATED CARE TRAINING REACHES VIRTUALLY ALL HEALTH FACILITIES Integrated quality care depends on providers ability to deliver a variety of services to standard, and the equipment and infrastructure to carry out those services. HSD trained providers on the essential integrated care package to ensure their capacity to deliver integrated care (see 1.1). According to HSD monitoring data, over 96% of target health facilities offer the complete package of integrated essential RMNCH+ services at the end of 2019, and 83% of health centers and 77% of hospitals were delivering services to performance standards, which exceeded project targets. HSD also assured essential equipment was available and provided minor upgrades to make possible the delivery of quality services. 38 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity of the facility or the Ministry of Health, and thus maintenance, oversight, and control were not included in the project. It was stated that the HSD Activity was modified to develop workable strategies for equipment and facility maintenance based on lessons learnt, although this was not evident to the assessment team nor to many of the respondents. The challenge with equipment oversight was felt by many stakeholders as “due to a lack of motivation on the part of individual DRS, DPS and facility managers.” (KII National; KII Providers) “In reality, the project will leave one day, regardless of the duration, but if the equipment is not looked after, the problem will always remain at the structural level.” (KII Facility Managers, Kindia) There was a common perception that essential project contributions for maintaining quality improvements in service delivery (e.g., equipment, training, and commodities) could be secured through continuing project support. As mentioned previously, sustainability of inputs were largely considered the continuing remit of USAID as the donor rather than the role of the Ministry of Health. “To overcome these difficulties, it was necessary to advocate with the project to continue providing these inputs on a permanent basis. Without the help of this project, it would be very difficult to overcome these difficulties.” (KII providers, Kankan) While the majority of providers felt that the HSD project inputs had created sufficient momentum for the Ministry of Health for maintaining the improvements made, this sentiment was not shared by COSAHs, facility managers, nor MoH managers or policymakers. “The HSD project has created momentum and a commitment from local partners to sustain the interventions. The Ministry of Health and the government have generally supported the improvement of the quality of services with a lot of effort by providing medicines and fuel, training providers, delivery kits, family planning [commodities], equipment (e.g., mattresses, x-ray, oxygenator, IV pole and consultation tables), free caesarean section and malaria treatment, supervision, … availability of drugs and staff. Training on advocacy, health promotion and organization of community meetings can be maintained after the end of the program through the COSAHs.” (KII Providers) Provider optimism in the sustainability and maintenance of project inputs may be attributable to the support they received from HSD staff through their frequent presence and mentoring. “Also noteworthy is the accompaniment that the project gives to the agents after the training, which contributes to a better organization of the services, the definition of work standards and eventually the improvement of standards. This process has enabled certain structures to obtain “stars”…. After obtaining the gold star, the HSD project has created a [good] dynamic.” (KII Facility manager, Boké) The majority of national stakeholders and implementing partner respondents (and selected providers and facility managers) did not share providers’ optimism that existing Ministry of Health structures, momentum, and commitment would sustain quality improvements at facilities that have been supported by HSD inputs. They felt strongly that a high level of impunity and a lack of control and accountability exists at facility, and Ministry of Health managerial levels that undermines efforts to sustain improvements in the health sector. Respondents stated that individuals can take public equipment, space, materials and run their own private clinic out of the public facility. As a result, some managers and providers have started to lock up equipment to avoid it being stolen or misused consequently limiting its availability in moments when there is no access to the key. Critical equipment for emergency obstetric and neonatal care such as a pelvic exam lamp, an ultrasound machine, and an autoclave were not available across all facilities surveyed (Figure 10), and where present at facilities, not all were functional. Such equipment is key for SBM-R quality assurance, and their absence limits progress in provision of emergency obstetric and neonatal care protocols. Integrated Health Service Delivery in the Post-Ebola Context 39 Figure 10. Availability and Functionality of Select Supplies and Equipment at HSD Sites Visited, December 2019 (n=35) 3.2.3. HSD’s efforts have contributed to increased availability of drugs and commodities at facilities but struggled to overcome the programming gap at district and regional levels that did not benefit sufficiently from project-specific support to address recurring and prolonged stock outs of RMNCH+ products in some facilities. HSD reported programming challenges with maintaining RMNCH+ product stocks in facilities over the course of the project, with prolonged stock outs experienced in some regions with health centers having a slightly greater challenge than hospitals (see Table 4 and Table 5). It was reported that the difficulty encountered is often related to the prolonged interruption of family planning commodities such as the implant and the intrauterine device. Over a third of providers reported that stock-outs of family planning commodities were a challenge in delivering modern methods. Some regions’ health centers reporting stock outs to be a challenge for more than half the facilities responding to the survey. Despite difficulties addressing the stock-outs, improvements were seen in project year four, when product availability rates were above 80% in four of the seven regions, according to data from the logistics management unit. Average product availability rate across all items was 83% in hospitals, improved health centers, and regional PCG depots (Table 8) and 79% in health centers (Table 9). CHW/RECO and COSAH credited HSD with improvements in stock availability, particularly for family planning commodities as they engaged more actively with district and regional managers to proactively work to solve the supply chain issues. While this engagement on supply chain was beyond the purview of HSD, local HSD staff worked with the districts to try and solve the problems at hand. “Medication interruptions were common in the past until the HSD project was able to fill this gap.” (KII CHW/RECO, Kindia). Particular challenges with commodity stocks were highlighted in Boké, Kindia and Kankan however. Management (GHSC-PSM), led by Chemonics. The GHSC-PSM project operates at a central level with support down to the district level but cannot resolve the facility-level problems stemming from human resources capacities and will. HSD worked to support and help to carry out stock monitoring during post-training follow￾up and supervision visits in the healthcare facilities 100 90 80 70 60 50 40 30 20 10 0 Vaginal speculum Proportion of Facilities Vaccine refrigerator Ultrasound Blood pressure apparatus Disinfectant solution At least 5 syringes Pelvic Exam Lamp IUD kit available Newborn Autoclave scale Percent facilities have available Percent of available are functioning Select Supplies & Equipment 40 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity and coordinated with the GHSC-PSM Project and DRS management to exchange product stock and inventory data. HSD advocated and coordinated with the central and regional levels and GHSC-PSM to improve product availability in facilities, including employing regional teams to share the results of their monthly product inventory with the healthcare facility managers and the DPS pharmacists. At weekly DRS meetings, HSD focal points encouraged them to: place product orders on time; link health facility personnel with GHSC-PSM to speed up the replenishment process and correct RMNCH+ product stock outs and over stocks noted during field visits; and relay stock out issues in the different regions to the Ministry of Health Logistics Management Unit for resolution and advocating for replenishment of facilities based on monthly average consumption calculated by GHSC-PSM. 3.2.4. Missed opportunity: HSD engaged in facility￾level stop gap measures to resolve stock-outs pending implementation of HSD activities, such as SBM-R quality improvement in family planning. They also participated in a national-level dialogue on supply chain management and improvements. However, HSD was not designed to intervene in supply chain management in a significant and systemic way as this was the purview of GHSC-PSM. As a result, supply chain delays and stock-outs persisted at the facility level to a greater or lesser degree depending on facility, DPS, Region Quarter Labé Mamou Faranah Kankan Kindia Boké Conakry 1 99% 100% 87% 76% 48% 73% 99% 2 100% 100% 97% 77% 65% 71% 100% 3 98% 100% 83% 87% 67% 51% 95% 4 100% 93% 85% 75% 73% 42% 93% Year 4 Average 99% 98% 88% 79% 63% 59% 97% Table 9. Reproductive Health, Family Planning, and Integrated Management of Childhood Illness Product Availability Rate in Hospitals, CSAs and Regional PCG Depots in HSD Activity Zone by Region, October 2018-September 2019 (PY4) Region Quarter Labé Mamou Faranah Kankan Kindia Boké Conakry 1 98% 100% 87% 72% 55% 67% 96% 2 99% 94% 90% 79% 63% 43% 98% 3 97% 95% 77% 79% 63% 55% 95% 4 100% 46% 83.2% 62% 67% 47% 95% Year 4 Average 98% 84% 84% 73% 62% 53% 96% Table 10. Reproductive Health, Family Planning, and Integrated Management of Childhood Illness Product Availability Rate in Health Centers in HSD Activity Zone by Region, October 2018-September 2019 (PY 4) Source: eLMIS, supervision visits, monitoring via telephone Source: eLMIS, supervision visits, monitoring via telephone Integrated Health Service Delivery in the Post-Ebola Context 41 and DRS management of drug and commodity stocks. HSD was designed to address quality improvement at the facility level while GHSC-PSM was designed to resolve national-level stock issues. While they ideally could be synergistic, a gap emerged at district and regional levels that both projects were not able to fully resolve as they were not planned or budgeted for in their project design. It was anticipated that the Ministry of Health would resolve supply chain issues, particularly at the facility level, and GHSC-PSM would support their overall strategic approach and planning for improved supply chain management. However, lack of resources within the Ministry of Health required the projects to buy essential drugs and commodities when possible to fill facility gaps. In this context, HSD found themselves pressured to resolve facility-level gaps, which were the financial remit of the government. DATA QUALITY AND USE 3.3.1. HSD has worked to improve data collection, quality, and reporting at the facility, prefectural and regional levels to support the availability of data use in decision making. However, the complexity of the national roll out of the DHIS2 health information management system has led to data quality and data use challenges, resulting in parallel reporting. The improvement of data collection practices in the facilities is based on the validation of quarterly data. At the health district level, data is collected, compiled and analyzed monthly in the presence of all the facility managers to check consistency. The results of each health center are prepared by the statistics officer and analyzed at the meetings to correct problems immediately. These meetings were originally supported by Stop Palu+ to review malaria data but quickly expanded to review all facility data. Currently HSD works with Stop Palu+ to facilitate the monthly meetings. DHIS2 indicators have improved in recent years, suggesting that data review at the monthly meetings, and focused quality improvement processes supported by HSD and others have contributed measurably. “Thanks to training on reporting tools, data collection, data entry, and collective analysis, the structures and authorities have dashboards to collect data on diseases under surveillance…. Performance according to each structure is visible every month in order to take steps to improve the quality of care.” (KII Providers, Boké) Figure 12 shows strong data quality performance across most data management domains on which HSD￾supervised sites were assessed among the 127 facilities supervised by HSD regional M&E advisors. Follow up on recommendations, conversely, must be led by the Ministry of Health and may thus be less responsive to HSD-supported activities. “The HSD project contributed to improvements in facility data collection practices through the implementation of the DHIS2 software. The data collected has been analyzed to identify key health and service delivery issues at the facility level, reports are provided and tracked, and death cases are audited and analyzed. Thanks to this project we understand the high rate of maternal deaths SUCCESS: DATA QUALITY IMPROVED THROUGH TRAINING, DIGITAL ENTRY, AND ANALYSIS Significant improvements in data quality was achieved through staff training, provision of collection materials in the various units, computerization of data entry and management, and data analysis sessions involving all staff. Appreciation and use of service data in monthly meetings has supported improvements in service delivery and facility performance. Engagement of managers from the facilities, DRS and DPS together with partners from HSD and other stakeholders around data further catalyzed change. At the district hospital level, the project “developed their capacity to analyze, interpret and disseminate service data but also to make decisions based on facts.” (KII Providers, Boké). 42 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity in the district and it allowed us to do a good follow￾up of pregnant women. Changes in data collection and analysis have influenced the way services are provided.” (KII Facility Manager, Kankan) “The analysis of the data and the recommendations made led to a commitment that enabled the structure to increase its performance in the use of the partogram from 19% to 77% and then to 94% thanks to HSD.” (KII Facility manager, Labé) Improvements in data collection and use for decision making to improve services was also recognized by CHW/RECO and COSAH members. At some HSD￾supported facilities, data reports and dashboards were shared with the community through meetings with community leaders and actors, health centers, DPS, DRS, and Ministry of Health, though this was not uniformly the case and breakdowns in data flow did take place. “The HSD Activity has improved the quality of data on health services and their use for better decision￾making through funding prefectural technical committees, and feedback reporting involving RECOs and CHW/RECOs directly in contact with communities. During this technical committee the data from the health posts of the DPS are reviewed in order to find recommendations for improving data quality…. Involvement of RECOs [helps us] to have good results.” (FGD COSAH, Labé) In Conakry, respondents reported that HSD helped to raised awareness in the community by including media representatives within their trainings and in meetings reviewing facility monitoring data. These sessions were complimented by the six-monthly review by a technical committee within the Ministry of Health that validates registers and documents to improve data collection and reporting. “Before there was no such thing. The data was not available. But thanks to this project, as soon as you need family planning service data, you [can] have it. The data collected is analyzed monthly with health staff to identify the problems [in service quality at the facilities] that exist.” (Facility Manager, Conakry). However, some did not agree, stating that there is a lack of feedback from the higher levels responsible for analyzing the data collected: “We do not analyze the data [ourselves]. The central level does the analysis and we do not have the feedback.” (KII Facility Manager, Boké). Despite improved data collection in many facilities and moving it upstream for analysis, getting the results back to facilities in real time from the central level to be reviewed in the monthly district meetings has proven difficult in some cases. DHIS2 data is not currently available in real-time for decision-making. 100 90 80 70 60 50 40 30 20 10 0 Global perf Average Performance Availability of Logistics management tools Figure 11. Average Performance of Data Quality Domains, PY4 Recommendations follow-up Health data display Data reporting and entry Human resources Data collection tools filling 96% 91% 94% 97% 83% 73% 58% 84% Integrated Health Service Delivery in the Post-Ebola Context 43 More capacity building of providers and analysts is desperately needed to allow for real-time data use for decision-making. DHIS2 is improving data collection but data quality issues and access to analysis platforms at facility and district levels remain, despite national roll-out. Other ongoing challenges are training and a lack of data collection tools at facility and district levels. USAID MEASURE Evaluation introduced DHIS2 but respondents said at least two more years of support, including funding, is needed to make it operational across the country. There was general agreement that HSD should continue to invest in the DHIS2 health information system. Specifically, more training on data collection and use for decision-making (to identify areas for improvement) and reporting tools are needed at all levels (e.g. community, health posts, health centers, hospitals) to make the system operational. Yet duplication of training on DHIS2 has been noted and should be avoided. (e.g., GIZ doing training on DHIS2 at the university). It was not clear whether the Ministry of Health has the commitment and resources to fully scale up the use of DHIS2 nationwide under its own initiative and leadership. National stakeholders varied in their perspectives on the status of DHIS2 implementation, ranging from perceiving it as completed to identifying the need for more donor support for training and tools, indicating more investment will be requested from a future project to continue implementation activities. IN-SERVICE TRAINING 3.4.1. Developing the capacity, curriculum, and tools (Skills Labs) at selected public and private nursing and midwifery schools has been very successful and important component of the project. Upgrading course materials and school capacities to train on the latest standards will have long-term effects beyond the life of the project. Ministry of Health Directorates responsible for the schools are engaged and active in finding ways to facilitate maximum use of the skills labs for example across to even non-beneficiary schools. Respondents reported that the initial training allowed them to “learn the trade and manage responsibilities associated with their jobs… Before, I didn’t know when to initiate the partogram when a woman is in labor, but I have now mastered the partogram and I feel proud.” (KII Provider, Kindia). Sustainability of HSD-supported in-service activities will be assured by the pool of trainers, and the continuing education courses they provide on an on-going basis to update standards and skills of providers (doctors and midwives). 3.4.2. The public and private medical, midwifery and nursing schools are under the direction of separate ministries. Private medical education is coordinated under the Ministry of Technical Education, Vocational Training, Public Service and Labor. The public nursing and midwifery schools are coordinated under the Faculty of Medicine under the Ministry of Education. Despite their different hierarchical structures, they collaborate well together to deliver pre and in-service training in Guinea. HSD-supported activities were very much appreciated by both departments as expressed by their motivation and commitment to expand and sustain the training activities begun under HSD. They noted, however that Ministry of Health support is needed, particularly from DPS and DRS, to own and sustain the investment after the end of the project. Maintaining service standards through in-service training, including skills lab sessions will require dedicated remuneration for the trainers. Maintenance of the materials afforded by the initial investment can be done by the schools themselves with Ministry of Health support, ensuring the training materials are available to all students is critical. Respondents agreed that there is a need to expand the creation of skills laboratories in the nursing and midwifery schools to all administrative regions beyond the four supported by HSD through Ministry of Health support. (KII National; KII Providers, KII Facility Managers) Scaling up the skills lab should be a priority for the Ministry of Health to demonstrate their engagement with sustaining the well-received pre- and in-service training currently provided within the four HSD￾supported schools. Other issues noted by respondents as important areas to expand in the curricula related 44 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity to managerial and leadership training and a training package on adolescent and youth health. POST-INVESTMENT SUSTAINABILITY: THE RESPONSIBILITY OF THE GOVERNMENT 3.5.1. Facility and community level respondents considered post-investment sustainability the responsibility of the government at all levels. Specifically, sustainability depends on quality of planning and oversight by DRS, DPS, and the facility managers, and dedicated funding either from the government or through a project. Some believe the Ministry of Health can sustain the progress achieved. However, it will need to allocate the necessary financial resources to maintain the current level of supervision, monitoring, and support of the quality improvement efforts including continuous training, which many considered unlikely. “After the project, the support that will be necessary for its sustainability is the regular monitoring by the Ministry of Health and the financial support.” (KII Provider, Kankan) As noted in 3.2.1, some providers find that the HSD project’s interventions to improve quality of care can be sustained after the end of the project because health workers have been trained and equipped and should be able to maintain their performance. Quality improvement processes such as SBM-R have demonstrated what can be achieved in terms of quality of health services to health center staff, managers, Ministry of Health management, and the community. “If the facility level works, the rest will follow. All staff must be responsible for the sustainability of the project’s achievements, but especially the head of the center. By following the guidelines; the ways of working instilled by the project has become habit now that will maintain the improvement in the quality of care after the end of the project.” (Facility Manager, Labé) Sustainability requires leadership at all levels that is not yet in place. The DPS and DRS are involved but not ready to take over the process. HSD is playing a leadership role by facilitating the quality improvement activities across project components. Government representatives and managers from national to regional and prefectural level accompany the process, often participating in training and supervision, but do not yet own or lead the processes nor the activities. It was suggested that in the planning for a next phase, a new approach to transferring responsibility for the project is needed based on performance measures and markers such as is being supported by the World Bank through performance-based financing schemes. Respondents stated that additional investment is needs to include water, electricity, and provider accommodation, which could be included in a performance-based financing scheme as has been in done Rwanda, Uganda, and elsewhere. Respondents were clear that while everyone has a role in sustaining quality improvements, ultimately “sustainability depends upon the government taking control of the ship by supervising the achievements of the project.” (KII Provider, Kankan) They stated that the MoH must “make available all the materials required to provide quality care on time, and not wait for a project to come to their aid.” (KII Provider, Labé) “The involvement of all providers at all levels of the health pyramid, even without remuneration [is necessary], the Ministry of Health, the DPS, DRS, department heads—everyone must be able to get involved for sustainability, because we have been given everything, it is now up to us to commit ourselves to continue the work.” (KII Facility Manager) Another reminded that the Ministry of Health and health staff remain after the project ends implying that they must take charge of their own systems. “The project may leave but the Ministry of Health and the health workers will not be able to leave.” (FGD CHW/RECO, Kindia) “Change starts from the top! [Le poisson pourri par la tête.] (KII Facility Manager, Boké) Integrated Health Service Delivery in the Post-Ebola Context 45 SUSTAINABILITY OF SUB-COMPONENTS 3.5.2. Many of the sub-components of the HSD Activity are not well-positioned to be sustained without external support. Clinical performance and SBM-R: Performance standards and the SBM-R quality improvement program are best placed to continue if adequate monitoring and supervision are made available and led by providers and facility managers themselves and supervised at the DRS and DPS level (with the requisite financial support for supervisory visits and validation assessments). Facility managers recalled quality improvements made by facilities (in the context of the SBM-R process) with the help of the project that can be leveraged to maintain the momentum. “After the initial training and assessment, we had a 30% [rating]: it was the infection prevention and control domain that had problems. With the help of HSD, we implemented new measures to improve quality at all levels, including data quality….HSD helped us to plan activities and supervise them each time. To maintain investments after the end of the project, each level of decision making must play its role (e.g. the Ministry for system maintenance, the DRS for supervision, and the providers for compliance with the rules of the trade in their work).” (KII Facility Manager, Kindia) Sustainability of SBM-R can only be assured if the Ministry of Health takes full ownership of the program, which has yet to be observed. GIZ is supporting some quality improvements but the Ministry of Health itself does not have a fully functioning Technical Committee to maintain the SBM-R program. SBM-R validation in facilities has not, as yet, been consistent due to problems in scheduling the members of the SBM-R Technical Committee to come together to conduct the visit to the facility. More detailed information on the SBM-R program and its sustainability is discussed in SBM-R Case Study. “Sustainability of performance standards and quality improvements through SBM-R requires supervision, … support, as well as ongoing training and donations of materials to maintain health center quality status. Sustainability could be facilitated by integrating the SBM-R process into the Guinean health system policy.” (KII Facility Manager, Mamou) Obstetric Fistula: Sub-components of the project such as obstetric fistula repair and reintegration are particularly vulnerable as they depend on support of the Ministry of Health and an external partner. At the community level, awareness raising and referral must be maintained if women are going to access care and treatment. While the project set up a pool of competent trainers that can be maintained, without external assistance, respondents concurred that the program cannot continue as it is now. Current health budgets do not include line items for fistula repair expenses. “If the HSD project stops right away and the MoH agrees to cover the costs, we will be able to continue to operate. But if not—even though we have mastered the technique of fistula repair— we don’t have the means to supply the patients with kits, to house and feed them, to pay for their transport to and from the hospital and to raise awareness in the radio stations.” (KII Providers, Faranah) Integration of gender-based violence: Investment in the integration of gender-based violence within the health service package has been minimal with sustainability of the gender-based violence intervention seen as the responsibility of the government. Partnership with the Ministry of Social Affairs and a local NGO requires additional funding from government or project sources to be sustained. Strategic planning and partnership are needed to ensure focal points are in place and systems are functional to support an adequate multisector response for sustainability of GBV investments to be maintained (see 5.2). Community Engagement: As noted previously (see section 2.4 on community engagement and sections 3.1.-3.1.3 and 3.2.1 above), community members are not yet sufficiently engaged in the quality improvement process of their local health facilities to support sustainability. Awareness raising by CHW/RECO and local NGOs has resulted in increased utilization of 46 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity services, especially post-Ebola (see 2.4) but they are not mobilized to claim their right to quality health services. COSAHs represent communities on the SBM-R assessment teams but broader participation in the quality improvement process remains limited. Some communities are engaging with their facilities to support quality improvements through the SBM-R process but these are notable exceptions. Respondents noted that when the community is engaged and confident of the service quality, they will continue to come to the health facilities. Changing community perceptions and trust in the health system will require the government to continue to support quality improvements process (e.g. SBM-R) and awareness raising through further investment—efforts also supported by USAID activities such as Stop Palu+ and CIHG. Providers and COSAH members stressed the role community and the COSAH, as their representatives, can play in sustaining project achievements. “If the people who have been trained take the project with open arms and continue sensitizing the communities, the project can be sustainable.” (KII Provider, Faranah) “The HSD project interventions can be maintained provided that there is regular monitoring and that health providers and RECOs continue to carry out the same project actions” (FDG COSAH, Kindia) Support to regional nursing and midwifery public and private schools (and to a lesser extent the medical faculty due to ongoing strikes and other challenges) were considered good investments which can be sustained by the schools themselves. New curricula and trained teachers, and the skills labs facilitate institutionalization of the new approaches within preservice training. Likewise, improvements in the in-service training programs are decentralized, with a pool of trainers at regional levels. Respondents felt that if there is sufficient leadership and support from the Ministry of Health and within the schools, capacity building and training investments could be maintained. EVALUATION QUESTION 4 – FISTULA CARE: To what extent has local technical and managerial capacity been sufficiently built to support fistula prevention and care in Guinea? 4.1. HSD activities have contributed to building fistula surgical capacity and coverage in Guinea through training providers and supporting surgical repair in existing and new repair hospitals. Obstetric fistula is a traumatic and debilitating birth injury that affects the poorest women and requires specialized surgery as treatment. HSD fistula-related activities incorporated prevention, treatment, and social reintegration programming, with EngenderHealth as the technical lead. HSD activities aligned with the national fistula strategy that promotes expanding coverage of surgical services to regional hospitals through decentralization of care. HSD activities focused on the three hospitals where fistula repair services had been developed under FistulaCare and FistulaCarePlus, prior USAID projects (Conakry,* Labé and Kissidougou). In addition, HSD supported the integration of fistula repair services into three regional hospitals where such services were not provided before (Kindia, Boké and Faranah).† HSD contributions included supporting 760 repairs from FY2016 to FY2019 (Figure 12), representing 70% of the target, and training a total of 232 individuals on fistula surgery, pre and postoperative care, screening and referral (including 17 fistula surgeons, 106 nurses/ midwives; 9 medical anesthetists and 17 nurse anesthetists). Community level fistula screening activities increased women’s access to surgical services through referral. * HSD support to John Paul II hospital in Conakry resulted in only 10 repairs due to operational challenges. † Two regions not covered by HSD (Mamou and Kankan) had commitments from UNFPA to support fistula repairs at the regional hospital level. Integrated Health Service Delivery in the Post-Ebola Context 47 Figure 12. Number of fistula repairs supported by HSD by Year 300 200 100 0 Nubmer of Fistula Repairs 2016 2017 2018 2019 92 288 220 160 However, HSD activities were limited in their achievement of fully integrated fistula repair services. A well-integrated hospital would maintain skilled surgeons and clinical staff along with equipment that would allow for fistula repair as a routine service. While HSD supported repair services at the newly established repair hospitals, it could be argued that an inadequate number of fistula repair surgeries were supported in these facilities to provide sufficient training to the new local fistula surgical providers. The project strategy was to bring expert surgeons from Conakry to conduct repairs at the new sites while training local surgeons during these repair sessions. According to the World Health Organization, practical training with 300 fistula repairs are required for a surgeon to acquire a full set of skills in fistula surgery.38 However, together, the three new sites achieved only 109 repairs over the duration of the project resulting in an inability for surgeons to conduct an adequate number of repairs to become certified in fistula repair. If unqualified surgeons are conducting fistula surgery, successful closure rates will reduce and the number of complex cases will increase. Kissidougou and Labé hospitals achieved more repairs as compared to the newly integrated hospitals because they already had local certified fistula surgeons. In addition, the Jean Paul II hospital was not willing to honor elements of the partnership which affected achievement of results in FY2017. Ownership by government is lacking and funding outside USAID is very limited with UNFPA providing surgical kits for about 100 repairs yearly and Fistula Foundation supporting up to 50 repairs per year. In such a context, the sustainability of fistula repair service provision is a major concern, as alternative future financing for surgery and reintegration services is uncertain. “There are qualified surgeons for the management of fistula that can continue [the work] after the end of the HSD project, provided that there is a third￾party payer since the management is costly, not less than 300 to 600 US dollars per repair.” (KII Provider) 4.2. HSD has effectively supported community and facility-based prevention activities to prevent the occurrence of fistula in Guinea. The ultimate goal is to prevent fistula through increasing the quality of delivery care and ensuring all women receive education around maternal danger signs, access skilled delivery care, and that those women who develop complications receive high quality and timely emergency obstetric care. HSD collaborated with the Ministry of Health, Ministry of Social Affairs and UNFPA to design, implement, monitor and evaluate health communication programs addressing fistula, including dissemination of health messages through several communication channels. This included radio messaging, community messaging through door-to-door visits with community health workers and community relays. Furthermore, HSD 50 40 30 20 10 0 Proportion of Fistula Repairs Figure 13. Number of fistula repairs by Region Kissidougou Labé Kindia Boké Faranah Conakry Region 3% 2% 4% 13% 28% 49% 48 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity activities strengthened family planning counseling and provision at the community and facility levels, increased use of the partogram, and improved quality and coverage of emergency obstetric and neonatal care services including cesarean section. (See Sections 1.1, 1.5) However, challenges exist to optimizing prevention of obstetric fistula development including a variety of delays which prevent women from accessing needed timely cesarean section. Community-based activities are weakened by ongoing changes in financial coverage for community health workers. In addition, cultural barriers and practices such as early marriage, early pregnancies, women’s low status, and logistical barriers remain a challenge to access and use quality health services, even for prolonged obstructed labor. “We don’t have an ambulance. Our women who are referred suffer a lot especially if it is late in the night.” (KII CHW/RECO) 4.3. Reintegration activities provided under the HSD Activity were largely effective in terms of supporting women to reenter their communities. Reintegration activities include adjunct programming to support women’s psychological, social, and economic recovery from fistula given the significant stigma associated with the condition and its consequences. Reintegration activities supported by HSD and headed by the Ministry of Social Affairs focused on continuing strategies developed under the prior FistulaCare project, including social immersion and economic development. Host families living near fistula care centers hosted women after fistula repair for several weeks in order for them to resume community daily life before returning to their families and villages. In addition, the project provided apprenticeship training in soap making to selected women (i.e., at Kankan and Mamou) that enabled them to start a new income generating activity after repair. These activities were appreciated by beneficiaries and community members and were felt to facilitate community reintegration. However, not all women participated in reintegration activities, and some key stakeholders believed that successful reintegration depends mainly on successful surgery, with additional services unnecessary. “Women who return to the community after obstetric fistula treatment are satisfied with their experience and their results.” (KII Provider) Women treated for fistula contributed to awareness raising within communities resulting in better self￾esteem. “We help women to reintegrate into their communities after undergoing treatment by raising awareness among the population.” (FGD Men, Faranah) 4.4. Missed Opportunities: Reintegration programming was considered to be helpful but insufficient, with not all women who underwent surgery receiving reintegration programming. A more targeted strategy based on identifying women in greatest need of reintegration support could expand services to the most vulnerable while improving cost￾efficiency. Furthermore, repair breakdown and adverse maternal and neonatal complications during post-repair childbirth is common in Guinea.39 Current international guidelines indicate women should not become pregnant in the year following fistula repair and should undergo elective cesarean section in any subsequent deliveries. Some follow-up was done after fistula repair within HSD-supported activities, but strategies to expand this long-term follow-up after fistula repair (i.e. 12-24 months minimum after repair) to prevent fistula recurrence and ensure women’s access to continued care for prevention of adverse health events would be supportive. 4.5. Sustainability of existing investments will be critically affected by the significant backlog of women living with fistula who are still awaiting surgery in Guinea. The latest Demographic and Health Survey suggests that 4.1% of Guinean women of reproductive age have experienced fistula-related symptoms (regional range 0.6%-12.1%), with only 21.5% having undergone surgery.5 Approximately 105,000 women of reproductive age have suffered from fistula, with about 82,000 women still needing surgery, and Integrated Health Service Delivery in the Post-Ebola Context 49 ensuring local surgical availability is essential. Given the lack of funds to support more repairs while community awareness raising efforts were maintained, it is plausible that many women seeking care for fistula are turned away. On the other hand, because technical and managerial capacities were not sufficiently built, it is unlikely that fistula care will be sustained after HSD as anticipated, as perhaps the investment was too limited to bring about a substantial and sustained effect. EVALUATION QUESTION 5 – GENDER-BASED VIOLENCE: How well have the limited financial resources been strategically directed to address gender-based violence issues under the HSD projec interventions? What could have been done differently to enhance achievement of results? As described in the ‘Guinea RMNCH+ Context’ section of this report, the rate of violence against women in Guinea is high according to available data. Prevalence of socio-cultural norms justifying intimate partner violence are trending downward; however, they still prevail in the majority of Guinean men and women (national average of 55% and 67% respectively) according to DHS data.5 HSD’s primarily efforts aimed to improve the screening and treatment for gender-based violence in facilities and availability of and linkages to supportive and legal services for gender-based violence survivors. A secondary focus of HSD’s gender-based violence efforts was in violence prevention through community educators and media messages focused on changing social norms and making women aware of available services. HSD’s gender-based violence efforts were a continuation of USAID-supported efforts to implement the nationally defined package of health legal, and support services for gender-based violence survivors that began in 2013-2014 under the Maternal and Child Health Integrated Program and continued through 2017 under the Maternal and Child Survival Project, both implemented by Jhpiego. HSD aimed to continue the activity where it was already being implemented in Conakry and expand to two additional regions - Boké and Labé. These regions were chosen because of the high rates of reported gender-based violence and early marriage, as well as because other development partners were intervening in other regions. Funding specifically earmarked for gender-based violence was about $300,000 per year. 5.1. The limited financial resources devoted to gender-based violence under the HSD activity have been reasonably effective in strengthening the health system’s capacity to address gender-based violence, particularly in the regions targeted. The gender-based violence issues that HSD was strategically addressing respond to key gender-based violence causes and challenges in Guinea. HSD’s work at both the policy level and the facility and community levels around strengthening the availability and quality of gender-based violence services in facilities as a part of the integrated package of essential care was directly in line with HSD’s overall RMNCH+ objectives, making HSD uniquely situated to focus in these areas. At the policy level, HSD supported validation of the gender-based violence elements within the Norms and Procedures for Reproductive Health and related training packages and provided technical support to the Ministry of Health for the revision of the national strategic plan end female genital mutilation for the 2019-2023 period. At the facility and community levels, HSD took over the Conakry-based provider and community educator training from the Maternal and Child Survival Project in 2017 (PY2) and was successful in expanding the gender-based violence intervention to Boké in 2016 (PY1) and then to Labé in 2018-2019 (PY4). Seven healthcare facilities in Labé region and 29 health facilities in five districts of the Boké region can now offer gender-based violence management services. HSD also reestablished the legal aid case management component of the gender-based violence intervention, which had been initiated under MCHIP but was unable 50 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity to be continued under the Maternal and Child Survival Project. This component was substantially delayed and was not implemented until HSD’s fourth year. The delays resulted largely from the decision to carry out the work directly through a Guinea-based partner Meme Droits pour Tous, rather than the international partner that had been identified, requiring new efforts to define the scope and budget for the implementing partner. These delays limited the effectiveness of the overall intervention in the early project years, but the direct engagement of Meme Droits pour Tous is supportive of the Journey to Self-Reliance objectives. Integration of gender-based violence screening and referral into the essential integrated care package through provider training was widely viewed as important and successful where it was done according to facility-level managers, COSAH, and community members. In the three regions where gender-based violence activities were focused, HSD efforts supported access to strengthened gender-based violence management services for women, beyond the gender￾based violence screening and care that is a part of the integrated package that HSD is working to ensure all facilities can offer. 5.2. Missed Opportunity: In addition to screening and case management, HSD efforts also focused on violence prevention to some degree, aiming to raise community awareness of the problem and counter negative socio-cultural norms through community educators and media messaging, however an enhanced focus on prevention may be needed (depending on what other partners are already doing). Facility and community-based stakeholders emphasized the need to raise awareness of the problem of gender-based violence as a high priority, in addition to the priority of supporting victims with treatment and access to supporting resources that can be used to hold perpetrators accountable. Stakeholders at community level expressed favorable views of the gender-based violence services made available in HSD supported facilities. At the same time, community￾based stakeholders voiced many of the harmful social norms and values that contribute to gender-based violence, citing women’s provocative dress, women’s lack of submission, jealousy, infidelity, and men’s irresponsibility as causes of gender-based violence, indicating these views remain widely held and socially acceptable among many. “Everybody shows off and that’s what it is all about. In order to avoid being raped, you have to wear the right clothes, that’s what is good for women. to dress properly, how to give priority to wearing the right clothes, because if you meet someone you’re not properly dressed if they want you, they’ll have you, so that’s what we have to let them know, is to dress properly to protect ourselves.” (FGD Women) “Because, what the eye cannot see the heart will not be able to claim” (FGD Women) Evaluation data clearly demonstrated that these harmful norms—that violence is justifiable—are common among stakeholders interviewed, including community members, COSAH representatives, and in some cases healthcare workers as well. This demonstrates the need to address norms and raise awareness of the problem. Other challenges, including unemployment, lack of education, poverty, and alcohol and substance abuse were also reported to contribute to gender-based violence by community- and facility￾level stakeholders. Stakeholders highlighted the importance of social and behavioral change communication in addressing gender-based violence and harmful cultural norms, including in relation to female genital mutilation, and including through high level support on the issue, e.g. mobilizing champions among Guineans to speak out against gender-based violence. In this area of social and behavioral change communication for violence prevention, many stakeholders indicated the importance of taking action. As one national stakeholder put it: “More investment is needed to be more proactive rather than reactive.” (KII National) HSD did produce some local successes in denouncing domestic violence and gender-based violence, Integrated Health Service Delivery in the Post-Ebola Context 51 which is an important building block. Given the high priority in this area, expansion of community awareness and behavior change communication efforts is indicated, either through HSD, a behavior change communication-focused USAID activity, or a combination, with close coordination between the two. In the context of limited resources, some balance of investments in both services for gender-based violence survivors and violence prevention is warranted, as both are so critically needed in Guinea. 5.3. The small amount of money available for gender-based violence under HSD, though used well according to stakeholders, may be under-leveraging other gender-based violence investments and, in spite of efforts to coordinate across partners, the absence of a broader nationally owned strategy focused on gender￾based violence presents a challenge.* HSD collaborations with the Ministry and UNFPA on gender-based violence were synergistic. In addition, HSD works with gender-based violence units in police, gendarmerie, and justice. UNFPA provides equipment (computers) to gender-based violence units. Regional offices of the Ministry of Social Affairs previously distributed gender-based violence care kits; now they are channeled through the Ministry of Health which is a new challenge as there are no payments for focal points. A key achievement nationally is the revision of the National Strategy to End Female Genital Mutilation for which HSD contributed technical assistance to the revision, pre-validation and dissemination workshops. In spite of strong coordination efforts on the part of HSD and government and non-government partners, coordination gaps remain. There is a need for more collaboration in the planning stage for gender-based violence activities to leverage other efforts and further support is needed to improve the availability of medical kits and establish an equipped data management unit and system. National stakeholders expressed concerns that there doesn’t seem to be sufficient governance across sectors and gender-based violence donors and partners to address gaps in a coordinated way (e.g. in health facilities and among security services). An area where HSD could invest is in provision of support to the Ministry to better coordinate the gender-based violence issue across health, security and legal sectors (including establishing and supporting focal points). Stakeholders indicated the lack of focal points severely limits effectiveness of the Ministries’ efforts. “Certainly there are results but without good coordination we cannot capitalize on the results” (KII National) HSD’s efforts to address gender-based violence from within the health sector were considered important by many stakeholders, and were appropriate given HSD’s objectives. Inclusion and validation of gender-based violence in RMNCH+ norms and standards, including in policies and training packages, and ensuring all facilities have the capacity to integrate the gender￾based violence components of the integrated package were important efforts. One national-level key informant indicated that addressing gender-based violence from a health perspective makes it more palatable for some. Nonetheless, doing more at the governance level to facilitate stronger coordination among the key sectors is a significant gap that HSD or another USAID partner would be well-positioned to fill. * The National Strategy to End Female Genital Mutilation was recently updated by the Ministry of Health with technical support from HSD and Guinea’s National Health Development Plan 2015-2024 includes objectives relating to gender-based violence: Zero tolerance for female genital mutilation and gender-based violence: development of psychosocial, medical, legal care of women and children through a) campaigns to sensitize political, health and community leaders, and the population about female genital mutilation and gender-based violence and consequences; b) push for legal regulation on female genital mutilation abandonment. However, there is not a national strategy focused on gender-based violence to organize interventions under a unified plan. 52 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity EVALUATION QUESTION 6 – FUTURE PERSPECTIVES: Complementarity between USAID activities 6.1. HSD-related activities were designed to build on prior USAID-supported activities and complement concurrent USAID and other development partner￾supported programming. Recent USAID funding to improve the Guinean health sector demonstrates continuity, reinforcement, and expansion of targets, geographies, and strategies across the Maternal and Child Health Indicator Program (MCHIP 2010-2014), the Maternal and Child Survival Program (MCSP 2015-17), and the HSD Activity (2015-2020), all targeting RMNCH+ care (Table 11). Complementary and supportive programming has been provided through the StopPalu (2013-17), StopPalu+ (2017-2022), Health Communication Capacity Collaborative (2012-17), Breakthrough Action (2017-2022), the Global Health Supply Chain Program (2012-20) and the Citizen Involvement in Health Governance (2017-2020) projects, in addition to Ebola￾specific funding. USAID financial investment has ranged from $15 million (2010) to $165 million (2015) over the last 10 years, with an overall investment of $394 million disbursed from 2010 year to 2020 year.6 Key stakeholders had a strong appreciation of USAID’s continued focus on RMNCH+. Project documents and interviewees noted Jhpiego’s commitment and thoughtful assessment of health system needs and challenges, as well as collective efforts among partners, with the goal of coordinating with organizations implementing similar programming to avoid duplication of effort. Similarly, stakeholders reported perceiving that the various USAID-sponsored activities were providing non-overlapping contributions to improving RMNCH+ in Guinea. They noted USAID’s practice of generally working with the same partners facilitates synergies in programming, particularly related to renovation of health facilities, training, and equipment provision across projects. While programming across HSD, StopPalu+, Breakthrough Action and Citizen Involvement in Health Governance was largely perceived as complementary, some interviewees perceived some activity redundancy (e.g., Breakthrough Action and HSD implementation of community-based health care demand creation activities) while others felt that opportunities for collaboration were not fully taken advantage of across projects. Indeed, recognition of the need for project specific demand creation activities was the rationale for inclusion of the community component in HSD alongside the HC3 and Breakthrough Action interventions. With Breakthrough Action, for example, common messages were created to reinforce each project’s community-based outreach activities. Striking a balance between redundancy and the sizable scope of work required to scale such activities across regions at multiple levels and through varying approaches was also recognized as a considerable challenge to reconcile (KII National). Despite efforts of implementing partners and USAID to facilitate collaboration, challenges in addressing differing agendas of partners and their respective projects and activities, as well as the inevitable staggering of project timelines sometimes resulted in misalignment between projects. “USAID really has projects that complement each other. If one begins, the other completes.” (Facility manager, Kindia) “The USAID projects that have supported the HCs have been complementary because when Jhpiego intervenes in obstetric planning and care, there is Stop Palu+, which is there for malaria prevention and management.” (Facility manager, Kindia) 6.2. Synergies across USAID activities were felt to be particularly effective during the Ebola epidemic, largely due to enhanced coordination across all actors and sectors. Respondents appreciated the increased communication and collaboration opportunities provided through regular meetings and shared tasks across partners. A mayor of one town, for example, started to participate in the local hospital’s monthly meetings which began during the Ebola crisis, a practice he maintained after the crisis subsided and which built greater community involvement in health. The monthly DRS and DPS meetings are another example Integrated Health Service Delivery in the Post-Ebola Context 53 of heightened engagement of Ministry of Health managers and local officials that began under the Ebola period and were facilitated to extend to today by USAID projects (HSD, Stop Palu+). HSD and Stop Palu+ also made complementary strides in support of antenatal care protocols and quality performance improvements through collaboration both in standards development as well as in joint monitoring and support for maternal health care at the facility level. Indeed, complementarity worked best between projects when they were intervening in common service levels and domains. 6.3. Missed opportunity: Efforts to design activities to be complimentary were recognized, but fixed boundaries and outcome measures between project activities created implementation gaps. As noted in section 3.2, the USAID multi-sectoral approach of allocating supply chain support and improvement interventions to different activities along the levels of the health system resulted in gaps in the commodities and other medicines at the health facility level. While HSD supported provider and managerial capacity development and equipment, supply chain support was implemented by the Global Health Supply Chain Project. Supply chain management support did not cover activities between the health facilities and districts, which was a primary cause of stock-outs at the facility level, particularly contraceptives. Activity concern for meeting internal outputs and outcome measures (indicators) and a perceived inflexibility in project design perpetuated this issue. Recently however, efforts have been made to bridge the supply chain gap, but the modifications in approach came late to the project. Greater efforts should be made for USAID-supported projects to reinforce each other, adapt to the evolving context when service delivery gaps emerge, and work with health systems partners to mitigate the challenges, in the future (see Q3). USAID regularly convenes partner meetings, annual planning and performance reviews where implementation challenges and gaps such as those noted above could be addressed. There appears to be some reticence on the part of partners and possibly USAID to significantly modify project implementation schedules as such changes can delay financial flows to ongoing activities or reflect poorly on implementers. Better communication, coordination, and license to adjust goal posts of projects could help to overcome such challenges and mitigate gaps in project implementation. Increased and collective planning by partners working together was requested by respondents to strengthen governance, planning, monitoring, evaluation, and health financing. Project/ Activity implementers were also reluctant to take responsibility for gaps that fall outside of their implementation plan and results framework, likely because these additional activities were not funded. Respondents noted the need for better problem-solving mechanisms to resolve such issues. For example, inclusion of annual course correction activities including assignment of responsibility, and specification within RFP/As, were suggested ways to overcome these challenges. 6.4. Cross-partner activities with USAID-funded and non-USAID funded activities were well-leveraged by USAID. For example, USAID support of cross-partner meetings including donors and implementers provided an opportunity for partners to leverage their activities amongst themselves to better align and coordinate common health sector interventions in support of government programs. For example, the support for district-level monthly review meetings where data is presented and key health system problems solved was initially supported by Stop Palu+, then increasingly shared the responsibility for support and facilitation (participation) with HSD. Despite positive efforts, it was acknowledged that government ownership of cross-project coordination must be increased, and an integrated plan of all project interventions must be developed to consolidate best practice. Government coordination of donor activities and support was extremely limited with no visible plan in place to organize bilateral cooperation. The effectiveness of the monthly meeting is not evident for the extensive management efforts required to convene them, particularly when Ministry of Health ownership is low. Nonetheless, ongoing monitoring between programs is necessary for optimizing gains across projects. 54 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Specific partnerships such as with UNFPA, GIZ, WHO, OPROGEM, Catholic Relief Services, ACOPED, Terre des Hommes, Red Cross and UNICEF were mentioned. Areas of complementarity included family planning, antenatal care, and infection prevention and control. Examples of partners with synergistic activities included the WHO, which intervenes on community prevention of diseases with epidemic potential, UNFPA, which intervenes in the framework of family planning and gender-based violence, and, UNICEF, which equips facilities with refrigerators, incinerators, and improved wells, and even with supportive services such as security to improve gender-based violence. HSD collaborated in various ways, such as following up with monitoring of activities originally implemented by other partners: “If we take, for example, GIZ, which came with the partograms at the beginning and trained the staff but HSD continued the monitoring and evaluation and the quality of the data.” (Facility Manager, Mamou) 6.6. Missed opportunities: USAID-supported projects were felt to be covering many issues, but support of other donors is still needed. USAID’s long-term commitment and investment in the primary health care sector and family planning is unparalleled and much appreciated. As a result, other donors give way to RMNCH+ as USAID is considered to be the provider of that support. Likewise, the government in part absolves itself of some responsibility for financing and coordinating specific health sector interventions such as training which they see as the purview of USAID-supported projects. Further dialogue, planning, and collaboration among donors and accountability measures with the government are needed. Stakeholders observed that there were missed opportunities for better synergy and advancement of results, both internally within USAID and externally, including development of laboratory capacity, community social and behavior change communication, human resources for health deployment and management, leadership and accountability of senior managers, and SBM-R data management. These needs are described further in previous sections (Sections 1.2, 2.2 and 2.6, 3.1, 3.2, and 3.3) 6.7 Post-investment sustainability of HSD: After nearly four years of USAID support under HSD, the Government of Guinea has an additional distance to travel on their “journey to self-reliance”. Despite USAID’s long-term and continued investments in RMNCH+ and FP, sustainability of inputs by the Ministry of Health at all levels has yet to be assured through substantial government financial and managerial resources and leadership. New ways of designing and supporting the Government of Guinea may be needed to ensure activities are led by the government with partners playing a supporting role. For example, use of composite indicators that are shared between USAID health and governance programs could facilitate partnership, complementarity, and be designed to create greater ownership by the Government of Guinea for supported activities’ results. As discussed in Findings 3.2.2, 3.5.1, and 3.5.2, there is a need to rethink the sustainability strategy due to continued accountability issues to secure USAID investments into the future. Integrated Health Service Delivery in the Post-Ebola Context 55 1. AVAILABILITY OF INTEGRATED CARE AND QUALITY OF CARE Delivery of timely, quality, integrated of RMNCH+ care. Recommendation 1.1. Target engagement of clients throughout the continuum of care, with a particular focus on improved engagement in the postpartum period, and institutionalize functional referral systems to strengthen quality of care improvements. Substantial progress was made in HSD-targeted facilities in terms of improving perinatal care. However, further efforts are needed to increase the proportion of women receiving the full continuum of perinatal care which has remained low. Targeted strategies are particularly needed to ensure continued engagement through the postpartum period, particularly for certain vulnerable populations. (Relates to Finding 1.5) Health cards used by providers and held by women could be leveraged for monitoring and promoting the continuum of care, as the postnatal period was most often found empty in these cards. CHW/RECO could play a larger role in monitoring and encouraging women to complete their full care continuum, beyond antenatal care attendance, and postpartum care could be better integrated with child vaccination services.40,41 Additional improvements to quality of care are possible through further investment in referral. Future USAID￾supported activities should prioritize supporting comprehensive referral systems, from the community through to tertiary care if needed. Institutionalization of a functional referral system through dedicated funding and facilitation of coordination, partnership with communities, and innovation is urgently needed to mitigate delays in access to emergency care, particularly for the poorest households. As referral systems are developed and strengthened, including support for transportation and community sensitization, other methods for bringing women closer to care could be facilitated, e.g., maternity waiting homes, support for accompaniment, etc.42,43 (Relates to Findings 2.1.2) Recommendation 1.2. Augment or continue support for integrated critical services for the most vulnerable, including adolescents and youth, women in need of fistula surgery, and survivors of gender-based violence. HSD has made strides towards improving integrated care but critical populations, particularly the most vulnerable, and specific services need further attention. Reorienting care to meet the specific needs of adolescents and youth, who represent 41% of the population, is critical. There are no specific service standards on counseling and service provision for younger women’s sexual and reproductive health needs. Given the demographic imperative, USAID should urgently work with the Ministry of Health and partners to develop service standards, training, and indicators to measure improvements in service quality for adolescent girls and young women.44 (Relates to Findings 2.6) Likewise, pre-service training at the midwifery, nursing, and medical schools and faculty should include curricula on youth friendly service provision. (Relates to Findings 3.4.2) Risks of obstetric fistula continue, and numerous Guinean women remain in need of fistula surgery, but fistula treatment, care, and reintegration programming cannot be sustained without targeted investment. As a gender and human rights issue, fistula services should continue to be a critical component of future investments in women’s health by donors and the government to ensure service availability. Innovative partnerships across government, donors, private sector, community, and civil society may be one way of securing the necessary resources. Support must also target women in need of reintegration services and consistently implement follow-up care to improve health outcomes in subsequent pregnancies. (Relates to Findings 4.1) KEY CONSIDERATIONS & RECOMMENDATIONS 56 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Gender-based violence, harmful practices and cultural norms continue to be pervasive in Guinea. HSD’s investment in gender-based violence was not meant to address the full scope of these challenges. Future investment in gender-based violence should seek to increase intervention scale and expand GBV prevention and treatment efforts to include community engagement and community-based training, national and regional governance, and stronger coordination and alignment across Ministries and development partners. Gaps in stewardship of cases between the implicated ministries need to be addressed at all levels. If funding limits persist, leveraging investment to support strengthened governance, coordination and social and behavioral change communication messaging should be a primary focus. Expansion of the focus of gender-based violence activities to include other prevalent harmful practices in Guinea such as female genital mutilation is needed. (Relates to Findings 5.1, 5.2, 5.3) 2. THE LAST MILE: Reaching the most vulnerable at the community level Despite significant contributions to improving service quality and utilization across Guinea, HSD had difficulty reaching the most remote and vulnerable. HSD was designed to focus on improving quality of emergency obstetric and neonatal care in health centers and hospitals. Opportunities remain for strengthening community-level components and targeting interventions to improve equity. Important groundwork was laid by HSD and the Ministry of Health in the recent year for establishing policy and normative documents to bring care closer to the community; planning that will now require significant decentralized investment to be implemented as envisioned. The following recommendations could assure future investments to address this gap. COMMUNITY-LEVEL INTERVENTIONS Recommendation 2.1. Support the Government of Guinea to implement the Community Health Policy and integrate community health services To foster implementation and sustainability of the Community Health Policy and the integrated community health service package, support is needed to create the conditions for CHW/RECO to perform their new tasks. Inputs to community health should promote integration as guided in the Norms and Procedures for Reproductive Health, to begin the shift from vertical CHW/RECO interventions to more integrated outreach. For example, vaccination/child health days facilitated by CHW/RECO can be linked to postpartum care and support for breastfeeding mothers. USAID support for the community health policy should be preceded by financial support for human resources that remain the responsibility of government through decentralized funds. To hold the government accountable to the policy and given the HSD experience, using project funds for paying community level workers is not recommended, but rather strengthen the capacity of the new cadres of CHW/RECO through tools, supervision, and training. (Finding 2.2) USAID should also consider support for innovative approaches to co-creation of community level integrated activities with community leaders and stakeholders to further support CHW/RECO in their new role. (Relates to Findings 2.2, 3.5) Recommendation 2.2. Strengthen community engagement through support for the Community Action Cycle, community groups, and local health posts The community action cycle and community groups including COSAH, community action groups, and mobilization teams working with CHW/RECO can engage with communities to raise awareness and rebuild trust in the health system and create demand for health services. Such entities should continue and expand their reach by working with community and religious leaders. Innovation in co-funding schemes with communities as an expansion of COSAH and community action group activities and the community action cycle Integrated Health Service Delivery in the Post-Ebola Context 57 (with training and supervision) could add value to the growing infrastructure of CHW/RECO and COSAH being scaled up as part of national policy. (Findings 2.1, 2.4, 2.6) Recommendation 2.3. Social accountability for removing financial, gender and cultural barriers for the most vulnerable including through addressing service fees for critical services and supporting community health mutuals. Barriers of finance, age, and gender continue to impede service utilization and RMNCH+ outcomes in Guinea. Meeting the needs of the most vulnerable individuals deserves greater attention in HSD and future programming to ensure equity in the design, implementation, measurement and reporting of activities. (Relates to Findings 1.3. 2.6, 5.1) Access to RMNCH+ services continues to be limited by out-of-pocket payment demands for specific services. There is an urgent need to improve policy and enforcement of free RMNCH+ financial access barriers. Upstream advocacy by USAID is needed to revisit the policy of 2011 attesting free RMNCH services to have better clarity and enforcement within the context of universal health care promotion in Guinea. (Finding 2.5) Downstream community efforts to promote community health mutuals should also be strengthened and scaled up to cover small costs, transportation, and meals for patients. Supporting communities by informing them of their health rights can help them to mobilise social accountability actions around illegal costs and other barriers further raising the stakes for the Government of Guinea to address access to quality care issues. Such awareness raising should go beyond the written/ poster communication channel that was employed by HSD and other programs to include a rights-based information, communication, and education campaign to make communities, as rights holders, aware of the entitlements and make health providers and managers aware of their obligations as duty bearers. (Relates to Findings 2.5, 3.1.2) 3. HEALTH SYSTEMS STRENGTHENING: BUILDING ON SUCCESSES Maintaining health system and service quality improvements achieved through HSD (and past USAID health systems strengthening projects) requires leadership and commitment from the Ministry of Health at all levels. Despite efforts to put in place quality improvement processes such as SBM-R, significant gaps remain. HSD implementers compensated for leadership gaps to maintain project progress and achievements. Local NGO and private sector partners were not stimulated to participate in quality improvement processes by local government or the Ministry of Health. GOVERNANCE Recommendation 3.1. Require accountability, leadership and ownership from implementing partners for sustainability through results-based measures for accountability HSD recognized gaps in health system leadership, oversight, and accountability that offset the sustainability of joint achievements. Capacity building efforts to increase commitment and ownership were acknowledged as insufficient yet course correction to overcome inaction and apathy was not done— possibly because they were considered unlikely to make a significant difference given the health system context. Inadvertently, HSD’s efforts and interventions to compensate for leadership gaps may have perpetuated the problem, as some managers became reliant on partners to lead. Similarly, supply chain ruptures at the facility-level are the responsibility of the Ministry of Health and were not within the purview of HSD. Yet, the lack of responsibility taken by Ministry of Health managers at the district and regional levels to address stock-outs, possibly waiting for HSD to solve the problem, is further evidence of challenged Ministry of Health capacity and/or complacency/willingness to let HSD lead. Future USAID activities can stimulate leadership and ownership by the government and Ministry of Health in their design. Results-based 58 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity measures can require accountability for project deliverables, including successful implementation of standards with functioning equipment, maintenance, and reporting. The design of future projects could also consider transferring responsibility for the project activities over time based on performance (results￾based) measures. Co-design of projects to address critical needs of health facility staff, such as access to water, electricity and provider accommodation, can also facilitate increased ownership. (Relates to Findings 1.3, 3.2.3, 3.5.1) QUALITY IMPROVEMENT PROCESSES The SBM-R quality improvement program has the potential to be an important, sustainable process by which health systems and service quality becomes a collective effort of communities, local government, facility staff and Ministry of Health managers at district, regional, and national levels. HSD’s approach already has yielded momentum, commitment, and success at some facilities, particularly at smaller hospitals. Some communities, providers, and health managers have voluntarily engaged in the process, offering a recognition model of quality improvement that can be managed within the constraints of the existing health system. Yet, some constraints exist. Communities are not fully engaged in the process, despite some strong examples of their responsiveness to the SBM-R stars as a symbol of quality. Referral hospitals with higher caseloads have struggled to engage and thus have benefited less. The national Technical Committee validating SBM-R is not yet fully functional, delaying external facility validation and limiting necessary oversight. Documentation of the SBM-R checklist results by facility is not readily available for analysis and use to inform facility- or district-level efforts to achieve “stars”— an overview that computerized data by targeted facility could provide. SBM-R has not been institutionalized by the Ministry of Health within existing efforts by the government such as the Improved Monitoring program (Monitorage amelioré); alignment between these programs is necessary for sustainability. SUSTAINABILITY Recommendation 3.2. Align SBM-R with existing quality improvement measures of the Government of Guinea and Ministry of Health and advocate for an institutionalized quality improvement approach system-wide To sustain SBM-R as an effective quality improvement measure, HSD should support the institutionalization and systematization (possibly through tablet-based data collection) of the SBM-R process. Targets and progress should be tracked and benchmarked, challenges should be discussed at subnational—and, if common—national level to be collectively addressed. SBM-R can be a unifying process to facilitate use of data for decision making and action by the Ministry of Health. HSD can facilitate the process by working with the Ministry of Health to own the process and provide the necessary leadership through the appropriate national directorates (Family Health and Nutrition, Bureau of Strategies and Development, national health information systems, and the university). Cultivating further facility-based support will be needed to expand and sustain the success of the SBM-R process given quality improvement is a continuous process that must be maintained by the facilities themselves. (Relates to findings 1.3, 1.5, 1.6, 3.1.1, 3.1.2, 3.5.2) Recommendation 3.3. Continue support to the DHIS2 health information system to achieve full implementation to facilitate evidence-based decision making HSD has made significant contributions to a number of critical health system structures that require additional future investment. Through the SBM-R process and data monitoring and review (at monthly meetings), HSD has contributed towards the institutionalization of data use for decision-making to advance quality improvement. Past investment in DHIS2 were further supported by HSD. To ensure DHIS2 is fully-functional and utilized to its potential in Guinea, additional technical and managerial support is needed. Beyond putting the software in place, emphasis should be on supervision of data quality, timeliness of reporting, analysis at sub- Integrated Health Service Delivery in the Post-Ebola Context 59 national level to facilitate use of data at district and facility levels to improve quality of care for better health outcomes. (Findings refer to 3.3.1) Recommendation 3.4. Scale up support for pre￾service midwifery and nursing education and skills labs as an important investment for ensuring the evidence-based, quality clinical service delivery HSD investment in public and private midwifery and nursing schools is novel and important. The activity has demonstrated success having put in pace updated curriculum with current RMNCH+ performance standards and a skills lab to provide practical training on critical procedures. The fact that the new skills labs are being used by other schools indicates the value placed by regional stakeholders on the learning resource. Ownership and pride by the school management of the skills labs assures maintenance and security of the equipment. Such investments show potential for sustainability by the schools and should be sustained and expanded as a model. The momentum created by this training component should be maximized. There is a need to create laboratory skills in health schools in all administrative regions with the support of Ministry of Health (Findings refer to 3.4.2) Recommendation 3.5. Strengthen facility-based management of equipment, infrastructure, and supplies, which is fundamental to the provision of quality integrated services. HSD facilitated implementation of performance standards by making the necessary equipment available for their implementation, and by contributing to small infrastructure improvements. These critical contributions were greatly appreciated by facility staff and community members who reported facility improvements as a strong motivator for engagement in quality improvement processes. Specific infrastructural investments identified as most influential by key stakeholders include those supporting electricity and water, and those mentioned as critically lacking include laboratories (Findings 3.4.2 and 6.6), and space for private and confidential care for family planning and antenatal care services in order to improve respectful care and adolescent-friendly care. (Findings 2.1.1, 3.2.1 and 3.2.2) Despite the value placed on such investments, they were not embedded into a broader strategy to ensure maintenance, oversight, and accountability for upkeep of equipment of the facility improvements. Equipment malfunction, misuse, and theft impacts quality of care provided despite best intentions. Feedback mechanisms and accountability with punitive measures are needed in the system to counter impunity. Lack of managerial accountability and control of equipment donations and facility improvements at facility and district level should be addressed through strategies to ensure respect for the agreements between projects and the Ministry of Health accompanied by close monitoring, recourse mechanisms, and reporting. (Relates to Findings 3.1.4) 4. USAID AND PROGRAM OVERSIGHT: Programming for sustainability HSD activities are built on prior USAID-supported activities and complement concurrent USAID and other development partner-supported programming. This approach was successful in terms of continuity from prior projects and collaborative efforts across partners to improve coverage, although challenges were noted in communication, scope, priorities, and logistics that were not managed sufficiently, creating implementation gaps. LEVERAGING, ADAPTATION, AND PROBLEM SOLVING Recommendation 4.1. USAID should leverage its health sector investments for improved collaboration, communication and impact between its projects/activities and others in the health development field Increased communication and more synergistic management of programming across RMNCH+ implementing groups during the Ebola outbreak was an important lesson for moving towards greater collaboration and impact. Building on these lessons from the Ebola period, USAID should improve collaboration and monitoring of activities between supported development partners and their activities/ 60 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity programs to collectively optimize gains across projects. Institutionalizing cross-project synergies is possible through the USAID mission by improving alignment of resources, timing, sectors, and actors. Regular meetings, cross project and joint milestones and indicators can facilitate cooperation and collective effort. Additional mechanisms could be established between projects (including other development partners and the Ministry of Health) through capacity development activities to avoid duplication of activities and approaches between projects and ensure appropriate programmatic coverage across demonstrated needs (e.g., topical areas, geographies, supervision, and management). USAID monthly meetings can be useful for aligning activities but they must be accompanied by continuous monitoring between programs to optimizing gains across projects. (Relates to findings 6.1, 6.3, 6.4, 6.5) Recommendation 4.2. Flexibility in project design, implementation and outcome measures are needed to adapt to changing context and needs Fixed programming boundaries across USAID￾supported activities limited the ability of projects to rapidly adapt when important systems components fell between the cracks. Lack of flexibility in activity design perpetuated this issue. Institutionalization of mitigation measures for improving adaptive response across USAID activities is needed as the lack of a clear contingency plan limits achievements of all projects. Greater efforts should be made for USAID-supported projects to reinforce each other, adapt to the evolving context when service delivery gaps emerge, and work with health systems partners to mitigate the challenges, in the future. Further, success of USAID activities could be improved through encoding shared responsibility (and measures of success) between projects (e.g. for supply chain functioning across the health system from facility to national level management) and ensuring regular meetings and annual course correction reviews. Assignment of responsibility and specification within the request for proposal were suggested ways to overcome these challenges, among others. Ensuring collaborating, learning and adapting strategies in future projects both involve key government, implementing partner, and USAID focal points in building adaptive management capacity, and include an evaluation of the collaborating, learning and adapting strategy, would help to address this. (Relates to finding 6.3) DESIGN FOR SUSTAINABILITY Recommendation 4.3. Design projects collaboratively to increase ownership and devolve responsibility to government partners HSD (and past activities) have focused on service delivery improvements and quality of care, with each new activity building on some components of the preceding activity. Sustainability, as observed by Guinean health service staff, has essentially become guaranteed through this generational project approach. HSD was not designed with a plan to devolve to the Ministry of Health. To move Guinea forward on the “Journey to Self-Reliance”, activities must be designed for sustainability with participation from Government counterparts to build in realistic plans for gradually expanding government ownership, commitment and leadership. Within project design, theories of change should be explicit about how the project will build ownership and leadership over time. (Relates to finding 3.5) Recommendation 4.4. Consider demonstrating the full effectiveness of investment activities by prioritizing depth of investment HSD targeted seven regions in the country with the aim of improving the quality of integrated RMNCH+ care in selected facilities. The breadth of the intervention package required continuous training, monitoring, data collection, and reporting. Continuous training efforts were needed to bring the full package of interventions to all the facilities and staff given staff turnover and retention issues. Given the quality of RMNCH+ delivery and care, the health status of the population, the financial resources available to the project, the timeline of resource availability, and the known contextual challenges, HSD activities emphasized breadth (reaching as many regions and facilities as possible) rather than depth (concentrating on proof of concept Integrated Health Service Delivery in the Post-Ebola Context 61 in a subset of regions and facilities). HSD and future activities could consider an alternative approach of providing the full package of integrated activities within a more limited geography. This would facilitate the provision of more specific attention and mentoring to a subset of facilities to demonstrate how success can be achieved as envisioned by the project. (Relates to finding 6.3) In further defining a strategy, USAID will need to consider certain priorities for breadth versus depth of coverage. 62 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity REFERENCES 1. Central Intelligence Agency. The World Factbook: Guinea. 2018. https://www.cia.gov/the-world-factbook/ countries/guinea/ (accessed October 10 2019). 2. Institut National de la Statistique, Ministère du Plan et du Développement Economique. Annuaire Statistique 2016. Conakry, Guinée: INS, 2017. 3. Institut National des Statistiques. Guinea Demographic and Health Survey 2012. Conakry: Institute National des Statistiques, MEASURE DHS, 2014. 4. Alkema L, Chou D, Hogan D, et al. Global, regional, and national levels and trends in maternal mortality between 1990 and 2015, with scenario-based projections to 2030: a systematic analysis by the UN Maternal Mortality Estimation Inter-Agency Group. Lancet (London, England) 2016; 387(10017): 462-74.PMC5515236 5. Institut National de la Statistique MdPedDE, République de Guinée,, The DHS Program I. Enquête Démographique et de Santé 2018 (EDSG V). Conakry, Guinée: Institut National de la Statistique, Ministère du Plan et du Développement Economique, République de Guinée 2019. 6. USAID Data Services. Foreign Aid Explorer. 2020. https:// explorer.usaid.gov/cd/GIN. 7. Balde MD, Bangoura A, Diallo BA, et al. A qualitative study of women’s and health providers’ attitudes and acceptability of mistreatment during childbirth in health facilities in Guinea. Reproductive health 2017; 14(1): 4.5237275 8. Balde MD, Diallo BA, Bangoura A, et al. Perceptions and experiences of the mistreatment of women during childbirth in health facilities in Guinea: a qualitative study with women and service providers. Reproductive health 2017; 14(1): 3.5225581 9. Maheu-Giroux M, Filippi V, Samadoulougou S, et al. Prevalence of symptoms of vaginal fistula in 19 sub￾Saharan Africa countries: a meta-analysis of national household survey data. Lancet Glob Health 2015; 3(5): e271-8 10. Evans DK, Goldstein M, Popova A. The Next Wave of Deaths from Ebola? The Impact of Health Care Worker Mortality. Policy Research Working Paper 7344. Accra, Ghana: World Bank Group, Africa Region. Office of the Chief Economist., 2015. 11. Institut National de la Statistique (INS), ICF. Enquête Démographique et de Santé en Guinée 2018. Conakry, Guinée et Rockville, Maryland, USA: INS et ICF, 2018. 12. Delamou A, Dubourg D, Delvaux T, et al. How the free obstetric care policy has impacted unmet obstetric needs in a rural health district in Guinea. PloS one 2015; 5(10): 6 13. Ministere de la Sante RdG. Plan National de Developpement Sanitaire (PNDS) 2015-2024. Conakry, Guinee: Ministere de la Sante, Republique de Guinee, 2015. 14. Chen L, Evans T, Anand S, et al. Human resources for health: overcoming the crisis. Lancet (London, England) 2004; 364(9449): 1984-90 15. Coordination Nationale Ebola Guinée, World Health Organization. Rapport de la Situation Epidémiologique, Maladie à Virus Ebola en Guinée du 29 Decembre 2015. Sit_Rep_no 623. Conakry: Coordination Nationale Ebola, Guinée, 2015. 16. World Health Organization. Health worker Ebola infections in Guinea, Liberia and Sierra Leone. Geneva, Switzerland: World Health Organization, 2015. 17. Centers for Disease Control and Prevention NCfEaZIDN, Division of High-Consequence Pathogens and Pathology (DHCPP), Viral Special Pathogens Branch (VSPB). 2014- 2016 Ebola Outbreak Distribution in West Africa. 2019 (accessed October 10 2019). 18. Coordination Nationale Ebola in Guinée O. Rapport de la Situation Epidémiologique, Maladie à Virus Ebola en Guinée du 28 Février 2015. Sit. Rep. No 319. Conakry, Guinea: Coordination Nationale Ebola Guinée and Organisation mondiale de la Santé, 2015. 19. Présidence de la République de Guinée. Troisième recensement général de la population et de l’habitat 2014., Décret D/2015/225/PRG/SGG. Conakry, Guinée: Institut National des Statistiques, 2015. Integrated Health Service Delivery in the Post-Ebola Context 63 20. Delamou A, Hammonds RM, Caluwaerts S, Utz B, Delvaux T. Ebola in Africa: beyond epidemics, reproductive health in crisis. Lancet (London, England) 2014; 384(9960): 2105 21. Menendez C, Lucas A, Munguambe K, Langer A. Ebola crisis: the unequal impact on women and children’s health. Lancet Glob Health 2015; 3(3): e130 22. Black BO. Obstetrics in the time of Ebola: challenges and dilemmas in providing lifesaving care during a deadly epidemic. BJOG 2015; 122(3): 284-6 23. Delamou A, Beavogui AH, Konde MK, van Griensven J, De Brouwere V. Ebola: better protection needed for Guinean health-care workers. Lancet (London, England) 2015; 385(9967): 503-4 24. Fischer WA, 2nd, Hynes NA, Perl TM. Protecting health care workers from Ebola: personal protective equipment is critical but is not enough. Ann Intern Med 2014; 161(10): 753-4 25. Tsoidras S, Tsakris A. Ebola virus and childhood immunication lapse: another hidden public health risk. BMJ 2014; 349(g7668): 26. Thiam S, Delamou A, Camara S, et al. Challenges in controlling the Ebola outbreak in two prefectures in Guinea: why did communities continue to resist? Pan Afr Med J 2015; 22 Suppl 1: 22.4695515 27. Omidian P, Tehoungue K, Monger J. Medical Anthropology Study of the Ebola Virus Disease (EVD) Outbreak in Liberia West Africa. Geneva, Switzerland: World Health Organization, 2014. 28. Brolin Ribacke KJ, Saulnier DD, Eriksson A, von Schreeb J. Effects of the West Africa Ebola Virus Disease on Health￾Care Utilization – A Systematic Review. Frontiers in public health 2016; 4: 222.PMC5056406 29. Ly J, Sathananthan V, Griffiths T, et al. Facility-Based Delivery during the Ebola Virus Disease Epidemic in Rural Liberia: Analysis from a Cross-Sectional, Population-Based Household Survey. PLoS Med 2016; 13(8): e1002096 30. Barden-O’Fallon J, Barry MA, Brodish P, Hazerjian J. Rapid Assessment of Ebola-Related Implications for Reproductive, Maternal, Newborn and Child Health Service Delivery and Utilization in Guinea. PLoS Curr 2015; 7.4542265 31. Lori JR, Rominski SD, Perosky JE, et al. A case series study on the effect of Ebola on facility-based deliveries in rural Liberia. BMC pregnancy and childbirth 2015; 15: 254.4603295 32. Iyengar P, Kerber K, Howe CJ, Dahn B. Services for mothers and newborns during the ebola outbreak in liberia: the need for improvement in emergencies. PLoS Curr 2015; 7.4404271 33. Delamou A, El Ayadi AM, Sidibe S, et al. Effect of Ebola virus disease on maternal and child health services in Guinea: a retrospective observational cohort study. Lancet Glob Health 2017; 5(4): e448-e57 34. Leno NN, Delamou A, Koita Y, et al. Ebola virus disease outbreak in Guinea: what effects on prevention of mother-to-child transmission of HIV services? Reproductive health 2018; 15(1): 60.PMC5891943 35. United States Agency for International Development. USAID/Guinea Country Development Cooperation Strategy 2015-2020. . Washington, DC: USAID, Updated Dec 19, 2019. 36. Marchal B, van Belle S, van Olmen J, Hoerée T, Kegels G. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation 2012; 18(2): 192-212 37. Jhpiego, EngenderHealth, Save the Children. Health Services Delivery Project Guinea, Annual Report October 2018-September 2019. Conakry, Guinea: Jhpiego, 2019. 38. WHO. Obstetric Fistula: Guiding Principles for Clinical Management and Programme Development, 2006. 39. Delamou A, Delvaux T, El Ayadi AM, et al. Fistula recurrence, pregnancy, and childbirth following successful closure of female genital fistula in Guinea: a longitudinal study. Lancet Glob Health 2017: 40. Djellouli N, Mann S, Nambiar B, et al. Improving postpartum care delivery and uptake by implementing context-specific interventions in four countries in Africa: a realist evaluation of the Missed Opportunities in Maternal and Infant Health (MOMI) project. BMJ global health 2017; 2(4): e000408.PMC5717926 64 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 41. Yugbare Belemsaga D, Goujon A, Bado A, et al. Integration of postpartum care into child health and immunization services in Burkina Faso: findings from a cross-sectional study. Reproductive health 2018; 15(1): 171. PMC6180606 42. Dadi TL, Bekele BB, Kasaye HK, Nigussie T. Role of maternity waiting homes in the reduction of maternal death and stillbirth in developing countries and its contribution for maternal death reduction in Ethiopia: a systematic review and meta-analysis. BMC health services research 2018; 18(1): 748.PMC6167854 43. Geleto A, Chojenta C, Musa A, Loxton D. Barriers to access and utilization of emergency obstetric care at health facilities in sub-Saharan Africa: a systematic review of literature. Systematic reviews 2018; 7(1): 183. PMC6234634 44. Mazur A, Brindis CD, Decker MJ. Assessing youth-friendly sexual and reproductive health services: a systematic review. BMC health services research 2018; 18(1): 216. PMC5870922 Integrated Health Service Delivery in the Post-Ebola Context 65 ANNEXES 1. Evaluation Statement of Work 2. Evaluation Matrix 3. Data Collection Instruments 4. Sources of Information 5. Strategy Reference Group 6. Disclosure of Conflicts of Interest 7. Data Tables 8. SBM-R Case Study 66 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity ANNEX I: EVALUATION STATEMENT OF WORK STATEMENT OF WORK FOR FINAL EVALUATION OF USAID GUINEA HEALTH SERVICES DELIVERY (HSD) ACTIVITY Revised July 2019 BACKGROUND AND CONTEXT The maternal, neonatal and child health situation in Guinea is dire despite targeted development interventions over the past years. Though preliminary results of the latest Demographic Health Survey (DHS 2018) indicate some improvements, mortality rate remains high. The child mortality rate among children under the age of five reduced to 111 deaths per 1,000 live births from 123 in 2012. The infant mortality rate is estimated to have been reduced from 67 in 2012 to 66 deaths per 1,000 live births. Neonatal mortality remains at 34 deaths per 1,000 live births. Fever, diarrhea and acute respiratory infections remain the main cause of neonatal and postnatal mortality. There are also recorded changes in key maternal and child health and family planning indicators. Levels of improvement across the different indicators vary across regions, with some of the poorest performance in current USAID regions of interventions. During this period, Guinea suffered the catastrophic 2014 – 2015 Ebola epidemic that claimed the lives of 2,254 people including xxx health service providers. A significant drop in health facility attendance during this period impacted all services, including maternity services, thus increasing the risk of maternal and infant mortality. USAID Guinea has invested substantial resources targeted at achieving improvement in the provision of maternal and child health services in Guinea over the past years. This has been through standalone maternal child health related activities aimed at rebuilding the health system and improving the delivery of services. Complementary activities on the prevention of malaria also aim at reducing malaria related mortality among women and children. Health systems strengthening activities focused on improving availability of essential medicines and availability and use of strategic information alongside targeted health governance at different levels have also been implemented. Additional investments through the USAID Washington Ebola Pillar II Recovery investments from 2015-2018 were also targeted to restoring critical services including reproductive, maternal and child health services and strengthening health systems including the adoption of effective infection prevention and control measures. Annex 1 contains a list of USAID activities implemented in Guinea since 2015. USAID Guinea launched a flagship five year Health Service Delivery (HSD) Activity in December 2015 to support the provision of an essential and integrated care package (EICP) for maternal, neonatal child health (MNCH) and family planning (FP) in a consistent and high-quality manner in health facilities and surrounding communities in seven out of eight the regions Guinea i.e., of Boké, Conakry, Kindia, Mamou, Faranah, Kankan and Labé. The integrated package is meant to be implemented through the household-to-hospital continuum of care, where the Activity’s key interventions will ensure that high-quality care is available at each level, as well as access to information and referral for serious illness, obstetric fistula (OF) and long-acting and permanent contraception. This will be achieved through three core objectives; a) delivery of quality health services improved; b) Healthy behaviors and demand for quality health services improved and c) health systems Integrated Health Service Delivery in the Post-Ebola Context 67 Boké* Gaoual* Koundara* Mali* Beyla Conakry* Coyah* Kérouané Kankan Ko Mandiana uroussa Siguiri Yomou NzérékoréLola Macenta Guéckédou Kissidougou Faranah Dinguiraye* Tougué* Koubia* Labé* Lélouma* Pita Télimélé Boa* Kindia Mamou Dabola Dubréka* Forécariah* Dalaba Fria* strengthened. HSD also implements activities in support of prevention, detection and treatment of GBV in the community. One of HSD’s cross cutting quality improvement approaches was the Standards Based Management and Recognition (SBM-R); implemented as a proactive, practical management approach for improving the performance and quality of services across 97 facilities. Facilities that meet a minimum of 80% of selected performance standards in FP, EMoNC and IPC are accredited with a Gold Star in a public ceremony signifying that the facility is providing quality services. Typically, SBM-R is accompanied by other interventions to improve providers’ performance and work environment, including in-service training and supportive supervision. In 2019, the HSD Activity was modified to include a component on mobile services outreach and maintenance of health facilities intended to further strengthen community trust and use of health services. The total estimated cost of the HSD Activity was thereby increased from $24,000,000 to $28,800,000 with Guinea Mission bilateral funds and the USAID Washington Global Health Ebola program.6 HSD Activity is implemented by Jphiego in partnership with Engender Health and Save the Children. By the end of the five-year period, the project, in collaboration with the MOH, will have contributed to increasing the availability and quality of an EICP for MNCH and FP at each level of the health system, from the community to the prefectural hospital, at regional and national levels, to mitigate the effects of the Ebola epidemic, restore the community’s confidence in the health system and its use, and improve the health of Guinea’s population. At least 80% of 272 targeted facilities will offer the full range of EICP services consistent with their level in the healthcare system. The HSD Results Framework is attached as Annex 2. As Guinea has emerged from the Ebola epidemic over the past three years, external development support and funding patterns have shifted back to near normal levels. Apart from USAID, other health sector development efforts in Guinea are now led by the Global Fund, UNFPA, the European Union, UNICEF, World Health Organization and the World Bank amongst others. Substantive leadership and policy changes in the country have emerged to create both opportunities and challenges for driving the health reform agenda. Likewise, similar changes within USAID to refocus our efforts on creating self-reliance are demanding a re-evaluation of the way we partner with local actors to build their capacity and commitment to provide and sustain health outcomes. The emergence of new DHS results, uncertain political environment and changing policy priorities coinciding with the end of USAID Guinea’s first Country Development Cooperation Strategy (CDCS) and HSD Activity in 2020 provide a unique opportunity for reflecting on ways USAID’s health resources can be more strategically directed in country to achieve more sustainable results over the next five years. 6 Total Global Health funding in the instrument was USD 7,500,000 initially targeted to be spent within the first two years of the Activity lifespan. Regional HQ in Bold HSD Focus Prefectures HSD Prefectures with extended outreach * PMI/USAID Supported Prefectures 68 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity PURPOSE AND USE OF THE EVALUATION As HSD comes to a close in December 2020, USAID is interested in independently verifying the results achieved and how core strategic interventions have contributed to their achievement.7 This end of activity performance evaluation will establish how USAID can focus its maternal child health and FP interventions to support greater country self-reliance. The evaluation shall point out promising practices / lessons learnt weaknesses / gaps, bright spots that we can build on /expand and new opportunities that should inform the design of a new health activity and overall health strategy for USAID in Guinea over the next five years. Findings and recommendations from this evaluation can also be used by government and other development partners to determine how efforts and resources can be better directed towards strengthening the health system in Guinea. EVALUATION QUESTIONS The evaluation will specifically address the following questions: 1. Essential Integrated care package: To what extent did HSD core approach (i.e. improving the household-to-hospital continuum of care, ensure the availability of high-quality care at each level, access to information and referral for serious illness, strengthened capacity of health care workers) contribute to improved availability and use of essential integrated care package of health services at targeted facilities (about 272 facilities)? Which internal and external factors have positively or negatively affected achievement of these results? 2. Engagement of local actors: How well did the HSD design and implementation engage and strengthen the ability of local actors i.e. government at different levels, civil society, communities and the private 7 This evaluation will focus on the core HSD scope; i.e. it does not cover the expanded outreach and maintenance component. Note that the Guinea Mission is pursuing a separate ongoing assessment of the new component. sector, other development partners and USAID activities to provide and sustain MCH / FP services? 3. Fistula care: To what extent has local technical and managerial capacity been sufficiently built to support fistula prevention and care in Guinea? 4. Gender Based Violence: How well have the limited financial resources been strategically directed to address GBV issues under the HSD Activity? What could have been done differently to enhance achievement of results? 5. Adaptation: How well did implementation of HSD adapt to shifts in the internal and external operating environment? What factors enabled or hindered the program’s ability to adapt to these shifts? 6. Value for money??:- there is some interest in pursuing this line but need to determine feasibility The evaluation team is encouraged to make suggestions/amendments to the evaluation questions in line with the evaluation purpose for approval by the USAID. EVALUATION DESIGN AND METHODOLOGY The evaluation will use mixed methods to collect and analyze relevant information required to respond to the evaluation questions. Suggested data collection methods include: ▶ Review of Program Documents and Related Literature: The team will review several project related documents including solicitation documents, annual work plans, monitoring, evaluation and learning plans, baseline assessment, and progress reports. The team will also review other USAID, district and health sector plans, strategies as well as other strategy policy and performance documents. An illustrative list of documents is attached as Annex 3. Integrated Health Service Delivery in the Post-Ebola Context 69 8 A GoG commissioned health facility survey and DHS, while ideal data sources will not be available in time for this evaluation. ▶ Key Informant Interviews (KII) and Focus Group Discussions: The evaluation team will conduct in-depth interviews and group discussions with program staff at the different levels, partner organizations, stakeholders from the Ministry of Health, district health staff (DPS, DRS), other related USAID implementing partners, USAID Guinea and Washington country team, UNFPA, UNICEF and other development partners, non-government organizations, health facility and health post staff, Health and hygiene committee members, community health workers and community members on their views and perceptions of the program and the kind of changes that have resulted from the program intervention. ▶ Case study: The evaluation team will develop a case study on HSD’s core quality improvement approach, the Standards Based Management and Recognition and its contributions to sustainable health service improvements. This case study will investigate the underlying drivers for quality service improvements and potential for replication, scale up and sustainability of this approach. ▶ Secondary data analysis: The USAID Ebola Pillar II Monitoring, Evaluation and Learning program conducted health facility surveys across targeted USG supported and non USG supported facilities in 2017/18.8 The evaluation team will analyze this data alongside data sets from HSD to determine the effect and contribution of USAID investments to improving service delivery and use. Prospective evaluators are encouraged to propose additional, alternative methodologies that they believe can yield stimulating, robust evidence in answering the evaluation’s questions. The evaluation team will develop tools and detailed guidance for data collection and work closely with HSD staff to identify appropriate respondents. All data collection instruments and guides will be approved by USAID prior to the beginning of fieldwork. Sampling: The evaluation team is expected to propose and use sound sampling techniques to determine prefectures / communes and facilities to be visited as well as stakeholders that will be consulted. Analysis: The evaluation team will propose data analysis strategies and tools for both the qualitative and quantitative data. The team will assess trends in availability and quality of services over the past years across the different HSD supported districts. Data disaggregation and analysis by gender and age to establish the differential effects of the project on men, women and different age groups will also be expected. The team will propose other analysis approaches. An evaluation design matrix is provided in Annex 4 to guide the evaluation team synthesize the evaluation design, approach and methods. This matrix will be provided as part of the scoping trip report POTENTIAL LIMITATIONS: 1. Completeness and quality of baseline data 2. Factoring other factors outside the project’s influence that affect the quality of care such as availability of central government and health facility staff, medicine and other medical products stock out. 3. Determining attribution for cross-cutting system strengthen components of the program that get contribution from other partners SCOPING This evaluation will be kicked off by an in country scoping visit aimed at gaining on ground understanding of the priorities and availability of data to inform the development of a realistic evaluation design and time plan. Through a consultative process, the Contractor will meet with USAID, HSD Implementing Partner staff, key government counterparts to get a preliminary understanding of the activity and how it has been implemented, refine the theory of change, determine 70 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity data availability (quality and quantity) and establish based on perceived interests and priorities for this evaluation. Based on these consultations, the firm shall develop and submit to USAID for approval a realistic evaluation design that includes; ▶ Preliminary findings of the desk review ▶ Articulated theory of change ▶ Detailed evaluation design matrix that links the Evaluation Questions from the SOW (in their finalized form) to data sources, methods, and the data analysis plan; ▶ Data collection tools including draft questionnaires, group discussion guides and other data collection instruments or their main features; ▶ List of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include sampling methodology and methods, including a justification of sample size and any applicable calculations); ▶ Limitations to the evaluation design; and ▶ Dissemination plan (designed in collaboration with USAID) ▶ Evaluation team, their roles and responsibilities ▶ Detailed work plan showing a timeline for each evaluation activity to be undertaken, including the field work and allocation of expertise efforts ▶ Evaluation budget proposal OTHER DELIVERABLES AND REPORTING REQUIREMENTS FOLLOWING THE SCOPING VISIT ▶ In country in-brief: Upon arrival in country the evaluation team will hold an in-brief with USAID, and HSD staff as part of the team planning exercise. The in brief will include an introduction of the evaluation team and discussion of the scope of 9 The Evaluation Contractor is expected to schedule team planning calls / meetings even before and after the team arrives in country to enable the evaluation team meet to form common understandings of the assignments, discuss individual roles and tasks, and plan for the assignment. work, initial presentation of the proposed evaluation methodology and work plan, and other emerging issues that may affect the evaluation.9 ▶ Weekly Progress Reports: Brief informal reports highlighting progress, challenges and constraints and describing evaluation team’s response. ▶ Exit Brief / oral presentation: Following field work and initial analysis, the evaluation team will present preliminary findings and conclusions to USAID and the Implementing Partner through an in-person presentation using Power Point. This meeting and presentation will be conducted in English. The oral presentation should, at a minimum, cover the major findings, conclusions, recommendations for improvement and key lessons. The evaluation team will liaise with the mission to agree on the dates, audience, venue and other logistical arrangements for this briefing. ▶ Preliminary for Recommendations Development: The evaluation team is expected to hold a preliminary [presentation/workshop] to discuss the summary of findings and conclusions with USAID and targeted stakeholders to draft collaboratively any requested recommendations. Specific stakeholders to be invited will be confirmed during the scoping visit. Any presentations or workshops will be scheduled as agreed upon during the in-briefing. From this meeting, a Findings/Conclusions/Recommendations will be developed and included as part of the final evaluation report. ▶ Draft Evaluation Report: The content should cover all the main elements of the report including major findings, conclusions, lessons learned, recommendations and relevant annexes including the case study. The inputs from the exit debrief and recommendations workshop should be incorporated in the report. The report should comply with the USAID’s Evaluation Report content and standards set out in Annex 5. The date for submission of the Integrated Health Service Delivery in the Post-Ebola Context 71 first draft report will be determined in the evaluation design plan. ▶ Final Report: The contractor will submit a 508 compliant final report (not exceeding 30 pages, excluding the annexes) incorporating final edits incorporating comments from USAID and other stakeholders. The approved final report should be cleared by USAID. The final report shall be provided in both English and French. ▶ Evaluation Summary: The evaluation team will produce a user friendly synthesis or summary of the evaluation in a format that is attractive and understandable for public use. This summary will be produced in both English and French. The Summary will be presented alongside the final evaluation report. ▶ After Action Review (AAR): The evaluation team together with USAID will participate in an After Action Review of the evaluation process as part of the out brief process to establish lessons and needed improvements for future evaluation exercises. This discussion will be organized and documented by USAID for sharing with the Evaluation Contractor. ▶ Submission of Dataset(s) to the Development Data Library: Per USAID’s Open Data policy (see ADS 579, USAID Development Data) the contractor must also submit to the COR and the Development Data Library (DDL), at www.usaid.gov/data, in a machine-readable, non-proprietary format, a copy of any dataset created or obtained in performance of this award, if applicable. The dataset should be organized and documented for use by those not fully familiar with the intervention or evaluation. Please review ADS 579.3.2.2 Types of Data To Be Submitted to the DDL to determine applicability. ▶ Submission of Final Evaluation Report to the Development Experience Clearinghouse: Per USAID policy (ADS 201.3.5.18) the contractor must submit the evaluation final report and its summary or summaries to the Development Experience Clearinghouse (DEC) within three months of final approval by USAID. TIMING Scoping Exercise including in country trip September 2019 10 days Final Evaluation Design September 2019 Two weeks after the in country scoping trip Internal Review Board (IRB) approval October 2019 In country data collection October/ November 2019 o/a 4-5 weeks Recommendation workshops November Draft Report December 2019 Final Report Jan/Feb 2020 SOW ANNEX 1: HEALTH SERVICES DELIVERY ACTIVITY RESULTS FRAMEWORK Sub-IR 1.1.1: Availability of Integrated Quality Maternal, Neonatal and Child Health Services Increased Sub-IR 1.1.2: Referral Linkages Strengthened Sub-IR 1.1.3: Availability of Prevention, Care and Treatment Services Increased Sub-IR 1.1.4: Access to Health Services Improved IR 1.2: Health Behaviors and Demand for Quality Health Services Improved IR 1.3: Health Systems Strengthened IR 1.1: Delivery of Quality Health Services Improved 72 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity SOW ANNEX 2: ILLUSTRATIVE LIST OF DOCUMENTS TO BE REVIEWED Program Documents ▶ RFA / Solicitation document ▶ Agreement and modifications ▶ Annual work plans, annual and quarterly progress and financial reports ▶ Reports and minutes of annual and quarterly program review reports ▶ Monitoring Evaluation and Learning Plan and data quality assessment reports ▶ Baseline reports ▶ Guidelines and other policy documents produced as a result of the HSD efforts ▶ Financial reports USAID ▶ Rapid Health Systems Assessment conducted by the MCSP-HSS Activity ▶ USAID Country Development Cooperation Strategy 2015-2019 ▶ Integrated Health Project Appraisal document 2015, amendment 2017 ▶ USAID Strategy for Ending Preventable Child and Maternal Deaths ▶ Local Systems Assessment Framework ▶ USAID Health Systems Strengthening Framework ▶ Guinea Self Reliance Country Roadmap ▶ National Supply Chain Assessment 2016, 2019 National and other relevant documents ▶ National Health Development Plan 2015-2024 ▶ National Ebola Recovery Strategy 2015-2017 ▶ Community Health Policy ▶ Guinea Multi Indicator Cluster Survey 2016 ▶ Preliminary DHS Results 2018 ▶ Guinea Family Planning Costed Implementation Plan ▶ Youth SOW ANNEX 3: FINAL REPORT FORMAT AND QUALITY CRITERIA 1. Abstract 2. Executive Summary 3. Evaluation Purpose 4. Background on the Context and the Strategies/ Projects/Activities being Evaluated 5. Evaluation Questions 6. Methodology 7. Limitations to the Evaluation 8. Findings, Conclusions, and (If Applicable) Recommendations 9. Annexes See the Evaluation Toolkit for the How-To Note on Preparing Evaluation Reports and ADS 201mah, USAID Evaluation Report Requirements. An optional Evaluation Report Template is also available in the Evaluation Toolkit. The evaluation abstract of no more than 250 words should describe what was evaluated, evaluation questions, methods, and key findings or conclusions. The executive summary should be 2–5 pages and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, and conclusions (plus recommendations and lessons learned, if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methods (e.g., in sampling; data availability; measurement; analysis; any potential bias such as sampling/selection, measurement, interviewer, response, etc.) and their implications for conclusions drawn from the evaluation findings. Integrated Health Service Delivery in the Post-Ebola Context 73 Annexes to the report must include: ▶ Evaluation SOW (updated, not the original, if there were any modifications); ▶ Evaluation methods; ▶ All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ▶ All sources of information or data, identified and listed; ▶ Statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team, if applicable; ▶ Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of or describing existing conflicts of interest; and ▶ Summary information about evaluation team members, including qualifications, experience, and role on the team. USAID EVALUATION STANDARDS Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure quality. ▶ Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity; ▶ Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly; ▶ The Executive Summary should present a concise and accurate statement of the most critical elements of the report; ▶ Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID; ▶ Evaluation methodology should be explained in detail and sources of information or data properly identified; ▶ Limitations to the evaluation should be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.); ▶ Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions; ▶ Conclusions should be specific, concise, and include an assessment of quality and strength of evidence to support them supported by strong quantitative and/ or qualitative evidence; ▶ If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females; and ▶ If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Checklist and Review Template from the Evaluation Toolkit for additional guidance. 74 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Evaluation Question and Data Collection Methods Primary Question Sub-question Monitoring data or data validation Document review Facility checklist & Obs. Provider Survey & KII KII CHW/ RECO s KII Facility Managers KII MoH Directors & Policy Makers KII Imp. Partners & Donors KII Profess. School managers Group Interviews - COSAH FGD Women FGD Men EQ1. To what extent has quality of health care services improved as a result of the HSD activity? 1.1 To what extent have HSD interventions contributed to increased availability of the complete package of integrated RMNCH+ care across targeted facilities (including FP, OF and GBV)? X X X X X X X X X X 1.2 To what extent has HSD strengthened provider capacity to deliver quality care to national performance standards? X X X X X X X EQ2. To what extent did HSD contribute to improved access and use of the essential integrated care package of health services at targeted facilities? 2.1 To what extent has HSD contributed to the delivery of the integrated community health package including referral linkages? To what degree did implementation of the integrated community health package and referral increase use of services? X X X X X X X X X X 2.2 To what extent has HSD interventions increased access and use of RMNCH+ services through training, capacity, coordination, and policy change? X X X X X X X X X 2.3 To what extent has HSD’s SBCC approach improved the targeting and scale of those reached with health promotion activities? Has the approach contributed to engaging and empowering community members and clients to access and use health care services? To what extent did community members shift from recipients to advocates/promoters of health? X X X X X X X 2.4 To what extent have community interventions worked to mitigate gender barriers and improve equity in access to care? X X X X X X X X ANNEX II: EVALUATION MATRIX Integrated Health Service Delivery in the Post-Ebola Context 75 Evaluation Question and Data Collection Methods Primary Question Sub-question Monitoring data or data validation Document review Facility checklist & Obs. Provider Survey & KII KII CHW/ RECO s KII Facility Managers KII MoH Directors & Policy Makers KII Imp. Partners & Donors KII Profess. School managers Group Interviews - COSAH FGD Women FGD Men EQ3. To what extent has HSD strengthened the health system to deliver an integrated package of RMNCH health care services across the continuum of care ? 3.1 To what extent did HSD strengthen governance by building leadership and commitment of local partners including government at different levels (e.g., MOH; Governorates; DPS' health management teams) civil society, communities, and private sector midwifery and nursing schools? X X X X X X X X 3.2 To what extent has the HSD activity contributed to the availability of equipment, commodities and drugs for the delivery of (integrated) prevention, care and treatment services? X X X X X X X X X X 3.3 To what extent has HSD improved the quality of health services data and its use for decision-making at all levels (e.g., SDP managerial, provider, and data collection capacity)? X X X X X X X 3.4 What was the added value of supporting pre￾service training through the Faculty of Medicine and midwifery schools? X X X X X X 3.5 To what extent have HSD initiatives demonstrated potential for post-investment sustainability (e.g. which are more likely to continue after HSD ends, what are the cost and quality implications if they do, and which ones will likely not be sustained)? X X X X X X X X EQ4: To what extent has local technical and managerial capacity been sufficiently built to support fistula prevention and care in Guinea? X X X X X X X X X X EQ 5:How well have the limited financial resources been strategically directed to address gender based violence issues under the HSD project interventions? What could have been done differently to enhance achievement of results? X X X X X X X X X X X X 76 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Evaluation Question and Data Collection Methods Primary Question Sub-question Monitoring data or data validation Document review Facility checklist & Obs. Provider Survey & KII KII CHW/ RECO s KII Facility Managers KII MoH Directors & Policy Makers KII Imp. Partners & Donors KII Profess. School managers Group Interviews - COSAH FGD Women FGD Men EQ 6. How well did HSD activities complement and leverage efforts of other USAID and development partner activities to advance RMNCH results in Guinea? 6.1 To what extent did HSD complement and reinforce other USAID programs in health and governance sectors, e.g. internal synergies with USAID programs? X X X X X X 6.2 To what extent do USAID health programs partner with the same health system levels and partners to facility synergies across partnerships? X X X X X X 6.3 To what extent do USAID programs add value to other RMNCH+ initiatives in Guinea e.g., value add (externally)? X X X X X X 6.4 What were missed opportunities for better synergy and advancement of results, both internally (within USAID) and externally? X X X X X X Integrated Health Service Delivery in the Post-Ebola Context 77 ANNEX III: DATA COLLECTION INSTRUMENTS 78 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 1 Instrument A: Facility Observation and Checklist INSTRUCTIONS: After introduction and interview with the facility manager, continue with the observational checklist. Section 1 includes introductory questions, and the following sections relate to observations. You may invite the provider to accompany you but it is not obligatory. Question Response Comment 1. Date and time of arrival __ / __ / __ (DD/MM/YYYY) __: __ (HH:MM) 2. Facility name: [Free text] 3. Interviewer name: [Free text] Section 1: Health Care Access 4. About how many patients were in the waiting room when you arrived? □ <5 □ 6-15 □ 15-30 □ More than 30 5. What are the opening hours of the facility? __ __: __ __ (HH:MM) __ __: __ __ (HH:MM) 6. How many patients were in the waiting room when you left? □ <5 □ 6-15 □ 15-30 □ More than 30 7. About how long did clients wait on average before being seen? □ <30 min □ 30 min – 1h □ 1h – 2h □ More than 2h 8. Are all services offered every day? □ Yes □ No 9. Is there at least one doctor available every day? □ Yes □ No 10. Are patient rights posted visibly on the wall? □ Yes □ No 11. Are payments made for integrated reproductive, maternal, neonatal, infant, and adolescent health services? □ Yes □ No Section 2. ENVIRONMENT 12. Is the counseling room for family planning, antenatal care, postnatal care, and vaccination clean and welcoming? □ Yes □ No 13. Are patient toilets functioning and in good condition? □ Yes □ No 14. Are information, education, and communication materials on family planning, antenatal care, and other subjects available? □ Yes □ No 15. Are posters and public health messages posted on the walls in good condition? □ Yes □ No 16. Are there chairs or benches in the waiting room? □ Yes □ No 17. Does the facility have functioning electricity today? □ Yes □ No 18. Does the facility have functioning water today? □ Yes □ No 19. Does the facility have an SBM-R tracking system publicly posted on the wall (indicating stars)? □ Yes □ No □ N/A 20. Does the facility have current data publicly posted on the wall (e.g., graphics, statistics, numbers)? □ Yes □ No □ N/A Note data Or take photos Section 3. Family planning. Ask to see the locations where family planning patients wait for services and are counselled. 21. Are family planning services offered at this facility? □ Yes □ No -> SKIP TO S4 22. Is a designated family planning counselor available? □ Yes □ No 23. Is there a private place for family planning counseling? □ Yes □ No Integrated Health Service Delivery in the Post-Ebola Context 79 2 24. Are job aids used for family planning counseling? □ Yes □ No 25. Are the following modern family planning methods available today? Select all that apply: □ Oral contraceptive pill □ Injectable □ Implant □ IUD □ Sterilization □ Male condoms □ Lactational amenorrhea counseling Explain stock ruptures or lack of availability 26. Is a trained provider available for provision of IUDs and implants? □ Yes □ No 27. Is parental consent required for family planning? □ Yes □ No 28. Is marital consent required for family planning? □ Yes □ No Section 4. Prenatal and postpartum care. Ask to see the locations where antenatal and postnatal care patients wait for services and where these services are provided. 29. Are there designated nurses or midwives for antenatal and postnatal care? □ Yes □ No 30. Is there a private location for antenatal care provision and counseling? □ Yes □ No 31. Are job aids used for antenatal and postnatal care? □ Yes □ No 32. Are the following supplements or tests available today? Select all that apply: □ Folic acid supplements □ Hemoglobin testing □ Iron supplements □ Diabetes testing Explain stock ruptures or lack of availability 33. Are breastfeeding support groups offered to new mothers? □ Yes □ No 34. Is there a functioning laboratory with the appropriate consumables for providing antenatal and postnatal care (e.g., laboratory technicians, equipment and consumables, tests, gloves)? Please verify equipment and consumables. □ Yes □ No 35. Is a laboratory technician available today? □ Yes □ No 36. For primary care, where are laboratory tests processed? [Free text] 37. How much time does it take to receive test results? [Free text] 38. Is there a registry for referral of complicated cases? □ Yes □ No -> SKIP TO Q40 39. Is this registry used? □ Yes □ No 40. Is fistula care or referral for fistula care provided? □ Yes □ No 41. Is care for sexual violence or referral for sexual violence care provided? □ Yes □ No Section 5. Physical and material resources. For each of the following, verify if they are in the exam room or adjacent room. 42. Pelvic exam lamp Available: □ Yes □ No Functioning: □ Yes □ No 43. Table or bed for gynecological exam Available: □ Yes □ No Functioning: □ Yes □ No 44. Gloves Available: □ Yes □ No Functioning: □ Yes □ No 45. Sharps container Available: □ Yes □ No Functioning: □ Yes □ No 46. At least 5 syringes of 2-3 ml (with 21 needles) Available: □ Yes □ No Functioning: □ Yes □ No 47. Disinfectant solution (mixed) Available: □ Yes □ No Functioning: □ Yes □ No 48. Blood pressure cuff Available: □ Yes □ No Functioning: □ Yes □ No 49. Stethoscope Available: □ Yes □ No Functioning: □ Yes □ No 80 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 3 50. Thermometer Available: □ Yes □ No Functioning: □ Yes □ No 51. Adult scale Available: □ Yes □ No Functioning: □ Yes □ No 52. Infant scale Available: □ Yes □ No Functioning: □ Yes □ No 53. Speculum Available: □ Yes □ No Functioning: □ Yes □ No 54. Doppler Available: □ Yes □ No Functioning: □ Yes □ No 55. Vaccine refrigerator Available: □ Yes □ No Functioning: □ Yes □ No 56. IUD kit Available: □ Yes □ No □ NA Functioning: □ Yes □ No □ NA 57. Instrument kit Available: □ Yes □ No Functioning: □ Yes □ No 58. Autoclave (may be elsewhere in the facility) Available: □ Yes □ No Functioning: □ Yes □ No 59. Surgical scrubs (pants, shirts, masks, glasses, aprons, shoes) Available: □ Yes □ No Functioning: □ Yes □ No 60. Surgical drapes Available: □ Yes □ No Functioning: □ Yes □ No 61. Waterproof surgical pads Available: □ Yes □ No Functioning: □ Yes □ No 62. Lift Available: □ Yes □ No Functioning: □ Yes □ No 63. Venipuncture arm support Available: □ Yes □ No Functioning: □ Yes □ No Thank the provider who has accompanied you (if accompanied). Integrated Health Service Delivery in the Post-Ebola Context 81 4 Instrument B. Health Provider Questionnaire Time estimated: 25 minutes Official use only: Interviewer ID: Sex: Region: Facility ID: □ Male □ Female Prefecture: Provider ID: Age: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Health facility information Region: Conakry Kindia Boké Mamou Labé Kankan Faranah Prefecture: [Free text] Facility name: Facility ID: [ | ] Facility type 01- National hospital 02 – Communal medical center 03 Regional hospital 04 Prefectoral hospital 05 Health center Provider type 01 – Doctor/General practitioner 02 – Nurse 03 – Midwife 88 – Other (specify) For how long have you worked in this facility? < 6 months 6-12 months 12-24 months >24 months SECTION A: Experience with HSD Project № Facility type QUESTION Response CODE SKIP 1 HC and hospital Did you participate in the HSD program? Yes 1 No 2 Skip to Q5 Partially 3 Skip to Q5 Don’t know 14 Skip to Q5 2 Which HSD activities did you participate in? (select all that apply) Training 1 Skip to Q4 82 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 5 Supervision 1 Skip to Q4 SBM-R 1 Skip to Q4 GBV or fistula activities 1 Skip to Q4 Community health 1 Skip to Q4 Other 1 None 0 Skip to Q5 3 Specify other 4 HC and hospital Are the HSD activities that you participated in the most important issues for improving access, availability and quality of reproductive, maternal, neonatal, child and adolescent (RMNCH+A) in your facility? Completely agree 1 Somewhat agree 2 Somewhat disagree 3 Completely disagree 4 Don’t know 14 Other (please specify) 88 SECTION B: Contribution to quality of care in family planning, prenatal care, obstetrical care, and postnatal care SECTION B1: Family planning 5 HC only Which family planning methods are currently available at your facility (select all that apply) Oral contraceptive pill (combined) 1 Oral contraceptive pill (progestin-only) 2 Injectables 3 Implant 4 IUD 5 Sterilization 6 Male condoms 7 LAM counseling 8 Don’t know 0 Other 88 6 HC only If certain methods are not available, are clients referred elsewhere? Always Sometimes Rarely Never Skip to QQ9 Don’t know 14 Skip to QQ9 No applicable 99 Skip to QQ9 7 HC only Where are the clients referred? (Select all that apply) Other health center Regional hospital Prefectoral hospital National hospital Centre Médical Communal Don’t know Other (specify) 8 HC only For which methods are clients referred? (Select all that apply) Oral contraceptive pill (combined) 1 Oral contraceptive pill (progestin-only) 2 Injectables 3 Implant 4 IUD 5 Sterilization 6 Male condoms 7 Integrated Health Service Delivery in the Post-Ebola Context 83 6 LAM counseling 8 Don’t know 0 Other (specify) 88 9 HC and hospital Do clients receive family planning counseling at this facility? Always Sometimes Rarely Never Skip to Q11 Don’t know 14 Skip to Q11 10 Who provides the family planning counseling? (Select all that apply) Nurse 1 Midwife 2 Doctor 3 Don’t know 14 Other (specify) 88 11 What are the main challenges that providers face in providing family planning counseling services in your facility? (Select all that apply) Lack of awareness by women (e.g., misconceptions, religion, etc.) 1 Lack of interest among women 2 Need to consult family members 3 Provider lack of time for FP counseling 4 Client lack of time for FP counseling 5 Lack of educational materials on FP 6 Lack of space/room for FP counseling 7 Not a priority for facility personnel 8 Facility personnel have not received training in FP counseling 9 Inappropriate or not necessary 10 Other (specify) 88 12 According to the female clients you receive at your health center, what are the main challenges they face in adopting/using contraceptives? (Select all that apply) Cultural or religious barrier 1 Family barrier (spouse or parents) Availability (including stock ruptures) 2 Safety/side effects 3 Cost 4 Trust in providers 5 Lack of awareness or knowledge 6 Lack of privacy 7 Provider availability 8 Miscommunication Other (specify) 8 8 Don’t know 13 Hospital only Are women regularly counseled on family planning methods before they are discharged from the facility after giving birth? Always 1 Sometimes Rarely Never 0 Skip to Q16 Don’t know 1 4 Skip to QQ16 14 Hospital If yes, who counsels them? (Select Doctor who assisted birth 1 84 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 7 only all that apply) Another doctor 2 Nurse/midwife who assisted birth 3 Another nurse/midwife 4 Other (specify) 8 8 15 HC and hospital Now I would like to know how frequently you discuss the following topics with your clients? Always (1) Someti mes (2) Rarely (3) Never (4) Don’t know (14) a. Current and prior use of family planning methods b. Side effects c. Concerns, rumors, misconceptions of clients d. GBV e. Birth plan a. Modern contraceptiv e methods 16 HC and hospital Do you have job aids or IEC materials for FP counseling? Yes 1 No 2 Skip to Q18 Don’t know 3 Skip to Q18 17 HC and hospital If yes, how frequently do you use job aids/IEC materials for FP counseling? Always 1 Sometimes 2 Rarely 3 Never 4 18 HC only In your experience, what is the most important factor for women in making a decision about contraceptive use? Comprehensive provider counseling 1 Written health education material/brochures 2 Prior negative experiences with a method 3 Prior positive experiences with a method 4 Advice from friends or family members 5 Fear of side effects 6 Client interest in FP 7 Spouse advice/guidance 8 Don’t know 14 Other 88 SECTION B2: Antenatal care Q19: When are the following essential ANC services provided? (Health Center only) Procedure 1st visit 1st trimester visit 2nd trimester visit 3rd trimester visit All visits Integrated Health Service Delivery in the Post-Ebola Context 85 8 Blood pressure Weight Risk assessment Anemia assessment Folic acid supplementation Iron supplementation Nutritional advice Iron administration and counseling FP counseling Breastfeeding guidance Blood typing Routine urine test Diabetes testing SECTION B3: Care provision 20 Hospital only With what frequency do women receive 10 units of oxytocin immediately after giving birth? Always 1 Sometimes 2 Rarely Never 0 Skip to Q 22 Don’t know 14 Skip to Q 22 21 Hospital only How long after giving birth is the oxytocin administered? Immediately after birth One hour after birth 1-12 hours after birth Before hospital discharge 22 Hospital How many births have there been at this facility within the past week? [number] 23 Hospital How frequently do you ensure immediate skin-to-skin contact after birth? Always 1 Sometimes 2 Rarely 3 Never 4 Aller to QQ 25 Don’t know 5 24 How many times in the past week was this done? SECTION B4: Postpartum and postnatal care 25 Hospital only Is breastfeeding supported in the postpartum period? Yes, always 1 Yes, sometimes 2 Yes, in few cases 3 No 0 Don’t know 14 26 Hospital only How frequently is breastfeeding guidance (e.g., position and latch) provided? Always 1 Sometimes 2 Rarely 3 Never 0 Don’t know 14 27 Hospital only How frequently is continuous positive airway pressure used for newborn respiratory problems? Always 1 Sometimes 2 Rarely 3 Never 14 Skip to Q29 28 Hospital When using CPAP, how frequently do you assess the neonate during the first two hours after birth? Every 5 minutes Skip to Q30 Every 10 minutes Skip to Q30 Every 15 minutes Skip to Q30 Every 30 minute Skip to Q30 86 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 9 29 Hospital If CPAP is not used for neonatal respiratory issues, why not? (select all that apply) Lack of training 1 Lack of CPAP equipment 2 Lack of CPAP consumables 3 Other (specify) 88 SECTION C: Community engagement 33 HC only Does your facility currently have a COSAH? Yes 1 No 0 Don’t know 14 Other 88 34 HC only What role does the COSAH play in the promotion of RMNCH+A care in your community? (Select all that apply) Health promotion 1 Promotion of communication between COSAH as representatives of clients and providers 2 Sensitization of community members to RMNCH+A through community events and interactive education 3 Support community evaluation activities 4 Other (specify) 88 SECTION D: Health equity 35 HC and hospitals To what extent are RMNCH+A services accessible to marginalized (e.g., refugees) and vulnerable populations (e.g. individuals less likely to have positive health outcomes such as those with low￾income). Ver accessible 1 Somewhat accessible 2 Somewhat inaccessible 3 Very inaccessible 4 Don’t know 98 36 HC and hospitals What barriers and challenges do marginalized and vulnerable populations experience in accessing RMNCH+A care at this facility? Transportation 1 Discrimination or perceived discrimination 2 Cultural or religious barriers 3 Not aware of service availability 4 Language barriers 5 Provider lack of time 6 Cost 7 Wait time 8 Lack of understanding of necessary documents for receiving care 9 Other (specify) 88 37 HC and hospitals In your opinion, what are the main barriers linked to gender which limit RMNCH+A care use? Opening times 1 Cost of service 2 Financial obstacles 3 Wait times 4 Decision-making ability 5 Spousal or parental permission for care 6 Uncomfortable with provider gender 7 None 0 Other (specify) 88 Integrated Health Service Delivery in the Post-Ebola Context 87 10 38 HC and hospitals Does your establishment work to reduce gender￾based barriers to care? Yes 1 No 0 Don’t know 1 4 SECTION E: Improvements in RMNCH+A care 39 HC and hospital s To what extent do you agree with the following statements? HSD training of managers and providers has improved the following: Completely agree (1) Somewha t agree (2) Somewhat disagree (3) Completely disagree (4) NA (98) a.RMNCH+A service delivery management, organization, or approaches b. Care organization c. Understanding of RMNCH+A services by the community d. IEC materials e. Provider behavior in responding to client needs Thank you for your participation 88 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 11 Instrument C. Provider interview Time estimated: 45 minutes Official use only: Interviewer ID: Sex: Region ID: Facility ID: □ Male □ Female Prefecture ID: Provider ID: Age: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 What services are included in the complete package [see facility list] of integrated RMNCH+A care (including FP, OF/GBV)? To what extent are these services available in your facility today? For those services which are not available, why? 1.1 2 Have you provided or received training on the following RMNCH+A quality of care and performance standard topics (list of broad topic areas)? When? How much? 1.2 3 How familiar are you with the services offered in the community? Please list the services that are supposed to be provided at the community level? 2.1 4 In your opinion, how functional is the service provision at the community level? In your opinion, how functional is the referral from the community to the health center? 2.1 5 What is your perspective on the utilization of health services by the community? How much of a role have the services available at the community and referral from the community to the health centers played in use of health services overall? 2.1 6 In your opinion, are there more women coming to the facility for MCH services since HSD intervention? FP services since HSD intervention? How about in the past 6 months? 2.2 7 In your experience, are female clients more informed and engaged around their health care needs since the HSD intervention began? In what ways? (list) 2.2 8 What specific barriers do women face in accessing health care at this facility? within this community generally? (list) What specific measures are you or your facility taking to help women overcome these barriers? Please explain. 2.4 9 How about youth - what barriers do they face? What measures are being taken? Please explain. 2.4 Section 2. Health systems strengthening 10 How would you characterize the leadership role of the [DPS health management team/MOH/NGOs/other] in improving the quality of services in the facilities they work with? To what extent have the MOH and the Governates more generally supported quality improvements? (separate) 3.1 11 In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 12 How has HSD contributed to availability of equipment, commodities, and drugs at your facility? What have these contributions impacted your facility's provision of integrated care? improvement in DHIS2 or SBM-R indicators? 3.2 13 What difficulties has your facility had in service provision? What are the most challenging aspects of these? How do you feel your facility can overcome these aspects? 3.2 14 To what extent did pre-service training prepare you/your frontline staff for the services that you offer? In what areas do you think additional training would have be helpful? 3.4 Integrated Health Service Delivery in the Post-Ebola Context 89 12 In your opinion, how can investments in pre-service training be sustained after the HSD project ends? 15 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? 3.5 16 What supports will be necessary for sustaining the quality improvements after the project ends? 3.5 Section 3: Special themes: fistula and GBV care 17 How consistently would you say partograms are used for women giving birth at this facility? What barriers are faced in ensuring that women who develop obstructed labor obtain timely cesarean surgery? Are obstetric fistula treatment services available at your facility? at the closest referral hospital? How would you rate the availability and quality of these fistula-related services? How may referrals for OF surgery have you made in the past 6 months? After referral, how long would you say women typically have to wait for OF surgery? 4 18 Approximately what proportion of women undergoing OF repair at your facility have successful fistula closure without residual incontinence? From your perspective, are women who return to the community after treatment satisfied with their experience? the result? In your opinion, how can fistula-related services continue after the HSD project has completed? 4 19 Are women seeking care at this facility regularly screened for GBV? How are GBV care, treatment, or referral integrated into the service package provided at your facility? 5 20 What challenges have you experienced in providing GBV-related care? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4: Cross-cutting Questions 21 What internal and external factors have affected achievement of results (positively or negatively)? How? Which interventions or activities have worked better than others? Why? 6 22 How has HSD adapted to shifts in the internal and external operating environments (e.g., national policies/strategies, governance structure changes, entry/exit of critical government/development partner/community/private sector/NGO actors)? 6 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 90 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 13 Instrument D. Facility Manager Interview Guide Time estimated: 45 minutes Official use only: Interviewer ID: Sex: Region ID: Facility ID: □ Male □ Female Prefecture ID: Provider ID: Age: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 What services are included in the complete package [see facility list] of integrated RMNCH+A care (including FP, OF/GBV)? To what extent are these services available in your facility today? For those services which are not available, why? 1.1 2 Have you provided or received training on the following RMNCH+A quality of care and performance standard topics (list of broad topic areas)? When? How much? 1.2 3 How familiar are you with the services offered in the community? Please list the services that are supposed to be provided at the community level? 2.1 4 In your opinion, how functional is the service provision at the community level? In your opinion, how functional is the referral from the community to the health center? 2.1 5 What is your perspective on the utilization of health services by the community? How much of a role have the services available at the community and referral from the community to the health centers played in use of health services overall? 2.1 6 In your opinion, are there more women coming to the facility for MCH services since HSD intervention? FP services since HSD intervention? How about in the past 6 months? 2.2 7 In your experience, are female clients more informed and engaged around their health care needs since the HSD intervention began? In what ways? (list) 2.2 8 What specific barriers do women face in accessing health care at this facility? within this community generally? (list) What specific measures are you or your facility taking to help women overcome these barriers? Please explain. 2.4 9 How about youth - what barriers do they face? What measures are being taken? Please explain. 2.4 Section 2. Health systems strengthening 10 How would you characterize the leadership role of the [DPS health management team/MOH/NGOs/other] in improving the quality of services in the facilities they work with? To what extent have the MOH and the Governates more generally supported quality improvements? (separate) 3.1 11 In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 12 How has HSD contributed to availability of equipment, commodities, and drugs at your facility? What have these contributions impacted your facility's provision of integrated care? improvement in DHIS2 or SBM-R indicators? 3.2 13 What difficulties has your facility had in service provision? What are the most challenging aspects of these? How do you feel your facility can overcome these aspects? 3.2 14 To what extent did pre-service training prepare you/your frontline staff for the services that you offer? In what areas do you think additional training would have be helpful? In your opinion, how can investments in pre-service training be sustained after the HSD project ends? 3.4 Integrated Health Service Delivery in the Post-Ebola Context 91 14 15 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? 3.5 16 What supports will be necessary for sustaining the quality improvements after the project ends? 3.5 Section 3: Special themes: fistula and GBV care 17 How consistently would you say partograms are used for women giving birth at this facility? What barriers are faced in ensuring that women who develop obstructed labor obtain timely cesarean surgery? Are obstetric fistula treatment services available at your facility? at the closest referral hospital? How would you rate the availability and quality of these fistula-related services? How may referrals for OF surgery have you made in the past 6 months? After referral, how long would you say women typically have to wait for OF surgery? 4 18 Approximately what proportion of women undergoing OF repair at your facility have successful fistula closure without residual incontinence? From your perspective, are women who return to the community after treatment satisfied with their experience? the result? In your opinion, how can fistula-related services continue after the HSD project has completed? 4 19 Are women seeking care at this facility regularly screened for GBV? How are GBV care, treatment, or referral integrated into the service package provided at your facility? 5 20 What challenges have you experienced in providing GBV-related care? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4. Leveraging Internal and External Programming/Partners 21 In your opinion, have the USAID projects that have supported your [facility/prefecture/region/MOH] been complimentary (e.g. built on one another)? How? 6.1 22 In your opinion, to what extent do USAID projects partner with the same health system levels and partners to facilitate synergies across partnerships? 6.2 23 In your opinion, have the USAID projects complimented (rather than duplicated) other donor and MOH RMNCH+ projects in your [facility/prefecture/region/MOH]? How? 6.3 24 In your opinion, were important internal or external opportunities missed in the USAID approach to imprpoving RMNCH+ care in Guinea? How? 6.4 Section 4: Cross-cutting questions 21 What internal and external factors have affected achievement of results (positively or negatively)? How? Which interventions or activities have worked better than others? Why? 7 22 How has HSD adapted to shifts in the internal and external operating environments (e.g., national policies/strategies, governance structure changes, entry/exit of critical government/development partner/community/private sector/NGO actors)? 7 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 92 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 15 Instrument E. Community health worker interview Time estimated: 45 minutes Official use only: Interviewer ID: Sex: Region: Code HC: □ Male □ Female Prefecture: Code ASC: Age: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are familiar with the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 To what extent are the integrated RMNCH care package services available in your health facility? (read list of services) Which services have no (or limited) availability? Where can women go to receive [the services that are lacking at local facility]? What is the referral process for women to access those services? 1.1 2 Please list the services that are supposed to be provided at the community level? In your opinion, how functional is the service provision at the community level? What challenges are faced? 2.1 3 In your opinion, how functional is the referral from the community to the health center? How much of a role have the services available at the community and referral from the community to the health centers played in use of health services overall? 2.1 4 In your opinion, are there more women coming to the facility for MCH services since HSD intervention? FP services since HSD intervention? How about in the past 6 months? 2.2 5 In your experience, are clients more informed and engaged around their health care needs since the HSD intervention began? In what ways? 2.3 6 What specific barriers do women face in accessing health care at this facility? within this community generally? (list) What specific measures are you or your facility taking to help women overcome these barriers? How about youth - what barriers do they face? What measures are being taken to help youth overcome these barriers? 2.4 Section 2. Health Systems Strengthening 7 How would you characterize the leadership role of the management team of the Ministry of Health, NGOs, and other organizations in improving the quality of services in the facilities they work with? 3.1 8 To what extent have the MOH and the Governates more generally supported quality improvements? 3.1 9 How has HSD contributed to availability of equipment, commodities, and drugs at your facility? What have these contributions impacted your facility's provision of integrated care? improvement in DHIS2 or SBM￾R indicators? 3.2 10 Which goals or criteria has your facility had the most difficulty achieving? What are the most challenging aspects of these goals or criteria? How do you feel your facility can overcome these aspects (probe re internal solutions with staff/management)? 3.3 11 In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 12 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? What supports will be necessary for sustaining the quality improvements after the project ends? 3.5 Section 3. Special Topics: Fistula and GBV 13 How consistently would you say partograms are used for women giving birth at this facility? Integrated Health Service Delivery in the Post-Ebola Context 93 16 What barriers are faced in ensuring that women who develop obstructed labor obtain timely cesarean surgery? Are obstetric fistula treatment services available at your facility? at the closest referral hospital? How would you rate the availability and quality of these fistula-related services? How may referrals for OF surgery have you made in the past 6 months? After referral, how long would you say women typically have to wait for OF surgery? Approximately what proportion of women undergoing OF repair at your facility have successful fistula closure without residual incontinence? From your perspective, are women who return to the community after treatment satisfied with their experience? the result? In your opinion, how can fistula-related services continue after the HSD project has completed? 4 14 Are women seeking care at this facility regularly screened for GBV? How are GBV care, treatment, or referral integrated into the service package provided at your facility? What challenges have you experienced in providing GBV-related care? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4. Crosscutting Questions 15 What internal and external factors have affected achievement of results (positively or negatively)? How? Which interventions or activities have worked better than others? Why? 6 16 How has HSD adapted to shifts in the internal and external operating environments (e.g., national policies/strategies, governance structure changes, entry/exit of critical government/development partner/community/private sector/NGO actors)? 6 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 94 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 17 Instrument F. COSAH group interview Time estimated: 45 minutes Official use only: Interviewer ID: Number of participants: Females: Males: Region: Code HC: Prefecture: Community ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are familiar with the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 To what extent has the HSD project contributed to the availability of the integrated RMNCH care package services available? In which areas has HSD had the most impact? The least impact? 1.1 2 In your opinion, how has the quality of services for RMNCH+ improved with HSD-initiated activities? In which areas have you seen the greatest improvement? The least improvement? 1.2 3 What health services are offered in your community? How would you rate the quality of these services, as provided by the CHW? Does the CHW regularly refer women to the health facility? For which services are referrals made? 2.1 4 Please describe how functional service provision is at the community level? How functional is the referral from the community to the health center? From your perspective, how could community-based health services be improved? referral services? 2.1 5 How has access and utilization of services increased due to HSD intervention? On what basis do you assess this? 2.2 6 How has HSD's approach contributed to engaging and empowering community members to access and use health care services? Are clients more informed and engaged about their health care needs since the HSD intervention began? In what ways? To what extent did community members shift from healthcare recipients to advocates/promoters of health? 2.3 7 What specific barriers do women in your community face in accessing health care? 2.4 8 What specific measures are being implemented at the community or facility level to help women overcome these barriers? Please explain. 2.4 9 How about youth? What barriers do they face? What measures are implemented to help them overcome these barriers? 2.4 Section 2. Health Systems Strengthening 10 How would you characterize the leadership role of the [DPS health management team/MOH/NGOs/other] in improving the quality of services in the facilities they work with? To what extent have the MOH and the Governates more generally supported quality improvements? In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 11 How has HSD contributed to availability of equipment, commodities, and drugs across target facilities? What have these contributions impacted the ability of health care facilities to provision of integrated care? improvement in DHIS2 or SBM-R indicators? 3.2 12 Which goals or criteria have facilities had the most difficulty achieving? What are the most challenging aspects of these goals or criteria? How do you feel facilities can overcome these aspects? 3.2 13 How has data been used to better understand the health needs of the community at your level (Ministry, donor, partner, COSAH) about the based on data? Please tell me more about the how the data were used, the specific actions suggested, and what happened following the discussion? 3.3 Integrated Health Service Delivery in the Post-Ebola Context 95 18 14 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? Yes: How? No: Why? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? What supports will be necessary for sustaining the quality improvements after the project ends? 3.5 Section 3. Special Topics: Fistula and GBV 15 Please describe the process of identifying women with OF in your community. What are the barriers to care that local women with OF face? Is local care for OF accessible? How are women facilitated to rejoin their community after treatment? Who facilitates their reentry? 4 16 How are GBV care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in integrating GBV care into services? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4. Leveraging Internal and External Partners 17 In your opinion, have the USAID projects that have supported your [facility/prefecture/region/MOH] been complimentary (e.g. built on one another)? How? 6.1 18 In your opinion, to what extent do USAID projects partner with the same health system levels and partners to facilitate synergies across partnerships? 6.2 19 In your opinion, have the USAID projects complimented (rather than duplicated) other donor and MOH RMNCH+ projects in your [facility/prefecture/region/MOH]? How? 6.3 20 In your opinion, were important internal or external opportunities missed in the USAID approach to improving RMNCH+ care in Guinea? How? 6.4 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 96 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 19 Instrument G. Focus Group Guide – Community Women Time estimated: 45 minutes Official use only: Interviewer ID: Age group: □ <20 yrs □ ≥ 20 yrs Region ID: Facility ID: N Participants: Prefecture: Community ID: N with Health Cards: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 To what extent are the integrated RMNCH care package services available in your health facility? (read list of services) Which services have no (or limited) availability. 1.1 2 Where can women go to receive [the services that are lacking at local facility]? What is the referral process for women to access those services? 1.1 3 What health services are offered in your community? How would you rate the quality of these services, as provided by the CHW? Does the CHW regularly refer women to the health facility? For which services are referrals made? 2.1 4 Please describe how functional service provision is at the community level? How functional is the referral from the community to the health center? 2.1 5 From your perspective, how could community-based health services be improved? referral services? 2.1 6 Have you or a member of your family ever been referred by the CHW to the health facility? Did you go? Why or why not? 2.1 7 Please describe any household barriers that women in your community face in using community health services? facility-based health services? 2.1 8 In your opinion, are there more women coming to the facility for MCH services since HSD intervention (past 4 years)? FP services since HSD intervention? How about in the past 6 months? 2.2 9 What health communication campaigns or activities have occurred in your community? Which populations have been targeted? How have these activities motivated you or someone you know to change their behavior or seek health care? 2.3 10 In your opinion, how has the level of community members engagement in accessing health care services changed with these activities? Have you noticed changes in promotion of health care use among members of your community following these activities? How? Among who? 2.3 11 What specific barriers do women in your community face in accessing health care? (list) What specific measures are being implemented at the community or facility level to help women overcome these barriers? Please explain. How about youth? What barriers do they face? What measures are implemented to help them overcome these barriers? 2.4 12 How about youth? What barriers do they face? What measures are implemented to help them overcome these barriers? 2.4 Section 2: Health Systems Strengthening 13 Has your local facility received any recognition for quality of care from central or regional health authorities? (e.g., SBM-R) How has this recognition improved the environment and quality of care at your local facility? How has it impacted community perspectives of the facility? 3.2 Integrated Health Service Delivery in the Post-Ebola Context 97 20 Section 3: Special topics: Fistula and GBV 14 Please describe the process of identifying women with OF in your community. 4 15 What are the barriers to care that local women with OF face? Is local care for OF accessible? How are women facilitated to rejoin their community after treatment? Who facilitates their reentry? 4 16 How are GBV care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in integrating GBV care into services? How might these challenges be overcome? 5 17 In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 98 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 21 Instrument G. Focus Group Guide – Community Men Time estimated: 45 minutes Official use only: Interviewer ID: Age group: □ 20-35 yrs □ ≥ 35 yrs + Region ID: Facility ID: N Participants: Prefecture: Community ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 To what extent are the integrated RMNCH care package services available in your health facility? (read list of services) Which services have no (or limited) availability. 1.1 2 Where can women go to receive [the services that are lacking at local facility]? What is the referral process for women to access those services? 1.1 3 What health services are offered in your community? How would you rate the quality of these services, as provided by the CHW? Does the CHW regularly refer women to the health facility? For which services are referrals made? 2.1 4 Please describe how functional service provision is at the community level? How functional is the referral from the community to the health center? 2.1 5 From your perspective, how could community-based health services be improved? referral services? 2.1 6 Have you or a member of your family ever been referred by the CHW to the health facility? Did you go? Why or why not? 2.1 7 Please describe any barriers that women in your community face in using community health services? facility-based health services? 2.1 8 Please describe any barriers that youth and adolescents in your community face in using community health services? facility-based health services? 2.1 9 In your opinion, are there more women coming to the facility for MCH services since HSD intervention (past 4 years)? FP services since HSD intervention? How about in the past 6 months? 2.2 10 What health communication campaigns or activities have occurred in your community? Which populations have been targeted? How have these activities motivated you or someone you know to change their behavior or seek health care? 2.3 11 In your opinion, how has the level of community members engagement in accessing health care services changed with these activities? Have you noticed changes in promotion of health care use among members of your community following these activities? How? Among who? 2.3 12 What specific barriers do women in your community face in accessing health care? (list) What specific measures are being implemented at the community or facility level to help women overcome these barriers? Please explain. How about youth? What barriers do they face? What measures are implemented to help them overcome these barriers? 2.4 13 How about youth? What barriers do they face? What measures are implemented to help them overcome these barriers? 2.4 Section 2: Health Systems Strengthening 14 Has your local facility received any recognition for quality of care from central or regional health authorities? (e.g., SBM-R) How has this recognition improved the environment and quality of care at your local facility? 3.2 Integrated Health Service Delivery in the Post-Ebola Context 99 22 How has it impacted community perspectives of the facility? Section 3: Special topics: Fistula and GBV 15 Please describe community-based activities for fistula prevention. Please describe the process of identifying women with OF in your community. 4 16 What are the barriers to care that local women with OF face? Is local care for OF accessible? How are women facilitated to rejoin their community after treatment? Who facilitates their reentry? 4 17 How are GBV care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in integrating GBV care into services? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 18 In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 100 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 1 Instrument H. In-depth interview guide: Fistula Patients Time estimated: 45 minutes Official use only: Interviewer ID: Region: Facility ID: Prefecture: Community ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Individual Fistula Experience 1 First, I would like to ask when you developed the fistula and when you received surgery? (estimate if needed) 4 2 Please tell me about the pregnancy and birth where you developed obstetric fistula. Did you access prenatal care? Where? Which type of facility? How many visits? Where did you give birth? [If facility] How long after the start of your labor did you arrive at the facility? Did you give birth vaginally or via cesarean? How long in total was your labor? [For cesarean} Were you referred to another facility for your cesarean section? Clarify if she went, and how long the trip took. Did you access postpartum care? If so, where? When? 4 3 Now I would like to ask when you noticed the fistula symptoms. How long after your birth did you notice urine leakage symptoms? 4 4 How did you access fistula surgery? Who helped you at the community level? How? How long after you noticed urine leakage did you consult with a doctor? Who did you first seek care from? Where? What happened next? (trace through surgery) What were the main barriers that you faced in accessing fistula care? What care facilitators did you experience? 4 5 Before your surgery, how did your community and family react to your developing fistula? 4 6 Please tell me about the fistula surgery and the care that you received at the facility where you underwent surgery. How long did you have to wait to be operated after arriving at the facility? Did you receive any health education at the facility (e.g., information on the cause of the fistula, how to prevent fistula and other complications, others?) Did you receive counseling at the facility? (e.g., discussions with social workers to help your mental health and perspectives) Did you receive physiotherapy at the facility? What was your surgical result? How satisfied are you with this result? How satisfied were you overall with your fistula care? 4 7 Now let’s talk about the period after you underwent fistula surgery. How long did you stay at the facility after having had surgery? Did you return directly to your home or did you stay with a host family? [If host family]: How long did you stay with this family before going home (or elsewhere)? What was your experience with this family? What advantages did you note through staying with a host family? What challenges did you experience? 4 Integrated Health Service Delivery in the Post-Ebola Context 101 2 [If host family] Are you still in contact with this family? Have you stayed with them again? Why? When? 8 Tell me about returning to your home after having fistula surgery. How was your return to your community and family facilitated? Did someone accompany you home? Who? What did they do? 4 9 How recovered and reintegrated do you feel now? What continuing concerns do you have about fistula? 4 Section 2. Perspectives on community-based fistula services. Read to participant: Thank you for sharing your fistula experience with me. Now I would like to talk about your perspectives on community-based resources for preventing fistula and helping women with fistula, and the effectiveness of these resources. 10 Which services are currently offered in your community for the prevention of obstetric fistula? Treatment of obstetric fistula? Reintegration after fistula repair? In your opinion, how functional are these services? How could they be improved? 2.1 11 What barriers do women in your community experience in accessing community-based fistula￾care services? Facility-based services? What measures are in place at the community level to help women overcome these barriers? 2.1 132 What barriers do adolescents and youth in your community experience in accessing community￾based fistula-care services? Facility-based services? What measures are in place at the community level to help women overcome these barriers? 2.1 17 What community campaigns have been held in your community targeting prevention, treatment and rehabilitation for fistula? What populations have been targeted for these campaigns? How have these campaigns motivated you or anyone you know to change your behavior or seek care? In your opinion, how has the engagement of community members in health care access changed with these campaigns? Youth? Female? Male? Have you noticed behavioral changes in the promotion and utilization of health care among community members following these campaigns? How? Among who? 2.1 Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 102 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 3 Instrument J. Focus Group Guide: Groups working on GBV Time estimated: 1 hour Official use only: Interviewer ID: N participants: ______ Females: Males: Region: Facility ID: Prefecture: Community ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 To what extent did HSD contribute to prevention or referral services for GBV within the integrated care package in your community? For which services do you think HSD had the biggest impact? How? Why? For which services do you think HSD had the least impact? How? Why? 1.1 2 What GBV services are included in your community (including education, prevention, and treatment)? Who is responsible for these activities? What is your opinion about the quality of community-based activities against GBV? Does the CHW refer women to the health facility for GBV-related services? To judicial services? How functional are community-based GBV services? facility-based GBV services? How? 2.1 3 In your opinion, how can we improve community-based GBV services? In your opinion, how can we improve community-based GBV education? In your opinion, how can we improve judicial GBV services? 2.1 5 In your opinion, how has access to GBV-related prevention, health, and justice services improved due to HSD activities? Since HSD began GBV-related activities, have you noticed any difference in the number of women seeking GBV-related care? Are there any differences in women’s characteristics between those who seek GBV-related care compared to those who don’t? (age, ethnicity, etc.) 2.2 6 How has HSD’s approach contributed to engaging community members to access GBV-related services? For adolescents and young adults? Would you say that clients are now better informed and more engaged in GBV prevention, treatment, and judicial services? How? How involved are community-members in GBV advocacy? 2.3 7 What challenges do women in your community face in relation to accessing GBV-related services? What measures have been put in place at the community or facility levels to help women overcome barriers to GBV-related services? Please explain. 2.4 8 What challenges do adolescents and youth in your community face in relation to accessing GBV-related services? What measures have been put in place at the community or facility levels to help adolescents and youth overcome barriers to GBV-related services? Please explain. 2.4 Section 2: Health Systems Strengthening 9 What is your perspective on leadership and management from DPS, MOH, NGOs and others who are seeking to improve GBV-related services? To what extent is the MOH/government pushing for improvement in the quality of GBV care? 3.1 10 In your opinion, has the HSD project created dynamism and engagement for local partners to continue 3.1 Integrated Health Service Delivery in the Post-Ebola Context 103 4 supporting these activities after the end of the project? How? Why? 11 Do you believe that the GBV interventions associated with the HSD project can be supported after the end of the project? How? Who would be the principal organization responsible for maintaining these interventions? What support will be necessary for supporting the continuity of these interventions after the end of the project? 3.5 Section 3: Special themes: Fistula and GBV 12 Please tell me about any activities linked to GBV which is also related to prevention, referral, treatment and reintegration of women affected by fistula? 4 13 How have care, treatment and referral for GBV been integrated in available services in your area? What challenges have you found related to GBV care and referral? How can these challenges be overcome? 4 14 In your opinion, what are the main causes of GBV in Guinea? How could these be overcome in the most efficient way? What activities do you believe USAID should prioritize to most efficiently address GBV in Guinea? 4 Section 3. Leveraging internal and external partnerships 15 In your opinion, have USAID project which have supported your group been complementary? How? 6.1 16 In your opinion, have important internal or external opportunities been missed in USAID’s approach to improving RMNCH+A in Guinea? Which? How? What are your suggestions to improve USAID’s approach? 6.4 Section 4: Crosscutting questions 17 Which internal and external factors have influenced the achievement of project results? Positive? Negative? How? Which interventions have worked better than others? How? Conclusion: I have no more questions for you, and thank you for your participation. Do you have any additional questions for me/us? Notes: Quality of interview: 104 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 5 Instrument K: SBMR (Standards based recognition and management)-specific questions Time estimated: 45 minutes Official use only: Interviewer ID: Sex: Region ID: Facility ID: □ Male □ Female Prefecture ID: Provider ID: Age: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1: Understanding the SBMR Process 1 The SBM-R indicator list includes many standards and performance domains. As your facility has participated within the SBMR program, please tell us, to the best of your knowledge, which of the domains are the most difficult to achieve? Why? 2 How does your facility prepare for the SBMR evaluation? 3 Who directs the process? Externally oriented (e.g., minister, district, facility, community)? 4 What costs are associated with implementing SBMR within your facility? Do these costs represent an obstacle to making progress? 5 To what extent does the SBMR establish the minimum necessary requirements for a health facility to be able to provide quality integrated services? 6 What are the strengths and weaknesses of the SBMR instrument and process? 7 Has the SBMR process motivated health providers to improve the quality of the health facility and services provided? How? Section 2: Achievements and Results 8 What have been the main achievements in terms of health care quality resulting from your engagement (or non￾engagement) with the SBMR process? 9 Has your facility reflected on the results of your SBMR evaluation and worked on the issues arising from this evaluation? 10 Please share an example of health care quality improvement associated with the SBMR process in your facility. 11 In your opinion, do clients recognize the improvements in health care service? Please share an example. Section 3: Performance factors 12 In your opinion, which factors affect SBMR performance of a health facility? 13 In your opinion, which factors result in regression of a health facility that was previously performing well? 14 How does facility leadership influence adoption and maintenance of SBMR-facilitated service improvements? 15 Do you believe that the performance-based incentive system (e.g., stars, public recognition) of the SBMR process is adequate? Section 4: Ownership, reproducibility, and sustainability 16 To what extent does different levels of government show ownership and leadership in the SBMR process? 17 To what extent has SBMR been institutionalized within the governmental process? 18 From your perspective, what potential do you think the government has to take charge of the SBMR process and continue implementation without USAID support? Probe on: capacity, will, ownership, leadership, cost Section 5: Facing forward 19 What other quality improvement approaches are in place in Guinea at health centers and hospitals? 20 Could SBMR become a standardized approach used by all partners and key stakeholders in the country? How? 21 How could USAID partners facilitate the adoption of the SBMR process? 22 Do you have any other comments or recommendations regarding how to improve the SBM-R process in Guinea? Conclusion: I have no more questions for you. Thank you for your participation. Do you have any further questions for Integrated Health Service Delivery in the Post-Ebola Context 105 6 me/us? Comments: Interview quality: 106 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 7 Instrument L. Interview Guide: Decision-makers/Key stakeholders Time estimated: 1 hr Official use only: Interviewer ID: Sex: Region ID: Respondent ID: □ Male □ Female Prefecture ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Integrated RMNCH+A service provision (quality and availability) 1 To what extent has HSD contributed to the availability of the complete package [see facility list] of integrated services (including OF and GBV)? In which areas do you feel HSD has been most impactful? Least impactful? 1.1 2 To what extent did the training on RMNCH+A provided by the HSD activity improve the quality of care and achievement of performance standards? How does your team evaluate the situation? 1.2 3 How familiar are you with the services offered at the community level? Please list those services which are supposed to be provided at the community level. In your opinion, how functional is community-level service provision? How functional is referral from the community to the health facility? How much of a role have the services available at the community and referral from the community to the health centers played in use of health services overall? 2.1 4 How has access and utilization of services increased due to HSD intervention? On what basis do you assess this? 2.2 Section 2. Health Systems Strengthening 5 How would you characterize the leadership role of the [DPS health management team/MOH/NGOs/other] in improving the quality of services in the facilities they work with? To what extent have the MOH and the Governates more generally supported quality improvements? (separate) 3.1 6 In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 7 How has HSD contributed to availability of equipment, commodities, and drugs across target facilities? What have these contributions impacted the ability of health care facilities to provision of integrated care? improvement in DHIS2 or SBM-R indicators? 3.2 8 Which goals or criteria have facilities had the most difficulty achieving? What are the most challenging aspects of these goals or criteria? How do you feel that health facilities can overcome these aspects? 3.2 9 In your opinion, how well has HSD support improved data collection practices at the facility/facilities? In what way (list) has data collected been analyzed to identify key health or service delivery issues at the facility level? How is this done? How have changes in data collection and analysis affected how services are provided or quality improvement efforts at the facility? 10 To what extent did pre-service training prepare you/your frontline staff for the work you/they are doing now in the facility? By provider type? In what areas do you think additional training would have be helpful? (List) In your opinion, how can investments in pre-service training be sustained after the HSD project ends? 3.4 Integrated Health Service Delivery in the Post-Ebola Context 107 8 11 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? What supports will be necessary for sustaining the quality improvements after the project ends? 3.5 Section 3: Special themes: fistula and GBV care 12 How are fistula-related care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in providing fistula-related care or referring for services? How might these challenges be overcome? 4 13 How are GBV care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in integrating GBV care into services? How might these challenges be overcome? In your opinion, what are the underlying causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4: SBM-R 14 What are the strengths and weaknesses of the SBM-R instrument and process, the quality improvement approaches implemented by the HSD project? 8 15 Did the SBM-R process motivate staff to improve the quality of the health facility and services offered? How? 8 16 What were the main achievements in care quality associated with your facility’s engagement (or non-engagement) in the SBM-R process? In your opinion, how do the clients and the community recognize the improvements in service? 8 17 Please provide an example of health care service improvement associated with the SBM-R process in your prefecture. 8 18 In your opinion, which factors affect the SBM-R performance of a health facility? 8 19 In your opinion, which factors are behind the decline of previously successful facilities? 8 20 How has facility leadership influenced the adoption and maintenance of service quality by SBM-R? 8 21 Do you think the performance-based incentive system (e.g., stars, public recognition0 of the SBM-R approach is adequate? 8 22 To what extent is the government, at different levels, demonstrating leadership and ownership in the SBM-R process? 8 23 To what extent has SBM-R been institutionalized within government processes? 8 24 What is the potential for the government to take charge and continue this process without USAID support? Probe: capacity, will, ownership, leadership, cost. 8 25 Which other quality improvement approaches are in place in Guinea at the level of health centers and hospitals? 8 26 How can SBM-R be standardized and used by all partners and stakeholder’s country-wide? 8 27 How can USAID partners facilitate the adoption of the SBM-R process? 8 28 Do you have any other comments or recommendations on how to improve the SBM-R process in Guinea? 8 Section 4. Internal and external partner engagement 29 In your opinion, have the USAID projects that have supported your [facility/prefecture/region/MOH] been complimentary (e.g. built on one another)? How? 6.1 30 In your opinion, to what extent do USAID projects partner with the same health system levels and partners to facilitate synergies across partnerships? 6.2 31 In your opinion, have the USAID projects complimented (rather than duplicated) other donor and MOH RMNCH+A projects in your [facility/prefecture/region/MOH]? How? 6.3 32 In your opinion, were important internal or external opportunities missed in the USAID approach to improving RMNCH+A care in Guinea? How? 6.4 Section 5. Crosscutting Questions 33 What internal and external factors have affected achievement of results (positively or negatively)? How? Which interventions or activities have worked better than others? Why? How has HSD adapted to shifts in the internal and external operating environments (e.g., national policies/strategies, governance structure changes, entry/exit of critical government/development partner/community/private sector/NGO actors)? Conclusion: I have no more questions for you. Thank you for your participation. Do you have any further questions for me/us? Comments: Interview quality: 108 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 9 Instrument M. Questionnaire for Jhpiego Coordinators Time estimated: 1 hr Official use only: Interviewer ID: Sex: Region ID: Respondent ID: □ Male □ Female Prefecture ID: My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Integrated RMNCH+A care provision (quality and availability) 1 Please tell me about your role as regional HSD/Jhpiego coordinator? For how long have you held this position? Who are your key partners? 2 To what extent has HSD contributed to the availability of the complete package [see facility list] of integrated services (including OF and GBV)? In which areas do you feel HSD has been most impactful? Least impactful? 1.1 3 Please share which services are supposed to be offered at the community level? In your opinion, how functional is service provision at the community level? How functional is the referral from the community to the health center? From your perspective, how could community-based health services be improved? referral services? 2.1 4 How has access and utilization of services increased due to HSD intervention? On what basis do you assess this? 2.2 Section 2. Health Systems Strengthening 5 How would you characterize the leadership role of the [DPS health management team/MOH/NGOs/other] in improving the quality of services in the facilities they work with? To what extent have the MOH and the Governates more generally supported quality improvements? (separate) In your opinion, has the HSD project built momentum and commitment of local partners to sustain project interventions after the end of the project? How? 3.1 6 How has HSD contributed to availability of equipment, commodities, and drugs across target facilities? What have these contributions impacted the ability of health care facilities to provision of integrated care? improvement in DHIS2 or SBM-R indicators? 3.2 7 Which goals or criteria have facilities had the most difficulty achieving? What are the most challenging aspects of these goals or criteria? How do you feel facilities can overcome these aspects? 3.2 8 In your opinion, how well has HSD support improved data collection practices at the facility/facilities? In what way (list) has data collected been analyzed to identify key health or service delivery issues at the facility level? How is this done? How have changes in data collection and analysis affected how services are provided or quality improvement efforts at the facility? 3.2 9 In what areas do you think additional training would have be helpful? (List) In your opinion, how can investments in pre-service training be sustained after the HSD project ends? 3.4 10 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, 3.5 Integrated Health Service Delivery in the Post-Ebola Context 109 10 providers and facility managers; facility managers; DPS; MOH directors, other)? What supports will be necessary for sustaining the quality improvements after the project ends? Section 3 : Special Themes : Fistula and GBV 12 Please describe the process of identifying women with OF in your community. What are the barriers to care that local women with OF face? Is local care for OF accessible? How are women facilitated to rejoin their community after treatment? Who facilitates their reentry? 4 13 How are GBV care, treatment and referral integrated into the services available in this region/prefecture? What challenges have you experienced in integrating GBV care into services? How might these challenges be overcome? In your opinion, what are the underling causes of GBV in Guinea? How might these most efficiently be addressed? 5 Section 4. SBMR 14 What are the strengths and weaknesses of the SBM-R instrument and process ? 8 16 What have been the main achievements in care quality in your facility associated with your engagement (or non-engagement) within the SBM-R process? In your opinion, do clients and the community recognize the improvements in care quality ? 8 17 Please share an example of improvement in care quality associated with the SBM-R process in your region. 8 18 In your opinion, which factors affect SBM-R performance in a health facility ? 8 19 In your opinion, which factors are associated with regression in facilities which had previously been performing well? 8 20 How does facility leadership influence adoption and maintenance of care quality through SBM-R ? 8 22 To what extend does the govenment at different levels show leadership and ownership over the SBM-R process? 8 26 Could SBM-R be standardized and used by all partners and key stakeholders in the country? 8 28 Do you have other comments or recommendations regarding how to improve the SBM-R process in Guinea? 8 Section 5. Leveraging of Internal and External Partners 29 In your opinion, have the USAID projects that have supported your [facility/prefecture/region/MOH] been complimentary (e.g. built on one another)? How? 6.1 30 In your opinion, to what extent do USAID projects partner with the same health system levels and partners to facilitate synergies across partnerships? 6.2 Conclusion: I have no more questions for you. Thank you for your participation. Do you have any further questions for me/us? Comments: Interview quality: 110 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity 11 Instrument Q. School Manager Questionnaire Time estimated: 45 minutes Official use only: Interviewer ID: Sex: Region ID: Facility ID: □ Male □ Female Prefecture ID: Provider ID : Age : My name is [INTERVIEWER NAME] and this is my colleague [NAME]. We work in collaboration with the Ministry of Public Health, the Health Service Delivery Project, and USAID. We are inviting you to participate in this evaluation as you are a beneficiary of the Health Services Delivery (HSD) activities, which includes health systems strengthening focused on an essential integrated care package for high quality family planning and maternal, newborn and child health within hospitals and health centers. The information that we are collecting will be used for research purposes only and will be maintained as confidential. The data we are collecting will be analyzed to inform improvements in maternal, neonatal and child health programming and family planning programming. This information will be used for reporting and shared; however, your personal identification will not be included in any results dissemination activities. There will be no consequences to you whether or not you choose to participate in this interview. If at any time during this interview you are uncomfortable with a particular question, that question can be skipped. on, that is fine. There are no direct risks or benefits associated with your participation in this study. If you choose to participate, we will ask you a series of questions regarding your perspectives on HSD project activities in maternal, neonatal, and child health and family planning. The interview will last approximately 45 minutes. If you are comfortable participating in this interview, we can start now. Your acceptance to participate in this interview signifies your consent to participate. Would you like to participate? □ Yes □ No Section 1. Provision of integrated RMNCH+A services (quality and availability) 1 Did you provide/receive training on the following MCH/FP quality of care and performance standard topics (list of broad topic areas)? When? How much? 1.2 Section 2. Health Systems Strengthening 2 To what extent are graduating midwives prepared to offer the complete package of integrated services for MCH and FP? (likert scale) In what areas do you think addtional training would have been helpful? (List) To what degree does your program offer the needed addtional training? Please explain. 3.4 3 In your opinion, can HSD project interventions to improve the quality of care be sustained after the project ends? How? 3.4 4 Who will be primarily responsible for sustaining the quality improvements (individual providers and CHWs, providers and facility managers; facility managers; DPS; MOH directors, other)? What supports will be necessary for sustaining the quality improvements after the project ends 3.5 Section 3. Special Themes : Fistula and GBV 18 Is training provided on GBV screening, treatment and referral through the midwifery training program? 5 Section 4. Leveraging of Internal and External Partners 20 In your opinion, have the USAID projects that have supported your [facility/prefecture/region/MOH] been complimentary (e.g. built on one another)? How? 6.1 21 In your opinion, to what extent do USAID projects partner with the same health system levels and partners to facilitate synergies across partnerships? 6.2 22 In your opinion, have the USAID projects complimented (rather than duplicated) other donor and MOH RMNCH+ projects in your [facility/prefecture/region/MOH]? How? 6.3 23 In your opinion, were important internal or external opportunities missed in the USAID approach to imprpoving RMNCH+ care in Guinea? How? 6.4 Section 5 : Crosscutting Questions 24 What internal and external factors have affected achievement of results (positively or negatively)? How? Which interventions or activities have worked better than others? Why? 7 25 How has HSD adapted to shifts in the internal and external operating environments (e.g., national policies/strategies, governance structure changes, entry/exit of critical government/development partner/community/private sector/NGO actors)? 7 Conclusion: I have no more questions for you. Thank you for your participation. Do you have any further questions for me/us? Notes : Quality of interview : Integrated Health Service Delivery in the Post-Ebola Context 111 Instrument V: Data Validation Region ID : __________ Region ID : _______ Prefecture ID : _______ HC ID :_______ Community ID : _______ Indicator 1 : 4 ANC visits (dates) Indicator 2 : Assisted Delivery (date) Indicator 3 : AMTSL (partogram) Indicateur 4 : Postnatal visit (1 visit) Client name Age Source : Obs Record Source : ANC Card Source : Obs Record Source : Birth registry Source : Obs Record Source : Obs Record Source : Postnatal registry 1 2 3 4 Instructions: Indicator 1: ANC (DO) record the date of the 4 most recent ANC visits (if less than 4, capture all). ANC Registry: Identify dates and namesof the client, record yes if found. Indicator 2: Assisted Delivery (DO) record the date of birth/admission/hospitalization. Registry: Identify the dates and find name of client, record yes if found. Indicator 3: GATPA : (DO) verify the completion of the partogram and record. Indicator 4: PP visit : (DO) record the most recent date of PP counseling. ANC: review in the PP registries and record yes if found. 112 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity ANNEX IV: SOURCES OF INFORMATION As described within the evaluation methods section, primary data collection activities took place within a set of purposively selected hospitals (n=10) and health centers (n=26), listed below in Table 10, and with key regional, national, and partner stakeholders. The full list of respondents by type of method and respondent are detailed in Table 1. Permission to share individual names was not obtained from in-depth interviewees or focus group participants, and thus is not listed within this report. Region Prefecture Facility Name Facility Type SBM-R Case Study Boké Boffa Boffa Health Center Yes Boké Boké Sangaredi Health Center No Boké Boké Bintimodiya Health Center No Boké Boké Boké Hospital No Boké Fria Sabende Health Center No Boké Gaoual Gaoual-Centre Hospital No Boké Koundara Youkounkoun Health Center No Conakry Conakry Macire Health Center No Conakry Conakry Lambanyi Health Center Yes Conakry Conakry Yimbaya Health Center No Conakry Conakry Ratoma Hospital No Conakry Conakry Ignace Denn Hospital Yes Faranah Kissidougou Dar-es-salam Health Center No Faranah Kissidougou Heremakono Health Center No Faranah Kissidougou Kissidougou Hospital No Kankan Kankan Missira Health Center No Kankan Kankan Moribayah Health Center No Kankan Kerouane Kerouane Hospital No Kankan Kouroussa Kouroussa Hospital No Kankan Mandiana Mandiana Health Center Yes Kankan Mandiana Faralako Health Center No Kankan Siguiri Niandankoro Health Center No Kankan Siguiri Niagassola Health Center No Kankan Siguiri Kintinian Health Center No Table 11. Primary Data Collection Sites by Region, Prefecture, and Type of Facility Integrated Health Service Delivery in the Post-Ebola Context 113 Region Prefecture Facility Name Facility Type SBM-R Case Study Kindia Coyah Wonkifong Health Center No Kindia Coyah Coyah Hospital No Kindia Dubreka Dubreka Health Center No Kindia Forecariah Kakossa Health Center Yes Kindia Kindia Wondy Health Center Yes Kindia Telimele Daramagnaky Health Center No Labé Labé Labé Hospital Yes Labé Lelouma Lelouma Health Center No Labé Tougue Tougue Health Center Yes Mamou Dalaba Kebali Health Center No Mamou Mamou Sabou Health Center No Mamou Mamou Mamou Hospital Yes Table 11. Primary Data Collection Sites by Region, Prefecture, and Type of Facility continued 114 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity ANNEX V. STRATEGY REVIEW GROUP As described in the methods section, a Strategy Reference Group (SRG) consisting of 5 individuals with strong contextual knowledge and who were previous end users of similar project evaluations was convened to critically review the findings compiled and analyzed by the evaluation team and to debate, prioritize, and strengthen the recommendations that emerged from the findings. SRG members provided written ratings and feedback on the report and preliminary recommendations. They participated in a group discussion to refine interpretation of our findings and translation into recommendations that are actionable and informative for USAID and which speak to USAID’s comparative advantage, strategic priorities, and contextual realities. Below we present our original recommendations, and the SRG ratings and specificity, responsiveness, priority, and feasibility, and comments that informed our revisions. Strategy review group members: France Donnay, Consultant, Women’s Health Global Policies, Programs and Practices; Elizabeth Kibour, Senior Advisor for Country Support Systems & Operations, USAID/GHB/DC; Joseph Mwanji, Independent Consultant, Program Analyst and Adaptive Management; Marcel Ouendeno, Technical Advisor, support to the National Health Security Agency (ANSS) Guinea; Regional Pandemic Prevention and Control Program (RPPP) in the ECOWAS region; and Tisna Veldhuijzen van Zanten, Former Chief Science Officer/Senior Vice President, URC, Retired. Integrated Health Service Delivery in the Post-Ebola Context 115 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.1. Ensuring the Continuum of care: While progress was made in HSD-targeted facilities in separate perinatal care indicators, the proportion of women receiving the full continuum of perinatal care was extremely low. Targeted strategies to ensure women receive the full continuum of perinatal care is needed for optimizing maternal health, particularly ensuring continued engagement through the postpartum period. (Relates to Finding 1.2) The mother and child health cards are being used by providers and held by women, these could be leveraged for monitoring and promoting continuum of care, as the postnatal period was most often found empty in these cards. With the scale up of CHW/RECO s/RECO, they could play a larger role in monitoring and encouraging the continuum, beyond the ANC attendance. S 3 3 4 3 4 3.4 This finding and recommendation was perhaps known by the design time and probably included in the solicitation or in the winning proposal. What could improve specificity is to find out if there were some population groups defined by a combination of variables/attributes such as location, age, education, income, urban/rural where we see significant improvement and groups with least progress. i.e. classify responsiveness to current strategy/ effort (example discriminant analysis, or simple more granular analysis of routine service delivery data) and recommend differentiated approach by these strata. As is, I don’t believe it makes any difference to add this to priority action plan, unless USAID thinks this is a completely new idea, that was not in previous solicitation. If specific groups are identified, it would become a priority 5. The recommendation only addresses one aspect of continuum of care and focused on post partum services. The data also show inconsistent use of antenatal services. It does not address any aspect of nutrition, neonatal and early child wellbeing care... In addition to the role of ASC/RECO, providers must also insist on the continuum of care during ANC, especially 3rd and 4th visit.. Maintaining continuum of care is known to be very difficult, especially in places where systems are not fully functional, and access to care is socially and financially difficult. The evaluation report mentions impressive results (83% of facilities report progress) however it is not clear whether findings are based on self-reporting by providers only, or directly observes ANC and births as well. Good information on challenges encountered during the evaluation is included in the report, and is helpful for understanding the limitations of the evaluation work. Use of postnatal care is always the lowest for a variety of reasons, including beliefs about confinement as well as religious and social significance about community/family care based care for newborns (feasibility rated 2). In no way does this comment reduce the importance of safe and competent postnatal care (priority rated 4). The recommendation suggest optimizing and scaling up CHW/RECO s, which is good, provided they are trained and supervised, and have a caseload/volume high enough to recognize at risk/fragile babies and sick mothers (responsiveness rated 3). Thus, postnatal care visits with trained professionals are still paramount. Maternity homes for pre- and postnatal stays are another solution that could be discussed. In addition, an in depth analysis of reasons behind low attendance and determinants of postnatal care seeking in these specific populations could have been included in the recommendation. R 4 3 4 3 3.5 P 2 4 5 4 4 3.8 F 4 2 4 4 2 3.2 116 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.2. Referral services: Referral mechanisms were inconsistently available for bringing women in urgent need to higher levels of care, and often required immediate out of pocket payment of the family which delayed care seeking, sometimes with dire effects. HSD and future USAID supported activities have not, as yet, given sufficient priority to the referral system. Support for the institutionalization of a functional referral system through dedicated funding and facilitation of coordination, partnership with communities, and innovation is urgently needed to mitigate delays in access to emergency care for the poorest households. (Relates to Findings 2.1.2). S 4 3 4 3 2 3.2 Effective and efficient referral systems are critical in maternal and perinatal health. An evaluation of a two-country proof of concept activity titled saving mothers giving life (Uganda and Zambia) showed that quick and timely referral of emergencies makes a lot of difference. It is a high priority. Most of the components of referral systems can be implemented easily but some may be stickier requiring more resources. Examples of the latter include access beyond regular work hours, competent staff, transport, etc. The recommendation includes several items, and is thus hard to score. Given the lack of central level funding and institutional support, the likelihood of the government taking on referral services (i.e. with ambulances) is very low. The reference to “innovation” is good, but lacks specificity. There are best practices from other countries that might be leveraged in Guinea. The referral system is neccesary in the Guinea context but its also neccesary to include the counter referal system and harmonization of referal and counter referal forms, as well as availability in sufficient quanitiy and quality. ..Please explain/ comment on your ratings: Referral systems are not functional anywhere, with very rare exceptions, such as The Netherlands, or heavily funded and supported initiatives, in Ethiopia and Bangladesh for instance. Such systems are very hard to organize, for logistical, geographical, behavioral and coordination reasons. Moreover, referring complicated cases is often not the solution: the patient after having travelled and often being denied access, bouncing from place to place, reaches the right place too late or never. Maternal health programs should help women who live away far from safe care move before labor or in early labor, before complications arise. This has been done in India with a large transportation system, and elsewhere with maternity homes, although both interventions have implementation challenges as always. Whether this is possible in Guinea is not discussed, but the recommendation is too high level, vague, and plagued with implementation bottlenecks. I rated the priority at 3 for equipoise, but this needs a much deeper engagement to begin to unveil the right solution for this setting. R 5 4 5 3 2 3.8 P 5 5 5 4 3 4.4 F 4 3 5 2 3.5 Integrated Health Service Delivery in the Post-Ebola Context 117 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.3. Obstetric fistula: The model employed for fistula surgical services did not evidence full integration of care due to limited geographical availability and most facilities providing care only intermittently. Despite project successes in equipping, training, and conducting surgeries, cost and scope of the investment in a few facilities and surgeons limited the number of surgeries possible which in turn impacted the skills development of surgeons under training. Obstetric fistula (OF) is a debilitating condition that leaves affected women stigmatized, marginalized, and often pushed to the fringes of their communities. As efforts continue to prevent incidence of fistula, a large number of women remain in need of surgery for the condition. Fistula treatment, care and reintegration cannot be sustained without external investment. As a gender and human rights issue, it should continue to be a critical component of future investments in women health by donors and the government to ensure services reach the most vulnerable. Consideration should be given to improve the delivery of adjunct (i.e., physical therapy, psychosocial and economic needs) and follow-up care for women who have undergone OF surgery to improve health outcomes in subsequent pregnancies. (Relates to Findings 4.1). S 2 4 4 5 3.8 Simply stating that “As a gender and human rights issue, it should continue to be a critical component of future investments” makes that sound like a standardized evaluation comment. Though it comes directly from the findings and OF can be included in future interventions, it makes little meaning as is. I think the recommendation should have dealt more with the “scope”, funding, scale and how communities, civil society, religious organizations, etc should come together and partner with the government, donors and private sector to address this challenge. High feasibility rating is for continuing to consider this a critical component of future investments. I would rate scope, funding, scale and civil society partnership lower (4). We must also envision measures against the stigmatization of these women and their marginalization in order to guarantee their reintegration into the community...Good job here. The recommendation fully describes the problem and the challenges to implementing solutions, and rightly mentions prevention, as well as treatment and reintegration. But given how hard this issue is to address, I gave a 3 for priority and a 2 for feasibility. Here too, striving to get women closer to a safe place for labor and delivery care is a key component of prevention activities. Surgery should be provided in a few highly specialized cases, to ensure caseload and maintain competence. R 4 5 5 3 5 4.4 P 2 5 5 4 3 3.8 F 5 4 4 5 2 4.0 118 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.4. Gender-based violence: Gender-based violence and harmful practices are pervasive in Guinea, even if some improvement has been reported in recent years. HSD’s pilot investment in GBV was insufficient to address the challenges. Future investment in GBV should expand the focus of GBV prevention and treatment efforts to include national and regional governance, coordination and alignment across Ministries, and development partners. Gaps in stewardship of cases between the implicated Ministries needs to be addressed at all levels. Expansion of the focus of GBV activities to include other prevalent harmful practices in Guinea such as female genital mutilation is needed. (Relates to Findings 5.1) S 2 3 4 3 3 3.0 I think the problem comes from the lack of a substantive or well analyzed finding. Verify the assumption that current GBV effort did not include national and regional governance – it is hard to believe that. What approaches did the pilot test? Anything that was found effective that should be scaled? Although GBV itself is high priority, I rated this a very low priority because I don’t think it says much that USAID should do in the future, that wasn’t (perhaps) already in the program PD (not reviewed). The recommendation as written is clear; however, it is a highly complicated topic and rooted in societal norms outside of the scope of this project. It can only be solved as part of a much broader multisectoral BCC/political campaign. Also important to consider early marriage....Another difficult area to address. The recommendation rightly mentions prevention as well as treatment, but could be more specific by adding the importance of attention and care for perpetrators, a key area for prevention. Still, it is important to note that GBV screening is now part of the service package. FGM is important as well but very different in terms of causes as well as program design and implementation. I am not sure that those harmful practices can and should be addressed together as a bundle. The focus here is on leadership action (Ministries). While policy changes are important, interventions should be co-created and co-implemented with families and communities. R 4 4 5 3 3 3.8 P 1 5 5 3 4 3.6 F 4 2 5 4 3 3.6 Recommendation 1.5. Adolescents and youth: Women of reproductive age that are adolescent and youth are increasing with 41% of the population under 15 years of age. Despite their numbers and need, there is a lack of dedicated service strategies, approaches, training and government level plans to meet this growing need. There are no specific service standards on how to counsel and provider services for younger women’s sexual and reproductive health needs appropriately. Given the demographic imperative, HSD should urgently work with the MoH and partners to develop service standards, training and indicators to measure improvements in service quality for adolescent girls and young women. Likewise, pre-service training at the midwifery, nursing and medical schools and faculty should include curricula on youth friendly service provision. (Relates to Findings 3.4.2) S 5 5 5 3 2 4.0 Could not verify that the finding includes weaknesses in curricula and primary training. Service standards and a responsive curricula are critical and basic, but their effects are more long￾term than short-term. This recommendation should include a short-to-medium term action..Its the initial in-service training that must be revised and adapted to the needs of adolescents and young people and the care practices in our health facilities..The first half of the recommendation merely mentions the problem. Next comes a vague, high level recommendation on standards – it is surprising that this is not done yet, given that health programs for youth have been studied, implemented and evaluated in many countries for years. On the other hand, highlighting pre-service training is an excellent recommendation. R 4 5 5 4 3 4.2 P 4 5 5 5 5 4.8 F 5 4 3 5 3 4.0 Integrated Health Service Delivery in the Post-Ebola Context 119 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.6. Equipment: HSD facilitated implementation of performance standards by making the necessary equipment available for their implementation. This critical contribution however was not embedded into a broader strategy (beyond a facility-based memorandum) to ensure maintenance, oversight and accountability for its upkeep. Equipment malfunction, misuse and theft impacted the quality of care provided despite best intentions. Feedback mechanisms and answerability with punitive measures are needed in the system to counter impunity. Lack of managerial accountability and control of equipment donations at facility and district level should be addressed through stringent agreements between projects and the MoH accompanied by close monitoring, accountability and recourse mechanisms, and reporting. (Relates to Findings 3.1.4) S 4 5 5 4 4 4.4 Though scored highly, “close monitoring, accountability and recourse mechanisms, and reporting” are necessary to maintain the equipment functional and providing the much needed services; but this may not be sufficient. I think this recommendation can be improved by adding some “carrots and sticks” to keep them. How about sensitization of community to demand accountability, more support tied to facilities utilizing the equipment efficiently, some punitive action required from MOH, etc. It is hard to discern from the report to what extent the central MOH was involved in deciding on and managing equipment needs, distribution and oversight. Projects should not manage equipment procurement and donations without close integration with government authorities. Maintenance system strengthening could be integrated in new program design..Strongly involve MOH in the monitoring and control of the equipment donated by the partners. Put in place a systematic information mechanism for the MOH on the donation of equipment to facilities to facilitate verification during joint supervision. Important recommendation, linked to the section on health systems strengthening – as well as rec 1.7 on labs and 1.9 on infrastructure – these could be grouped together in one single section since challenges and potential solutions are very similar. I rated feasibility very low because unless leadership and accountability issues are not solved, I don’t think much can change in all these areas. Obviously the problems mentioned heavily impact on results, both in terms of processes and health outcomes. Similarly, as mentioned in the report, individual, facility level leadership can be very effective. R 4 5 5 4 4 4.4 P 5 5 5 5 4 4.8 F 5 3 5 5 2 4.0 Recommendation 1.7. Laboratories: Lack of investment in laboratories was a missed opportunity within the HSD activity. Within maternal care services, particular components of antenatal care could not be provided where laboratories were not operational. Future projects that aim to improve the quality RMNCH care require concomitant investment into functional laboratory services. In addition to a facility-specific focus, laboratory skills development must be supported within the professional health schools. (Relates to Findings 3.4.2 and 6.6). S 3 4 5 3 3 3.6 The recommendation directly addresses the finding but it is not clear what “require concomitant investment into functional laboratory services” means. Making the labs functional is critical to getting better RMNCH results but the first part of the recommendation is not easy to implement. Recommending a critical equipment gap analysis and advising that future projects include that set of equipment, training and supplies can improve specificity of this recommendation. Missing in this recommendation is one major aspects and that is that the government ought to participate in the WHO-led accreditation system for laboratories. Just building staff capacity is inadequate to ensure high quality and reliable laboratory services... But also the allocation of commodities (reagents) and laboratory equipment to enable quality care. Establishment of a self￾financing mechanism for these commodities by benefitting facilities to ensure sustainability after the project..Here again it is hard to believe that functional labs can be established given the context. So the recommendation while specific, is rated low in terms of feasibility. Other, innovative strategies for the continued R 5 4 5 3 3 4.0 P 4 5 5 4 4 4.4 F 2 3 5 4 2 3.2 120 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments continued relatively small range of antenatal checks could have been explored: sending samples with drones, with a stringent zipline to make sure results come back in time, with short delays; or grouping of samples transported by car to hospital labs… the recommendation does not specify where future labs could be installed, cost implications and only mentions need for training, which is not effective in isolation. Recommendation 1.8. Supply chain management: HSD invested in service quality improvements that were hampered by supply chain ruptures at facility level. Though supply chain management was not within the scope of the HSD activity, and other supply chain projects and activities are on-going, HSD and future RMNCH+ projects must create strategic partnerships and adaptation strategies to overcome fundamental gaps in essential medicines and supplies necessary for the delivery of care to standard at the facility level. Further engagement is necessary beyond advocacy with the Ministry of Health to adjust the facility commodity provisioning process, avoiding stock-outs. (Relates to Findings 1.3, 3.2.3)...“Further engagement is necessary beyond advocacy with the Ministry of Health to adjust the facility commodity provisioning process, avoiding stock-outs.”It is not clear what further engagement means. S 2 3 3 5 2 3.0 Yes, effective supply chain is essential, and the finding did reveal current performance was inadequate. But did the data support the conclusion that supply chain weaknesses were there because of weak or inexistent partnerships? A recommendation like “create strategic partnerships and adaptation strategies to overcome fundamental gaps in essential medicines and supplies” is not specific and will be hard to measure. “Further engagement is necessary beyond advocacy with the Ministry of Health to adjust the facility commodity provisioning process, avoiding stock-outs” is not specific at all. Non-specificity makes feasibility, hence the prioritization for this particular recommendation, though in an area very important to HSD. “Further engagement is necessary beyond advocacy with the Ministry of Health to adjust the facility commodity provisioning process, avoiding stock-outs.”It is not clear what further engagement means. Allocation of commodities (reagents) and laboratory equipment to enable quality care. Establishment of a self-financing mechanism for these commodities by benefitting facilities to ensure sustainability after the project... One cannot disagree with such a vague recommendation. Maybe the reason behind it is that HSD was not designed to address supply chains issues, as mentioned in the report. Again, linkages with health system strategies, including equipment and infrastructure… could have been made. Perhaps the format of the document could be amended to include a section where potential synergies could be highlighted, whereas we have here a series of discrete topics, which necessarily leads to repetition. Feasibility is low as the implementation of such activities is contingent on government‘s interest and investments in public health. R 4 5 5 4 3 4.2 P 3 5 5 4 4 4.2 F 3 3 5 4 2 3.4 Recommendation 1.9. Infrastructure: HSD contributed small refurbishments and infrastructure upgrading which was greatly appreciated by community members, providers, and MoH managers who reported the facility improvements were a strong motivator for engagement in quality improvement processes at their continued S 3 4 5 3 4 3.8 Could have done better at specifying which facilities need what infrastructure. This leaves it too general and difficult to plan for, and implement. For example, which geographic location suffers most and what are the other characteristics that guide infrastructure decisions? What are the most critical infrastructure? “A focus on providing the location for private and confidential care for family planning, and antenatal care services” Need to phrase this recommendation more explicitly. Renovations should where possible include adding or refurbishing space for private consultations. continued R 4 5 5 3 3 4.0 P 3 4 4 4 3.8 F 3 5 3 or 4 4 3 3.8 Integrated Health Service Delivery in the Post-Ebola Context 121 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 1.9 continued facilities. While recognized as the remit of the government, and cost intensive, some investments can stimulate change in practice. A focus on providing the location for private and confidential care for family planning, and antenatal care services will improve respectful care practices, and is a critical part of adolescent-friendly care. (Relates to Findings 2.1.1, 3.2.1 and 3.2.2) S 3 4 5 3 4 3.8 continued Take good stock/inventory before investing in infrastructure in certain structures : I personally observed that the infrastructure did not meet the priority needs of the health facility (i.e: in Mamou a storage shed was built while the priority need was the extension of maternity by building another hospital room)... Indeed, small investments can go a long way. Good recommendation. The last sentence is a bit unclear, as it mentions “private care “meaning privacy not privately owned clinics? Why mention privacy only? while important, it is not the only issue. As well, providers’ perspectives are hardly mentioned anywhere except where recommendations specifically address training. R 4 5 5 3 3 4.0 P 3 4 4 4 3.8 F 3 5 3 or 4 4 3 3.8 Recommendation 2.1. Community Health Policy: To foster implementation and sustainability of the Community Health Policy and the integrated community health service package, project support is needed for tools, supervision, and training of CHW/RECO /RECO. It is expected that these contributions will be preceded by financial support for human resources that remain the responsibility of government through decentralized funds. In order to hold the government accountable to the policy and given the HSD experience, it is not recommended to use project funds for paying community level workers, but rather strengthening the capacity of the new cadres of CHW/ RECO s. The inputs should especially target promoting integration as guided in the Norms and Procedures for Reproductive Health, to begin the shift from vertical CHW/RECO interventions to more integrated outreach. As vaccination/child health days are regularly implemented and engaging CHW/RECO s and mothers, guidance could be created on using this as an entry-point for integrating ANC/Postpartum follow-up with pregnant and lactating women for integrated community health days. (Relates to Findings 2.1.1; 2.2.1; 3.5) S 4 4 4 3 3 3.6 A fairly good quality recommendation but I was not able to follow how the details in the recommendation come from data, conclusions and findings. Raising consistent funding for payment for CHW/RECO s will not be easy. Integrating CHW/ RECO s in public health services is challenging. It is clear that the government’s approach can only work if adequate and consistent funding is provided to CHW/RECO s. It is not evident from report that this is the case. Given the absence of Save the Children’s voice in the evaluation, it is also not clear to what extent HSD did put in adequate technical and financial resources to strengthen community health services... Clear recommendation, particularly on payments for CHW/RECO . Then the second part on integration is wishful thinking, since incentives, structures, reporting lines … follow vertical programs: immunization versus family planning versus antenatal care, … integration is much talked and written about, and community health is a natural locus for it, but trends pulling the other way are quite powerful. Vaccination as an entry point for ANC for instance has been mentioned since decades and have not become a reality. This type of recommendation is not very helpful. R 3 4 5 3 4 3.8 P 5 5 5 3 4 4.4 F 4 4 4 3 2 3.4 122 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 2.2. Health Posts: With the Community Health Policy and Norms and Procedures for Reproductive Health in place, future investments should prioritize assessing and revitalizing Health Posts as the most accessible entry point for communities into higher levels of the health service delivery system. Although Health Posts have significant short falls in supplies, infrastructure, and human resources, they were not a focus of HSD. Given the strides made in increased demand and use in the post-Ebola context, the quality of health posts must be improved to maintain momentum of increasing demand and access in part as a result of HSD interventions in the health system. (Relates to Findings 2.1.1) S 2 5 4 3 4 3.6 I did not find a finding or support for a finding that Health Posts are ‘the most accessible entry point for communities into higher levels of the health service delivery system”? “assessing and revitalizing” is not specific. What is in that package? Does it change from facility or location to another? Same applies to “quality of health posts must be improved”. If the finding exists or research supports this fact, the specificity rating can be higher (4).. Are health posts really a solution? They are difficult to maintain, staff…and as the population increases, and motivation of a shrinking health force decreases, they will become even more so. Why not focus on health centers for basic care, now that communications and logistics have improved in most places, and they have a high degree of autonomy? If such a recommendation is valid and it might be the case, then it should explain how quality could be ensured, thus including not only WHAT to do but HOW to do. R 3 3 5 3 3 3.4 P 4 3 4 4 2 3.4 F 3 3 4 or 5 4 2 3 Recommendation 2.3. Fee for Services: Access to RMNCH+ ser￾vices continue to be limited by out of pocket payment demands for specific services. There is an urgent need to improve policy and enforcement of free RMNCH+ financial access barriers. Upstream advocacy is needed to revisit the policy of 2011 attesting free RMNCH services to have better clarity and enforcement. Likewise, down￾stream community efforts to promote Community Health Mutuals should be strengthened and scaled up. It is also necessary to work with communities on social accountability actions to inform people of their rights and call services to account for illegal costs. Such awareness raising should go beyond the written/poster communi￾cation channel that was employed by HSD and other programs to include a rights-based information, communica￾tion and education to make commu￾nities, as rights holders, aware of the entitlements, and health providers, aware of the obligations as duty bear￾ers. (Relates to Findings 2.3.2) S 3 2 5 3 4 3.4 This recommendation has a wide range. The first part about policy and upstream advocacy is not clear, and implementing “truly” free services policy would not be easy. The part about work with communities on social accountability is sufficiently clarified with the statement that recommends going “beyond the written/poster communication channel that was employed by HSD and other programs to include a rights-based. ”..Well written and specific recommendation- it is critically important that the government policy of free RMNCH services is clearly communicated and adherence enforced.... Here as well, an important priority for action. The recommendation does not mention accelerating policy changes around Universal Health Coverage, free services, national health insurance policy, probably because it pertains to community level. A more holistic set of recommendations around that issue would be helpful. Concrete and novel suggestions would be welcome a well, as for instance, using social media to claim for rights, quality and respectful care. R 2 3 5 3 4 3.4 P 3 4 5 3 4 3.8 F 5 2 4 3 4 3.6 Integrated Health Service Delivery in the Post-Ebola Context 123 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 2.4. Youth economic empowerment and family planning: Use youth economic empowerment approaches to achieve FP uptake by youth. Youth face ongoing economic challenges as barriers to utilization of health services and male/spouse acceptance also remains a challenge. Entrepreneurship and economic development were consistently mentioned in male (and female) FGDs. This demand should be harnessed in revamping FP messages promoting the economic benefits of FP and in creating innovative models of combining entrepreneurship opportunities with health messaging to increase demand and access. (Relates to Findings 2.3.3) S 3 2 5 3 4 3.4 Though integration of FP with youth economic empowerment activities can be quick win (easy to implement), I rated priority lower because I didn’t see enough empirical evidence that this would increase demand and uptake for FP. That it is frequently mentioned in FGDs is not enough support that it works. Do we have evidence of situations where it worked well? ..A good, short and action oriented recommendation, on a key integration opportunity between health services and income generation and empowerment activities. R 2 3 5 3 4 3.4 P 3 4 5 3 4 3.8 F 5 2 4 3 4 3.6 Recommendation 2.5. Gender bar￾riers: Gender barriers continue to im￾pede service utilization and RMNCH+ outcomes in Guinea but hardly recog￾nized or accepted by the majority of respondents. Gender inequality, along with the persistent early marriage and FGM practices, deserves greater atten￾tion in HSD and future programming. Strong attention and importance to the issues are needed to address gender equity in design, implementation, measurement and reporting. (Relates to Findings 1.3, 2.3.3, 5.1) S 2 4 3 3 2 2.8 “Gender barriers” is a critical challenge but I do not find the statement “deserves greater attention in HSD and future programming. Strong attention and importance to the issues are needed ….” specific enough for implementation. What worked and what didn’t? where and at what scale? ...Yes, but how could this be done? Maybe the issue here is that programs like HSD as well as the evaluation are not meant to address deeply rooted societal issues — so the evaluation question is misplaced. R 4 4 5 3 3 3.8 P 4 5 5 3 4 4.2 F 3 4 5 3 2 3.4 Recommendation 3.1. Leadership and Ownership: Lack of leadership, oversight and accountability offsets the sustainability of achievements. HSD recognized gaps in health system leadership but did not change programmatic course to address the issue. In fact, HSD’s interventions to compensate for leadership gaps may have perpetuated the problem, as managers become can become reliant on partners to lead. Future USAID activities can stimulate leadership and continued S 5 3 3 3 4 3.6 Recommendation is straightforward, direct from a finding, important and easy to implement. As written the recommendation combines observations and recommendations. It is outside of the scope of HSD to ensure GOG leadership and commitment. This is really the purview of USAID. HSD’s role and possible overcompensation is a direct consequence of project design. HSD is responsible fore results, which are not measured in terms of government engagement and leadership.There are approaches that can foster increased engagement from MOH staff from the outset. The report does not go into whether HSD worked closely with and gave responsibility to central and regional leaders and managers. USAID has successful project design models including performance based grants to regions continued R 5 4 5 3 4 4.2 P 5 5 5 4 5 4.8 F 5 4 5 4 3 4.2 124 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 3.1 continued ownership by the government and MoH in their design. Results-based measures can require accountability for project deliverables including successful implementation of standards with functioning equipment, maintenance, and reporting. The design of future projects could also consider transferring responsibility for the project activities over time based on performance measures. Co-design of projects to address critical needs of health facility staff, such as access to water, electricity and provider accommodation, can also facilitate increased ownership. (Relates to Findings 3.5.1) continued that foster increased leadership and oversight of results by GOG staff... This recommendation emphasizes the crux of the problem with health care delivery in Guinea. Although the recommendation includes several avenues for addressing this very difficult issue, a more pointed analysis of the current leadership environment might be necessary. It is possible that substitution is still the solution for now, while gradually moving towards ownership and coordination by government authorities. Another issue mentioned in the report but not addressed here is collaboration between public and private, as well as NGO run facilities, while most providers have dual practice, and equipment obviously travels between places. Recommendation 3.2. Social Accountability: There were missed opportunities to embed accountability within the health system. Working with COSAHs or NGOs at the community level could have stimulated social accountability for health system responsiveness to community needs, strengthened linkages between project interventions and the community, and increased community expectations for quality care. HSD and future projects should involve the community and their representatives more actively in quality improvement mechanisms and facilitate their understanding, appreciation and demand for quality health care services. (Relates to Findings 3.1.2) S 4 4 4 4 3 3.8 Providing illustrative interventions would have made this more specific than leaving it at “should involve the community and their representatives more actively ….”. Are there examples of interventions that had success increasing social accountability? Quote them and what they did. Are NGOs recognized partner for public health services in the country? Not clear from report... Community participation (beneficiaries) at all stage of the project is essential for the success of the project... Good recommendation, with action points that in part overlaps with 2.3 Fees for services. I too believe that in such context, community based work might be the most promising way to achieve social accountability and better quality of services. R 4 3 5 4 4 4 P 5 4 5 5 4 4.6 F 4 4 4 5 3 4 Integrated Health Service Delivery in the Post-Ebola Context 125 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 3.3. Institutionalization of SBM-R: To sustain SBM-R as an effective quality improvement measure, HSD should support the institutionalization and systematization (possibly through tablet-based data collection) of the SBM-R process. Targets and progress should be tracked and benchmarked; challenges should be discussed at subnational, and benchmarked; challenges should be discussed at subnational, and if common national level to be collective addressed. SBM-R can be a unifying process to facilitate use of data for decision making and action by the MoH. HSD can facilitate the process by working with the MoH to own the process and provide the necessary leadership through the appropriate National directorates (Family Health and Nutrition, Bureau de Strategies et Developpement, National health information systems, and the University). Cultivating further facility-based support will be needed to expand and sustain the success of the SBM-R process given quality improvement is a continuous process that must be maintained by the facilities themselves. (Findings refer to 1.3, 1.5, 1.6, 3.1.1, 3.1.2, 3.5.2) S 5 3 4 5 4.25 Did not find proof from literature or other data on the effectiveness of SBM-R interventions. Besides are they easy to administer? Absent a national policy for quality improvement, and resources to assist the MOH implement national guidelines and training, oversight and accountability, no project can put in place lasting QI as integral to health service delivery. The efforts by HSD are well intended and have made positive changes, at facility levels, but are too dependent on individual rather than institutional leadership... Could not agree more, and in this section, recommendations are more specific and more actionable. R 4 5 5 4 5 4.6 P 3 4 4 5 4 F 4 2 3 or 4 4 4 3.5 126 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 3.4. Health information systems: HSD has made significant contribution to a number of critical health systems structures that require additional future investment. Through the SBM-R process and data monitoring and review (at monthly meetings), HSD has contributed towards the institutionalization of use of data for decision-making towards quality improvement. Past investment in DHIS2 was further supported by HSD. To ensure DHIS2 is fully functional and utilized to its potential in Guinea, however, additional technical and managerial support is needed. Beyond putting the software in place, emphasis should be on supervision of data quality, timeliness of reporting, analysis at sub-national level to facilitate use of data at district and facility level to improve quality of care for better health outcomes. (Findings refer to 3.3.1) S 3 5 5 4 5 4.4 Data analysis that gave rise to this recommendation should have identified performance differences by region and identify the critical factors for success. Giving “additional technical and managerial support” generally is not specific enough. For example is there a national enterprise architecture that development partners are contributing to? Are there centers of excellence for quality, timeliness and data use? What exactly should be adopted? HIS is a heavy lift that takes resources and requires strong coordination and leadership. Recommendation is on point... Again, fully agree. While making data systems fully functional, timely, and connected to implementation improvements is obviously a big task, it is of high priority. It sounds that resource mobilization to make it possible is highly necessary, as well as increasing local/national capacity to analyze data for action. R 5 5 5 4 5 4.8 P 4 5 5 4 5 4.6 F 3 5 5 4 4 4.2 Recommendation 3.5. Public and private midwifery and nursing schools: HSD investment in public and private midwifery and nursing schools is novel and important. The activity has demonstrated success having put in pace updated curriculum with current RMNCH and FP performance standards and a skills lab to provide practical training on critical procedures. The fact that new skills labs are being used by other schools indicates the value placed by regional stakeholders on the learning resource. Ownership and pride by the school management of the skills labs assures maintenance and security of the equipment. Such investments show potential for sustainability by the schools and should be sustained and expanded as a model. The momentum created by this training component continued S 3 5 4 4 5 4.2 Recommendation here is that the momentum created by this training component should be maximized. It misses the how, and though it looks like a successful pilot, recommendation falls short of stating how scale up would look like. How many schools? Students? Regions? Any indication of effectiveness of these students when they return to their places of normal duty? What should it go with to be more effective? We must use these good results as an opportunity to revise/improve all the teaching programs of these schools in collaboration with the competent authorities. ..Yes! in my view, PRE service training is a priority and could more effective than in service (although refresher courses and continuous education are still needed of course). I also agree with the need to promote labs skills, with the caveat mentioned above that a mapping of labs services should be done, in order to make strategic decisions about locations and optimize staffing and equipment. R 3 5 5 4 4 4.2 P 4 5 3 or 4 4 5 4.6 F 3 4 5 5 4 4.2 Integrated Health Service Delivery in the Post-Ebola Context 127 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 3.5 continued should be maximized. There is a need to create laboratory skills in health schools in all administrative regions with the support of MoH (Findings refer to 3.4.2) Recommendation 3.6. Design for sustainability: USAID should improve monitoring of sustainability beyond design through using tools or strategies to ensure that sustainability is considered throughout project implementation despite changing circumstances or adaptation needs. Persistent fundamental issues such as stock-outs or equipment maintenance in a facility level project must be addressed or mitigated as part of the evolving context of the project interventions. Finally, indicators that measure increased ownership and sustainability could shift the focus from inputs to health system outcomes needed to sustain short term gains in other process and outcome areas. (Findings refer to 3.2.2 and 3.5.1) S 4 3 4 4 3.75 This is a great recommendation with moderate specificity which could be improved by recommending thresholds in sustainability indicators that would trigger changes in resource structure and levels. May also recommend to add flexibility guided by adaptive management at design and in award. The recommendation is multi-faceted. The design and structure of HSD make it difficult to implement with long term sustainability focus. USAID has to weigh both need for direct investment in improving health outcomes with building long term strategies for increased GOG leadership and ownership... Good recommendation in principle, but hard to deliver on. A further analysis of sustainability prospects [how will the Guinean health system look like 10 years from now ? ] in order to prepare a forward looking plan of action could have been added to the recommendation? Without government‘s commitment and meaningful expenditures for health, progress will remain dependent on international support. R 4 3 5 4 4 4 P 4 5 5 4 5 4.6 F 4 3 3 or 4 4 2 3.25 Recommendation 4.1. USAID Leveraging: Increased communication and more synergistic management of programming across RMNCH+ implementing groups during the Ebola outbreak was seen as an important lesson for moving towards greater collaboration and impact. Further collaboration and ongoing monitoring between development partners and programs is necessary for optimizing gains across projects. These efforts must be facilitated with other development partners and the Ministry of Health in capacity development activities to avoid duplication of activities and approaches, and ensure appropriate programmatic coverage across demonstrated needs (e.g., topical areas, geographies, supervision and management). (Findings refer to 6.1) S 2 5 5 4 4 4 Not clear how USAID and its IPs can provide leadership for coordination and synergistic collaboration across different actors. What is USAID’s and HSD’s leverage? Pertinent for future programming. Additionally, mention was made in the report that USAID was seen as having predominance in one technical intervention area, by other donors. For future leadership of the GOG for sustainability it is important for a “rebalancing”... Agree, without reservation, but this is hard work! However, putting staff time and resources behind it is paramount. Publications, reports and communications about how this was achieved during the Ebola crisis should be very helpful. Usually, coordination mechanisms are stronger and work much better during emergencies, than “normal “times. In fact, representing the current health status of the population in Guinea and the current response of the health system as a crisis is necessary in my view. R 4 5 5 4 4 4.4 P 4 5 5 5 5 4.8 F 3 5 5 5 3 4.2 128 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 4.2. Synergistic communication: Increased communication and more synergistic management of programming within USAID and across RMNCH+ implementing partners during the Ebola outbreak was seen as an important lesson for moving towards greater collaboration and impact. Institutionalizing cross-project synergies is possible through the USAID mission through better alignment of resources, timing, sectors, and actors. It is unclear how effective these monthly meetings are for the extensive management required, particularly where MoH ownership is low; ongoing monitoring between programs is necessary for optimizing gains across projects. (Findings refer to 6.3-6.5) S 1 2 2 or 5 4 4 2.75 Not clear what USAID is being asked to do and whether to do that through HSD or other mechanisms. While the finding points to the need for improvement, more discussion with actors (a learning event) may have helped to sharpen the recommendation. As written, the feasibility is low but could be re-constructed to increase feasibility.. Recommendations 4.1 and 4.2 are too similar, confusing... Beyond the existing, it is important to create a platform of technical and financial partners in all the intervention regions to promote consultation and harmonization of interventions in order to avoid duplication of activities to make interventions more efficient...This is a good recommendation, but it is repetitive. The topic overlaps with rec 4.1 R 4 3 5 4 4 4 P 5 4 5 5 4 4.6 F 3 4 5 5 3 4 Recommendation 4.3. Flexibility: Fixed programming boundaries across USAID supported activities limited the ability of projects to rapidly adapt when important systems components fell between the cracks. The success of USAID activities could be improved through encoding responsibility; and ensuring more regular meetings and shared tasks across partners. Lack of flexibility in project design, and responsibility by involved partners perpetuated this issue. Greater efforts should be made for USAID-supported projects to reinforce each other, adapt to the evolving context when service delivery gaps emerge, and work with health systems partners to mitigate the challenges, in the future. (Findings refer to 6.3) S 3 1 2 4 4 2.8 Looks like this recommendation is alluding to implementation of learning and adaptive management. Could it be added to 3.6? The recommendation is for general USAID action, not HSD specific. “USAID-supported projects to reinforce each other, adapt to the evolving context when service delivery gaps emerge, and work with health systems partners to mitigate the challenges, in the future” Implementation will need buy in, coordination and monitoring that may be slow to achieve... A muddled formulation of the recommendation. It is outside of the manageable interest of HSD (and other projects) to determine where modifications can be made in design or execution o avoid things falling through the cracks other than strong advocacy with USAID and the MOH.... This is an interesting recommendation, which pertains to USAID’s internal working processes. In the absence of a strong Government ‘s capacity to lead and coordinate, more flexibility in program design, implementation and evaluation will be needed, but it is up to USAID to make recommendations about this issue. R 5 3 5 4 4 4.2 P 4 4 5 4 4 4.2 F 3 2 5 4 3 3.4 Integrated Health Service Delivery in the Post-Ebola Context 129 Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 4.4. Problem solving and course correction: Institutionalization of mitigation measures for improving adaptive response across USAID activities is needed as the lack of a clear contingency plan limits achievements of all projects. Problem solving strategies to be developed and implemented include inclusion of annual course correction activities including assignment of responsibility, and specification within RFP, were suggested ways to overcome these challenges, among others. (Findings refer to 6.3) S 4 4 4 3 4 3.8 This too belongs to the learning and adaptive management category. I would lean towards recommending a CLA task group for the Health Office (including Health Office, Program Office, OAA, OFM) to develop CLA templates for use in solicitations. Consider recommending course corrections more frequently than annual – may be quarterly – after every performance review meeting with stakeholders...First part of the recommendation is very pertinent. Do not see how the second part, “specification within RFP” can be done...This is important as well. Analysis of failures as well as co-design of course correction measures surely help achieving better results, even if it takes more time than expected. R 5 4 5 3 4 4.2 P 4 5 5 3 5 4.4 F 4 2 5 3 3 3.4 Recommendation 4.5. Innovation: USAID has supported MNCH Activities for decades in Guinea with some success, however too little government level ownership has yet to sustain achievements without intensive donor support. New approaches that have greater implication of government in activity development are needed. New ways of working together and a new vision may result in new partnerships, and may help to influence other partners to better support MoH, including setting clear and measurable governance indicators and milestones in USAID activities. (Findings refer to 3.5.1) S 3 5 5 4 2 3.8 Bad governance takes a long time to change. Effectiveness of USAID TA approach is not uniform and Guinea may be one of the harder places. But there’s always some place somewhere that results can be achieved. I believe the recommendation should have found some success case which USAID could scale up... Of course, but more specificity on these “new ways” would have been expected. Only at the end of the recommendation do we get an indication about how to go about this. R 4 5 5 4 4 4.4 P 5 5 5 4 4 4.6 F 2 4 5 4 3 3.6 Recommendation 4.6 Design: HSD (and past activities) have focused on service delivery improvements and quality of care with each new activity building on some components of the preceding activity. Sustainability, as observed by Guinean health service staff, has essentially become guaranteed through this generational project approach. HSD was not designed with a plan to devolve to the MoH. To move Guinea forward on the “journey to self-reliance”, activities continued S 4 5 3 4 2 3.6 Is this recommendation similar to that under “innovation”? I have similar comments, except this is more important and higher priority...Very to the point... Sustainability of actions is a major challenge.. Do we know whether self- reliance is possible now, or by when it might become a reality? Designing for something not achievable within the lifetime of a project only leads to frustrating results. Small, doable steps are possible for sure, and recommendations could be more precise in mentioning what they are. The language here is vague in that respect. R 4 5 5 4 4 4.4 P 5 5 5 4 4 4.6 F 3 5 5 4 2 3.8 130 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Recommendations Scores (Specificity, Responsiveness, Priority, Feasibility) Comments Recommendation 4.6. continued must be designed for sustainability through government ownership, commitment and leadership. Within project design, theories of change should be explicit about how the project will build ownership and leadership over time. (Findings refer to 3.5) Recommendation 4.7. Breadth or depth of investment: HSD targeted seven regions in the country with the aim of improving the quality of integrated RMNCH+ care in selected facilities. The breadth of the intervention package required continuous training, monitoring, data collection and reporting. Continuous training efforts were needed to bring the full package of interventions to all the facilities and staff given staff turnover and retention issues. Given the quality of RMNCH+ delivery and care, the state of the health of the population, the financial resources available to the project, the timeline of resource availability, and the known contextual challenges, HSD activities emphasized breadth (reaching as many regions and facilities as possible) rather than depth (concentrating on proof of concept in a subset of regions and facilities). HSD and future activities could consider an alternative approach of providing the full package of integrated activities within a more limited geography. This would facilitate the provision of more specific attention and mentoring to a subset of facilities to demonstrate how success can be achieved as envisioned by the project. (Findings refer to 6.3) S 4 5 5 4 4.5 Should have added illustrative analysis showing exactly where effort can be focused. May be 20% of facilities are reporting 80% of the RMNCH+ results, or some reporting most challenges on one component (eg malnutrition) and not the others... The scope of the project was set by USAID and HSD implemented it as such. From a humanitarian and impact perspective it is not sure whether it is politically acceptable to focus on proof of concept and have lesser impact. There are other countries’ experiences that can be used for future design with more explicit sustainability benchmarks, and recognizing the week overall health system... The recommendation falls short in analyzing why depth should be chosen over breadth, or in addition to it. It only mentions what solutions targeted implementation will bring, but not how these activities could be scaled up in the absence of the necessary underpinning conditions, such as government commitment? R 5 3 5 4 4.25 P 5 3 5 5 4.5 F 5 4 5 5 4.75 Integrated Health Service Delivery in the Post-Ebola Context 131 ANNEX VI: DISCLOSURE OF ANY CONFLICTS OF INTEREST The evaluation team reports no conflicts of interest. 132 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity ANNEX VII: DATA TABLES Activity & IP (prime) Description Geographical Coverage Dates Budget (USD) Stop Palu RTI International To assist the Government of Guinea (GOG) to reduce malaria-associated mortality through improving malaria prevention in support of the National Malaria Strategy, improving diagnostic testing and malaria treatment capacity, enhancing the National Malaria Control Program’s technical capacity to plan, design, manage, and coordinate a comprehensive malaria control program. 14 prefectures,a and 5 communes of Conakryb May 13 – Dec 17 $21,596,539 Stop Palu RTI International To assist GOG efforts to bring the country to pre￾elimination of malaria by the end of 2022 by reducing malaria-related morbidity and mortality by 75% through increasing use of insecticide treated nets, intermittent preventive treatment for pregnant women; increased prompt care seeking and treatment and delivery of an increased full dose of seasonal malaria chemoprevention. 14 prefectures,a and 5 communes of Conakryb Dec 17 – Dec 22 $27,815,368 Maternal and Child Survival Program (MCSP) – Service Delivery Jhpiego Support healthcare workers and facilities to offer high quality health services in a safe environment by strengthening infection prevention and control practices through training, supportive supervision, and complementary monitoring and evaluation. Prefecturesc in Conakry (5), Kindia (6), Boke (2), N’Zérékoré (6), Faranah (1). Jun 15 – May 16 $2,400,000 Maternal and Child Survival Program (MCSP)— Health Systems Strengthening Jhpiego Support updating of national level policies and procedures for IPC and its integration into RMNCH; support national, regional and prefectural engagement in the quality improvement process; ensure engagement with national health management information system initiatives to strengthen data collection and analysis; support improved national immunization program coordination; support MOH to complete the OIC-supported health systems bottleneck analysis for maternal and newborn health Conakry, Boké, N’Zérékoré, Kissidougou, Kindia Apr 16 – Dec 17 $2,750,000 Table 12. USAID-funded Projects in Guinea during HSD Activity (2015 - 2020) a Coyah, Forécariah, Kindia, Dubreka, Fria, Boffa, Boké, Gaoual, Koundara, Lélouma, Koubia, Labé, Tougué, Mali; b Ratoma, Matoto, Dixinn, Matam, Kaloum; c Conakry: Ratoma, Matoto, Dixinn, Matam Kaloum, Kindia: Coyah, Forécariah, Kindia, Télimélé, Dubréka, Fria, Boké: Boffa, Boké, N’Zérékoré: Lola, Yomou, Macenta, Guéckédou, N’Zérékoré, Beyla, Faranah: Kissidougou; Integrated Health Service Delivery in the Post-Ebola Context 133 Activity & IP (prime) Description Geographical Coverage Dates Budget (USD) System for Improved Access to Pharmaceuticals and Services (SIAPS) Management Sciences for Health Strengthen pharmaceutical sector governance, increase/enhance capacity for pharmaceutical supply management and services, ensure pharmaceutical management information available and used for decision making, strengthen financing strategies and mechanisms to improve access to medicines strengthened, improve pharmaceutical services to achieve desired health outcomes National Oct 11 – Sep 17 $10,304,800 Maternal and Child Survival Program (MCSP)— Health Systems Strengthening Jhpiego Support updating of national level policies and procedures for IPC and its integration into RMNCH; support national, regional and prefectural engagement in the QI process; ensure engagement with national HMIS initiatives to strengthen data collection and analysis; support improved coordination of the NIP; support MOH to complete OIC supported health systems bottleneck analysis for MNH Conakry, Boké, N’Zérékoré, Kissidougou, Kindia Apr 16 – Dec 17 $2,750,000 Health Service Delivery Jhpiego To ensure that a package of essential and comprehensive care for FP and MNCH is consistently provided with high quality in health facilities in Guinea. Conakry, Kindia, Boké, Mamou, Labé, Kankan, Faranah Dec 15 – Dec 20 $28,800,000 MEASURE DHS ICF International Availability of reliable health and demographic data National 2018 – 2023 $1,400,000 Strengthening Data Availability and Use in Guinea John Snow Inc. Strengthen the HMIS to increase availability and quality of health service delivery data and institutionalize a culture of data driven decision making National; Pilot in Conakry, roll out in Labé Oct 15 – Dec 17 $2,500,000 Promoting the Quality of Medicines (PQM/USP) United States Pharmacopeial Convention, Inc. To improve quality assurance and quality control of pharmaceuticals and systems through continuing to build the technical capacity of the (LNCQM); strengthen DNPL functions by improving the existing registration and inspection processes National Jan 09 – Sep 19 $350,000 Leadership, Management and Governance National Malaria Control Program Capacity Building Project (LMG/ NMCP) Management Sciences for Health Effective management of human, financial and material resources, develop and direct policy and norms for the implementation and surveillance of the national malaria control strategy. Mobilize stakeholders to participate in national malaria control coordination and implementation efforts. Nationwide Sep 13 – Sep 17 $1,624,272 Table 12. USAID-funded Projects in Guinea during HSD Activity (2015 - 2020) continued 134 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Activity & IP (prime) Description Geographical Coverage Dates Budget (USD) APC Ebola Transmission Prevention and Survivor Services John Snow Inc. Reduce or eliminate Ebola transmission from survivors to others and to ensure access to and effective delivery of health care and psychosocial services for survivors Conakry￾N’Zérékoré￾Kindia Jun 16 – Jun 18 $6,000,000 HRH2030: Human Resources for Health in 2030 – Capacity Building for Malaria Chemonics Strengthen the institutional and managerial capacities of the NMCP to improve country performance on grants from GFATM through changes in policies or guidelines, improvement in M&E systems, or reduced stockouts. National 2017 – 2020 $2,400,000 ENVISION Hellen Keller International Provide assistance to the national NTD program for the control and elimination of seven targeted NTDs: lymphatic filariasis, trachoma, onchocerciasis, schistosomiasis, and three soil-transmitted helminths. ENVISION contributes to the global goal of reducing the burden of targeted NTDs. See listd 2012 – 2019 $14,000,000 USAID Global Health Supply Chain Program – Procurement and Supply Management Chemonics To ensure uninterrupted supplies of health commodities in support of USG-funded public health initiatives through the provision of health commodity procurement services and related systems-strengthening technical assistance; to provide effective and environmentally benign treatment and disposal of UMPP at Guinean warehousing facilities and supports the design of a long-term strategy to improve the country’s ability to properly manage hazardous waste Nationwide 2016 – 2023 $49,000,000 Breakthrough Action Johns Hopkins Breakthrough-ACTION will build upon USAID investment in SBCC, including both global and bilateral projects, to simultaneously guide new learning and drive broader application of proven practices and tools in SBCC to promote health behavior change in Guinea Forecariah, Boké, Kindia, Mamou, N’Zérékoré, Macenta, Kissidougou, Faranah. 09/18 – 09/20 $2.5 million for FP, MCH and PMI and $1.2 million for GHSA Emergency Pandemic Threat (EPT) FAO To support efforts to combat emerging and or re￾emerging high impact infectious diseases. It will also address relations between livestock and livelihoods in terms of reducing negative impacts of nutrition security. It also aims at building and pandemics on sustainable food strengthening basic capacity in animal health to detect early and respond rapidly to emerging disease crises threatening animal and human health. Conakry, Kankan, Kissidogou, Kouroussa, Forécariah, Kindia, Mamou, N'Zérékoré 2015 – 2023 $4,095,000 d Koundara, Gaoual, Boké, Boffa, Fria, Télimélé, Lélouma, Kindia, Forécariah, Dinguiraye, Dabola, Siguiri, Kouroussa, Faranah, Mandiana, Kankan, Kérouané, Kissidougou, Gueckédou, Macenta, Beyla Table 12. USAID-funded Projects in Guinea during HSD Activity (2015 - 2020) continued Integrated Health Service Delivery in the Post-Ebola Context 135 Activity & IP (prime) Description Geographical Coverage Dates Budget (USD) Civil Society Epidemic and Pandemic Preparedness International Federation of the Red Cross Strengthen communities’ One Health preparedness and response for epidemics and pandemics and continue to establish trust between government and civil society and train community workers for One Health community surveillance and response so that they can rapidly respond to both animal and human diseases and outbreaks. Faranah. Gueckédou 06/18 – 09/22 $2,000,000 Infectious Disease Detection & Surveillance (IDDS) Strengthening animal health laboratory systems by expanding technical proficiency for diagnosing priority zoonotic infectious diseases, specimen transport, facilitating a One Health network and increasing linkage to surveillance. National Sep 18 – Sep 23 $3,000,000 PREDICT-2 ICF International Improve characterization of biological and behavioral risks to better understand which locations, “epidemiological zones, or interfaces are most associated with spillover, amplification, and spread of zoonotic viruses with pandemic potential. Conakry, Forécariah, Boké, Kindia, Mamou, Macenta, N'Zérékoré 2016 – 2019 $6,000,000 Comprehensive Health Systems Strengthening Activity for Guinea Abt Associates Strengthen Guinea's health system to enhance functionality of institutions and programs and improve their capacity to deliver better services to the population of Guinea through a) improving institutional capacity and health governance across the health system; b) enable health financing; c) improve institutional capacity of Ministry of Health to manage effectively human resources National Jun 16 – Jun 18 $7,000,000 Citizens Involvement in Health Governance FHI 360 Improve citizens’ understanding of and participation in Guinea’s health system reforms and build public trust in Guinea’s health sector and governance through expanding opportunities for elected and appointed officials and citizens to engage in constructive dialogue and improving the quality of public discussion on health issues. Nationwide Jun 17 – Jun 20 $12,170,000 Table 12. USAID-funded Projects in Guinea during HSD Activity (2015 - 2020) continued 136 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Table 13. Observed Characteristics of Selected HSD-Participating Facilities, by Facility Type Facility Type CS/CMC (n=25) n (%) Hospital (n=10) n (%) Total (n=35) n (%) Access characteristics Clients rights posted on wall 14 (56.0) 8 (80.0) 22 (62.9) All services offered everyday 21 (84.0) 10 (100.0) 31 (88.6) At least one doctor available everyday 17 (68.0) 10 (100.0) 27 (77.1) No User Fees for RMNCH 17 (68.0) 5 (50.0) 22 (62.9) Do not require spouse consent for FP 19 (76.0) 9 (90.0) 28 (80.0) Environmental characteristics Toilets functioning, in good state 19 (76.0) 8 (80.0) 27 (77.1) Electricity 25 (100.0) 10 (100.0) 35 (100.0) Running water 14 (56.0) 8 (80.0) 22 (62.9) Waiting room has chairs/benches 20 (80.0) 7 (70.0) 27 (77.1) Educational and data materials IEC materials on wall 21 (84.0) 10 (100.0) 31 (88.6) Facility data on the wall 16 (64.0) 7 (70.0) 23 (65.7) Laboratory Lab functioning with necessary supplies 15 (60.0) 10 (100.0) 25 (71.4) Lab technician available 17 (68.0) 10 (100.0) 27 (77.1) Lab samples sent elsewhere for processing 19 (76.0) 8 (80.0) 27 (77.1) Average lab processing time (n=21) (n=8) (n=29) Within 24 hrs 17 (81.0) 7 (87.5) 24 (82.8) Over 24 hrs 4 (19.1) 1 (12.5) 5 (17.2) Family Planning FP exam room clean and welcoming 13 (52.0) 6 (60.0) 19 (54.3) IEC materials for FP available 25 (100.0) 10 (100.0) 35 (100.0) FP Job Aids used for counseling 23 (92.0) 10 (100.0) 33 (94.3) FP services and counseling offered 25 (100.0) 10 (100.0) 35 (100.0) FP counselor/provider designated and available 21 (84.0) 10 (100.0) 31 (88.6) Private and confidential space for FP 12 (48.0) 9 (90.0) 21 (60.0) 7 Modern FP Methods Available Today 19 (76.0) 8 (80.0) 27 (77.1) Trained provider present who can insert IUDs 21 (84.0) 10 (100.0) 31 (88.6) ANC Services ANC/PNC services offered today 24 (96.0) 7 (70.0) 31 (88.6) Clean and welcoming room for ANC/PNC 20 (80.0) 9 (90.0) 29 (82.9) Nurse/Midwives designated for ANC/PNC 22 (88.0) 5 (50.0) 27 (77.1) Integrated Health Service Delivery in the Post-Ebola Context 137 Facility Type CS/CMC (n=25) n (%) Hospital (n=10) n (%) Total (n=35) n (%) Private Confidential Place for ANC/PNC 17 (68.0) 6 (60.0) 23 (65.7) Job Aids used for ANC/PNC 25 (100.0) 10 (100.0) 35 (100.0) BF support group for lactating mothers 17 (68.0) 7 (70.0) 24 (68.6) Referrals Referral Register in Place 22 (88.0) 7 (70.0) 29 (82.9) Referral Register Filled out/used 22 (100.0) 7 (100.0) 29 (100.0) ANC Services FA supplements available today 23 (92.0) 7 (70.0) 30 (85.7) Anemia screening available today 20 (80.0) 8 (80.0) 28 (80.0) Iron supplements available today 23 (92.0) 8 (80.0) 31 (88.6) Glucose screening available today 13 (52.0) 7 (70.0) 20 (57.1) Availability and Functionality of ANC Supplies Pelvic Exam Lamp 5 (20.0) 7 (70.0) 12 (34.3) Functioning 4 (80.0) 6 (85.7) 10 (83.3) Exam Table/Bed 25 (100.0) 10 (100.0) 35 (100.0) Gloves 25 (100.0) 10 (100.0) 35 (100.0) Sharp Object Container 23 (92.0) 10 (100.0) 33 (94.3) Functioning 23 (100.0) 10 (100.0) 33 (100.0) Ultrasound 13 (52.0) 8 (80.0) 21 (60.0) Functioning 9 (69.2) 8 (100.0) 17 (81.0) General supplies Vaccine Refrigerator 23 (92.0) 6 (60.0) 29 (82.9) Functioning 23 (100.0) 6 (100.0) 29 (100.0) IUD kit available 22 (88.0) 10 (100.0) 32 (91.4) Functioning 32 (100.0) 10 (100.0) 32 (100.0) At least 5 syringes available 21 (91.3) 9 (90.0) 31 (88.6) Functioning 22 (100.0) 9 (100.0) 31 (100.0) Disinfectant solution 17 (73.9) 10 (100.0) 29 (82.9) Functioning 18 (94.7) 10 (100.0) 28 (96.6) Blood Pressure Apparatus 21 (91.3) 10 (100.0) 33 (94.3) Functioning 23 (100.0) 9 (90.0) 32 (97.0) Stethoscope Available 23 (100.0) 10 (100.0) 35 (100.0) Functioning 24 (96) 10 (100.0) 34 (97.1) Thermometer 17 (73.9) 10 (100.0) 29 (82.9) Functioning 17 (100.0) 10 (100.0) 29 (100.0) Table 13. Observed Characteristics of Selected HSD-Participating Facilities, by Facility Type continued 138 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Facility Type CS/CMC (n=25) n (%) Hospital (n=10) n (%) Total (n=35) n (%) Adult scale 23 (100.0) 9 (90.0) 34 (97.1) Functioning 22 (88.0) 8 (88.9) 30 (88.2) Newborn scale 19 (82.6) 10 (100.0) 31 (88.6) Functioning 20 (95.2) 10 (100.0) 30 (96.8) Vaginal speculum 21 (91.3) 10 (100.0) 33 (94.3) Functioning 22 (95.7) 10 (100.0) 32 (97.0) Obstetric surgery supplies Instrument trolley 16 (64.0) 9 (90.0) 25 (71.4) Functioning 15 (93.8) 9 (100.0) 24 (96.0) Autoclave 15 (60.0) 10 (100.0) 25 (71.4) Functioning 11 (73.3) 10 (100.0) 21 (84.0) Surgery clothing 17 (68.0) 10 (100.0) 27 (77.1) Functioning 16 (94.1) 9 (90.0) 25 (92.6) Champs 7 (28.0) 5 (50.0) 12 (34.3) Functioning 11 (91.7) 10 (100.0) 21 (95.5) Surgical pads 24 (96.0) 10 (100.0) 34 (97.1) Functioning 6 (85.7) 5 (100.0) 11 (91.7) Brackets 24 (96.0) 10 (100.0) 34 (97.1) Functioning 22 (91.7) 10 (100.0) 32 (94.1) Arm support for venipuncture 6 (24.0) 9 (90.0) 15 (42.9) Functioning 6 (100.0) 9 (100.0) 15 (100.0) Table 13. Observed Characteristics of Selected HSD-Participating Facilities, by Facility Type continued Integrated Health Service Delivery in the Post-Ebola Context 139 Table 14. Characteristics of Provider Survey Respondents at Selected HSD-Targeted Facilities Number Percent Region Boké 20 19.2 Conakry 19 18.3 Faranah 10 9.6 Kankan 23 22.1 Kindia 12 11.5 Labé 10 9.6 Mamou 10 9.6 Facility Type Health Center (CS)a 51 49 CMCa 7 6.7 Prefectural Hospitalb 25 24 Regional Hospitalb 15 14.4 National Hospitalb 6 5.8 Provider Type Doctor 15 14.4 Midwife 47 45.2 Nurse 20 19.2 Tech Agent 18 17.3 Other 4 3.8 Time working in facility Over 2 years 83 79.8 12-24 months 9 8.7 6 – 12 months 10 9.6 Less than 6 months 2 1.9 a Combined in subsequent tables b Combined in subsequent tables. 140 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Table 15. Perspectives and Practices of Selected Providers, by Facility Level Health Center/CMC (n=58) n (%) Hospital (n=46) n (%) Total (n=104) n (%) HSD Participation and Perspectives on Impact Provider participated in HSD activities Yes 34 (58.6) 35 (76.1) 69 (66.4) Partially 3 (5.2) 1 (2.2) 4 (3.9) No 21 (36.2) 10 (21.7) 31 (29.8) HSD activities participated in Training 30 (51.7) 33 (71.7) 63 (60.6) Supervision 2 (3.5) 6 (13.1) 8 (7.7) SBMR 8 (13.8) 9 (19.6) 17 (16.4) GBV 8 (13.8) 11 (23.9) 19 (18.3) CHW/RECO 5 (8.6) 1 (2.2) 6 (5.8) Other 1 (1.7) 0 (0.0) 1 (1.0) Strongly Agree that HSD has Impacted Management of RMNCH services, approaches, organization 53 (91.4) 45 (97.8) 98 (94.2) Workflow, provider time management 53 (91.4) 45 (97.8) 98 (94.2) Community Understanding of RMNCH 51 (87.9) 41 (89.1) 92 (88.5) Documentation for IEC 53 (91.4) 45 (97.8) 98 (94.2) Provider skills for responding to client need 50 (86.2) 42 (91.3) 92 (88.5) Family Planning Counseling always available 96 (92.3) Counselor Nurse 18 (31.6) 12 (26.1) 30 (29.1) Midwife 45 (79.0) 44 (95.7) 89 (86.4) Doctor 12 (21.1) 24 (52.2) 36 (35) Other provider 17 (29.8) 6 (13.0) 23 (22.3) IEC/Job Aids for FP In place 54 (93.1) 41 (89.1) 95 (91.4) Always used 49 (90.7) 36 (87.8) 85 (89.5) Contraceptive methods available Oral contraceptive pill, combined 52 (89.7) Oral contraceptive pill, progestin 38 (65.5) Injectable 52 (89.7) Implant 57 (98.3) IUD 47 (81.0) Sterilization 6 (10.3) Integrated Health Service Delivery in the Post-Ebola Context 141 Health Center/CMC (n=58) n (%) Hospital (n=46) n (%) Total (n=104) n (%) Condom 52 (89.7) LAM 33 (56.9) Other 4 (6.9) Always referred if preferred method not available 20 (34.5) Challenges in provision of family planning for providers Lack of awareness (false beliefs, religion) 31 (53.5) 27 (58.7) 58 (55.8) Lack of interest by women 9 (15.5) 9 (19.6) 18 (17.3) Need to consult family members 23 (39.7) 21 (45.7) 44 (42.3) Provider lacks time for counseling 1 (1.7) 1 (2.2) 2 (1.9) Client lacks time for counseling 4 (6.9) 7 (15.2) 11 (10.6) Lack of FP orientation materials 5 (8.6) 3 (6.5) 8 (7.7) Lack of space for counseling 13 (22.4) 14 (30.4) 27 (26) Not considered a priority for staff 2 (3.5) 0 (0.0) 2 (1.9) Staff have not received training 5 (8.6) 1 (2.2) 6 (5.8) Inappropriate or unnecessary 1 (1.7) 0 (0.0) 1 (1.0) Other 3 (5.2) 6 (13.0) 9 (8.7) None 5 (12.8) 4 (12.9) 9 (8.7) Factors perceived to influence contraceptive adoption among women Cultural or religious barriers 44 (75.9) 31 (67.4) 75 (72.1) Stock outs 20 (34.5) 15 (32.6) 35 (33.7) Side effects 29 (50.0) 25 (54.4) 54 (51.9) Cost 2 (3.5) 2 (4.4) 4 (3.9) Trust in providers 3 (5.2) 5 (10.9) 8 (7.7) Lack awareness 15 (25.9) 11 (23.9) 26 (25.0) Lack privacy 6 (10.3) 5 (10.9) 11 (10.6) (un)Availability of providers 1 (1.7) 1 (2.2) 2 (1.9) Miscommunication 6 (10.3) 8 (17.40 14 (13.5) Other 5 (8.6) 4 (8.7) 9 (8.7) None 3 (7.7) 1 (3.2) 4 (5.7) Postpartum family planning counseling Always 29 (63.0) Sometimes 13 (28.3) Rarely 3 (6.5) Never 1 (2.2) Table 15. Perspectives and Practices of Selected Providers, by Facility Level continued 142 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Health Center/CMC (n=58) n (%) Hospital (n=46) n (%) Total (n=104) n (%) Antenatal Care Services conducted at any ANC visit Weight 51 (87.9) Risk assessment 29 (50.0) Anemia 32 (55.2) Folic acid supplementation 48 (82.8) Iron supplementation 45 (77.6) Nutritional counseling 45 (77.6) Family planning counseling 31 (53.5) Breastfeeding education 29 (50.0) Urinalysis 21 (36.2) Gestational diabetes assessment 1 (1.7) Delivery Care Always provide 10U oxytocin after delivery 43 (93.5) Always initiate skin-to-skin contact after delivery 42 (91.3) Mean % births in prior week where skin-to-skin contact was initiated 61.7 (36.8) Always evaluate postpartum breastfeeding 41 (89.1) Always use positive pressure ventilation for neonates with respiratory difficulty 33 (71.7) COSAH Presence of COSAH at facility Yes 48 (82.8) No 3 (5.2) Don’t know 7 (12.1) Role of COSAH (n=48) Develop and implement health promotion plans 23 (47.9) Promote communication between COSAH and providers 34 (70.8) Awareness raising and education activities for community members regarding RMNCH 42 (87.5) Support evaluation of services by community 24 (50.0) Other 5 (10.4) Table 15. Perspectives and Practices of Selected Providers, by Facility Level continued Integrated Health Service Delivery in the Post-Ebola Context 143 Health Center/CMC (n=58) n (%) Hospital (n=46) n (%) Total (n=104) n (%) Vulnerable populations and adolescents Accessibility to RMNCH among vulnerable populations Very accessible 44 (75.9) 38 (82.6) 82 (78.9) Somewhat accessible 9 (15.5) 4 (8.7) 13 (12.5) Somewhat inaccessible 2 (3.5) 1 (2.2) 3 (2.9) Very inaccessible 2 (3.5) 0 (0.0) 0 (0.0) Don’t know 1 (1.7) 3 (6.5) 4 (3.9) Barriers faced by vulnerable populations in accessing RMNCH care Transportation 34 (77.6) 35 (76.1) 80 (76.9) Actual or perceived discrimination 3 (5.2) 7 (15.2) 10 (9.6) Religious or cultural barriers 19 (31.0) 14 (30.4) 32 (30.8) Unaware of available services 10 (17.2) 10 (21.7) 20 (19.2) Language barriers 4 (6.9) 4 (8.7) 8 (7.7) Lack of provider time 1 (1.7) 1 (2.2) 2 (1.9) Cost/financial barriers 11 (19.0) 9 (19.6) 20 (19.2) Long wait time 10 (17.2) 10 (21.7) 20 (19.2) Misunderstanding of needed documentation 3 (5.2) 1 (2.2) 4 (3.9) Other 4 (6.9) 7 (15.2) 11 (11.6) Barriers faced by adolescents in accessing RMNCH care Hours of services 2 (3.5) 2 (4.4) 4 (3.9) Cost of services 3 (5.2) 2 (4.4) 5 (4.8) Financial obstacles 18 (31.0) 11 (23.9) 29 (27.9) Waiting time 10 (17.2) 9 (19.6) 19 (18.3) Decision-making power 14 (24.1) 13 (28.3) 27 (23.0) Authorization of spouse/father 27 (46.6) 20 (43.5) 47 (45.2) Uncomfortable with sex of provider 10 (17.2) 5 (10.9) 15 (14.4) Other 2 (3.5) 0 (0.0) 2 (1.9) Facility working to reduce gender-based obstacles to care Yes 43 (74.1) 31 (67.4) 74 (71.2) No 5 (8.6) 4 (8.7) 9 (8.7) Don’t know 8 (17.3) 11 (24.0) 21 (20.2) Notes: certain questions were asked only of health centers or hospitals. Where questions were not asked of particular respondents, they are shaded. Table 15. Perspectives and Practices of Selected Providers, by Facility Level continued 144 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Table 16. Indicators of Maternal Health Care, Guinea Demographic and Health Survey, 2018 Demand for family planning satisfied by modern methods (%) ANC at least 4 (%) Skilled attendant at birth (%) Institutional delivery (%) Cesarean delivery (%) Postnatal care of mothers within 2 days (%) ANC, Inst. Deliv. & PNC (%) Total 11.8 35.3 55.3 52.6 2.7 48.6 23.8 Residence Urban 17.0 51.4 92.0 84.1 5.8 68.1 40.6 Rural 8.6 28.4 40.7 40.0 1.4 40.7 18.0 Educational attainment no education 10.1 30.3 48.1 46.0 1.8 43.6 20.8 primary 10.6 41.5 68.6 62.0 3.8 54.6 29.1 secondary 16.5 57.0 88.0 83.7 5.6 69.1 46.7 higher 26.6 68.3 98.7 95.0 15.0 73.4 56.7 Household wealth poorest 4.8 18.8 26.5 25.1 0.6 30.4 10.4 poorer 8.9 26.5 40.4 40.6 1.1 43.6 16.1 middle 10.8 32.6 50.8 50.7 2.2 45.5 21.9 richer 15.4 48.2 82.1 75.6 4.5 63.4 38.2 richest 17.8 58.4 94.9 87.1 6.6 69.5 46.3 Age (maternal) 15-19 8.6 34.3 62.5 58.6 3.3 48.2 20.4 20-24 15.5 39.3 56.4 54.0 1.6 51.5 24.2 25-29 14.7 36.5 55.8 54.4 3.7 49.2 23.2 30-34 13.7 34.3 54.7 50.9 2.1 47.8 24.2 35-39 13.0 32.9 53.9 50.2 2.8 44.4 23.8 40-44 8.8 31.5 49.4 46.9 2.7 49.1 24.5 45-49 5.7 30.7 49.2 46.0 1.9 41.1 24.8 Integrated Health Service Delivery in the Post-Ebola Context 145 Demand for family planning satisfied by modern methods (%) ANC at least 4 (%) Skilled attendant at birth (%) Institutional delivery (%) Cesarean delivery (%) Postnatal care of mothers within 2 days (%) ANC, Inst. Deliv. & PNC (%) Region Boké 5.3 28.2 45.5 42.1 3.2 45.6 17.4 Conakry 17.8 56.5 96.0 89.8 5.9 72.1 44.6 Faranah 9.8 27.6 38.4 34.4 2.1 36.8 15.2 Kankan 18.2 36.1 56.2 51.6 1.5 49.6 25.3 Kindia 12.1 47.4 55.9 50.7 2.4 55.3 34.5 Labé 3.0 29.6 33.7 30.3 2.2 18.8 16.2 Mamou 2.7 32.0 41.8 37.5 1.8 41.5 17.9 N'zérékoré 15.0 21.5 64.3 71.5 2.7 61.4 16.5 Table 16. Indicators of Maternal Health Care, Guinea Demographic and Health Survey, 2018 continued 146 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Table 17. Indicators of Infant and Child Health Care, Guinea Demographic and Health Survey, 2018 Postnatal care of newborns within 2 days (%) Early initiation of breast- feeding (%) BCG vaccine, 12-23 mo (%) DPT1 vaccination 12- 23 mo (%) All basic vaccinations, 12-23 mo (%) Advice/tmt sought for diarrhea,a under 5y (%) Advice/tmt sought for ARI,a under 5y (%) Total 42.8 46.9 73.4 62.3 23.9 67.6 83.2 Residence Urban 65.7 56.5 87.7 78.6 30.5 78.4 96.6 Rural 33.7 42.6 67.3 55.3 21.0 63.1 78.3 Educational attainment no education 37.2 46.2 69.7 57.4 21.9 64.4 84.2 primary 50.0 42.9 77.5 65.6 21.8 68.6 74.9 secondary 64.1 52.6 87.0 81.4 29.9 85.1 84.4 higher 78.6 60.2 91.8 90.5 58.3 92.7 100.0 Household wealth poorest 22.9 36.2 56.0 44.6 15.2 47.8 74.0 poorer 34.1 43.2 70.8 58.7 21.8 63.7 72.9 middle 41.5 46.7 72.2 58.4 22.6 75.0 87.5 richer 57.7 53.9 83.6 74.6 26.4 75.9 89.1 richest 70.0 57.8 89.6 81.0 36.3 79.5 100.0 Age (maternal) 62.3 23.9 15-19 43.7 44.5 75.2 64.0 25.2 62.3 70.6 20-24 45.1 47.2 75.7 63.7 25.0 66.0 87.4 25-29 43.1 46.7 73.1 66.8 25.7 69.3 89.9 30-34 41.8 48.4 74.3 60.7 22.9 71.7 82.8 35-39 40.2 47.4 70.7 57.0 23.1 67.8 74.5 40-44 41.7 41.5 72.4 60.6 18.9 62.5 81.7 45-49 39.0 54.8 54.1 25.5 4.6 67.4 65.5 Integrated Health Service Delivery in the Post-Ebola Context 147 Postnatal care of newborns within 2 days (%) Early initiation of breast￾feeding (%) BCG vaccine, 12-23 mo (%) DPT1 vaccination 12- 23 mo (%) All basic vaccinations, 12-23 mo (%) Advice/tmt sought for diarrhea,a under 5y (%) Advice/tmt sought for ARI,a under 5y (%) Region Boké 42.1 39.5 61.6 51.8 16.9 64.4 44.5 Conakry 72.2 62.9 87.2 77.4 36.5 74.4 100.0 Faranah 29.4 33.4 72.9 62.8 20.6 46.4 36.5 Kankan 38.8 48.2 84.1 75.2 36.2 76.9 97.0 Kindia 50.3 51.5 75.8 62.8 11.7 76.7 85.2 Labé 16.3 27.6 54.6 41.3 8.4 40.2 100.0 Mamou 34.7 34.1 72.2 53.3 13.3 57.8 64.6 N'zérékoré 54.8 60.7 69.5 60.0 35.0 82.6 100.0 Table 17. Indicators of Infant and Child Health Care, Guinea Demographic and Health Survey, 2018 continued 148 Final Evaluation of the USAID Guinea Health Service Delivery (HSD) Activity Table 18. Problems Accessing Health Care among Women of Reproductive Age, Guinea Demographic and Health Survey, 2018 ≥1 Problem % Permission % Money % Distance % Accompaniment % Total 68.1 29.5 60.1 46.1 32.2 Residence Urban 52.9 19.9 46.0 25.3 20.4 Rural 77.3 35.3 68.6 58.7 39.3 Educational attainment No education 73.3 32.5 65.3 52.0 34.5 Primary 66.4 26.8 59.1 44.3 32.8 Secondary 54.8 22.4 44.9 28.6 26.3 Higher 35.1 13.2 30.8 17.6 12.8 Region Boké 68.2 32.0 61.1 54.5 31.9 Conakry 48.7 19.7 40.8 23.9 20.6 Faranah 79.8 29.0 62.1 53.4 32.3 Kankan 65.9 22.9 53.5 31.9 18.5 Kindia 76.4 35.1 70.5 58.9 37.5 Labé 72.5 30.4 67.9 50.3 38.9 Mamou 85.9 52.9 79.5 76.8 61.8 N'zérékoré 64.4 25.8 59.8 41.6 30.6 Household wealth Poorest 82.1 39.5 72.8 66.7 43.7 Poorer 77.8 33.6 69.0 59.5 38.8 Middle 75.7 34.8 68.4 53.2 36.9 Richer 61.1 23.2 55.1 31.6 22.9 Richest 47.8 18.7 39.2 24.3 21.0 Age 15-19 70.2 33.6 61.0 46.0 38.6 20-24 65.3 25.5 56.5 41.5 27.0 25-29 66.1 26.5 58.0 44.4 27.5 30-34 66.0 28.5 59.7 46.4 30.0 35-39 68.7 29.2 61.3 47.1 31.0 40-44 70.6 31.5 62.2 51.1 35.2 45-49 71.5 32.0 65.2 52.2 35.5 Integrated Health Service Delivery in the Post-Ebola Context 149 ANNEX VIII: SBM-R CASE STUDY PURPOSE As part of the final performance evaluation of the USAID Guinea Health Service Delivery (HSD) Activity, a case study of the Standards-based Management and Recognition (SBM-R) quality improvement approach was requested to explore in detail the extent to which it is leading to improvements in health service quality. Specifically, this case study aims to shed light on the underlying drivers for quality service improvements and the potential for replication, scale up, and sustainability of this approach. BACKGROUND In the aftermath of the 2014-2016 Ebola epidemic, Guinea continued to be challenged with unacceptably high maternal and child morbidity and mortality.1 The USAID Guinea HSD Activity implemented the SBM-R approach to advance health service quality improvements. By the end of 2019, HSD had rolled out SBM-R to 97 facilities in the regions of Boké, Conakry, Faranah, Kankan, Kindia, Labé, and Mamou. Developed and refined by Jhpiego, SBM-R is a proactive, practical management approach for improving the performance, availability, quality, and use of January 2021 USAID’s Health Evaluation and Applied Research Development Project (HEARD) is funded by United States Agency for International Development (USAID) under cooperative agreement AID-OAA-A-17-00002. The project team includes prime recipient University Research Co., LLC (URC) and subrecipient organizations. USAID’S HEALTH EVALUATION AND APPLIED RESEARCH DEVELOPMENT (HEARD) PROJECT THE STANDARDS-BASED MANAGEMENT AND RECOGNITION (SBM-R) APPROACH IN GUINEA: PROCESSES, OUTCOMES, AND POTENTIAL FOR REPLICATION A Realist Case Study services in health care facilities that has been supported by successive USAID-funded and Jhpiego-led health programs in Guinea since 2009 with the start of Maternal and Child Health Integrated Program (MCHIP) . SBM-R is a checklist-based quality assessment benchmarking system whereby staff, managers, and community partners target improvements in health service quality and the enabling environment in three primary domains: emergency obstetric and neonatal care (EmONC); family planning (FP); and infection prevention and control (IPC). In this approach, targeted health facilities implement and monitor performance standards in the three domains to improve their adherence to national clinical and managerial protocols in an integrated manner. In 2018, HSD attempted to expand the criteria to include elements of clinical governance. Facilities participating in the SBM-R process conduct self￾assessments of their service availability and performance standards on a regular basis. Once internal assessment results meet minimum criteria, a national validation committee led by the Ministry of Health (MoH) externally verifies the results. Facilities successfully meeting a minimum average score of 80% of the standards are accredited with a gold star in a public ceremony 1 Maternal and Child Health Integrated Program (2009 to 2014). Maternal Child Survival Project / Health Systems Strengthening (2016-2018); Health Services Delivery Activity (2015 to 2020). 2 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study signifying that the facility is providing quality services. Facilities can earn a second star by meeting 86% of the standards. Where facilities do not maintain performance, losing a star for poor performance is possible; however, external assessments are not conducted systematically, they are done upon facility request. Typically, SBM-R is accompanied by other interventions to improve providers’ performance and work environment, including infrastructure improvements, in-service training and supportive supervision. SBM-R and the HSD activity are complemented by a broader USAID Guinea governance activity (Citizens Involvement in Health Governance; FHI 360) that includes efforts to improve the availability, accessibility, quality and ultimately the use of health services and reduction of mortality. CASE STUDY METHODOLOGY The case study utilized both qualitative and quantitative data (primary and secondary) collected at the national, facility, and community levels for the HSD evaluation. The main evaluation purposively sampled ten hospitals and 26 health centers, ensuring that MOH Hospitals selected included five high and five low-performing SBM-R hospitals2 , with three in highly populous districts, two in less populous districts3 , and variability in timing of entry into the SBM-R program. The health centers in the sample were selected for variation in region, activity performance (SBM-R)4 , length of time participating in SBM-R, location (urban and rural), and service volume. Also included were facilities that have benefited from infrastructure improvements and receipt of equipment, and that had active community components associated with the facility. The SBM-R case study sample included six facilities: two hospitals, and four health centers (HCs) with variable characteristics (performance, volume, and timing of adoption) (see Table 1). We also sought to have some regional balance. To respond to USAID interest in a broader understanding of other factors around adoption/ non-adoption of quality improvement approaches in this context, we included two health centers and one hospital that were not using SBM-R. Non-SBM-R health centers were both selected for high volume and performance variability, using HMIS indicators. In all facilities included within the HSD evaluation, the evaluation team collected qualitative data through key informant interviews (KII) and focus group discussions Table 1: Demand for Health Care Services, Guinea Demographic and Health Survey (2012-2018) Facility SBM-R Performance Volume Entry into SBM-R program Health Centers Boffa HC Lower Lower Early Mandiana HC Lower (variable) Higher Early Wondy HC Higher Higher Early Tougue HC Higher Higher Later Lambanyi HC - Higher Not using SBM-R Kakossa HC - Higher Not using SBM-R Hospitals Labé Regional Hospital Higher (variable) Earlier Mamou Regional Hospital Medium Later Ignace Deen Hospital - Began SBM-R program early but dropped out 2 Cumulative scores from the hospitals’ most recent SBM-R assessment were used to determine performance, with scores of .87 or lower classified as lower-performing and scores of .88 or higher as higher-performing. 3 Facilities serving populous catchment areas were determined based on service volume - number of maternity services provided per month. 4 Cumulative scores from the health centers’ most recent SBM-R assessment were used to determine performance, with scores of .86 or lower classified as lower-performing and scores of .87 or higher as higher-performing. Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 3 (FGD), and quantitative data through observation checklist, provider survey, and evaluation of monitoring and service data. In the six SBM-R case study facilities, the team asked additional questions regarding the facility’s experience with the SBM-R process. Following data analysis, the team conducted additional phone interviews with the SBM-R case study facilities for further clarification of findings. The evaluation team reviewed relevant national and subnational documents. We included secondary data within our evaluation including quantitative program and facility-level data. HSD shared databases containing results of internal self-assessments and external validations for each SBM-R facility for the entire duration that the facility has been implementing SBM-R. The databases contained total scores for each of the three domains (EmONC, FP, IPC), and the composite score across the three domains. These data were analyzed across the 97 facilities and for the six case study facilities to demonstrate trends within each domain and overall across assessments. Case study questions The SBM-R case study investigated the following issues and questions: w Understanding the SBM-R process: How has SBM-R been implemented and how has it evolved over time? w Achievements and outcomes: What has SBM-R achieved and what were the outcomes (intended and unintended) of the SBM-R approach on service quality and use? w Uncovering the internal and external performance drivers: What are the key factors affecting the implementation and achievement of performance improvements through the SBM-R process? w Ownership, replicability, and sustainability: To what extent is the Guinean government demonstrating leadership and ownership in the SBM-R process? What other quality improvement approaches are being used by other development partners in Guinea? How can this process be standardized to be used by the government, USAID partners, and other development partners, especially where USAID partners operate in the same facilities? How can other USAID partners leverage these efforts? Analysis A realist evaluation approach informed the analysis of the case study to explore the SBM-R program.2 Specifically, we sought to answer what mechanisms within the approach generated program outcomes, and what features of the context affected the operation of those mechanisms. The context-mechanism-outcome configuration is the main structure for realist analysis in the discussion of the case study. Limitations of the analysis Facilities varied in the criteria that were included from internal/external assessment to assessment, meaning the scores used to compare facilities in this case study, were based on similar but not identical criteria. Facilities also varied in number of assessments conducted due to different timing of entry into the SBM-R program and differences in the frequency of assessment across facilities. As a result, some facilities have more data points for analysis, which gives a clearer understanding of performance and trajectory in those facilities compared to others with fewer data points available. In addition, the trend analysis depicts performance from first to most recent assessment, capturing assessments at different points in time (i.e., facilities’ first assessment dates ranged from 2009 to 2017), some of which took place during HSD and others prior to HSD’s start, making that part of the performance trend unrelated to the HSD intervention. SBM-R PERFORMANCE ACROSS ALL PARTICIPATING FACILITIES Engagement Of 272 HSD-supported facilities, 97 (36%) implement the SBM-R program, including 64 (66%) HCs (including urban and rural HCs and ‘improved’ HCs); six (6%) communal medical centers (CMCs); and 27 (28%) hospitals (including six regional, 21 district hospitals) (Figure 1). Of the 97 SBM-R facilities, 43 began implementing SBM-R prior to the start of HSD, under the predecessor projects, Maternal and Child Health Integrated Program (MCHIP) and Maternal and Child Survival Program (MCSP), with baseline assessments between 2009 and 2014. Under HSD, 54 additional facilities began implementing SBM-R, 4 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study with baseline assessments between 2016 and 2018; nearly meeting their overall HSD target of 98 facilities. We examined facilities’ level of engagement during the most recent two years of available data (from October 2017 through September 2019) using the number of SBM-R assessments facilities had done as proxy for engagement in the SBM-R approach. Facilities aimed for quarterly self￾assessments, so perfect adherence over two years would mean eight assessment scores over this period. Facilities ranged from zero to eight assessments (see Figure 2), with most – nearly 70 percent – achieving four or fewer. Nearly half of facilities (47%) had three or fewer assessments from 2017-2019 and were considered to have low engagement (46 total: 31 health centers and 15 hospitals), A higher proportion of facilities in Kankan, Boké, and Faranah had low engagement relative to the other regions (Table 2; range 53-76%). Low engagement was more common among facilities that initiated SBM-R between 2009-2014, with 70% of low-engagement facilities beginning SBM-R during that period. Several low engagement facilities (46%) had earned one or more stars, including: w Fifteen that earned their first or second star recently (between October 2017 and September 2019), suggesting that despite infrequent assessment during this period, these facilities were sufficiently engaged in the process to earn stars (Table 2; Column D); w Six that earned their first or second star between 2012 and 2015, suggesting that some facilities have earned stars but may not be continuing to engage sufficiently Figure 1: HSD-supported facilities implementing SBM-R by type Figure 2: Number of SBM-R assessements facilities completed (October 2017-September 2019) 22 20 18 16 14 12 10 8 6 4 2 0 0 1 2 3 4 5 6 7 8 Assessments Facilities              Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 5 in the SBM-R approach, bringing into question whether these facilities have maintained sufficient performance levels to merit maintaining the stars earned. Stars Of the 97 SBM-R facilities, 49 (51%) have earned one or more stars; 39 (40%) have earned one star and 10 (10%) have earned two stars. Across all SBM-R facilities, CMCs and hospitals have been more successful in earning one or more stars compared to health centers; 83% of participating CMCs and 67% of hospitals implementing SBM-R have earned one or more stars, compared to only 41% of health centers. Of the 54 SBM-R facilities that began in 2016 or later (i.e., within the HSD activity), 20 (37%) have earned one or more stars, with only one having earned two stars. Of the 43 facilities that began in 2014 or earlier, 29 (67%) have earned one or more stars (20 have earned one star; 9 have earned two stars), suggesting the duration of time exposed to the SBM-R intervention may play a role in success. Further, among the 10 facilities that have achieved two stars, three achieved their first star within the HSD activity (2016 or later), whereas seven received their first star prior to HSD, indicating HSD’s capacity to support facilities in further advancing their quality improvement processes many years after they had initially became involved (See Annex 1). Performance On average, facilities reached the 80% performance threshold by the third (hospitals) or fourth assessment (health centers) (Figures 3 and 4), though the amount of time between assessments varied across facilities. Table 2: SBM-R and low-engagement facilities by region Region A. # SBM-R facilities B. # low engagement facilities C. % of facilities in region: low engagement D. # low engagement facilities without recent star E. % of facilities in region with low engagement, without recent star Boké 14 9 64% 9 64% Conakry 10 3 30% 2 20% Faranah 15 8 53% 3 20% Kankan 17 13 76% 11 65% Kindia 15 4 27% 3 20% Labé 16 5 31% 1 6% Mamou 10 4 40% 2 20% Total 97 46 47% 35 36% For example, the number of years between the baseline assessment and the fourth assessment ranged from 0.67 to 6.83 years (median 1.45, interquartile range 1.00- 2.50). After reaching the 80% performance threshold, on average, facilities’ assessment scores tended to hover around 80% for several more assessment periods before eventually rising to a range of 85-95%. To examine ‘current’ performance, we analyzed facilities’ most recent assessment scores as of early December 2019, which took place between March 2017 and September 2019. Most facilities’ latest assessments (69 facilities) were conducted in the last two quarters of fiscal year 2019 (between April and September 2019; see Table 2). The average overall score from the most recent assessments was 84% (range 40-98%). Mean IPC scores were slightly lower than the average overall score and mean FP scores slightly higher than the average overall score. The mean most recent performance scores did not vary substantially between hospitals and HCs/CMCs, though the range of scores was narrower among hospitals (60-95%–35 percentage points) compared to HCs/CMCs (40-98%–nearly 60 percentage points). SBM-R Initiation by Region: Regional trends in SBM-R initiation suggest 3 categories: w Predominantly recent (2016 or later) initiation (under HSD): Facilities in Boké, Labé, and Mamou began implementing SBM-R relatively recently. All 14 SBM-R facilities in Boké, 14/16 in Labé, and 8/10 in Mamou logged their first assessment between late 2016 and February 2018; 6 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study Table 3: Mean (and range) facility scores on most recent SBM-R assessment All facilities Hospitals Health Centers, CMCs EmONC 84% (23%-99%) 86% (56%-98%) 83% (23%-99%) FP 88% (37%-100%) 90% (65%-100%) 87% (37%-100%) IPC 82% (27%-100%) 81% (50%-97%) 82% (27%-100%) Overall 84% (40%-98%) 86% (60-95%) 84% (40%-98%) Figure 3: Mean SBM-R Performance Scores Among Hospitals* (n=27) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 7 8 9 10 11 12 Assessment Number Figure 4: Mean SBM-R Performance Scores Among Health Centers* (n=70) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 1 2 3 4 5 6 7 8 9 10 11 12 Assessment Number EmONC FP IPC Overall EmONC FP IPC Overall Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 7 w Predominantly early (2013 or prior) initiation (under MCHIP/MCSP): Facilities in Faranah and Kankan have largely been implementing SBM-R for longer. Most SBM-R facilities in these regions (13/15 in Faranah and 13/17 in Kankan) logged their first assessment in 2013 or prior; and w Mixed initiation timing: Facilities in Conakry and Kindia are mixed in terms of start of SBM-R, with more than half (5/5 in Conakry and 7/15 in Kindia) initiating SBM-R in 2016 or later. SBM-R Performance by region: Faranah, Mamou, and Kindia had the highest proportion of facilities having earned one or two stars, with 80%, 70%, and 60% of facilities in those regions having met that achievement (see Table 4). Based on performance at last assessment (Figure 5), average performance of facilities in Mamou, Kankan, and Boké lagged in comparison to other regions. SBM-R PERFORMANCE ACROSS CASE STUDY FACILITIES Health centers in the case study sample included three facilities which had not yet achieved any stars as of December 2019 (Mandiana HC in Kankan, Boffa HC in Boké and Tougue HC in Labé ) and one facility that had earned one star (Wondy HC in Kindia). Hospitals in the case study sample (Labé Regional Hospital and Mamou Regional Hospital) had both earned one star. Table 4: Star-earning facilities by region # SBM-R facilities # with any stars (1 or 2) % of facilities with any stars (1 or 2) # with 2 stars % of facilities with 2 stars Boké 14 2 14% 0 0% Conakry 10 5 50% 2 20% Faranah 15 12 80% 3 20% Kankan 17 6 35% 1 6% Kindia 15 9 60% 1 7% Labé 16 8 50% 2 13% Mamou 10 7 70% 1 10% Total 97 49 51% 10 10% Figure 5: Performance Overall at Last Assessment 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Boké Conakry Faranah Kankan Kindia Labé Mamou 8 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study In these six facilities, we examined the key factors affecting success in achieving SBM-R performance criteria in the areas of engagement, context, performance, analysis and perspectives (of MoH managers, facility staff, COSAHs and or community members). The overall findings of this analysis are reported below. Detailed facility-by-facility findings are available upon request. Engagement The SBM-R process was designed to be a facility-led, quality improvement process. Jhpiego first implemented SBM-R in Guinea in 2009 under MCHIP and continued through the MCSP and HSD activities. Facilities that have participated (n=97) vary considerably in level of engagement, number of assessments, progress, and results. The 97 SBM-R facilities have been implementing the approach on average for four years (median 2.6 years, range six months to 9.8 years). Wide variation existed in the frequency of facility assessments, ranging from one assessment per quarter at the high end to 0.2 assessments per quarter, or less than one per year at the low end (Figure 6). A minority of facilities conducted the expected one assessment per quarter (17 of 97, 18%). Thirty-six of 97 facilities (37%) were in the mid-range and conducted 0.5-0.8 assessments per quarter, equivalent to two to three assessments per year. The highest proportion of facilities (45%) succeeded in conducting only 0.2-0.4 assessments per quarter – equivalent to between 1.5 and less than one assessment per year. Context The SBM-R approach under HSD has been implemented within the context of major structural challenges facing the health system, including chronic staff turnover, staff shortages, and inadequate infrastructure and equipment. The Ebola epidemic that was subsiding at the start of the HSD activity is also considered an external contextual factor influencing implementation of SBM-R during and immediately following the epidemic. Specifically, community mistrust in the health system that resulted from the epidemic complicated efforts to achieve community engagement generally, including in SBM-R efforts at facility level. In each of the six facilities examined within the case study, equipment was donated (e.g., examination tables, delivery tables, instrument trays, delivery kits, vacuum extractors, Dopplers, etc.). Only one of the six case study facilities received rehabilitation with HSD support (Labé Regional Hospital), and support of space management was minor in comparison to many other HSD-supported facilities which received improvements to water and electricity supply and other repairs to the facility structure. Process Core steps of the SBM-R process at the facility level include regular self-assessment, often by providers, senior managers from the facility manager to district and regional focal points, and COSAH representatives. These assessments yield a score, which, together with facility performance data, indicates whether the facility is ready to be evaluated by the national committee. When ready, the facility requests the external evaluation and validation. This process is not always linear, and self-assessment scores do not always align with other facility performance data (i.e. self-assessment scores may be high while evidence from other facility data indicates lower performance). When self-assessment of progress did not align with health system data (which may show less progress in terms of health outcomes), commitment and motivation Figure 6: Number of assessments per quarter among SBM-R facilities 45 40 35 30 25 20 15 10 5 0 Proportion of facilities High (0.9-1.1) Mid (0.5-0.8) Low (0.2-0.4) 17 36 44 Assessments per quarter Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 9 of staff and managers could be affected. For example, health facilities such as Tougue HC in Labé, which has been actively engaged in the SBM-R process, registering self-assessed progress above 80% since mid-2018, has not yet had their self-assessment of high performance validated, likely due to their efforts not yet being validated by service delivery data (which usually take time to reflect improvement due to better clinical performance, for example). There appear to be varying degrees to which the actors (providers, facility, district and regional managers, COSAH and community representatives) participate in self-assessments. COSAH members and community actor involvement tended to generate excitement from community members. As was the case in Wondy HC in Kindia and Boffa HC in Boké, community members became partners in the quality improvement process by helping with the maintenance and cleaning of facility grounds. Observation: Engaging communities as partners in quality improvement activities increases commitment and motivation of health staff and managers to achieve progress at the facilities as they feel valued by their clients. Observation: The national committee that validates facility performance and awards stars has not been sufficiently proactive and empowered to support the SBM-R process as facilities progress. Progress could be noted and made available as dashboards, with incremental steps to maintain engagement in the process. Sitting on the SBM-R Committee needs to be valued and validation visits need to take place at regular intervals. The current ad hoc convening for the committee does not valorize the process. Cost: Review of the case study facility responses indicate that financial cost associated with the SBM-R is not a significant factor, unless equipment or infrastrure improvements or purchases are needed. The process is staff and management-led and implemented. Costs for cleaning supplies (to implement IPC protocols) and assessment forms were negligible; however, broader service-related costs (related to equipment maintenance, and other supplies needed to deliver service to standard) may impede progress. Observation: Some facility managers noted that infrastructure and equipment costs cannot be underestimated if quality improvement is to be achieved. (Mandiana HC, Kankan) True quality improvement costs for training, equipment purchase or repair, and infrastructure investments (e.g. waste pits, latrines, renovation or expansion of the building, borehole for water etc.) are expensive and will likely continue to require private sector, government, and donor support. Observation: Cost should not, however, be considered requisite for quality improvement processes to be implemented on existing services and structures with existing resources. When the facility severely lacks the basic equipment and infrastructure to achieve basic quality criteria, motivation and commitment of staff to continue to participate in the process is likely. Performance criteria: The performance criteria assessed through SBM-R is extensive, possibly exaggeratedly so. Although they correspond to national and international guidelines, providers and facility managers suggested that performance criteria should be refined and reduced to focus on the most critical issues. Many criteria are interdependent, particularly those related to IPC, and as a result, if one factor is not achieved (e.g., lack of functioning autoclave), numerous other criteria cannot be met. As present, the list of criteria do not detail the minimum requirements needed in place at each facility. The emphasis on breadth and depth results in many questions. Positively, this provides multiple areas where small improvements can be made, which is motivating. Negatively, the sheer number of criteria can be overwhelming and demotivating. A balance needs to be struck. As it stands, the tool measures results, not progress. If a facility goes from 30% to 65% in one domain, they still fail to be sufficient (80% required) but they have made tremendous improvement. If the issues that impede further progress are out of their control (e.g., the incinerator is not working, infrastructure problems, etc.), motivation can wane. Observation: Each domain would be helped by creating a list of essential equipment and infrastructure by level needed to achieve the performance standards. This could focus pressure on the districts and regional MoH to ensure minimum requirements are in place within each facility. 10 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study Observation: To facilitate regular internal SBM-R assessment, a tablet-based checklist could be established to allow assessment of performance and essential equipment, supplies, consumables, and other resources. This would allow for an efficient response through skip pattern implementation to avoid interdependent questions and could support improved use of data for decision making through improved availability at the facility and at higher levels. Observation: Progress on indicators should also be valued by the process demonstrating effort and incremental achievement of criteria. Making progress visible might motivate staff and communities to engage more actively. This would require organization by subdomain such as equipment, infrastructure, client friendly, community engagement, clinical quality standards – some of which the staff can affect more readily than others. Such a process would also be supported by the switch to a tablet￾based platform. Performance (Achievements and Outcomes) Of the 272 facilities supported by HSD, 97 (36%) implemented the SBM-R program (66% of health facilities; 6% of CMCs, and 28% of hospitals), nearly meeting their overall phase target of 98 facilities; of those, 43 began under the prior MCHIP/MCSP programs, and 54 began under HSD. As noted earlier, of the 97 participating SBM-R facilities about half (49, 51%) have earned 1 or more stars , nearly meeting their HSD target of 98 facilities, 10 of which (10% of all SBM-R facilities) have earned 2 stars CMCs and hospitals being more successful than health centers (based on proportion earning 2 stars). Two-star facilities are a mix of facility types and are located across various regions, but have in common that all but one began implementing SBM-R early – between 2009 and 2013. While time of exposure to the process does appear to pay a role in star achievement, review of the case study facilities also indicates that motivation may wane over time. Motivation seems to reflect facility managerial leadership (and their capacity to motivate facility staff), community and district engagement in the process, and importantly, the mentoring they receive from HSD staff to make progress. For example, of the 14 facilities that earned 1 star between 2012-2015, only half of those went on to earn a second star, 4 of which were in 2019 – possibly indicating a late push by HSD (See Annex 1). Our review indicates that interest and commitment at initiation usually leads to short-term improvements and measurable progress at the first or second assessment, which may then be followed by a leveling or slight decline as the more challenging criteria are reckoned with. This was the case in Mandiana, Wondy and Boffa Health Centers, while Tougue HC rapidly improved over the first 2 years and then leveled off. Labé and Mamou Regional Hospitals also showed quick gains only to drastically decline after the first year (Labé) or level off (Mamou). In all cases, after the first couple of years, progress slowed and became incremental, with specific problems on individual criteria affecting overall ratings such as staff turnover or equipment malfunction. Notable is the high number of targeted facilities that are not fully engaged despite HSD efforts. While the evaluation did not explore non-participation, efforts to include control facilities in the sample in Conakry were unsuccessful because of lack of interest, time and commitment from facility managers and staff to answer our questions. In Conakry, it was suggested by managers/decision makers that it is not possible for a referral hospital such as Ignace Deen to participate in SBM-R given its considerable resource and patient burden. Observation: A constellation of factors contributes to quality improvement achievements more than simply the length of time the facility has been engaged in the process. Facilitation by HSD contributed to reinvigorating some facilities to engage in the SBM-R process that started during the previous programs. Clinical performance Clinical performance on IPC standards appear to be the most variable domain within SBM-R indicators as they are provider-centered and can thus be improved through training and supervision. These criteria effect the IPC standards within the EmONC and FP domains more than those related to sanitation and hygiene within the IPC domain. During the case study facility assessment period, marked improvements in self-assessment scores for their clinical practice were reported ranging from a 25 percentage point increase at Mamou Hospital to a 70 percentage point increase at Wondy HC. Boffa HC, Tougue HC, and Labé Hospital also saw rapid improvements, usually within 1 year of starting their SBM-R program. Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 11 Mamou Hospital had improvements in clinical practice but with inconsistencies in achievement of particular clinical standards across assessments (i.e. at one assessment, eclampsia treatment was inadequate; at another, it was postpartum hemorrhage treatment) in contrast to Labé Regional Hospital that showed steady but consistent improvement. Mandiana HC was the only facility where there was a rapid decline in performance standards from 80% to 51% between 2012 and 2016, prior to the start of HSD. Apparently, staff capacity to perform services to clinical standards was inconsistent as many had not been trained due to staff turnover. Improvements in clinical practice largely were related to obstetric interventions. Essential newborn care, respectful care (including client provider communication and sharing of information about their care or for the care of their newborn) and use of the partograph are all areas where more attention is needed in both health centers and hospitals. These findings align with other reviews of EmONC implementation in Africa.3,4 In a number of health centers, the HSD activity brought considerable training and focus on performance standards. These early trainings likely improved the provision of clinical maternal health care to protocol. Studies have shown that more complex newborn health interventions that are more difficult to perform and less frequent – such as newborn resuscitation – and care criteria that requires a shift in provider behavior and values – such as components of respectful care (e.g., provider-client interaction and informing women of their rights) – saw less progress and were slower to change.5 The marginal improvement in neonatal care may also have suffered from a tendency for health centers to refer families to hospitals to manage the sick newborns, and possibly fatalism6 to neonatal survival rates. Family planning performance standards generally improved across the case study facilities. Challenges in providing evidence-based information on specific FP methods such as the oral contraceptive, IUD, and implants were observed at health centers. Critical education on the need for dual method use, information on the fertility cycle, and method-related side effects was provided inconsistently. Boffa and Tougue improved performance markers, reaching the 80% threshold in 2018 and 2019 respectively. Performance issues in these facilities also are likely due to the need for continuous training as staff relocate or disengage if not motivated to perform all aspects of the clinical protocol. Observation: The SBM-R process demonstrates that clinical performance criteria can be improved through continuous training and supportive supervision at the facility level. Dramatic improvements were observed shortly after the start of the HSD activity. HSD has facilitated the process by preparing a team of trainers around the country. To maintain progress, regular investment in training on standards and refresher training will be needed in future and can be driven by the District and Regional Directorates of MoH. Observation: The SBM-R quality improvement process was more successful for the less- complicated interventions than those that require more skill (some neonatal care) or behavior change of providers. Provision of incorrect information on FP may be due to provider values or opinions rather than lack of knowledge. Managerial performance Managerial standards are those that control facility processes such as cleanliness, record keeping, job descriptions, posting on walls of information, and review of services indicators. Compared to clinical performance indicators, performance against managerial standards was the most variable as they took into consideration client and staff views of managerial processes. Responses ranging from client-provider interaction experience to leadership and availability of protocols differed significantly between and within the same facility over time likely reflecting change of staff and management. Comparability of the results is limited across data points and should rather be understood a simple indication of lack of continuity of management style and effectiveness over time. Managerial issues focused on improving the client experience fared least well with missed criteria throughout assessment periods for poor client provider communication, information giving and posting; availability of IEC materials, registers, patient files, client flow protocols, and job descriptions for staff; and lack of attention to waiting times in all case study facilities except Wondy HC (which provided too little information to assess). In Tougue HC particularly, and to a slightly lesser 12 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study degree in CSU Boké and Labé Hospital, missed criteria were achieved by 2019. Exploration of client satisfaction through comment boxes or forms was generally not pursued in any facilities except Wondy HC. Specific efforts were made to improve these indicators in Tougue HC, and Labé and Mamou Regional Hospitals. Management of IPC through regular monitoring and oversight of cleanliness was an issue in Mandiana, Boké, and Tougue Health Centers, and in both regional hospitals with basic issues such as hand washing before performing essential newborn care (Mamou Hospital) and cleaning staff not using gloves to manage medical waste (Labé Hospital) indicating gross infractions of protocol. Generally, IPC management was less strong in hospitals, particularly related to organization and standards in the morgues. Observation: When in place and functional, managerial standards (e.g. cleanliness, record keeping, job descriptions, posting information on walls, and review of service indicators) can motivate, and inspire improvements in clinical performance. Many are quite visible thus contribute to an overall perception of progress, motivating staff and community to engage and support quality improvement. Observation: The role of facility, district, and regional managers cannot be underestimated. SBM-R facilitated managerial leadership through training tools and protocols, which was effective in some districts and facilities. Given the role they play in the quality improvement process, more could be done to measure managerial performance and problem solving within the SBM-R assessment process. Observation: Facility in-charges and managers can advance quality improvement by prioritizing systemic solutions through guidelines, checklists, supervision, and control measures. Oversight is ad hoc in some facilities. Use of assessment data to make an action plan focused on specific issues is done inconsistently. Routinized behaviors should be emphasized. The SBM-R program should develop adapted tools that can facilitate management of IPC, including basic sanitation and hygiene measures in all service areas. The community is supposed to be part of this process and should be enlisted to participate in support of such measures. Equipment, supplies, and infrastructure Shortages of Equipment and supplies required for implementation of clinical or managerial standards are particularly difficult challenges to overcome as facility staff feel disempowered to affect change in these domains. To some extent, this is true; however, small improvements such as ensuring availability of disinfectants, protective wear, and basic supplies can contribute significantly towards improved IPC and EmONC criteria. Many of the case study facilities struggled in the initial years to put in place small equipment, such as scales, blood pressure gauges, resuscitation tables, trash receptacles, and in an extreme case, even tables for vaginal births. Mandiana had experienced the most difficulties during the MCSP project of the case study facilities as they had not yet benefited from the HSD￾related equipment contributions which were done between 2016-2017. Mamou Hospital also reported difficulties, particularly related to equipment and supplies needed for sterilization and hygiene throughout the assessment period. Inappropriate storage and ruptures in essential medicines, commodities and products such as oxytocin, magnesium sulfate, and blood also occurred in Mandiana and Labé Hospitals. HSD-supported facilities relied on family planning commodities facilitated by other program partners, including USAID’s Global Health Supply Chain-Procurement and Supply Management Project and UNFPA, however supply gaps remained a considerable challenge for many facilities. Facilities’ capacity to perform sterilization and waste management to standard was particularly problematic. Lack of disinfectants and antiseptics (Mandiana, Boké, and Labé Hospitals), gloves and protective wear (Mandiana, Wondy, and Labé Hospitals), basic equipment such as tables or dedicated space for sterilization and packaging of sterile equipment was often not available (Mandiana, Wondy, Labé, and Mamou Hospitals). These challenges are exacerbated if there is no functional autoclave or capacity to do high-level disinfection as was the case in Mandiana, Boké, and Mamou Hospitals. Lack of an incinerator was also recorded in Mandiana, Boké and Tougue Health Centers (though later resolved in Tougue HCe in 2018), and the use of open waste pits in CSU Boké and Mamou Hospitals. Even in facilities that have functional equipment and the capacity Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 13 to properly dispose of medical waste, it is often not done properly, as was the case in CSU Boké and Labé Regional Hospitals. In both Labé and Mamou Hospitals, the laundry machines were also not functioning properly, contributing to inadequate infection prevention and control standards, particularly in the hospital. Infrastructure issues played a significant role in terms of space to provide care, space to manage client flow, and receptiveness of the facilities. Specific challenges related to lack of an admissions area (Mandiana, Boké, Labé and Mamou Hospitals), a waiting area (Tougue, Labé Hospitals), a space for immediate post-partum (Boké), and abortion care (Tougue), a private space for family planning counseling and exams (Boké, Tougue and Labé Hospitals), and a space for staff to rest (Mandiana, Boké, Tougue, and Labé Hospitals). Essential amenities such as water and functional toilets were almost universally unavailable (except in Wondy). In addition, there was reported a lack of a functional climatized laboratory that further complicated performance of EmONC and FP as observed at Mandiana, Boké and Tougue Health Centers and Labé Hospital. Despite these challenges, most facilities (except Mandiana and Mamou Hospital) managed to overcome infrastructure limitations within a year or two of their engagement in the SBM-R process, as was the case in Boké in 2017, Labé Hospital in 2018 and Tougue by 2019. Although many of the challenges listed above seemed unsurmountable, leadership by the “in-charge”, hospital director or manager played a significant role. As noted above, achievement of performance criteria in these areas were in fact largely influenceable though managerial measures, systems, budget allocations, and supervision. Ensuring adequate staff, training, and materials and supplies can advance progress. Infrastructure issues, sanitation, and waste management were managed in some settings without major investment. Indeed, where MoH managers at facility, district and regional levels are supportive, quality improvements were achieved, as was seen in Kindia. In the regional hospitals where volume is high however, managerial and staff interventions around cleanliness beyond cosmetic solutions may not be sufficient and require higher levels of intervention with commitment and resources from the district, regional, or central level. Observation: Facility managers must be proactive in identifying equipment and infrastructural deficiencies that limit quality improvements. The SBM-R program needs to include stock taking exercises that go beyond the facility raising the issue and responsibility for quality improvement at district and regional levels for inputs that cannot be addressed locally. The SBM-R process needs to implicate the MoH at all levels to own the achievements and challenges that they must help to overcome. The needs and obstacles to achievement need to be made visible in the process. Observation: Critical facility improvements including water, sanitation (toilets and waste management) and sterilization need to be supported to provide the basis for quality improvements. Investment in these areas also helps to build staff commitment and momentum for further improvements. Analysis (Internal and external performance drivers) Key factors affecting the implementation and achievement of performance as evidenced by the case study facilities include inputs by the HSD activity such as training, equipment donations, tools and managerial processes; and facility level receptivity of SBM-R. HSD (and MCSP previously) inputs cannot be underestimated as they created the impetus to engage in quality improvement in many facilities and districts. Training and the necessary equipment needed for clinical practice to standard provides the basis for engagement and commitment. Training and equipment alone, however, are insufficient to drive quality improvement. Leadership by facility managers and supervisors, coupled with the engagement, commitment and appreciation of district and regional actors within the MoH in the SBM-R process is essential. While HSD facilitated this broader engagement, maintenance over time must be managed locally. Participation of other government officials and community leaders such as mayors, village chiefs, COSAHs, and communities stimulate facility staff to engage and improve. Sustaining interest and engagement by staff, however, requires continuous monitoring, self￾assessment, and action planning for targeted improvements of specific indicators. Involving the whole staff seems to be a positive factor in facilities that have shown improvements. Community appreciate and engagement further motivates staff. 14 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study While data to determine why some facilities regress was insufficient, indications from case study facilities suggest that changes in staff and or managers played an important role. Facility performance is highly dependent on close monitoring and follow up of the process. Peer reviews, COSAH meetings, and DPS monitoring are all important to maintain progress. Staff motivation is critical for success. This can be inspired by internal recognition of staff members who are leading the change process and contributing to quality improvements. Other key issues include consistency and retaining of staff in the process. Division of tasks also supports progress as the gaining (or maintaining of stars) becomes a team effort. Close follow up by project staff from MCHIP/MCSP and HSD also has been a factor for improvement, and conversely, when they are not available, for abandonment of the process. Unattended infrastructural deficiencies and high volume of clients can demotivate staff to engage, particularly when these shortcomings structurally impede quality improvements. In hospitals such as Ignace Deen, such conditions have limited engagement. Observation: Building teams to engage in quality improvement processes requires more than training on clinical and managerial skills so that they can work together effectively towards common aims. This continual process will need to be embedded in the organization’s culture. To change organizational culture in facilities and motivate those that are not currently engaged, new or adapted (non-monetary) incentives need to be developed with providers to ensure they are valued and appreciated by the staff on the frontline of quality improvement Perspectives on Ownership, Replicability and Sustainability Currently, facilities that have engaged in the SBM-R process show that engagement itself positively affects motivation. Positive competition is recognized as a sustainable way to improve quality without considerable cost to the system. Despite the positive perceptions of the process however, some facilities do not engage, likely because of the perception of additional work it entails from staff and managers, or as mentioned above, a sense of fatalism that improvements cannot be achieved given the state of the facility and skills of the staff. District and regional managers, and central MoH has not fully owned the process nor provided the leadership needed to maintain it. While the SBM-R process is appreciated by staff and communities in many locations, higher-level leadership from the MoH is needed to drive participation in the quality improvement process more broadly in Guinea. Currently SBM-R is in-place in about one-third of the HSD￾supported health facilities in Guinea. Other approaches (“Monitorage Amelioré”) to quality improvement currently being trialed in Guinea include results-based financing and improved monitoring approaches. The SBM-R process has now been adopted by the MoH, but it remains unclear how SBM-R will be aligned with the other QI approach of the MoH. To expand the program, full participation is needed across the health sector. Hospitals’ quality assurance departments need to also engage and support a common approach. USAID is organizing consultation meetings with other partners to involve them to support and facilitate adoption of the SBM-R process across health care thematic areas and levels. One manager noted: A star can bring respect and appreciation to the entire staff. It can increase skills of the staff and create a culture of self-assessment to differentiate what to do from what to avoid, improving behaviors (for instance, separating biological waste)” – Mandiana HC Staff “ It’s not even in hospitals alone that the SBM-R approach can be used; it’s in all departments, even in governance, because it’s a quality [improvement] approach.” – Labé Hospital manager “ Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 15 Observation: To fully engage communities and facility staff in the improvement cycle, support needs to be grounded locally. Government representatives from all sectors and levels need to engage in healthy competition and team up with the facility to make the improvements. Facilities, even with the support of the community, cannot go it alone. HSD and future USAID activities need to work more closely with multi-sectoral stakeholders beyond the MoH to build support and commitment to the program, particularly with the support of local governance structures. CONCLUSIONS: A REALIST VIEW A realist review of SBM-R asks, “What works, for whom, in what respects, to what extent, in what contexts, and how?” In this case study we have explored the contexts in which SBM-R is an effective quality improvement process. We have detailed the generative mechanisms that facilitate some health centers and hospitals to achieve success while others regress or do not engage. Finally, we have reviewed the outcomes achieved generally and more specifically in the selected case study facilities. The context-mechanism-outcome of the SBM-R process is described below. Context HSD reports that SBM-R currently engages 97 facilities across Guinea. This case study found that although each has some level of engagement since the start of the program in 2009 under MCHIP, not all are active. The following conditions affect engagement: w The health status of mother and children is precarious, and existing health services are not of sufficient quality to quickly improve MCH in Guinea. w The MoH and partners support improvement through training but it is not enough to change quality of services. w Contextual factors including poor infrastructure (limited or no water, toilets, sufficient physical space) and shortages or dysfunction of materials, supplies, essential medicine and equipment undercut participation, particularly when the factors are out of the control of facility managers or staff. w Human resource constraints (in number, capacity, and motivation) limit the perspective for rapid improvement in the quality of service delivery. w Community trust in the health system after Ebola is being rebuilt, albeit slowly. w Organizational culture in health facility, DPS, DRS and at the central level of the MoH is generally passive, fatalistic, self-serving, and opportunistic. w Accountability in the health system is low. Generative mechanism The SBM-R program is predicated on the belief that through recognition, staff, managers, community members and MoH and other local government leaders at District and Regional level can be motivated to improve service quality through public recognition. The mechanism assumes that: w Healthy competition between facilities should generate commitment and motivation to engage. w If providers are given training and equipment, they will choose to participate. w Managerial tools and training can be used by facility managers to lead the process in the facility. Through this process, leadership skills will be built. w Close follow-up, mentoring, and monthly meetings with the DPS and DRS, community leaders, COSAHs and facility managers help to sustain the process. w Gaps in infrastructure, equipment and other material deficiencies can be overcome largely by the facilities, districts and regions themselves. w Being awarded a Star will yield community recognition, appreciation, and reputation. Outcomes The SBM-R program further hypothesizes that in the Guinea health system context, the generative mechanisms of the SBM-R program will be sufficient to catalyze service and facility quality improvements that can be measured in improved service systems and health outcomes. Specifically, when facilities succeed in gaining a star, the progress will result in the following outcomes: w Participation in the quality improvement process and achievement (in stars) will change organizational culture towards service performance and monitoring; 16 Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study w Collective action, engagement and leadership at facility and district level with demonstrable results will sustain momentum and stimulate ownership of the program by the MoH; w MoH engagement will release investment and support for facility needs to sustain improvements; w Achievements and associated potential investments by the MoH will motivate providers and other staff to stay committed and engaged in the quality improvement cycle; w Community recognition of improved service quality at their local facility leads to greater utilization of health services; and w Maternal and child health will improve. We conclude that in the Context of the Guinean health services at HSD-supported SBM-R facilities, existing engagement of the MoH at all levels, and past distrust by the community of the health sector, quality improvement processes through SBM-R (generative mechanism) can lead to modest improvements that contribute to selected health Outcomes. Prospective outcomes could include: w Positive short-term improvements in clinical performance leading to improved delivery of specific health services and the resulting improved immediate health outcomes; w Improved managerial systems needed to begin to address hygiene and sanitation challenges but not sufficiently to significantly change IPC outcomes (due to structural limitations in terms of water, waste management, and equipment needed for sterilization, as well as difficulty in changing providers’ behavior); w Increased linkages and ownership of facility quality improvement efforts by communities and local government representatives including the DPS and DRS; and w Community satisfaction, trust, and utilization of the facility and its services improve through QI process. However, we do not see the context changing significantly through SBM-R to sustain and expand the program by the MoH without a change in the approach to increase their leadership and accountability to the process. RECOMMENDATIONS FOR SUSTAINABILITY Context w Small renovations of facilities are needed for the provision of integrated quality care that can meet IPC standards. w There is a need to expand these processes to rural and smaller health centers that still do not have consistent supplies and lack basic equipment like beds and scales. “Strategies avancés” (Outreach) and referral systems also need to be improved. w Better understanding how to adapt the SBM-R approach to the range of facilities, from small health centers to large hospitals is needed to ensure successful engagement by these facilities in the approach. A ‘one size fits all’ approach will not work. Process w Staff and manager engagement for quality improvement should be intrinsic in the job descriptions. Additional resources should not be needed to engage and participate in quality improvement processes to be implemented on existing services and structures with existing resources. w Engaging communities as partners increases commitment and motivation to achieve quality improvements at the facilities. w The national committee that validates SBM-R at the facility level needs to be formalized as part of the job descriptions or compensated to ensure its regular functioning. Making progress more visible through public dashboards with incremental steps will help to maintain engagement in the process. The committee must meet and conduct assessments at regular intervals that staff and communities can depend upon and plan for. The current ad hoc convening for the committee does not valorize the process. It also does not allow for the removal of stars for unsatisfactory performance which is also critical for the validity of the process. w The performance criteria assessed through SBM-R is extensive, and many are interdependent, particularly related to IPC such that if one factor is out of order, numerous other criteria cannot be met. A minimal set of Final Evaluation: USAID Guinea’s HSD Activity | SBM-R Case Study 17 criteria that capture the most critical aspects of quality care could replace the current list to make achievement (and maintenance of core standards) within reach for all facilities. w For the performance criteria in the tool it would be helpful to have an essential equipment and infrastructure list, that all facilities should have in place to achieve the performance standards. This will focus pressure on district and regional MoH representatives to ensure the minimum requirements are in order in each facility. w To facilitate regular internal assessment using the criteria, a tablet-based version could be established that would allow the team to check if essential equipment, supplies, consumables, and other resources are in place, and, if not, have a skip pattern to avoid answering interdependent questions. Transition to tablet-based technology will also support data sharing at all levels. w Progress on indicators also should be valued by the process, demonstrating effort and incremental achievement of criteria. If this was made visible, it may motivate staff and communities to engage more actively. This would require a steps or points to be given or ratings by subdomains such as equipment, infrastructure, client friendly, community engagement, clinical quality standards – some of which the staff can affect more readily than others. Engagement w Skills-building and close follow up is needed with managers at all levels to encourage ownership and commitment in the process. Many staff-based criteria could be met with closer follow up and small improvements in the conditions in which they work, beginning with waste management and hygiene. This will also require continuing education and engagement of staff and COSAH. w Better managerial leadership in conducting monthly monitoring and supervision could re-engage facility staff in SBM-R. w Given their commitment, MoH managers at the DPS and DRS levels should consider providing additional skilled staff and facility improvements. To not lose momentum, an evaluation should take place soon as they have made significant progress since they began. To ensure momentum is not lost, DPS and the DRS managers need to provide basic support through repair and renovation of critical equipment and facilities. In Labé, repair of the incinerator is urgent and deserved. Performance (Achievements and Outcomes) w A constellation of factors contributes to quality improvement achievements more than simply the length of time the facility has been engaged in the process. The specific context of a facility needs to be taken into account. Consider simplification of the QI process to focus on managerial process indicators linked to adherence to managerial standards for all facilities (at a high level)) rather than the granular investigation that can misfocus attention on detail when larger process and leadership issues mostly determine the effectiveness of the facility engagement. w The SBM-R process demonstrates that clinical performance criteria can be improved through continuous training and supportive supervision at the facility level. Dramatic improvements were observed shortly after the start of the HSD activity. HSD has facilitated the process by preparing a team of trainers around the country. To maintain progress, regular investment in training on standards and refresher training will be needed in future and can be driven by the District and Regional MoH. w When clinical performance is found to be unsatisfactory or even harmful, mechanisms for immediate recourse need to be in place with disciplinary measures that are clear to everyone. For example, management must respond to the discovery that some providers give personal, biased opinions on family planning messages that are not evidence-based. w The role of facility, district, and regional managers cannot be underestimated. SBM-R facilitated managerial leadership through training tools and protocols, which was effective in some districts and facilities. Given the role they play in the quality improvement process, more could be done to measure managerial performance and problem solving within the SBM-R assessment process. w Facility managers can advance quality improvement by prioritizing systemic solutions through guidelines, checklists, supervision, and control measures. Currently in some facilities, oversight is ad hoc. Using assessment data to make an action plan focused on specific issues was not done consistently. Routinized behaviors should be emphasized. The SBM-R program should develop, with managers, adapted tools that can facilitate management of IPC, including basic sanitation and hygiene measures in all service areas. The community could be enlisted to participate in support of such measures. w Facility managers must be proactive in identifying equipment and infrastructural deficiencies that limit quality improvements. The SBM-R program needs to include stock taking exercises that go beyond the facility raising the issue and responsibility for quality improvement at district and regional levels for inputs that cannot be addressed locally. The SBM-R process needs to implicate the MoH at all levels to own the achievements and challenges that they must help to overcome. The needs and obstacles to achievement need to be made visible in the process. w Critical facility improvements including water, sanitation (toilets and waste management), and sterilization need to be supported to provide the basis for quality improvements. Investment in these areas also helps to build staff commitment and momentum for further improvements. Analysis (Internal and external performance drivers) w Building teams to engage in quality improvement processes requires more than training on clinical and managerial skills, it also requires providing the training and resources that staff and managers need, so that they can work together effectively towards common aims. This is a continual process that will need to be embedded in the organization’s culture. To change the organizational culture in facilities and motivate those that are not currently engaged, new or adapted (non-monetary) incentives need to be developed with providers to ensure they are valued and appreciated by the staff on the frontline of quality improvement efforts. Perspectives on Ownership, Replicability and Sustainability w HSD efforts to establish regional SBM-R committees is important for local ownership, sustainability, and mentoring and should be sustained and reinforced by any future USAID activity in the sector. w As it is unlikely that the district and facilities can maintain the level of inputs of the HSD activity, it will be necessary to advocate for other sources of external support, possibly through local government and community resources to maintain the momentum that SBM-R has created. w To fully engage communities and facility staff in the improvement cycle, support needs to be grounded locally. Government representatives from all sectors and levels need to engage in the healthy competition and team up with the facility to make the improvements. Facilities, even with the support of the community, cannot go it alone. HSD and future USAID activities need to work more closely with multi-sectoral stakeholders beyond the MoH to build support and commitment to the program, particularly with the support of local governance structures. University Research Co., LLC w 5404 Wisconsin Avenue, Suite 800, Chevy Chase, MD 20815 w www.urc-chs.com