USAID/Senegal Final Mid-Term Evaluation Report 9 January 2015 Contract No.: AID-685-C-14-00001 Contracting Officer’s Representative: John Bernon, USAID/Senegal Health Office Development Objective (DO): DO 2 - Improved Health Status of the Senegalese Population Contractor: Ernst & Young LLP Author: Ernst & Young LLP This work product/document is intended solely for the information and use of the management of USAID and is not intended to be and should not be used by anyone other than these specified parties. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Acknowledgements This evaluation was conducted by Ernst & Young LLP (EY) in collaboration with Senegal’s Ministry of Health and Social Welfare (MOH) and the United States Agency for International Development in Senegal (USAID/Senegal). We would like to thank Abt Associates Inc. (Abt), IntraHealth Int’l (IntraHealth), the ChildFund consortium, Family Health International 360 (FHI 360), and the Agency for the Development of Social Marketing (ADEMAS) for their cooperation during the evaluation. Information was gathered from all 14 of Senegal’s regions at the district, regional, and Headquarters (HQ) level. Disclaimer: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government (USG). Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Acronyms Abt Abt Associates Inc. ACA Association Council for Action (Association Conseil pour l’Action) ACI Africa Consultants International ACT Artemisinin-Based Combination Therapy ADEMAS Agency for the Development of Social Marketing AFM Administrative Financial Manager AICPA American Institute of Certified Public Accountants AmQ Quality Improvement Approach (Approche pour l’Amélioration de la Qualité) AMTSL Active Management of Third Stage Labor ANC Antenatal Care ANCS National Alliance for the Fight Against AIDS (Alliance Nationale Contre le SIDA) AOR Agreement Officer’s Representative APL Acceptable Performance Level ARI Acute Respiratory Infection ART Antiretroviral Therapy ARV Antiretroviral ASC Associations Sportives et Culturelles (Sport and Culture Associations) AWP Annual Work Plan BCC Behavior Change Communication BG Bajenu Gox BICIS International Bank of Commerce and Industry of Senegal BNDE National Bank for Economic Development (La Banque Nationale pour le Développement Économique) BREIPS Regional Office of Education and Information for Health (Bureau régional de l'éducation et de l'information pour la santé) BTC Belgian Technical Cooperation Agency (Agence de Cooperation Technique Belge) CABECA Capacity Building for Electronic Communication in Africa CACMU Support Unit for Universal Health Coverage (Cellule d'Appui à la Couverture Maladie Universelle) CBC Communication for Behavior Change CBO Community-Based Organization CCA Counseling Center for Adolescents (Centre Conseil pour les Adolescents) CDCS Country Development Cooperation Strategy CDEPS District Center for Popular Education and Sports (Centre pour le Departemental pour l’Education Populaire et Sportive) CEFOREP Regional Center for Training and Research in Reproductive Health (Centre Régional de Formation, de Recherche et de Plaidoyer en Santé de la Reproduction) CH Community Health CHW Community Health Worker CMRO Chief Medical Region Officer CNCAS Advisory Council for Health and Social Action (Concertation Nationale pour la Santé et l’Action Sociale) CNLS National Committee for the Fight Against AIDS (Comité National de Lutte contre le SIDA) COP Chief of Party COR Contracting Officer’s Representative Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report CPJ Center for Youth Promotion (Centre de Promotion de la Jeunesse) CPS Center for Health Promotion (Centre de Promotion de la Sante) CRDH Centre de Recherche pour le Développement Humain CRS Catholic Relief Services CSNP Child Survival National Plan CSO Civil Society Organizations CSW Commercial Sex Worker CT Counseling and Testing CYP Couple Years of Protection DAGE Division of General Administration and Equipment (Direction de l’Administration Générale et de l’Equipement) DAN Division de l’Alimentation et de la Nutrition DECAM Decentralization and Health Insurance (Decentralisation de l’Assurance Maladie) DF Direct Financing DHMT District Health Management Team DHR Division of Human Resources DHS Demographic and Health Survey DLSI Ministry of Health and Social Welfare’s Division of HIV/AIDS and STIs (Division de Lutte contre le Sida et les IST) DMC District Management Committee DMO District Management Office DO Development Objective (Développement Organisationnel) DOTS Directly Observed Treatment Short-Course DPL Directorate for Pharmacy and Laboratories DPPD Multiyear Programming Document Expenditure DPRS Division on Planning, Research, and Statistics DSISS Department of Health and Social Information System (Division du Système d’Information Sanitaire et Social) DSRSE Department of Reproductive Health and Child Survival (Direction de la santé, de la reproduction et de la survie de l'enfant) ECD District Leadership Team (Equipe Cadre de District) EIPS Health Sector Policy Initiatives Team (Equipe d’Initiative et de la Politique de Sante) EMOC Emergency Obstetric Care EPI Expanded Program on Immunization EPS Public Health Establishments (Établissements Publiquescs de Santé) EY Ernst & Young LLP FBO Faith-Based Organization FHI 360 Family Health International 360 FM Financial Manager FoQus Framework for Qualitative Research in Social Marketing FP Family Planning FP/RH Family Planning/Reproductive Health GFATM The Global Fund to Fight for AIDS, Tuberculosis, and Malaria GHI Global Health Initiative GIS Geographic Information System GOS Government of Senegal HCP Health Communication and Promotion HIS Health Information System Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report HIV Human Immunodeficiency Virus HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HKI Helen Keller International HPNSP Health Promotion National Strategic Plan HQ Headquarter HSI Health Services Improvement HSS Health System Strengthening ICP Chief Nurse, Health Post (Infirmière Chef de Poste) ICT Information Communication Technology IEC Information, Education, and Communication IGA Income Generating Activity iHRIS Electronic Human Resources Information Software IntraHealth IntraHealth Int’l IP Implementing Partner IPC Interpersonal Communication IPT Intermittent Preventative Therapy IR Intermediate Result ISSA Innovations Group and Health Systems in Africa (Innovations et Systèmes de Santé en Afrique) ITN Insecticide Treated Net IUD Intrauterine Device JICA Japan International Cooperation Agency JPR Joint Annual Portfolio Review KPCF Key Population Challenge Fund LuxDev Luxembourg Development Agency M&E Monitoring & Evaluation MARP Most-at-Risk Population MCD Chief District Medical Officer (Médecin Chef du District) MCH Maternal and Child Health MCHIP Maternal and Child Health Integrated Program MDG Millennium Development Goals mHealth Mobile Health MIS Management Information System MHO Mutuelle Health Organization MNCH Maternal, Newborn, and Child Health MOH Ministry of Health and Social Welfare MOU Memorandum of Understanding MSM Men who have Sex with Men MTEF Medium Term Expenditure Framework NGO Non-Governmental Organization NTD Neglected Tropical Disease OI Opportunistic Infection ONAMS National Office of Mutuelles in Senegal (Office National des Associations Mutualistes de Sante au Sénégal) ORS Oral Rehydration Salts ORT Oral Rehydration Therapy PAC Post-Abortion Care PANPF National Advocacy Strategy for Family Planning (Plan d’Action Nationale de Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Planification Familiale) PAQ Package of Quality Services (Paquet Intégré de Service de Qualité) (PAQ) PBF Performance-Based Financing PCAOB Public Company Accounting Oversight Board PE Principal Evaluation PEPFAR President’s Emergency Plan for AIDS Relief PLWHA People Living With HIV/AIDS PLWHIV People Living With HIV PMI President’s Malaria Initiative PMTCT Prevention of Mother-to-Child Transmission PNA Central Medical Stores PNDS National Plan for Health and Development (Plan national de développement sanitaire) PNT National Tuberculosis Program (Programme National de Lutte contre le Tuberculose) POC Point of Contact PPP Public-Private Partnership PRA Regional Medical Stores PRIM Regional Integrated Multi-Sectorial Plan (Plan Régional Intégré Multisectoriel) PSC ChildFund-led Community Health Program (Programme Sante Communautaire) PSI Population Services International PTA Pavilion Outpatient Treatment (Pavillon Traitement Ambulatoire) RB Regional Bureau RH Reproductive Health RHMT Regional Health Management Team RMC Regional Management Committee RMNCH Reproductive, Maternal, Newborn, and Child Health RNP+ Rajasthan Network of People Living with HIV ROI Return on Investment RVC Regional Verification Committee SDM Standard Day Method SDP Service Delivery Point SEAD System for Exchanging Automatic Data SG Secretary General (Secretaire General) SI Strategic Information SME Small and Medium Enterprise SNEIPS The National Health Education and Information Service (Service National d’Education et Information pour la Sante) SPA Service Provision Assessment SSP Primary Health Care STI Sexually Transmitted Infection SUN Scaling Up Nutrition SWAA Society of Women and AIDS in Africa TB Tuberculosis TDR Rapid Diagnostic Testing TFP Technical and Financial Partner TPM+ Smear-Positive Pulmonary Tuberculosis (Tuberculose Pulmonaire a Frottis d’expectoration positif) TRaC Tracking Result Continuously TTBA Trained Traditional Birth Attendant Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report TWG Technical Working Group UEMOA West African Economic and Monetary Union (Union Économique et Monétaire Ouest-Africaine) UHC Universal Health Coverage UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund URC University Research Co., LLC US United States USAID/Senegal United States Agency for International Development in Senegal USD United States Dollars USG United States Government UTA Ambulatory Treatment Unit (Unité de Traitement Ambulatoire) VCT Voluntary Counseling and Testing WASH Water, Sanitation, and Hygiene WHO World Health Organization WV World Vision Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Table of Contents EXECUTIVE SUMMARY ............................................................................................................. I 1.0 INTRODUCTION ..................................................................................................................1 2.0 BACKGROUND ....................................................................................................................2 2.1 USAID/Senegal Health Program Overview .................................................................3 3.0 PURPOSE, DESIGN, AND METHODOLOGY OF THE EVALUATION ..........................7 3.1 Purpose and Objectives of the Mid-Term Evaluation ..................................................7 3.2 Principal Evaluation Questions ....................................................................................8 3.3 Evaluation Scope and Critical Assumptions ................................................................8 3.4 Data Collection Methodology Process Overview ......................................................11 3.5 Data Collection Tool Development ............................................................................15 3.6 Process for Data Collection, Management, and Analysis ..........................................18 4.0 HEALTH PROGRAM INTEGRATED APPROACH .........................................................22 4.1 Key Findings and Recommendations/Benefits ..........................................................22 4.2 Principal Evaluation Question #1: How effective has the structure of USAID’s overall health program and the division of the program into five components been in helping achieve the health development objective? ......................................23 4.3 Principal Evaluation Question #2: How have interventions been coordinated and implemented in synergy across components, with other USAID/Senegal programs, and with other development partners? ......................................................28 4.4 Principal Evaluation Question #3: To what extent has DF to the three regions been implemented successfully and what could be improved? ..................................30 4.5 Principal Evaluation Question #4: To what extent has the system of RBs and integrated work plans improved coordination among the five components? .............38 4.6 Principal Evaluation Question #5: To what extent has the program strengthened government ownership and demonstrated sustainability? ..........................................42 5.0 COMPONENT #1: HEALTH SYSTEM STRENGTHENING ...........................................43 5.1 Background ................................................................................................................43 5.2 Key Findings and Recommendations/Benefits .........................................................49 5.3 Sub-component A: Improving management and system performance at regional and district levels ........................................................................................................51 5.4 Sub-component B: Expanding alternative financing mechanisms and improving their sustainability ......................................................................................................56 5.5 Sub-component C: Ensuring that national level policies and systems support improved performance of the health system throughout Senegal ..............................59 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report 5.6 Sub-component D: Coordinating USAID/Senegal health program components to ensure that they work effectively together to support improved performance of the health system ........................................................................................................65 5.7 Response to Evaluation Questions .............................................................................68 5.8 Data Sources ...............................................................................................................72 6.0 COMPONENT #2: HEALTH SERVICES IMPROVEMENT ............................................72 6.1 Background ................................................................................................................72 6.2 Key Findings and Recommendations/Benefits ..........................................................77 6.3 Sub-component A: Increase access to an integrated package of quality health services .......................................................................................................................78 6.4 Sub-component B: Improve functioning of health services in public health posts and health centers and regional hospitals for related priority services provided in the Integrated Package ................................................................................................84 6.5 Sub-component C: Improve Human Resource Management at Public Health Facilities .....................................................................................................................86 6.6 Sub-component D: Outreach to Private Health Facilities ..........................................88 6.7 Response to Evaluation Questions .............................................................................89 6.8 Data Sources ...............................................................................................................92 7.0 COMPONENT #3: COMMUNITY HEALTH ....................................................................92 7.1 Background ................................................................................................................92 7.2 Key Findings and Recommendations/Benefits ..........................................................98 7.3 Sub-component A: Improving the quality of and access to information, products, and services at health huts and outreach sites ............................................................99 7.4 Sub-component B: Fostering community ownership and improving linkages and collaboration between the regional, district medical teams, development partners, and community level actors ......................................................................................104 7.5 Sub-component C: Fostering national MOH and other sector ministry ownership for CH and harmonizing the linkages with national policy initiatives .....................108 7.6 Response to Evaluation Questions ...........................................................................109 7.7 Data Sources .............................................................................................................113 8.0 COMPONENT #4: HIV/AIDS ...........................................................................................113 8.1 Background ..............................................................................................................113 8.2 Key Findings and Recommendations/Benefits ........................................................117 8.3 Sub-Component A: Support national efforts in prevention of sexual transmission of HIV .......................................................................................................................119 8.4 Sub-Component B: Reinforce comprehensive package of treatment, care, and support for PLWHIV ................................................................................................122 8.5 Response to Evaluation Question .............................................................................132 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report 8.6 Data Sources .............................................................................................................138 9.0 COMPONENT #5: HEALTH COMMUNICATION AND PROMOTION ......................139 9.1 Background ..............................................................................................................139 9.2 Key Findings and Recommendations/Benefits ........................................................145 9.3 Sub-component A: Strengthening capacity for effective BCC programs ................146 9.4 Sub-component B: Supporting implementation of quality BCC interventions leading to the adoption of healthy behaviors and the increased use of health services .....................................................................................................................151 9.5 Sub-component C: Strengthening the capacity of key actors to advocate for political and social engagement for health programs ...............................................155 9.6 Sub-component D: Social marketing of key health products resulting in their increased sale and use ...............................................................................................156 9.7 Sub-component E: Technical capacity building and organizational development of the recipient ..........................................................................................................158 9.8 Response to Evaluation Questions ...........................................................................161 9.9 Data Sources .............................................................................................................164 10.0 CONCLUSION ...................................................................................................................164 Annex A: Map – Implementing Partner Presence in Senegal Regions .............................167 Annex C: Data Point/Site Interview List ...........................................................................172 Annex D: Evaluation Work Plan .......................................................................................180 Annex E: Alignment of Principal Evaluation Questions and Component Specific Questions ..................................................................................................................195 Annex F: Literature Review Annotated Bibliography ......................................................206 Annex G: Bibliography .....................................................................................................216 Annex H: Implementing Partner Survey ...........................................................................223 Annex I: In-Depth Interview Guides .................................................................................225 Annex J: Focus Group Interview Guides ..........................................................................267 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. i EXECUTIVE SUMMARY A. Introduction The United States Agency for International Development in Senegal (USAID/Senegal) works in partnership with the Government of Senegal (GOS) to support the ten year National Plan for Health and Development (PNDS 2010 - 2018). In FY 2011, USAID/Senegal launched its integrated five component program aligned to the PNDS. The five component health program includes Health System Strengthening (HSS), Health Services Improvement (HSI), Community Health (CH), Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome (HIV/AIDS), and Health Communication and Promotion (HCP) with an overarching goal of “Improved Health Status of the Senegalese Population.” Each component is led by a different Implementing Partner (IP). USAID/Senegal is currently at the mid-way point of its five year health program and engaged Ernst & Young LLP (EY) to provide technical assistance for a mid￾term evaluation of its health program. In support of the evaluation, EY teamed with Dr. Ruth Kornfield and a local Senegalese based Non-Governmental Organization (NGO), Africa Consultants International (ACI), collectively forming Team EY. B. Evaluation Background USAID/Senegal’s health activities are categorized into three Intermediate Results (IR), which were validated by the Ministry of Health and Social Welfare (MOH). These include: 1) increased use of an integrated package of quality health services; 2) improved health-seeking and healthy behaviors; and 3) improved performance of the health system. The IRs and subsequent sub-IRs can be referenced in the Health Program Results Framework in Figure E.1 below: Figure E.1: Health Program Results Framework Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. ii USAID/Senegal’s primary objective of the mid-term health evaluation was to assess how each of the five program components, their IRs, and sub-IRs all link to achieve the overall goal of improved health status of the Senegalese population. The evaluation covered the period from October 2011 to June 2014. To achieve this objective, the evaluation:  Assessed progress toward achieving the expected results of the USAID/Senegal Health Program (2011- 2016).  Assessed effectiveness of program design, implementation, and sustainability mechanisms.  Identified lessons learned and proposed actionable recommendations to guide implementation for the remaining period of the program to improve performance. The evaluation will be used by various stakeholders (e.g., USAID/Senegal, the MOH, IPs, and the other United States Government (USG) agencies) to help inform USAID/Senegal’s future strategic plan. C. Evaluation Design and Methodology Team EY’s evaluation focused on gathering and assessing information directly related to responding to the Principal Evaluation (PE) questions outlined in Section 3.2 of the report. Five of the ten PE questions were focused on program integration and the remaining five were component specific. Any information pertaining to the USAID/Senegal Health Program that was not directly linked to a PE question was considered out of scope. Team EY employed a mixed￾method data collection approach which was inclusive of collecting qualitative and quantitative data concurrently to cross-validate and corroborate findings within the evaluation. Qualitative information was sourced from IP surveys, individual in-depth interviews, focus groups discussions, and observations conducted at selected data collection points/site visits. Quantitative data was primarily sourced from existing documentation, IP Annual Work Plans (AWP), contracts, annual reports, and supported with a literature review. A detailed bibliography of the 143 quantitative sources used to prepare this evaluation is referenced in Annex G. It was assumed that all available quantitative data provided from USAID/Senegal, the IPs, and other key stakeholders was reliable and valid. Additionally, Section 3.3 describes the interviews conducted with key stakeholders and in-depth interviews and focus group discussions by region, with a complete list of interviews and data points/site interviews available in Annex B and Annex C. Prior to the field data collection, Team EY developed a set of 19 quantitative data collection tools to conduct the in-depth interviews and focus group discussions (a complete set of data collection tools is referenced in Annex I and Annex J). To support the data analysis process, Team EY created an alignment matrix which linked each PE question, to the IRs, sub-IRs, and data collection tools (the matrix is referenced in Annex E). Additional details regarding Team EY’s purpose and objectives of the evaluation, scope, critical assumptions, data collection methodology, data collection tool development, and the process for data collection, data management, and data analysis can be found in Section 3.0 of the evaluation report. D. Key Findings and Recommendations/Benefits Table E.1 below provides an overview of the key findings and recommendations/benefits identified during the evaluation. The complete list of findings and recommendations are provided in the health program integrated approach and each of the five components in Sections 4.0 – 9.0 of the Final Evaluation Report. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. iii Table E.1: USAID/Senegal Key Findings and Recommendations/Benefits Overview # Key Findings Recommendations/Benefits Health Program Integrated Approach 1 Team EY concluded that although the projects are meeting most of their contractual obligations in terms of indicators and deliverables, they are primarily operating vertical programs with examples of strong integration across the other components, but not through a standardized approach. Overall, interview respondents believe the five components funded by USAID/Senegal were aligned with the goals and objectives of the MOH, and covered a set of important interventions that contributed to improving the health of the Senegalese population. Recommendation: During the next program design phase, USAID/Senegal may want to consider defining and developing the essential package of health services needed to support a continuum of care approach. Research has demonstrated that more comprehensive coverage of effective interventions is possible when health services interventions are integrated throughout the life cycle using a continuum of care approach. One key aspect to implementing a continuum of care approach is the focus on strengthening linkages between the household, community, and facility levels and the second aspect is strengthening the health system and the skills of all human resources integral to the functionality of the continuum of care (i.e., medical facility staff and community health workers). Benefit: Redesigning the health strategy with an emphasis on a continuum of care approach with fewer, more comprehensive components, could help improve integration and coordination of services among IPs, which could lead to increasing the impact of USAID/Senegal’s health funding. 2 To help facilitate coordination, Abt Associates (Abt) manages the Regional Bureaus (RB) and has placed a full-time RB Coordinator in each one. However, the current process of coordination with the other IPs through the RB does not function as effectively as it could, both internally (among IPs) and externally (in relation to regional and districts committees). Interviews with IPs indicated that they do not always receive information on the achievement of AWP milestones from the RBs. Recommendation: USAID/Senegal may want to reconsider the design of the RBs by reviewing the RB coordinator function and developing a communication plan for information dissemination from the RBs to each of the IPs as well as the region and district health teams. Benefit: This may improve coordination and clarify the role and responsibilities of the RB and IP staff, as well as address gaps in supervision and M&E data collection by better organization and more consistent, scheduled follow-up for regions and districts. 3 Challenges within the supply chain system are affecting the ability of the components to implement aspects of their programs related to commodity procurement and security. Recommendation: USAID/Senegal may want to consider technical assistance from a supply chain field support partner who can focus exclusively on resolving supply chain issues affecting USAID/Senegal Health Programs. . Benefit: A supply chain technical assistance partner could work with each IP and focus on Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. iv # Key Findings Recommendations/Benefits issues related to the processes and systems of forecasting, procuring, storing, and distributing health commodities. In addition to the support provided to the IPs, the benefit of having a common supply chain partner is that USAID/Senegal could provide direct assistance to the Central Medical Stores (PNA) and Regional Medical Stores (PRA) for capacity building and human resource planning. Component #1: HSS 1 Feedback received on PBF was positive from health providers, who believed it was motivating and changing the way they delivered care. It was noted that PBF is contributing to positive behavioral change in the region, emphasizing an increase in service quality and ownership over health services. The success of the PBF pilot provides a basis for the MOH and other donors to financially support PBF scale-up. However, in the majority of interviews with MOH and Abt staff, the general consensus was that the process of data collection and indicator verification from start to finish was cumbersome. Recommendation: USAID/Senegal may want to consider using mobile technology for PBF data collection, verification, and bonus payments to increase the indicator validation process and the speed at which incentives are received by service providers. Benefit: More efficient and streamlined PBF processes can improve decentralized services and reduce the burden on regional and district health teams. 2 September 2014 marked the one year anniversary of the launch of Universal Health Coverage (UHC) by the GOS. The Mutuelle Health Organization’s (MHO) coverage to the informal and rural sector remains the strategic priority for progressing toward wider health coverage. Implementation of the various UHC components will likely result in a considerable increase in the use of health services in a context where there is a lack of service providers. Recommendation: USAID/Senegal may want to consider recommending that the MOH implement programs aimed at strengthening the delivery of health care services with an emphasis on the recruitment of qualified health care personnel and procurement of sufficient equipment for health facilities. Additionally, USAID/Senegal may want to develop creative strategies to engage the private sector (including assessing Public-Private Partnership (PPP) opportunities) and support employment and job creation for beneficiaries. Benefit: These recommendations may help address the growing demands placed on the health workforce and facilities as more of the population benefits from UHC over the next decade. 3 It appears the GOS’s financial contributions and donor resources are insufficient to achieve the ambitious health objectives. With the goal of the GOS to improve efficiency and transparency in budget allocation, there is an opportunity to Recommendation: USAID/Senegal may want to consider providing technical assistance to the MOH to implement a detailed financial resource gap assessment supported by the development of a strategy to mobilize resources within the public and private sectors. USAID/Senegal may also want Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. v # Key Findings Recommendations/Benefits better understand where current resources are being expended and what gaps remain. to consider developing a process to validate that the strategy includes sustainability measures, such as developing reliable revenue streams to decrease the level of dependence on donor resources. Benefit: This approach would allow the GOS’s financial contributions and donor resources for health to be more efficiently allocated and achieve the health objectives, while beginning the process of developing the GOS financial streams to move toward sustainability. 4 A number of actual results were unavailable or unreported in Abt’s year one and year two annual reports. This is because Abt accesses the data from the GOS public finance management systems which were greatly impacted due to data collection strikes which affected the entire country of Senegal, from the GOS to donor partners. The data collection systems are just beginning to recover and there are gaps in data collected and the quality of the data available. Abt’s responsibility was not to collect the data for each indicator, but to collate and verify the information received from the MOH, whose responsibility it is to collect the data. This is because these are national level indicators not USAID specific indicators. Abt noted in their annual report that despite their efforts, they did not have enough influence over the MOH leaders to gather the data needed to respond to the indicators and therefore believed it was best to leave them blank. Recommendation: Given that the entire data collection and management system was damaged by the prolonged national strike, Team EY recommends that USAID/Senegal may want to perform a data quality and controls assessment of the current indicators to make sure they are back on track and being collected correctly and consistently, especially those indicators related to public financial expenditure. Benefit: A data quality and controls assessment of the current indicators, especially those related to public finance and transparency could help restore the GOS’s promise to its citizens for improved visibility on tracking public spending and increasing their confidence in the government that funding is going to serve the needs of the people. The assessment may also provide insight into key control gaps related to collection, data transmission, reconciliation, and monitoring regarding financial expenditures. Component #2: HSI 1 The pilot phase of the electronic Human Resource for Health Information Software (iHRIS) was noted in interviews for supplying health managers and practitioners with information to assess human resource constraints and to subsequently plan and evaluate interventions. Recommendation: With the expansion of the iHRIS software system to support the MOH and Regional Health Management Teams (RHMT), USAID/Senegal may want to conduct an independent assessment of staffing needs and develop a strategy and action plan for staffing of the Service Delivery Points (SDP) in the five regions where the iHRIS is operating. Benefit: This assessment could contribute to Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. vi # Key Findings Recommendations/Benefits enhancing the use of more effective human resources in these regions by providing information on how iHRIS is being used and what additional human resources, infrastructure, or other support is needed to improve usage. 2 Interviews demonstrated that there may be more opportunities to better apply strategies that focus on improving the quality of services and care within private practices, primarily the Non-Governmental Organizations (NGO) and Faith-Based Organizations (FBO) sector within a more standardized continuum of care approach. Recommendation: USAID/Senegal and IntraHealth may want to consider forming networks of private providers to serve as champions and thought leaders on how the sectors can support each other in the management of care for Senegalese communities using a continuum of care approach. Benefit: This could support complementary care and contribute to achieving better outcomes at the district and regional level for beneficiaries, in line with standardized care from both private and public sector providers that are serving populations. 3 It appears that overall Demographic and Health Survey (DHS) 2012 health indicators are weakest in regions where the integrated packages are limited. The two regions of Matam and Tambacounda, specifically, show low overall health indicators that could benefit from the expanded integrated package services. Recommendation: USAID/Senegal may want to consider expanding the integrated package of services and the integrated package for malaria currently provided by IntraHealth to regions with weaker indicator results. Benefit: The expansion of integrated service packages to urban areas or higher-populated facilities may increase access of the population, which could lead to improved uptake of services. Component #3: CH 1 The functionality of the health huts and their outreach activities appear to be highly dependent on the consistent support of the ChildFund consortium. There is consensus among key stakeholders, including the MOH (regional and district health offices), service providers, and CH Management Committees that without the support, the quality and continued maintenance of services will decline or in some instances cease. Recommendation: USAID/Senegal may want to continue to support the current services provided at health huts, but not expand to additional sites in order to focus on increasing the quality of services. Benefit: The continued support will assist with the quality and maintenance of services until the local government and elected officials can sustain services. By not expanding services, the ChildFund consortium can focus on skills transfer, capacity building, and sustainability measures to reduce dependency on the ChildFund consortium. 2 The quality of services at the health huts appear to be compromised by inconsistent availability of medical supplies, poor Recommendation: USAID/Senegal may not want to expand the number of services in the integrated package in order to first improve the quality of Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. vii # Key Findings Recommendations/Benefits infrastructure, and lack of electricity and water. Project reports indicate that some health huts are in such disrepair that they are no longer functional (i.e., roof has collapsed) and that consultations lack privacy as the consultation rooms have no doors. To solve these problems, communities built enclosures and additional rooms where possible. It was reported that villagers call on their relatives who live abroad to send money for construction support and several health huts leveraged funds from local Community-Based Organizations (CBO) to support the installation and cost of electricity. existing interventions, increase supervision and monitoring, and support renovations of health huts. Benefit: The focus on existing interventions may allow the ChildFund consortium to improve the existing quality of services (i.e., fully functional health huts) and develop a better process for supervision and M&E of services. 3 The commitment from local government authorities to take over and support existing CH activities appears to be low. As USAID/Senegal transfers financial responsibility to the communities for the health huts, they become dependent on local government authorities for the majority of their resources. Recommendation: USAID/Senegal may want to consider placing more emphasis on advocacy efforts targeting locally elected officials to create a more favorable political environment for health huts. USAID/Senegal may also want to consider this as a priority agenda item for the donor coordination group to discuss solutions on how the GOS can increase financial support to health huts and CH services in general. Benefit: This approach may increase locally elected officials’ commitment and interest in supporting and allocating funds to health huts once USAID/Senegal resources are withdrawn. 4 The lack of incentives to support Community Health Workers (CHW) appears to be a barrier to the consistent availability of support in their communities. CH is based primarily on volunteer CHWs and outreach workers who are dedicated, but need to earn an income. Recommendation: The MOH, USAID/Senegal, and other donors may want to develop more effective strategies to incentivize and motivate CHWs and outreach workers. Benefit: This approach may increase consistent volunteer availability by helping to incentivize CHWs and outreach workers (e.g., financial, recognition, training). Component #4: HIV/AIDS 1 Men who have Sex with Men (MSM) and Commercial Sex Worker (CSW) face additional barriers to seeking and utilizing health services besides discrimination and stigmatization. Fear of prosecution, community exclusion, and weak laws to protect MSM and CSWs in cases of police Recommendation: Health communication approaches need to take into account these complex factors with subtle and inclusive messaging and outreach activities. Since MSM and CSWs in Senegal are disproportionately affected by HIV/AIDS, USAID/Senegal may want to consider increasing mobile services for key Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. viii # Key Findings Recommendations/Benefits abuse and unfair treatment, affect their decision-making. HIV testing yields for MSM are low compared to the general population. This is likely attributed to issues of stigma in addition to general negative views regarding People Living with HIV/AIDS (PLWHA), as well as the lack of societal support for those who come forward for services. populations and organizing services around a “hotspot” strategy. Additionally, USAID/Senegal may want to consider expanding and more closely evaluating neutral Behavior Change Communication (BCC) campaigns such as messaging about partner reduction and risks associated with multiple concurrent partnerships. Benefit: The program benefit of targeting young men in major cities or hubs for Counseling and Testing (CT) through mobile services includes encouraging the use of extended testing hours, and providing privacy to attract them to services. Other than CSWs and MSM who require this support, a benefit of targeted interventions for discordant couples and transient populations (i.e., truck drivers, fishermen, gold miners, and migrant workers) could strengthen tailored health communications. 2 Team EY’s understands that FHI 360 supports and is planning on initiating test and treat in Senegal. The scientific rationale for opting for a test and treat approach is that it is a proven intervention for reducing transmission and is being used in many countries with high prevalence rates. The intervention is generally based on testing everyone in 'high risk' groups and areas of generalized epidemics, and then immediately treating all of those diagnosed positive, regardless of whether their immune system is damaged or meets the clinical definition to initiate ART (CD4 count of less than 350). Taking into account the very low prevalence rate in the general population, the resource constraints in Senegal, and the commitment to patients’ lifelong needs for ART, Team EY believes that an alternative to the test and treat model deserves further consideration. Recommendation: A focus on maintaining high quality ART services for those already on treatment, meeting the unmet need for those who clinically should be on ART, and supporting more cost-effective interventions such as the treatment of opportunistic infections could be a more feasible public health approach for managing the HIV/AIDS epidemic. Benefit: The program benefit of considering alternative interventions to test and treat could be more cost-effective while not significantly damaging the needs of PLWHA. There are many interventions that might be better investments and meet the needs of the epidemic in Senegal. Component #5: HCP 1 Social marketing of health products was the most successful part of this component according to ADEMAS indicator results. The social marketing of products supported the promotion of the integrated package of services in the USAID/Senegal Health Recommendation: USAID/Senegal may want to continue to expand social marketing of key products through ADEMAS and strengthen the BCC campaigns developed to support the products. In addition, USAID/Senegal may want to request that ADEMAS use innovative and integrated Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. ix # Key Findings Recommendations/Benefits Program. platforms like social media and mobile technology to reach target populations, especially youth and Most-at-Risk Population (MARP). Benefit: Social marketing may increase the knowledge, demand for, and use of life-saving products through targeted marketing. 2 According to constraints cited by ADEMAS in their annual reports, ADEMAS struggled to establish relationships and leverage influence over central government agencies to improve institutional capacity building. Recommendation: USAID/Senegal may want to consider engaging another partner with expertise and experience in institution capacity building to provide this support directly to the MOH. Another option USAID/Senegal may want to consider is providing technical assistance to ADEMAS to improve their internal capability to provide institutional capacity building. Benefit: Bringing in another partner may allow ADEMAS to focus on their strengths (i.e., social marketing) while another partner can focus on the gap of institutional capacity building to better strengthen management and financials. However, since ADEMAS is already beginning year three of their cooperative agreement, it may not be worth replacing this sub-component with a new partner and thus providing technical assistance to ADEMAS may be a more feasible solution. 3 Challenges in collaboration between ADEMAS and the ChildFund consortium are hindering performance of both IPs’ components. For ADEMAS, this is limiting its ability to effectively collaborate with outreach workers and local CBOs. Recommendation: USAID/Senegal may want to consider discussing with ADEMAS more effective ways for collaboration at the community level for activity implementation. To support this, USAID/Senegal may want to consider facilitating a discussion with the ChildFund consortium and ADEMAS on methods to improve collaboration at the community level. During this discussion, USAID/Senegal may want to clearly delineate between the responsibilities of ADEMAS and the ChildFund consortium in their work with the CBOs (e.g., have the ChildFund consortium responsible for mobilization in the CBOs and ADEMAS be responsible for HCP activities). USAID/Senegal may also want to consider building required indicators for improved collaboration between the USAID/Senegal supported IPs into the structure of the program. Benefit: This may improve ADEMAS’s ability to effectively communicate and collaborate between partners, outreach workers, and local CBOs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 1 1.0 INTRODUCTION Improving the health of the Senegalese population is a priority Development Objective (DO) of the United States Agency for International Development in Senegal (USAID/Senegal). In order to achieve this objective, USAID/Senegal continues to build upon decades of United States Government (USG) investment and partnership with the Government of Senegal (GOS) as the largest bi-lateral donor in the health sector. In 2009, the GOS approved a ten year National Plan for Health and Development (PNDS 2009 – 2018) with a vision for Senegal where all individuals, households, and communities have universal access to quality curative and preventative health services without any form of exclusion. The PNDS vision is directly aligned to Senegal’s Millennium Development Goals (MDG) for health which focus on 1) reducing child mortality; 2) improving maternal mortality; and 3) combatting malaria, Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome (HIV/AIDS), and other diseases. The USAID/Senegal Health Program is a critical partner in the Global Health Initiative (GHI) and is aligned with the Country Development Cooperation Strategy (CDCS) for 2012-2016. In 2011, USAID/Senegal began implementing an integrated five component health program. The five components are listed in Table 1 below. Table 1: USAID/Senegal Five Health Components Number Name Component #1 Health System Strengthening (HSS) Component #2 Health Services Improvement (HSI) Component #3 Community Health (CH) Component #4 HIV/AIDS Component #5 Health Communication and Promotion (HCP) USAID/Senegal is currently at the mid-way point of its five year health program and engaged Ernst & Young LLP (EY) to provide technical assistance for a mid-term evaluation of its health program covering the period from October 2011 to June 2014. In support of the evaluation, EY teamed with Dr. Ruth Kornfield and a local Senegalese based Non-Governmental Organization (NGO), Africa Consultants International (ACI), collectively forming Team EY. Team EY anticipates that the findings from this evaluation will be shared with the Ministry of Health and Social Welfare (MOH) and utilized by USAID/Senegal to inform future strategic planning decisions. The primary audience for this evaluation is USAID/Senegal, the GOS, and Implementing Partners (IP). The information contained within this Final Evaluation Report details Team EY’s evaluation design, data collection methodology, evaluation tools, scope, assumptions/limitations, findings, and recommendations based on USAID/Senegal’s Principal Evaluation (PE) questions. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 2 2.0 BACKGROUND Senegal is considered one of the most politically and economically stable countries in the West Africa region. It has a population estimated at 12.9 million and is growing at a rate of 2.5% annually. The United Nations Human Development Index (2012) ranks Senegal as 154 out of 177 countries worldwide. Significant progress was achieved in reducing child mortality over the last several years, but substantial challenges related to maternal mortality and Family Planning (FP) remain. According to the 2012 - 2013 Demographic and Health Survey (DHS), between 2005 and 2012 infant mortality decreased from 61 to 42 per 1,000 live births, under-five mortality decreased from 121 to 68 per 1,000 live births, and the maternal mortality ratio dropped from 401 to 392 per 100,000 live births. Fertility has slowly and consistently decreased but remains high, at five children per woman. Only about 16% of married women of reproductive age use modern methods of contraception, and 30% of married women have an unmet need for FP. Under-nutrition is a major underlying cause of maternal and child death and disability, with 17% of children in Senegal suffering from stunting1 . Many of the top causes of death for children under-five years of age are preventable, including malaria, neonatal causes, pneumonia, diarrheal disease, and measles. In recent years, immunization coverage has declined and measles and polio outbreaks have recurred. Some of the greatest barriers to health care utilization in Senegal include insufficient numbers of health care workers, inadequate access to essential medicines and social, cultural, and religious beliefs that continue to influence gender inequities. The World Health Organization’s (WHO) health factsheet on Senegal states that there is less than one doctor (0.6%) per 10,000 people.2 Senegal is recognized for containing the HIV/AIDS epidemic and maintaining one of the lowest prevalence rates in Africa at 0.5%.3 Currently, there is a strong need for strategies to better support the GOS and to develop partners in HSS or service integration and narrow the gap between urban and rural health care access for the most vulnerable populations.4 The decentralized health management structures, community committees, and national commitment to the provision of high quality services offer a platform to build on and scale-up successful interventions. For example, there are early indications of success among interventions focused on Performance-Based Financing (PBF) and community-based health insurance models to improve access, quality, and costs associated with services.5 The health care structure in Senegal, as depicted in Figure 1, consists of a network of public health facilities that includes 22 regional hospitals, 78 district health centers, 986 public health posts, 144 private health posts, and approximately 2,000 health huts.6 Regional hospitals provide relatively advanced care; district health centers are intended to provide first-level referrals and limited hospitalization services; and health posts provide preventive and primary curative 1 “Senegal, 2010-11 Demographic and Health Survey and Multiple Indicator Cluster Survey Key Findings,” USAID/Senegal, 2012. 2 “Senegal Health Profile,” WHO, 2012. 3 “Epidemiological Fact Sheets on HIV/AIDS and Sexually Transmitted Infections,” WHO, 2004. 4 “Senegal Country Development Cooperation Strategy 2012-2016,” USAID/Senegal, February 2012. 5 "Composante Renforcement du Système de Sante – Draft- Rapport Annuel de la Période Octobre 2012- Septembre 2013," Abt, November 7, 2013. 6 “The United States Health Strategy, Senegal Global Health Initiative Strategy,” USAID/Senegal. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 3 services, care for chronic patients, prenatal care, FP, and health promotion activities. It is important to note the MDs at health posts are often replaced by a nurse or housewife. Health huts, the foundation of Senegal’s health care pyramid, are managed by local communities and account for approximately 19% of the country’s health seeking population. Figure 1: USAID/Senegal Decentralized Health Structure 2.1 USAID/Senegal Health Program Overview USAID/Senegal DO for health as outlined in the CDCS is to improve the health status of the Senegalese population by reinforcing the GOS’s efforts to reduce infant and under-five mortality, rates of under-nutrition, and the number of hospital visits due to malaria. In order to accomplish these goals USAID/Senegal categorized its health activities into three Intermediate Results (IR), validated by the MOH. These include: 1) increased use of an integrated package of quality health services; 2) improved health-seeking and healthy behaviors; and 3) improved performance of the health system. The IRs and subsequent sub-IRs can be referenced in Figure 2 below. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 4 Figure 2: Health Program Results Framework Each IR is linked to the following high-level USAID/Senegal indicators: Table 2: USAID/Senegal Indicators Number Name IR 1  Expanded Program on Immunization (EPI) coverage  Number of deliveries with a skilled birth attendant  Couple-Years-of-Protection (CYP) IR 2  Percentage of children between six to 23 months receiving minimal acceptable diet  Use of Insecticide Treated Nets (ITN) by household members  Percentage of target population who know how to prevent key illnesses (e.g., HIV/AIDS, malaria IR 3  Percentage of increase in GOS contribution  Percentage of facilities with stock-outs of essential drugs  Data used to guide program design The USAID/Senegal Health Program was designed with the intention of being implemented as a unified whole, with each component contributing to the achievement of the overall health strategy for effective integration and implementation. Figure 3 below provides a mapping of the health interventions by IP per region. Additional maps can be referenced in Annex A. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 5 Figure 3: USAID/Senegal Health Interventions7 USAID/Senegal’s CDCS recognizes the importance of the GHI principles and the President’s Malaria Initiative (PMI) in achieving its DO, improved health status of the Senegalese population. USAID/Senegal is recognized for its partnership with the GOS and efforts to align programs with the policies and objectives outlined in the PNDS 2009-2018.8 USAID/Senegal’s health DO incorporates gender equality and seeks impact through strategic coordination with other donors. USAID/Senegal continues to collaborate with key multilateral organizations like the World Bank and the Global Fund for AIDS, Tuberculosis, and Malaria (GFATM). The CDCS also states the goal of engaging the private sector in global health partnerships, particularly in the area of health communications and FP promotion. To enhance the likelihood of sustainability, the program is fostering country ownership of, and investment in approaches and interventions, particularly in health service delivery at the community level. The current USAID/Senegal Health Program design is built off the platform of previous projects focused on increasing access to basic health services and facilitating policy reforms aimed at increasing health resources and training staff at the community and district levels. It was designed as a complete strategy, divided into five components and engaged a broad range of actors (national and international) to address the complexity of health challenges in Senegal. This approach 7 Data sources used to produce Figure 3 were IP cooperative agreement and workplans. 8 “Senegal Country Development Cooperation Strategy 2012-2016,” USAID/Senegal, February 2012. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 6 requires facilitative leadership and considerable investments in communication and coordination activities. All IPs are headquartered in Dakar, and many have decentralized offices in regions where they are implementing services or providing technical assistance. Additionally, there are three RBs in Thiès, Kaolack, and Kolda, which house a Regional Coordinator and technical advisors from each of the five IPs. Abt is responsible for leading the coordination, planning, and logistical support of the RBs. As part of the mid-term evaluation, Team EY assessed several financial aspects to include DF, the decentralized funding approach, and PBF. As DF was one of the main PE questions related to program design and integration, Team EY’s findings/analysis related to this can be found in Section 4.5.1 and lessons learned/recommendations can be found in Section 4.5.2. The analysis and findings related to the decentralized financing approach regarding the RB coordination PE question can be found in Section 4.6.1. The lessons learned related to decentralized financing can be found in Section 4.6.2. Lastly, Team EY’s analysis related to PBF can be found in Section 5.3 as part of the HSS sub-component A. Financial information was gathered through various sources including a review of existing documentation (e.g., budgets in Annual Work Plans (AWPs), financial management plans), IP survey responses, data point/site interviews, and interviews between Team EY’s financial analyst and IP FMs. An assessment of how each IP is progressing is included in the component specific sections, beginning in Section 5.0. The component specific sections provide an overview of the indicators that were available to Team EY for analysis and summarize the key activities and finding from the AWPs. Team EY used the DHS 2010/2011 and 2012 data to develop a baseline understanding of how the various regions were performing based on population-based indicators. Based on the indicator regional analysis, Team EY then compared the data against each of the five components funded by USAID/Senegal. It is important to note that the analysis was constrained due to several limitations related to the data. For example, not all regions are implementing a uniformed set of indicators as seen in other USAID or President’s Emergency Plan for AIDS Relief (PEPFAR) programs and IPs often report on a common set of high-level process indicators that help management teams compare results across technical areas and geographic zones. Team EY is not suggesting that these process level indicators be the sole indicators that are reported on by IPs, but that having a few standardized measures can help to maintain a macro-view perspective of how implementation is proceeding. PEPFAR specifically uses a set of Next Generation Indicators to help manage data and a similar approach may be helpful for USAID/Senegal to consider. In addition, there was significant variance in the way IPs defined a SDP. For example, SDPs could be a hospital, health post, or a health hut, but not all interventions were housed in a physical structure (e.g., outreach by CHWs, mass media campaigns). However, based on the analysis Team EY was able to conduct, the most notable findings were from the DHS 2010-2012. The indicators below were chosen based on their applicability to USAID/Senegal’s high-level indicators as stated in Section 2.1.  The regions of Tambacounda, Louga, Matam, and Kaffrine have modern contraceptive use rates of less than 10%.  Diourbel (46.6%), Tambacounda (41.2%), and Fatick (45.8%) had the lowest percentage of pregnant women who received Tetanus toxoid injection. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 7  The percentage of women who reported a serious problem accessing health care for themselves when they were sick exceeded over 50% in all regions. In Kolda, Tambacounda, Kédougou, and Kaffrine percentages exceed 70%.  The West and North regions have the lowest malaria prevalence (1% each) while malaria prevalence is highest in the South grand region (9%).  Condom use among women was lowest in Matam (43%) and Tambacounda (51.5%) and highest in Dakar (80%). For men, condom use was highest in Dakar (90%) and lowest in Tambacounda (58.4%) and higher in men in Matam (72.2%). According to the DHS 2012, male condoms are provided at 70% of facilities, although only 1% of married women use male condoms. Team EY noted that in the regions of Fatick, Thiès, Ziguinchor, Sédhiou, and Kaffrine, there were improvements in the area of malnutrition care. In the regions of Louga, Kolda, and Sédhiou service provider capacity to perform malaria lab testing (microscopy and Rapid Diagnostic Testing (TDR) improved. However, other regions are lagging in the provision of the integrated package of malaria and other clinical services. USAID/Senegal funded activities in Saint-Louis are almost exclusively CH, HSI, with HCP occurring in only one district. Similarly in Louga, only the CH and HSI components are being implemented, and these are concentrated in the northwest of the region, providing very limited coverage. In addition, the RBs are too far to actively and effectively support the regions of Matam, Tambacounda, and Saint-Louis. This resulted in low levels of supervision and support for M&E. In interviews with stakeholders in these regions, the distance from the RB was cited as a key barrier in the perception of a RB’s utility. 3.0 PURPOSE, DESIGN, AND METHODOLOGY OF THE EVALUATION Team EY’s evaluation was designed to identify the factors which both facilitated and impeded improvements, as well as those that supported efficiency, effectiveness, relevance, and sustainability with regard to HSS within the context of regional decentralization. It also assessed coverage, access, and quality issues that provided both qualitative and quantitative information, as available, to identify catalysts for change and areas for further improvement. One aspect of the evaluation was to assess the sustainability and cost-effectiveness of USAID/Senegal’s initiatives to strengthen both service delivery as well as Health Information Systems (HIS). Sections 3.1 – 3.6 below provide details regarding Team EY’s purpose and objectives of the evaluation, evaluation scope, critical assumptions, PE questions, data collection methodology, data collection tool development, and the process for data collection, data management, and data analysis. 3.1 Purpose and Objectives of the Mid-Term Evaluation USAID/Senegal’s primary objective of the mid-term health evaluation was to assess how each of the five program components, their IRs, and sub-IRs all link to achieve the overall goal of improved health status of the Senegalese population. To achieve the objective, Team EY:  Assessed progress toward achieving the expected results of the USAID/Senegal Health Program (2011 - 2016). Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 8  Assessed the effectiveness of program design, implementation, and sustainability mechanisms.  Identified lessons learned and proposed actionable recommendations to guide implementation for the remaining period of the program to improve performance. 3.2 Principal Evaluation Questions As noted above, per the contract, Team EY’s evaluation focused on the PE and component specific questions listed in Table 3 below. During the course of the evaluation, it was decided by USAID/Senegal to remove the component question related to gender because an in-depth gender assessment was occurring simultaneously. Table 3: PE Questions and Component Specific Questions # PE QUESTIONS PROGRAM DESIGN AND INTEGRATION 1 How effective has the structure of USAID’s overall health program and the division of the program into five components been in helping achieve the health DO? 2 How have interventions been coordinated and implemented in synergy across components, with other USAID/Senegal programs, and with other development partners? 3 To what extent has DF to the three regions been implemented successfully and what could be improved? 4 To what extent has the system of RB and integrated AWPs improved coordination among the five components? 5 To what extent has the program strengthened government ownership and demonstrated sustainability? COMPONENT SPECIFIC QUESTIONS 1 To what extent have the components achieved their objectives? 2 To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component? 3 What are the constraints and challenges that have hindered successful implementation of each sub￾component, and how has the IP dealt with those challenges? 4 Are there interventions that should be added or removed? Are there changes that could be made to improve performance? 3.3 Evaluation Scope and Critical Assumptions Team EY’s evaluation focused on gathering and assessing information directly related to the PE questions outlined in Section 3.2. Any information pertaining to the USAID/Senegal Health Program that was not directly linked to a PE question was considered out of scope. Another critical factor that impacted scope was the duration of the assessment. Per the contract, Team EY had three months to prepare, conduct data collection, and develop the evaluation report. Due to the limited time frame, Team EY was restricted to utilizing information that was available during the three month period. If documentation was not available (e.g., year three annual reports or Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 9 indicators from each component) or key resources were not available within the time frame allotted, Team EY was not able to utilize the information for analysis. Data and other information available to Team EY for the evaluation included background information sent to Team EY by USAID/Senegal and the IPs, IP survey responses, key stakeholder interviews (e.g., informational interviews with IPs, interviews with IP Financial Managers (FM), USAID Agreement Officer’s Representatives (AOR), and MOH representatives (as seen in Figure 4 below)), and field interviews across all 14 regions. The complete list of key stakeholder interviews conducted can be referenced in Annex: B. Figure 4: Key Stakeholder Interviews Conducted Figure 5 below provides an overview of total interviews conducted with key stakeholders. A minimum of 10 data point/site interviews per region were included as part of the scope. All primary data collected was qualitative. Any quantitative data received was through secondary sources (e.g., component AWPs, annual reports). The complete list of data point/site interviews conducted is referenced in Annex: C. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 10 Figure 5: Data Point/Site Interviews Conducted The following assumptions/limitations are critical to note when reviewing the evaluation report:  The structure of the evaluation report was provided by USAID/Senegal per contract requirements. It is assumed that the layout and flow of the report is the preferred format to display the findings of the evaluation.  Due to the short timeline for the evaluation and the need to design the evaluation tools to initiate data collection, the literature review was not exhaustive in terms of pursuing a weighted categorization of randomized controlled trials, quasi-experimental, and non￾experimental studies. A deliberate focus on evidence which was quickly obtainable, authored by known sources, and other evaluations, was adopted instead. With a longer time frame, Team EY would have executed a more in-depth review.  The sites selected for field interviews were identified through both random selection (based on a set of criteria referenced in Section 3.6.1), as well as a selection of preferred sites. The preferred sites were identified by a USAID/Senegal Point of Contact (POC) as they were believed to have information and input to support the evaluation. It is assumed that sites identified for field visits provided an overall representation of comparable sites, to assist with evaluating the PE questions.  It is assumed that all data gathered through surveys, interviews, as well as data reported in secondary sources (e.g., annual reports) were accurate and valid.  It is assumed that all available quantitative data related to the PE evaluation questions were submitted to Team EY and were reliable and valid.  It is assumed that relevant information from USAID/Senegal pertaining to the information was provided, rather than withheld. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 11  Financial management documentation reviewed was limited to that which was provided by the IPs. Not all IPs provided supporting documentation to verify practices stated in interviews. USAID/Senegal advised Team EY that the financial assessment was of lesser importance relative to the other objectives of the evaluation.  Team EY’s review of decentralized financing was limited by its inability to directly observe the full funds distribution process due to time constraints, as well as a lack of availability of sub-contractor interviews.  A gender evaluation was not part of Team EY’s scope, as USAID/Senegal awarded another contractor to perform a detailed gender assessment. However, some gender data was collected (e.g., amount of men vs. women who completed trainings). High-level gender information gathered is included in component specific sections when applicable. 3.4 Data Collection Methodology Process Overview Team EY employed a mixed-method data collection approach which was inclusive of collecting qualitative and quantitative data concurrently to cross-validate or corroborate findings within the evaluation. Team EY’s approach to the evaluation, the data collection methodology, timeline, and evaluation tools was outlined in the Evaluation Work Plan approved by USAID/Senegal in October 2014 and is included in Annex D. Sections 3.4 – 3.5 below. These sections provide further information regarding literature review findings, in-depth interview guides, focus group discussion guides, observations, and IP surveys. Quantitative information was primarily sourced from existing documentation, IP AWPs and annual reports, and supported with a literature review. The documents reviewed consisted of USAID/Senegal’s program documents as well as other externally relevant sources. The literature review informed the development of the evaluation tool design and facilitated Team EY’s contextual understanding behind the key health interventions being implemented by each component. Please reference Section 3.4.2 for a detailed summary of Team EY’s literature review. The data collection tools developed were directly aligned with USAID/Senegal’s PE questions and were also aligned to the health components, IRs, and sub-IRs for analysis purposes as referenced in Annex E. Further information regarding Team EY’s data collection tools can be referenced in Section 3.5. Feedback from all standardized IP surveys, individual in-depth interviews, focus groups discussions, and observations conducted at selected data collection points/site interviews was transcribed, summarized, and then catalogued by component for analysis. 3.4.1 Alignment of Principal Evaluation Questions A complete alignment of questions to the PE and component specific questions can be referenced in Annex E. 3.4.2 Literature Review Summary A literature review was conducted to inform the development of the evaluation tool designs described and facilitate Team EY’s understanding of the context behind the key health interventions being implemented by each component. Team EY’s objectives were to document Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 12 existing intervention strengths and challenges through a review of existing literature provided by USAID/Senegal and identify best practices through a review of external sources. The literature review supported findings, by comparing the literature review results to the qualitative data collected during the evaluation. Evidence-based practices are summarized in Table 4, with an annotated bibliography on each of the 25 additional external sources referenced in Annex F. The literature review focused on the following areas relating to the five health components: • Senegal specific intervention focus: It is important to note that the literature review took into special consideration four programmatic areas relevant for the Senegal program. These were PBF, community case management, quality improvement methodologies, and innovations for programs linked to decentralization models. The above practices were identified because, at each component level, they inform how managers and communities foster joint ownership, understand indicators of success, and execute evidence-based service delivery strategies. PBF and Mutuelle Health Organizations (MHO), two interventions being scaled up in Senegal, build on the globally recognized areas of results-based financing, and community-based health insurance studies. • Linkages, relationships, and integration: The literature review focused on how the five components supported the GOS initiatives which aim to strengthen integration at national and local levels. This referred to integration across all levels of the health pyramid and across sectors. For example, the IPs all contribute to an integrated work plan across the five component areas with the goal of using resources more cost-effectively and improving synergies across service delivery. • Planning, budgeting, and cost-effectiveness: During the literature review, Team EY analyzed linkages associated with decentralized funding mechanisms and direct regional financing. Our approach included a broad assessment of financial flows and their relationships to performance outcomes. For example, in the Final Evaluation Report, information is provided on the success of each IP’s interventions, factoring in variables that contribute to cost such as geographic location, proximity to a central medical store, and access to local transportation. • Process: The first phase of analysis focused on sources provided by USAID/Senegal including the CDCS, the GHI Strategy, the DHS, Service Provision Assessment (SPA), the PNDS (2010-2018), IP Annual Work Plans (AWP), IP quarterly and annual reports, and indicators reported in the National Health Management Information System. Please reference Annex G for a complete bibliography of all documents reviewed. The second phase of the literature review included a database search of Google Scholar, Open Access public health journals, and other publications and presentations. A total of 25 articles were identified as a result of health components similar to USAID/Senegal’s interventions, regional donor funded health or development programs in West Africa, and Senegal specific documents. Additionally, search terms included the health component by name and these key words: “best practices,” “quality,” “decentralization,” “innovation,” “community,” and “health outcomes.” The three key parameters of the literature review were the focus on multi-country studies, multi￾year literature reviews, and implementation of science case studies from low-income/ low- Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 13 resource settings. Additionally, another inclusion criterion was that the article was original research drafted in English and the source discussed aspects of service provision relevant to the USAID/Senegal Health Program implementation. The selection process of health specific components’ publications generated information that allowed the field team to note key points and questions for consideration as they designed the evaluation tools and later in the analysis of the data. For the data collection tools, elements such as the stratification of interviews and sequencing in each region, beneficiary and community-sensitive questions, and culturally nuanced execution were enhanced by authors’ perspectives from the literature review. Findings:9 Key findings from the external literature review are summarized by health component in the table below. Table 4: Literature Review Key Findings Health Component Evidence-Based Practices HSS Scale-up PBF: PBF is an approach for structuring the flow of resources to pay for results—desired goals, outcomes, and even impact. In the PBF approach, the purchaser transfers to the provider (or facility) significant power and authority over strategies and activities, and also the potential for reward or loss. Countries that have developed national PBF programs, such as Rwanda and Burundi, are seeing significant changes in the attitudes of health providers as they are held accountable for the services they deliver. As responsibility shifts to health facilities to increase the quality of services linked to tangible results, the improved financial sustainability is also achieved. This is accomplished through the gradual transfer of payment on PBF indicators from donors to the host government. This practice is relevant to the USAID/Senegal Health Program as they move from the pilot phase of PBF to scale￾up. Develop an integrated data collection platform: The challenges of data collection and reporting systems are well documented in developing countries. Many of the current systems lack data for decision-making, have significant issues related to data quality, limited access to data by the general public, and place burden on government structures and staff for reporting large amounts of data to international organizations. The concept of a technology-based integrated local health management information system, which harmonizes reporting of patient and program data and operates off of one consolidated platform, is gaining traction with donor agencies and IPs. A well￾functioning consolidated data platform can lead to increased data access, thereby enhancing transparency, accountability, and governance. It can also improve data quality through enhanced adherence to international standards, and promote regional sharing of lessons learned and best practices. This practice is relevant to the USAID/Senegal Health Program given the strike of data collectors and how it hindered consistent and reliable data collection in the country for over a year. HSI Standardize Monitoring & Evaluation (M&E) processes: Evidence suggests that the use of a set of agreed-upon standard program monitoring indicators to measure progress across multiple IPs can help improve the burden of data collection and reporting. These practices put in place a systemized approach that supports providers, 9 All data sources for the Literature Review are included in Annex G as well as in the Annotated Bibliography in Annex F. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 14 Health Component Evidence-Based Practices managers, and technicians in health program planning, course-correction, and implementation. Often data is collected for reporting purposes but not used to improve decision-making. Standardization across processes, reporting, and supervisory roles/responsibilities can build strong teams, provide clarity of purpose, and create efficiencies (i.e., costs saving and reduced burden on human resources). This practice is relevant to USAID/Senegal because with the expansion of the TutoratPlus program there is an opportunity to use data to improve performance. Integrate performance improvement: Systems left unchanged may be expected to continue to produce the same results. Documented programs that integrated performance improvement interventions at regional and district health facility, health post, and health hut levels recorded improvements in the motivation of health providers and the quality of services delivered. An example of performance improvement interventions is the integration of on-site supportive supervision systems for all cadres of health service providers. USAID has documented many models of performance improvement methods including, University Research Co., LLC’s (URC) Collaborative Improvement Model for low and middle income countries. This practice is relevant to USAID/Senegal because there may be aspects of the URC Collaborative Improvement Model that could expand the TutoratPlus Model. A desk review could provide USAID/Senegal with some insight into how to further strengthen on the TutoratPlus Model to include additional performance improvement modules. CH Support community case management: For Universal Health Coverage (UHC) to be successful, services must be accessible to all people and should be provided in cost-effective ways. A key approach to addressing both of these challenges lies within community-based services, which are provided closer to where people live and are deemed to be more affordable and sustainable through the use of volunteers or low￾cost community workers. Many simple, affordable, and effective disease control measures have limited impact on the burden of disease due to their inadequate distribution in poor and remote communities. This practice is relevant to USAID/Senegal given the priority of moving toward UHC. Greater integration is particularly relevant for the delivery of those community-level interventions in which the community itself participates. Senegal is implementing community-based case management for expanded treatment of diarrhea, pneumonia, and malaria. Expand community-based health insurance: Community-based health insurance models are promising alternatives for a cost sharing health care system which can lead to better utilization of health care services, reduce illness related income shocks, and eventually lead to a sustainable and functioning universal health care system. Solutions that bring health services closer to the end user, and utilize different contracting schemes typically increase access to services by beneficiaries. This practice is relevant to USAID/Senegal because Senegal is supporting MHOs which are a type of CH insurance intervention. Findings from case studies suggest that MHOs can be successful in increasing utilization of modern health care services and reducing catastrophic health related expenditures. HIV/AIDS Focus on a continuum of care model: The rapid implementation of vertically driven programs was noted as successful in providing life-saving treatment for large numbers Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 15 Health Component Evidence-Based Practices of patients in need. However, this success was achieved with insufficient attention given to the establishment of the necessary linkages and integration of HIV care within the broader health system. These are essential for the establishment of an effective continuum of care both before and during Antiretroval Therapy (ART). The care pathway is not a simple linear process and the dynamic nature of linkage, retention, loss and re-engagement in care, especially in the pre-ART stage, makes this a challenging pathway to assess. Interventions to increase HIV testing include task￾shifting (e.g., testing through lay health care workers); provider-initiated testing as well as mobile, community, home-based, and workplace offerings. These offerings bring the services nearer to the patient and thus increase accessibility. Community￾based strategies for HIV testing and ART delivery are important to include in the continuum of care as implementers develop ways to further expand access to care. This practice is relevant to USAID/Senegal as HIV is currently not included in the health systems improvement component which limits the integration of services with the same target audience such as pregnant women, Counseling and Testing (CT) clients, Most-at-Risk Population (MARP), and those seeking STI services. HCP Target behavior change campaigns to reach key populations: The use of mass media as a vehicle for behavior change interventions has demonstrated that it can reach substantial portions of target populations and mobilize community activities. In addition, it can stimulate dialogue and awareness among its audiences, specifically in relation to the appropriateness of strategies to change social and gender norms and individual behaviors. However, evaluations and studies show that interventions can fall short in addressing some of the prevalent harmful cultural practices that have led to individuals making better decisions, especially when related to sexual health practices. This practice is relevant to USAID/Senegal because key populations such as Men who have Sex with Men (MSM) and Commercial Sex Workers (CSW) are often not reached with Behavior Change Communication (BCC) messages and services that are relevant to their needs in Senegal. In countries where religion plays an important role in influencing people’s behaviors, it is critical that they are involved in the development and process of health communications. Evaluations of faith-based programs demonstrate that donors and partners should focus on several key areas: 1) capitalize on the trust developed between Faith-Based Organizations (FBO) and local communities to build stronger, more complete, and integrated prevention effort; 2) develop the capacity for FBOs to advocate for improved health care for all citizens and hold governments accountable; 3) leverage the existing organizational infrastructure of faith-based health systems to reach communities, including vulnerable, hard-to-reach, and most at-risk populations; and 4) develop the capacity to communicate in ways that are relevant and meaningful to religious communities, donors, and governments. This practice is relevant to USAID/Senegal given the influential role of FBOs and religious leaders and their potential to mitigate the negative perceptions surrounding marginalized groups such as MSM and CSWs. 3.5 Data Collection Tool Development Team EY utilized the quantitative data findings from the literature review to inform the development of several different data collection tools to include: focus group discussion guides, in-depth interview guides, IP surveys, and observations as seen in Table 5 below. The focus Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 16 group discussion guides and in-depth interview guides were refined based on feedback from USAID/Senegal as well as information gathered during the course of data collection. This form of data collection was conducted in order to better understand the qualitative factors which both facilitate and hinder high-quality interventions at the decentralized level. Based upon trends that emerged from the quantitative data, the selection of individuals and groups for qualitative interviews led to an understanding of the processes by which high and low performing sites/regions achieve different outcomes. As noted above, please reference Annex C for a complete list of data points/sites interviewed during the field visits. Table 5: Data Collection Tools Name of Tool Description IP Survey Short survey disseminated to the five IPs prior to the site visits that was utilized as an important first step for assessing IP perspectives in order to maximize the use of their time when conducting the in-depth interview. In-Depth Interviews Interviews with service providers and community level stakeholders to elicit their perspectives on barriers to behavior change and uptake of services. The interviews allowed for a better understanding of the impact and effectiveness of programs on: technical capacity (including training, task shifting, and management of local budgets), linkages and referrals within the health pyramid, resource allocation, health care workers’ workload, job satisfaction and motivation, sustainability (including engagement with local government and local community), and gender￾specific and equity considerations. Focus Group Discussions Discussions with health center staff at all levels to identify factors which motivate staff as well as impede their ability to fulfill their responsibilities. The focus groups conducted allowed Team EY to better understand beneficiaries’ perspectives with regard to: access to services, quality (including availability of commodities, confidentiality), gender and equity-based differences on use and impact, ownership, accountability, and sustainability. Observations Informal and formal observations conducted to understand what happens when patients attend health facilities. Types of observations conducted by Team EY included observing how long beneficiaries wait for services, what type of information beneficiaries receive, provider client interactions, and how clinical data is captured. The IP survey sent to the five IPs can be referenced in Annex H. The list of in-depth interview guides is included in Table 6 below and each individual tool can be referenced in Annex I. The list of focus group discussion guides is included in Table 6 below and each individual tool can be referenced in Annex J. Table 6: In-Depth Interview Guides and Focus Group Discussion Guides Tool Name of Individual Tools In-Depth Interview Guides 1. AOR/IP 2. MOH – Component #1: HSS 3. MOH – Component #2: HSI 4. MOH – Component #3: CH 5. MOH – Component #4: HIV/AIDS 6. MOH – Component #5: HCP 7. Health structures (i.e., hospitals, centers, and posts) 8. Health huts Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 17 Tool Name of Individual Tools 9. National/regional pharmacies (GOS) 10. Private pharmacies 11. Regional and district health offices (GOS) 12. Regional and district coordinating offices (USAID/Senegal) 13. Public-Private Partnerships (PPP) Focus Group Discussion Guides 1. CH workers 2. CH insurance (committee) 3. CH insurance (leader/manager) 4. CH insurance (beneficiaries) 5. CH Management Committee 6. Associations of People Living with HIV/AIDS (PLWHA) Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 18 3.6 Process for Data Collection, Management, and Analysis Team EY executed formal data collection for the evaluation. Specifically, our methodology identified how the data was collected, where and when data was collected, and how the data was analyzed. The process as depicted in Figure 6 below describes Team EY’s approach for site selection, data collection, data management and security, and data analysis. Detailed information for each phase is described in the sections below. Figure 6: Overview of Process for Data Collection, Management, and Analysis Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 19 3.6.1 Site Selection Criteria The site selection process took into account that data needed to be collected at the national, regional, and district levels, and include facility-based, community-based, and administrative sites (i.e., regional health offices). Data was collected from a variety of sources including Abt Associates Inc. (Abt), IntraHealth Int’l (IntraHealth), the ChildFund consortium, Family Health International 360 (FHI 360), and Agency for the Development of Social Marketing (ADEMAS), MOH (i.e., policy makers, managers), RB representatives, Community Health Workers (CHW), Bajenu Gox (BG) (female volunteers supporting women’s health), representatives of local NGOs and local implementing organizations, community stakeholders (i.e., local community leaders, religious leaders, women’s leaders, and health advocacy groups), mothers of children under-five, FP users, adolescents, CSWs, MSM, PLWHA, men accessing health services, health management committees, and the private sector. The additional criteria for site selection were:  At least one district per region.  Equivalent health interventions implemented in regions with a regional coordination office compared to those without a regional coordination office.  Facility-based sites that represent urban, semi-urban/rural, and rural populations.  Community-based sites that represent urban, semi-urban/rural, and rural populations.  Representation of sub-recipients to include a minimum of one project under each of the prime IPs.  Representation of regions with PBF interventions.  Representation of regions where there are CH insurance interventions.  Representation of high-performing regions versus low performing regions (based on health indicators).  Representation of private sector pharmacies and health facilities. 3.6.2 Data Collection In order to capture data from the 14 regions, Team EY’s in-country team divided into three field teams, comprised of three team members each. Table 7, below provides an overview of the team breakout. Table 7: Team Coordination and Regional Assessments Team Team Lead Team Members Regions Visited* A Dr. Ruth Kornfield  Mr. Ibrahima Diallo  Dr. Rene Carvalho  Mr. Gabriel Diouf  Thiès  Louga  Matam  Saint-Louis  Dakar Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 20 Team Team Lead Team Members Regions Visited* B Dr. Ismail Diene Thioye  Mr. Abdoulaye Konate  Mr. Lafi Charles Diatta  Ms. Yama Sy  Fatick  Diourbel  Tambacounda  Kaffrine  Kaolack C Dr. Georges Tiendrebeogo  Mr. Djiby Sow  Mr. Moustapha Dieng  Ms. Virginie Kantoussan  Sédhiou  Kolda  Kédougou  Ziguinchor *Note: All Team Leads conducted interviews in Dakar (the 14th Region) A number of factors were considered when deciding which regions the different teams visited. These factors included guidance from the USAID/Senegal AOR and technical advisors, local regional contextual knowledge and experience, local language capabilities, as well as appropriate linkage of each team member’s technical skills with the health activities being implemented in the regions where they collected data. Prior to data collection, Team EY conducted a three day internal training for all sub-team members, led by the Evaluation Team Leader. The training facilitated a standardized data collection process between teams. This was important given the integrated nature of the data collection, whereby the data collected was interpreted and used for analysis by all team members. In the training the Evaluation Team Leader reviewed the process to be followed (as depicted in Figure 6 above) for data collection, including the tools to be used and the mandatory quality assurance reviews. As noted in Figure 6, Team EY developed note templates to capture field interview notes. The notes in the templates were then copied into off-line web-based forms which were created for each in-depth interview and focus group discussion. The forms were installed on each team member’s computer during the training, and were subsequently uploaded into Team EY’s data collection tool that is described in Section 3.6.3. The training also provided guidance on effective methods for conducting in-depth interviews as well as focus group discussions. The Evaluation Team Leader emphasized the use of a multi￾vantage point approach, where a conscious effort was made to not let the data collector’s biases shape how they interpreted responses. All data collectors operated using the same set of assumptions, definitions, and approaches to limit biases and pre-conceptions. The primary mode for collecting and analyzing data was progressive focusing. Progressive focusing allowed for adjustments during the data collection process when it appeared that additional concepts needed to be investigated or new relationships explored. The teams started with a set of defined questions, but realized flexibility was required as new data led to additional questions, concepts, focus area, and information that were not previously considered. The teams were encouraged to discuss their data with each other on a regular basis to encourage consideration of alternative data interpretations and challenge pre-existing views. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 21 3.6.3 Data Management Following the Evaluation Team Leader’s review and approval of the data collected, each field team’s designated data collector entered the data from the notes template into the corresponding off-line web-based forms, which were automatically uploaded into Team EY’s data collection tool (i.e., an automated database stored on a HTML 5 server) when internet connectivity allowed. This database was a secure site that required a user name and password for log-in. Only members of Team EY were provided log-in access to view the web-based tool. Team EY user access varied based on capabilities needed including upload, edit, and read only access. No other third parties were provided access to the data. The final step in the Team EY data management process was a quality check performed on the data uploaded on the HTML 5 server. The Headquarters (HQ) team conducted a review of the data point/site interviews conducted against the data collected in the tool on the server. Any discrepancies noted were discussed with the Evaluation Team Leader and any remaining data needed or identified was uploaded to the server. 3.6.4 Data Analysis Team EY utilized the quantitative data available through IPs’ quarterly and annual reports to review their indicator results over the past two years of implementation and assessed how each IP was progressing. This included a comparison of proposed AWPs and actual activities, as well as tracking the achievement of indicators and milestones annually. In each component section, high-level indicator tables and corresponding results are included. Additionally, the most relevant challenges and findings highlighted in the IP’s annual reports are integrated into each component section. For the analysis of qualitative data, the field team leaders divided up the program’s five components as well as the integrated approach among each other. Each team leader took the lead on the analysis for two sections. Field team leaders were paired with members of the headquarters team to extract and analyze applicable data per section. Once all sections were drafted, the Evaluation Team Leader reviewed and approved all final sections prior to report compilation. To analyze the data, Team EY used the following processes:  Documentation of the data and the process of data collection.  Organization/categorization of the data into concepts (e.g., health components, IRs, and sub￾IRs).  Connection of the data to show how one concept may influence another.  Data validation, by evaluating alternative explanations, disconfirming evidence, and searching for negative cases.  Representing the account (reporting the findings). Team EY utilized the field data collected to perform a comparison analysis between regions/locations and cross-sectional analysis with other descriptive factors such as demographic data to understand whether socioeconomic, age, or gender impacts were influential factors. To help this type of analysis, the team included structured questions in addition to open-ended Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 22 questions, in the data collection tools. Where possible, questions were structured in a yes/no format and followed up with the opportunity for further explanation. Having questions in this structured format helped the team to get an understanding of frequency of responses for specific observed variables. In addition, for each data collection point, the team gathered demographic and geographic information. The team was able to filter responses by geographic/demographic variable and subsequently observe frequency of response for structured variables. This allowed for comparison of the response across interventions, regions or respondent-type. For example, it allowed Team EY to see beneficiaries’ satisfaction with a particular service received, broken down by region or demographic information, such as a responder’s gender. Team EY’s data collection tools interfaced with Drupal 7, which is a software analytics program. Team EY utilized Drupal 7’s Views Application as its primary tool for data analysis. This tool allowed Team EY to view data point/site interview responses by question and sort each response by variables including region, intervention, and respondent-type. The Views Application included a number of features to allow the team to modify and export data in different forms to facilitate analysis. The ‘Report’ function allowed the team to create separate reports by question or to filter data by a specific variable and then generate a report of the data. The ‘Analysis’ feature was utilized to classify information by frequency of response for those questions of a structured and dichotomous nature. Moreover, the application allowed the team to download data into excel for easier viewing and manipulation for those team members who were more comfortable with using the data in that format. 4.0 HEALTH PROGRAM INTEGRATED APPROACH10 4.1 Key Findings and Recommendations/Benefits Table 8: Health Program Integrated Approach Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefits 1 Team EY concluded that although the projects are meeting most of their contractual obligations in terms of indicators and deliverables, they are primarily operating vertical programs with examples of strong integration across the other components, but not through a standardized approach. Overall, interview respondents believe the five components funded by USAID/Senegal were aligned with the goals and objectives of the MOH, and covered a set of Recommendation: During the next program design phase, USAID/Senegal may want to consider defining and developing the essential package of health services needed to support a continuum of care approach. Research has demonstrated that more comprehensive coverage of effective interventions is possible when health services interventions are integrated throughout the life cycle using a continuum of care approach. One key aspect to implementing a continuum of care approach is the focus on strengthening linkages between the household, community, and facility levels and the second aspect is strengthening the health system and the skills of all human resources integral to the functionality of the continuum of care (i.e., medical facility staff and community health workers). Benefit: Redesigning the health strategy with an emphasis on a continuum of care approach with fewer, more comprehensive 10 Data from this component was derived from all sources including: The bibliography (Annex G), key stakeholder interviews (Annex B), and data point/site interviews (Annex C). Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 23 # Key Findings Recommendations/Benefits important interventions that contributed to improving the health of the Senegalese population. components, could help improve integration and coordination of services among IPs, which could lead to increasing the impact of USAID/Senegal’s health funding. 2 To help facilitate coordination, Abt manages the RBs and has placed a full-time RB Coordinator in each one. However, the current process of coordination with the other IPs through the RB does not function as effectively as it could, both internally (among IPs) and externally (in relation to regional and districts committees). Interviews with IPs indicated that they do not always receive information on the achievement of AWP milestones from the RBs. Recommendation: USAID/Senegal may want to reconsider the design of the RBs by reviewing the RB coordinator function and developing a communication plan for information dissemination from the RBs to each of the IPs as well as the region and district health teams. Benefit: This may improve coordination and clarify the role and responsibilities of the RB and IP staff, as well as address gaps in supervision and M&E data collection by better organization and more consistent, scheduled follow-up for regions and districts. 3 Challenges within the supply chain system are affecting the ability of the components to implement aspects of their programs related to commodity procurement and security. Recommendation: USAID/Senegal may want to consider technical assistance from a supply chain field support partner who can focus exclusively on resolving supply chain issues affecting USAID/Senegal Health Program. Benefit: A supply chain technical assistance partner could work with each IP and focus on issues related to the processes and systems of forecasting, procuring, storing, and distributing health commodities. In addition to the support provided to the IPs, the benefit of having a common supply chain partner is that USAID/Senegal could also provide direct assistance to the Central Medical Stores (PNA) and Regional Medical Stores (PRA) for capacity building and human resource planning. 4.2 Principal Evaluation Question #1: How effective has the structure of USAID’s overall health program and the division of the program into five components been in helping achieve the health development objective? The USAID/Senegal Health Program was designed with the intention of being implemented as a unified whole, with each component contributing to the achievement of the overall health strategy and for effective integration and implementation. USAID/Senegal is currently supporting health services in all 14 regions of Senegal. A brief summary of USAID/Senegal’s integrated components is detailed below and a breakout of the budget per component can be found in Figure 7: Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 24  Component #1: HSS - Improve the performance of the decentralized public health system of Senegal (district and regional levels) and validate that it is supported by effective and efficient policies, planning, and budgeting at the central level of the MOH.  Component #2: HIS - Improve linkages to community-based services and to regional hospitals to support a well-coordinated continuum of care. The component seeks to improve not only the availability of the integrated package of services but also the functioning of facilities and teams that deliver these services.  Component #3: CH - Improve access to and quality of CH services and information, support the active engagement of beneficiaries in seeking and using health care options at the community level (urban and rural), contribute to the establishment and sustainability of technical linkages and referral networks for community-based health service providers, and strengthen the relationship between CH structures and the formal GOS health system.  Component #4: HIV/AIDS - Provide targeted, relevant technical assistance and institutional support to the GOS and partners in order to maintain a low national prevalence of HIV/AIDS, reduce transmission in high prevalence areas and among most vulnerable populations, improve the quality and availability of treatment, care and support PLWHA, and improve detection and treatment of Tuberculosis (TB) among PLWHA.  Component #5: HCP - Support a range of communication activities to influence the social and behavior changes needed to improve outcomes in the priority technical areas of Reproductive Health (RH), Maternal and Child Health (MCH), malaria, HIV/AIDS, TB, and other infectious diseases to maximize the use of relevant approaches, materials/tools, and media products already developed and used successfully in Senegal.11 Figure 7: USAID IP Budget 11 “USAID/Senegal Request for Proposal-mid-term health evaluation,” February 2014. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 25 4.2.1 Findings and Analysis Each of the five independent IPs use their own respective institutional policies and procedures with distinct scopes of work and targets for which they are accountable. There is one IP for each component and USAID/Senegal expects that the IPs should collaborate across components and seek opportunities to better integrate services where appropriate. Interviews with the MOH authorities at the national, regional, and district levels indicated satisfaction with the division of the USAID/Senegal Health Program into the five components. The overall perspective was that the components covered a set of important interventions that contributed to the Senegalese government’s health program. Team EY’s perspective was that the five components funded by USAID/Senegal were aligned with the goals and objectives of the MOH, and covered a set of interventions that contributed to improving the health of the Senegalese population. There were no suggestions for adding additional components, but rather recommendations for improved support within components and a recognition that there were opportunities for better collaboration and integration between components. An example of this is that the HCP approach could be more cross-cutting by targeting community educational needs for promoting mutuelles and for targeting high risk groups for HIV prevention especially in the southern regions. According to the stakeholders identified by USAID/Senegal, the five components addressed the needs and expectations of the Regional and District Health authorities. IntraHealth, the ChildFund consortium, ADEMAS, and FHI 360 offered no suggestions for an alternative structure for the components. Abt suggested that conceptually it could be beneficial to collapse the five components into three components: 1) HSS, 2) HSI (which would include HIV/AIDS), and 3) CH merged with HCP. Team EY concluded that although the projects are meeting their contractual obligations in terms of indicators and deliverables, they are primarily operating vertical programs with examples of strong integration across the other components, but not through a standardized approach. There are several examples including:  Limited integration of HIV/AIDS (FHI 360) in the integrated package of services (implemented by IntraHealth).  Lack of cross-over in training between facility-based services (IntraHealth) and community￾based services (the ChildFund consortium) even though health huts are being given additional responsibilities in the areas of EMOC, early warning signs of pregnancy complications, and treatment of under-five diarrhea.  Limited supervision and commodity supplies coordination between the health post (IntraHealth) and health hut (the ChildFund consortium) levels. As USAID/Senegal continues the process of strategic planning, consideration should be given to restructuring the components under a new design of the health program. 4.2.2 Lessons Learned and Recommendations ► One recommendation is for USAID/Senegal to define and develop the essential package of health services needed to support a continuum of care approach. Research has demonstrated that more comprehensive coverage of effective interventions is possible when health service Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 26 interventions are integrated throughout the life cycle using a continuum of care approach.12 One key aspect to implementing a continuum of care approach is the focus on strengthening linkages between the household, community, and facility levels. The second aspect is strengthening the health system and the skills of all human resources integral to the functionality of the continuum of care (i.e., medical facility staff and community health workers). Practically, Team EY would suggest that USAID/Senegal reduce the number of components to focus on integrated clinical facility-based services, integrated community￾based services, and centralized HSS support (e.g., PBF, human resource recruitment and planning, M&E, policy reform, and institutional capacity building) as depicted in Figure 8 below all within the larger framework of a continuum of care approach. In doing so, both the integrated clinical services and the integrated community services (in addition to implementing a services package) would each include interventions that focus on performance-based incentives, human resource capacity building and development, and health promotion. Additionally, these components could be reinforced with targeted technical assistance by partners that focus on supply chain technical assistance, innovative technology to improve data collection and reporting, and social marketing of health products that could all be linked to the community and facility-based service delivery programs. Supply chain technical assistance can be further referenced in several sections, including 5.5.2 and 8.4.2. A continuum of care approach would make fewer partners responsible for a larger set of integrated services and would help to reduce the current environment where partners are understandably, focused on meeting their specific, component activities. It could allow the program to reach its target audience with more complete interventions and would facilitate the same set of standardized interventions being offered in all geographic areas where USAID/Senegal is implementing services. The challenge of creating linkages between clinical and community based services is demonstrated in many global health programs (i.e., PEPFAR programs, FP/HIV integration). There are several strategies to help mitigate or improve linkages, suggestions include, but are not limited to 1) develop metrics or indicators that measure linkages that IPs are expected to report on in their quarterly and annual reports (there are several large globally funded USAID and PEPFAR M&E partners that have developed metrics to address how to measure linkages between community and facility￾based services), 2) reduce the number of IPs to allow for more oversight and field visits from the USAID/Senegal project managers to verify that coordination is occurring at the decentralized level, and 3) design new follow-on procurements that are based on a continuum of care approach and request applicants to specifically address how their approaches will strengthen linkages and with other USAID/Senegal Health Programs. The final determination about the appropriate structure and mix of interventions would also be dependent on available resources. 12 Joseph de Graft-Johnson, Kate Kerber, Anne Tinker et al. Section II: The maternal, newborn, and child health continuum of care. Opportunities for Africa’s Newborns. The Partnership for Maternal, Newborn, and ChildHealth. 2006. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 27 Figure 8: Potential Design Structure for USAID/Senegal Targeted Technical Assistance (e.g. supply chain, innovative technology for data collection, and social marketing programs) Centralized HSS Integrated Package of Clinical Services Integrated Package of Community Services RBs Another option to consider would be to redesign the health strategy using a continuum of care approach, with a geographic focus and to develop one health strategy that could include all five of the current health components (HSS would be central level support). This could be awarded to several different partners that would implement the same program across each of the regions (i.e., north, central, and south), as seen in Figure 9 below. Some of the PEPFAR programs (primarily the former focus countries) implemented this type of strategy for clinical service delivery. Figure 9: Potential Geographic Structure for USAID/Senegal 1 January 2014 Presentation title North Central South C e n t r a l i z e d H S S Integrated Continuum of Care Approach (Integrated Clinical Services, Community-based Services, Health Promotion & BCC) Integrated Continuum of Care Approach (Integrated Clinical Services, Community-based Services, Health Promotion & BCC) Integrated Continuum of Care Approach (Integrated Clinical Services, Community-based Services, Health Promotion & BCC) Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 28 4.3 Principal Evaluation Question #2: How have interventions been coordinated and implemented in synergy across components, with other USAID/Senegal programs, and with other development partners? 4.3.1 Findings and Analysis While USAID/Senegal is the largest donor agency operating in the health sector in Senegal, other development partners are also contributing to advancements in health. The World Bank provides budget support to the GOS and is implementing a program for health financing, policy reform, and nutrition. The GFATM provides funding for malaria, HIV/AIDS, TB programs, and HSS. UN organizations including the United Nations Population Fund (UNFPA), United Nations Children’s Fund (UNICEF), and WHO provide technical support for immunization, RH, WASH, and MCH technical assistance and services. Other bilateral donors include Germany, Japan, France, Luxembourg, and Belgium. Several foundations are also active in providing health sector support, notably the Bill and Melinda Gates Foundation, which is funding an Urban Reproductive Health Initiative to support the expansion of FP in major urban areas and supports the Ministerial Leadership Initiative to build capacity at the MOH. Coordinated leadership of the donor committee helps spearhead policy dialogue and reform in sectors critical for Senegal’s development. A number of coordination groups exist among donor agencies. The G-50 coordination group is the largest and includes all donors present in Senegal. The group is co-chaired on a rotating basis and meets monthly to report on or raise issues of interest to all donors. USAID/Senegal also participates in the G-12, a working group of the twelve largest bilateral and multi-lateral donors that serves as the G-50’s Secretariat. The G-12 donor group also acts as an interface between the GOS and the donor community, supporting the transmission of joint messages to the GOS on matters relating to reforms, policies, as well as the Poverty Reduction Strategy. USAID/Senegal has a strong collaborative relationship with UNFPA in which they both lead policy dialogue in RH within the Donor Coordination Group and the new Reproductive, Maternal, Newborn, and Child Health (RMNCH) cluster group. USAID/Senegal with UNFPA leads the FP cluster which is under the thematic group on health led by WHO. Under the leadership of USAID/Senegal, UNFPA facilitated the joint implementation of the National Plan on FP including institutional diagnosis, coordination, and alignment of action plans. As a result of the institutional diagnosis and technical assistance, the Division of FP and Child Survival was upgraded from a Division to a Directorate in the MOH. Additionally, USAID/Senegal and UNFPA coordinate the platform for commodity management and security. They lead the biannual meeting for commodity forecasting and the monthly meetings of the sub-committee for RH commodity security. They facilitate an annual planning workshop to forecast drugs and contraceptive needs for Senegal, and share the costs for condoms and contraceptives. A Memorandum of Understanding (MOU) was signed for coordinated action to increase coverage and efficiency of community health. This cooperation led to the scaling up of innovations in FP such as the design and implementation of the M&E tools for injectable contraceptives. USAID/Senegal has a strong partnership with UNICEF in which they collaborate on the RMNCH cluster group. USAID/Senegal and UNICEF developed a consensus on their approach and are often able to work effectively as “one voice” on many issues directly with the MOH. In Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 29 interviews with UNICEF staff, appreciation was expressed for their inclusion in the USAID/Senegal program planning meetings and AWP development. ADEMAS and Abt are also collaborating directly with UNICEF on communication programs by providing assistance to the National Health Education and Information Service (SNEIPS) for the Development of the National CH Strategy and vaccination and Ebola campaigns. In some instances where the ChildFund consortium support is limited for health huts, UNICEF training has provided support for small equipment, medications and training materials. Under the USAID/Senegal Health Program, several IPs have actively engaged with development partners, however, there may be further opportunities for collaboration that could be explored. Team EY was restricted by time during this evaluation and was not able to conduct a donor landscape analysis and cross-walk the results against USAID/Senegal’s inventory of activities to identify additional opportunities for improved collaboration. The program did support the creation of a Technical and Financial Partners (TFP) Forum at the regional level. The forum contributed to information sharing and synergy in the implementation of interventions with other development partners. In addition, the program supported the MOH in the development of the integrated package of services. The support facilitated the mapping of health needs, setting priorities and indicators, and monitoring progress. This mapping is regularly updated and is used by the MOH to inform the donor community of funding gaps. The top four health priority areas where donor assistance is focused include: malaria, RH, nutrition, and HIV/AIDS. For partnerships that do exist, it helped IPs to enhance their activities, cost-shares, and coverage of their programs. Please note the list of examples below is not intended to be exhaustive.  Abt coordinates with The World Bank in the implementation and scale-up of PBF. Abt worked with The World Bank and the Luxembourg Corporation for the adoption of a single approach to Universal Coverage. It also supported The World Bank for the finalization of the Project for Health Financing and Nutrition. Abt worked with the Japan International Cooperation Agency (JICA) for capacity building in the areas of planning and service quality. Abt and FHI 360 collaborated with the Belgian Technical Cooperation Agency (BTC) in the design and implementation of the pilot programs for UHC.  WHO and all five components of the USAID/Senegal Health Program collaborated with the University Gason Berger de Saint-Louis for the use of the mapping software (Quantum Geographic Information System (GIS)) for the central and regional level of St. Louis. WHO and Abt supported the Department of Pediatrics at Cheikh Anta Diop University and the Unit of Training and Research of Thiès in introducing WHO Anthro software, which is used for growth monitoring.  USAID/Senegal collaborates with UNICEF to strengthen the quality of child health services and the support of the management of in-patient severe acute malnutrition centers.  Members of the FHI 360 consortium (ENDA and National Alliance for the Fight against AIDS (ANCS)) coordinated and received funding with the GFATM for cost-sharing to increase geographic coverage for MSM interventions with the Luxembourg Development Agency (LuxDev) for cost-sharing of cross border interventions. In addition, Sis Afrique receives support from the Netherland’s Ministry of Development Cooperation for its interventions which use mobile and internet technology to reach youth and MARPs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 30 Information and data on how interventions were coordinated and implemented in synergy across components and with other USAID/Senegal programs are primarily referenced in the component specific sections. For example, in Section 7.4.1, the report discusses how the ChildFund consortium developed linkages with several development partners mainly for activities concerning reproductive health and child survival. The USAID/Yaajeende agriculture and food security project is referenced as well as information on how the Peace Corps is linked to the CH component. It is important to note that Team EY was constrained by the time allotted to complete the data collection and analysis for the evaluation and therefore was limited in their ability to inventory each example of how synergy and coordination is occurring across all aspects of the health program. 4.4 Principal Evaluation Question #3: To what extent has DF to the three regions been implemented successfully and what could be improved? DF is a method of financing where funds are disbursed to the health regions to implement their activities directly. The introduction of DF in Senegal was designed to increase the impact of USAID/Senegal assistance, in line with the principles of USAID Forward and the Paris Declaration on Aid Effectiveness (2005). The goal of DF is to strengthen the management capacity of regional institutions to better align USAID/Senegal assistance with national priorities, reduce transaction costs, increase accountability and ownership, and promote sustainability. Team EY was asked to evaluate the effectiveness of USAID/Senegal’s DF to the regions as one of the key PE questions. Due to time constraints, the team was mindful to stay within this scope and did not evaluate other types of DF related to IPs, such as sub-grants. In Senegal, as of 2011, government spending represents about 60% to 70% of financing of health care expenditures. Government financing mechanisms are centralized and based on inputs and budgets prepared on a historical basis. They are often not harmonized with the strategic directions of the PNDS, particularly in terms of decentralization of health care services, empowerment of technical and operational support staff, and performance-based management.13 Moreover, national budget appropriations are managed at the central level with regions and districts have limited flexibility in the allocation and use of resources. To help mitigate these issues, USAID/Senegal’s integrated health program introduced DF with the following objectives:  Contribute to strengthening the decentralization of health services.  Improve the planning, budgeting, and implementation process of health interventions at all levels of the public health system.  Increase transparency and accountability in the health sector.  Strengthen the management capacity of medical regions, health districts, and local stakeholders. In its first year (2012), DF started in the three regions of Thiès, Kaolack, and Kolda and expanded in the second year to include Ziguinchor, Sédhiou, and Diourbel. Ultimately, it is intended that DF will be extended to the regions implementing the integrated package, as 13 “USAID/Senegal 2011-2016 Health Program, Inter-Agency Working Grouping Group on Financing,” USAID/Senegal, 2011. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 31 described in the original project design, however; the rate of enrollment of the regions will be at the discretion of USAID/Senegal. The Health Regions are the recipients of DF with contracts signed between each of the five IPs, on behalf of USAID/Senegal, and the Chief Medical Region Officer (CMRO). The CMRO then signs subsequent performance-based agreements with each district in the region. The regional office coordinator is the interface between the IP and medical region. The DF structure and system was designed to function as a decentralized mechanism to support more efficient and effective delivery of funding to the regions. The model was designed with six key operating principles: transparency, accountability, flexibility, efficiency, additivity, and predictability:  Transparency: DF shall facilitate information sharing among stakeholders to maximize its impact, avoid duplication of activities, and raise awareness.  Accountability: Validate that actors are responsible for their own results and commitments in reaching objectives in conformity with the guidelines and priorities identified by the MOH.  Flexibility: Clear management rules are to be implemented and adapted to address operational issues.  Efficiency: Reducing transaction costs associated with DF by streamlining and adapting procedures to the capacities of beneficiaries, aligning with objectives and systems of the MOH, and maximizing inputs at the operational level.  Additivity: Validate that the DF does not replace resources and internal financing mechanisms of the MOH and local government units with outside resources.  Predictability: Predictability so that that DF is based on a multi-year commitment to support capacity-building over several years and not through a “one shot” intervention. The total amount of financing for each region, as well as each IP’s proportion of payment for that region, is determined by USAID/Senegal as follows: fifty percent of a region’s DF budget is allocated based on the regional population and the other 50% is based on the number of health facilities in the region (e.g., hospitals, health districts, health centers, health posts, and health huts). This formula was determined so that DF resources are allocated to strengthen the capacities of the existing health services. The IP contributions are directly proportional to their overall project budget. Table 9 below provides a breakdown of each IP’s contribution to DF in year one and year two. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 32 Table 9: IP DF Contributions Total DF Contribution (in CFA francs) Abt IntraHealth ChildFund consortium FHI 360 ADEMAS 14 Year 1 (2012) 496,950,592 18% 25% 25% 15% 17% Year 2 (2013) 1,000,000,000 13% 27% 29% 14% 17% Prior to the signing of contracts, the IPs meet with regional stakeholders to develop AWPs and determine a set of activities and corresponding milestones against which the DF will be paid on a quarterly reimbursable schedule. Upon submission of payment requests, the Regional Verification Committee (RVC), led by the Regional Governor and comprised of key stakeholders from the regions as well as IP representatives, validates the level of completion of each milestone. Validation reports are sent to the three RBs, where payments are made on behalf of the IPs. The DF Joint Technical Monitoring Committee provides guidance and monitors implementation of the DF mechanism. Committee responsibilities include validation of activity areas eligible for DF, recommendation on the distribution of funds among regions, validation of the contract model for DF, and adoption of procedures manual. In addition, the committee monitors implementation of the DF mechanism in beneficiary regions based on the RVC’s reports, decides upon measures to be taken in disputes, and follows-up on audit recommendations. If a milestone is not achieved, the associated monetary value is deducted from the quarterly payment request. If a portion of that milestone is achieved, an associated percentage of the full value is paid. Figure 10 depicts the actors involved in the DF scheme and their associated roles. Figure 10: DF Mechanism 14 Each IP is requested to contribute a portion of their annual budget to support DF. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 33 4.4.1 Findings and Analysis Analysis of the DF mechanism is based on interviews with IP FMs, Regional Health Management Teams (RHMT)/District Health Management Teams (DHMT), and RB coordinators. In total, 31 interviews were conducted where DF was a central topic of discussion. In addition, Team EY reviewed and assessed documentation provided by Abt on the concept behind DF as well as annual and quarterly reports for critical challenges and lessons learned. Since DF is in its second year of implementation, an impact evaluation has not been conducted. In its assessment, Team EY was able to gather perceptions and examples of the strengths and weaknesses of the mechanism. According to the IP survey, four out of five IPs stated that DF contributed positively to the implementation of their component activities. When Team EY followed up with FM interviews, four of the five IP FMs confirmed their satisfaction with the model’s concept and design. This is in part because the IPs were consulted and contributed to the DF Concept Paper, which outlined processes, roles and responsibilities, activities, and monitoring for the model. The initial design accounted for the need to contribute to cross-cutting activities, primarily those that are not component-specific but impact the functionality and quality of all the components such as training, coordination, and supervision. The five specific areas that can be supported by DF include:  Area 1: Strengthening the capacities of medical regions, districts, and health committees to manage and supervise activities.  Area 2: Planning, monitoring, and evaluation.  Area 3: Coordination of interventions.  Area 4: Health promotion activities.  Area 5: Management system of medicines and essential commodities. DF is also well regarded at the regional and district coordination levels where local staff receive funds directly to implement programs impacting their region/district’s activities. It was noted in the region of Sédhiou that staff are better coordinated under DF than in regular activities budgeted by the MOH because clear processes were established for joint monitoring and supervision of activities being implemented in the same districts and sites. A number of benefits were highlighted by staff at the regional and district levels including:  Reduced dependence on the Senegalese government for financing.  AWPs guide the region’s activities and assess which were completed which, in turn, facilitates better distribution of tasks and prevent overlap of activities.  Milestones serve as a motivating factor for regional staff to complete work.  Formalized processes and tools to manage regional activities have helped staff to better identify roles and responsibilities and increase accountability. Abt conducted a number of trainings to facilitate capacity development in the medical regions and districts to properly manage DF funds. Thus far, 12 trainings on DF management were conducted that included six in Kaolack, three in Kolda, and three in Thiès. The trainings focused Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 34 on health governance and financial management. Interviews at the regional and district levels indicated that these trainings resulted in more credible financial management processes, enhanced role definition, increased knowledge for regional health management staff, better tracking of resources, and more targeted health spending. Interviewees noted some positive practices implemented in these trainings, to include:  In addition to the RHMT, training workshops were open to public sector, private health community representatives, and stakeholders including local authorities, health committees and CBOs.  Training modules were distributed, allowing participants to re-familiarize themselves with the skills learned once they are back in the field. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 35 Constraints and Challenges While DF received support in several areas, there were a number of constraints and challenges identified during the assessment. Greater detail is provided below in Table 10, however, the main challenges include: Table 10: DF Constraints/Challenges Constraints/Challenges Description Low level of ownership by some IPs, as a result of weak harmonization between activities At the conclusion of year one, significant issues were raised in the implementation of the funding. Lack of ownership from all IPs of the DF model is an issue that poses a problem of credibility in the regions. The Kolda Regional Coordinator noted that some IPs believe that DF is under the purview of Abt and, therefore, participate less in the process. One key issue that contributed to the lack of ownership was the insufficient alignment between activities financed under DF and the specific technical components. The ChildFund consortium, responsible for implementing community-based services, stated that they would like to see a greater proportion of funding to support community-based structures rather than primarily clinical services at the facility-level since health huts are not considered part of the formal health pyramid. This feedback was incorporated in year two and lengthy negotiations with the regions occurred prior to contract signing to include more milestones focused on community￾based activities. Delays in signing contracts and AWPs impeded implementation The process of required approvals by all IPs at the various DF stages (i.e., contract signing, AWP approval, and securing the quarterly milestone payment) contributed to significant implementation delays. Several regions reported that delays in approval of AWPs prevented regions from meeting their Q1 milestones. During this period, regions were utilizing their own funds to execute activities, unaware if those activities would be approved and subsequently reimbursed. This process makes planning of activities and absorption of DF funds very difficult for the regions. In addition, the delay meant that regions needed to execute the remainder of activities in a shorter time frame. Furthermore, some regions described having to forward their fund activities as a significant burden on their budget. As a result of these issues, it was decided that for 2015 DF, Abt will have sole responsibility for signing DF contracts, AWP approvals, and administering quarterly payments. Low absorption rates experienced by some regions and local regional and district medical structures lack the systems and/or capacity to properly manage decentralized funds Despite delays, a portion of DF was provided during this pilot project, as shown in Table 11. Kaolack was able to absorb the majority of funds allocated, while Kolda lagged behind at approximately 69% of funds executed from what was originally financed. The Kolda Regional Coordinator indicated that this was the result of a lack of motivation and commitment from the region. It was noted during data point/site interviews that MOH did not provide adequate guidance under DF and, as a result, the region did not feel compelled to deliver results. Another factor which may have impeded absorption is a lack of capacity in the regions, for it was noted that medical regions and health districts are lacking professional managers. Individuals serving as health officers come from a range of backgrounds and are often provided on the job training. They are also lacking administration and supervision support staff which is important given the level of details required to support the DF process. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 36 Constraints/Challenges Description Cumbersome monitoring processes overburdens local capacity and results in payment delays Significant issues were noted in the M&E Framework established for DF. The process of monitoring is too cumbersome and contributes to delays in payment and subsequently in milestone achievements. In order to verify milestones, the RVC must go to each health facility in the region where activities are occurring and generate a report. This results in more than 100 reports generated on a verification mission. This process can take several months, thereby delaying payment. Regional coordinators note that these reports are not read by the IPs before payment, instead IPs rely on the succinct reports generated by the RVC which include minimal detail on activities conducted. Although verification is meant to occur on a quarterly basis, since the process is so time consuming, the practice of milestone verification is continual throughout the year. Insufficient feedback loop communicating the DF impact and results to the IPs There is a perception that there is no consistent feedback that informs IPs as to how the DF funding was disbursed and what was achieved. Three of five FMs interviewed noted a desire for more transparency on how the funding was expended, what results are being achieved, and the impact that DF is having. Abt conducted a review and assessment of the DF mechanism and its implementation at the end of the first year, as well as generated quarterly reports on financing focused on contract monitoring, milestone achievement, status of payments, and level of absorption. Various interviews found that not all IPs are aware of these reports or received copies. Furthermore, the concept paper references that an audit will be delivered by the DF Joint Technical Committee annually for review. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 37 As noted, Table 11 below shows the absorption rates for the first year of DF in the regions of Kaolack, Thiès, and Kolda. Table 11: DF Absorption Rates for Year 1 Region Amount CFA francs Kaolack Total Contracted 146,000,000 Total Executed 138,700,000 Percentage Absorbed 95.00% Thiès Total Contracted 233,195,980 Total Executed 171,869,073 Percentage Absorbed 73.70% Kolda Total Contracted 117,754,612 Total Executed 81,157,979 Percentage Absorbed 68.92% 4.4.2 Lessons Learned and Recommendations One of the key PE questions was focused on the integration and coordination of DF implementation. Significant lessons were learned in the first two years of the project, with one of the critical lessons being that the mechanism designed was too complex to facilitate timely disbursement of funds. Complexity around the IP’s approval process, as well as milestone monitoring and verification, were cited as major burdens in implementation. The course-correction of the mechanism to be coordinated by Abt on behalf of all IPs should not only simplify implementation but also reduce delays in approval and disbursement of funds to the regions. However, a number of areas exist to further improve implementation and streamline monitoring procedures. Recommendations include: Implementation  USAID/Senegal may want to consider adding a budgetary line-item specifically for DF in subsequent cooperative agreements with IPs. Although the expectation for IPs to contribute to DF was clearly stated in cooperative agreements, this would address the perspective voiced by some IPs that DF is detracting from their component activities, and increase support for and ownership over DF.  USAID/Senegal may want to request that each region hire professionals with management experience to be dedicated to the DF activities. It is recommended that the regions validate that the manager is provided with enough support staff to properly administer and supervise the program. The cost of these individuals could be paid for by funds under the DF mechanism.  USAID/Senegal may want to engage the MOH more actively to support and promote DF in the regions. This may assist with regional adoption and utilization of DF.  USAID/Senegal may want to consider supporting an information campaign on the benefits of DF in the regions where the absorption rate is low. Although awareness of DF exists, the benefits are not clear to all stakeholders in the regions. This may help increase regional understanding of DF and its objectives, goals, and benefits. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 38  USAID/Senegal may want to consider setting a deadline for IP approval of AWPs (i.e., such as two weeks after receipt) in order to avoid derailing regional activities and require subsequent contract modifications.  USAID/Senegal may want to provide Q1 financing upfront so that regions can commence AWP activities without needing to utilize their own funding. The Q1 milestone monitoring can then verify that activities were conducted as planned. Funds already disbursed for milestones not achieved in Q1 can be withheld from the Q2 tranche. Performance measures for process cycle time and the definition of internal controls and regular testing should be considered. M&E  USAID/Senegal may want to simplify the M&E process to reduce cumbersome and unnecessary reporting. One option to consider is involving regional and district DF management teams in gathering reports from health facilities to reduce burdens on the RVC. Another option to consider is the use of a mobile application for health facilities to send receipt/activity verification and reduce time spent in the field gathering data.  USAID/Senegal may want to develop a new process in which all IP applicable reports developed are shared on a quarterly or annual basis with all IPs to promote transparency and communicate relative to the impact that the program is having and the results achieved. If the new process is implemented and it is found that the reports are not read by the IPs, USAID/Senegal may want to consider coordinating biannual meetings (led by Abt) with all IPs to share findings and discuss issues from reports and implementation experiences, in order to enhance communication. An alternative solution would be to utilize quarterly RB coordination meetings to share DF results.  USAID/Senegal may want to request that Abt conduct the audit (required annually) as soon as possible so that the verification activities spearheaded by the RVC are having the desired impact. 4.5 Principal Evaluation Question #4: To what extent has the system of RBs and integrated work plans improved coordination among the five components? 4.5.1 Findings and Analysis The USAID/Senegal Health Program utilized three RBs in Thiès, Kaolack, and Kolda in their decentralized approach to facilitate better coordination and disbursement of funds. Through the use of those hubs, IPs disbursed funds to the regions and districts for activities in a more efficient and systematic manner. The funding flow starts from their US-based headquarters through the Senegal national office in Dakar and then out to the RBs, where funds are disbursed both for the component￾specific activities, as well as the DF initiatives. Figure 11 depicts the flow of funds. The RBs house a Regional Coordinator, as well as an Administrative and Financial Manager (AFM), for joint use by all components, hired through and financed by Abt. Other IPs finance the hiring of assistants and drivers in each of the RBs, also at the disposal of all components for their use. In addition to the shared staff, each IP also has a regional advisor that sits in the office. Under this structure, Abt is responsible for the coordination functions of the RBs. Team EY assessed the RBs effectiveness in terms of their financing as well as their coordinating functions. As discussed in detail below, interviews with IPs suggest that the use of RBs for more timely distribution of funds for field activities has been successful. However, IPs identify less benefit in relation to the RB’s coordinating function, citing that the coordination does little to enhance their activities or integrate component activities into a more cohesive program. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 39 RB Financing Function The IPs, with the exception of the ChildFund consortium, utilize the decentralized RBs as their hub for financing and coordinating activities in the regions, with designated bank accounts in those regions. The ChildFund consortium cites the nature of the CH activities, and its need to be close to partners working at the grass-roots level, as the reason it needs an office in each region and therefore does not utilize the RB decentralized infrastructure. Although the ChildFund consortium contributes to the financing of the RBs operating expenses, it established a separate arrangement with its consortium partners for financing component activities, thereby bypassing the decentralized bureaus. As seen in Figure 11, a member of the ChildFund consortium maintains an office in each of the 14 regions through which funds are disbursed. According to an interview with the ChildFund consortium’s FM, these offices do not coordinate with the RBs. Figure 11: Flow of Funds While new to USAID/Senegal’s Health Program and the IPs, the process of a decentralized financing structure is received well overall from a funding perspective. Of the five IPs that responded to the IP survey, 60% agreed or strongly agreed that decentralization is helping them to achieve their component goals. At the beginning of the program, Abt and the IPs met to discuss how the RBs would function and, in conjunction with other IPs, developed a procedures manual to outline the roles and responsibilities of the bureaus. The manual describes in detail procedures for financial management, human resources, and logistics and office equipment. It loosely describes the relationship between the Regional Coordinator and the IPs technical advisors in each bureau. Interviews noted a level of frustration among staff in the bureaus which was attributed to a lack of clarity around the lines of authority and communication between the Regional Coordinator and IP’s technical advisors. It is not clear to whom the technical advisors are accountable, the Regional Coordinator or their Chief of Party (COP). This sometimes results in conflict or confusion. Additionally, Team EY did not observe clearly defined mission statements in each of the RBs. Office expenses for the RBs are shared by each of the five IPs. At the end of each month, the AFM consolidates the office expenses and submits the bill for their portion of the operating budget to each component FM in Dakar. It was noted by the FMs that this is considered a time consuming Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 40 process from the perspective of the FM, who drafts and sends invoices. It also often takes time to receive the funds from each of the partners. The RB in Kolda noted that their office expenses are more expensive due to the remote nature of the region. Abt hired three drivers for their 11 staff members and the drivers are overextended. Additionally, the region lacks individuals with the required technical skills needed to effectively support the office, necessitating that they outsource some services also adding to increased costs. The RB Coordinator does not believe the office has adequate funds to function effectively. There was consensus among the IPs that some of the RBs provide good coverage of the country, facilitating quicker disbursement and access to the regions as well as the USAID/Senegal health program activities. Thiès has one of the largest regional populations in Senegal and is in close proximity to Dakar. Kaolack is in the central region of Senegal and Kolda and supports the southern regions. Decentralization is effective at helping finances to reach the field faster and is recognized as a positive enhancement to their activities by most IPs. In addition, having a local presence helps them to more closely monitor activities and correct issues as they arise. The AFM is utilized by the IPs to transfer financing and validate that the funds are used for their intended purposes and according to budget. The transfer of funds from IPs’ Dakar offices to the regional offices is simplified by the use of the same bank. The IPs’ FMs are in close contact with those in the regions. The AFM reports back to the IP FM located in their Dakar offices on a regular basis. The IPs provide the AFMs with specific reporting requirements and documentation that they submit on a monthly basis to adequately track components’ expenditures. Upon submission of monthly budgets and AWPs, the Dakar offices approve payments and funds are transferred for the implementation of planned activities. Transfers can be completed within a day and checks signed by AFMs to support respective field activities. RB Coordination Function At the RBs, Abt coordinates the development of integrated work plans with all IPs. These integrated work plans are in addition to the individual work plans that IPs are required to submit to USAID/Senegal annually as a deliverable. Interviews found that many of the IPs consider the integrated work plans to be a secondary priority to meeting their own work plan deliverables and implementing their specific activities. This placed additional workload on Abt to coordinate and organize time to complete integrated work plans, which are sizable in content and nature. In addition, multiple interviewees indicated that the integrated work plans are not being used as a real tool for coordination and synergy across IPs. Interviewees noted that the RB’s coordination is centered mostly on logistics and daily activities and does not focus enough on technical exchanges among IP advisors to discuss the implementation of their activities and potential for collaboration. This was due mainly to the time being directed toward the operations and logistics of the RBs, and less toward resolving technical challenges IPs are managing. The RBs schedule quarterly meetings where such technical exchanges could occur, as these meetings include participation from the regions and COPs are invited to attend. However, interviews with COPs indicated that they felt that RBs devoted insufficient time to organizing and structuring quarterly meetings to focus on technical matters, limiting the value of these meetings. COPs and field staff also noted that they are often not all able to attend meetings due to scheduling conflicts. The system of coordination could be enhanced through electronic scheduling of meetings. In addition, lack of communication between the IP COPs in Dakar and the RBs has created two parallel systems of coordination. Team EY understands that the COPs do not invite the Regional Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 41 Coordinators to participate in their meetings in Dakar either in person or virtually (i.e., teleconference, Skype). As a result, there is limited communications between the HQ level and the RBs. The evaluation team was not provided with an explanation as to why the COPs did not include the coordinators in their HQ based meetings. Additionally, sharing best practices and information between the three RBs was also limited. Overall, IPs expressed that the RB Coordinator, as well as the AFM hired by Abt, were of good quality and generally worked well on behalf of all five components. One IP expressed concern that the coordinator and AFM were primarily focused on meeting the priorities set by Abt and did not serve all component partners equally. Several interviewees noted that the shared staff are overextended, which resulted in delays in delivering financial management and other central services to the IPs. IPs report that there is sometimes slow collection and reporting to the Dakar office for receipts of activity expenditures. In some cases, the office assistants assist the AFM when the workload becomes overwhelming. Some of the remote regions in Senegal, including Matam or Saint-Louis, are not easily reached from the RBs, thereby limiting regular site visits to those areas and causing delays in the verification of financial results. IPs vocalized that in the future USAID/Senegal should consider adding additional RBs, to increase coverage of the country, particularly among those regions with considerable distance from the current hubs and are thereby not easily accessed. Suggested regions included Matam, Kédougou, Tambacounda, and Saint-Louis. 4.5.2 Lessons Learned and Recommendations Based on the findings and analysis above, Team EY identified key ways in which the process of decentralization through the regional offices could be more streamlined in the future: • USAID/Senegal may want to increase the number of qualified staff placed in the RBs to better accommodate requests for field monitoring of the IPs. This does not necessarily require hiring additional FMs, but could be accomplished through hiring assistant level staff specifically to support the documentation of financial transactions and gathering of supporting paperwork. • USAID/Senegal may want to work with Abt to define a clear vision and mission statements for the RBs to support IPs to better operate within the system and improve their functionality. In addition, USAID/Senegal may want to encourage Abt to review the procedure manuals with all IPs so that job descriptions are clearly defined and lines of authority and responsibilities for the coordinator and IP’s technical advisors are known. • USAID/Senegal may want to enhance communication between RBs and Dakar-based COPs to eliminate a dual-coordination system and better integrate the IP activities at the regional level. • USAID/Senegal may want to consider the use of a mobile application to help with the collection of receipts/verification documentation from more remote regions. • USAID/Senegal may want to consider increasing the number of RBs to better reach remote regions including Matam, Saint-Louis, or Kédougou as cited by interviewees. USAID/Senegal may want to also consider taking advantage of pre-existing IP field staff in regions to set up ‘mini RBs’ or request co-location to facilitate coordination between regions. For example, field staff from IntraHealth, the ChildFund consortium, and ADEMAS are all located in the region of Kaffrine and IntraHealth has field staff located in Tambacounda. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 42 • USAID/Senegal may want to consider re-allocating the office operating costs under one component and IP to reduce administrative burden and time spent by the AFM in recovering office expenses. • USAID/Senegal may want to utilize the experiences of the three existing RBs to estimate office operating costs, particularly in remote regions where past experience has shown that costs are higher. 4.6 Principal Evaluation Question #5: To what extent has the program strengthened government ownership and demonstrated sustainability? 4.6.1 Findings and Analysis The high dependence on foreign aid, particularly in health is a concern for the sustainability of interventions. For example, according to the CNLS, 80% of the total HIV/AIDS budget is funded through external resources. The National Strategic HIV/AIDS Plan 2014 - 2017 notes a risk of decreased external funding related to the low burden of HIV in Senegal, which ranks the country as one of the low priority countries by the GFATM. Without continual prevention interventions, there is a risk that the HIV prevalence will increase. This is only one example, but applies to other aspects of the health sector. To enhance the likelihood of sustainability, the USAID/Senegal Health Program continues to foster country ownership of, and investment in, proven approaches and interventions, particularly in health service delivery at the community level. Sustainability depends on the ownership of processes, skills and the quality of human resource with the necessary funding to sustain implementation over time. Ownership At the community level, beneficiaries such as health huts service providers, clients, CBOs, volunteer outreach workers, and the CH management committees appreciated the participatory approaches adopted by USAID/Senegal’s Health Program IPs. For example, FHI 360 and the ChildFund consortium operated in a manner that takes into account the beneficiaries’ perspectives (situational analysis, needs assessments, strategies). Team EY observed instances where communities took responsibility for providing support to CHWs and Trained Traditional Birth Attendants (TTBA) for the maintenance of health huts. For example, a community would build a new health hut and provide for a monthly allowance for the CHW and TTBAs. However, despite the participatory processes meant to stimulate participation and increase ownership, it was expressed by communities, CBOs, volunteers and members of committees that sustainability was a concern. For example, the community in Kédougou was reluctant to financially support CHW and TTBA, which led the CHW to leave the health hut to seek employment at a mining site. Skills Development Many people benefited from trainings in different areas (e.g., health management, health financing, good clinical practices, CH, strategic planning, design of health education messages, health promotion). As a result, linkages, referrals, and counter referral systems were organized and as a result, the coverage of and access to quality services and care improved. Such capacity and skills building were supported by the provision of institutional support (e.g., office equipment, mobile clinics, laboratory equipment). The HSS approach to cross-cutting issues introduced new processes and systems to improve coordination meetings, planning exercises, supervision, M&E, and reporting. Support was provided for both technical and financial capacity building. All stakeholders Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 43 from the central level (MOH) to local level (e.g., governors, préfets, RHMT, DHMTs) confirm the relevance and effectiveness of the support, especially when followed by funding for implementation and monitoring. Human Resource Availability While ownership appears to be strengthened, activities could not be sustained without funding. General issues in the health sector included the lack of planning and control over staff mobility, which enhances risks for any project. The southern regions are not considered as attractive to civil servants due to the enduring insecurity. Additional challenges encountered in retaining civil servants include remoteness and difficult working conditions. As noted by the governor in Kédougou, the turnover of civil servants is on average two to three years. The GOS is aware of its commitments and the General Secretary of the MOH explained the need for international partners to work together to phase out assistance programs in a manner that the GOS can progressively assume responsibility for financing and supporting. Further discussion on sustainability and ownership in relation to individual component activities is provided throughout the report. Further discussion on the sustainability of mutuelles and PBF is provided in Section 5.0. In addition, sustainability relative to community-health activities, such as the issue of lack of funding by the MOH to maintain health hut services, is discussed in Section 7.0. This section also includes information on the over-reliance on volunteers and the ChildFund consortium at the community-level. While an analysis of health hut graduation is not provided, there is discussion on varying levels of functionality and resources available for health huts. Throughout the Final Evaluation Report, references are made to capacity building of CBOs. However, Team EY did not have the opportunity to go into depth with each sub-component on CBOs, as this was not a stated priority in the PE questions. Finally, human resource issues such as capacity and lack of qualified staff are discussed at length in several of the component sections. 4.6.2 Lessons Learned and Recommendations • Human resources are a major constraint throughout the health system. To improve the MOH’s capacity to deliver services and strengthen systems, service providers and CHWs require additional support. USAID/Senegal may consider requesting Abt to develop recruitment, deployment, and retention strategies with medical and nursing institutions to increase enrollment and then subsequent deployment of health personnel to regions where there are acute shortages. 5.0 COMPONENT #1: HEALTH SYSTEM STRENGTHENING 5.1 Background USAID/Senegal awarded Abt a five-year $22M United States Dollars (USD) cooperative agreement in 2011 to implement the HSS component of the USAID/Senegal Health Sector Strategy. The HSS component contributes to the achievement of IR 3: Improved performance of the health system. The project includes interventions that facilitate effective and efficient policies, planning, and budgeting at the central level of the MOH, and improved management of district and regional health teams. Activities carried out emphasize capacity development and targeted assistance related to health governance and the overall functioning and performance of the health system. Abt is implementing activities in ten regions: Diourbel, Fatick, Kaffrine, Kaolack, Kolda, Louga, Sédhiou, Thiès, Ziguinchor, and Dakar (Departments of Pikine and Ruisque only). Abt’s primary sub-partners as Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 44 outlined in their original cooperative agreement are Innovations Group and Health Systems in Africa (ISSA), Association Council for Action (ACA), Centre de Recherche pour le Développement Humain (CRDH), FHI 360, PATH, and Broad Branch Associates. Abt leads the implementation of the following four sub-components in collaboration with partners:  Sub-component A: Improving management and system performance at regional and district levels.  Sub-component B: Expanding alternative financing mechanisms and improving their sustainability.  Sub-component C: Ensuring that national level policies and systems support improved performance of the health system throughout Senegal.  Sub-component D: Coordinating USAID/Senegal Health Program components to validate that they work effectively together to support improved performance of the health system. The HSS component focuses on operational interventions at various levels of the health system. At the decentralized level (regional and district levels), the assistance provided by Abt is primarily technical assistance with implementation support for piloting new financial approaches such as PBF. Additional objectives included in Abt’s portfolio are strengthening the capacity and sustainability of community-based health insurance models, and technical assistance to improve the performance and functionality of the DHMTs and RHMTs. Abt is also responsible for planning and coordinating and integrated approach with the four other USAID/Senegal IPs. As mandated in their contracts, all IPs contribute a portion of their funds to DF of the regions, with Abt acting in a coordinating role. Team EY recently learned that, in the following year, Abt’s cooperative agreement will be modified and additional financing added for DF so that all funds for this mechanism will be executed through a single partner, Abt. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 45 Table 12 below shows the standard indicators related to the HSS component in the first two years compared with targets. Although Team EY understands that Abt provided its year three annual report to USAID/Senegal in November 2014, Team EY did not receive the report in time to include the 2014 indicator data into the table below. However, information derived from the annual report that was verified by data point/site interviews and corroborated by alternative sources was utilized in our analysis. Table 12: USAID/Senegal Indicators for HSS # Indicator Target Actual Target Actual 2012 2013 1 Proportion of health districts where the functions of the district medical officer and those of the chief doctor at the health center are separated * 19% 25% 31% 2 Proportion of Service Delivery Points (SDP) that have displayed drug prices and tariffs for services 95% 55% 95% 64% 3 Proportion of health districts with a technical implementation of AWPs ≥ 80% 100% * 100% * 4 Number of medical regions that have organized a high quality Joint Annual Portfolio Review (JPR) 100% 100% 100% 100% 5 Number of audit reports delivered on time (PBF-related) 100% 0% 100% 0% 6 Proportion of reimbursement requests paid on time (PBF-related) 100% 0% 100% 0% 7 Number of health districts involved in PBF 3 3 7 7 8 Number of MHOs that received public subsidies following the establishment of mechanisms by the government * * 50 * 9 Number of beneficiaries covered by community-based MHOs * 263,343 330,000 337,872 10 Number of vulnerable persons covered through MHOs with the support of a third￾party payer * 22,438 33,000 31,876 11 Number of policy documents approved and regulatory acts adopted for the implementation of policy initiatives developed by the Health Sector Policy Initiatives Team (EIPS) ≥ 1 13 ≥ 1 2 12 Health sector budget as a percentage of the national budget 8% 10.2% 15% 10.7% 13 Deadline for development of the performance report of the health sector Medium Term Expenditure Framework (MTEF) for year n-1 is met (May) Yes Yes Yes Yes 14 Amount allocated (in CFA francs) to districts and medical regions through the DF mechanism * * * 519,400, 000 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 46 15 Amount allocated (in CFA francs) to districts, medical regions and Public Health Establishments (EPS) through the PBF mechanism for the payment of bonuses 96,312,310 * 260,269,000 23,564,521 16 Proportion of progress reports of the component prepared within the required time￾limit 100% 100% 100% 100% *Data not available Note: Target and actual data included in the table above is derived directly from annual reports provided by IPs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 47 The data from the 16 indicators above are derived from Abt annual reports. These indicators are closely aligned with USAID/Senegal’s IR 3: Improved performance of the health system and support USAID/Senegal’s high-level indicators, the percentage increase in GOS contribution to health spending and percentage of facilities with stock-outs of essential drugs. The indicators in the table above represent Abt’s core activities in HSS, from improving management at the regional and district levels, expanding alternative financing mechanisms such as PBF, and supporting national policy and health systems development and functioning. The indicators also address the critical PE questions of whether the program has strengthened government ownership and demonstrated sustainability. Additional information on indicator achievement status can be referenced in the Figure 12 below. ► Targets Met: According to the 2012-2013 annual report, success in this area can be attributed to the development of tools and technical assistance for capacity building of MHOs and within the MOH. In addition, select indicators show that notable achievements were made in government ownership, with the health sector budget as a percentage of the national budget as shown in the indicator table. Policy documents and regulatory acts adopted by the EIPS also surpassed the numbers planned. Successes in this area are attributed to capacity building for policy reform and implementation of health reforms by Abt. The central and regional services of the MOH are being restructured and governance and universal health coverage was included as a priority on the political agenda, according to the 2012-2013 annual report. In total, Abt nearly met, met, or exceeded targets for 10 indicators in year two including: 1. Proportion of health districts where the functions of the district medical officer and those of the chief doctor at the health center are separated. 2. Proportion of SDPs that have displayed drug prices and tariffs for services. 3. Number of medical regions that have organized a high quality JPR. 4. Number of health districts involved in PBF. 5. Number of beneficiaries covered by community-based MHOs. 6. Number of vulnerable persons covered through MHOs with the support of a third-party payer. 7. Number of policy documents approved and regulatory acts adopted for the implementation of policy initiatives developed by the EIPS. 8. Health sector budget as a percentage of the national budget. 9. Deadline for development of the performance report of the health sector MTEF for year n-1 is met (May). 10. Proportion of progress reports of the component prepared within the required time-limit. ► Targets Not Met: While indicators show that PBF-related activities have met targets for the establishment of programs in the regions, they also suggest that challenges around the implementation of this mechanism exist. The amount allocated (in CFA francs) to districts, regions, and EPS through the PBF mechanism for the payment of bonuses is lagging as compared to the target. In addition, the following indicators reported low performance against targets: Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 48 1. Number of audit reports delivered on time (PBF-related). 2. Proportion of reimbursement requests paid on time (PBF-related). 3. Amount allocated (in CFA francs) to districts, medical regions and EPS through the PBF mechanism for the payment of bonuses. Challenges identified in meeting the targets include lack of availability of drugs and difficulty with procurement procedures as well as low capacity in planning, management and monitoring at the operational level. These constraints can inhibit leadership in carrying out the strategic directions of the PNDS on health governance and effective oversight to attain desired results. ► Data Not Received: A number of actual results were unavailable or unreported in Abt’s year two annual reports because they access the data from the GOS public finance management system. Three indicators had particularly large data gaps: 1. Proportion of health districts with a technical implementation of AWPs ≥ 80% 2. Number of MHOs that received public subsidies following the establishment of mechanisms by the government. 3. Amount allocated (in CFA francs) to districts and medical regions through the DF mechanism. The absence of this data may be explained by Abt’s limited influence at the central government level to gather accurate data on resource allocation during the well documented data collection strike. Team EY received no additional information from the AWPs, annual reports or interviews to substantiate the reasons for missing standard indicators. Lastly, Team EY noted the absence of indicators that track progress on implementation of sub-component D: Coordination of USAID/Senegal’s Health Program components to validate that they work effectively together to support improved performance of the health system. This gap is critically important for consideration because it represents one of Abt’s primary functions, to work collaboratively and effectively with other components to support improved performance of the health system. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 49 Figure 12: HSS Indicator Status 5.2 Key Findings and Recommendations/Benefits The HSS component is achieving its objectives and meeting the majority of agreed-upon indicators as stated in their contract. Table 13 below summarizes the most important key findings and lessons learned/recommendations relevant to strengthening the overall technical assistance of the HSS component. Table 13: HSS Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefits 1 Feedback received on PBF was positive from health providers, who believed it was motivating and changing the way they delivered care. It was noted that PBF is contributing to positive behavioral change in the region, emphasizing an increase in service quality and ownership over health services. The success of the PBF pilot provides a basis for the MOH and other donors to financially support PBF scale-up. However, in the majority of interviews with MOH and Abt staff, the general consensus was that the process of data collection and indicator verification from start to finish was cumbersome. Recommendation: USAID/Senegal may want to consider using mobile technology for PBF data collection, verification, and bonus payments to increase the indicator validation process and the speed at which incentives are received by service providers. Benefit: More efficient and streamlined PBF processes can improve decentralized services and reduce the burden on regional and district health teams. 2 September 2014 marked the one year anniversary of the launch of UHC by the GOS. The MHO’s coverage to the informal and rural sector remains the strategic priority for progressing toward wider health coverage. Implementation of the Recommendation: USAID/Senegal may want to consider recommending that the MOH implement programs aimed at strengthening the delivery of health care services with an emphasis on the recruitment of qualified health care personnel and procurement of sufficient equipment for health Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 50 # Key Findings Recommendations/Benefits various UHC components will likely result in a considerable increase in the use of health services in a context where there is a lack of service providers. facilities. Additionally, USAID/Senegal may want to develop creative strategies to engage the private sector (including assessing PPP opportunities) and support employment and job creation for beneficiaries. Benefit: These recommendations may help address the growing demands placed on the health workforce and facilities as more of the population benefits from UHC over the next decade. 3 It appears the GOS’s financial contributions and donor resources are insufficient to achieve the ambitious health objectives. With the goal of the GOS to improve efficiency and transparency in budget allocation there is an opportunity to better understand where current resources are being expended and what gaps remain. Recommendation: USAID/Senegal may want to consider providing technical assistance to the MOH to implement a detailed financial resource gap assessment supported by the development of a strategy to mobilize resources within the public and private sectors. USAID/Senegal may also want to consider developing a process to validate that the strategy includes sustainability measures, such as developing reliable revenue streams to decrease the level of dependence on donor resources. Benefit: This approach would allow the GOS’s financial contributions and donor resources for health to be more efficiently allocated and achieve the health objectives, while beginning the process of developing the GOS financial streams to move toward sustainability. 4 A number of actual results were unavailable or unreported in Abt’s year one and year two annual reports. This is because Abt accesses the data from the GOS public finance management systems which were greatly impacted due to data collection strikes which affected the entire country of Senegal, from the GOS to donor partners. The data collection systems are just beginning to recover and there are gaps in data collected and the quality of the data available. Abt’s responsibility was not to collect the data for each indicator, but to collate and verify the information received from the MOH, whose responsibility it is to collect the data. This is because these are national level indicators not USAID specific indicators. Abt noted in their annual report Recommendation: Given that the entire data collection and management system was damaged by the prolonged national strike, Team EY recommends that USAID/Senegal may want to perform a data quality and controls assessment of the current indicators to make sure they are back on track and being collected correctly and consistently￾especially those indicators related to public financial expenditure. Benefit: A data quality and controls assessment of the current indicators, especially those related to public finance and transparency could help restore the GOS’s promise to its citizens for improved visibility on tracking public spending and increasing their confidence in the government that funding is going to serve the needs of the people. The assessment may also provide insight into key control gaps related to collection, data transmission, reconciliation, and monitoring regarding financial Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 51 # Key Findings Recommendations/Benefits that despite their efforts, they did not have enough influence over the MOH leaders to gather the data needed to respond to the indicators and, therefore, believed it was best to leave them blank. expenditures. 5.3 Sub-component A: Improving management and system performance at regional and district levels 5.3.1 Findings and Analysis Sub-component A made progress in strengthening the management of medical regions and health district through capacity building training, DF support, and PBF activities. DF is a shared mechanism, mandated by USAID/Senegal with procedures developed by the five IPs collectively, to improve the management and coordination of USAID/Senegal’s Health Program in six regions. The main objective of the DF mechanism is to contribute to adapting the way USAID/Senegal delivers its assistance to strengthen the decentralization of health services and performance-based management in the regions. Even though this component contributes to the coordination of DF, it was decided to include information on DF in Section 4.4 because the PE question related to the overall effectiveness of the approach. PBF The PBF is an incentive tool to accelerate progress toward achieving health-related results. Senegal piloted PBF indicators in year one in the regions of Louga, Kaffrine, and Kolda. Thirty￾eight performance contracts were signed in Kaffrine and Kolda. However, strikes by health workers, which withheld health data, prevented signing of contracts in Louga. In year two, PBF was expanded to 77 health facilities in five additional health districts: Médina Yoro Foulah, Vélingara, Birkelane, Koungheul, and Malem. Strikes have ended and the MOH is planning to expand PBF to four additional regions; Ziguinchor, Sédhiou, Tambacounda and Kédougou, in conjunction with The World Bank. The pilot project is under the authority of the MOH and contracts are signed annually between the MOH and the health facilities (e.g., posts, centers, and hospitals). Approximately 123 PBF performance contracts were signed for a total estimated amount of 322,272,000 CFA francs ($608,431 USD) in 2014. In each PBF region, an Abt coordinator works with a national PBF counsellor who is responsible for supporting the program’s implementation. Abt provides technical assistance to the MOH in its implementation of the program and supports MOH trainings in the regions where the PBF pilots are being executed. The program was designed by Abt in conjunction with other IPs and is largely based on the Rwanda experience, which is a noted best practice case in the literature review section. The PBF is a positive example of collaboration between USAID/Senegal and other donor agencies as well as with the GOS. The World Bank as well as the BTC, is working with USAID to support the MOH in this initiative. USAID/Senegal and The World Bank initiated discussions to determine a co-financing joint agreement in the future, with The World Bank leading this Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 52 initiative. Additionally, feedback received on PBF was positive from health providers, who believed it was motivating and changing the way they delivered care. The Regional PBF Coordinator in Kolda noted a behavioral change in the region, citing a greater focus among workers on how they analyzed quality and coverage of health care services. In addition, greater ownership over health services was noted along with a sense of connection and control over the outcome of health services provided because health care workers see a direct benefit when milestones are achieved. Bonus payments can contribute to a health worker’s monthly salary, as shown in Figure 13 below. A potential 20% bonus, even one totaling as low as $11 USD, is a meaningful sum of money, relative to a low monthly salary. If objectives are met, 75% of the health facility’s bonus is shared between health staff while the other 25% is invested back into the facility to cover operating expenses. For health posts, this portion is divided between the health post and its associated health huts. Although the concept paper for PBF mandates this, there is no formalized process to provide funding to health huts and it is done sporadically. Distribution among staff will be conducted at the level of the health facility on the basis of consensual criteria such as basic salary level, number of days present at place of work, unit where staff is assigned, etc. Figure 13 below shows an example of a distribution method that was agreed upon by a health facility in the region of Kolda. Due to time constraints, Team EY was not able to further investigate how bonus payments were actually used by staff and facilities. Figure 13: PBF bonus scale implemented in Kolda However, in the majority of interviews with MOH and Abt staff, the general consensus was that the process from start to finish was cumbersome. Part of the reason why the process is considered “heavy” is due to required steps in the cycle, which were not always completed on time, leading to delays in the indicator related payments or bonuses. For example, a district may have 20 health posts of which seven of which would be checked at random (i.e., validators typically verify 30% of activities). The PBF indicators are verified to cross-validate that the reported records are consistent with actual activities/services, as well as CBO activities with Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 53 reference to the health hut records. The next step is a household survey to cross-check if the beneficiaries actually exist and received the recorded services. There could be over 100 beneficiaries located in multiple villages across the district. CBOs are utilized to conduct these household surveys, a good practice drawn from the Rwanda experience. Once these verifications are complete, a check list is sent back to the Regional Management Committee (RMC). Following receipt of verification, the RMC then sends a request for payment to the MOH. Theoretically, the complete process is scheduled to take one week, when in reality it often takes over a month. As a result of this burdensome and time-consuming verification process, significant delays were noted in the payment of bonuses. Table 14 below contains the health services and health management indicators. Currently no PBF indicators were discovered that focused on community-based services or data collection at the community-level such as commodity-tracking. Additionally, there were no indicators that focused on collecting data directly on maternal mortality or neonatal mortality. Team EY believes it would be important to include indicators that capture data and incentivize heath providers in the PBF model. During the PBF pilot evaluation, it was found that the late verification process attributed to the delayed payment of bonuses, which was in turn due to delays in the selection of the audit firm and CBOs. There were also delays in the signing of the order establishing the national PBF program, which resulted in delays in contract signings at the health facility level. While contracts are intended to be signed in January for one calendar year, it was noted that in many cases contracts were not signed until June. Due to the delays any contract that was signed in June was considered retroactive and the indicators were observed and bonuses were paid for the first six months. This delay caused significant confusion and frustration among workers in the region. Not only were bonus payments delayed, but staff did not have a clear understanding of their objectives for the year. Table 14: Health Services and Health Management Indicators INDICATOR I – Health facility (Health center/EPS 1, Health post) A) Maternal, Newborn, and Child Health (MNCH)  Immunization coverage rate of children aged 0-11 months  Coverage rate for nutrition and weight monitoring of children aged 0-24 months  Proportion of children aged between six and 59 months who have received two doses of vitamin A  Rate of skilled attendance at birth  Rate of new users of FP services (recruitment)  Rate of Prenatal Care attendance  Proportion of HIV-positive pregnant women under Antiretroviral (ARV) B) Disease control  Intermittent Preventive Therapy (IPT) 2 coverage rate for pregnant women  TB screening rate (in health centers and EPS 1)  Rate of TB cases successfully treated (in health posts) II – DHMT  Proportion of PBF funds available for all health facilities in the district that was effectively transferred to these facilities Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 54  Rate of timely delivery of periodic HSI reports  Comprehensiveness of periodic HSI reports Improved Regional and District Health Planning Regional and district health committees strengthen capacity in health governance, planning, as well as accounting and financial management of medical regions and health districts. The development of high quality AWPs at the regional and district levels requires the presence of TFPs during the planning process. The AWP targets the results expected from health structure staff through a milestone process. When milestones were met, regions and districts are paid through DF. It is important to note that AWP milestones are different than the indicators used in PBF. The evaluation team observed that the AWP process was not decentralized, despite the Procedures Manual indicating that the RMC was responsible for managing activities at the local level. Reality demonstrated that all major decisions were made at the central level. The central level is “micro managing” the work plan implementation, and reliable and regular communication between the RMC and the Program Chief of Bureau is inconsistent. The AWP quarterly review sessions were not viewed as productive because they were more focused on ceremony and speeches, rather than achieving clear, tangible objectives. Additionally, delays continue to threaten the RMC ownership in the implementation of the national plan for FP due to the insufficient capacity of the Department of Reproductive Health and Child Survival (DSRSE), specifically related to limited human resources and coordination. A few other notable findings were:  Abt supported the training sessions for MHO managers on the administrative and financial management handbook to strengthen professional management. The empowerment of local communities and health care providers in the implementation of the pilot Decentralization and Health Insurance (DECAM) project resulted in providing clear understanding of the roles and responsibilities of the MHO teams in expanding their coverage. Significant details on MHOs are included under sub-component B in Section 5.4 below.  The substantial involvement of IPs in organizing national consultations on health issues increased the level of trust between the MOH and USAID/Senegal and supported the validation of a number of approaches and initiatives in health governance and UHC. Continued collaboration and joint planning between MOH and USAID/Senegal activities is important to continue to support successful implementation of health programs and prevent any future potential for redundancy in the allocation of resources collaboration occurring between MOH/USAID activities.  While capacity has improved over the last two years through Abt’s foundational training, challenges still remain. Limited regional and district capacity for planning, management, and monitoring at the operational level continues to be a major challenge to achieving the strategic objective for health governance and management as outlined in the PNDS. 5.3.2 Lessons Learned and Recommendations ► USAID/Senegal may want to consider making the multiple reporting and data collection processes of the AWP, PBF, DF milestones, and the IP project indicators (USAID/Senegal Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 55 specific) more efficient and streamlined in order to improve decentralized services. Best practices indicate that there are efficiencies in aligning parallel processes, so that AWPs directly link to PBF and DF outcomes. For example, Team EY recognizes that it would be difficult to collapse all of the processes for AWP, PBF, DF, etc. into a single process. However, USAID/Senegal may want to consider reviewing all the parallel processes and perhaps supporting technical assistance for a Lean Six Sigma or process decomposition analysis to identify areas for process reduction or combination, where possible, to minimize burden on regional and district health teams. One example identified by team EY for process streamlining could be under DF. Although DF is paid against milestones, regions are also tracking a separate set of indicators which do not serve as a basis for payment. There may be potential to eliminate one set of reporting in this mechanism. Another option would be to tie DF not to milestones, but to link it with PBF indicators, since at least four indicators relate to regional and district management. This approach could be piloted in districts where both DF and PBF are functioning. In an effort to reduce all the cumbersome PBF steps, USAID/Senegal may want to consider using mobile technology for PBF data collection, verification, and bonus payments to increase the incentive and the speed at which they are received by service providers. One suggestion would be to develop an application that can be downloaded by PBF managers and verifiers so that PBF payments could be paid out monthly, and be based on real-time data collection. This system, which is being rolled out in Rwanda, would replace aspects of the current process, such as data verification. In Rwanda, the district health teams conduct data entry through a mobile internet solution into one central database. The database allows users to input data and print quarterly invoices for the verification committee. Further, it gives district staff the possibility to analyze their own PBF results by a built-in pivot chart function.15 ► USAID/Senegal may want to consider establishing a formalized mechanism for integration of health huts into the PBF mechanism, in order to support the PBF funding in reaching community health programs. USAID/Senegal may want to consider piloting indicators linked to data collection at the health hut level, such as data on community activities against which a portion of PBF bonuses could be paid. ► USAID/Senegal may want to consider creating a network of PBF health providers, in order to share lessons learned and improve PBF implementation. Social media platforms are valuable tools being used globally to share and improve knowledge in health in both developed and developing countries. Mobile phone apps could also be used as a platform for increased information sharing and communications among the PBF community. 15 Fritsche, G., and Rusa, L. "Rwanda: Performance-Based Financing in Health,” Sourcebook: Second Edition. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 56 5.4 Sub-component B: Expanding alternative financing mechanisms and improving their sustainability 5.4.1 Findings and Analysis Progress was made under sub-component B, social financing mechanisms with regard to strengthening the institutional framework, extending health risk coverage, and providing coverage for vulnerable groups through community-based health insurance or MHOs. Abt supported the MOH to monitor the signing of decrees on the organization and functioning of the National Office of Mutuelles in Senegal (ONAMS) and the Guarantee Fund for MHOs. According to interviews, Abt plans to continue providing support to these organizations going forward. The West Africa Economic and Monetary Union (UEMOA), of which Senegal is a member, was established to promote a common accounting system, periodic reviews of macroeconomic policies based on convergence criteria, a regional stock exchange, and the legal and regulatory framework for a regional banking system. The decrees developed for the establishment of MHOs in Senegal were adapted to the UEMOA regulations following a mission conducted in July 2012, to assess the implementation for community regulation in Senegal. Adoption of these decrees should contribute to strengthening the governing structure for MHOs in Senegal. Abt’s assistance to the MOH helped to draft legislation that resulted in the establishment of a UHC inter-ministerial steering committee at the national level. Subsequent to this inter￾ministerial council meeting, the official launch of UHC by the President of Senegal was an important milestone which demonstrated the political commitment to extending health care coverage to the entire population, particularly to those in the informal sector. In support of UHC, Abt assisted the MOH in the development of the 2013-2017 action plans for basic UHC through MHOs. The action plan aims at extending health coverage to 65% of the population employed in the informal and rural sectors by 2017, in line with the target of the 2013-2017 national strategy for economic and social development. It was adopted at the inter-ministerial steering committee on UHC chaired by the Prime Minister and held in April 2013. Since then, there was a noted increase in the flow of government financial and technical resources earmarked for the MHO coverage. Furthermore, the MOH established an advocacy mechanism so that increased budgetary volume, revised procedures in managing public resources, regular transfers, and transparency in financial transactions were apparent. Additionally, Abt supported the MOH in operationalizing MHO as an alternative funding mechanism. Abt provided support to validate additional awards of partial subsidies to 75 MHOs located in 10 of the 14 pilot departments involved in the DECAM initiative, thereby bringing the total amount of partial subsidies mobilized to 191,250,500 CFA francs ($100,341 USD) representing 50% of premiums for the 54,360 beneficiaries who are up-to-date with their payments. Furthermore, Abt helped 25 MHOs obtain targeted subsidies in the amount of 41,692,000 CFA francs ($79,465 USD) for the enrolment of 5,956 people residing in the Diourbel, Fatick, Kaffrine, and Kaolack regions. In an effort to enhance regulations governing MHOs, Abt assisted the Support Unit for Universal Health Coverage (CACMU) in the organization of supervision missions in the pilot departments of Kolda and Louga, thus facilitating the assessment of the absorption rate of initial subsidies. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 57 Community participation in these eight MHOs in the Thiès region was extensive across the various dimensions, but also revealed certain patterns of participation. Two of the MHOs in the study were established in the early phase of Senegalese MHO development and built upon experiences in a neighboring MHO (Fandene). These did not apply what are now widely accepted mechanisms for social participation in the early stages (e.g., formalized initial working group, feasibility study, development of alternative scenarios). However, through existing networks and experiences of the initiators, and the use of more informal groups, both these MHOs were able to engage their members in sensitization and policy making. In contrast, the newer MHOs developed their legitimacy through formal working groups representing the various groupings in the community, and participation in feasibility studies. The MHOs embedded within an existing association had high levels of participation during the set-up/design phase, building on their already existing associative structures, which facilitated delegation, engagement, and democratic processes. However, several of the MHOs had difficulty holding general assemblies (a key mechanism for social participation) and larger MHOs (in terms of size or geographic span of their catchment areas) that did not create specific decentralized structures appear to have more difficulty maintaining participation. The public perception of MHOs is favorable. However, due to economic difficulties, support from political and health authorities is needed. There was an overwhelming sense from the focus group discussions that members and MHO management had not really thought through the role that the general membership could or should play in awareness raising and recruitment of new members. Most members felt that they could contribute constructively in this regard, but needed support to identify opportunities to raise awareness or recruit new members. Engagement and recruitment activities, in which external stakeholders and MHO promoters participated in, indicated that providers and government external stakeholders generally had limited involvement. It also indicates the extent of promoter involvement in initial and ongoing activities. It should be noted that external stakeholder participation is crucial. The MOH and local government structures are important in terms of creating an institutional and regulatory environment conducive to the MHOs and providing political and financial resources that facilitate the viability of the MHO. The absence of the MOH in the ongoing support to MHOs was discussed frequently by MHO managers, members, and providers. MHO officials saw many ways in which the state could assist them. In most MHOs, the MOH or its decentralized units (the regional or district health offices) were minimally involved with MHO creation or functioning in their areas. However, once they were created, the regional and district authorities committed themselves to supporting the MHOs. They provided technical assistance to the MHO and to other localities that were willing to develop their own MHO. MHOs also sought assistance from local government structures, which they felt could provide a variety of possible subsidies such as help with provision of office space, sensitization, and support for premiums for poorer families in the community who experienced difficulty in joining the MHO because the membership fees and/or premiums were too high. Many MHOs mentioned that, although local governments had promised support to MHO operations, little materialized in terms of financial or political support. There was also a lack of technical capacity and experience among volunteers managing the MHO organizations to respond to the growing functioning requirements and departmental unions of MHOs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 58 5.4.2 Lessons Learned and Recommendations ► September 2014 marked the one year anniversary of the launch of UHC by the GOS. The MHO coverage of the informal and rural sector remains the strategic priority for progressing toward wider health coverage. Currently, vulnerable groups make up only 9.4% of beneficiaries supported by Abt during 2012 to 2013. The GOS may want to consider creating a new revenue stream to support subsidies so that there is sustainable funding for universal health coverage. Team EY acknowledges that Abt and FHI 360’s programs with microfinance were unsuccessful. As such, Team EY is recommending that USAID/Senegal work with MOH or the Ministry of Finance, at a higher level to create a macro-revenue stream to support UHC. One suggestion would be to levy an incremental tax on cell phone minutes, as was done in with the Kenya M-Pesa program. ► Additionally, USAID/Senegal should consider developing creative strategies that can engage the private sector and support employment and job creation. It may be useful to conduct a regional landscape analysis to assess the industries where there may be a viable PPP opportunity. For example, one company or a network could be approached to cover the upfront cost for a specific community to join a MHO. It is suggested that PPPs focus on Return on Investment (ROI) instead of social corporate responsibility in order to maximize opportunities for profit, productivity, and efficiency gains which could be part of the dialogue. The ROI under discussion is not associated directly with the MHO, but would be the focus of the private health insurance provider who might be interested in participating in a PPP with an MHO. ► USAID/Senegal may want to develop cost-effective strategies which help Small and Medium Enterprises (SME) to subsidize or purchase inexpensive health insurance or MHO memberships for their employees through the informal and formal sector. USAID/Senegal may also want to review opportunities to use PBF funds to create a resource pool to hire more health care providers. ► Implementation of the various UHC components may result in an increase in the use of health services in a context where there is a lack of service providers. The MOH may want to consider advocating for the implementation of programs aimed at strengthening the delivery of health care services regarding the recruitment of qualified health care personnel and procurement of sufficient equipment for health facilities, in anticipation of the growing demand. ► The significant flow of government resources to MHOs calls for the establishment of a mechanism that will support the regularity of transfers and transparency of financial transactions, in compliance with procedures on the management of public funds. However, MHOs do not yet have the capacity to meet these new requirements. The definition of resource allocation criteria and the development of a quarterly budget monitoring system are the Division of General Administration and Equipment’s (DAGE) responsibilities but were delayed, despite frequent reminders by Abt. USAID/Senegal may want to consider prioritizing Abt support for the DAGE in the upcoming year to draft, validate, and disseminate the resources allocation criteria this year. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 59 5.5 Sub-component C: Ensuring that national level policies and systems support improved performance of the health system throughout Senegal 5.5.1 Findings and Analysis Abt is supporting the policy reform and reorganization of the MOH to improve their ability to effectively implement UHC. Abt’s approach to policy reform is a multi-step, iterative process that has impact throughout the health system that all steps in the process must be anticipated and appropriately planned. Part of this support includes budgetary and financial reforms driven by the need to harmonize public financial management within the parameters set by the UEMOA. The process to reform public financial management in UEMOA countries began in 2009 with the objective of establishing new regulations for rigorous management of public resources. Flexibility of financing mechanisms is an asset for the successful alignment of aid to national priorities, provided that risks of redundancy in the allocation of resources are minimized through an integrated planning system. Key activities and initiatives related to sub-component C include 1) the replacement of the MTEF by the Multiyear Programming Document Expenditure (DPPD), 2) the reactivation of the interagency workgroup and the development of AWPs, 3) the development of a CH policy document, 4) support to the PNA for strategic planning and supply chain improvements, and 5) the organization of an Advisory Council for Health and Social Action (CNCAS). Support for the Development of the DPPD The replacement of the MTEF by the DPPD calls for a paradigm shift in line with PBF, regarding budget allocation of resources and accountability. Details regarding the DPPD can be found in Table 15. Table 15: DPPD Overview DPPD Overview: The MTEF is expected to be replaced by the DPPD by the end of 2014. The DPPD includes appropriations for and expected results of each Senegal health based program on the targeted objectives over a period of three years. Pursuant to article 12 paragraph 4 of the Finance Act 2011-15 of July 8, 2011, “a program includes funds for the implementation of an action or a coherent set of actions representing a clearly defined medium-term public policy.” Impact: This reform introduces major innovations in comparison with the MTEF. Firstly, the role of program managers is reinforced. Program managers are now “funds administrators” and “authorizing officers by delegation.” The appointment of program managers will now be done by decree. Secondly, the structure of the DPPD introduces the notion of “actions.” The program is conducted through a certain number of precise “actions,” for which budgetary allocations are made. Consequently, the HSS component will have to rethink its support to the MOH for the implementation of budgetary and financial reforms and adapt it to the new context of DPPD. Capacity gaps have already been identified at the Division of Planning, Research, and Statistics (DPRS) designated to steer this reform process. Furthermore, the roles and responsibilities of new program managers to be appointed should also be considered. The HSS component will help the MOH define these new roles and develop systems and capacities to address these new public finance management reform requirements in the health sector. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 60 Inter-Agency Working Group of Development of AWPs Faced with the changes brought about by the DPPD in the MOH planning process, Abt reactivated the inter-agency coordination working group to better prepare for the development of the 2015 AWPs. Based on the planning meetings, the working group decided 1) to disseminate the USAID/Senegal Health Program’s 2013-2014 integrated action plan at the regional level in order to provide guidance on the program’s priorities, 2) to inform RBs of component milestones for 2014-2015 to clearly identify activities to be proposed for USAID/Senegal’s financing, 3) to formally request that USAID/Senegal inform the IP’s COP on the budget amounts for their projects in the upcoming 2014-2015 fiscal year, and 4) to update regions on any changes made to the DF process. The RMC and District Management Committee (DMC) developed their AWPs for 2015 in the first quarter of 2014. It was noted that they were assisted by the IPs who were not informed on USAID/Senegal’s financial priorities for 2015; therefore the AWPs were not fully aligned with USAID/Senegal’s financial targets. As a result, the AWP activities and budget will go through a second round of review, which is a time-consuming process. The IPs are now aware of USAID/Senegal’s priorities and will guide the RMCs and DMCs in the revisions of the AWPs. Once completed, Abt will organize a validation meeting with all stakeholders to officially approve the AWPs. The process is viewed by most interviewees as a process that needs to be redesigned to reduce the amount of time spent preparing the AWPs. Additionally, this may be a good opportunity to harmonize the IP annual planning cycle with the government planning cycle and eliminate parallel work plan development processes. CH Policy Abt provided the MOH with technical assistance to develop a CH policy document in close collaboration with the CH component. Technical assistance was administrative in nature, including organizing of meetings, developing agendas and taking meeting minutes. The CH strategy included revised FP advocacy tools and a detailed plan for support to increase advocacy, awareness-raising, and monitoring activities at the national and regional levels. The MOH decided to develop a CH policy in 2012. Other USAID/Senegal IPs were invited to join the technical committee established by the EIPS to develop the CH policy paper, which was a precursor to the strategy. Advocacy for Family Planning The component is providing support for the MOH to implement a national FP plan through Group ISSA. In addition to technical assistance for revised FP advocacy tools, Abt and DSRSE worked to jointly adopt a strategy to harmonize the process of developing advocacy plans at a regional level through a participatory and inclusive approach utilizing three steps: 1) organization of a special committee on FP chaired by the Governor of the region, 2) evaluation of supply chain bottlenecks with FP commodities and identification of how they can be overcome through advocacy, and 3) the development of a regional advocacy plan based on the results of the committee. At the end of the 2014 fiscal year, Abt organized preparatory technical meetings to support advocacy efforts related to FP in nine of its 10 intervention regions, with the exception of Dakar. Of these nine regions, five held their special advocacy meetings (Kaolack, Kaffrine Sédhiou, Fatick, and Ziguinchor). Abt intends to keep working with the committees to implement FP advocacy plans in the other four regions. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 61 PNA Strategic Plan Support and Supply Chain Improvements Due to weak supply chain systems, various GOS programs and projects supported by development partners have established separate supply processes and systems for forecasting, procuring, storing, and distributing various health commodities in the country. Even programs that contract with the PNA for the storage and distribution of commodities still maintain independent processes for planning, forecasting, and procurement without much coordination. From a pharmaceutical management perspective, 85% of drugs are imported and distributed through two supply chains, a public supply chain, and a private supply chain. The supply chain pattern is therefore complex, and is modeled on the international donor system, which means it is highly compartmentalized by specific disease. Five wholesalers supply 921 private pharmacies. Apart from vaccines managed by UNICEF, drugs, including those procured by private pharmacies, are primarily managed by the PNA. The private sector is not authorized to import drugs without approval from the MOH and from the Directorate for Pharmacy and Laboratories (DPL). Additionally, all generic drugs are managed by the PNA. A cost recovery system was introduced into Primary Health Care (SSP) in the early 1980s. It was based on fee for service; a small fee of a few CFA francs was paid by the client after receiving the services. The system is based on the Bamako initiative, whose core objective is providing a basic package of integrated services through revitalized health centers that employ user fees and community co-management of funds.16 Recently, the cost recovery system was reviewed to allow essential drugs (generic drugs) and other subsidized health products from multilateral and bilateral sources to be managed and distributed by the PNA and PRAs. This was decided in order to comply with the international donors supply chain directives focusing on improving host countries health systems, instead of supporting parallel processes. The goal was to support the PNA and PRAs to scale-up national drug distribution coverage to around 40%. This structure places the PNA and PRAs on the front line for drugs and commodities for hospitals, health centers, posts and huts, throughout the health pyramid. The PNA traditional operational approach is a pull model, however, PRAs are moving towards a push model, with a target of six regions by 2014. In order to increase the quality of services at each level of the pyramid, the PNA and PRA implemented a push model in select pilot regions. The push model was primarily implemented by the Gates Foundation, and the launch of the National Family Planning Action Plan. According to the directorate of the PRA in Fatick, in the push model, a professional logistician who is managing stock and deliveries estimates the needs for health structures first, and then the health products are identified, selected according to the demands received, and packaged accordingly. The products are distributed to the services equipped to provide care for emergencies, FP, and MCH services. The health facilities receive the supplies needed on a delivery truck, without the facilities needing to come and pick up the supplies themselves. The PNA and PRA receive a small margin but are not aiming for significant profit. They receive their payments after the service provision and the product is distributed to the client. This means the drugs are no longer idle in large quantities in warehouses, waiting for health facilities to “pull” them when needed. Once distribution is 16 “The Bamako Initiative.” UNICEF. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 62 completed, the PNA and PRAs monitor the stock levels on a monthly basis and at that point the costs are recovered. According to Abt’s annual report, the PNA is autonomous in its management, which is unique because most other national services transactions are controlled by the Ministry of Finance. The drug distribution is decentralized to the region in order to facilitate health products reaching the beneficiaries faster. Although the PNA and the 11 PRAs are managed by qualified pharmacists, the other logistics and support staff is not professionally trained to occupy the positions they are currently filling. It was noted through interviews with individuals in various aspects of the supply chain including warehousing, distribution, and quantification of health commodities that pharmacy staff is often hired through political, social or religious connections and their work performance is weak. However, they do have high expectations of financial reward, which is their primary motivation. A few months of this combination of a low and inadequate skillset and high financial expectation leads to disappointment and a feeling of “being trapped” by the support staff and their production declines further. In reality, the PNA and the PRAs are covering only 15% of drugs distributed nationwide, due to the constraints of the national budget. To increase the operating budget for distribution, the MOH decided that the PNA and the PRAs should join the cost recovery system to take advantage of additional funds recovered from the distribution of products made available through multilateral and bilateral donors and through SSP cost recovery mechanisms. One recent study by the Management Sciences for Health demonstrated that with these changes the PNA is increasing its productivity but cannot handle the increased workload with its current work force capabilities. Weak supply chains among PRAs affect the ability of public hospitals to receive needed medicines and other essential commodities, which limits the services they can provide to their patients. Most public hospitals (including health centers and health posts) in Senegal, like many other public services, are dealing with considerable management issues. These issues range from the lack of human resources with the appropriate skillsets to a need for infrastructure support to improve health structures including buildings renovations, furniture, medical supplies, and essential equipment, including basic office furniture. Purchasing furniture, equipment, and commodities (excluding essential drugs) are procured through a tedious process that takes between 180 to 300 days. Additionally, the logistics systems are broken, with cars and ambulances unaccounted for and limited petrol available to use in the few vehicles that are available (there is not enough funding available to buy petrol). Hospitals are facing financial management constraints because they are implementing services at a volume they cannot afford, leaving them with significant back payments or debt. Abt provided technical assistance to the PNA to develop a strategic plan to improve coordination and reduce supply chain issues. The plan focused on strengthening pharmaceutical management, improving the capacity of the EIPS, increasing the availability of drugs, and resolving issues relating to the supply chain. The finalization and validation of the PNA procurement manual and development of the management and information system were the two key results achieved in 2013. A workshop was held in Dakar in June 2014 to validate the 2014-2018 Strategic Plan of the PNA. The main results achieved during the workshop were: i) strong advocacy by social Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 63 partners to provide the PNA with appropriate status; ii) a consensus on the need to simplify procedures for the award of contracts for the acquisition of drugs; iii) identification of flagship projects linked to the relocation of central store and the creation of large regional centers to supply fixed and mobile PRA, and iv) the commitment of donors to help fund the plan. The final document is expected to be approved by the Minister for Health by December 2014. Table 16 below provides an overview of the procurement manual and its impact. Table 16: PNA Procurement Manual Overview PNA Procurement Manual Overview: Development of the PNA procurement manual was entrusted to the consulting firm DMA, recruited through Abt’s sub-partner, PATH. The PNA led this activity from the preparation of the consultant’s scope of work to the validation of the final document in April 2013. The objective of the PNA procurement manual is to facilitate the geographic availability of essential commodities and other health products taking into account all policies and requirements regarding cost and efficiency. The procedure manual provides guidance and parameters for those responsible for the procurement of goods and services at the PNA, with information on the procurement rules and processes to be followed as well as on other related services such as insurance and distribution of products. Procedures will help those in charge of procurement and stock management to:  Comply with the procurement process.  Familiarize themselves with the key stages in the supply chain.  Familiarize themselves with the principles for the selection of products.  Quantify needs for drugs and other health products.  Monitor the quality of drugs.  Conduct a physical inspection of products at delivery.  Perform best storage and distribution practices.  Familiarize themselves with the distribution circuit.  Involve all persons who should be involved in the process.  Keep stock management tools up-to-date.  Monitor standard indicators on stock management. Impact: The procurement manual was widely distributed and the PRAs were trained on the new procedures. The manual was supplemented by the Management Information System (MIS). The MIS was created to supplement the procurement manual which was produced by SENINFOR, a consulting firm hired through Abt’s sub-partner PATH. Organization of the CNCAS Abt was active in the organization of the CNCAS in January 2013, which was designed as an advisory council to support health and social reforms. Abt provided support to the MOH throughout the lengthy process of preparing, organizing, and holding CNCAS meetings. Support included the preparation of the terms of reference to identify reforms needed to improve health governance, as well as the legal and technical documentation to accompany the reform strategy. Table 17 below provides an overview of the process and key outcomes. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 64 Table 17: CNCAS Overview CNCAS Overview: The objective of the CNCAS was to “build national consensus for health and social action sector reform to improve access to quality, sustainable health care in a context of improved governance and performance-based management.” The specific objectives were to:  Identify reforms needed for improved health governance at all levels of the health system.  Establish a system to operationalize health care cards to improve equitable distribution of health care service delivery.  Create the legal, material, and technical conditions to accelerate implementation of the national strategy on expanding health coverage for sustainable UHC.  Identify the appropriate strategies to be implemented to improve disease prevention and control through a multi-sectorial framework where the relevant ministerial departments and other non￾governmental stakeholders will play their part in full.  Improve advocacy and better protection for people with disabilities and other vulnerable groups through an integrated system built on the discussions with the community affected by disabilities and consistent with the CH policy. Impact: The event was a major turning point for the MOH as it translated into concrete actions for the Head of State. A total of 20 key measures were adopted at the CNCAS. Some progress in the implementation of two of these include: 1) Measure six on CH and 2) Measure 19 on accelerating expansion of health insurance coverage.* Note: A total of 20 key measures were adopted at the CNCAS. Some progress in the implementation of two of these include: 1) Measure six on CH and 2) Measure 19 on accelerating expansion of health insurance coverage. This data was received from Abt’s year two annual report. Team EY does not have additional information on details on the particulars of measures 6 and 19. 5.5.2 Lessons Learned and Recommendations ► Considering the GOS’s financial contributions and donor resources for health are insufficient to achieve the ambitious health objectives, the issue of budget allocation based on objective criteria is now a requirement. With the goal of the DPPD to validate efficiency in budget allocation and complete transparency in the management of funds, there is an opportunity to better understand where current resources are being used and what gaps remain. USAID/Senegal may want to provide technical assistance to the MOH to implement a detailed financial resource gap assessment supported by the development of a strategy to mobilize resources within the public and private sectors. If the recommendation is implemented, USAID/Senegal should consider developing a process to validate that the strategy is complete and includes sustainability measures, such as developing reliable revenue streams to decrease the level of dependence on donor resources. Additionally, USAID/Senegal may want to consider bringing in a local finance organization to provide targeted technical assistance in the areas of financial management. ► USAID/Senegal may want to consider bringing in technical assistance from a supply chain field support partner who can focus entirely on resolving all supply chain issues throughout the decentralized system, since there are significant concerns/issues reported by Abt and the other IPs related to supply chain. Experiences from the USAID West Africa funded regional Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 65 project, the Systems for Improved Access to Pharmaceuticals and Services, can also help inform the selected partner’s priorities and offer existing Francophone tools to monitor commodity stock status, anticipate future funding gaps, respond to projected medicine shortages and expiries, and make decisions based on accurate information. Additionally, some aspects of the private supply chain may be leveraged to optimize the public supply chain and there could be opportunities for skill transfer and coaching. USAID/Senegal may consider assessing the private supply chain further to identify good practices. ► Human resources are a major constraint throughout the health system. Given the challenges facing the PNA and PRAs, USAID/Senegal may want to consider embedding or seconding Abt supported staff with the appropriate skillsets to support Abt’s efforts and improve their capacity to deliver on their primary objectives of implementing a functional supply chain. USAID/Senegal has supported this type of model in many countries to mitigate the human resources constraints and there are examples that exist of how to eventually transfer embedded or seconded staff to the MOH through a gradual process over a reasonable amount of time for the MOH to absorb the costs. It is also recommended that USAID/Senegal consider integrating the performance-based indicators into the PNA and PRA management structure to incentivize MOH staff to improve the quality of their work. 5.6 Sub-component D: Coordinating USAID/Senegal health program components to ensure that they work effectively together to support improved performance of the health system 5.6.1 Findings and Analysis Although USAID/Senegal is using five separate contracts for each of the five components of its health program, there is an expectation that the five components should function as one program and be seen as a single program by the GOS and other stakeholders. Each of the five IPs has their own respective institutional policies and procedures with separate scopes and targets for which they are accountable. The five IPs and their consortia members have no formal grant relationship with the other components and no one component has legal authority over the others. Abt is responsible for working with the other four IPs and their partners to improve coordination and facilitate communication, coordination, and compromise. To help facilitate coordination, Abt manages the RBs and has placed a full-time RB Coordinator in each one. Abt participates and leads multiple meetings including the RB steering committee, and interagency coordination meetings in order to support and validate that the components are working together effectively to support improved performance of the health system. RB Coordination Meetings The three USAID/Senegal RBs located in the regions of Kaolack, Kolda, and Thiès are intended to play a significant role in coordination of the health program. Although coordination activities are occurring, interviews have shown that RBs are not being utilized effectively as they could be. Detailed information on the RBs is available in Section 4.5. The RBs hold quarterly coordination meetings with all five components to discuss implementation of activities, upcoming activities, recommendations to improve activity implementation, and to finalize their activity reports. Participants who attended the April 2014 meeting in Thiès included the RMO, District Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 66 Management Offices (DMO) or their representatives, members of the USAID/Senegal Health team (Director and AOR/HSS), and representatives of CACMU. These meetings are an opportunity to assess implementation of recommendations made during previous meetings, review activities undertaken during the quarter, discuss constraints and difficulties that impede activity implementation, and propose recommendations to enhance program implementation. Quarterly reports of the five components are also validated during these meetings. RBs also participated this year in quarterly coordination meetings of medical regions and health districts, providing both technical and financial support. These meetings are opportunities to share and discuss information for the effective implementation of activities. Coordination meetings contributed in the development of the DF concept paper as well as the procedure manual. Abt supported the identification of milestones and related indicators, which are the basis of DF contracting arrangements. Abt continued supporting coordination efforts in year three, building off the planning, implementation, management and financing instruments developed during the first two years, to continue improving the efficiency and effectiveness of health system performance. Although the procedure manual outlines a clear process for DF, it is clear from discussions with IPs that the communication around DF is weak. Only five IP FMs (Abt) could explain how the regions for DF were selected or how each component’s contribution to DF was determined. The response provided by other interviewees was that USAID/Senegal made the decision, but they did not know or were not able to communicate on what basis. Moreover, IPs do not fully understand the benefits of DF and how it can support health performance in the regions and are not aware of the impact or results of DF, as noted in Section 4.4 above. Steering Committee Meetings The Health Program’s Steering Committee met twice. The first meeting, held in October 2012 at the MOH, reviewed the accomplishments of year one and perspectives for year two. The second, held in May 2013, focused on the integrated action plan, new financing mechanisms initiatives (PBF and DF), and challenges and recommendations for improved implementation of program interventions. Coordination of USAID Health Program interventions was enhanced with the development of the first integrated action plan, as well as the DF procedure manual. The integrated action plan is now the tool being used by the Health Program's Steering Committee to summarize the program’s commitments and implementation of its activities. The RBs rely on the integrated action plan to support the development process, monitor districts and regions, and to inform local stakeholders on the various commitments of the USAID/Senegal Health Program's five components. The Steering Committee also supported the process of MHO development, which was led by the CACMU. There were significant challenges related with this process, due to weak capacity of the CACMU to support the process. Although Abt is supposed to take on a coordinating role for the Steering Committee meetings, reports indicate that USAID/Senegal sets the agenda and drives the meetings. Team EY understands that USAID/Senegal is driving the agenda, however, per the activities in the work plan this function should be performed by Abt. Interagency Coordination Meetings Inter-agency coordination continued during year three. Key inter-agency coordination activities conducted involved DF implementation in the first three regions and preparations for implementation in extension regions. An inter-agency coordination meeting was held in April Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 67 2014 at the component’s offices to discuss draft amendments to close out implementation letters for the Kaolack, Kolda and Thiès regions as well as DF implementation letters for the regions of Diourbel, Kaolack, Kolda Sédhiou, Thiès and Ziguinchor. Progress in the production of the summary report on the rapid review of DF activities was also assessed. It was decided to close￾out initial DF implementation letters on December 31, 2013 and sign amendments as well as new implementation letters by April 30, 2014 at the latest. A meeting with USAID/Senegal to discuss DF was scheduled on April 30, 2014 as well as a meeting between the committee in charge of monitoring DF and coordinators of RBs. The guidance note on planning was updated. One of the major challenges in program coordination is the inconsistent frequency of inter￾agency meetings and internal coordination meetings (national office and RBs) as well as the participation of RBs in coordination meetings of medical regions and health districts. For monitoring, the issue is the availability of quality information in a timely fashion. Regarding inter-agency meetings, the challenge could be addressed through exchanges between COPs, effectively convening meetings on a rotational basis which involves all components, and complying with established norms (e.g., duration of meetings, participants, frequency). Agencies agree that COP meetings could be held at more regular intervals and for shorter durations with a limited number of attendees. To address this issue, RBs are currently increasing their support for the development and sharing of quarterly work plans and monitoring implementation of AWPs. According to the DHMT the concept behind the creation of RBs for coordination offices is conceptually sound. They were expected to bring the IPs closer to their regional and district counterparts for improved coordination and synergy across the decentralized health sector. In Tambacounda, Team EY was told that the bureaus brought the IPs closer to the RHMT and DHMTs only in regions like Thiès, Kaolack, and Kolda where an RB existed. It was also noted by the Regional Medical Officer of Tambacounda that the RB in Kaolack was too far away to be effective in supporting Tambacounda and that he used his own personal vehicle to monitor and supervise health teams in the region. Governors in Tambacounda and Kaffrine expressed to the evaluation team that they were also too far from an RB to receive any real benefit from the coordinator and team of experts, and that they primarily relied on staff from IntraHealth, the ChildFund consortium, Africare, and World Vision (WV) who were available in their region. According to the interviews, it is the chief of the health region who attends the coordination meetings; and many of them reported that they have not seen the coordination team in their region more than once or twice a year, making them question how effective the RB coordination teams are outside of their regions in monitoring the AWPs. It was strongly suggested that the RB system be more equitable in distributing technical assistance and improve overall methods of communication. 5.6.2 Lessons Learned and Recommendations The current process of Abt coordinating with the other IPs in the RB could function more effectively, both internally among IPs and externally in relation to regional and districts committees. Suggestions to improve coordination include: ► USAID/Senegal may want to determine alternative ways to structure the health program or components so that integration is built into the project design. Exploring a different approach Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 68 from the current model where one IP without legal or budgetary authority over the others coordinates activities, has the potential to be more systematic, holistic, and efficient. As mentioned in Section 4.2 of the integrated approach, Abt has suggested that the components be collapsed into three more integrated components instead of the current five. Team EY supports this approach and in Section 4.2.2, we offer two options for restructuring the health program. In order to identify potential ways to restructure, USAID/Senegal may want to consider conducting stakeholder surveys to identify relevant and alternative ways to restructure the health components. ► Interviews have shown that RBs are not functioning well, in terms of being able to provide support to those outside of the regions that house the RBs. There are a number of inter￾agency coordination structures that could serve as examples for how the RBs may provide better coverage and services. However to determine the best structure for Senegal, further in￾depth assessment of RBs, including the current management plan is required. In addition, Team EY would need more time to do a desk review of potential coordination structures that are being used elsewhere and could be applied to the Senegal context. Although Team EY is not prepared to give explicit examples, one aspect that has been successful in improved coordination in several other countries is when coordination is led by the donor itself, instead of the partner. This has been seen in Haiti and several PEPFAR countries including Rwanda, Côte d’Ivoire, and South Africa. USAID/Senegal may want to establish clear standard operating procedures to facilitate regular communication between Abt and the other IPs on the results and benefits of DF and PBF to increase support for these initiatives as well as a sense of ownership by other components, which currently does not exist. This could also facilitate monitoring of the AWPs and improve the quality of USAID/Senegal’s integrated health package. USAID/Senegal may want to request that Abt work with the RBs to develop a clear and consistent communication plan that includes a more effective way to provide guidance and support to the RHMTs and DHMTs. USAID/Senegal could also consider procuring technical assistance to develop a more feasible and effective strategy for regional coordination. 5.7 Response to Evaluation Questions The table below provides information to respond to the component-specific evaluation questions as stated in the scope of work. It is not intended to be exhaustive, but rather aims to highlight notable successes, constraints, and challenges that have been experienced during implementation by Abt Associates; and key interventions that may be added, continued, or removed within Component 1. Team EY and USAID/Senegal agreed that this analysis would be conducted by component rather than by individual sub-components. Table 18: Component Table Component Specific Question Analysis To what extent have the components achieved their objectives? Abt made progress towards achieving their objectives as outlined in Section 5.1. The USAID standard indicators outlined in Table 12 were selected as a sample, as they are closely aligned with USAID/Senegal’s IR 3: Improved performance of the health system, and support USAID/Senegal’s high-level indicators, the percentage increase in the GOS’s contribution to health spending and percentage Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 69 of facilities with stock-outs of essential drugs. According to Abt indicator data, Abt is meeting the majority of its targets for those indicators where data was reported. To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component?  Strengthening the management of medical regions and health districts: Abt has contributed to the strengthening of skills and management capacity within the regions and districts through trainings, DF support, and PBF activities. One of the factors contributing to success is that in each PBF region, an Abt coordinator works with a national PBF advisor who is responsible for supporting the program’s implementation in a coordinated approach. Qualitative interviews at the regional and district levels indicated that training related to PBF and direct financing resulted in more credible financial management processes; enhanced role definition; and increased knowledge for regional health management staff, better tracking of resources, and more targeted health spending.  Implementation of a PBF pilot: Abt supported the MOH in piloting PBF in seven districts in three regions since the inception of the program. The provision of bonus payments based on performance has done more than just augment health worker income. Feedback received on PBF showed that health care providers believe PBF was motivating and changing the way they delivered care. The Regional PBF Coordinator in Kolda noted that one of the key factors contributing to success was the behavioral change in the region, citing a greater focus among workers on how they analyzed quality and coverage of health care services. In addition, greater ownership over health services was noted along with a sense of connection and control over the outcome of health services provided because health care workers saw a direct benefit when indicators were achieved.  Support to the GOS for coordination and sustainability: As a result of Abt assisting the MOH, legislation was drafted that resulted in the establishment of a UHC inter-ministerial steering committee at the national level. Abt is also supporting policy reform and reorganization of the MOH to improve their ability to effectively implement UHC. Part of this support includes budgetary and financial reforms driven by the need to harmonize public financial management within the parameters set by the UEMOA. The key success factor was the substantial involvement of partners in organizing national consultations for health issues, which increased the level of trust between the MOH and USAID/Senegal. It also supported the validation of a number of approaches and initiatives related to rolling out UHC. What are the constraints and challenges that have hindered successful implementation of each sub-component, and how has the IP dealt with those challenges?  Data strike: A number of actual results on Abt’s activities were unavailable or unreported in Abt’s year one and year two annual reports. Abt access to the data from the GOS public finance management systems was greatly impacted due to the data strike that affected the entire country. This strike delayed implementation of the PBF pilot and prevented the signing of PBF contracts in Louga, as health care workers in this region refused to provide the data that would be necessary for measuring performance. As a result, Abt focused implementation in years one and two in regions where health workers were willing to provide the necessary data.  Reporting burden on regional and district teams: The various funding mechanisms being employed in the regions has resulted in multiple reporting and data collection processes for the AWP, PBF, and DF milestones. Regional Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 70 and district health teams are responsible for gathering data and reporting on each of these individual processes, several of which are quite cumbersome and time consuming. The negative impact that the reporting burden has had on Abt is that it detracts from time spent on other implementing other activities. The Abt-hired Regional Coordinator and RB staff are in place to work with the regional district teams on execution of these activities. In addition, Abt has conducted trainings at the medical region and district levels to facilitate capacity development to manage DF and PBF processes.  Weak supply chain system: Due to weak supply chain systems, various projects supported by development partners established separate supply processes and systems for forecasting, procuring, storing, and distributing various health commodities in the country. Even programs that contract with the PNA for the storage and distribution of commodities still maintain independent processes for planning, forecasting, and procurement without much coordination. Abt provided technical assistance to the PNA to develop a strategic plan to improve coordination and reduce supply chain issues. The plan focused on strengthening pharmaceutical management, improving the capacity of the EIPS, increasing the availability of drugs, and resolving issues relating to the supply chain. The finalization and validation of the PNA procurement manual and development of the management and information system were the two key results achieved in 2013. It is important to note, that the weak supply chain systems did not affect Abt’s program directly since they are providing service delivery technical assistance, but the problems with the supply chain did affect other partners that are depending on medicines and other commodities. These specific supply chain issues are referenced in the other component sections. Are there interventions that should be added or removed?  Financial resource analysis to facilitate effective budget allocation: Considering that the GOS’s financial contributions and donor resources for health are insufficient to achieve the ambitious health objectives, the issue of budget allocation based on objective criteria should be reviewed. With the goal of the DPPD to validate efficiency in budget allocation and complete transparency in the management of funds, there is an opportunity to better understand where current resources are being used and what gaps remain. USAID/Senegal may want to provide technical assistance to the MOH to implement a detailed financial resource gap assessment supported by the development of a strategy to mobilize resources within the public and private sectors. If the recommendation is implemented, USAID/Senegal should consider developing a process to validate that the strategy is complete and includes sustainability measures, such as developing reliable revenue streams in order to decrease the level of dependence on donor resources. Additionally, USAID/Senegal may want to consider bringing in a local financial organization to provide targeted technical assistance in the area of financial management.  Capacity building of PNA and PRAs to improve supply chains: Although the PNA and the 11 PRAs are managed by qualified pharmacists, the other logistical and support staff is not professionally trained to occupy the positions they are currently filling. Given the challenges facing the PNA and PRAs, USAID/Senegal may want to consider embedding or seconding to the PNA Abt-supported staff with the appropriate skillsets to improve their Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 71 capacity to deliver on their primary objectives of implementing a functional supply chain. It is also recommended that USAID/Senegal consider integrating the performance-based indicators into the PNA and PRA management structure to incentivize staff to improve the quality of their work.  Specialized technical assistance for supply chain: Given the supply chain weaknesses identified, USAID/Senegal may want to consider bringing in technical assistance from a supply chain field support partner. It is expected that this partner would be able to focus entirely on addressing all supply chain issues throughout the decentralized system. Additionally, some aspects of the private supply chain may be leveraged to optimize the public supply chain and there could be opportunities for skill transfer and coaching. USAID/Senegal may consider assessing the private supply chain further to leverage or identify good practices. Are there changes that could be made to improve performance?  Expansion of PBF: USAID/Senegal may want to consider strengthening the role of health huts in the PBF mechanism. This could contribute to increased quality of services and improved linkages between the health posts and health huts. USAID/Senegal may want to consider piloting indicators linked to data collection at the health hut level, such as data on community-based activities against which a portion of PBF bonuses could be paid.  Expand peripheral health services to respond to demand generated by MHOs: The MHO’s coverage to the informal and rural sector remains the strategic priority for progressing toward wider health coverage. Implementation of the various UHC components will likely result in a considerable increase in the use of health services in a context where there is a lack of service providers. USAID/Senegal may want to consider recommending that the MOH implement programs aimed at strengthening the delivery of health care services with an emphasis on the recruitment of qualified health care personnel and procurement of sufficient equipment for health facilities.  Increased collaboration across USAID/Senegal Health Program components through strengthened regional bureaus: Based on the challenges of the RBs, USAID/Senegal may want to assess the structure of RBs to better understand how to improve the functionality. In order to determine the best structure for Senegal, further in-depth assessment of the RBs (including the current management plan) is required. In the near-term, USAID/Senegal may consider working with Abt to define clear vision and mission statements for the RBs to support IPs to better operate within the system and improve their functionality. This would include a review of procedure manuals with all IPs so that job descriptions are clearly defined and lines of authority and responsibilities for the coordinator and IP’s technical advisors are known. USAID/Senegal may also want to request that Abt work with the RBs to develop a clear and consistent communication plan that includes a more effective way to provide guidance and support to the RHMTs and DHMTs.  Strengthen support to regions and districts for reporting for the AWPs: Abt and USAID/Senegal may want to increase the number of qualified staff placed in the RBs to better accommodate requests for field monitoring of the IPs. This does not necessarily require hiring additional FM (staff that can support the internal financial controls and other administrative aspects Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 72 managing the RBs), but could be accomplished through hiring assistant level staff specifically to support the documentation of financial transactions and gathering of supporting paperwork. In addition, USAID/Senegal may consider the use of a mobile application to help collection of receipts/verification documentation from more remote regions to increase the indicator validation process and the speed at which payments are received. 5.8 Data Sources In addition to sources cited in Annex G: Bibliography, data collected in the field was used for analysis of this component, including: In-depth interview with Chiefs of Party, RB coordinators, key MOH stakeholders, regional and district health officers, regional and district coordinating offices, community-based health insurance managers, Directors of the Regional Office of Education and Information for Health (BREIPS), service providers at health facilities; services providers at health huts, pharmacists, and private firms; as well as Focus Group Discussions with CHWs, CH committees, community-based health insurance beneficiaries, and associations of PLWHA. Data collection tools for field interviews are available in Annex I and Annex J. 6.0 COMPONENT #2: HEALTH SERVICES IMPROVEMENT 6.1 Background USAID/Senegal awarded IntraHealth a five-year, $32M USD cooperative agreement in 2011 to implement the HSI of the health sector strategy and contribute to the achievement of IR 1: Increased availability of an integrated package of quality health services. IntraHealth is implementing this project with three primary sub-partners, as outlined in their original cooperative, Helen Keller International (HKI), Medic Mobile, and the Siggil Jigeen Network. The principal objective of this component is to increase the use of the integrated package of services in health posts and health centers with strong linkages to community-based services and regional hospitals to support a well-coordinated continuum of care. This component is expected to improve the availability of the integrated package of quality health services, the functioning of the facilities and teams that deliver these services, performance of health personnel in these facilities, and increased engagement of the private sector in offering the integrated package of services. The HSI component has two main service packages: the integrated package and the malaria package. The integrated package of services being implemented in 12 regions and the malaria package is being implemented in 14 regions. In 2012 there was an important shift in focus regions Louga to Kédougou. The project is organized into four main sub-components:  Sub-component A: Increase access to an integrated package of quality health services.  Sub-component B: Improve functioning of health services in public health posts, health centers, and regional hospitals for related priority services provided in the integrated package.  Sub-component C: Improve human resource management at public health facilities.  Sub-component D: Outreach to private health facilities. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 73 IntraHealth is working to strengthen the capacity of MOH divisions at the central level where staff is engaged in the planning and oversight of activities supported in their respective program areas (e.g., RH, FP, malaria). Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 74 Table 19 below shows the standard indicators related to the HSI component in the first two years compared with targets. Although Team EY understands that IntraHealth provided its year three annual report to USAID/Senegal in November 2014, Team EY did not receive the report in time to include the 2014 indicator data into the table below. However, information derived from the annual report that was verified by data point/site interviews and corroborated by alternative sources was utilized in our analysis. Table 19: USAID Indicators for HSI # Indicator Target Actual Target Actual 2012 2013 1 Percentage of SDPs that offer the integrated package of services as part of a program supported by the USG 28% 33% 73% 74% 2 Number of FP counseling locations supported by the USG 339 357 882 855 3 CYP total (broken down by method Family Planning/Reproductive Health (FP/RH) 130,276 259,857 143,303 154,740 4 Percentage of women having received IPT in prenatal consultations during their last pregnancy 45.2% * 52.8% 41.3% 5 Number of contraceptives distributed 739,219 1,266,846 813,141 922,447 6 Percentage of women receiving Active Management of Third Stage Labor (AMTSL) as part of a program supported by the USG 81.4% 95.6% 90% * 7 Percentage of newborns who received a post-natal visit within three days of birth 60% * 70% * 8 Percentage of newborns who received immediate newborn care by the USG-supported program 81.4% 41.6% 90% * 9 Percentage of children under 12 months that received DPT3 immunization as part of a program supported by the USG 87% * 88% 88.5% 10 Percentage of cases of child diarrhea treated with Oral Rehydration Salts (ORS) 40% * 50% 17% 11 Number of malaria cases treated as part of programs supported by USAID (e.g., broken down by target group: children under-five, pregnant women, general population) 157, 401 6117 166, 146 * 12 Number of providers trained in child health and nutrition as part of programs supported by the USG 1640 257 1770 1032 13 Number of providers trained in treating malaria with Artemisinin-Based Combination Therapy (ACT), supported by the USG 670 672 309 277 14 Percentage of SDPs assisted by USAID that experience stock shortages of contraceptive products 50% 97% 30% 74% 15 Number of supervisory visits by personnel in health care establishments at the community level 976 585 1,176 1,400 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 75 16 Number of advanced strategies implemented by public health posts and centers 600 590 2,000 1,586 17 Number of SDPs covered by the program that have a job description 285 64 500 276 18 Number of private establishments having enrolled in TutoratPlus 17 74 34 89 *Data not available Note: Target and actual data included in the table above is derived directly from annual reports provided by IPs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 76 The data from the 18 indicators noted in Table 19 are derived from IntraHealth’s quarterly and annual reports. These select indicators are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services. In addition, they support USAID/Senegal’s high-level indicators: EPI coverage, number of deliveries with a skilled birth attendant, and CYP. The indicators in the table represent IntraHealth’s core activities in his including:  Increasing access to an integrated package of quality health services.  Improving functioning of health services in public health posts, health centers and regional hospitals for related priority services provided in the integrated package.  Improving HR management at public facilities, and outreach to private health facilities. Additional information on indicator achievement status can be referenced Figure 14. ► Targets Met: According to the IntraHealth year two annual report, the HSI component nearly met, met, or exceeded targets for 12 indicators to include: 1. Percentage of SDPs that offer the integrated package of services as part of a program supported by the USG. 2. Number of FP counseling locations supported by the USG. 3. CYP total. 4. Percentage of women having received IPT in prenatal consultations during their last pregnancy. 5. Number of contraceptives distributed. 6. Percentage of children under 12 months that received DPT3 immunization as part of a program supported by the USG. 7. Percentage of cases of child diarrhea treated with ORS. 8. Number of providers trained in treating malaria with ACT, supported by the USG. 9. Percentage of SDPs assisted by USAID/Senegal that experience stock shortages of contraceptive products. 10. Number of supervisory visits by personnel in health care establishments at the community level. 11. Number of advanced strategies implemented by public health posts and centers. 12. Number of private establishments having enrolled in TutoratPlus. ► Targets Not Met: Two indicators reported low performance against targets during year two to include: 1. Number of providers trained in child health and nutrition. 2. Number of SDPs covered by the program that have a job description. Challenges identified in meeting these targets include delays in training at the decentralized level due to high demand and limited time for technical teams in the regions. In addition, challenges noted in IntraHealth's report included limited introduction of postpartum Intrauterine Devices (IUD) in trainings, lack of coordination at the central level, as well as limited integration of the TutoratPlus in districts’ work plans. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 77 ► Data Not Received: Team EY did not receive full reports after 2013 to include in the table. Additionally, a number of actual results were unavailable or unreported in year two of IntraHealth’s annual reports to include: 1. Percentage of women receiving AMTSL as part of a program supported by the USG. 2. Percentage of newborns who received a postnatal visit within three days of birth. 3. Percentage of newborns who received immediate newborn care by the USG-supported program. 4. Number of malaria cases treated as part of programs supported by USAID/Senegal. Figure 14: HSI Indicator Status 6.2 Key Findings and Recommendations/Benefits Overall the HSI component is achieving some of its objectives and meeting some of the agreed upon indicators as stated in their cooperative agreement based on the data received. Given that this evaluation was focused on two years, Team EY did not evaluate the impact of IntraHealth but reviewed the USAID/Senegal strategy as a whole. Overall, a key takeaway from the evaluation was the opportunity for a greater focus on the quality of implementation. Key findings and recommendations are provided in further detail in Section 6.3 – 6.6. The table below summarizes the most important key findings and recommendations/benefits relevant to strengthening the overall quality and delivery of the HSI component. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 78 Table 20: HSI Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefits 1 The pilot phase of the electronic Human Resource for Health Information Software (iHRIS) was noted in interviews for supplying health managers and practitioners with information to assess human resource constraints and to subsequently plan and evaluate interventions. Recommendation: With the expansion of the iHRIS software system to support the MOH and RHMT, USAID/Senegal may want to conduct an independent assessment of staffing needs and develop a strategy and action plan for staffing of the SDP in the five regions where the iHRIS is operating. Benefit: This assessment could contribute to enhancing the use of more effective human resources in these regions by providing information on how iHRIS is being used and what additional human resources, infrastructure, or other support is needed to improve usage. 2 Interviews demonstrated that there may be more opportunities to better apply strategies that focus on improving the quality of services and care within private practices, primarily the NGO and FBO sector within a more standardized continuum of care approach. Recommendation: USAID/Senegal and IntraHealth may want to consider forming networks of private providers to serve as champions and thought leaders on how the sectors can support each other in the management of care for Senegalese communities using a continuum of care approach. Benefit: This could support complementary care and contribute to achieving better outcomes at the district and regional level for beneficiaries, in line with standardized care from both private and public sector providers that are serving populations. 3 It appears that overall DHS 2012 health indicators are weakest in regions where the integrated packages are limited. The two regions of Matam and Tambacounda, specifically, show low overall health indicators that could benefit from expanding the integrated package of services. Recommendation: USAID/Senegal may want to consider expanding the integrated package of services currently provided by IntraHealth to regions with weaker indicator results. Benefit: The expansion of integrated service packages to urban areas or higher-populated facilities may increase access of the population, which could lead to improved uptake of services. 6.3 Sub-component A: Increase access to an integrated package of quality health services 6.3.1 Findings and Analysis Implementation of the TutoratPlus Performance Improvement Method The TutoratPlus model is a site-based mentoring and performance-improvement process, which mobilizes essential actors in an effort to identify shortfalls in both public and private sector Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 79 service delivery, clinic functioning, and individual/clinic performance. The TutoratPlus model specifically aims to develop and strengthen the clinical and management capacity of health staff and to address identified shortfalls through skills upgrade, attention to systems, new partnerships, action planning, and the establishment and tracking of performance targets. The TutoratPlus process has shown success as an intervention and a training approach. The model utilizes a three step process. The first step is to conduct a health situational analysis which produces a work plan with key recommendations and action steps to improve the quality and performance of services provided. Mentoring of providers is a critical component of the next steps after all are aware of results from the analysis. The second step is implementing the recommendations from the work plan over a period of time. On-the job mentoring builds the capacity of health staff to execute recommendations effectively and facilitates skills reinforcement. The last step is an on-site performance assessment to compare if the recommendations in the work plan were implemented. The TutoratPlus model is implemented throughout Senegal in 60 districts, in all 14 regions. This coverage represents 80% of the country’s health districts, exceeding planned coverage by 118% of the target indicator. According to annual reports, in year one, 15 districts were enrolled and by the end of year two, 41 health districts were enrolled, representing 54 % of the countries districts. Multiple trainings and workshops were conducted on TutoratPlus at the regions and districts to improve adaption and use of the model. Trainings/workshops included TutoratPlus mentors working directly with clinic staff, results-sharing workshops, performance-based training sessions, and orientation workshops.  Mentor trainings: Over the implementation period, district and regional health staff with past experience from similar exercises were selected to mentor colleagues in performing the health situation analyses. The trained mentors worked with clinic staff to apply TutoratPlus, with the assistance of district supervisors, regional trainers, and project staff, until they were able to manage the process on their own. As a result of the mentor training, resources from the regional and district health teams developed clinical mapping and quality of care research capacity and knowledge. Mentors also created and facilitated results-based workshops, performance-based training sessions, and orientation workshops to assist with adoption of the TutoratPlus model.  Results-based workshops: The results-based workshops were attended by multiple stakeholders. The workshops facilitated discussion on key health issues related to a particular district or region and approaches and interventions to address challenges, risk factors, and other access barriers for communities.  Performance-based training sessions: The performance-based training sessions were conducted in 80% of the health districts in 11 of the 14 regions. Regional and district level service providers expressed their satisfaction with the approach which allowed them to improve their interpersonal communication skills and establish an integrated service delivery approach for MNCH, FP, and malaria. They also voiced a level of confidence and competency in their ability to apply principles learned when they return to work. Health providers prioritized services and changing the negative or undesirable behaviors that can limit community access to services.  Orientation workshop: An orientation workshop was conducted on monitoring TutoratPlus for regional and district health team members in 14 regions. Of the women that participated Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 80 in the workshop, 40% reported that their on the job skills were strengthened by on-site coaching and supervision tools. On-site surveys and supervision assessments are part of step three in the TutoratPlus process. These exercises are conducted to assess performance of the TutoratPlus model. Based on an on￾site survey, 97% of 96 district health managers and health care providers in multiple districts, expressed their satisfaction with the ongoing coaching visits, as reported by IntraHealth’s year two annual report. Service providers cited strengthened professional competency and improved service provision and performance through involvement of local collectivities, health committees, and MHOs in identifying problems and issues as a positive aspect of TutoratPlus. Beginning in 2014, IntraHealth made minor changes in the manner in which tutors were trained. The tutors were grouped in teams of 15 to 16, and a total of 152 tutors were made available to the 21 districts trained this year. As seen in Table 21 below, tutors were equipped to coach sessions for the following modules: Table 21: Training Sessions Training Description Module 1 Pregnancy with delivery and post-partum kits, and mannequin aids that demonstrate techniques to help infants breathe when resuscitating neonates. Module 2 FP with pelvis and arm mannequins, IUD kits for insertion and withdrawal, and insertion of subcutaneous implants (implants). Module 3 Managing preventable disease with ORS kits, and health communication materials for the prevention of malaria, diarrhea, and other key illnesses Module 4 Management and organization of services, FM, and management of human resources. Module 5 Management information systems, supervision, and monitoring and evaluation of the action plans. Module 6 Health promotion using community-based approaches, BCC messaging, and creating demand for the increased uptake for the integrated package of health services. Improving Access to Quality Family Planning Services IntraHealth identified ways to improve new clients’ access to FP and strengthen the competencies of qualified providers. A main strategy to increase FP service uptake was to target high-volume health sites where FP services could be linked to where women were already receiving other services. Where FP services were regularly linked to immunization services and outreach strategies at health posts, results showed that 33,060 people (2,527 men and 30,533 women) received FP messages while attending information sessions during immunization service provision. A total of 10,905 new FP method users were enrolled, with variation across regions, at an average rate of 35%. In year two, a total of 12,456 new users of FP methods were enrolled. FP training and supportive supervision activities continued as the DSRSE received FP tools and assistance which could then be transferred to the regional and district health teams. IntraHealth and the DSRSE collaborated on one study protocol and strategies for introducing post-partum IUDs into obstetric services of ten districts (e.g., Guédiawaye, Pikine, Dakar West, Dakar South, Sédhiou, Touba, Bambey, Thiadiaye, and Rufisque). The study protocol was submitted to the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 81 MOH Committee of Ethics, pending approval in 2015. Figure 15 below outlines the trainings delivered to providers by subject. Figure 15: Provider Trainings by Subject *Note – disaggregated data by gender not available for immunization guidelines Data collected on the number of men and women trained was available for IntraHealth’s childhood illness, malaria, and maternal health trainings. Figure 15 above summarizes the above disaggregated information. More women than men were trained in each provider training category (52%, 68%, and 67% respectively) with the exception of the immunization guidelines training where information was not collected by gender. No additional analyses of these gender￾related findings were conducted since a detailed gender evaluation was not part of Team EY’s scope, as USAID/Senegal is currently performing a separate gender assessment. Childhood Illness: IntraHealth contributed to the implementation of the Child Survival National Plan (CSNP) through the training of 170 providers (88 of which were women). The focus of the training included the provision of services to manage childhood illness in the health regions of Thiès, Fatick, Dakar, Kaffrine, Kaolack, and Diourbel. The Division de l’Alimentation et de la Nutrition (DAN) also organized training sessions of 247 health workers in which 188 were women. According the qualitative data, trainings were perceived to raise the level of competency or knowledge of CHWs to provide better care for those under-five affected by diarrhea, a major contributing factor to child mortality. For example, in a focus group discussion with CHWs in Sédhiou, they believed they were better able to care for uncomplicated simple diarrhea (e.g., attributable to malaria or dehydration), facilitating the initial delivery of basic medication, the supporting the treatment of pneumonia. Additionally, the training they received provided the practical support they needed to help them do their day-to-day jobs better. In the regions of Sédhiou and Kédougou, where acute malnutrition cases were high, capacity development focused on training 58 providers (45 men and 13 women) from 51 SDPs. Ample time was devoted to helping staff understand how to use the WHO’s Anthro Software for better monitoring of child growth and detecting cases of acute malnutrition. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 82 Malaria: Throughout Senegal, 509 health service providers (208 women) were trained on malaria prevention. The TPI2 (an IntraHealth partner-developed approach) focuses on skills development, improved understanding, and knowledge for effective cross-sector partnering. The TPI2 approach is being used to prevent malaria among pregnant women in the districts of Mbao and Touba, showing promising results. To scale-up its use, five districts in Dakar as well as nine districts in the region of Thiès have adopted the TP12 approach. In these districts, once the situational analysis is completed, advocacy and outreach plans will be developed to target local groups and mobilize them to facilitate better communication around malaria prevention and other CH issues. Maternal Health: To improve support to pregnant women in the regions that provide the integrated package of services (excluding the region of Louga), 333 providers (inclusive of 230 women) received training to strengthen competency in the provision of quality prenatal care, focused on detection and prevention of pregnancy-related complications, interpersonal communication, and counselling for women and couples. According to IntraHealth’s annual report, training was well received and affected the expansion of services, readily available in the 14 SDPs of Sédhiou, Kaolack, Kolda and Diourbel. In year two, the program strengthened the management of eclampsia and pre-eclampsia in 350 SDPs through training 448 providers. Providers reported that through assistance in supervision, they contributed to improvements in maternal and neonatal health such as more complete services offered to pregnant women and examinations during ANC visits and counselling related to maternal nutrition, safe deliveries and healthy timing and spacing of births. Immunization Guidelines: In order to assist with meeting immunization goals, IntraHealth trained providers on the use of the routine immunization guidelines in preparation for national immunization days in all 14 regions of Senegal. The trainings included updating their knowledge and skills on how to manage the vaccines requiring cold chain, injection safety, waste management, and the integrated surveillance of illness and epidemic outbreak control. In addition, sessions on the use of newly introduced anti-pneumococcal vaccine in 2013, anti￾Rubella, and the second dose of measles in 2014, were conducted. In order for the new guidelines to be understood and applied rapidly, 1,001 providers received training, which was double the original planned target. Reaching every district consisted of sharing the health action plans with all stakeholders, including the networks of CHWs and outreach volunteers, MHO, local authorities and local administrators (e.g., Préfets, Presidents of Rural Councils, and Mayors), and regional and district health teams. IntraHealth also supported the introduction of immunization management tools including 500 registers to track vaccines, 920 delivery order books, 1,500 monthly stock reports, 1,500 daily data collection journals, and 200,000 immunization cards. TB: According to the most recent annual report, Team EY noted the TB-related activities under the HSI component were linked to the IntraHealth’s partnering with the FHI 360 to validate high-quality diagnosis and Directly Observed Treatment Short-Course (DOTS) management of TB cases at the facility level. Other key focus areas were the inclusion of TB drugs in monitoring of facility-level drug supply and shared updates of contacts, coordinating work plans, and updating TutoratPlus when the HIV/AIDS component introduced relevant advances. Given the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 83 decrease in TB funding and that services are primarily included in the HIV/AIDS component, no further analysis was conducted on IntraHealth’s TB activities. Support for Direct Funding and District Grants Institutional support to partners was provided through joint investments and technical assistance spearheaded by IntraHealth. According to the annual report, IntraHealth distributed approximately 101,244,073 CFA ($192,253 USD) to the regions of Thiès, Kolda, Kaolack, Diourbel, Sédhiou, and Ziguinchor for institutional support via the DF mechanism. In addition, funds were distributed through 38 district grants, which provided the opportunity for districts to implement priority activities where they lacked the necessary resources. These activities included: 1) Training health committees in better understanding of their roles and responsibilities, 2) Procuring essential equipment and commodities (i.e., implant insertion and withdrawal kits) and 3) Signing memoranda of understanding between districts and private health structures to improve collaboration through coordination meetings, supervision and submission of health service information. This is a slight increase from year two, where direct funding was provided in three regions (Thiès, Kolda, Kaolack) for 24.867.529 CFA ($46,522 USD) and 15 districts (for TutoratPlus Milestone payments) of 74.694.206 CFA ($39,926 USD) for a total of 99.561.735 CFA ($186,260 USD). 6.3.2 Lessons Learned and Recommendations ► Recipients of the TutoratPlus training described the interactions with mentors as positive in affecting their ability to adopt new practices of clinical care in addition to changing past behaviors that compromise clients’ satisfaction and quality of the integrated package of services. The major barrier noted with the implementation of the TutoratPlus model related to staffing issues. USAID/Senegal, in its leadership role, should continue supporting IntraHealth to advocate for and support the MOH in filling clinical staffing gaps throughout the focus regions for this component. ► USAID/Senegal may want to consider conducting a review of FP integration in services such as HIV care and treatment platforms, since the linking of FP services to immunization services and outreach in the community has demonstrated positive results. USAID/Senegal may also want to consider additional opportunities to offer FP counselling, contraceptive methods, and other linkages within a continuum of care approach. Experiences from a number of larger PEPFAR countries such as South Africa, Malawi and Tanzania, show that FP service integration in Prevention of Mother-to-Child Transmission (PMTCT) services, infant wellness visits, and CT encounters, may provide experiences to increase the opportunity to reach women and couples with integrated health services. These multi-country successes across Africa suggest that better use of these service delivery platforms may also be effective in Senegal given FHI 360’s current focus to strengthen HIV services across the care continuum. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 84 6.4 Sub-component B: Improve functioning of health services in public health posts and health centers and regional hospitals for related priority services provided in the Integrated Package 6.4.1 Findings and Analysis IntraHealth implemented a range of management and technical interventions that affected the functioning of health services at different levels in the context of systemic challenges faced at health posts, health centers, and regional hospitals. In general there were improvements in the availability of contraceptives and essential drugs at the SDPs. However, 40% stock-outs for magnesium sulfate were noted nationwide. Support was provided to the MOH to improve the availability of drugs and essential tracer products. Targeted assistance for supply chain issues included:  Support to the DSRSE for two workshops on RH, commodity planning, and security (co￾financed with UNFPA) to help reach the goal of a 27% contraceptive prevalence rate in 2015. Support was also provided to districts on the decentralized management of contraceptives through testing of a new approach to eradicate stocks-outs in four new districts, reaching 75 SDPs.  Capacity development of 160 staff (70 men and 90 women) in the regions (in collaboration with DSRSE and PNA) to improve logistics systems through supervision visits to 11 PRAs, 76 districts depots, seven regional hospital centers, five public health services, and 74 districts health centers. In year two, supervision visits were conducted in 11 PRAs, 73 district depots, depots of 14 regional health centers, two public health services and one regional hospital. IntraHealth supported the functioning of health services in public health posts, health centers, and regional hospitals by being involved in several activities including supporting EMOC through involvement in pilots, promoting infection control and provider safety through trainings, and implementing a model to improve the quality of services through strengthened partnerships between communities and health facilities (PAQ).  Referral system for emergency cases: According to the MOH policies for maternal health, regional hospitals and more advanced health centers are supposed to offer EMOC, as part of a broader system of emergency care for patients, while health posts are supposed to deliver basic EMOC services such as AMTSL, PAC, and referral (if danger signs are recognized). In order to contribute to the decrease in the high maternal, neo-natal, and child mortality rate, IntraHealth supported the Medical Assistance Emergency Service (SAMU) of the MOH, by developing a pilot project for decentralizing the referral and counter-referral system between the district and the health post levels. The pilot project was expected to provide support for obstetric and pediatric emergencies during mobile team field visits. During data collection it was noted in one sample that 34 patients were transferred from 10 local collectivities of which six were due to pregnancy complications. The emergency system has existing protocols, and tools (i.e., kits and emergency carts), but it was noted that there were insufficient emergency kits available at the PRA as well as a significant variance in the skillset of health providers. In year three, training was conducted to improve the working Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 85 conditions for emergency patient care. This began practices to address gaps in health personnel skills and develop/apply referrals and counter referrals, where possible.  Infection control and provider safety: Infection control and provider safety in the health facility environment of SDPs were focus areas of the HSI component. IntraHealth, through training and community partnerships, focused on promoting an alcohol-based hand sanitizer, a sterilizing agent for preventing and managing the spread of nosocomial infections and multi-resistant bacteria. In year two, IntraHealth trained 1,912 providers in infection prevention and environmental protection. According to the year three annual report, IntraHealth trained 529 providers on infection control and environmental protection in 167 public facilities and five private facilities in Louga, Diourbel, Kaolack, Tambacounda, Kédougou, Matam, Kolda, and Ziguinchor. Findings noted in the annual report related to hazardous practices and health safety risks revealed that providers did not consistently follow the steps to limit their exposure to blood and did not adhere to the protocols in using the hydro-alcoholic solution consistently for appropriate sanitation. IntraHealth also supported the creation of facilities that are actually producing hydro-alcoholic solution. The report did not indicate to what extent this problem persisted. To improve the production of the sterilizing agent, 24 pharmacists and lab technicians in Dakar, Louga, Touba, and Diourbel completed training on how to create the solution. Health committees and religious leaders were encouraged to adopt the use of the agent to prevent infection and promote improved health and hygiene practices at the community level.  PAQ Model: IntraHealth is implementing a PAQ model (Partenariat pour l’Amélioration de la Qualité des Services), through partner Siggil Jigeen, for improved linkages between communities and health facility services. This model was first used in Rwanda, and then in Senegal under the HSI component. This is a participatory approach which brings together representatives from health facilities and communities (e.g., health committees or individuals) to identify and address barriers to quality and use of services. For example, IntraHealth focused on improving ANC attendance, and mobilizing women and their families to attend the four ANC visits, starting with ANC 1 in the first trimester. IntraHealth emphasized community and facility-based provider counseling on the purpose/benefits of ANC, and highlighted why each visit was a critical step in fostering a woman’s healthy pregnancy. The PAQ model is being used to identify barriers for why women do not seek four ANC visits. Community-provider partnerships support the success of continuous improvement of quality of services under the PAQ model. In year two, IntraHealth established 276 PAQ teams around targeted SDPs. In the past year, PAQ trainers, different from the TutoratPlus actors, made 320 SDP visits to support the establishment of PAQ teams and PAQ committees, as well as the extension of SDP action plans to improve the quality of services for a total of 581 SDPs, according to the IntraHealth’s annual report. Training sessions focused not only on improving service provision, but also role playing and practicing, and as well as receiving and providing constructive feedback. A total of 46 were trained, 24 of which were women. The Siggil Jigeen Network continues to work with health committees to expand membership; establish their definition of quality and how it will be measured; and build capacity of health committees to define and improve quality through these trainings and follow-up support. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 86 The System for Exchanging Automatic Data (SEAD) is also assisting with improving the flow and oversight of health services with information shared through mobile phones. This tool is currently being used in 21 health districts following field testing. IntraHealth introduced SEAD as a pilot within the Department of Health and Social Information System (DSISS), the DSRSE, and the health district of Foundiougne. Following successful use, the system was scaled up and SEAD was introduced in 21 districts in three regions (i.e., Thiès, Ziguinchor, and Fatick), covering 382 SDPs or 29% of the country, according to the 2013 IntraHealth annual report. IntraHealth’s cooperative agreement states that they will use mobile technology to strengthen referral and counter-referral systems. The cooperative agreement noted that this system would be based on providers sending text messages to refer a patient to a hospital with their information. The hospital would then send a follow-up message to the referring clinic when the client checks in, advising about treatment that he/she received and necessary follow-up. This system could be instituted through Frontline SMS forms. During the evaluation, Team EY did not see an example operationalized, nor did service providers discuss this with interviewers as an intervention. USAID/Senegal confirmed that the SMS system for testing patient referrals will be initiated in year four. 6.4.2 Lessons Learned and Recommendations ► USAID/Senegal may want to provide support to the MOH to monitor the process and scale￾up of the SEAD system planned for next year especially regarding data collection and reporting. The pilot phase was completed by IntraHealth and the initial results were positive, according to the 2013 IntraHealth annual report. The use of SEAD data may possibly improve information for decision-making at the district level and regional levels. IntraHealth may consider monitoring different outcomes in the use of mobile technologies and phones to facilitate data collection and capture information in a more real-time capacity, as part of plans to scale-up across their many development programs. 6.5 Sub-component C: Improve Human Resource Management at Public Health Facilities 6.5.1 Findings and Analysis Human Resource Management According to the CDCS, the severe shortage of health providers undermines Senegal’s ability to meet the health needs of its population. Interviews with health providers indicated that while policies, standards, and norms forming the basis of quality assurance practices exist, the MOH was understaffed, and standardized, accountable, quality care provided at health facilities is inconsistent. To facilitate human resource management capacity building at public health facilities, IntraHealth supported the MOH in training the regional and district health teams. Trainings were led by MOH’s Division of Human Resources (DHR) training arm. The MOH collaborated with TFPs who were familiar with the needs of the regions to develop appropriate training plans. Training modules focused on performance management, health governance, management leadership, human resource management, management of financial resources, service management and organization, and M&E. Although originally intended to support all 14 regions, IntraHealth supported three regions in the south (i.e., Kolda, Ziguinchor and Sédhiou), two regions in the east (i.e., Tambacounda and Kédougou), and Dakar in the west for a total of Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 87 six regions. Approximately 26 regional health team members (13 men and 13 women), district health teams, and the EPS around Saint-Louis were trained on human resources management. Trainers teaching the module on service management and organization provided support to 116 SDP (114 public and four private) in eight regions.  Human resource software suite: IntraHealth’s focus for this sub-component was to impact human resource issues at the district and facility level - improving individual health provider productivity, performance, job satisfaction, allocation of providers to different facilities, and retention within a context of improved HR and clinic management. The early success in the introduction of human resource for health software developed by IntraHealth is noted for supplying health managers and practitioners with the information they need to assess human resource problems, plan effective interventions, and evaluate those interventions. IntraHealth encouraged the use of Information Communication Technology (ICT) to monitor training and performance. Multiple interventions utilizing ICT were conducted or initiated with the objective of improving the availability and utilization of information for decision-making at various levels of the health pyramid, including along the supply chain to control drug and contraceptive stock-outs. IntraHealth, utilized technology to contribute to management improvements at public sector health facilities. Coaching on Organizational Services According to interviews with service providers supported by the IntraHealth program, health personnel in the regional offices were satisfied with the training they received and stated that they are able to better execute and manage their work. Providers interviewed in the districts were pleased with their coaching and supervisory skills gained through the trainings. At the health huts, BG, Associations Sportives et Culturelles (ASC), volunteers, relais, and CBO representatives were pleased to be receiving coaching from the Chief Nurse, Health Post (ICP). They stated that the method of learning-by-doing and working together was effective and promoted improved collaboration between the facility and the community.  Performance assessments/reward systems: The training on performance management conducted in the districts of Diamniadio, Rufisque, Popenguine, Goudomp, Ziguinchor, and Diourbel led to the establishment of performance assessments and a reward system in these districts, according to the IntraHealth 2013 annual report. These districts assessed the performance of the SDPs utilizing an assessment and reward system based on the TutoratPlus platform. The rapid assessment indicated the extent to which gaps existed and services were missing. These were utilized to establish competencies to perform new services and improve the quality of existing services. Coaching reports were used to facilitate the development of competences and strengthen weakness in job performance. In follow-up assessments, the tool was used to designate the highest performing SDP in the health area covered. Well performing SDPs received awards during public celebration for all SDPs, service providers, local and administrative authorities, and health committee members. For example, in the district of Rufisque, the health posts of Diorga and Tivaouane Peulh were awarded because the majority of gaps identified during the TutoratPlus needs assessment were resolved and services were strengthened, which for the most part did not exist, according to the IntraHealth 2013 annual report. During the interview, Team EY noted that these awards are viewed as an honor by health staff and served as a motivation tool for staff in other districts. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 88 6.5.2 Lessons Learned and Recommendations ► With the expansion of the iHRIS software system to support the MOH, USAID/Senegal should consider conducting an independent assessment of staffing needs and develop a strategy and action plan for staffing of the SDPs in the five regions where the iHRIS is operating. The assessment should include how the software is being used and what additional manpower, infrastructure, or other needs will enhance its use. It is also suggested that USAID/Senegal consider conducting an assessment of iHRIS utilization in the five regions prior to scaling up in other regions. ► USAID/Senegal may want to consider scaling up incentive programs, as the qualitative interviews indicated that incentivizing staff, regardless of whether the incentive was financial, resulted in a positive degree of motivation. Models like PBF can be expanded to include more health facilities across the country, and should be considered as one of the incentive programs to scale-up. USAID/Senegal may want to consider, including additional benefits that are cost effective, like mobile phone minutes, to the SDP reward system, in order to keep the program interesting, innovative, and engage the staff’s interests. 6.6 Sub-component D: Outreach to Private Health Facilities 6.6.1 Findings and Analysis IntraHealth’s primary objective in sub-component D was to link the public and private sector, establishing partnerships that allow for-profit and not-for-profit private entities to enhance their contribution, through public sector inputs for training, standards, performance improvement, monitoring, and commodities. Public and private sector linkages were strengthened through the signing of 144 MOUs by the end of the year, according to the 2013 IntraHealth annual report. In year two, 43 MOUs were signed between districts and private SDPs. Achievements include the training of for-profit providers in the integration of the integrated package of services. In partnership with the Association of Corporate Paramedics, IntraHealth trained 31 private providers (including eight women) from 27 private companies on counselling and communication techniques. This training utilized various role playing exercises and techniques to help providers to acquire the skills necessary to convey key information about the integrated package of services. The HSI component also provided training to 91 private pharmacists in the following areas of care: malaria, treatment of diarrhea with ORS-zinc, FP, and acute respiratory infections in children. Furthermore, private sector service delivery sites provided care in treating pediatric patients with acute malnutrition, malaria, and diarrhea (ORS with zinc) and improving the quality of prenatal visits, and contraception services. According to the year two annual report, 89 private sector establishments enrolled in TutoratPlus as compared to the target of 39. This suggests opportunities to continue efforts in the provision of comprehensive services in these sites, strengthened by the TutoratPlus model. An advocacy visit by MOH leaders was conducted with business leaders to disseminate information on the integrated package of services. The Division of Private Health facilities with the technical and financial support of IntraHealth organized the meeting and promoted the availability of the integrated package of services in for profit facilities. These health promotion forums targeted private providers in the regions of Kédougou, Tambacounda, Kolda, and Ziguinchor with 72 heads of private practices participating. Interviews showed that subsequent advocacy meetings and events held with private Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 89 sector entities appear to have some value in promoting the understanding and adoption of the integrated package of services in private health centers and facilities. High level meetings, joint provider trainings and technical exchanges among private and public sector providers were initial activities implemented to strengthen the linkage across the two systems. These efforts illustrate how USAID/Senegal could play a complementary role in supporting the GOS oversight of the private sector. According to the year two IntraHealth annual report, IntraHealth, in partnership with association of private pharmacies and private sector providers in the region of Dakar, expanded the integrated package of service with a focus on contraceptive technology. For example, 60% of pharmacists expressed interest in more comprehensive training methods following a regional training in Dakar (67 pharmacists were trained: 32 males and 35 females) on an integrated package of services. The component collaborated with the HCP component led by ADEMAS to identify ways to better engage pharmacists in the offering of an integrated package of services since they are working with pharmacies and private sector partners for the distribution and sales of socially marketed health commodities that are linked to the services supported by IntraHealth. 6.6.2 Lessons Learned and Recommendations ► USAID/Senegal and IntraHealth may want to consider forming networks of private providers to serve as champions and thought leaders on how the sectors can support each other in the management of care for Senegalese communities. For example, private nurse midwife associations in a number of countries have played a large role in complementary care and building on their success in maternal, neonatal and child health. In Botswana and Kenya, for example, they have served as an intermediary between the facility-based and home-based care, visiting clients in the community who cannot access emergency and other services. ► USAID/Senegal and IntraHealth may want to consider supporting the GOS in their role of oversight of the private sector providers and supporting their contributions to bridge the gap in services for communities that access them and those that frequent specific private outlets (e.g., stand alone, mobile pharmacists). A mapping of private sector services and providers could inform the strategy of who to engage and what types of partnerships are the most beneficial for a particular community. 6.7 Response to Evaluation Questions The table below provides information to respond to the component-specific evaluation questions as stated in the scope of work. It is not intended to be exhaustive, but rather aims to highlight notable successes; constraints and challenges that have been experienced during implementation by IntraHealth; and key interventions that may be added, continued, or removed within Component 2. Team EY and USAID/Senegal agreed that this analysis would be conducted by component rather than by individual sub-components. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 90 Table 22: Component Table Component Specific Question Analysis To what extent have the components achieved their objectives? Overall the HSI component is achieving some of its objectives and meeting 14 of the agreed-upon indicators as stated in their cooperative agreement based on the data received. Overall, a key takeaway from the evaluation was the opportunity for a greater focus on the quality of implementation. The HSI interventions are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services. In addition, they support USAID/Senegal’s high-level indicators: EPI coverage, number of deliveries with a skilled birth attendant, and CYP. Given that this evaluation was focused on two years, Team EY did not evaluate the impact of IntraHealth interventions but reviewed the USAID/Senegal strategy as a whole. To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component?  TutoratPlus Model Implementation: The TutoratPlus model is a site-based mentoring and performance-improvement process, which mobilizes essential actors in an effort to identify shortfalls in both public and private sector service delivery, clinic functioning, and individual/clinic performance. Qualitative data gathered from IntraHealth staff in the field indicates that capacity building training is scheduled at the right time in order to meet the demands of service providers, and therefore better received. One of the success factors contributing to achievement is the coverage rate of the intervention. It currently represents 80% of the country’s health districts, which exceeded planned targets.  Training on childhood illnesses: IntraHealth contributed to the implementation of the CSNP through the training of 170 providers. The focus of the training included the provision of services to manage childhood illness in six regions. The DAN also organized training sessions comprised of 247 health workers. According to qualitative data, trainings were perceived to raise the level of competency or knowledge of CHWs who work at the health post to support the ICP to provide better care for those under-five affected by diarrhea, a major contributing factor to child mortality. For example, in a focus group discussion with CHWs in Sédhiou, it was noted that they believed they were better able to care for uncomplicated simple diarrhea (i.e., attributable to malaria or dehydration), facilitating the initial delivery of basic medication, and supporting the treatment of pneumonia. One of the success factors contributing to achievement was that those trained received the practical support they needed to help them do their day-to-day jobs better.  Training on acute malnutrition: In the regions of Sédhiou and Kédougou, where acute malnutrition cases were high, service providers that participated in the training stated that adequate time was devoted to helping staff understand how to use the WHO’s Anthro Software for better monitoring of child growth and detecting cases of acute malnutrition.  Coaching of organizations/individuals: According to interviews with service providers supported by the IntraHealth program, health personnel at the regional level were satisfied with the training they received and stated that they are able to better execute and manage their work. Providers interviewed in the districts were satisfied with their coaching and supervisory skills gained through the trainings. One health provider in Matam noted that “we appreciate the coaching because it shortens our work. A job that took us one hour now takes 30 minutes. It has improved the quality of our work.” Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 91 What are the constraints and challenges that have hindered successful implementation of each sub-component, and how has the IP dealt with those challenges?  Inconsistent availability of commodities and equipment: Based on qualitative interviews, Team EY’s understanding is that there are concerns regarding the availability of commodities at the health facility as well as the maintenance and repairs to equipment. Several health facilities reported having broken refrigerators for over three months as well as other essential equipment which had fallen into disrepair for over a year. Additionally, the lack of incinerators or knowledge on how to dispose of biological waste was brought up in several interviews. IntraHealth is aware of the issues regarding stock-outs of commodities and has worked with the MOH to make improvements over the past two years. For example, IntraHealth has worked with the DSRSE and PNA to conduct semi-annual logistics supervision visits across the country. These visits focus on assessing the stock situation of essential tracer drugs and identifying challenges.  Constraints affecting the TutoratPlus implementation: The major barrier noted with the implementation of the TutoratPlus model is related to staffing issues. USAID/Senegal, in its leadership role, should continue supporting IntraHealth to advocate for and support the MOH in filling clinical staffing gaps throughout the focus regions for this component. Are there interventions that should be added or removed?  Focus project interventions to core set of activities: In qualitative interviews with IntraHealth regional staff, several challenges emerged including the following: 1) the project was too “ambitious” and needed to take into account the human resource constraints that exist in the field, 2) concerns with the quality of technical assistance provided to facilities, which affects the quality of clinical services provided, especially for infection prevention and environmental protection; and 3) lack of community involvement and engagement. In interviews with regional and district health teams as well as facility-based providers there was an appreciation for the support that IntraHealth provides. However, the quality of services being provided at health facilities varied depending on the region. While IntraHealth was trying to resolve the challenges in program implementation, USAID/Senegal might want to consider conducting a more in-depth analysis of IntraHealth’s activities per region and district to assess whether it would be more cost￾effective and improve the quality of interventions if the scope of services offered were more focused, and specific activities were removed. It is important to note that Team EY did not conduct a detailed analysis of all of IntraHealth’s activities due to the scope and duration of the evaluation and therefore this would need to be further analyzed to verify and validate the concerns noted above. Are there changes that could be made to improve performance?  iHRIS software assessment: The pilot phase of the electronic iHRIS was noted in interviews for supplying health managers with information to assess human resource constraints and to subsequently plan and evaluate interventions. With the expansion of the iHRIS software system to support the MOH and the RHMT, USAID/Senegal should consider supporting the MOH to conduct an independent assessment of staffing needs and develop a strategy and action plan for staffing of the SDPs in the five regions where the iHRIS is operating. The assessment should include how the software is being used and what additional manpower, infrastructure, or other needs will enhance its use.  Increase the role of the private sector in implementing the integrated package of services: The Division of Private Health facilities with the technical and financial support of IntraHealth targeted private providers in the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 92 regions of Kédougou, Tambacounda, Kolda, and Ziguinchor. During interviews it was noted that subsequent advocacy meetings and events held with private sector entities appear to have some value in promoting the understanding and adoption of the integrated package of services in private health centers and facilities. USAID/Senegal and IntraHealth may want to consider forming networks of private providers to serve as champions and thought leaders on how the sectors can support each other in the management of care for Senegalese communities.  Improved alignment of services to geographic needs: It appears that overall DHS 2012 health indicators are weakest in regions where the integrated package is not being implemented. The two regions of Matam and Tambacounda, specifically, show low overall health indicators that could benefit from the integrated package of services (rather than only the malaria package that is currently being implemented in these regions). USAID/Senegal may want to consider shifting and prioritizing the integrated package of services currently provided by IntraHealth to geographic areas with weaker indicator results. 6.8 Data Sources In addition to sources cited in Annex G: Bibliography, data collected in the field was used for analysis of this component, including: in-depth interviews with the COP, RB coordinators, key MOH stakeholders, Regional and district health officers, regional and district coordinating offices, community-based health insurance managers, Directors of the BREIPS, service providers at health facilities; services providers at health huts, pharmacists, private firms; as well as Focus Group Discussions with CHWs, CH committees, community-based health insurance beneficiaries, and associations of PLWHA. Data collection tools for field interviews are available in Annex I and Annex J. 7.0 COMPONENT #3: COMMUNITY HEALTH 7.1 Background USAID/Senegal awarded the ChildFund consortium a five-year $40M USD cooperative agreement in 2011 to implement the CH component of the USAID/Senegal Health Program. The component was designed to contribute to the achievement of IR 1: Increased availability of an integrated package of quality health services, IR 2: Improved health seeking and healthy behaviors, and IR 3: Improved performance of the health system. The current program builds off the investments made in the first phase of CH implemented between the years of 2006 to 2011. The previous project focused on implementing a package of integrated services to impact the public health of the Senegalese population focused on FP/RH, MCH, nutrition, malaria, TB, and HIV/AIDS (awareness) at community-based health huts. The second phase of the CH project is focused on continuing to provide the package of services from phase one, but to a larger geographic area and with greater emphasis on sustainability. The component is working in all 14 regions and 72 health districts. ChildFund International leads the consortium and is responsible for the overall project coordination and management. The other consortium members as stated in their original cooperative agreement are Africare, Catholic Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 93 Relief Services (CRS), Plan International USA (Plan), WV, Enda Sahel/West Africa (Enda Graf), and Enda Santé. The project is organized into three sub-components:  Sub-component A: Improving the quality of and access to information, products, and services at health huts and outreach sites.  Sub-component B: Fostering community ownership and improving linkages and collaboration between the regional, district medical teams, development partners, and community level actors.  Sub-component C: Fostering national MOH and other sector ministry ownership for CH and harmonizing the linkages with national policy initiatives. Figure 16: CH Overview As seen in the Figure 16 above, the CH program is designed to support the health huts system, which provides basic SSP outside of the facility-based structure. Health huts are staffed by a volunteer CHW, and TTBAs. The current CH project provides services referred to as the “minimum integrated package,” which include the management of diarrhea with ORS-zinc, management of pneumonia with cotrimoxazole, prenatal and neonatal care, initial offer of contraceptive pills, and rapid diagnosis of malaria and treatment with ACT. According to MOH health policies, health huts should be a minimum of five kilometers from the nearest health post to encourage populations to utilize them as the first entry point for care-seeking services. The health huts are supported by community mobilizers (relais), which include health promoters and home-based care visitors, community members, and the CH Management Committee. The CH committees are supposed to manage the health hut, including aspects related to financial management. Additionally, they support the health hut in leveraging community and local government support. Outreach sites exist where there are no health huts, either in remote rural areas where health huts do not exist or in urban areas, especially Dakar, where health huts are not part of the overall health system. They function as a base from which volunteer outreach workers carry out community communication and health promotion and supervise CHWs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 94 Table 23 below shows the standard indicators related to the CH component in the first two years compared with targets. Although Team EY understands that the ChildFund consortium provided its year three annual report to USAID/Senegal in November 2014, Team EY did not receive the report in time to include the 2014 indicator data into the table below. However, information derived from the report that was verified by data point/site interviews and corroborated by alternative sources was utilized in our analysis. Table 23: USAID Indicators for CH # Indicator Target Actual Target Actual 2012 2013 RH/FP Indicators 1 CYP in USG supported programs 5,482 1,849 9,602 7,696 2 CYP with oral contraceptives (birth control pills) * * 3,120 2,986 3 CYP with cycle beads /Standard Day Method (SDM) * * 2,732 1,431 4 CYP with condom * * 3,361 2,682 5 CYP with injectable * * 390 597 6 Year one: Number of cycles of pills distributed Year two: Number of contraceptive methods distributed by oral contraceptives 39,000 12,577 46,798 44,795 7 Year one: Number of cycle beads distributed Year two: Number of contraceptive methods distributed by SDM 999 361 1,821 954 8 Year one: Number of condoms distributed Year two: Number of contraceptive methods distributed by condoms 222,720 82,022 403,320 321,909 9 Year one: Number of injections administered Year two: Number of contraceptive methods distributed by injectables 1,440 342 1,560 2,386 10 Year one: Percent of USG-assisted SDPs that experience a stock-out at any time Year two: Number of SDP that reported shortage of contraceptive methods during the report period 0% 66% 838 654 11 Year one: Number of USG-assisted delivery service websites providing FP counseling and/or services Year two: Number of huts and sites supported with USAID funds providing counseling and FP services 3,463 3,256 3,258 3,890 Malaria Indicators 12 Year one: Number of health workers trained in malaria laboratory diagnostics (rapid diagnostic tests or microscopy) with USG funds Year two: Number of community stakeholders trained on TDR for malaria 478 524 409 800 13 Number of community actors retrained on TDR for malaria * * 2,128 812 14 Year one: Number of health workers trained in case management with ACTs with USG 478 524 409 800 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 95 # Indicator Target Actual Target Actual funds Year two: Number of people trained on fever management with ACT 15 Number of people retrained on fever management with ACT * * 2,128 812 Nutrition Indicators 16 Number of zero to five year old children reached by nutrition programs (e.g., screening, weighing) * * 323,420 1,042,210 17 Number of people trained in child health and nutrition through USG-supported health area programs 1,647 4,745 1,575 5,702 MCH Indicators 18 Year one: Number of postpartum/newborn visits within three days of birth in USAID￾assisted programs Year two: Percentage of newborns receiving postnatal care two days after birth 39,871 20,476 38,003 47,905 19 Year one: Number of children under-five with diarrhea treated with Oral Rehydration Therapy (ORT) Year two: Percentage of 0-5 year old children with diarrhea receiving ORS (only) or ORS-zinc 111,659 111,360 87,552 52,162 20 Number of children with pneumonia taken to appropriate care 55,680 * 48,744 * *Data not available Note: Target and actual data included in the table above is derived directly from annual reports provided by IPs. Also, for some indicators, the description changed between year one and year two, and therefore both descriptions have been added. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 96 The data from the 20 indicators selected above are derived from the ChildFund consortium’s annual reports for year one and two. These indicators were selected as a sample because they are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services, IR 2: Improved health seeking and healthy behaviors, and IR 3: Improved performance of the health system. The indicators in the table represent key activities/interventions across CH with a focus on malaria, RH/FP, nutrition, and MCH. The select indicators are an important subset of the ChildFund consortium’s total number of indicators because they address critical PE questions of whether clients are receiving a comprehensive package of services and whether health facilities and providers are equipped to offer these services. Overall, the ChildFund consortium showed positive results of improving indicators from year one to year two. An analysis for the year two indicators can be found below. Additional information on indicator achievement status can be referenced in Figure 17 below. ► Targets Met: The results indicate that the ChildFund consortium met and exceeded targets for two of the four malaria related indicators including: 1. Number of community stakeholders trained on TDR for malaria. 2. Number of people trained on fever management with ACT. These indicators exceeded the expected results and may be attributed to training of all CHWs and TTBAs in health huts as opposed to just one community actor per new health hut as was originally planned. For RH/FP, the ChildFund consortium nearly met, met, or exceeded targets for nine of the 11 indicators. These include: 1. CYP in USG supported programs. 2. CYP with oral contraceptives. 3. CYP with condoms. 4. CYP protection with injectables. 5. Number of contraceptive methods distributed by oral contraceptive type. 6. Number of contraceptive methods distributed by condoms. 7. Number of contraceptive methods distributed by injectable type. 8. Percentage of SDPs that reported a shortage of contraceptive methods during the reporting period. 9. Number of huts and sites supported with funds from USAID/Senegal that provide counseling and FP services. From a nutrition indicator perspective, the ChildFund consortium showed positive results by exceeding initial targets for the two indicators to include: 1. Number of zero to five year old children reached by nutrition programs. 2. Number of people trained in child health and nutrition through USG-supported health area programs. The year two annual report noted that the number of children under-five reached by nutrition programs may be inaccurate due to double counting, possibly affecting data quality. The ChildFund consortium continues to address this issue through a focus on coaching in data Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 97 cleaning and steps to validate reliabilitywith staff and community actors. Lastly, from a MCH indicator perspective, the ChildFund consortium exceeded the percentage of newborns receiving postnatal care two days after birth and almost met the target for percentage of zero to five year old children with diarrhea receiving ORS (only) or ORS-zinc indicator. According to the year two annual report, significant efforts have been made through the community-based diarrhea management strategy that has showed positive trends in improving the results. Team EY noted that USAID/Senegal may want to review this ChildFund consortium target to set it higher since it appears that the actual results are exceeding targets beyond an acceptable range (i.e., 10% to 20%) in many of the indicators. ► Targets Not Met The two malaria indicators with low performance included: 1. Number of community actors retrained on TDR for malaria. 2. Number of people retrained on fever management with ACT. This indicates that while the ChildFund consortium exceeded targets for initial training, they may be lacking processes to improve continuous or refresher training. The two FP/RH indicators with lower performance include: 1. CYP with cycle beads/SDM. 2. Number of contraceptive methods distributed by SDM. According to the ChildFund consortium year two annual report, a shortage rate of nearly 10% impacted meeting some of the RH/FP targets. The annual accumulation shows that 66% of health huts had at least one day of stock shortage during the year. These contraceptive commodity shortages are related to the weakness of the health system supply mechanism, which USAID/Senegal is aware of and is discussed earlier in the report. ► Data Not Received: Team EY did not receive full reports after 2013 to include in the table. Additionally, a number of actual results were unavailable or unreported in year one. Only one result was not reported in year two: number of children with pneumonia taken to appropriate care. Team EY received no additional information from the work plans, annual reports, or interviews to substantiate the reasons for missing standard indicators; however, it is evident that the ChildFund consortium improved their process of tracking and reporting information in year two as only one indicator was not reported on ascompared to eight in year one. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 98 Figure 17: CH Indicator Status 7.2 Key Findings and Recommendations/Benefits Overall, the CH program is achieving some its objectives and meeting some of the agreed-upon indicators as stated in their contract. Key findings and recommendations are provided in further detail in Section 7.3 – 7.5. Table 24 below summarizes the most important key findings and recommendations/benefits relevant to strengthening the overall quality and delivery of the CH component. Table 24: CH Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefits 1 The functionality of the health huts and their outreach activities appear to be highly dependent on the consistent support of the ChildFund consortium. There is consensus among key stakeholders, including the MOH (regional and district health offices), service providers, and CH Management Committees that without the support, the quality and continued maintenance of services will decline or in some instances cease. Recommendation: USAID/Senegal may want to continue to support the current services provided at health huts, but not expand to additional sites in order to focus on increasing the quality of services. Benefit: The continued support will assist with the quality and maintenance of services until the local government and elected officials can sustain services. By not expanding services, the ChildFund consortium can focus on skills transfer, capacity building, and sustainability measures to reduce dependency on the ChildFund consortium. 2 The quality of services at the health huts appear to be compromised by inconsistent availability of medical supplies, poor infrastructure, and lack of electricity and water. Project reports indicate that some health huts are in such disrepair that they are no longer functional (i.e., roof has collapsed) and that consultations lack privacy as the consultation rooms have no enclosure. To solve these problems, communities built Recommendation: USAID/Senegal may not want to expand the number of services in the integrated package in order to first improve the quality of existing interventions and increase support for supervision and monitoring. Benefit: The focus on existing interventions may allow the ChildFund consortium to improve the existing quality of services (i.e., fully functional health huts) and develop a better process for Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 99 # Key Findings Recommendations/Benefits doors and additional rooms where possible. It was reported that villagers call on their relatives who live abroad to send money for construction support and several health huts leveraged funds from local Community￾Based Organizations (CBO) to support the installation and cost of electricity. supervision and M&E of services. 3 The commitment from local government authorities to take over and support existing CH activities appears to be low. As USAID/Senegal transfers financial responsibility to the communities for the health huts, they become dependent on local government authorities for the majority of their resources. Recommendation: USAID/Senegal may want to consider placing more emphasis on advocacy efforts targeting locally elected officials to create a more favorable political environment for health huts. USAID/Senegal may also want to consider this as a priority agenda item for the donor coordination group to discuss solutions on how the GOS can increase financial support to health huts and CH services in general. Benefit: This approach may increase locally elected officials commitment and interest in supporting and allocating funds to health huts once USAID/Senegal resources are withdrawn. 4 The lack of incentives to support CHWs appears to be a barrier to the consistent availability of support in their communities. CH is based primarily on volunteer CHWs and outreach workers who are dedicated, but need to earn an income. Recommendation: The MOH, USAID/Senegal, and other donors may want to develop more effective strategies to incentivize and motivate CHWs and outreach workers. Benefit: This approach may increase consistent volunteer availability by helping to incentivize CHWs and outreach workers (e.g., financial, recognition, training). 7.3 Sub-component A: Improving the quality of and access to information, products, and services at health huts and outreach sites 7.3.1 Findings and Analysis The integrated package of services provided at health huts includes treatment and prevention of malaria with ACT and ITNs, treatment of diarrhea with zinc and ORS, treatment of ARI, pre￾and post-natal services, nutritional supplements for under-fives, and the availability of FP commodities. At the end of the third year of implementation, the ChildFund consortium expanded the availability of the integrated package of services to 2,303 health huts (89 of health huts were then transformed into health huts), which represented 103% of their planned target for year three. Additionally, they increased their support to 1,649 health sites, which represented 91% of their planned target (target 1,806). Twenty-four health huts and 15 health sites graduated Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 100 to health posts17. Observations at the 14 health huts visited by Team EY indicated that the service providers understood and could explain simple protocols provided for each element of the package. However, it is important to note that the health huts visited were close in proximity to a health post and received frequent supervision visits by the ChildFund consortium staff. According to the ChildFund consortium’s year three annual report, all 2,214 health huts were monitoring pregnancies and deliveries. The report data states that 97% of mothers and their newborns received a postnatal visit from either the TTBA or CHW within three days of delivery. According to focus group discussions, Team EY held with CHWs, in circumstances where health huts did not have birthing facilities and were distant from health posts, there was a tendency toward home deliveries attended by a TTBA. One of the key reasons for this was the lack of access to consistent means of transportation to a health facility for delivery and concerns about emergencies during labor which began at home. Methods of finding transportation included using horse and buggy, trying to wave down passing vehicles on the side of the road, or making arrangements with a villager (or relative of a villager) who had a vehicle (in some instances patients would pay a fee). Mobile phones are commonly used to resolve communication problems, and in the rare occasion that a health post offered functional ambulance services, villagers would use it. There was an increase in the availability and use of oral contraceptives for FP from the onset of the project to the time of the evaluation. At the end of year two, only 48.7% of health huts offered oral contraceptives, compared to the end of year three where 90% of health huts offer several methods of FP. According to the ChildFund consortium annual report, the significant increase is partly attributed to the decreased number of stock-outs in the past year. Tracking of oral contraceptive use is now possible with the introduction of client registers indicating provision of oral contraceptives. With improved availability, the ChildFund consortium was able to reach 87.24% of their targeted number of users. The CHWs and TTBAs in all the health huts were trained in the treatment of diarrhea, malaria, and counseling for oral contraceptives. Training on the use of misoprostol was provided to TTBAs and CHWs in the prevention of postpartum hemorrhaging in 28 health huts in the regions of Kolda and Thiès. The trained TTBAs who were interviewed by Team EY were interested in administering medication to improve their services. The MOH recently (August 2014) provided full authorization for the use of misoprostol at the community level via the health huts, and there is an indication that the TTBAs could integrate the intervention into their services. Additionally, it was noted that many of the CHWs are providing specific information on the integrated package to the catchment area population. However, there are no studies to confirm the positive impact of their health promotion on the health status of the population. Field data suggests that the quality of services of health huts depends on several elements which vary from place to place and include: 1) frequent supervision of health huts, which is often correlated with close proximity to the health post and to the base from which the ChildFund consortium field staff works; 2) strong commitment of the CH Management Committee; 3) 17 “Programme Sante USAID, Sante Communautaire Phase ii, Plan D’Action, An 4, Octobre 2014-Septembre 2015,” Childfund, October 2014. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 101 positive cultural attitudes toward volunteers and a sense of community solidarity; and 4) increased availability of resources within the catchment area of the health hut. The quality of services at health huts is compromised by inconsistent availability of medical supplies, poor infrastructure, and lack of electricity and water. Additionally, some health huts lack privacy for consultations and have only one room with no enclosure. Project reports indicate that some health huts are in such disrepair that they are no longer functional (i.e., roof has collapsed). To solve these problems, communities built enclosures and additional rooms where possible. It was reported that villagers call on their relatives who live abroad to send money for construction support and several health huts leveraged funds from local CBOs to support the installation and cost of electricity. Communities try to replenish medical supplies with the money they receive from the consultation fees and from contributions by the CH Management Committee. Additional challenges as noted in Table 25 below include: Table 25: CH Constraints and Challenges Constraint/Challenge Description There was a lack of amoxicillin for ARI at the community level, which is primarily due to the lack of MOH administrative authorization The ChildFund consortium plans to accelerate the process of making amoxicillin available by developing protocols for treatment of ARIs with amoxicillin by CHWs. They will collaborate with UNICEF and the DSRSE to define the methodology for the initial introduction of amoxicillin into the health huts, track the transport of product, train CHWs in treatment of ARIs, and organize supervision for CHWs providing this service. Scaling-up the use of misoprostol Interviews with some of the medical personnel indicated concern over the ability of TTBAs to correctly administer misoprostol. This will pose a challenge and require close monitoring for quality control when misoprostol is first introduced at health huts beyond the initial pilot sites until there is confidence that TTBAs are administering it correctly. Lack of understanding of the new oral contraceptive tracking records and patient side effects While the new tracking tool for oral contraceptives has provision for tracking drop-outs and refills, Team EY noted that some of the TTBAs do not understand that aspect of the register and may not be filling it out correctly. While TTBAs and CHWs were trained to provide oral contraceptives, it is not clear that they know how to manage and mitigate patients who present with side-effects. Lack of supervision between the health post and the health huts According to the project, the health huts are supervised and provided outreach services monthly by the head nurse of the health post. However this is not occurring consistently. The two key reasons given during interviews with the nurses were that they lack transportation or the time due to the quantity of patients and responsibilities they are obligated to perform. Delays in commodities and quality of storage space CHWs do not consistently provide drug orders to health posts on a monthly basis, resulting in the delay of commodities. Reasons given by the CHWs were lack of transport or recognition that this responsibility is a priority. The ChildFund consortium field staff is trying to address these gaps by taking the orders for supplies themselves, or accompanying the CHW to the health post. When Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 102 Constraint/Challenge Description health huts are fully transferred to the responsibility of the community, the concern remains as to how this gap will be filled. Observations revealed that the quality of commodity storage space varied significantly, from an orderly and clean cupboard to a dilapidated carton kept under a cot in a small, locked, dirty room. The final evaluation of the first phase of the CH project had similar observations. Inconsistent availability of volunteer health resources Data collected through interviews with TTBAs, CHWs, volunteer outreach workers, and members of the CH Management Committees revealed several challenges in staffing the health huts. The TTBAs and CHWs are volunteers who on an inconsistent basis receive a very small amount of money (3,000-4,000 CFA francs/$6-$8 USD) at the end of the month from the sale of drugs and a 200 CFA francs ($.40 USD) consultation fee. Most volunteers are women with household responsibilities and in many cases also contribute to earning household revenue through farming. Therefore, they are not able to keep the health huts opened consistently on a daily basis and would open them on an irregular basis, with no specific schedule. Since the clients could not count on them being open at any specific time, they were discouraged to use it at all. A solution provided during interviews with TTBAs and CHWs was to keep the health hut open for only a few hours a day at specific times and then to be available by mobile phone should someone need their services. 7.3.2 Lessons Learned and Recommendations • The functionality of the health huts and their outreach activities appear to be highly dependent on the consistent support of the ChildFund consortium. There is consensus among key stakeholders, including the MOH (regional and district health offices), service providers, and CH Management Committees that without the support, the quality and continued maintenance of services will decline, or in some instances, cease. The sustainability of the health huts depends on the commitment of local elected officials and local government, which is low. The dependency of the health huts on the ChildFund consortium leaves them vulnerable if support is withdrawn or decreased. This vulnerability has direct consequences on the benefits provided to the population that relies on the health huts for primary health services. • USAID/Senegal may want to continue to support the current services provided at health huts, but not expand to additional sites. They may want to consider focusing on increased capacity building, skills transfer, and sustainability measures to reduce dependency on the ChildFund consortium’s support. In order to improve the enabling environment for sustainability, it is necessary for the MOH and the next phase of the USAID/Senegal Health Program to strengthen interventions to increase commitment by local government offices, government staff, and elected officials for CH. Additionally, financial management capacity will need to be strengthened to help the appropriate allocation and governance of funding for CH services. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 103 • USAID/Senegal may not want to allow any service additions to the integrated package in order to improve the quality of existing interventions. USAID/Senegal may want to focus its attention on the quality of existing services throughout the existing health huts system. Since the existing field staff of the ChildFund consortium cannot fully supervise and monitor the number of health huts already in existence, it may be more effective to develop the full functionality of those existing by the end of the project. • The program demonstrated that it is possible for TTBAs and CHWs with limited education to learn to administer basic medical services correctly, as outlined in the integrated package, by following simple protocols. As such, USAID/Senegal may want to continue to support the MOH in developing and piloting simple protocols for administering basic medical services at the health hut level, and provide frequent in-service refresher trainings and constructive supervision to maintain quality. Given the human resource constraints in the health sector, task shifting strategies that use TTBAs and CHWs to decrease the burden on doctors and nurses could lead to improved care provided at the community level. • Activities that depend on the collaboration of the health posts and the health huts do not appear to function effectively. For example, monthly supervision and the provision of outreach services to the health huts by the health post nurses are inconsistent. The system of the CHWs ordering commodities through the health post is not working well and at times results in stock-outs even when medical supplies are available at the central level. One reason cited is difficulty in accessing transportation. To assist with a more effective collaboration between the health post and the health hut, USAID/Senegal may want to request the ChildFund consortium conduct a deeper analysis of the relationship between the CHW and ICP in order to determine if there are underlying factors other than transport and patient overload that may be causing the problems. Contributing factors could include: 1) a general lack of motivation and interest in the health hut on the part of the nurse; 2) possible personality conflicts between the nurse and the CHW; or 3) the fact that IntraHealth supervises the IP and the ChildFund consortium supervises the CHW. Based on the findings, the ChildFund consortium can work with the CHWs and IntraHealth to develop practical solutions. Where there is PBF in health posts, USAID/Senegal may want to consider including health hut supervision as one of the standard indicators. This could strengthen the sustainability of the health hut’s quality of services after transfer. A suggestion was made that the community field agents, who are currently the ChildFund consortium field staff, could be replaced and taken by the MOH for sustainability of the health huts. However, Team EY does not think this is practical as it would mean adding new staff to the MOH system when the MOH is not planning to allocate sufficient funds to pay a regular salary to the CHW as the health huts become more fully integrated into the MOH system according to the new CH strategy. It may be more sustainable to support increasing stipends to the CHWs rather than adding new personnel to the MOH. The ChildFund consortium may want to consider exploring opportunities to integrate the use of mobile technology to improve the commodity procurement and distribution of commodities at the community level. This may increase coordination and consistency between health huts and health posts. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 104 7.4 Sub-component B: Fostering community ownership and improving linkages and collaboration between the regional, district medical teams, development partners, and community level actors 7.4.1 Findings and Analysis A key target in the second year of the CH project was the transfer of 270 health huts. However, only 74 health huts were transferred which represented 27.4% of their planned targeted. At the end of the third year, a total of 386 health huts out of 479 were transferred, representing 78% of the target completed. The CH component established a process for the transfer of responsibility of health huts to the community, local government, and the MOH. The methodology guide and the tools used to transfer health huts were finalized and shared with the various stakeholders in all regions, and each developed a transfer plan. The objective of this process is to gradually transfer responsibility until the community and the MOH are ready to assume full responsibility for the continued functioning of health huts and the outreach health promotion. This is a complex plan which, if done correctly, could be effective in managing the transition; however, it is acknowledged that it may be time-consuming. The first step involves the identification of strengths and deficiencies within the health hut system, and associated corrective actions necessary to facilitate autonomous management by the community and the health districts. The second step includes realigning the health huts to function more effectively. The final step in the process involves defining the roles and responsibilities of the different players in the monitoring and support of the transferred health huts. This step is vital to validate that health huts remain functional after USAID/Senegal transfers support and responsibility. While dependence on the ChildFund consortium is high, discussions with the TTBAs, CHWs, CH Management Committees, and outreach workers demonstrated that there was a strong sense of ownership of their health huts. From the point of view of the beneficiaries, it was conveyed that what they need from USAID/Senegal is resources to run the health huts, since they already “own” them. The BGs are an important type of CHW in Senegal that provide support for linking members of the community to health services. Based on focus group discussions with the BGs and ChildFund annual reports, there were 3,406 functional BGs of which 1,207 (35%) reported activities that reached a total of 117, 394 people for an average of approximately 97 activities per person per year. Activities reported included 15,788 mothers and babies accompanied to health structures; 33,464 reached with advocacy activities; and 37,051 mothers/infant pairs visited for postnatal follow-up. Discussions with outreach workers demonstrated that there were varying levels of training received on components of the integrated package. Some individuals received training in all of the components, while others received training in just one aspect. They all expressed appreciation and desire for further training to increase their knowledge on the integrated package, as well as on other health issues relevant to their community. Discussions with the outreach workers indicated that they received personal satisfaction from their work and considered it to be very important and effective in improving the health of their community members. Part of the newly developed CH strategy is the strengthening of the CH information system and its integration into the national information management system. The ChildFund consortium supported this process from the beginning, and CH information is now being transmitted to the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 105 health posts, which then combine their data and submit it to the district for transmittal up through the hierarchy. The coordinator of the CH unit confirmed the existence of the disaggregated data in the national information management system. Data for CH includes numbers and services of clients frequenting the health huts and the activities of the outreach volunteer workers. The ChildFund consortium supplied tools for data collection to the health huts and trained health hut staff in its use. Team EY confirmed the presence of these registers during health hut visits and found them to be, in most cases, completed. Parallel to the MOH information system is the ChildFund consortium reporting system which is required by USAID. The MOH has access these records as well. The ChildFund consortium and USAID/Senegal reporting system, functions like a backup to that of the MOH. It is important to note that the CH registries used by the health huts are paper-based. The ChildFund consortium developed linkages with several development partners mainly for activities concerning reproductive health and child survival. The USAID/Yaajeende agriculture and food security project was designed to link with the CH component by having Yaajeende’s target population overlap in the catchment areas of the health huts. The ChildFund consortium collaborates directly with USAID/Yaajeende so that their activities are complementary and the teams collaborate. USAID/Yaajeende works with rural producers through nutrition-led agriculture, whereby improved agricultural and wild food products are promoted within the rural value chain that would diminish identified nutritional deficiencies. Team EY visited the USAID/Yaajeende office in Matam and found the way in which the staff implemented their partnership to be highly effective. The field staff of both the ChildFund consortium and USAID/Yaajeende work together in the communities where all the activities take place around the health huts. The activities are based on horticulture, swamp farming, and the introduction of high-yield gardening techniques. They include demonstration gardens at the health huts, schools, community, and household gardens where vegetables are grown to fill in nutritional gaps in the local diet. During year three of the project, the ChildFund consortium conducted several activities including developing joint plans with USAID/Yaajeende between zonal offices of Matam, Bakel, and Kédougu. Mobile units are being used for awareness building, screening for malnutrition, nutritional monitoring, distribution of food stuffs, and the installation of drying racks for agricultural products produced by the community. One aspect of this collaboration is the plan for USAID/Yaajeende in Matam to train the CH outreach workers in agriculture so they can benefit from earnings made from the produce. The ChildFund consortium is also linked with the Micronutrient Initiative for a program of community activities for maternal and neonatal health in Kolda. UNICEF is contributing to the supply of equipment for health huts and supervision of community outreach workers. Additionally, Peace Corps volunteers collaborate with the ChildFund consortium in the catchment areas of the health huts on programs for outreach workers concerning hygiene, sanitation, and promotion of small gardens, malaria prevention, FP, and nutrition. The mechanism for transferring health huts requires them to meet a set of criteria defined in the health hut transfer strategy, which is considered necessary in order for them to be self-sustaining. Regional and district health officers interviewed did not think the health huts would be able to continue to function effectively without USAID/Senegal or other donor support. The ChildFund consortium tried to address these problems. Their system of waiting to begin the process of fund withdrawal until the community determines it is ready is a viable way to address the problem to a Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 106 certain extent. In addition, they developed a monitoring tool with 25 indicators which they used as a baseline before transfer, and if a decline in services is detected, they try to rectify the problem. Advocacy work with local government authorities is continuing. In addition, the ChildFund consortium provides management training to CH Management Committees. The low rate of transfer in the second year as compared to that of the third year reflects the time it takes to prepare the health huts when following the transfer mechanism implemented by the ChildFund consortium. It may be beneficial to investigate contributing factors to the success of the health huts which have been functioning continually in Thiès for the two years since transfer. This may bring awareness to the factors that should be emphasized during the process of strengthening them for appropriation. Discussions with the CH Management Committees, which are supposed to manage the funds for the health hut and provide advocacy for support, indicated that they did not believe there would be sufficient resources to maintain the health huts and their outreach activities. This belief was consistently shared by the volunteer health outreach workers, the TTBAs, and the CHWs that were interviewed. Additionally, discussions with the volunteer health outreach workers indicated a strong demand for some type of incentive to allow them to conduct their outreach activities consistently. Additional challenges that were captured during the interviews are noted in Table 26 below include: Table 26: CH Constraints and Challenges Constraint/Challenge Description Inconsistent completion of registers and transmittal of data without the consistent supervision of the ChildFund consortium Interviews with the ChildFund consortium staff, confirmed by the TTBAs and CHWs, revealed that for the registries that were completed it was because the ChildFund consortium provided direct supervision. Interviews with the ChildFund consortium indicated that health huts at greater distances from the post were less frequently supervised and had greater difficulty in maintaining these registers. Also, where weak cooperation from the nurse at the health post exists, TTBAs and CHWs may stop filling out the registers. Many health huts depended on the ChildFund consortium staff to physically bring data to the health posts. Team EY is concerned that once the ChildFund consortium staff is no longer supporting this activity, the CHWs will not consistently deliver reports. Low commitment of local government authorities to CH18 Field data corroborated the information in the ChildFund consortium cooperative agreement concerning the dependence on local government and its resources after the withdrawal of USAID/Senegal resources from the health huts. This poses a challenge because of the low commitment of local government authorities to CH and the lack of capacity in financial management. Another concern was the dependence on local government authorities for funding medical supplies. Funding from the MOH is allocated to the local government authority to buy health products for the health huts. However, local government personnel frequently do not have the capacity to manage finances or an interest in allocating those funds to the health huts. 18 “CH Program Component Cooperative Agreement,” USAID/Senegal, September 28, 2011. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 107 Constraint/Challenge Description High drop-out rate of BG There continues to be a large drop-out rate of BGs in all zones except in Sédhiou where 100% were functional. The reason given for this in the annual reports was that they enrolled BGs late and did not promise any incentives. To address this problem, it was agreed that the DSRSE would write a letter to the regional and district medical officers reminding them of their responsibility for the supervision and monitoring of activities of the BG program. Lack of incentives for volunteer health outreach workers In discussions with regional and district medical officers, members of community management teams, and volunteer health outreach workers, the issue of the lack of incentives was continually raised. Some volunteers insisted on monetary payment. However, others expressed the opinion that while monetary remuneration was desirable, it was not necessarily the central issue. Volunteers understood the importance of their work and that funds were not available, and could be somewhat satisfied with other types of incentives such as badges for identification, recognition of their work, and more training and information materials to use for their health education activities. However, as long as they were volunteering, it was clear that they could not commit to working as consistently as if they were being paid. The MOH and the ChildFund consortium are well aware of this problem and are trying to work out solutions. The CH unit in the MOH has responded to some of these requests. Identification badges were given to some volunteers and there are plans for a “Volunteer Recognition Day.” In one district visited, the director of the BREIPS gave promotional materials to the volunteers. Lack of educational materials or visual aids and demonstration items for the outreach education The outreach workers are not given informational materials for themselves or visual aids or demonstration items to use during their outreach activities. It could compromise the effectiveness and knowledge transfer associated with the outreach. 7.4.2 Lessons Learned and Recommendations • The commitment from local government authorities to take over and support existing CH activities appears to be low. The CH Management Committees are concerned about the negative impact of the withdrawal of USAID support with the transfer of health huts, which could result in closures or the reduction of services. Once the health huts become dependent on local government authorities for the majority of their resources, it will be a significant undertaking to secure the commitment from local government to support CH activities. USAID/Senegal may want to consider placing more emphasis on advocacy efforts targeting locally elected officials to create a more favorable political environment for health huts. This may increase locally elected officials’ commitment and interest in supporting and allocating funds to health huts once USAID/Senegal resources are withdrawn. In addition, the strategy for the transfer of USAID/Senegal support should include an Income Generating Activity (IGA) for the CH Management Committees and assist them in developing a plan for leveraging alternative sources of funding. • CH education appears to rely mainly on volunteer outreach workers who, while dedicated and willing to do the work, also need and ask for incentives so they can more consistently Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 108 provide support to their communities. The MOH, USAID/Senegal, and other donors may want to develop effective strategies to incentivize and motivate CHWs and outreach workers. This could be an opportunity to develop incentive programs through creating pilot PBF community-based indicators, or consider adding supervision and quality indicators to the list of current PBF indicators that are currently being expanded. Other incentive “bonuses” may include rewarding supervision teams with phone card minutes if they consistently perform at a higher level for at least one quarter in a row. Another option would be to create a small incentive fund (less than $5,000 USD a year) to be included as part of the DF mechanisms. Suggestions given by the volunteer workers should be taken into consideration such as recognition of their work and more training and information materials to use for health education activities. The ChildFund consortium could collaborate with ADEMAS and develop a social marketing scheme where the volunteers could sell their products such as water purifiers (aquatabs) and ITNs (MILDA). Implementation of this recommendation it may increase consistent volunteer availability. • The partnership of the ChildFund consortium and USAID/ Yaajeende appears to function well because the collaboration is built into the structure of the project. The population benefits from the joint efforts and the health and agricultural interventions improve the overall status of the population. To increase the benefits from partnership collaborations, USAID/Senegal may want to consider following the model of partnership between USAID/Yaajeende and the ChildFund consortium into future project design, and require IPs to report on structural and operational aspects of the relationship as a deliverable. This will allow USAID/Senegal to leverage best practices of existing partnerships and better track progress of partnership collaborations. 7.5 Sub-component C: Fostering national MOH and other sector ministry ownership for CH and harmonizing the linkages with national policy initiatives 7.5.1 Findings and Analysis The ChildFund consortium held planning workshops with all offices and national programs in the MOH to define priority activities for the CH program. To date, two planning workshops were organized. The first brought together the CH unit, and the MOH programs and services involved in CH. The second, also with the participation of the CH unit, was for planning with national partners who contributed to controlling Neglected Tropical Diseases (NTD). The ChildFund consortium also participated in the development of the USAID integrated work plan with Abt, IntraHealth, FHI 360, and ADEMAS. Other planning workshops were held separately with the HSI component (IntraHealth) and with the HCP component (ADEMAS) resulting in collaborative action plans. These are sub-sets of specific action plans within the overall integrated work plan. The main activity under this sub-component was the development of the National CH Strategy, which was officially launched with the MOH in August 2014, and subsequently followed by a national workshop organized with the technical directors and national programs of the MOH and their partners. Regional dissemination workshops were conducted in 10 regions (Ziguinchor, Matam, Sédhiou, Diourbel, Louga, Tambacounda, Fatick, Kaffrine, Kaolack and Saint-Louis) in August and September 2014. Dissemination workshops were planned for the other regions in Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 109 October 2014. The ChildFund consortium with the CH Unit and other relevant MOH divisions developed an action plan to operationalize the strategy. Interviews with the MOH Secretary General (SG) and regional and district health officers indicated their support for the strategy, which recognizes the health hut level of the MOH pyramid of services as an integral part of the health system to be supported and included in their scope. The development of the National CH Strategy is an example of a successful collaboration between the ChildFund consortium and Abt. The challenge that remains will be the operationalization of the strategy with the development of a specific work plan. Furthermore, there still are some doctors who are reticent to accept the idea that TTBA should be permitted to administer long-term contraceptives or intervene in prevention of postpartum hemorrhaging. While interviews with key stakeholders indicate their support, it is still not clear as to the degree of support that the MOH will provide at the central, regional, and district levels when action and resources are required. As noted in earlier sub-components, the ChildFund consortium continues to provide support for these types of task-shifting interventions. While linkages are established with other relevant MOH entities, the challenge is to maintain those linkages on the operational level and to raise the leadership capacity of the CH component unit. The challenge concerning planning meetings is implementing the action items that follow. 7.5.2 Lessons Learned and Recommendations • The dissemination of the National CH Strategy appeared to be successful and resulted in a positive perception toward the value of CH activities. In order to receive continued support of the National CH Strategy, it is important to continue to maintain awareness of the benefits of the strategy. USAID/Senegal may want to consider providing continued support to the MOH to operationalize the CH strategy and to reinforce linkages between the MOH and CH unit. This will increase momentum and build on the national awareness of CH activities. 7.6 Response to Evaluation Questions The table below provides information to respond to the component-specific evaluation questions as stated in the scope of work. This section is not intended to be exhaustive, but rather aims to highlight notable successes; constraints and challenges that have been experienced during implementation by the ChildFund consortium; and key interventions that may be added, continued, or removed within Component 3. Team EY and USAID/Senegal agreed that this analysis would be conducted by component rather than by individual sub-components. Table 27: Component Table Component Specific Question Analysis To what extent have the components achieved their objectives? The ChildFund consortium made progress towards achieving their objectives as outlined in Section 7.1. The USAID standard indicators outlined in Table 23 were selected as a sample as they are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services, IR 2: Improved health seeking and healthy behaviors, and IR 3: Improved performance of the health system. Overall, the ChildFund consortium showed positive results of improving indicators from year one to year two. Areas where targets were not met included retraining on RDT for malaria and fever management with ACT. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 110 To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component?  Expanded availability of the integrated package: The ChildFund consortium expanded the availability of the integrated package of services to 1,627 health sites, which represented 91% of their planned target (1,919). Factors contributing to success are attributed to the training of providers on interventions/protocols, which allowed them to improve their skillset and expand access to services.  Increased availability and use of oral contraceptives: There was an increase in the availability and use of oral contraceptives for FP from the onset of the project to the time of the evaluation. At the end of year two, only 48.7% of health huts offered oral contraceptives, compared to the end of year three where 90% of health huts offered several methods of FP. The increase is partly attributed to the decreased number of stock-outs in the past year, which was confirmed by the evaluation team when they interviewed facility-based staff at the health posts. In addition, tracking of oral contraceptive use is now possible with the introduction of client registers indicating provision of oral contraceptives.  Improved community ownership and linkage of community members to health services: The BGs are an important type of CHW in Senegal that provide support for linking members of the community to health services. Interviews with outreach workers demonstrated that there were varying levels of training received on components of the integrated package. Some individuals received training in all of the components, while others received training in just one aspect. They all expressed appreciation and desire for further training to increase their knowledge on the integrated package, as well as on other health issues relevant to their community. Furthermore, outreach workers indicated that they received personal satisfaction from their work and considered it to be very important and effective in improving the health of their community members.  Successful collaboration for RH and child survival interventions: The ChildFund consortium developed linkages with several development partners mainly for activities concerning RH and child survival. The USAID/Yaajeende agriculture and food security project was designed to link with the CH component in the catchment areas of the health huts. The ChildFund consortium collaborates directly with USAID/Yaajeende in Matam, Bakel, and Kédougou so that their activities are complementary. Team EY found that one of the key factors contributing to the success of ChildFund and Yaajeende’s collaboration was the highly effective manner in which the staff implemented their partnership.  Development of National CH Strategy: The National CH Strategy was officially launched with the MOH in August 2014. Interviews with the MOH SG and regional and district health officers indicated their support for the strategy, which recognizes the health hut level of the MOH pyramid of services as an integral part of the health system to be supported and included in their scope. Factors contributing to this success include the collaboration between the ChildFund consortium and Abt and the participatory and inclusive approach implemented to coordinate the relevant MOH and other stakeholders in the process of the development of the national CH strategy. What are the constraints and challenges that have  Lack of amoxicillin for ARI at the community level: There was a lack of amoxicillin for ARI at the community level, which is primarily due to the lack of MOH administrative authorization. This limited the ChildFund Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 111 hindered successful implementation of each sub-component, and how has the IP dealt with those challenges? consortium’s ability to treat ARIs at the health huts. In order to address this challenge, the ChildFund consortium plans to accelerate the process of making amoxicillin available by developing protocols for treatment of ARIs with amoxicillin by CHWs. They will collaborate with UNICEF and the DSRSE to define the methodology for the initial introduction of amoxicillin into the health huts, track the transport of product, train CHWs in treatment of ARIs, and organize supervision for CHWs providing this service. The ChildFund consortium expects that with bottlenecks resolved at the MOH, more amoxicillin will be made available at the community level for use to treat ARIs.  Scaling-up the use of misoprostol: Training on the use of misoprostol was provided to TTBAs and CHWs in the prevention of postpartum hemorrhaging in 28 health huts as part of a pilot program in the regions of Kolda and Thiès. Interviews with several medical service providers indicated concern over the ability of TTBAs to correctly administer misoprostol. It was noted that this will pose a challenge and require close monitoring for quality control when misoprostol is first introduced at health huts (beyond the initial pilot sites) until there is confidence that TTBAs are administering it correctly. Since TTBAs were only recently authorized to use misoprostol at the community level (outside of the pilot), the ChildFund consortium has not dealt with this challenge. It will be important for the ChildFund consortium to note the concerns of medical services providers and integrate quality control measures into their training and supervision activities.  Delays in commodities and quality of storage space: CHWs do not consistently provide drug orders to health posts on a monthly basis, thereby resulting in the delay of commodities. Reasons given by the CHWs include lack of transport or failure to recognize that this responsibility is a priority. The ChildFund consortium field staff is trying to address these gaps by taking the orders for supplies themselves, or accompanying the CHW to the health post. When health huts are fully transferred to the responsibility of the community, the concern remains as to how this gap will be filled. Observations revealed that the quality of commodity storage space varied significantly, from an orderly and clean cupboard to a dilapidated carton kept under a cot in a small, locked, dirty room. The final evaluation of the first phase of the CH project had similar observations.  The ability of health huts to manage themselves without significant outside support: Data collected through interviews with TTBAs, CHWs, volunteer outreach workers, and members of the CH Management Committees revealed several challenges in staffing the health huts. The TTBAs and CHWs are volunteers who receive a very small amount of money (3,000 - 4,000 CFA francs/$6 USD - $8 USD) inconsistently at the end of the month from the sale of drugs and a 200 CFA francs ($.40 USD) consultation fee. Most volunteers are women with household responsibilities and in many cases also contribute to earning household revenue through farming. Therefore, they are not able to keep the health huts opened consistently on a daily basis and only open them on an irregular basis, with no specific schedule. Since the clients could not count on them being open at any specific time, they were discouraged to use them at all. A solution provided during interviews with TTBAs and CHWs was to keep the health hut open for only a few hours a day at specific times and be available by mobile phone should Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 112 someone need their services. Additionally, interviews with the ChildFund consortium staff, confirmed by the TTBAs and CHWs, revealed that for the registries that were completed, it was only because the ChildFund consortium provided direct supervision. Interviews with the ChildFund consortium staff indicated that health huts at greater distances from the post were less frequently supervised and had greater difficulty in maintaining these registers. Also, where there was weak collaboration between the nurse at the health post and the health hut, TTBAs and CHWs sometimes stopped filling out the registers. Many health huts depended on the ChildFund consortium staff to physically bring data to the health posts on their behalf. Team EY is concerned that once the ChildFund consortium staff is no longer supporting this activity, the CHWs will not consistently deliver reports. Team EY did not receive information on how the ChildFund consortium plans to address this challenge in the future. Are there interventions that should be added or removed?  Integrate the use of mobile technology: It was noted during interviews with Team EY that the system of the CHWs ordering commodities through the health post is not working well. At times this process results in stock-outs even when medical supplies are available at the central level. One reason cited is difficulty in accessing transportation. The ChildFund consortium may want to consider exploring opportunities to integrate the use of mobile technology to improve the commodity procurement and distribution of commodities at the community level in order for CHWs and TTBAs to deliver services at the household level by working with commodities point of contact at the health posts to communicate regularly on what commodities are needed. Additionally, this may increase coordination and consistency between health huts and health posts.  Leverage the USAID/Yaajeende and the ChildFund consortium partnership model for future structures: The partnership of the ChildFund consortium and USAID/ Yaajeende appears to function well because the collaboration is built into the structure of the project. The population benefits from the joint efforts and the health and agricultural interventions improve the overall status of the population. To increase the benefits from partner collaborations, USAID/Senegal may want to consider adopting the model of partnership between USAID/Yaajeende and the ChildFund consortium for future project design. USAID/Senegal may also want to consider requiring IPs to report on structural and operational aspects of the relationship as a deliverable. This will allow USAID/Senegal to leverage best practices of existing partnerships and better track progress of partnership collaborations. Are there changes that could be made to improve performance?  Consistent quality of services at health huts: It was noted that the quality of services at the health huts appear to be compromised by inconsistent availability of medical supplies, poor infrastructure, and lack of electricity and water. Project reports indicate that some health huts (a percentage is not available since Team EY did not visit every health hut in the each region) are in such disrepair that they are no longer functional (i.e., roof has collapsed) and that consultations lack privacy as the consultation rooms have no enclosure. USAID/Senegal may not want to add additional services to the integrated package in order to first improve the quality of existing interventions, increase support for supervision and monitoring, and provide the necessary renovations to render the health huts more functional. The focus on existing interventions may allow the ChildFund consortium to improve the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 113 existing quality of services (i.e., fully functional health huts) and develop a better process for supervision and M&E of services. 7.7 Data Sources In addition to sources cited in Annex G: Bibliography, data collected in the field was used for analysis of this component including: in-depth interview with COP, RB coordinators, key MOH stakeholders, regional and district health officers, regional and district coordinating offices, community-based health insurance managers, Directors of the BREIPS, service providers at health facilities, services providers at health huts, pharmacists, private firms; as well as focus group discussions with CH workers, CH committees, community-based health insurance beneficiaries, and associations of PLWHA. Data collection tools for field interviews are available in Annex I and Annex J. 8.0 COMPONENT #4: HIV/AIDS 8.1 Background USAID/Senegal awarded FHI 360 a five- year $22M USD cooperative agreement in 2011 to implement the HIV/AIDS component of the USAID/Senegal Health Program. The component contributes to IR 1: Increased availability of an integrated package of quality health services, and IR 2: Improved health seeking and healthy behaviors. FHI 360 is implementing this project with two primary sub-partners, ANCS and Enda/Santé. Team EY recognizes that FHI has many other sub-partners implementing activities, however, for purposes of consistency between each component section, Team EY is only listing the original sub-partners as named in the cooperative agreement. In the original design of the project, TB was included as a separate sub￾component of the project. However, in 2012 a decision was made by USAID/Washington to only include services exclusively for HIV/TB co-infection. USAID/Senegal was not consulted in the decision-making process. Therefore, USAID/Senegal advised FHI 360 to only include services for HIV/TB co-infection. The FHI 360 team is focused on strengthening and expanding successful interventions, while tailoring targeted prevention messages through Mobile Health (mHealth) solutions in eight focus regions of Senegal (i.e., Kaolack, Kédougou, Ziguinchor, Sédhiou, Kolda, Dakar, Thiès, Diourbel). The project is organized into three sub-components:  Sub-component A: Support national efforts in prevention of sexual transmission of HIV.  Sub-component B: Reinforce a comprehensive package of treatment, care, and support for People Living With HIV (PLWHIV).  Sub-component C: Strengthen planning, coordination, and management of the National HIV and TB programs and advocacy and policy dialogue for HIV programming. According to the DHS, Senegal has a concentrated HIV epidemic with a low prevalence in the general population (0.72%) and higher rates in MARPs. The 2010 Combined Sexually Transmitted Infection (STI)/HIV/AIDS National Surveillance Survey found prevalence as high as 18.2% among CSWs, and a 2007 study by Helios found a prevalence rate of 21.8% among MSM. Stigma associated with HIV/AIDS is a considerable barrier in Senegal, and according to Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 114 the DHS 2010-2011, it is a major obstacle to achieving universal access to HIV prevention and treatment. Among people aged 15-24, only 50.5% of women and 40.1% of men would buy fresh vegetables from a shopkeeper who has the AIDS virus.19 The focus of FHI 360’s project is to provide targeted technical assistance and institutional support to the GOS and its partners in order to maintain low HIV prevalence and improve the quality and availability of treatment, care, and support for PLWHA. Additionally, according to the DHS 2012, less than 10% of health facilities in Senegal offer ART services with significant disparities by regions (27% in Kédougou and 3% in Ziguinchor region). 19 “Senegal Demographic and Health Survey 2005,” Macro International, Inc. 2006. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 115 Table 28 below shows the standard indicators related to the HIV/AIDS component in the first two years compared with targets. Although Team EY understands that FHI 360 provided its year three annual report to USAID/Senegal in November 2014, Team EY did not receive the report in time to include the 2014 indicator data into the table below. However, information derived from the report that was verified by data point/site interviews and corroborated by alternative sources was utilized in our analysis. Table 28: USAID Indicators for HIV/AIDS # Indicator Target Actual Target Actual 2012 2013 1 Number of PLWHIV currently receiving ART at the service outlets in the target regions 4,100 3,799 4,650 3,591 2 Number of adults and children with advanced HIV infection newly enrolled on ART * * * 642 3 Number of individuals who received CT for HIV and received their test results 32,677 17,084 24,850 31,770 4 Number of eligible adults and children provided with a minimum of one care service 7,329 6,477 7,000 17,294 5 Total number of individuals provided with HIV-related palliative care (including TB/HIV) * 511 490 558 6 Number of health care workers who successfully completed an in-service training program within the reporting period 110 389 264 357 7 Number of health facilities that offer comprehensive HIV care (e.g., ART, TB/HIV, psychosocial, nutritional, malaria, RH, palliative care) in an ambulatory setting 7 4 5 5 8 Number of the targeted population reached with individual and/or small group level HIV prevention interventions that are evidence-based and/or meet the minimum standards required * 73,852 89,775 92,063 9 Number of MARP reached with individual and/or small group level HIV preventive interventions that are evidence-based and/or meet the minimum standards required * 4,134 7,306 9,288 10 Percent of registered new Smear-Positive Pulmonary Tuberculosis (TPM+) cases that were cured and completed treatment under DOTS nationally (treatment success rate) 86 * * * 11 Case notification rate in new sputum TPM+ cases per 100,000 population 73 82 * * 12 Percent of TB patients tested for HIV 67 72% * * 13 Number of participants in a country trained in the components of the WHO Stop TB Strategy with USG funding 346 661 * * *Data not available Note: Target and actual data included in the table above is derived directly from annual reports provided by IPs. TB indicator data was not collected since FY 2012 when all TB funding ended. Information provided by FHI 360 is what is available. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 116 All data in the above table, including percentages, was directly pulled from the IP annual reports. These indicators were selected as a sample because they are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services and IR 2: Improved health seeking and healthy behaviors. Additionally, these indicators support USAID/Senegal’s high-level indicator, percentage of the target population who know how to prevent key illnesses (e.g., HIV/AIDs, malaria). The indicators in the table represent core HIV interventions across the HIV continuum from prevention activities to diagnosis, treatment, and care for adults and children; health workforce skills development, and targeting of priority populations in this component such as MSM and CSW. The selected indicators are an important subset of FHI 360’s total number of indicators because they address critical PE questions of whether clients are receiving a comprehensive package of services, whether health facilities and providers are equipped to offer these services, and which specific services reach individuals and small groups in the areas of HIV care, HIV prevention information, and education and counseling. Additional information on indicator achievement status can be referenced in Figure 18 below. ► Targets Met: The results indicate that FHI 360-supported sites near met, met, or exceeded targets during year two for the following: 1. Number of PLHIV currently receiving ART at the service outlets in the target regions. 2. Number of individuals who received CT for HIV and received their test results. 3. Number of eligible adults and children provided with a minimum of one care service. 4. Total number of individuals provided with HIV-related palliative care (including TB/HIV). 5. Number of health care workers who successfully completed an in-service training program within the reporting period. 6. Number of health facilities that offer comprehensive HIV care in ambulatory setting. 7. Number of the targeted population reached with individual and/or small group level HIV prevention interventions that are evidence based and/or meet the minimum standards required. 8. Number of MARPs reached with individual and/or small group level HIV preventive interventions that are evidence-based and/or meet the minimum standards required. According to the 2012 - 2013 annual report, factors which lead to reaching targets were attributed to the existence of multiple HIV service delivery sub-partners that executed on their activities, dynamic and motivated PLWHA associations that worked to mobilize populations, especially MARPs, and effective target setting informed by available demographic data. These gains may also be associated with better functioning health centers where capacity was built through HIV component trainings and investments in supportive supervision and quality of service improvements. ► Targets Not Met: There were two targets during year one that did not meet the target. During year two, all targets were achieved. Based on quarterly reports, challenges identified in meeting the HIV-related palliative care target describe a need for training tools and limited capacity of providers and Civil Society Organizations (CSO) to deliver these specialized services. FHI 360 actively sought to address these issues by forming a Technical Working Group (TWG) for palliative care and Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 117 implementing trainings through a pilot project which makes use of those newly skilled in service provision. For HIV prevention programming, FHI 360 continues to explore activities that combat stigma and discrimination and create an environment for MARPs to seek HIV care. ► Data Not Received: A number of actual results were unavailable or unreported in FHI 360’s year one and year two annual reports. Five indicators did not have full information available during year two to include: 1. Number of adults and children with advanced HIV infection newly enrolled on ART. 2. Percent of registered new TPM+ cases that were cured and completed treatment under DOTS nationally. 3. Case notification rate in new sputum TPM+ cases per 100,000. 4. Percent of TB patients tested for HIV. 5. Number of participants in a country trained in the components of the WHO Stop TB Strategy with USG funding. All TB-related indicators were not collected after FY2012 because TB funding ended. Additionally, although performance data was provided for year one achievement for certain three indicators, no target was set or available in year one for these indicators. Team EY received no additional information from the work plans, annual reports, or interviews to substantiate the reasons for missing standard indicators. FHI 360, as stated in the note under Table 28, was not required to provide further information on the four indicators which explains why data was not reported on in year two. Figure 18: HIV/AIDS Indicator Status 8.2 Key Findings and Recommendations/Benefits Overall, the FHI 360 project is achieving some of its objectives and meeting some of the agreed upon indicators as stated in their contract. Key findings and recommendations are provided in further detail in Section 8.3 - 8.5. Table 29 below summarizes the most important findings and recommendations relevant to strengthening the overall quality and delivery of the HIV/AIDS component. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 118 Table 29: HIV/AIDS Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefit 1 MSM and CSWs face additional barriers to seeking and utilizing health services besides discrimination and stigmatization. Fear of prosecution, community exclusion, and weak laws to protect MSM and CSWs in cases of police abuse and unfair treatment affect their decision-making. HIV testing yields for MSM are low compared to the general population. This is likely attributed to issues of stigma in addition to general negative views regarding PLWHA, as well as the lack of societal support for those who come forward for services. Recommendation: Health communication approaches need to take into account these complex factors with subtle and inclusive messaging and outreach activities. Since MSM and CSWs in Senegal are disproportionately affected by HIV/AIDS, USAID/Senegal may want to consider increasing mobile services for key populations and organizing services around a “hotspot” strategy. Additionally, USAID/Senegal may want to consider expanding and more closely evaluating neutral BCC campaigns such as messaging about partner reduction and risks associated with multiple concurrent partnerships. Benefit: The program benefit of targeting young men in major cities or hubs for CT through mobile services includes encouraging the use of extended testing hours, and providing privacy to attract them to services. Other than CSWs and MSM who require this support, a benefit of targeted interventions for discordant couples and transient populations (i.e., truck drivers, fishermen, gold miners, and migrant workers) could strengthen tailored health communications. 2 Team EY’s understands that FHI 360 is anticipating the initiation of test and treat in Senegal. The scientific rationale for opting for a test and treat approach is that it is a proven intervention for reducing transmission and is being used in many countries with high prevalence rates. The intervention is generally based on testing everyone in 'high risk' groups and areas of generalized epidemics, and then immediately treating all of those diagnosed positive, regardless of whether their immune system is damaged or meets the clinical definition to initiate ART (CD4 count of less than 350). Taking into account the very low prevalence rate in the general population, the resource constraints in Senegal, and the commitment to patients’ lifelong needs for ART, Team EY believes that an alternative to the test and treat model deserves further consideration. Recommendation: A focus on maintaining high quality ART services for those already on treatment, meeting the unmet need for those who clinically should be on ART, and supporting more cost-effective interventions such as the treatment of opportunistic infections could be a more feasible public health approach for managing the HIV/AIDS epidemic. Benefit: The program benefit of considering alternative interventions to test and treat could be more cost-effective while not significantly damaging the needs of PLWHA. There are many interventions that might be better investments and meet the needs of the epidemic in Senegal. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 119 8.3 Sub-Component A: Support national efforts in prevention of sexual transmission of HIV 8.3.1 Findings and Analysis FHI 360 implemented a range of activities that support national efforts in the prevention of sexual transmission. These included BCC for the most-at-risk groups, CT services, consultations on STIs, and distribution of female condoms. Close to half (49%) of clients targeted for HIV prevention services were reached during year two. Early indication of success was visible in the outreach activities conducted and tailored to MSM and youth. Positive feedback on the mHealth innovations and strategies suggest a potential entry point for expansion to reach more male clients and youth through these electronic platforms. The data from year one and year two depict a steady increase in the number of people reached as a result of prevention interventions. In year two, FHI 360 exceeded their target of MARPs reached with individual and/or small group level HIV preventive interventions. However, the package of services differs from one district to the other (e.g., laboratories for CD4 count and viral load, ART, Voluntary Counseling and Testing (VCT), MSM prevention) and the differences in access to services between urban and rural areas are evident. PLWHAs in Kédougou in need of specific HIV/AIDS related services most likely need to travel to Kaolack, Dakar, or the Tambacounda hospital, which is not currently supported by the FHI 360 program. Team EY did note that the geographic distribution leaves out the Tambacounda region, which has a similar population and health issues as Kédougou and Kolda, but is not covered by the FHI 360 project. Providing targeted prevention interventions to CSWs and MSM is a priority in Senegal as they are considered MARPs. It is important to note that there is limited population-based data, specifically on the CSW and MSM populations. FHI 360’s approach is to identify high-need districts for expansion and integration of comprehensive services for both CSW and MSM through an open dialogue process with key stakeholders including RMHT, services providers, MSM, CSWs, religious leaders, and civil society. In collaboration with the Netherlands Ministry of Development Cooperation, FHI 360 facilitated the sensitization of religious and community leaders and journalists with visibility around MSM and the development of an Advocacy Resource Book, which is in the process of being finalized. FHI 360 also focused on reaching MSM outside of these various networks by integrating counseling capacity through a hotline service. However, due to the insufficient number of calls the hotline was closed. Despite the growth in the number of MSM associations over the past five years, and donor support from GFATM and USAID/Senegal, there are still MSM who are not being reached. The USAID/Senegal Health Program also promoted access to care, medical treatment, and condoms for CSWs. The success of mHealth interventions from recent indicators demonstrate that a total of 11,833 youth and 1,200 MSM were reached. FHI 360 works with the CNLS and partners to develop mobile text messages, blogs, and website platforms to: 1) revise messages based upon target groups and gender issues, 2) use innovative communication channels, 3) introduce an active “positive prevention” program among PLWHIV(further discussed in Section 8.4 below under sub-component B), and 4) integrate HIV into other development programs. Building on this foundation, the FHI 360 team is focusing on strengthening and expanding successful Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 120 interventions, while tailoring targeted solutions such as “quality blog,” “gender observatory,” and a youth focused platform titled “cliqinfoado” to address the contextual circumstances of HIV/AIDS in eight focus regions of Senegal. The mHealth strategies are also being used to promote services and improve linkages between HIV and RH at Counseling Center for Adolescents (CCA) in Kolda, Kédougou, and Sédhiou in collaboration with selected local CBOs and NGOs. RH information is provided through socio-educational activities in the CCAs using internet and mobile messaging. The CCAs were established by the Ministry of Youth and exist in all districts with support from UNFPA and previous investments from USAID. The CCAs are managed by the District Center for Popular Education and Sports (CDEPS). The activities target youth with education on contraceptive use, prevention of early pregnancies, and STI/HIV prevention. These strategies include both mobile phone messaging and internet information that are currently being piloted by One World/UK, which is a civil society social network platform. The One World/UK intervention is implemented in collaboration with Sis Afrique and uses MSM internet-based peer facilitators for live-chat websites and discussions around HIV prevention, care, and stigma. The peer prevention animators are members of MSM Associations and receive training from Sis Afrique to provide internet-based information services to MSM, including basic facts on HIV, STIs, ART, as well as testimonials. They also provide information on MSM-friendly referral health facilities or doctors trained specifically on MSM health needs. This appears to be an attractive method for reaching MSM with information and explains some of the positive feedback reported by the program through interviews with members of the Association of PLWHA. Many efforts are being made to create an enabling environment for civil society organizations who continue to advocate for PLWHAs. ENDA Santé, one of FHI 360’s sub-partners, supports the creation of groups and associations of CSWs in the regions covered by the program. Members of groups and associations of CSWs organize health talks and promote STI/HIV risk reduction, condom distribution, and HIV testing. ENDA Santé currently has five outreach mobile clinics, two of which were provided with FHI 360 support, for STI diagnostic and treatment. Mobile clinic operations started in Dakar and were expanded to Mbour, Kolda, and Ziguinchor. CSWs receive information on diseases and benefit from follow-up home visits. These visits allow them to receive counseling, psychosocial support, and financial support to purchase prescribed drugs. The CSW leaders are members of the HIV response coordination bodies, which facilitates their advocacy effort with local authorities and the police in case of arrests. ENDA Santé also carries out cross-border interventions targeting CSWs. The approach includes dialogue with health authorities across three countries (i.e., Senegal, Gambia, and Guinea￾Bissau), harmonization of protocols in service provision, and sharing of information (contact details of service providers) and experiences. A regional conference is planned for 2015. Currently, HIV/AIDS is concentrated in MARPs; however, according to the National Strategic Plan 2014-2017, the dynamics of the epidemic are shifting and raising new concerns. Seventy￾nine percent (79.1%) of new infections occur in heterosexual ‘stable’ couples. Owing to taboos and stigma, many MSM revert to bisexuality, which should be considered in future prevention strategies. The social environment in Senegal is not very favorable to MARPs, even if CSWs are tolerated. Stigma and discrimination toward MSM in communities and among religious leaders Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 121 and health providers remains a major barrier and programmatic gap in FHI 360’s project. 20 Among religious leaders there is a range of views regarding MSM. The major obstacle is conservative religious leaders and organizations who act as watchdogs regarding MSM, resulting in condemnation, violence, and imprisonment of MSM. Insufficient sensitization campaigns on tolerance regarding MARP groups may explain why MSM, CSWs, and youth do not feel safe to disclose their status or to seek health services. During the interviews with PLWHAs, concerns of health staff potentially sharing information about patients seeking HIV, STI, and other medical services were seen as a barrier to patients accessing health care from a particular health center. MSM mostly suffer from family, community, and even professional discrimination and rejection once their sexual orientation is disclosed; therefore, many feel the need to hide their sexual orientation. It is still considered a taboo to be gay or an MSM in Senegal even though it is slowly becoming more tolerated. Additionally, the media coverage on MSM issues mostly echo religious discourse, reverting to sensationalism on topics related to morality. FHI 360 should continue to address the unmet health needs of MSM, especially young men with a greater focus on research by region and the provision of wrap around services that support positive living, disclosure, and ways to manage stigma and discrimination at the household and community Couples Counseling and Gender FHI 360 supported the development of the National Plan on Gender and training of CBOs and health care providers at the national and decentralized levels. FHI 360 supported the Society of Women and AIDS in Africa (SWAA) gender approach to couple counseling which contributes to the promotion of male involvement. In some health posts, nurses’ support couples CT. However, despite the efforts made toward the promotion of couples CT, the uptake and performance of the strategy is still low (2,095 couples were tested: accounting for 16% of the target). According to the FHI 360 annual report, the main challenges include the lack of ownership of the strategy by health care decision makers and workers; insufficient number of trained health care providers; limited number of trained CHWs involved due to the lack of incentives; and insufficient quantity of communication materials (promotional and educational leaflets). Qualitative data collected by Team EY supported the challenges in the information provided in FHI 360’s annual report and also noted the lack or inconsistent capacity/skills in how to approach couples counseling by service providers outside of specific HIV services. Opinions expressed by regional and district health officers in the southern regions were that couple counseling approaches that may work in Dakar need to be contextualized to regional and local realities. For example, women are heavily influenced (financial dependency was noted as a key barrier) by their husbands’ social norms, which affect their individual behaviors including asking their husbands to accompany them for CT. Therefore, couple counseling strategies need to take into account the sociocultural barriers by involving community and religious leaders that work to shift cultural norms and practices. Additionally, gender inequalities should be approached from a multi-sectorial perspective to address the root causes of gender inequalities (e.g., literacy, economic dependency ratio, inheritance laws). The Men Care+ approach, which 20 “Senegal Demographic and Health Survey and Multiple Indicator Cluster Survey (EDS-MICS) 2010-2011,” Agence Nationale de la Statistique et de la Démographie (ANSD), and ICF International, 2012. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 122 has shown to be effective for male involvement in Rwanda and South Africa, may be worth considering as an approach in Senegal as well.21 8.3.2 Lessons Learned and Recommendations • HIV testing yields for MSM are low in comparison with the general population. This is likely attributed to issues of stigma and other challenges reaching this group, general negative views regarding PLWHA, as well as the lack of societal support for those who seek services. Since MSM in Senegal are disproportionately affected by HIV/AIDS, USAID/Senegal may want to consider creating special programs focused on strategies to improve access and use of HIV CT interventions. It may be beneficial if these programs target young men in major cities or hubs for CT through mobile services such as implementing “hot spot strategies” and BCC outreach campaigns that encourage use of extended testing hours, so that there is privacy to attract them to services. • USAID/Senegal may want to request that FHI 360 work more closely with the GOS to provide solutions that address quality and access for key populations and PLWHA. The availability of safe and reliable CT services is required to help clients to know their status and link into prevention, treatment, and care services. The limited number of CT sites outside of Dakar presents a challenge for prevention programming, given that many people do not know their HIV status and, therefore, do not receive the necessary follow-up support. Working with the RHMTs and DHMTs to advocate for and deploy resources in CT activities should be prioritized. Investments in the use of innovations that use mobile technology could be explored with NGOs, public sector providers, and private sector partners. • The progress made with eHealth and mHealth interventions shows promising opportunities for engaging hard-to-reach groups with HCP around prevention. These activities can be strengthened when connected to information and key population-friendly services. USAID/Senegal may want to consider exploring the use of social networking to share experiences and better utilize PLWHIV as models for positive living. • Civil society organizations are also effective mechanisms for reducing stigma. The CNLS has set up a working group on MSM to lobby government officials, religious and other community leaders, as well as the media for more tolerance toward MARPs, including MSM. USAID/Senegal may want to request that FHI 360 increase support for this working group. 8.4 Sub-Component B: Reinforce comprehensive package of treatment, care, and support for PLWHIV 8.4.1 Findings and Analysis The comprehensive package of services supported by FHI 360 and partners includes HIV care and support, HIV/TB/Malaria/RH integrated clinical services, psychosocial support, nutritional services, and socio-economic support. Although USAID/Senegal is no longer funding TB as a separate component under FHI 360, resources are used to support the Ministry of Health and Social Welfare’s Division of HIV/AIDS and STIs (DLSI) in the area of TB and HIV co- 21 MenCare+, Engaging Men in a four country initiative. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 123 infection. FHI 360 is supporting the TWG for TB-HIV co-infection which includes members from the DLSI, the National TB Program, and FHI 360. The TWG carries out periodic reviews where they exchange strategies and new developments for addressing HIV and TB co-infections. The DLSI conducts supportive supervision of health providers at the regional level. FHI 360 also facilitated the drafting of the TB policy document, norms, and protocols and the facilitator and participant training manual. This manual was updated to take into account management of HIV and TB co-infection. The ending of TB funding and the focus on TB-HIV only activities has negatively affected the MOH’s ability to implement its activities. According to interviews with MOH stakeholders, the decrease of funding for TB has created a larger gap in services and contributed to frustration among the GOS health staff. Team EY understands that this decrease in TB funding was not attributed to FHI360’s performance and was mandated by USAID/Washington and, therefore, it is not the responsibility of FHI 360 to resolve this challenge. The current findings from year two show 91% of eligible clients were reached with ART services at FHI 360 supported sites. PLWHA were provided with at least one care service, as part of the integrated package of services. The comprehensive package of services is valued by all stakeholders interviewed during the field data collection phase of the evaluation (e.g., PLWHIV, MSM, health care providers, NGOs). Community meals were raised during interviews as a positive intervention provided to PLWHA. Community meals were initially established to provide guidance and advice on the nutritional value of various foods, but it was found during the interviews and focus group discussions that they were helpful in many other ways. For example, they are also used as opportunities for peer support, ART adherence tracking, and for providing meals to those who could not afford to eat a proper meal each week. Project indicators reported to USAID/Senegal in the annual reports demonstrate that more women than men were tested for HIV and received their test results as shown in Figure 19 below. Initial findings and gaps in disaggregated data also show that younger clients are missed in the targeting of HIV services and therefore require prioritization in future efforts. HIV testing is especially difficult for youth to take advantage of, and in turn they are at risk for infection, not knowing their status and not accessing the health care system. Figure 19: Gender Differences in HIV Testing Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 124 For 2014, the goal was to ramp up Gender and HIV in policy documents and further operationalize the strategy, as well as addressing Gender-based violence (GBV). However, while some progress was made, because of a funding gap in fiscal year 2014, activities have not yet been started. Several gender achievements supported by FHI 360 were:  Production of the National Policy on Gender, which included a practical manual and guidelines on Gender and HIV (a manual) and Community Guidelines on Gender and HIV for use by community volunteers.  Implementation of national and regional training-of-trainers on gender and HIV, targeting key actors such as the HIV Regional and District Steering committees, the Nursing Association, gender units at MOH, Hope for African Children Initiative (HACI), Rajasthan Network of People Living with HIV (RNP+), AIDS Service, and FHI 360 staff.  Implementation and promotion of couples CT approach from a gender perspective in the four southern regions.  Discussion with relevant line ministries and the Association of Female Lawyers to introduce Legal Assistance Desks called 'boutiques de droit.’ Legal assistant desks can be effective in addressing violence against women and equalities. USAID/Senegal may want to consider supporting the piloting of these desks, documenting their progress, and sharing lessons learned. FHI 360 stated that they would work with the HSS component IP (Abt) to develop partnerships with structures outside the health sector, including agricultural programs and microfinance institutions that work with women’s groups and associations to help vulnerable women (e.g., primarily clandestine sex workers) gain access to IGAs. FHI 360 intended to implement and then evaluate an IGA pilot, and based upon those results, they planned to expand to other regions. As a result, Abt and FHI 360 worked together to establish a CH insurance pilot for PLWHA to provide PLWHAs with money for their insurance contributions. A pilot was carried out in Kaolack where the membership was established and 33 people were supported in IGAs. However, the IGAs failed and members were not able to reimburse their initial loans, nor make their insurance contributions. Team EY is not clear on why the IGAs failed and did not receive an IGA assessment from FHI 360 evaluating the factors contributing to the lack of success of the intervention. Another form of IGA currently supported by FHI 360 is micro-gardens. Micro-gardens are small spaces that are cultivated with planting containers (such as wooden boxes) to grow vegetables that can then be sold or consumed. According to the cooperative agreement, the FHI 360 team will evaluate the impact on income and/or nutritional status of the approximately 30 gardens that were established and how to pursue this activity for scale-up. In collaboration with local authorities and other development partners, including the Peace Corps, FHI 360 expanded this intervention through PLWHA associations to try to establish five to 10 gardens per association per year. Team EY confirmed that FHI 360 worked with the Infectious Disease Department of the Fann Hospital in Dakar to set up vegetable gardens to provide food for 115 female and 140 male patients on ART. In Kédougou, micro-gardens were started to support 59 PLWHA (54 females and five men), and in Kolda five household micro-gardens were established which supported 17 people. Another collective micro-garden was started at the Ambulatory Treatment Unit (UTA) of Ziguinchor that supports 20 male and three female PLWHA. Six individual Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 125 micro-gardens were started, which resulted in six community meals and furnished 54 kilos of vegetables to 60 people. Unfortunately, FHI 360 found that there was no ownership or sustainability of this activity, and handed it over to the Ministry of Agriculture. USAID/Senegal informed Team EY that all nutrition funds were pulled at the mandate of USAID headquarters in Washington, DC. USAID/Senegal informed Team EY that nutrition funds "should be used to address malnutrition among children and breastfeeding women in Feed the Future target regions and not specifically for the HIV program. HIV funds can be used to support HIV-related nutrition activities such as community meals and microgardens. Given the challenges of sustaining these nutrition activities, the project is no longer directly supporting community meals and micro-gardens. FHI 360 continues to focus on strengthening the capacity to deliver home-based, community￾based, and mobile services. Development of manuals resulted in improvements in care, especially in reinforcing the quality of interactions between health care staff, CHWs, and clients to validate compassionate and timely care. Discussions with PLWHA and observations of health workers demonstrate a transformation in attitudes of health workers toward PLWHA, including MSM. There have been significant advances in disease-specific testing for TB and HIV; however, the use of the integrated treatment approach is still weak. As seen in Figure 20 below, only 67% of TB patients were tested for HIV (of which 10% of TB patients were co-infected with HIV). For those patients that were co-infected, 80% received the medicine cotrimoxazole to prevent against Opportunistic Infections (OI). However, only 37% of these individuals are currently under ART, indicating that HIV treatment services are not well integrated with TB treatment. Among HIV patients there is not sufficient systematic detection of TB due to difficulties of diagnosis in the absence of radiology.22 Figure 20: Treatment of HIV among TB patients FHI 360 utilized quality improvement methods which helped to create indicators to track patient encounters within the HIV treatment and care system. These activities aim to address the critical issue of lost-to-follow-up. The Quality Improvement Approach (AmQ) is a process that starts 22 “Health HIV/AIDS and TB Program Component Cooperative Agreement,” USAID/Senegal, September 27, 2011. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 126 from a situational analysis to better understand the problems at the UTA by reviewing the indicators (e.g., number lost-to-follow-up, number of children, number of PLWHA, weight charts with body mass index) and then setting targets for specific indicators. Practitioners lead the design of the appropriate training, plan coaching, and field monitoring meetings. At the beginning of the process, the number of people receiving treatment rose and the number lost-to￾follow-up declined. Moving HIV services closer to the community level helps PLWHA to access a more complete set of services. The ability to attract and serve more clients in this structure and offer a more complete set of services may contribute to better quality and possibly lower costs, as accessing care becomes easier with the referral center. A number of health facilities offer comprehensive HIV care (i.e., ART, TB/HIV, psychosocial, nutritional, malaria, RH, and palliative care) in ambulatory settings. The UTA has dedicated staff (e.g., doctor, nurse, social worker, mediators), equipment, and commodities to support HIV care. This is an example of success in the management of HIV integration that other health facilities might consider modeling their programs after, since clients reported benefits of receiving ART care and treatment services in a “one-stop-shop” approach. An issue that continued to surface, noted by FHI 360 in their 2013 annual report, was frequent stock-outs of consumables (e.g., lab reagents, test kits, pediatric ARVs). Health workers in the southern regions confirmed that access to laboratory services especially for the PMTCT was a barrier because of the lack of reagents available for the protocol testing (CD4 count). Concern was raised by doctors at the district levels and nurses at health posts sending samples to Ziguinchor for CD4 counts or Dakar for viral load testing, which resulted in increased waiting times for results. Team EY’s understands that FHI 360 is anticipating the initiation of test and treat in Senegal. The scientific rationale for opting for a test and treat approach is that it is a proven intervention for reducing transmission and is being used in many countries with high prevalence rates. The intervention is generally based on testing everyone in 'high risk' groups and areas of generalized epidemics, and then immediately treating all of those diagnosed positive, regardless of whether their immune system is damaged or meets the clinical definition to initiate ART (CD4 count of less than 350). Although this approach is effective and can be a strong public health intervention in specific setting, there are important factors and barriers to initiating test and treat in Senegal that deserves further consideration. For example, in an interview with the Governor of Sédhiou, he noted that one of the major challenges for the treatment of PLWHA was the inability to consistently perform lab tests, specifically the analysis of viral load testing across the southern regions because there was a lack of equipment and the capacity of lab technicians to perform the tests was weak. Coordination across donors is critical to sustaining gains made in HIV diagnosis, treatment, and care. For example in the UTA in Kolda, it was noted during interviews that there is collaboration between GFATM, USAID/Senegal, and the GOS in funding HIV testing of individuals. Figure 21 displays the percentage contribution of each partner to testing. Similar cost sharing also occurs in Dakar and Kaolack.23 23 Programme Sante de L’USAID Composante VIH/SIDA et Tuberculose- Rapport Annuel FY 2013,” FHI 360, January 2014. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 127 Figure 21: Percentage of HIV Testing Financed by Partners in Kolda The majority of FHI 360’s sub-partners conducted their interventions through 2013. Thereafter, a number of external factors associated with FHI 360 operations delayed funding disbursement to FHI 360’s partners which affected transfers, mostly to governmental institutions (regional and district). These partners reported problems with delays and missed opportunities to implement activities at the current planning levels. The situation is primarily attributed to changes in the model for sub-contracting to fixed obligation grants. This took time to organize and coincided with the departure of the Director of FHI 360 Senegal, which further delayed the signing of the contract with the RMHTs. FHI 360’s partners had access to other funding sources and, as a result, despite the delay in resources, the continuity of the interventions was maintained. For example, Sis Afrique benefits from funding by the Netherlands Ministry of Development Cooperation for MARPs activities. ENDA Santé and ANCS also received support from the GFATM (for MSM and CSWs) and LuxDev (Luxemburg) for cross-border interventions. Therefore, these resources from other donors indirectly benefited stakeholders when challenges with FHI 360’s resource flows may have stalled service delivery. 8.4.2 Lessons Learned and Recommendations • The provision of a comprehensive package of HIV treatment, care, and support services is dependent on the availability of health commodities at the service delivery site. Concerns were raised over the long waiting time for laboratory results for viral load and CD4 count.. USAID/Senegal may want to increase its role in verifying functional laboratory services and efficient supply chain operations for the proper diagnosis (e.g., CD4 count and viral load) and treatment. USAID/Senegal may want to receive supply chain expertise in the form of a key technical assistance partner to improve forecasting, distribution, and transfer of HIV commodities in FHI 360 supported sites. • The sustainability concerns related to a lack of ownership and direct involvement in MSM and CSW program by the GOS and sub-national levels are worth noting.24 NGOs and civil society organizations are key providers and leaders in interventions targeting CSW and MSM. USAID/Senegal may want to consider developing a strategy to engage national, 24 “Health HIV/AIDS and TB Program Component Cooperative Agreement,” USAID/Senegal, September 27, 2011. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 128 regional, and district government counterparts to understand the value, costs, and impact of these efforts. For example, contracting a firm or consultants for epidemiological modelling of MARPs groups could inform the future targeting of resources, identify size estimation of MSM and CSWs potentially in need of preventative and treatment services, and raise national awareness about the positive effects of action and the effects of inaction. Team EY was informed that FHI is actually conducting size estimation surveys in the South under their KPCF grant and may want to conduct similar activities in other regions. Meetings that facilitate conversations around this specific program are recommended in the future and as part of transitioning at the end of the USAID/Senegal funded HIV/AIDS program. Sub￾Component C: Strengthen planning, coordination and management of National HIV and TB program and advocacy and policy dialogue for HIV programming. 8.4.3 Findings and Analysis FHI 360 was critical in supporting CNLS in the development of the National Strategic Plan for HIV/AIDS. FHI 360 also provided both technical and financial support for the annual review of CNLS and the National Multi-Sectorial HIV Forum. Despite advances by the HIV program there are challenges in the functioning of the coordination and planning structures that exist at the regional and district levels. According to stakeholders from the MOH, and confirmed by interviews with regional and district MOH officials, FHI 360 has successfully strengthened the coordination of the HIV response. However, these coordinating bodies do not function in an inclusive and effective manner to support advocacy activities, or to create an environment to address and resolve policy issues, such as standardized per diem rates for community activities, financing options for HIV care at decentralized levels, or establishing a condom policy. FHI 360 supported the functioning of the regional and district coordinating bodies described below:  The Regional Technical Steering Committee is the executing agency of the CNLS and coordinates and monitors the implementation of the program at the decentralized level.  The District Technical Steering Committee is the executing agency for the district AIDS Council which coordinates and monitors program implementation at the district and local level.  The District "AIDS Pool" coordinates and monitors local actors implementing on behalf of the District Technical Steering Committee. It is the center of synergy between all the actors involved in district HIV/AIDS programs. FHI 360’s approach is to work at three levels to build upon existing partnerships and successful strategies currently being implemented as seen in Figure 22 below. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 129 Figure 22: FHI Approach In an effort to improve coordination, FHI 360 developed a partnership with the CNLS to help plan activities, provide adequate resources, and improve the visibility of support from the USAID/Senegal Health Program. At the CNLS level, the focus is on coordination, including advocacy and policy issues, and use of Strategic Information (SI) for reporting and decision￾making. At the DLSI and the National Tuberculosis Program (PNT) level, the focus is on program management. At the decentralized level, the focus is on planning, monitoring, and coordination. At all levels, the FHI 360 team will invest resources in a manner that complements the GFATM funding and activities with the focus on creating a collaborative environment. Various accomplishments reported in the annual report include FHI’s leadership in contributing to the improvement of collecting and reporting data to partners, which is linked to the RHMTs and DHMTs. Findings from the annual report included information on the 20 central level and decentralized institutions that jointly carry out SI activities (i.e., data analysis, data management, and data use for decision-making). The work of 61 decentralized coordination bodies (i.e., region, department, and district levels) that receive annual support for integrated management of the HIV response (e.g., coordination, planning, reporting, supervision, reviews) also improved the quality of service provision and linkages across the HIV care continuum. Improved Opportunities for Coordination It is unclear how the design of the integrated approach has contributed to FHI’s effectiveness overall and administratively. The HIV program of the MOH is a vertical program and it appears to be similar to the USAID/Senegal Health Program. Beyond coordination for the development of unified budgets and work plans at the regional levels, partners have not systematically taken advantage of opportunities to leverage strengths and fill gaps across other USAID/Senegal Health Program components. For example, FHI 360’s funds are used for the Regional Integrated Multi-Sectorial Plan (PRIM) based on vulnerability mappings and intervention gaps. However, the time lag between the planning and the mapping exercise, which was finalized after the PRIM development, means the findings of the mapping exercise were not used to inform the PRIM. Finally, apart from clinical aspects of the HIV response, FHI 360 mainly operates through NGOs (e.g., ANCS and ENDA Santé) with linkages to the health centers and health posts. Although not explicitly stated at the design phase, some opportunities that Team EY noted to improve coordination include:  The regional CNLS coordination could benefit from human resource strengthening to manage the HIV response and Abt could identify needs for future RH planning. Collaboration with Abt could also help stimulate health facility-based HIV interventions, Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 130 institutional strengthening and the planning/use of the mapping exercises (cartographies) for Integrated Multi-Sectorial Management. Another example is the inclusion of relevant HIV￾related indicators to PBF, which currently include CT and ANC for PMTCT, but does not address CSWs and MSM issues.  Collaboration with the HSI component (IntraHealth) could increase access to and coverage of quality HIV services such as routine CTs, treatment adherence support, defaulter tracing, and STI diagnosis and treatment (CSW and MSM). Most of these activities are still in the pilot phase and implemented by NGOs. FHI 360 could have benefited from using the opportunities provided by HSI’s approach of TutoratPlus, which shares some commonalities with FHI 360’s approach to coaching. Using both approaches could help to rapidly scale-up the training on care and treatment for doctors and nurses in anticipation of integrated HIV care and treatment into routine health care provision at health centers and post levels.  Collaboration with the CH component (the ChildFund consortium) for community mobilization and HIV activities at health huts level would be an important shift to the current program. ANCS and ENDA Santé mainly receive funding from FHI 360 through sub￾contracts. These two partners report to FHI 360 (financial and program data) and their own donors. The CNLS and DLSI are in charge of coordinating the governmental response, which is done through the different coordination bodies of the CNLS. All the IPs meet quarterly to discuss action plans and quarterly and annual reports. Capacity Building of Sub-Partners FHI 360 supports the capacity building of the DLSI in financial management. FHI 360 mainly provides sub-partners with technical and financial support to implement interventions and to help them to pilot innovative interventions (e.g., couple counseling, palliative care) or to scale-up promising practices such as eHealth for MSM. The partnership with ANCS (CBO/NGO HIV development umbrella organization) provides an opportunity to improve learning, linking, and sharing across regions and organizations. The RNP+ and the newly established network of MSM are in need of capacity building support to consolidate their approaches for empowering and managing networks. FHI 360 provided coaching and organizational development support through ACI during the period of 2006-2011. Now RNP+ is in a better position to provide capacity building to its members. Currently, FHI 360 is working with ANCS to help support capacity building of the MSM associations. At the operational level, regional advisors coordinate and oversee the planning and integration of sub-partner’s interventions as well as their implementation of these, including reporting (e.g., program data and financial data). FHI 360 also has a focal point for each region that works out of the Dakar office and backstops the regional advisors. The FHI 360 regional staff participates in all the coordination activities of the district and regions through their existing coordinating bodies (e.g., AIDS district and regional steering committees) with plans being consolidated at each level. The ANCS has its own coordination mechanism and channels for communication in the regions and districts. ANCS liaise with the FHI 360 regional advisor and their own headquarters. Subsequently, FHI 360 triangulates data it receives from the national coordination bodies with those it receives directly from ANCS, ENDA Santé and sub-partners. Delays in the Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 131 signing of the MOUs and changes to the fixed obligations grant between FHI 360 and its sub￾partners resulted in a lack of funding of these partners for almost a year. Although FHI 360 has explained the reason for this delay, interviews with these partners demonstrated that they are dissatisfied and do not fully understand the situation, resulting in displeasure with FHI 360. One of the key reasons cited for the delay in disbursement stated to Team EY during the evaluation was that since the departure of the COP, the validation or approval of key documents must “be decided at the Headquarter level.” This contributed to the lengthy process and time for decisions to be made. FHI 360 staff further noted that the change in funding levels (i.e., decrease in TB funding) and the departure of the COP disrupted a number of other planned activities for 2014. Policy Dialogue: HIV/AIDS and Key Population Challenge Fund (KPCF) In year three (according to interviews with FHI 360 staff), FHI 360’s work emphasized the KPCF with special attention to access quality care for CSW and MSM, and advocacy toward the creation of a supporting cross-border environment for MARPs. The KPCF’s interventions focused on three select regions (i.e., Kolda, Ziguinchor, and Sédhiou) with relatively higher HIV prevalence and the need to address cross-border migration effects in driving the epidemic. Target populations include CSW and MSM to facilitate access and use of sexual health and HIV prevention and care services. The main planned activities include research on estimates of the MSM and CSW population sizes, improving access to quality STI/HIV prevention services, and engaging in cross-border capacity building and advocacy (e.g., decision makers and health care providers) to integrate key populations in HIV services provision. Year one and year two data show a steady increase in the number of MARPs reached with prevention interventions (e.g., ANCS, DLSI, RNP+ and Sis Afrique) with over performance in the second year. According to interviews, there was a decrease in performance (below 80% for all target populations) in year three. Reaching out to the MSM community is currently a priority of the Senegalese response to HIV/AIDS. MARPs are explicitly mentioned in the National Strategic Plan 2014-2017. According to the CNLS and the DLSI, the contribution of the USAID/Senegal Health Program could be traced to ACI/ENDA Santé’s earlier advocacy work in the 2006-2011 phase during which FHI 360 provided support to the working group on MSM (Groupe de Reflexionsur les MSM). FHI 360 has provided support for the sensitization of religious leaders and decision makers, health care providers, as well as journalists with limited exposure to the problem and thus, perpetuated negative stereotypes on MSM. During an outreach event, issues of stigma against MSM were discussed thereby creating greater visibility of MSM (an initiative led by the Dutch Ministry for Development Cooperation). As such, FHI 360, with USAID/Senegal support is among the major development partners to support interventions addressing the unmet health needs of MSM. Starting in 2011 FHI 360’s involvement has decreased due to the sensitivity of the issue, and the GFATM is now leading the policy dialogue. Although FHI 360 contributed to the initial discussions and to the development of the Conceptual Framework for addressing HIV among MSM, based on field interviews, they did not follow-up with support to MSM associations with the same level of support provided to associations of PLWHA. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 132 8.4.4 Lessons Learned and Recommendations • Fund disbursement in 2014 was partly compromised by delays in reporting (e.g., financial, data collection, program indicators). USAID/Senegal may want to request that FHI 360 invest in training regional and district medical officers, CBO, and NGO stakeholders in financial management, data collection, program reporting, M&E, and strategic planning. Based on the qualitative data collected by Team EY, USAID/Senegal may want to consider further exploring the internal issues related to administrative bottlenecks with the FHI 360 in￾country office since they appear to be affecting the disbursement of funding to sub-partners. USAID/Senegal might be able to facilitate resolving some of the bottlenecks in the process by discussing this issue with the in-country office. • The absence of resources devoted to meetings, shared site visits, and strengthening health management information systems has led to concerns regarding which activities will continue and which would cease without USAID/Senegal or another international donor’s financial support. USAID/Senegal may want to consider having the costs for regional coordination be built into the budgets of decentralized structures (i.e., local government structures) rather than solely managed and funded by international development partners. 8.5 Response to Evaluation Question The table below provides information to respond to the component-specific evaluation questions as stated in the scope of work. It is not intended to be exhaustive, but rather aims to highlight notable successes; constraints and challenges that have been experienced during implementation by FHI 360; and key interventions that may be added, continued, or removed within Component 4. Team EY and USAID/Senegal agreed that this analysis would be conducted by component rather than by individual sub-components. Table 30: Component Table Component Specific Question Analysis To what extent have the components achieved their objectives? FHI 360 made progress towards achieving their objectives as outlined in Section 8.1. The USAID standard indicators outlined in Table 28 were selected as a sample as they are closely aligned with USAID/Senegal’s IR 1: Increased availability of an integrated package of quality health services and IR 2: Improved health seeking and healthy behaviors. Additionally, these indicators support one of USAID/Senegal’s high-level indicators: percentage of the target population who knows how to prevent key illnesses (e.g., HIV/AIDS, malaria). The indicators in the table represent core HIV interventions across the HIV continuum including prevention activities, diagnosis, treatment, care for adults and children, health workforce skill development, and targeting of priority populations such as MSM and CSWs. According to FHI 360 indicator data, FHI 360 is meeting the majority of its targets for those indicators where data was reported, however, multiple indicators do not have data. The root cause of the lack of data varies indicator by indicator and should be addressed and remediated Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 133 by FHI 360 as feasible. To what extent has each sub￾component been successfully implemented? What are the factors contributing to the achievement of each sub￾component?  Support for GOS Coordination: FHI 360 provided leadership and support, which resulted in the development of three coordinating bodies (Regional Technical Steering committees (executing agencies of the CNLS), the District Technical Steering Committees, and the District “AIDS Pool”). Regular meetings by these bodies appear to help facilitate coordination and improve HIV program implementation. Factors contributing to success are attributed to an inclusive approach and providing the organizational support for these bodies to meet regularly to share information and knowledge, which improves coordination.  Implementation of UTA Interventions: FHI 360 supported various MARP (i.e., MSM) civil society partners (e.g., NGOs, CBO), which contributed to improved access to HIV services for the MARPs in specific communities. FHI 360 supports one mobile clinic in per-urban Dakar (for RH services and VCT for CSW) and recently opened a second one in Ziguinchor as part of KPCF. The UTAs were highlighted in interviews as a positive intervention that provided services to MARP populations, notably MSMs and CSWs. USAID/Senegal has helped to build and maintain the infrastructure and Global Fund supported staff, reagents, equipment, etc. to support the UTA. For example, in interviews with regional and district medical teams in Kolda, it was reported to Team EY, that FHI 360 was vital in supporting their HIV program through the UTA. Specifically, the training of human resources (e.g., doctors, nurses, social assistants, and care and support staff) and providing necessary equipment and commodities for care and treatment services were stated as examples of how access was improved for PLWHA. One of the factors contributing to success is that services provided by the UTA are aligned with the specific needs of MARP populations and health staff is trained on how to respond to issues that pertain to MARPs that might not be relevant to the general population. However, it was highlighted in an interview with the AOR for FHI 360 that this model could potentially be problematic as there are concerns with the sustainability aspect of the model.  Mapping of Most Vulnerable Populations: FHI 360 implemented a mapping of the most vulnerable populations in in five regions nationwide (according to FHI’s quarterly reports: Thiès, Ziguinchor, Sedhiou, Kolda and Kédougou). This mapping was considered “innovative” by local stakeholders as a tool to use data and information to inform programming of HIV services in the region. One of the factors contributing to success was that stakeholders in the region believed they had a better understanding of where the most vulnerable populations in their communities were located based on actual evidence and not anecdotal data. Therefore, stakeholders encouraged the actual use of the data when making program decisions, and stakeholders specifically in Sédhiou stated some concern that the data would not be Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 134 critically applied to course-correct or tailor interventions to the specific needs of the region but rather a blanket set of HIV interventions would be implemented.  Focus on loss to follow-up of ART patients: FHI 360 in collaboration with the associations of PLWHA and staff of the UTAs are working on defaulter tracing through activities of the mediators and the weekly ‘repas communautaires.’ According to the RM of Ziguinchor, in collaboration with the LuxDev program (Luxembourg), ENDA Santé and the RMT have developed a list of service providers and referral and counter referral tools that are used across the various border countries (i.e., Guinea Bissau and the Gambia). These tools have led to improved communication and allowed service providers and CHWs to better manage the loss to follow-up of mobile populations on ART. Additionally, FHI 360 utilized quality improvement methods to create indicators to track patient encounters within the HIV treatment and care system. These activities led to helping providers in addressing the critical issue of lost-to-follow-up. Factors contributing to this success was 1) identifying and understanding the need for cross-border interventions to manage transient populations that were on ART to decrease the potential for loss to follow-up 2) supporting the coordination of RMTs to develop the tools to help mitigate loss to follow-up, and 3) supporting the utilization and updating of the tools to more effectively manage transient ART patients. What are the constraints and challenges that have hindered successful implementation of each sub￾component, and how has the IP dealt with those challenges? Cross-cutting constraints and challenges include:  Funding disbursement delays: Funding issues in the last 10 months resulted in delays in FHI 360’s funding disbursements to sub-partners and stalled some activities implemented by the regional medical health teams. One of the key reasons cited for the delay in disbursement stated to Team EY during the evaluation was that since the departure of the COP, the validation or approval of key documents must “be decided at the Headquarter level.” This contributed to the lengthy process and time for decisions to be made. FHI 360 staff further noted that the change in funding levels (i.e., decrease in TB funding) and the departure of the COP disrupted a number of other planned activities for 2014. FHI 360 did not address how they will deal with this challenge in the future; however, Team EY suggests further in the document that USAID/Senegal explore opportunities to help the in-country office facilitate improving the current situation.  Decreased TB funding: The ending of TB funding and the focus only on TB-HIV activities has negatively affected the MOH’s ability to implement its activities. According to interviews with MOH stakeholders, the decrease of funding for TB has created a larger gap in services and contributed to frustration among the GOS health staff. Although USAID/Senegal is no longer funding TB as a separate component under FHI 360, resources are used to support the DLSI in the area of TB and HIV co-infection. FHI 360 is supporting the TWG Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 135 for TB-HIV co-infection, which includes members from the DLSI, the National TB Program, and FHI 360. The TWG carries out periodic reviews where they exchange strategies and new developments for addressing HIV and TB co-infection. Team EY understands that this decrease in TB funding was not attributed to FHI360’s performance and was mandated by USAID/Washington and, therefore, it is not the responsibility of FHI 360 to resolve this challenge.  Management burden of sub-partners: FHI primarily operates through NGOs/CBOs implementing HIV/AIDS interventions in the regions. It is Team EY’s understanding that FHI has 17 local partners implementing HIV interventions. While FHI 360 is focused on capacity-building of its sub-partners, USAID/Senegal noted some potential concerns with the management burden of supporting a large number of sub-partners. Additionally, there might be issues of sustainability since the majority of these sub-partners are dependent on FHI 360 and do not have a diverse set of funding sources. If HIV funding decreases, in the same manner that occurred with TB funding, these local sub-partners are vulnerable to closing. Despite the success of the UTAs, the RM from Kolda noted that the health center in Kolda is struggling because of the delays in funding from FHI 360. This is impeding their ability to increase the number of beneficiaries that are being supported. It is important to note that Team EY did not conduct a detailed analysis of FHI 360’s capacity building activities due to the scope and duration of the evaluation and, therefore, this would need to be further analyzed to verify and validate the concerns noted above. Therefore, Team EY is not able to provide additional information on how and if FHI 360 has dealt with this challenge. Technical constraints and challenges include:  Low uptake of couples counseling: FHI 360 supported SWAA’s gender approach to couples counseling, which contributes to the promotion of male involvement. In some health posts, nurses’ support couples counseling and testing (CT). However, despite the efforts made toward the promotion of couples CT, the uptake and performance of the strategy is still low (i.e., 2,095 couples were tested: accounting for 16% of the target). FHI 360 stated that not reaching its targets was partly due to the delay in launching new BCC campaigns focused on couples counseling, which they believe will be resolved in the first quarter of 2015. According to the FHI 360’s annual report, the main challenges include the following: o Lack of ownership of the strategy by health care decision makers and workers. o Limited number of trained CHWs involved due to the lack of incentives. o Insufficient number of trained health care providers. o Insufficient quantity of communication materials (promotional Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 136 and educational leaflets). Qualitative data collected by Team EY supported the challenges in the information provided in FHI 360’s annual report and also noted the lack or inconsistent capacity/skills in how to approach couples counseling by service providers outside of specific HIV services.  Ineffective IGA interventions: FHI 360 stated that they would work with the HSS component IP (Abt) to develop partnerships with structures outside the health sector, including agricultural programs and microfinance institutions that work with women’s groups and associations to help vulnerable women (e.g., primarily clandestine sex workers) gain access to IGAs. FHI 360 intended to implement and then evaluate an IGA pilot and, based upon those results, then expand it to other regions. As a result, Abt and FHI 360 worked together to establish a Community Health insurance pilot for PLWHA to provide PLWHAs with money for their insurance contributions. A pilot was carried out in Kaolack where the membership was established and 33 people were supported in IGAs. However, the IGAs failed, and members were not able to reimburse their initial loans, nor make their insurance contributions. Team EY is not clear on why the IGAs failed and did not receive an IGA assessment from FHI 360 evaluating the factors contributing to the lack of success of the intervention.  High levels of stigma and discrimination towards MARPs and PLWHA: MSM and CSWs face additional barriers to seeking and utilizing health services, besides discrimination and stigmatization. Fear of prosecution, community exclusion, and weak laws to protect MSM and CSWs in cases of police abuse and unfair treatment affect their decision-making. HIV testing yields for MSM are low compared to the general population. This is likely attributed to issues of stigma, general negative views regarding PLWHA, and the lack of societal support for those who come forward for services. FHI 360 is one of the largest development partners supporting services to address the unmet health needs of MSM. However, since 2011 support began decreasing due to the sensitivity around the issues of MSM. The Global Fund has now taken over the primary leadership role for MSM interventions. Additionally, interviews noted that the perception of MSMs is that many of the services supported through FHI 360 for MSM are clinical and do not directly address issues of disclosure of HIV status and the “secrecy” surrounding MSM in Senegalese communities. Are there interventions that should be added or removed?  Targeted concentration on MARPs services: Health communication approaches need to take into account the complex factors (i.e., stigma and discrimination, fear of disclosure) that affect MARPs with subtle and inclusive messaging and outreach activities. Since MSM and CSWs in Senegal are disproportionately affected by HIV/AIDS, USAID/Senegal may want to consider increasing services (UTA) for key populations. However, they should consider linking mobile services to fixed facilities to reinforce the continuum of care in the regions and Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 137 districts. FHI 360 should continue to address the unmet health needs of MSM, especially young men with a greater focus on research by region and the provision of wrap around services that support positive living, disclosure, and ways to manage stigma and discrimination in the household and community. The program benefit of targeting young men in major cities or hubs for CT through mobile services includes encouraging the use of extended testing hours, and providing privacy to attract them to services. Other than CSWs and MSM who require this support, a benefit of targeted interventions for discordant couples and transient populations (i.e., truck drivers, fishermen, gold miners, and migrant workers) could strengthen tailored health communications.  Expand focused CT for MARPs: USAID/Senegal may want to request that FHI 360 work more closely with the GOS to provide solutions that address quality and access for key populations and PLWHA beyond KPCF. The availability of safe and reliable CT services is required to help clients to know their status and link into prevention, treatment, and care services. The limited number of CT sites outside of Dakar presents a challenge for prevention programming, given that many people do not know their HIV status and, therefore, do not receive the necessary follow-up support. FHI 360 should consider prioritizing their work with the RHMTs and DHMTs to advocate for and deploy resources in CT activities. Investments in the use of BCC through innovative strategies that use mobile technology could be explored with NGOs, public sector providers, and private sector partners to increase demand for CT services among MARPs and to improve the uptake of couples counseling.  Reconsider beginning ‘test and treat’ interventions: Team EY’s understands that FHI 360 is anticipating the initiation of test and treat in Senegal. The scientific rationale for opting for a test and treat approach is that it is a proven intervention for reducing transmission and is being used in many countries with high prevalence rates. The intervention is generally based on testing everyone in high risk groups and areas of generalized epidemics, and then immediately treating all of those diagnosed positive, regardless of whether their immune system is damaged or meets the clinical definition to initiate ART (CD4 count of less than 350). Taking into account the very low prevalence rate in the general population, the resource constraints in Senegal, and the commitment to patients’ lifelong needs for ART, Team EY believes that an alternative to the test and treat model deserves further consideration. Additionally, in an interview with the Governor of Sédhiou, he noted that one of the major challenges for the treatment of PLWHA was the inability to consistently perform lab tests, specifically the analysis of viral load testing across the southern regions because there was a lack of equipment and the capacity of lab technicians to perform the tests was weak. A focus on maintaining high quality ART services for those already on treatment, meeting the unmet need for those who clinically should be on ART, and supporting more cost-effective interventions Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 138 such as the treatment of opportunistic infections could be a more feasible public health approach for managing the HIV/AIDS epidemic. Are there changes that could be made to improve performance?  IGA Interventions: Based on the challenges and poor performance of IGAs, FHI 360 could re-focus on other community level activities and should consider no longer allocating additional funding for IGAs. An alternative could be to further assess why the IGA interventions failed and redesign them with the lessons learned from the assessment and best practices from other programs where IGA interventions succeeded.  Strengthen the Decision-Making Authority of FHI 360: Based on the qualitative data collected by Team EY, USAID/Senegal may want to consider further exploring the internal issues related to administrative bottlenecks with the FHI 360 in-country office since they appear to be affecting the disbursement of funding to sub-partners. USAID/Senegal might be able to facilitate resolving some of the bottlenecks in the process by discussing this issue with the in-country office.  Increased Collaboration across other USAID/Senegal supported Health Components and Partners: FHI 360 may want to consider focusing more efforts on strengthening areas of collaboration with other IPs working in the same regions. One example that was noted in many qualitative interviews and focus group discussions was the interest in expanding MHO (mutuelles) in the southern regions. Team EY’s understanding is the MHO program was expanded to Kolda and Ziguinchor in FY 2014 and is no longer linked to IGA. This would be a potential area of further expansion (beyond the one pilot of MHO linked to IGAs) with Abt after the MHO interventions in Kolda and Ziguinchor were implemented and then evaluated. MHOs were noted in several interviews as useful, with one MHO stating (MHO manager in Kaolack) that “MHO’s are very useful. They allow us to sustain services, which is why we want to promote membership. Adding more members will also help us because now our budget is inadequate.” USAID/Senegal may want to consider adding activities that increase the coordination and the expansion of MHOs supported by Abt and FHI 360 that are not linked to IGA (since the pilot was not successful). 8.6 Data Sources In addition to sources cited in Annex G: Bibliography, data collected in the field was used for analysis of this component, including: in-depth interview with the COP, RB coordinators, key MOH stakeholders, regional and district health officers, regional and district coordinating offices, community-based health insurance managers, Directors of the BREIPS, service providers at health facilities; services providers at health huts, pharmacists, private firms; as well as focus group discussions with CHWs, CH committees, community-based health insurance beneficiaries, and associations of PLWHA. Data collection tools for field interviews are available in Annex I and Annex J. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 139 9.0 COMPONENT #5: HEALTH COMMUNICATION AND PROMOTION 9.1 Background USAID/Senegal awarded ADEMAS, a five-year $22M USD cooperative agreement in 2012 to implement the HCP component of the USAID/Senegal Health Program. The HCP component is intended to foster positive health practices in households and communities and support changes in social norms that will reinforce and maintain these practices. The component was designed to contribute to the achievement of IR 1: Increased availability of an integrated package of quality health services, IR 2: Improved health seeking and healthy behaviors, and is also expected to contribute to IR 3: Improved performance of the health system. ADEMAS has partnered with Population Services International (PSI) and the Regional Center for Training and Research in Reproductive Health (CEFOREP) for the execution of this component as stated in their original cooperative agreement. It is important to note that ADEMAS’s sub-contract with CEFOREP was terminated in October 2014 for “material non-conformity.” The HCP component supports a range of activities in all 14 regions to support social and behavior change to improve outcomes in RH, MCH, malaria, HIV/AIDS, TB, and other infectious diseases. The program works in close partnership with the MOH, in particular, the SNEIPS, the DSRSE, and the National Disease Control Programs for AIDS and STIs, malaria, and TB, as well as other ministries (i.e., the Ministry of Education), USAID IPs, NGOs, FBOs, private sector entities, and various other local partners. The project is organized into five sub￾components.  Sub-component A: Strengthening capacity for effective BCC programs.  Sub-component B: Supporting implementation of quality BCC interventions leading to the adoption of healthy behaviors and the increased use of health services.  Sub-component C: Strengthening the capacity of key actors to advocate for political and social engagement for health programs.  Sub-component D: Social marketing of key health products resulting in their increased sale and use.  Sub-component E: Technical capacity building and organizational development of the recipient. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 140 Table 31 below shows the standard indicators related to the HCP component in the first two years compared with targets. Although Team EY understands that ADEMAS provided its year three annual report to USAID/Senegal in November 2014, Team EY did not receive the report in time to include the 2014 indicator data into the table below. However, information derived from the report that was verified by data point/site interviews and corroborated by alternative sources was utilized in our analysis. Table 31: USAID Indicators for HCP25 # Indicator Target Actual Target Actual 2012 2013 Sub-component A: Strengthening capacity for effective BCC programs 1 A National Health Promotion Plan finalized with best practices in BCC incorporated into the stated policies and procedures Plan Prepared Plan Prepared but not validated Plan validated 0 2 # of individuals trained as DELTA (tool used for designing health promotion campaigns for social marketing) trainers * N/A 10 15 3 # of regional workshops held for partners to present evidence based methodology for developing communication strategy * N/A 3 4 Sub-component B: Supporting implementation of quality BCC interventions leading to the adoption of healthy behaviors and the increased use of health services 4 % of women of reproductive age with positive beliefs about FP * * * * 5 Average score on Likert scale statements on husband’s support of their use of FP * * * * 6 % of mothers or caregivers of children under-five who know aquatabs is a home water treatment product * * * * 7 # of national health communication campaigns developed and implemented, led by SNEIPS * * 1 1 8 # of PPP in place to support HCP activities * 0 2 0 9 # of individuals reached through Interpersonal Communication (IPC) and community outreach that promotes HIV/AIDS prevention beyond abstinence and/or being faithful * No systematic collection 25,000 0 10 # of individuals of the target reached through IPC and community-based outreach activities for FP No systematic collection No systematic collection 20,000 0 11 # of individuals of the target reached through IPC and community-based outreach activities for child survival No systematic collection 0 20,000 0 12 # of radio spots aired promoting healthy behaviors 1,307 4,553 2,000 4,553 13 # of TV spots aired by promoting healthy behaviors * 79 133 502 25 “Performance Monitoring Plan (PMP)/Results Framework for Health Communication and Promotion Program in Senegal,” ADEMAS, February 4, 2013. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 141 # Indicator Target Actual Target Actual 2012 2013 Sub-component C: Strengthening the capacity of key actors to advocate for political and social engagement for health programs 14 # of training events for media and on health communication * 0 0 0 15 # of annual coordination and advocacy meetings held with one of the identified key stakeholders * 1 2 0 16 # of local “champions” identified for each priority health area * 0 1 0 17 % increase in budget support for HCP at the district level * 0% * 0 Sub-component D: Social marketing of key health products resulting in their increased sale and use 18 % of women of reproductive age in project areas knowledgeable of availability of injectable contraceptives (Depo Provera) * N/A * * 19 % of women of reproductive age in project areas who report knowledge of availability of oral contraceptives (Securil) * N/A * * 20 % of women with children under the age of five who report knowledge of availability of water treatment solution (aquatabs) * N/A * * 21 % of respondents knowledgeable of availability of long lasting treated ITNs N/A * 24.3% * 22 # of tablets of aquatabs distributed * 322,200 1,985,962 1,293,520 23 # of liters of drinking water disinfected with aquatabs * 6,444,000 39,719,260 25,870,400 24 # of long-lasting ITNs distributed * N/A 115,000 13,604 25 # of Securil oral contraceptives distributed over life of project * 299,444 407,865 485,136 26 # of Depo-Provera injectable contraceptives over life of project * 11,298 26,118 23,193 27 Total CYP delivered over the life of project with USG support * 46,615 85,120 89,066 Sub-component E: Technical capacity building and organizational development of the recipient 28 # of ADEMAS staff trained in research with USG assistance 1 1 2 40 29 # of research activities conducted by ADEMAS * 0 6 6 30 # of ADEMAS staff completed management training * 1 3 3 31 # of ADEMAS staff trained as DELTA facilitators * 0 2 2 32 # of marketing plans developed and implemented by ADEMAS using DELTA 0 2 2 5 33 Proportion of ADEMAS staff who have achieved annual performance goals * N/A 100% 100% 34 ADEMAS assessment tool completed and action plan updated annually 1 1 0% 0% *Data not available. Note: Target and actual data included in the table above is derived directly from annual reports provided by IPs. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 142 The data from the HCP indicators above are derived from ADEMAS quarterly and annual reports available. These 34 indicators consist of the full set reported and align with USAID/Senegal’s IR 2: Improved health seeking and healthy behaviors. They are expected to contribute indirectly to the achievement of IR 1: Increased availability of an integrated package of quality health services and IR 3: Improved performance of the health system. Additionally, these indicators support USAID/Senegal’s high-level indicators of 1) percentage of target population who know how to prevent key illnesses (e.g., HIV/AID, malaria), 2) use of ITNs by household members, 3) data used to guide program design, and 4) CYP. The sub-components, as a collective set of indicators provide performance data on efforts that lead to the adoption of healthy practices in households and communities linked to improved outcomes in the areas of RH, MCH, malaria, HIV/AIDS, TB co-infections, and other infectious diseases. The indicators under sub-component A, strengthening capacity for effective BCC programs, address critical PE questions related to whether national, partner, and individual efforts produce positive changes in policy and increased knowledge. The indicators under sub-component B, implementation of quality BCC interventions, support understanding of PE questions associated with the quality of health communication activities that reach women, partners, caregivers, community members, and messaging within national campaigns and small group Information, Education, and Communication (IEC) encounters. Sub-component C, strengthening the capacity of key actors to advocate, looks at the introduction and use of political and social engagements to facilitate an enabling environment for prioritizing health issues and care. Sub-component D, social marketing of key health products, addresses PE questions of whether levels of knowledge on and availability of, injectable and oral contraceptives, aquatabs, and ITNs increased as a result of component interventions. The final sub-component E indicators, technical capacity, and organizational development of ADEMAS show to what extent the staff and institutional capacity of ADEMAS were developed and improved over the course of the project. Overall, ADEMAS showed some positive results of improving and reporting on indicators from year one to year two. A detailed analysis for the year two indicators can be found below. Additional information on indicator achievement status can be referenced in Figure 23 below. ► Targets Met: The results indicate that ADEMAS supported sites almost met, met, or exceeded targets across the five sub-components for 17 indicators in year two. No targets were met for sub-component C. In sub-component A: 1. Number of individuals trained as DELTA trainers. 2. Number of regional workshops held. For sub-component B: 1. Number of national health communication campaigns developed and implemented, led by SNEIPS 2. Number of radio spots aired promoting healthy behaviors. 3. Number of TV spots aired by promoting healthy behaviors. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 143 For sub-component D: 1. Number of tablets of aquatabs distributed. 2. Number of liters of drinking water disinfected with aquatabs. 3. Number of Securil oral contraceptives distributed over Life of Project. 4. Number of Depo-Provera injectable contraceptives over Life of Project. 5. Total CYP delivered over the Life of Project with USG support. Lastly, for sub-component E, all of the seven indicators were met or exceeded expectations. These include: 1. Number of ADEMAS staff trained in research with USG assistance. 2. Number of research activities conducted by ADEMAS. 3. Number of ADEMAS staff that completed management training. 4. Number of ADEMAS staff trained as DELTA trainers. 5. Number of marketing plans developed and implemented by ADEMAS using DELTA. 6. Proportion of ADEMAS staff who have achieved annual performance goals. 7. ADEMAS assessment tool completed and action plan updated annually. According to the 2012-2013 annual report, key factors which influenced positive performance against the sub-component A through D targets possibly included the wide community involvement and acceptance of BCC interventions in many regions, the perceived benefits of medical products and commodities, and a high interest in FP information and methods such as preferences for select contraceptives (e.g., Securil contraceptive and condoms) among clients. These gains across components may also be associated with the effects of capacity built from HCP trainings and investments in the creation of materials, tools, and media products developed and leveraged by ADEMAS, especially in regions with multiple partners. ► Targets Not Met: Ten indicators reported low performance against targets. One was in sub-component A as it was not confirmed in the data that the National Health Promotion Plan was finalized with best practices in BCC incorporated into the stated policies and procedures. Four were in sub￾component B and include: 1. Number of PPP in place to support HCP activities. 2. Number of individuals reached through IPC and community outreach that promotes HIV/AIDS prevention beyond abstinence and/or being faithful. 3. Number of individuals of the target reached through interpersonal IPC and community-based outreach activities for FP. 4. Number of individuals of the target reached through IPC and community-based outreach activities for child survival. None of the four sub-component C indicators were met, including: 1. Number of training events for media and on health communication. 2. Number of annual coordination and advocacy meetings held with one of the identified key stakeholders. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 144 3. Number of local champions identified for each priority health area. 4. Percent increase in budget support for HCP at the district level. One indicator in sub-component D was not met and includes the number of long-lasting ITNs distributed. Challenges identified in meeting these different indicators, described in the annual report, ranged from constraints in human resources with respect to convening authority for multi￾stakeholder planning and BCC strategy development; varying capacity of CBOs’ carried out BCC activities and gaps in supervisory support across the regions; as well as ADEMAS’s limited influence with ministries and private sector leaders which strain its ability to scale-up effective interventions in all 14 regions. ADEMAS continues to work through these issues in building relationships, expanding its research activities, and disseminating information on progress made in HCP efforts. These efforts have the ability to better establish the organization as a trusted leader and implementer in communities and among local governance structures. ► Data Not Received: A number of targets were not set and actual results were unavailable or unreported in year one and year two of ADEMAS reports. However, the number of indicators with data not received decreased from year one (31 indicators) to year two (seven indicators). The seven indicators that did not have data available for year two were found in sub-components B and D. Three were in sub-component B and include: 1. Percent of women of reproductive age with positive beliefs about FP. 2. Average score on Likert scale statements on husband’s support of their use of FP. 3. Percent of mothers or caregivers of children under-five who know aquatabs is a home water treatment product. Four were in sub-component D and include: 1. Percent of women of reproductive age in project areas knowledgeable of availability of injectable contraceptives. 2. Percent of women of reproductive age in project areas who report knowledge of availability of oral contraceptives. 3. Percent of women with children under-five who report knowledge of availability of water treatment solution. 4. Percent of respondents knowledgeable of availability of long-lasting ITNs. Gaps in target setting may be explained by a lack of baseline data collected or existing knowledge, attitudes, and practices in certain regions. Team EY received no additional information from the work plans, annual reports, or interviews to substantiate the reasons for missing standard indicators. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 145 Figure 23: HCP Indicator Status 9.2 Key Findings and Recommendations/Benefits ADEMAS is making progress in meeting some its objectives (please reference Table 31 above for achievement of indicators to-date). The analysis indicates that ADEMAS’s strength lies in social marketing of health products and as technical advisors for community-based HCP. However, there are significant concerns regarding their ability to effectively contribute to a leadership role in building the institutional capacity of the MOH, as noted in the ADEMAS annual reports. Key findings and recommendations are provided in further detail in Section 9.3 – 9.7 below. Table 32 below summarizes the most important findings and recommendations relevant to the HCP component. Table 32: HCP Key Findings and Recommendations/Benefits # Key Findings Recommendations/Benefits 1 Social marketing of health products was the most successful part of this component according to ADEMAS. The social marketing of products supported the promotion of the integrated package of services in the USAID/Senegal Health Program. Recommendation: USAID/Senegal may want to continue to expand social marketing of key products through ADEMAS and strengthen the BCC campaigns developed to support the products. In addition, USAID/Senegal may want to request that ADEMAS use innovative and integrated platforms like social media and mobile technology to reach target populations, especially youth and MARPs. Benefit: Social marketing may increase the knowledge, demand for, and use of life-saving products through targeted marketing. 2 According to constraints cited by ADEMAS in their annual reports, ADEMAS struggled to establish relationships and leverage influence over central government agencies to improve institutional capacity building. Recommendation: USAID/Senegal may want to consider engaging another partner with expertise and experience in institution capacity building to provide this support directly to the MOH. Another option USAID/Senegal may want to consider is providing technical assistance to ADEMAS to Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 146 # Key Findings Recommendations/Benefits improve their internal capability to provide institutional capacity building. Benefit: Bringing in another partner may allow ADEMAS to focus on their strengths (i.e., social marketing) while another partner can focus on the gap of institutional capacity building to better strengthen management and financial aspects. However, since ADEMAS is already beginning year three of their contract, it might not be worth replacing this sub-component with a new partner and thus, providing technical assistance to ADEMAS may be a more feasible solution. 3 Challenges in collaboration between ADEMAS and the ChildFund consortium are hindering performance of both IPs’ components. For ADEMAS, this is limiting its ability to effectively collaborate with outreach workers and local CBOs. Recommendation: USAID/Senegal may want to consider discussing with ADEMAS more effective ways for collaboration at the community level for activity implementation. To support this, USAID/Senegal may want to consider facilitating a discussion with the ChildFund consortium and ADEMAS on methods to improve collaboration at the community level. During this discussion, USAID/Senegal may want to make a clear delineation between the responsibilities of ADEMAS and the ChildFund consortium in their work with the CBOs (e.g., have the ChildFund consortium responsible for mobilization in the CBOs and ADEMAS be responsible for HCP activities). USAID/Senegal may also want to consider building required indicators for improved collaboration between the USAID/Senegal supported IPs into the structure of the program. Benefit: This may improve ADEMAS’s ability to effectively communicate and collaborate between partners, outreach workers, and local CBOs. 9.3 Sub-component A: Strengthening capacity for effective BCC programs 9.3.1 Findings and Analysis The findings and analysis for sub-component A centered around training activities, the creation of the CBO network, the national communications framework, PPPs, and the M&E activities for communication campaigns. Details are described below. Training Activities Training of government agencies in the development, implementation, and monitoring of evidence-based BCC campaigns was a core activity under this sub-component. In particular, Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 147 ADEMAS worked closely with SNEIPS to raise its capacity in HCP. The institutional diagnosis was an important component of this capacity building. The institutional diagnosis assesses an organization’s capacity in the following areas: management, communication, health promotion, finance, research, and M&E. Based on this assessment, capacity needs are identified and a training plan developed. A “train the trainers” session was conducted for 10 SNEIPS agents and five directors of the BREIPS in the DELTA method approach for the development, implementation, and monitoring of marketing and communication campaigns. The DELTA method is a comprehensive planning process for social marketing, rooted in a deep understanding of target audiences and the underlying factors influencing their behavior. It helps when planning the delivery of appropriate interventions to understand when, where, and how the target audiences would like to receive them. The “train-the-trainers” session was subsequently followed by a training of decentralized MOH staff in all 14 regions. The participants trained the regional and district health teams responsible for HCP activities. In total, 12 of 14 BREIPS directors and 68 of 70 agents for EIPS were trained. With one BREIPS staff per region, there is a need for continuous support to Directors in their oversight of the 75 district level EIPS implementing health communication interventions. ADEMAS could help BREIPS improve the quality of BCC campaigns through closer monitoring of health messaging, campaigns, and interventions carried out with NGOs, FBOs, and CBOs in their region. In Matam, the BREIPS Director expressed a desire for support from ADEMAS staff citing inconsistent and infrequent communication and visits by ADEMAS. ADEMAS may want to consider assisting BREIPS through monthly or quarterly meetings (in person and/or virtual support). According to the SNEIPS director, the training on the DELTA method was found to be useful and applicable to their work. No explanation was given to Team EY as to why the two directors of BREIPS and two agents for EIPS were absent. When discussing with BREIPS concerns of what happens when an agent for EIPS is transferred and a new, untrained, one is assigned to the post (or if one was absent from the training) BREIPS said they would train the EIPS agent. Unfortunately, there is no provision made for training the BREIPS directors who were not present. In year two, a TWG was established to assist in the development of a framework to coordinate all interventions in health communication. The TWG benefited from an orientation on the DELTA method. During year three, the terms of reference, structure, mandate, and operational procedures of the group were finalized. Although established, the TWG is not fully functional and meetings are sparsely attended by members. Neither SNEIPS nor ADEMAS is well positioned in the MOH institutional hierarchy to leverage the support needed to actively engage members. Creation of a CBO Network According to the project description included in the year two annual report, ADEMAS was supposed to work with the ChildFund consortium to assist the health districts in creating CBO networks. However, there was miscommunication and ADEMAS expected that the ChildFund consortium would lead and organize the CBO networks within the districts and that ADEMAS would subsequently provide training instead of also creating the CBO networks. According to Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 148 the reports from the ChildFund consortium, the CBO networks were developed in all 14 regions, but ADEMAS is not necessarily taking advantage of these networks for training HCP activities. An interview with the ADEMAS COP indicated that goodwill exists; however, poor communication and coordination between partners was a constraint. Despite many efforts to identify the core issues around the weak communication, the evaluation team was not able to determine the root cause. According to annual reports, ADEMAS is working with individual CBOs and has trained CBOs engaged in communication campaigns in 40 districts. In addition, 498 outreach workers were trained on themes included in their contracts (please reference Section 9.4 below). A challenge experienced in the creation of a CBO network is ADEMAS’s inexperience working with newly formed CBOs. ADEMAS primarily contracted with existing CBOs which already had experience and adequate managerial capacity. ADEMAS was able to focus trainings in BCC methods to implement their health promotion activities. However, many of the CBOs identified by the district to form the networks are newer and, therefore, need capacity building in governance and financial management prior to training in BCC. ADEMAS is not specialized in this area and, therefore, lacks the capacity to strengthen the management and financial aspects needed for new CBOs to operate efficiently and effectively. National Communication Framework The process of finalizing the National Communication Framework took two years. In May 2012, ADEMAS supported the SNEIPS by organizing a workshop for the establishment of the Health Promotion National Strategic Plan (HPNSP). It was held in Thiès and attended by representatives of the MOH, Ministry of Education, WHO, UNICEF, USAID/Senegal, and ADEMAS. Unfortunately, the National Health Promotion Plan was recently finalized, but is still not validated by stakeholders. According to ADEMAS, they are unable to leverage sufficient authority to motivate the MOH to accelerate the process. ADEMAS continues to work on the process for validation and hopes that this will not be delayed further. PPPs A separate, stand-alone strategy for PPPs does not appear to exist through USAID/Senegal or the MOH. Therefore, ADEMAS’s activities in this area are unstructured and ad hoc. Currently two types of PPPs are currently being supported: 1) social marketing activity contracts with private companies to package and distribute the products and 2) longer-term relationships with companies to encourage their support for one-off, day-long campaigns. According to the annual reports, ADEMAS is in the process of gaining signatures for agreements with several organizations to promote behavior change health messages within their organizations, contribute to ADEMAS’s health campaigns, recruit people to sell the socially marketed products, and to identify wholesalers who would agree to sell the products. Please reference Table 33 below for further information. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 149 Table 33: PPPs Currently in Place or Planned Organizations Role International Bank of Commerce and Industry of Senegal (BICIS) An agreement is being negotiated for BICIS to promote ADEMAS’s products related to malaria. BICIS requires official approval by the National Malaria Control Program before signing. National Bank for Economic Development (BNDE), Tigo (mobile phone company), and La Ville de Dakar, World Funds Contact with each of these companies was made to support campaigns to promote socially marketed products, but there are currently no formal agreements or contracts signed. National Union of Businessmen and Industrialists of Senegal The goal through this partnership is to recruit local businesses to promote the use of LLINs, aquatabs, and condoms. A relationship was also developed with the Chamber of Commerce, Industry and Agriculture of Kaolack in March 2014 to promote those products. Currently, there are no contracts signed. The challenge for developing PPPs is in identifying companies that are willing to market the products. ADEMAS is trying to establish relationships with companies, which will lead to agreements, but the process is long and often unsuccessful. During Team EY’s discussions with ADEMAS staff, ADEMAS noted that they do not appear to have a broad concept of PPPs in the context of the larger USAID/Senegal Health Program and suggested that USAID/Senegal develop a strategy that would provide guidance to implementing agencies. According to ADEMAS, USAID/Senegal requested they explore PPPs with the MOH and it was discovered that SNEIPS already partnered with Colgate and Proctor and Gamble, and there was no role for ADEMAS. SNEIPS collaborated with Colgate and Proctor and Gamble to develop posters on washing hands (thus promoting Colgate’s soap) and infant diarrhea prevention (promoting the use of Proctor and Gamble’s disposable diapers). The District Health Officer of Popenguin in Thiès is interested in utilizing PPPs to improve the health of its population. This is an area where private enterprise (e.g., factories, gravel pits) is expanding with the expectation of increasing the numbers of workers with expanded knowledge, including male mobile groups, who are at a higher risk of HIV. These companies could be mobilized to provide HIV/AIDS prevention activities in the workplace. ADEMAS contributes to the health promotion campaign which takes place every year during the Pentecost in Popenguin and could potentially expand activities with private enterprises. M&E activities for communication campaigns including qualitative and quantitative research ADEMAS established an extensive research program for social marketing, based off of the PSI system. It is used to identify points of sale, determine marketing messages for promoting products, and to identify potential clients and targeted marketing messages to promote products and track sales. A five-year research plan was finalized in year one of the project,26 which includes studies to measure the basic indicators of the program, and those coming from the information gaps identified during the development of the different marketing plans for the HIV 26 Health Communication and Promotion Program Component Cooperative Agreement,” USAID/Senegal, March 1, 2012. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 150 and condoms, FP, and aquatabs. This activity was conducted in conjunction with the MOH Research Department and presented to SNEIPS upon completion. ADEMAS carries out many studies, the results of which they use for social marketing purposes to design their promotions for the socially marketed products. A number of studies were completed in years two and three of the project, including:  Investigation of a “bottom-up” approach to identify wholesalers in order to switch from a direct distribution mode to an indirect one for consumer products investigation will be through a recording of all points of sales in supervised areas.  A survey of FP service providers. This study complemented a prior study conducted on the profile of target groups, motivations, and barriers which affect the adoption of a modern method of contraception.  Pre-tests were carried out for the condom brands of Protec and Fogaru with the assistance of PSI. Three of the five qualitative studies for Framework for Qualitative Research in Social Marketing (FoQus) were completed and the reports were finalized and shared. They include the perception of women on FP, perception of youth on condom use, and the perception of men on condom use. Two additional reports are still being finalized which include the reproductive health service providers and distributors of health products.27 A series of Tracking Result Continuously (TRaC) quantitative studies were conducted in FP, nutrition, Water, Sanitation, and Hygiene (WASH), diarrheal diseases, and malaria.  Pre-tests for radio and television spots for the MILDA campaign and a review of HIV and FP marketing plans were completed, or were undertaken, to analyze data indicators to measure access and performance of the distribution network.  A review of ADEMAS’s M&E system for social marketing was conducted to improve management of the distribution system of socially marketed products. The review revealed that data collection tools and scoreboards needed to be revised to improve the quality and time spent on data collection. Revisions were made in the data collection tools and a set of indicators targeted for the new scoreboards. 9.3.2 Lessons Learned and Recommendations • ADEMAS’s health promotion is directly tied to levels of success in the promotion of USAID/Senegal’s package of health services; specifically, IR 1: Increased availability of an integrated package of quality health services. Focusing activities on smaller groups facilitates repeat messaging, which is shown to be more effective for behavior change. However, it may strengthen the program if ADEMAS were also empowered to provide assistance to SNEIPS and other organizations to respond to emergencies that arise (such as Ebola) or location￾specific issues which could easily be addressed through community radio or CBOs. Team EY recognizes that awards are not written to allow flexible spending for unforeseen issues or emergencies and that activities are approved in AWPs and require AOR approval if changes are made. However, if possible, USAID/Senegal may want to consider providing a 27 ”Programme Sante USAID 2011-2016, Composante Communication et Promotion de la Sante, Plan d’Actions 2013-2014,” ADEMAS. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 151 mechanism for flexible spending in ADEMAS’s budget to help respond to other health issues and emergencies as they emerge. • ADEMAS’s capacity may be strengthened through increased participation of ADEMAS’s research team in technical trainings, conferences, and exchange visits so that the research team remains up-to-date with best practices and innovations in research methodologies and M&E, including evolution of the data management system. The data from these research studies could be more effectively used for developing revised materials for CBO group discussions. ADEMAS may want to consider developing a process to better utilize the research studies for the group discussions. • To receive cooperation from the MOH in the execution and validation of the National Health Promotion Plan, ADEMAS may want to identify or recruit champions within the institutions who are supportive of the interventions and who can use their influence to encourage others to collaborate. • USAID/Senegal may want to request that more emphasis be placed on leadership and organizational development. Team EY recommends that USAID/Senegal conduct further analysis to explore the best solution among those suggested for consideration with ADEMAS. USAID/Senegal may want to add a senior staff member with leadership, management, and governance expertise to the staff or improve collaboration with Abt on this component. Another consideration suggested by the ADEMAS COP was to remove this intervention from their component altogether and allow them to “do what they are best in,” social marketing and providing technical assistance in health communication. Team EY believes that USAID/Senegal should consider focusing ADEMAS’s efforts on social marketing, national communication campaigns, and providing technical assistance in health communication and lastly, discuss their proposed plans to address other challenges. Team EY was unable to corroborate if there was duplication of activities specifically with CBOs working directly with the ChildFund consortium. Concerns raised about ADEMAS's ability to fulfill their capacity development role of its CBOs could not be explored further as the scope and duration of this evaluation did not permit an in-depth analysis. Team EY is advising that a more in-depth analysis be undertaken before decisions are made on ADEMAS’s future course of action in these different areas. 9.4 Sub-component B: Supporting implementation of quality BCC interventions leading to the adoption of healthy behaviors and the increased use of health services 9.4.1 Findings and Analysis Within this sub-component, ADEMAS’s BCC activities centered around four areas: 1) Development of integrated, evidence-based communication campaigns to promote health behaviors, 2) Increased use of health services including FP and malaria prevention, 3) Prevention and treatment of water-borne disease, and 4) Infant and early childhood malnutrition, hygiene, and sanitation. ADEMAS focused its efforts on two critical campaigns: FP contraception use and long-lasting ITNs (MILDA).  FP Campaign: During year two and year three of the project, ADEMAS developed various communication campaigns in the area of FP. The national communication campaign on FP with the logo MoytouNef was launched by the MOH in 2013. The campaign consists of a Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 152 package of activities designed to reach the entire country, through radio broadcasts, television spots, billboards, advertisement in newspapers, and field activities. ADEMAS signed contracts with CBOs for FP promotion added a dimension which strengthened the effect of this campaign.  Campaign for the use of ITNs: The campaign to promote the use of ITNs was launched in August 2013. It included television and radio spots in targeted areas of high malaria prevalence such as Dakar, Thiès, Kaolack, Fatick, Kédougou, Sédhiou, Kolda, Velingara, and Kaffrine. For this campaign, ADEMAS is collaborating with the National Malaria Control Program. ADEMAS initiated a mass media and community-based communication campaign for FP. The number of radio spots on FP for the three quarters of 2014 totaled 63,336, the equivalent of 4.55 spots a day. In addition, 2,488 small group discussions were held. The CBOs are contracted to conduct anywhere from six to 84 small group discussions over a nine month period. Focus group discussions and in-depth interviews with broadcasters of radio stations supported by ADEMAS on the value of radio as a medium of health communication suggest that it was effective. Interviews conducted at three community radio stations in Kaffrine, Kaolack, and Louga indicate that community radio can be a strong influential force for behavior change and is an effective means of reaching religious leaders and men for FP advocacy. Call-ins during broadcasts in Kaffrine and Louga included conversations with men who stated an understanding that contraceptives are not harmful to their wives nor do they prevent future pregnancies. The participation of medical staff from the health district in radio broadcasts is a good collaboration between the districts and the intervention. However, radio is less effective in areas such as the district of Richard Toll in Saint-Louis where the radius covered by radio broadcasting is limited. During the fourth quarter of year three, community-based interventions included the areas of FP, malaria, diarrhea, and breastfeeding. However, it is important to note that all health areas were not addressed in all regions. Table 34 below demonstrates the estimated number of people reached. Table 34: Number of Persons Reached for each Theme28 Theme Target Actual Percentage FP 28,725 30,222 105% Malaria 6,375 7,370 116% Diarrhea 5,550 6,125 110% Early Breastfeeding 275 287 104% ADEMAS focused significant time on developing the capacity of existing CBOs to deliver BCC messages. It signed individual contracts with 97 CBOs and 58 community and local radio stations for the FP campaign (see Table 35 below). To facilitate these relationships, ADEMAS conducted a study on the partnership identification process and the terms of partnership. A consultation workshop took place to finalize the package of activities and information collection tools used by CBOs. Workshops included: 28 "Programme Sante USAID Sénégal 2011-2016, Composante Communication et Promotion de la Sante, Rapport trimestriel d’activités Avril- Juin 2014," ADEMAS, July 31, 2014. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 153  The identification of basic contracting elements with CBOs.  The contracting process with community and local radio.  The profile of the IEC/BCC agents to be hired/contracted by ADEMAS to support monitoring and supervision of CBOs. Table 35: Contracts signed with Community Organizations29 Region CBOs Contracted Community and Local Radio Stations Contracted Dakar 0 3 Diourbel 12 2 Fatick 0 2 Kaffrine 12 2 Kaolack 10 5 Kédougou 7 3 Kolda 7 5 Louga 0 5 Sédhiou 5 5 Saint-Louis 15 3 Tambacounda 0 10 Thiès 25 7 Ziguinchor 5 7 ADEMAS is not currently working with CBOs in Dakar, Louga, Tambacounda, and Fatick for the delivery of messages through the HCP project. ADEMAS is however, working with community radio stations in each of Senegal’s 14 regions, including Dakar, as well as national media channels also based in Dakar for the delivery of messages via television and radio spots. ADEMAS launched community-based outreach activities in the three regions outside of Dakar where the RBs are located. The reason noted by ADEMAS in data point/site interviews was that working in regions where the Health Program had a greater permanent presence and close relationship with the Regional Medical Office and District Medical offices allowed for closer monitoring and supervision over the course of the roll-out of this new activity for ADEMAS. The second wave of regions (e.g., Sédhiou, Ziguinchor, Saint-Louis) was based on a mix of health indicators (need), population density, and discussions with ministry of health counterparts concerning priority geographic areas. The use of clearly defined parameters in contracts with CBOs supported the successful implementation of CBO activities. Contracts included a defined set of activities, the exact number of community contacts, the type of BCC intervention, and number of people expected to be reached. This made it possible for the CBO to track what they are doing with a simple reporting tool that was then sent to ADEMAS every quarter. Reports were accompanied by photographs as proof that the event took place. Focus group discussions with the outreach workers indicated that they appreciate the format of the contract as it helped them plan and achieve their work. They also expressed enthusiasm about the content of the work, but were concerned about delays of payment once they submitted their reports. 29 Information provided by ADEMAS during interviews. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 154 A number those activities are occurring with CBOs; however, Team EY could not validate the quality of activities. Interviews with CHWs revealed that ADEMAS does not furnish CBOs with visual aids or demonstration items to utilize in discussions. In ADEMAS’s annual report, a concern was expressed about the quality of the discussion groups and noted that sometimes the number of attendees is greater than would be effective for a discussion. This may be a positive reflection as it indicates the interest of the population in these discussions. Additionally, there are a number of challenges related to the monitoring of CBO activities.30 While some CBOs have their own monitoring system to validate that their volunteers are carrying out the activities, they lack the expertise to judge the quality or accuracy of the information they are providing or the communication techniques they use. Although CBOs submit reports to ADEMAS with photographs verifying their activities, these photos have limitations. For example, discussions with one CBO indicated that they sometimes forget to take the picture and subsequently organize another meeting for the sole purpose of photographing the event after the fact. It was clear to the evaluation team that ADEMAS has insufficient supervisory staff in some of the regions. There is only one supervisor per region to monitor all the activities. In some regions such as Saint-Louis, the distances between the locations where CBO activities occur are very far (>5km) and it is difficult to cover the area to observe the activities in monitoring visits. In addition, it was reported in interviews with CHWs that the support material, such as visual aids, provided to the ADEMAS field supervisors is sometimes insufficient for them to do their work properly. 9.4.2 Lessons Learned and Recommendations • Behavior change requires frequent contact through various methods of health campaigns. Moreover, coordination between campaigns and recurring campaigns has proven to be more effective than one-off or isolated group discussions. This was practiced in the FP campaign that presented continual and various communication activities. USAID/Senegal may want to request that ADEMAS works with the MOH to develop this type of multifaceted approach for each area of health promotion and help them to build these into regular activities of the National Strategy for Health Promotion. • A recurring issue is the quality of CH promotion activities, especially the CBO small group discussion. To support high quality health communication interventions on the community level, USAID/Senegal may want to consider conducting a review of the methodology for health discussions and their supervision in order to develop or adapt effective participatory tools for promoting behavior change in small groups. In addition, results from studies of behavior related to health from the social marketing sector can be used to develop materials for use within the discussion groups. Subsequently, USAID/Senegal should consider training CBO outreach workers in these new methods for implementation. • To establish an effective system for M&E activities in CBOs, USAID/Senegal may want to consider requesting that ADEMAS develop a strategy for monitoring the quality of small group discussions. ADEMAS may want this strategy to include more frequent observation, clearly defined roles and responsibilities, and establishment of set indicators and expected 30 Lillie, Tiffany. “USAID/Senegal: Social Marketing Program Performance Evaluation,” Global Technical Assistance Bridge Project, June 2012. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 155 milestones. In addition, observation of CBO activities should include on-the-spot coaching to support the continued skill development of behavior change communicators. ADEMAS may want to consider hiring additional staff to support monitoring activities. In larger and more remote regions such as Saint-Louis, it is recommended that mobile communication units be considered for use. 9.5 Sub-component C: Strengthening the capacity of key actors to advocate for political and social engagement for health programs 9.5.1 Findings and Analysis ADEMAS focused on two aspects of advocacy under this sub-component: 1) developing an overall operational plan for advocacy aimed at political and social leaders, and 2) developing an advocacy plan to change individuals’ attitudes and behaviors related to health. Core activities under this sub-component included:  Advocacy training for MOH staff.  Development of national advocacy strategies in priority health areas, including FP and child survival.  Support of advocacy activities targeting opinion leaders including journalists, religious leaders, and elected officials as appropriate to achieve advocacy strategy goals. ADEMAS has not been able to achieve its objectives at the central level in strengthening the advocacy capacity of political and social leaders to advance health communication programs. ADEMAS was most successful in media training events on health communications issues where they held three of the four planned events. It was planned that by project year three, ADEMAS would hold eight annual coordination and advocacy meetings with identified stakeholders, such as health officers, religious leaders, public-private providers, and national and local government officials. However, thus far, they have only held two such meetings, achieving only 25% of their target. Moreover, ADEMAS has not yet identified local “champions,” who would act as peer educators or spokespersons in support of promotion of a particular health behavior. Their plan was to identify five champions by this point in the project. The lack of influence at the MOH level is the main challenge faced by ADEMAS in executing its activities. The lack of authority or influence by CEFOREP or SNEIPS over the divisions of the MOH adds an additional level of bureaucracy and is a key reason cited for delays. Progress under the National Advocacy Strategy for Family Planning (PANPF) was also delayed. Although the strategy was developed, it has yet to be approved by the MOH and adopted. According to the SNEIPS director, the national-level advocacy plan was to be followed by a decentralized roll-out. CEFOREP was responsible for providing technical assistance for advocacy training at the central, regional, and district levels. Although this was initiated, progress is slow. Once the strategy is approved, ADEMAS plans to accelerate advocacy activities. The development of an action plan for this strategy is in process and it is expected to be completed in December 2015. The interview with the ADEMAS COP reported that the delay was due to CEFOREP’s failure to fulfill its contractual agreement. ADEMAS terminated their contract with CEFOREP in October 2014. It will then contract with a senior consultant to strengthen the advocacy team and accelerate implementation of the National Advocacy Action Plan. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 156 9.5.2 Lessons Learned and Recommendations • ADEMAS relied on CEFOREP in the area of institutional advocacy building and did not possess this capability internally, as stated in the ADEMAS cooperative agreement. Without strong expertise provided systematically by ADEMAS, the achievements of this sub￾component may continue to lag behind expectations. This is an area of capacity building that PSI tried to address but it was not effective in increasing ADEMAS’s capacity sufficiently. Team EY realizes that it may not be possible at this stage of the project to shift the advocacy sub-component to another partner. It may be a more realistic solution to increase USAID/Senegal technical assistance efforts following FY 14 AWP to improve results and indicators around this activity. • The ADEMAS COP has concerns that ADEMAS is not well placed to provide technical assistance in advocacy work at the central level and suggests reconsidering this role for ADEMAS. USAID/Senegal and ADEMAS may want to jointly review ADEMAS’s advocacy role and consider shifting it to another partner with more multi-faceted expertise in these areas. This would allow ADEMAS to continue to focus on its core strength, which is implementing advocacy interventions for behavior change at the community level. Other potential solutions could include hiring a senior-level specialist for the ADEMAS staff or contracting a senior consultant to work at strengthening support and commitment with senior executives within the MOH. 9.6 Sub-component D: Social marketing of key health products resulting in their increased sale and use 9.6.1 Findings and Analysis The USAID/Senegal Heath Program donates products for social marketing and ADEMAS is responsible for the packaging, branding, labeling, and distribution of the products. The products marketed correspond to those required for the integrated package of services promoted by the USAID/Senegal Health Program. ADEMAS is making progress in achieving goals in terms of social marketing indicators for health products. ITNs were socially marketed by ADEMAS in year two. In addition, another proposed marketing effort is co-packaging ORS therapy with zinc as a way to facilitate the correct treatment of diarrhea for mothers. The social marketing campaigns as a whole were successful in generating funding. In year two, the project generated $712,126 USD of funding, 125% of the set target ($577,880 USD). This money was returned to the project and used for other activities. Table 36 below summarizes achievements in sales distribution of social marketing products. Core activities under this sub-component include:  Social marketing for specific brands of health products, including condoms, ORS, ITNs, and water purifiers.  Exploration of opportunities to introduce new products and services using social marketing approaches to increase demand for and access to life-saving products and services for populations with an unmet need. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 157 Table 36: Sales and Distribution of Social Marketing Products31 Products FY 2014 Sales Goal FY 2014 CYPs Goal Accumulated FY 2014 sales Accumulated FY 2014 CYPs Percent realization of sales target Percent realization of CYPs target Condoms 8,155,695 67,964 5,557,698 46,314 68% 68% Sécuril 428,252 28,550 379,728 25,315 89% 89% Depo Provera 27,426 6,857 15,227 3,807 56% 56% aquatabs* 1,985,962 NA 1,635,200 NA 82% NA ITNs 115,000 NA 122,106 NA 106% NA * Distribution of aquatabs includes free distribution of 1,451,000 tablets due to an approaching expiration date. ADEMAS stated it has concerns about the level of sales of their socially marketed products. For legal reasons, ADEMAS was only able to directly distribute the socially marketed products to select pharmacies and non-pharmaceutical outlets where ADEMAS has no influence over the way products are displayed. However, products go through wholesalers who have the direct relationship with the pharmacies. In many pharmacies, products are not visible, especially condoms. A discussion with pharmacists revealed that some pharmacists may choose not to display them because they are less expensive than others they sell. When a customer comes in and asks for condoms, they will just offer the more expensive brand and the customer is unaware that there is a less expensive option. Additionally, the level of penetration of products outside pharmaceutical circuits and certain products including aquatabs were almost absent from the point of sale locations. Interviews with ADEMAS also highlighted the lack of choices in pharmaceutical distributors and packagers in Dakar resulting in insufficient storage capabilities. ADEMAS has a contract with Valdafrique who packages, stores, and distributes the socially marketed products to wholesale distributers. Until recently, Valdafrique was the only pharmaceutical distributor and packager in Dakar, earning between 50% and 85% of revenues from the sale of socially marketed products. Additionally, ADEMAS noted the storage conditions provided by Valdafrique did not meet minimum storage standards required by ADEMAS. To address these challenges, ADEMAS was recently able to negotiate a contract with a new distributor, Laboratoire DIDY, who began operations in Dakar. It is expected that this distributor will provide lower costs and have improved warehouse conditions, where ADEMAS will have control over the storage. The relationship with Laboratoire DIDY is expected to help ADEMAS to provide direct distribution to pharmacies, allowing for greater influence in the promotion of their products within the pharmacies. ADEMAS is also conducting studies to identify more points of sale to increase the distribution of products. 31 "Programme Sante USAID Sénégal 2011-2016, Composante Communication et Promotion de la Sante, Rapport trimestriel d’activités Avril- Juin 2014," ADEMAS, July 31, 2014. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 158 9.6.2 Lessons Learned and Recommendations 9.7 Sub-component E: Technical capacity building and organizational development of the recipient 9.7.1 Findings and Analysis During the first two years of project implementation, PSI assisted ADEMAS with capacity building in three core areas: 1) organizational functionality, 2) evidence-based decision-making, and 3) program management and leadership. This technical assistance was to support ADEMAS’s long-term sustainability as a leading Senegalese social marketing organization. PSI’s institutional development support aimed to provide ADEMAS with the skills and tools necessary for project implementation. As part of their support, PSI helped ADEMAS address challenges associated with a rapidly expanding technical and operational knowledge base, management structures, and systems and tools to absorb an increase in annual budget over $4M USD per year. Through PSI, ADEMAS was integrated into the global network of social marketing organizations.32 Table 37 below summarizes technical assistance provided to ADEMAS by PSI. Table 37: ADEMAS Capacity Building Accomplishments Technical capacity building Corporate development  Development of five marketing plans through a participatory approach and based on evidence. This included the development of a marketing plan for the introduction of a new contraceptive technology - Sayana Press - in partnership with the DSRSE and technical partners. The Sayana Press has not yet been socially marketed by the project. It is still pending GOS approval.  “Train-the-trainer” sessions on the DELTA method for marketing campaigns, planning, and communication for behavioral change.  Development of a comprehensive system for monitoring and tracking based on a GIS for data management in sales, marketing, and strategic decision-making.  Development and implementation of an annual research plan that meets the programmatic needs identified during development sessions for marketing and communication strategies of BCC.  Creation of an institutional diagnosis of SNEIPS through a participatory approach based on self-assessment principle.  Restructuring of the institution into a corporate structure based on functional departments.  Implementation of an Executive Management Team to decentralize decision-making at management level.  Implementation of a framework for the development and monitoring of annual performance objectives at the corporate level.  Development and implementation of a plan for staff retention and motivation.  Revision of ADEMAS’s strategic plan for the 2014 -2019 period.  Revision of ADEMAS’s status from Association to public utility NGO.  Restructuring of the partnership with Valdafrique to expand the market for its social marketing products.  Integration of technical committees into ADEMAS to better coordinate its interventions with technical partners involved (including BCC TWG for malaria).  The launch of a new social marketing product, MILDA. 32 “Health Communication and Promotion Program Component Cooperative Agreement,” USAID/Senegal, March 1, 2012. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 159 Technical capacity building Corporate development  Development of a conceptual framework (e.g., business plan) for private sector involvement in campaigns and health promotion.  Facilitation of exchange visits to Mali and Cameroon to build technical capacity in marketing and distribution.  The introduction of two concepts for funding intervention to address under nutrition (ORS￾zinc and Vitamin A). Operational support to ADEMAS emphasized increasing efficiency and decentralizing leadership and decision-making authority for a more complex organization, while validating that appropriate controls are in place for adherence to minimum quality standards. This was achieved through development and consolidation of institutional management and operations systems, including the development of a robust and capable Executive Team and Board of Directors responsible for validating adoption of best practice. Key activities completed in this area included:  Transformation of ADEMAS’s institutional management framework from a project structure to an organizational structure. This included the change in legal status from “Association” to “NGO” granted by the GOS and implementation of a two-year technical and institutional capacity building plan.  Adoption of a five-year strategic plan (2014 – 2019) incorporating institutional objectives in key capacity building areas already identified through the Private-Sector Partnership-One Assessment of 2009 and the ADEMAS organizational assessment carried out by PSI at the start of the project.  Donation of two hectares of land by the GOS for the installation of a permanent office and warehouse space.  Development of a resource mobilization plan to help ADEMAS seek opportunities to leverage other funding opportunities, which may also serve as cost-share for this project. To date, ADEMAS submitted 10 proposals for complementary funding in existing and new health areas including nutrition, WASH, malaria prevention, HSS, and the delivery of quality health services. PSI’s technical assistance placed an emphasis on the rapid development of ADEMAS’s sales, marketing, communications, and research teams. Focused technical assistance in these key technical areas provided the ADEMAS team with methodologies and tools to improve the quality of social marketing and BCC campaigns using an evidence-based approach. This included initial training and ongoing coaching in the use of the DELTA evidence-based methodology for marketing and communications planning, qualitative and quantitative research methodologies oriented toward understanding determinants of behavior and market development, and putting into place an information management system to allow for routine data collection and analysis using GIS mapping coordinates. Technical assistance to build program management and leadership skills was provided through a mix of technical and leadership trainings, exchange visits, and day-to-day mentoring. By the end of year three, the process of transferring the management of the COP to the Deputy COP was Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 160 completed and he assumed the role in October 2014. The former COP will continue to provide technical assistance for the 2015 fiscal year. Additionally, technical assistance was provided to build ADEMAS’s knowledge base and leadership in new health areas including malaria, WASH, and nutrition. PSI provided technical support and guidance to ADEMAS to carry out situational analysis, formative research, concept development, and activity monitoring for these emerging priorities as well as the addition of new products to the social marketing portfolio. Key capacity building achievements by technical area include:  Development, implementation, and annual revision of evidence-based marketing plans for all socially marketed products.  Implementation of a framework for analyzing and managing sales and distribution.  Launching of social marketing products described in sub-components.  Development, implementation, and evaluation of a national FP communications campaign.  Use of an evidence-based methodology to develop marketing and communications strategies with training-of-trainers. The evaluation team noted that ADEMAS’s capacity was strengthened in areas linked directly to their core mandate, namely social marketing and health promotion through targeted campaigns. The area of capacity building in leadership poses the greatest challenge to ADEMAS. Discussions with the ADEMAS COP revealed that ADEMAS is still aiming to position itself as a leader in relation to the MOH and has limited visibility in many of the regions of the country, even in those places where it is working with CBOs and community radio. Moving forward, focus will be placed on consistent use of newly acquired technical and operational skills and tools, implementation of the ADEMAS five-year strategic plan, mobilization of resources through new business development, enhanced external communication highlighting ADEMAS’s achievements, and addressing priority areas for improvement highlighted in the organizational capacity assessment. 9.7.2 Lessons Learned and Recommendations • The capacity building of ADEMAS through shadowing or direct coaching during this project appears effective in bringing about positive institutional changes. ADEMAS is making progress in their internal capacity building. However, as skills are being developed, ADEMAS is experiencing difficulties implementing all of its required tasks under the contract. As such, USAID/Senegal may want to continue providing ADEMAS with support in reinforcing technical and organizational management capacity to reflect best practices and high quality standards with an emphasis on innovation, leadership, research, and M&E. The Technical Organization Capacity Assessment tool, developed as part of the USG’s New Partners Initiative, is a tool that could be applied with ADEMAS staff to address its technical strengths, areas for improvement, and execute action plans for the remainder of the project. Team EY was informed that an ADEMAS OCA was already completed with the USAID/Senegal Financial Management Officer. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 161 9.8 Response to Evaluation Questions The table below provides information to respond to the component-specific evaluation questions as stated in the scope of work. It is not intended to be an exhaustive but rather aims to highlight notable successes; constraints and challenges that have been experienced during implementation by ADEMAS; and key interventions that may be added, continued, or removed within Component 5. Team EY and USAID/Senegal agreed that this analysis would be conducted by component rather than by individual sub-components. Table 38: Component Table Component Specific Question Analysis To what extent have the components achieved their objectives? ADEMAS made progress towards achieving their objectives as outlined in Section 9.1. The USAID standard indicators outlined in Table 31 were derived from ADEMAS annual reports. These 34 HCP indicators consist of the full set reported and align with USAID/Senegal’s IR 2: Improved health seeking and healthy behaviors. They are expected to contribute to the achievement of IR 1: Increased availability of an integrated package of quality health services and IR 3: Improved performance of the health system. According to the indicator data, ADEMAS is meeting some of its targets (i.e. in areas of social marketing and health campaigns) for those indicators where data was reported; however, multiple indicators were not met. The causes of varying performance and gaps in data should be addressed and remediated by ADEMAS as feasible. To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component?  Implementation of BCC interventions through local sub-partners: ADEMAS focused on working with existing CBOs to deliver BCC messages tailored to the context of client needs, which was successfully implemented overall. For example, ADEMAS signed individual contracts with 97 CBOs and 58 community and local radio stations for the FP campaign. Interviews showed, specifically, the deliberate effort on the part of ADEMAS to implement BCC programs in regions where the Health Program had a greater presence and close relationship with the Regional Medical Office and District Medical offices. This allowed for closer monitoring and supervision over the course of the roll-out of BCC activities. Focus group discussions with outreach workers, implementing the health campaigns, indicated that they appreciate the format of ADEMAS’s contract and remained enthusiastic about their BCC work with communities. Two factors noted as contributing to achievements were the well-chosen intervention locations and the clear components and metrics outlined in the CBO contracts.  Social Marketing of Family Planning Products: Research was conducted in social marketing to identify points of sale, determine marketing messages for promoting products, identify potential clients, and tracking sales for FP products by ADEMAS. Factors identified that affected the current success in the social marketing of FP products (noted in year two annual report data and corroborated through interviews) were that the ADEMAS trainings of district and regional health teams provided the MOH with methodologies for targeting social marketing programs appropriately. Trainings on the DELTA approach were cited by SNEIPS as useful and applicable to their work.  Internal technical assistance to strengthen capacity in marketing: ADEMAS received technical support from PSI to build their internal capacity. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 162 PSI’s technical assistance placed an emphasis on the rapid development of ADEMAS’s sales, marketing, communications, and research teams. Focused technical assistance provided the ADEMAS team with methodologies and tools to improve the quality of social marketing and BCC campaigns using an evidence-based approach. Specifically, this included initial training and ongoing coaching in the use of the DELTA methodology for marketing and communications planning, qualitative and quantitative research oriented toward understanding determinants of behavior and market development, and resources in the creation of an information management system to allow for routine data collection and analysis using GIS mapping coordinates. The evaluation team noted that ADEMAS’s capacity was strengthened in areas linked directly to their core mandate, namely social marketing and health promotion through targeted campaigns. Interviews with ADEMAS staff recognized the maturity and growth since the organization moved from an association to a registered NGO; however, they also noted the need for ADEMAS to stay technically current and build its reputation, brand and visibility in many of the regions. What are the constraints and challenges that have hindered successful implementation of each sub-component, and how has the IP dealt with those challenges?  Institutional capacity building: ADEMAS struggled to establish relationships and leverage influence over central government agencies to improve institutional capacity building. It does not appear that ADEMAS technical staff is well positioned to leverage the support needed to move agendas forward and routinely engage members at the national level. Interviews with ADEMAS staff cited this as a challenge that affected the infrequency of meetings with the MOH, which they continue to try to address through follow-up. The ADEMAS COP cited a plan being developed to help address the challenges of the organization’s limited visibility and influence at the central level and a concerted effort to continue its political engagement.  Limited Staff Resources within ADEMAS: ADEMAS has a mandate to build CBO capacity to carry out effective BCC activities. Distinct from its HCP technical leadership roles, the challenges of management and oversight across its 97 local partners and implementation sites arose in annual reports. This challenge was corroborated by the evaluation team that ADEMAS has insufficient supervisory staff in some of the regions. There is only one supervisor per region to monitor all the activities. In some regions such as Saint-Louis, the distances between the locations where CBO activities occur are far (>5km) and it is difficult to cover the area to observe the activities in monitoring visits. While there is focused work on HCP, the level of effort required for 1) meaningful capacity-building for its CBO partners, 2) building relationships with local government and 3) a commitment to maintain multiple small group contacts with communities to reinforce health messages, represent competing priorities for ADEMAS staff to deliver. Interviews with ADEMAS staff show a commitment to improving the areas of deficiencies as well as an understanding of the internal capacity limitations that need attention.  Public Private Partnership Formation: As per sub-component B, supporting the implementation of quality BCC interventions leading to the adoption of healthy behaviors and the increased use of health services, ADEMAS was supposed to develop PPPs. The type of PPPs include 1) social marketing activity contracts with private companies to package and distribute the products and 2) longer-term relationships with companies to encourage their support for one-off, day-long campaigns. ADEMAS developed contacts Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 163 with the MOH and different for-profit groups to initiate conversation about potential areas of private sector collaboration; however, ADEMAS’s activities in this area are unstructured and appear ad hoc. Team EY noted that none of the PPPs explored, corroborated by staff interviews, have materialized. ADEMAS will continue to try to develop concrete partnerships in the upcoming year. Details of current opportunities in early discussion with the BNDE, Tigo, and La Ville de Dakar, World Funds, and the National Union of Businessmen and Industrialists of Senegal, are provided in Table 29.  Development of the National Advocacy Strategy for FP: Despite the completion of the draft National Advocacy Strategy for Family Planning, the adoption by the MOH and its decentralized roll-out has stalled. ADEMAS faced challenges such as delays in its review and signing by the MOH. The evaluation team noted the following actions by ADEMAS as they aim to move the FP Advocacy Strategy forward: o Follow-up activities with the MOH (i.e., meetings and virtual support to finalize content, the signing process and plans for roll-out) to reinvigorate interest and acceptance of the National Advocacy Strategy Plan. o Termination of the ADEMAS contract with CEFOREP in October 2014 due to their failure to deliver as the partner responsible for providing technical assistance for advocacy training at the central, regional, and district levels. o Plan to contract with a senior consultant to strengthen the advocacy team and accelerate implementation of the National Advocacy Action Plan. Are there interventions that should be added or removed?  Use of Technology for Targeted Populations: USAID/Senegal may want to request that ADEMAS add interventions that use innovative and integrated platforms like social media and mobile technology to reach target populations, especially youth and MARPs. As ADEMAS continues to strengthen social marketing of key products and the accompanying BCC campaigns to support the products, activities such as group text messages and follow-up community outreach tied to specific health events should be considered. Effective integration and use technology can broaden the reach of health communication interventions and reinforce contacts made with clients with customized, targeted messages.  Increased Collaboration across other USAID/Senegal supported Health Components: ADEMAS has demonstrated positive performance in BCC and social marketing activities. Additional collaboration with other health components can better leverage the community platform and CBO network to place different social marketing and BCC campaigns into priority health initiatives in a region or district. Team EY noted ADEMAS’s efforts to identify ways to better engage pharmacists in the offering of the integrated package of services where pharmacies and private sector partners led distribution and sales of socially marketed health commodities that are linked to the services supported by IntraHealth. Building on these experiences and utilizing joint work plans developed across components, ADEMAS should add interventions that integrate health communication programs with the CH, HSS, and HIS components where appropriate. A discussion about current activities and opportunities to increase collaboration, convened by USAID/Senegal or Abt in its’ coordination role, could fast track ADEMAS’ capacity to take on this effort. Are there changes  Technical Assistance to the Regional Bureaus for Health Education and Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 164 that could be made to improve performance? Information: ADEMAS should continue to strengthen the BREIPS by providing direct TA to the 14 regional level offices. With one BREIPS per region, there is a need for continuous support to Directors in their oversight of the 75 district level EIPS officers implementing health communication interventions. ADEMAS should help the BREIPS improve the quality of BCC campaigns through closer monitoring of health messaging, campaigns, and interventions carried out with NGOs, FBOs, and CBOs in their region. Given the challenges noted earlier with ADEMAS building institutional capacity at the central level, Team EY suggests that ADEMAS refocus its efforts at the regional level. In Matam, the BREIPS Director expressed a desire for support from ADEMAS staff citing inconsistent and infrequent communication and visits by ADEMAS. Additionally, a lack of IEC materials on malaria, FP, and HIV/AIDS, was raised in a BREIPS interview in Thiès and in multiple focus group interviews with CHWs. Team EY suggests that ADEMAS consider assisting BREIPS through monthly or quarterly meetings (in person and/or virtual support) and provide TA to help develop IEC materials and job aids for health workers at regional and district levels to implement specific BCC campaigns. USAID/Senegal, in its leadership role, should continue supporting ADEMAS to advocate for and support the BREIPS in quality improvement activities throughout the focus regions for this component. 9.9 Data Sources In addition to sources cited in Annex G: Bibliography, data collected in the field was used for analysis of this component, including: in-depth interview with the COP, RB coordinators, key MOH stakeholders, regional and district health officers, regional and district coordinating offices, community-based health insurance managers, directors of the BREIPS, service providers at health facilities; services providers at health huts, pharmacists, private firms; as well as focus group discussions with CHWs, CH committees, community-based health insurance beneficiaries, and associations of PLWHA. Data collection tools for field interviews are available in Annex I and Annex J. 10.0 CONCLUSION The evaluation assessed the effectiveness of the program design, coordination, and implementation with the objective of identifying lessons learned and proposed recommendations to help inform decision-making on the current program and to facilitate discussion for future strategic planning. In its assessment, Team EY confirmed that the five components funded by USAID/Senegal were aligned with the goals and objectives of the MOH and helped contribute to improving the health of the Senegalese population. However, a key finding was that components are operating in vertical silos, rather than supporting a horizontal, integrated approach, leading to missed opportunities for collaboration. Team EY does not propose expanding the number of components, but instead recommends focusing on improved collaboration and streamlining of processes for integration between components. Each of the five health components is meeting some of its overall objectives as stated in the cooperative agreements and contributing to achievements made in the three IRs – IR: 1- Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 165 increased use of an integrated package of quality health services, IR 2: Improved health seeking and healthy behaviors, and IR 3: Improved performance of the health system. However, within each component there are opportunities for improvement, especially with meeting planned indicators as can be seen in Figure 24 below. Figure 24: USAID/Senegal Indicator Status Figure 24 includes the tally of all indicators that were reported in each of the detailed component sections (i.e., Sections 5.0 – 9.0). As demonstrated, there was improvement from year one to year two with meeting or exceeding targets. A total of 23 additional targets were met from year one to year two. In addition, the amount of targets that did not have data to report reduced from 54 in year one to 20 in year two, which indicates IPs are improving their processes of tracking and reporting data for key indicators. However, as noted above, there are still several areas for improvement including tracking and providing the data for those indicators where data was not provided as well as meeting indicators. For those indicators where data is not available, USAID/Senegal may want to work with the IPs and reevaluate the feasibility for tracking and reporting those indicators. If it is determined the indicator is no longer applicable or cannot be tracked, they may want to find a replacement indicator. In addition, for those indicators that have not been met, USAID/Senegal should work with the IPs to determine the root causes for not meeting the indicators and determine if the targets should be adjusted for future years and/or develop remediation plans to achieve targets. In addition to the key indicators, Team EY identified some key findings impacting the program that are described throughout the Final Evaluation Report. These include:  Availability of data and accurate, timely reporting are core challenges for all the components. When data is collected, it is often used for reporting purposes but not to improve decision￾making. The strike by health workers (primarily members of the nurses’ union) was a contributor to constraints related to reporting data as it hindered consistent and reliable data collection. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 166  While Abt manages the RBs to help facilitate coordination, it was noted that the current process of coordination with the other IPs through the RB does not function as effectively as it could, both internally (among IPs) and externally (relation to regional and districts committees).  Positive feedback was received on PBF from health providers, who believed it was motivating and changing the way they delivered care. It was noted that PBF is contributing to positive behavioral change in the region, emphasizing an increase in service quality and ownership of health services.  The reporting system of work plans, milestones, and indicators established in the regions/districts is overly complex and burdensome to local partners. Dueling infrastructures established for PBF and DF create a strain on the system and focus the attention of regional/district management teams on data collection and reporting rather than implementing and supervising service delivery.  Weak supply chain systems throughout the country are a consistent constraint affecting many of the components’ ability to implement aspects of their program. Notable issues were related to the processes and systems of forecasting, procuring, storing, and distributing various health commodities. The establishment of a functional supply chain requires the standardization of processes and the development of staff with the appropriate skillsets.  The CH program is challenged as a result of its reliance on volunteers for the complete functionality of services and health promotion. Incentivizing community-based workers (or creating opportunities for employment) could be implemented to support consistent, dedicated, and more sustainable health services.  The functionality of the health huts and their outreach activities appear to be highly dependent on the consistent support of the ChildFund consortium. There is consensus among key stakeholders, including the MOH (regional and district health offices), service providers, and CH Management Committees that without the support, the quality and continued maintenance of services will decline or in some instances cease.  MSM and CSWs in Senegal are disproportionately affected by HIV/AIDS and deal with additional barriers (e.g., laws to protect against police abuse, unfair treatment, stigma and discrimination) to seeking and utilizing health services. Additional interventions may be needed to reach this population such as mobile services or key BCC campaigns.  The core competency of ADEMAS is the social marketing of health products, which was the most successful part of this component according to their indicator results and interviews. The social marketing of products contributed to the integrated package of services promoted in the USAID/Senegal Health Program. Overall, the five components funded by the USAID/Senegal Health Program are meeting some of their set goals and objectives; however, Team EY believes that the program could enhance its efficiency and effectiveness if some of the recommendations listed throughout the Final Evaluation Report are considered and implemented into USAID/Senegal’s future strategy. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 167 Annex A: Map – Implementing Partner Presence in Senegal Regions Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 168 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 169 Annex B: Key Stakeholder Interview List The information below comprehensively lists the interviews conducted with key stakeholders. # Interview Date of Interview Region Interview Type Type EY Team Lead 1 USAID/Senegal - AOR 9/10/2014 Dakar In-Depth Interview AOR Dr. Ruth Kornfield 2 USAID/Senegal - AOR 9/10/2014 Dakar In-Depth Interview AOR Dr. Ruth Kornfield 3 USAID/Senegal - AOR 9/11/2014 Dakar In-Depth Interview AOR Dr. Ruth Kornfield 4 USAID/Senegal - AOR 9/11/2014 Dakar In-Depth Interview AOR Dr. Ruth Kornfield 5 USAID/Senegal - AOR 9/12/2014 Dakar In-Depth Interview AOR Dr. Ruth Kornfield 6 Abt – COP and Senior Team 9/12/2014 Dakar Focus Group Discussion IP Dr. Ruth Kornfield 7 The ChildFund consortium - COP and Senior Team 9/12/2014 Dakar Focus Group Discussion IP Dr. Ruth Kornfield 8 IntraHealth -COP and Senior Team 9/16/2014 Dakar Focus Group Discussion IP Dr. Ruth Kornfield 9 ADEMAS - COP and Deputy 9/16/2014 Dakar Focus Group Discussion IP Dr. Ruth Kornfield 10 FHI 360 - COP 9/17/2014 Dakar Focus Group Discussion IP Dr. Ruth Kornfield 11 MOH/DSRSE 9/25/2014 Dakar In-Depth Interview MOH Dr. Ismaila Thioye 12 MOH/Center for Youth Promotion (CPJ) 9/25/2014 Dakar In-Depth Interview MOH Dr. Ismaila Thioye 13 ADEMAS - COP 9/25/2014 Dakar In-Depth Interview IP Dr. Ismaila Thioye 14 MOH/DLSI 9/26/2014 Dakar In-Depth Interview MOH Dr. Georges Tiendrebeogo 15 MOH/PNA 9/29/2014 Dakar In-Depth Interview MOH Dr. Ruth Kornfield 16 FHI 360 - AIDS Service Partner 9/30/2014 Dakar In-Depth Interview IP Dr. Georges Tiendrebeogo 17 Abt - COP and Senior Team 10/2/2014 Dakar In-Depth Interview IP Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 18 MOH/SG 10/2/2014 Dakar In-Depth Interview MOH Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 19 Abt- FM 10/15/2014 Dakar In-Depth Interview Finance Ms. Christina Tippmann 20 The ChildFund consortium - FM 10/20/2014 Dakar In-Depth Interview Finance Ms. Christina Tippmann Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 170 # Interview Date of Interview Region Interview Type Type EY Team Lead 21 FHI 360- FM 10/21/2014 Dakar In-Depth Interview Finance Ms. Christina Tippmann 22 IntraHealth- FM 10/27/2014 Dakar In-Depth Interview Finance Ms. Christina Tippmann 23 ADEMAS- FM 10/28/2014 Dakar In-Depth Interview Finance Ms. Christina Tippmann 24 Abt- FM 11/20/2014 Dakar In-Depth Interview IP Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 25 Abt- FM 11/20/2014 Dakar In-Depth Interview IP Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 26 Abt- FM 11/20/2014 Dakar In-Depth Interview IP Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 27 Abt- FM 11/20/2014 Dakar In-Depth Interview IP Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 28 MOH-RBF 11/20/2014 Dakar In-Depth Interview MOH Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 29 UNFPA-Senegal Off 11/21/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 30 UNFPA-Senegal Off 11/21/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 31 UNFPA-Senegal Off 11/21/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 32 MOH- CACMU 11/21/2014 Dakar In-Depth Interview MOH Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 33 UNICEF- Health Specialist 11/25/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 171 # Interview Date of Interview Region Interview Type Type EY Team Lead Tiendrebeogo 34 UNICEF- Specialist in Communication for Development 11/25/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo 35 UNICEF- Nutrition Specialist 11/25/2014 Dakar In-Depth Interview Donor Dr. Ruth Kornfield, Dr. Ismaila Thioye, Dr. Georges Tiendrebeogo Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 172 Annex C: Data Point/Site Interview List The information below comprehensively lists the data point/site interviews conducted in the field. Team A Interview Date of Interview Region Interview Type EY Team Lead USAID/Senegal -Regional Office Coordinator / Thiès 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield The ChildFund consortium -Advisor for the RB 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Regional Chief MD 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Coordinator of USAID/Senegal RB of Thiès 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Officer of BREIPS Thiès 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Advisor for HSS 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Regional Program Officer 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Advisor for Social Financing 10/7/2014 Thiès In Depth Interview Dr. Ruth Kornfield Community Relay and BG 10/8/2014 Thiès Focus Group Discussion Dr. Ruth Kornfield District Primary Health Supervisor 10/8/2014 Thiès In Depth Interview Dr. Ruth Kornfield Chief Nurse of Sindia 10/8/2014 Thiès In Depth Interview Dr. Ruth Kornfield Matron and CHW 10/8/2014 Thiès In Depth Interview Dr. Ruth Kornfield Manager of the CH Insurance of Popenguine Ndayane 10/8/2014 Thiès In Depth Interview Dr. Ruth Kornfield Beneficiaries of the CH Insurance of Ndayane and Popenguine 10/8/2014 Thiès Focus Group Discussion Dr. Ruth Kornfield Advisor in SI 10/8/2014 Thiès In Depth Interview Dr. Ruth Kornfield Pharmacist 10/09/2014 Louga In Depth Interview Dr. Ruth Kornfield Focal Point IntraHealth in Louga 10/09/2014 Louga In Depth Interview Dr. Ruth Kornfield Focal Point IntraHealth in Louga 10/09/2014 Louga In Depth Interview Dr. Ruth Kornfield Officer of BREIPS in Louga representing the Regional MD 10/09/2014 Louga In Depth Interview Dr. Ruth Kornfield Supervisor of SSP 10/10/2014 Louga In Depth Interview Dr. Ruth Kornfield Supervisor of Pan International 10/10/2014 Louga In Depth Interview Dr. Ruth Kornfield Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 173 Interview Date of Interview Region Interview Type EY Team Lead Program Director/ Director of community radio 10/10/2014 Louga In Depth Interview Dr. Ruth Kornfield Commercial Director 10/10/2014 Louga In Depth Interview Dr. Ruth Kornfield CHW and Matrone 10/10/2014 Louga Focus Group Discussion Dr. Ruth Kornfield Chief Nurse of Fass touré 10/10/2014 Louga In Depth Interview Dr. Ruth Kornfield Chief District MD 10/13/2014 Matam In Depth Interview Dr. Ruth Kornfield Responsible BREIPS in Matam 10/13/2014 Matam In Depth Interview Dr. Ruth Kornfield Program Manager 10/13/2014 Matam In Depth Interview Dr. Ruth Kornfield Coordinator of Matam Bureau and Bakel YaaJeende 10/13/2014 Matam In Depth Interview Dr. Ruth Kornfield Nutrition officer Bureau and Bakel YaaJeende 10/13/2014 Matam In Depth Interview Dr. Ruth Kornfield Chief Region MD 10/14/2014 Matam In Depth Interview Dr. Ruth Kornfield Chief Nurse 10/14/2014 Matam In Depth Interview Dr. Ruth Kornfield Community Relay and BG 10/14/2014 Matam Focus Group Discussion Dr. Ruth Kornfield Matron and CHW 10/14/2014 Matam In Depth Interview Dr. Ruth Kornfield Village chief representative members of the health committee box Sinthiou mogo 10/14/2014 Matam Focus Group Discussion Dr. Ruth Kornfield Chief Pharmacist of PRA 10/16/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Coordinator of Reproductive Health of Saint-Louis 10/16/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Planning officer of the Medical Regional in Saint- Louis 10/15/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Focal Point the ChildFund consortium of Saint-Louis 10/16/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield District Chief MD 10/17/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield ADEMAS Supervisor 10/17/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Matron CHW 10/17/2014 Saint-Louis Focus Group Discussion Dr. Ruth Kornfield Matron Nurse of Savoigne 10/17/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Committee president 10/17/2014 Saint-Louis In Depth Interview Dr. Ruth Kornfield Health Committee of Ndiol Maure 10/17/2014 Saint-Louis Focus Group Discussion Dr. Ruth Kornfield Regional Chief MD 10/20/2014 Dakar In Depth Interview Dr. Ruth Kornfield District Chief MD FM 10/20/2014 Dakar In Depth Interview Dr. Ruth Kornfield Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 174 Interview Date of Interview Region Interview Type EY Team Lead Reproductive Health Coordinator SSP Community Relay BG 10/21/2014 Dakar Focus Group Discussion Dr. Ruth Kornfield Chief Nurse 10/21/2014 Dakar In Depth Interview Dr. Ruth Kornfield Pharmacist 10/22/2014 Dakar In Depth Interview Dr. Ruth Kornfield Director of SNEIPS and team 10/27/2014 Dakar In Depth Interview Dr. Ruth Kornfield ADEMAS director 10/27/2014 Dakar In Depth Interview Dr. Ruth Kornfield Coordinator of CH Cellule 10/27/2014 Dakar In Depth Interview Dr. Ruth Kornfield National Pharmacy 10/29/2014 Dakar Focus Group Discussion Dr. Ruth Kornfield Team B Interview Date of Interview Region Interview Type EY Team Lead Health Hut of Keur Ndiouga 10/8/2014 Fatick Focus Group Discussion Dr. Ismaila Thioye Health Hut of Gossas 10/8/14 Fatick In Depth Interview Dr. Ismaila Thioye WV International 10/7/14 Fatick In Depth Interview Dr. Ismaila Thioye Health Committee of Patar Lia 10/8/2014 Fatick In Depth Interview Dr. Ismaila Thioye Beneficiaries of the CH Insurance of NDamal Gossas 10/8/2014 Fatick Focus Group Discussion Dr. Ismaila Thioye Bokk Yakaar Association of Gossas 10/8/2014 Fatick Focus Group Discussion Dr. Ismaila Thioye CH Insurance of Ndamal Gossas 10/8/2014 Fatick Focus Group Discussion Dr. Ismaila Thioye IntraHealth Fatick 10/7/2014 Fatick In Depth Interview Dr. Ismaila Thioye CH Insurance of Gossas 10/8/2014 Fatick In Depth Interview Dr. Ismaila Thioye Health Post 10/8/2014 Fatick In Depth Interview Dr. Ismaila Thioye PRA 10/7/2014 Fatick In Depth Interview Dr. Ismaila Thioye Regional Medical Office 10/7/2014 Fatick In Depth Interview Dr. Ismaila Thioye CBO 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye Health hut Ngokhothie 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye Health Center Bambey 10/10/2014 Diourbel In Depth Interview Dr. Ismaila Thioye IntraHealth 10/9/2014 Diourbel In Depth Interview Dr. Ismaila Thioye Health Committee of Bambey 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye Association Farlu Daan AIDS 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 175 Interview Date of Interview Region Interview Type EY Team Lead CH Insurance JAPPOO 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye MHO - Bëg Fallou -de Ngogom 10/10/2014 Diourbel Focus Group Discussion Dr. Ismaila Thioye District of Bambey 10/10/2014 Diourbel In Depth Interview Dr. Ismaila Thioye Pharmacy Djily Borom Bagdad 10/10/2014 Diourbel In Depth Interview Dr. Ismaila Thioye Health post NDangalma 10/10/2014 Diourbel In Depth Interview Dr. Ismaila Thioye PRA 10/9/2014 Diourbel In Depth Interview Dr. Ismaila Thioye Medical Region 10/10/2014 Diourbel In Depth Interview Dr. Ismaila Thioye AOR 10/13/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Health hut Gambie 10/14/2014 Tambacounda Focus Group Discussion Dr. Ismaila Thioye Health Committee 10/14/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye AOR 10/13/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Health center of Goudiry 10/13/2013 Tambacounda In Depth Interview Dr. Ismaila Thioye CH Insurance of MGES 10/14/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Health Insurance for Teachers in Senegal and General Movement of Teachers of Senegal 10/14/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Governor’s Office 10/13/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye AOR 10/13/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye AOR 10/13/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Health Post Kothiary 10/14/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Community Radio Goudiry FM 10/14/2014 Tambacounda Focus Group Discussion Dr. Ismaila Thioye Radio com Goudiry FM 10/14/2014 Tambacounda In Depth Interview Dr. Ismaila Thioye Mutual Bokk Faj 10/17/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye Health Committee of Kaffrine 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye Health Committee of Keur Lahine 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye CH Insurance Bokk Faj 10/17/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye Health committee of Keur Lahine 10/17/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye CH Insurance Committee Bokk Faj 10/18/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye AOR WVI 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye AOR ADEMAS 10/16/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye RM 10/15/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye Radio Kaffrine FM 10/15/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 176 Interview Date of Interview Region Interview Type EY Team Lead Governor’s Office 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye District of Kaffrine 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye OCB RADEC 10/16/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye Private pharmacy Mame Ousmane 10/17/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye OCB and Taxaw, Association of PLWHIV 10/17/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye OCB ARDIC 10/17/2014 Kaffrine Focus Group Discussion Dr. Ismaila Thioye Health post of Santhie Gal Ngoné 10/15/2014 Kaffrine In Depth Interview Dr. Ismaila Thioye AOR Abt 10/21/2014 Kaolack In Depth Interview Dr. Ismaila Thioye AOR ADEMAS 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye CBO: ASC Kanda 10/21/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye BR 10/24/2014 Kaolack In Depth Interview Dr. Ismaila Thioye Health Hut Kanda 10/21/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye BR 10/20/2104 Kaolack In Depth Interview Dr. Ismaila Thioye AOR Child Fund 10/21/2014 Kaolack In Depth Interview Dr. Ismaila Thioye Health Committee 10/21/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye PRA Kaolack 10/24/2014 Kaolack In Depth Interview Dr. Ismaila Thioye CBO Red Cross 10/21/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye Health Insurance Oyofal Paj de Kahone 10/19/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye CBO Bokk Lepp Association of PLWHIW 10/19/2014 Kaolack Focus Group Discussion Dr. Ismaila Thioye Health Insurance Oyofal Paj de Kahone 10/19/2104 Kaolack Focus Group Discussion Dr. Ismaila Thioye Governor's office 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye AOR IntraHealth 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye Private Pharmacy Djily Borom Bagdad 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye Health Post Kahone 10/21/2014 Kaolack In Depth Interview Dr. Ismaila Thioye Community Radio Al Fayda 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye RM Kaolack 10/20/2014 Kaolack In Depth Interview Dr. Ismaila Thioye AOR CRS 10/23/2014 Dakar In Depth Interview Dr. Ismaila Thioye Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 177 Team C Interview Date of Interview Region Interview Type EY Team Lead Africare 10/8/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo Governor’s Office 10/8/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo Supervisor of SSP, Coordinator of Reproductive Health Manager of EPS 10/8/2014 Sédhiou Focus Group Discussion Dr. Georges Tiendrebeogo District Chief MD / interim of the Regional Chief MD 10/8/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo ANCS Focal Point 10/8/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo AIDS service Focal Point 10/8/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo Group of PLWHIV 10/9/2014 Sédhiou Focus Group Discussion Dr. Georges Tiendrebeogo MSM Leader 10/9/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo CHW and Matron of the Health Hut of Badiary 10/9/2014 Sédhiou Focus Group Discussion Dr. Georges Tiendrebeogo Community Volunteer and BG 10/9/2014 Sédhiou Focus Group Discussion Dr. Georges Tiendrebeogo Abt Advisor in the RB of USAID/Senegal Kolda 10/9/2014 Sédhiou In Depth Interview Dr. Georges Tiendrebeogo FHI 360/Abt/the ChildFund consortium Advisors in the RB of USAID/Senegal in Kolda 10/10/2014 Kolda Focus Group Discussion Dr. Georges Tiendrebeogo Prefet of Kolda 10/10/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Deputy Governor in charge administrative affairs 10/10/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Chief Nurse in the Health Post of Bagadadji 10/10/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Beneficiaries of the CH Insurance of Bagadadji 10/10/2014 Kolda Focus Group Discussion Dr. Georges Tiendrebeogo Coordinator USAID/Senegal RBof Kolda 10/16/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Group of PLWHIV 10/11/2014 Kolda Focus Group Discussion Dr. Georges Tiendrebeogo FBR Advisor 11/10/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo UAR Kolda 10/16/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Chief of Medical District /interim of Regional Chief MD the Regional Coordinator reproductive health 10/11/2014 Kolda In Depth Interview Dr. Georges Tiendrebeogo Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 178 Interview Date of Interview Region Interview Type EY Team Lead Team CSW 10/14/2014 Kolda Focus Group Discussion Dr. Georges Tiendrebeogo FHI 360 Advisor 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Planning with the Regiona Chief MD 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Prefet of Kolda 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo District Chief MD and the MD in charge of PLWHIV 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo IEC/CCC Supervisor for ADEMAS 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Director of community of Kédougou FM 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Regional Chief MC 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Governor and his deputies in charge of administrative affairs and of community development 10/13/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Matron/CHW Koundoukhou 10/14/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Health Committee of Koundoukhou 10/14/2014 Kédougou Focus Group Discussion Dr. Georges Tiendrebeogo Chief Nurse of Dalaba 10/14/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo Group of CSWs of Kédougou 10/13/2014 Kédougou Focus Group Discussion Dr. Georges Tiendrebeogo Group of PLWHIV Kédougou 10/14/2014 Kédougou Focus Group Discussion Dr. Georges Tiendrebeogo Community Radio of Kédougou 10/12/2014 Kédougou In Depth Interview Dr. Georges Tiendrebeogo MCD and Team 10/17/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo Community Radio AWAGNA FM – Department of BIGNONA 10/18/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo Health Post of Néma 10/17/2014 Ziguinchor Focus Group Discussion Dr. Georges Tiendrebeogo Health Post of Djiguinoume 10/18/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo Deputy Governors 10/17/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo Deputy Governors 10/17/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo Group of CSW in Ziguinchor 10/18/2014 Ziguinchor Focus Group Discussion Dr. Georges Tiendrebeogo FHI 360 Partners 10/18/2014 Ziguinchor Focus Group Discussion Dr. Georges Tiendrebeogo Group of PLWHIV Ziguinchor 10/18/2014 Ziguinchor Focus Group Discussion Dr. Georges Tiendrebeogo Chief MD of Ziguinchor 10/18/2014 Ziguinchor In Depth Interview Dr. Georges Tiendrebeogo SIS Afrique 10/22/2014 Dakar In Depth Interview Dr. Georges Tiendrebeogo SIS Afrique 10/22/2014 Dakar Focus Group Discussion Dr. Georges Tiendrebeogo CNLS Executive Secretary CNLS (National Committee for the Fight against 10/24/2014 Dakar In Depth Interview Dr. Georges Tiendrebeogo Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 179 Interview Date of Interview Region Interview Type EY Team Lead AIDS FHI 360 10/29/2014 Dakar In Depth Interview Dr. Georges Tiendrebeogo Network for PLVHIV 10/30/2014 Dakar In Depth Interview Dr. Georges Tiendrebeogo Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 180 Annex D: Evaluation Work Plan Disclaimer: Deliverable I: Evaluation Work Plan included below was previously approved by USAID/Senegal during October 2014. 1.0 PROJECT OVERVIEW 1.1 Project Purpose The United States Agency for International Development in Senegal (USAID/Senegal) works in partnership with the GOS to support the ten year National Plan for Health and Development (PNDS 2010 - 2018). To provide support, USAID/Senegal is implementing a large five component health program (i.e., Health System Strengthening, Health Services Improvement, CH, HIV/AIDS & TB, and Health Communication and Promotion) with an overarching goal of “Improved Health Status of the Senegalese Population.” In September 2014, USAID/Senegal engaged EY to provide technical assistance for a mid-term evaluation of its health program. 1.2 Project Objectives USAID/Senegal’s primary objective of the mid-term health evaluation is to assess how each of the five program components, their intermediate result (IR), and sub-IRs all link to achieve the overall goal of improved health status of the Senegalese population. The evaluation covers the period from October 2011 to June 2014. To achieve this objective, the evaluation will: • Assess progress toward achieving the expected results of the USAID/Senegal Health Program (2011 - 2016) • Assess effectiveness of program design, implementation, and sustainability mechanisms • Identify lessons learned and propose actionable recommendations to guide implementation for the remaining period of the program to improve performance The evaluation will be used by various stakeholders (e.g., USAID/Senegal, the Ministry of Health (MOH), Implementing Partners, and the United States Government (USG)) and will help inform USAID/Senegal’s upcoming strategic plan. To support these assessment requirements, EY teamed with Dr. Ruth Kornfield and Africa Consultants International (ACI), collectively forming Team EY to conduct this mid-term evaluation. The information contained within this Evaluation Work Plan details Team EY’s program management approach, evaluation design, data collection methodology, evaluation tools, and assumptions and constraints. 2.0 PROJECT ORGANIZATION Team EY is comprised of a group of experienced and qualified personnel with significant years of health evaluation experience and knowledge of the Senegalese country context. The Engagement Executive, Ms. Aloha McBride, is responsible for the overall management of this engagement. For this project we defined a management structure that supports the planning, implementation, and reporting for required services; and emphasizes close monitoring of cost, project plan, and performance. The Team EY engagement structure is provided in Figure #1 Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 181 below. EY, as the prime contractor, will serve as lead on project administrative and quality control activities. The EY support team is a virtual team that will remain at EY headquarters to help manage the volume of data being collected, support data analyses, and assist in the development of the Final Evaluation Report deliverable to USAID/Senegal. Figure 1: Team EY Organizational Chart Project delivery and day-to-day project management will be executed under the guidance of the Evaluation Team Leader, Dr. Kornfield. In addition, Dr. Georges Tiendrebeogo and the Program Analyst/Site Visit Lead from ACI will serve as team leaders while conducting data collection activities in the field. The in-country team is responsible for stakeholder interviews, field work preparation, scheduling and logistics, data collection at field sites, and drafting the Final Evaluation Report. The dual process of the in-country and headquarters team will support the synthesis of information, identification of lessons learned, and the development of recommendations. 3.0 PROJECT MANAGEMENT APPROACH To accomplish the objectives of the mid-term health evaluation, our approach as depicted in Figure #2 below will include Evaluation Preparation, Data Collection, Data Analysis and Evaluation, and Report Finalization. Our project approach includes the specific work streams as specified in the contract. Included within each work stream are the associated activities and deliverables Team EY will execute to achieve the project objectives. Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 182 Our team will collect data from a minimum of 10 sites in each of the 14 regions. We will gather data using a mixed method approach, beginning with a literature review of existing material as well as collecting new data through key stakeholder interviews, observations, and focus groups. Our team will formulate conclusions and propose key recommendations that USAID/Senegal can utilize to design subsequent health programs. Evaluation Preparation ► Post-award call and kick-off with the Evaluation Team Leader and headquarter team to review project understanding, confirm scope, and solidify processes ► Review all data provided by USAID/Senegal ► Conduct a literature review ► Perform information gap analysis ► Develop management tools including tools for qualitative and quantitative data collection ► Develop and submit the Evaluation Work Plan to USAID Senegal ► Plan, organize the schedule for site visits, and conduct in￾country team meetings with the Agreement Officer’s Representative (AOR), implementing partners, and other key stakeholders ► Begin development of the draft Evaluation Report Outline ► Travel to and conduct data collection in 10-15 sites in all 14 regions ► Conduct progressive data analysis, including financial analysis ► Review all existing data collected for information gaps and request additional existing data from USAID/Senegal and Implementing Partners, as needed ► Submit Evaluation Report Outline to USAID/Senegal ► Analyze and synthesize data into manageable sections of information that can be further developed into preliminary findings, lessons learned, and recommendations ► Develop Draft Evaluation Report ► Translate the Draft Evaluation Report into French ► Submit Draft Evaluation Report to USAID/Senegal ► Incorporate feedback from USAID/Senegal into the Final Evaluation Report ► Translate the Final Evaluation Report into French ► Submit Final Evaluation Reports to USAID/Senegal ► Develop Briefing Document ► Translate the Briefing Document into French ► Submit final Briefing Documents to USAID/Senegal ► Submit Final Evaluation Reports to the Development Experience Clearinghouse (DEC) Associated Deliverables: – Evaluation Work Plan Associated Deliverables: – Final Evaluation Reports – Briefing Documents – Electronic copies of the Final Evaluation Reports to the DEC Data Collection Data Analysis and Evaluation Report Finalization Associated Deliverables: – Draft Evaluation Report Associated Deliverables: – Evaluation Report Outline Figure 2: Project Approach Mid-Term Evaluation of USAID/Senegal Health Program Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 183 3.1 Key Deliverables Table #1 below describes the key deliverables required for this project and their expected due dates. Deliverable Title Requirements Expected Due Date Evaluation Work Plan  An operational work plan that includes the proposed evaluation design and data collection methodology, timeline, and the tools (including questionnaires) to be used for the evaluation. 22 SEP 2014 Evaluation Report Outline  Proposed outline for the Final Evaluation Report for USAID/Senegal’s approval. 20 OCT 2014 Draft Evaluation Report  A Draft Report in French and English. 10 NOV 2014 Briefing Document  A PowerPoint document (French and English) for briefing representatives from USAID/Senegal, implementing partners, and the MOH. 26 NOV 2014 Final Evaluation Report  The Final Evaluation Report will be submitted in French and English. 8 DEC 2014 Table 1: Deliverable Schedule Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 184 3.2 Project Milestones Key project milestones are depicted in Figure #3 below. The project plan for the assessment is in Section 6.0: Project Plan. Figure 3: Project Milestones Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 185 3.3 Contract Management 3.3.1 Deliverable Quality Management The goal of Team EY’s quality control plan is to actively engage with USAID/Senegal’s to support meeting its needs within realistic cost and schedule parameters. Team EY’s approach to quality management will start by understanding USAID/Senegal’s quality requirements and measures of performance and to then clearly define our quality policy, objectives, standards, roles, responsibilities, and quality management processes to meet USAID/Senegal’s requirements. Our internal quality management process for reviewing deliverables includes multiple levels of reviews by a peer reviewer, the Evaluation Team Leader, the Technical Advisor, and the Engagement Executive. Following the incorporation and validation of all changes, the deliverable will be reviewed by an independent EY pre-issuance reviewer and then submitted to USAID/Senegal. 3.3.2 Stakeholder and Communication Management It is critical to correctly identify and actively engage internal and external stakeholders as to promote timely and effective communication across all individuals and organizations involved in the project. Specifically, stakeholders are defined as people or organizations invested in the program, interested in the results of the evaluation, and/or with a vested stake in the results of the evaluation. As such, accurately addressing and representing their needs and interests throughout the process will be fundamental to a successful program evaluation. Stakeholder management analysis is an important aspect of this assessment. Our team will work with the COR to identify key stakeholders from USAID/Senegal, the GOS, MOH, the Implementing Partners, and beneficiaries for our interviews. Our team is working with USAID/Senegal to establish a common understanding of the project purpose, scope, and frequency of communication with USAID/Senegal and other key stakeholders. As Team EY conducts the assessment, we will develop and manage an interview document to keep track of all stakeholders interviewed. In addition to the tracker, we will facilitate coordination and structured communication with USAID/Senegal throughout project execution. Communication, coordination, and integration will be accomplished throughout the project using several mechanisms such as weekly internal Team EY meetings and weekly meetings with USAID/Senegal, in conjunction with other scheduled meetings conducted as part of the assessment. Our team will work continuously with USAID/Senegal to further identify communication mechanisms and determine the most effective ways to communicate. 3.3.3 Project Plan Management Insight gained from our discussions with initial stakeholders assisted with the development of the project plan and management approach which includes: confirming intent of specific deliverables, defining the tasks and activities required to develop the deliverable, sequencing tasks and activities appropriately, understanding the level of effort and duration required to complete each task and activity, and understanding the necessary resource requirements and associated costs as inputs to creating the program schedule. The full project plan is included in Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 186 Section 6.0: Project Plan. The Engagement Executive, in conjunction with the Evaluation Team Leader, is responsible for the successful development and execution of the project plan and team members will be responsible for compliance with the project plan. The project plan will be regularly updated as logistics are finalized. 3.3.4 Risk Management Team EY utilizes the Project Management Body of Knowledge methodology to mitigate risk. EY’s risk management approach is part of our corporate history of solid risk planning and identification processes, which begin at project inception and will continue throughout the project life cycle. EY utilizes a secure, web-enabled collaboration platform called EY Delivers, which will allow us to protect information being provided to us for analysis in the evaluation. The core activities included in our risk management approach include: planning, identification, assessment, mitigation, and monitoring. At the weekly USAID/Senegal meeting, Team EY will inform the COR of identified risks, determine potential impacts, and discuss mitigation strategies. 3.3.5 Cost Management Our team understands the importance of properly managing costs. We will closely monitor costs and the schedule throughout the period of performance to deliver projects on time and on budget. The Engagement Executive is ultimately responsible for monitoring expenditures so they do not exceed authorized funding. We will use periodic project and task financial reports to identify and address cost variance and take corrective actions with the goal of re-establishing performance in line with approved cost baselines. Additionally, our project has a dedicated professional assigned to monitor project costs, provide regular reporting analysis to the project team, and track/submit subcontractor and USAID invoices per contract requirements. 4.0 PROPOSED EVALUATION DESIGN 4.1 Overview Our evaluation design will identify the factors which both facilitate and impede improvements, and support efficiency, effectiveness, relevance, and sustainability with regard to health system strengthening within the context of regional decentralization. It will also assess coverage, access, and quality and provide both qualitative and quantitative information as available to identify catalysts for change and areas for further improvement. Therefore, one aspect of the evaluation is to assess the sustainability and cost-effectiveness of USAID/Senegal’s initiatives to strengthen both service delivery as well as HIS. Our evaluation design includes principal evaluation questions and our data collection methodology. In our data collection methodology, we will describe the sampling techniques, plan for data collection points/site selection criteria, data collection tools, and process for data management and analysis. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 187 4.2 Principal Evaluation Questions We will focus on the following principal evaluation questions in Table #2 below as stated in the contract. Additionally, we utilized USAID/Senegal’s Data Source Matrix to help our team organize our approach to responding to the evaluation questions. PRINCIPAL EVALUATION QUESTIONS PROGRAM DESIGN AND INTEGRATION How effective has the structure of USAID’s overall health program and the division of the program into five components been in helping achieve the health development objective? How have interventions been coordinated and implemented in synergy across components, with other USAID/Senegal programs, and with other development partners? To what extent has direct financing to the three regions been implemented successfully and what could be improved? To what extent has the system of RBs and integrated work plans improved coordination among the five components? To what extent has the program strengthened government ownership and demonstrated sustainability? COMPONENT SPECIFIC QUESTIONS To what extent have the Components achieved their objectives? To what extent has each sub-component been successfully implemented? What are the factors contributing to the achievement of each sub-component? What are the constraints and challenges that have hindered successful implementation of each sub￾component, and how has the implementing partner dealt with those challenges? Are there interventions that should be added or removed? Are there changes that could be made to improve performance? To what extent have gender considerations been integrated into the design and implementation of each component? Have women and men benefitted from each component equitably or differently, and how? Table 2: Principal Evaluation Questions 4.3 Data Collection Methodology For this evaluation, our team will execute formal data collection. Specifically, our methodology identifies how the data will be collected, where/when data will be collected, and how the data will be analyzed. Our process describes a project plan for collection and how each step of the process will occur and support that data collected remain in a reliable format. Our methodology will employ a mixed-method approach whereby the team will collect qualitative and quantitative data concurrently to cross-validate or corroborate findings within our assessment. Quantitative information will be primarily sourced from existing data, surveys, and supported with a literature review. Quantitative data will be continuously captured and analyzed throughout Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 188 the assessment. Quantitative data will be collected from various sources including the DHS, SPA, USAID/Senegal Health Strategy 2011 - 2016, The National Health Development Plan (PNDS) 2010-2018, Implementing Partners’ AWPs, Implementing Partners’ quarterly and annual reports, and indicators reported in the National Health Management Information System. Qualitative information will be derived from feedback from standardized, individual in-depth interviews, focus groups discussions, and observations conducted at selected data collection points/site visits. The qualitative data collection will be conducted in order to better understand the factors which both facilitate and hinder high quality interventions at the decentralized level. Based upon trends emerging from the quantitative data, the selection of individuals and groups for qualitative interviews will lead to an understanding of the processes by which high and low performing sites/regions achieve different outcomes. Qualitative data methods are: • Focus group discussions with clients and health center staff. Since there are a limited amount of health post/center staff for a focus group discussions, we will also include beneficiaries and community members Focus groups will allow us to better understand beneficiaries’ perspectives with regard to access to services, quality (including availability of commodities, confidentiality etc.), gender, and equity based differences on use and impact, ownership, accountability, and sustainability. • In-depth interviews with service providers and community level stakeholders to elicit their perspectives on barriers to behavior change and uptake of services. The interviews will allow us to better understand the impact and effectiveness of programs on technical capacity (including training, task shifting, and management of local budgets), linkages and referrals within the health pyramid, resource allocation, health care workers’ workload, job satisfaction and motivation, sustainability (including engagement with local government and local community), and gender-specific and equity considerations. • Observations to understand what happens when patients attend health facilities. For example, how long they wait for services, what type of information they receive, provider￾client interactions, and how clinical data are captured. 4.3.1 Plan for Data Collection Points/Site Visits and Selection Criteria Our data sampling plan will be a design that includes both certainty cases, those that are important and purposely needed in the sample, and random site selection in order to extrapolate data. We anticipate using a two stage sampling method where we sample facilities in each of the regions. In order to capture data from the 14 regions, our in-country team will divide into three field teams, comprised of three team members each. The Team Leader, Dr. Kornfield, will lead one of the field teams, the Deputy Team Leader, Dr. Georges Tiendrebeogo, will lead the second team, and the third team will be led by the Program Analyst/Site Visit Lead from ACI. A number of factors are included in deciding which team members and regions the teams will visit. Factors include guidance from the USAID/Senegal COR and technical advisors, local regional contextual knowledge and experience, local language capabilities, as well as validating that we are linking the appropriate technical skills of each member to the health activities being implemented in the regions where they will be collecting data. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 189 Data will be collected from the national, regional, and district levels, and will include facility￾based, community-based, and administrative sites (i.e., regional health offices). Data will also be collected from a variety of sources to include Abt, IntraHealth, ChildFund, FHI 360, ADEMAS, MOH (policy makers, managers), RB representatives, CHWs, BG , representatives of local non￾governmental organizations (NGOs) and local implementing organizations, community stakeholders (i.e., local community leaders, religious leaders, women’s leaders, health advocacy groups), mothers of children under five, FP users, adolescents, commercial sex workers, men having sex with men, PLWHA, men accessing health services, health management committees, and the private sector. Team EY is currently mapping out USAID/Senegal’s interventions across regions by Implementing Partner. This mapping is expected to be finalized during week three and will assist Team EY in further refining the criteria for data point/site selection. The initial criteria for site selection are: • At least one district per region • Equivalent health interventions implemented in regions with a regional coordination office compared to those without a regional coordination office • Facility-based sites that represent urban, semi-urban/rural, and rural populations • Community-based sites that represent urban, semi-urban/rural, and rural populations • Representation of sub-recipients to include a minimum of one project under each of the prime Implementing Partners • Representation of regions with PBF interventions • Representation of regions where there are CH insurance interventions (Mutuelles Communautaires) • Representation of high-performing regions versus low performing regions (based on health indicators) • Representation of private sector pharmacies and health facilities 4.3.2 Data Collection Tools Data will be collected using several different data collection tools, consisting of focus group discussion guides, in-depth open-ended interview guides, and an Implementing Partner survey. The data collection tools are directly aligned with USAID/Senegal’s Principal Evaluation Questions and the five components of the overall health program. The data collection tools will be continuously refined based on feedback from USAID/Senegal as well as information gathered during the course of data collection. We are including the private sector in our data collection by assessing existing public-private partnerships, beginning with interviews of private pharmacies involved in social marketing and product distribution. Private health facilities as well as Centres de Conseil Adolescents, beneficiaries, and CHWs delivering USAID/Senegal supported services will also be included in our sample. The qualitative data will be conducted in order to better understand the factors which both facilitate and hinder high quality interventions at the decentralized level. This will also allow for the comparison of high-performing and low Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 190 performing health regions to better understand the factors that contribute to or mitigate successful health programs. The list of focus group discussion guides is included below: • CH workers • CH insurance (Committee) • CH insurance (Leader/Manager) • CH insurance (Beneficiaries) • CH management committee • Associations of PLWHA The list of in-depth interview guides is included below: • AOR/Implementing Partner • MOH – Component #1: Health System Strengthening • MOH – Component #2: Health Services Improvement • MOH – Component #3: CH • MOH – Component #4: HIV/AIDS & TB (Co-infection) • MOH – Component #5: Health Communication and Promotion • Health facilities (i.e., hospitals, centers, and posts) • Health huts • National pharmacy (GOS) • Private pharmacies • Regional and district health (GOS) • Regional and district coordinating offices (USAID/Senegal) • Public-private partnerships • Team EY has included in the submission of the work plan an excel file, USAID_French Guides_9.29.14_DRAFT_FINAL.xls, which includes all the interview and focus group guides listed above. The last tool is the Implementing Partner survey which can be found in Annex A: Implementing Partner Survey. This short survey is an important first step for assessing the implementing partners’ perspectives in order to maximize the use of their time when conducting the in-depth interview. 4.3.3 Process for Data Management and Analysis Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 191 The first step in the data management and analysis process is to inventory and compile all background documentation into a central repository. Team EY will utilize our secure EY Delivers site which serves as a central repository to control project management and project delivery activities. The EY Delivers site will be used to track the status of key activities (e.g., submission and completion of surveys), store background documentation, and store other key project information such as team meeting minutes, lessons learned, etc. It will also be used to maintain version control of the most up to date data collection tools. The development of the data collection tools outlined in Section 4.3.2 is a critical step in the data management and analysis process. Each question in the data collection tool is coded with a unique identifier which links back to the principal evaluation questions, components, IRs, and sub-IRs. While the tools and question alignment are a key step in the data management and analysis process, Team EY understands that tools alone will not yield the evidence needed to draw conclusions or lessons learned. The data collectors are integral to the process of collecting and interpreting data so that data integrity is present and can be demonstrated to external audiences. In order to maintain data integrity, the Team Leader will train all of the data collectors on how to conduct in-depth interviews as well as focus group discussions. In the trainings, she will emphasize that data will be collected using a multi-vantage point approach, where a conscious effort will be made to not let the data collector’s biases shape how they interpret responses. All data collectors will operate using the same set of assumptions, definitions, and approaches to limit biases and pre-conceptions. The teams will be required to discuss their data with each other on a frequent basis so that they are considering alternative interpretations of data and challenging viewpoints. Team EY developed a process map that was used during the training to clearly demonstrate all the data collection steps and feedback loops in order to decrease the potential gaps in data collected and to clean the data prior to analysis beginning. After the training is complete, the team will be ready to begin data collection in the field. All data collected will be reviewed and approved by the Evaluation Team Leader prior to analysis. Our primary mode for collecting and analyzing data will be progressive focusing. Progressive focusing is defined as adjusting the data collection process when it begins to appear that additional concepts need to be investigated or new relationships explored (Parlett & Hamilton 1976). Team EY will begin with a set of defined questions, but realize flexibility is required as new data may lead to additional questions, concepts, and information that was not previously considered. During the site visits, our three field teams will compile data progressively. While in the field, each team will assign a member to enter the data collected during the day into a data collection template. The data collection template will then be sent to the field team leader who will review it for errors and accuracy before sending it to the Evaluation Team Leader. Once the Evaluation Team Leader reviews and approves the quality of the data in the data collection template, it will be sent to the EY Support Team and the ACI Data Collection and Analysis Team. These teams will use software such as STATA or SPSS to upload the data for analysis. This software will allow the team to sort qualitative data to understand the frequency of specific observed variables or responses across interventions, regions, or Implementing Partners. This will allow the team to draw conclusions regarding where activities are effective and where challenges exist. For example, it will allow the team to see beneficiaries’ satisfaction with a Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 192 particular service received, broken down by region or even district. If data allows, such responses could be further sorted based on demographic information, such as responder’s gender. We anticipate that each field team will conduct four interviews and two focus group discussions per day. At the conclusion of field visits, the field team leaders will continue the process of data analysis by dividing up the program components into IRs and sub-IRs. Team EY will utilize the data collected from the observations and surveys to perform a comparison analysis with other regions/locations and cross-sectional analysis with other descriptive factors such as demographic and geographic data to understand whether socioeconomic, age, or gender impacts are apparent. We anticipate that we will conduct 28 focus groups, two in each region and there will be seven to twelve (12) participants in each group. To analyze the data we will use the following academically accepted process of: • Documentation of the data and the process of data collection • Organization/categorization of the data into concepts (e.g., health components, IRs, and sub￾IRs) • Connection of the data to show how one concept may influence another • Data validation, by evaluating alternative explanations, disconfirming evidence, and searching for negative cases • Representing the account (reporting the findings) 5.0 ASSUMPTIONS AND CONSTRAINTS Team EY listed assumptions and constraints relative to this project in Table #3 and #4 below. 5.1 Assumptions Assumption Description Period of performance The project will end on 8 DEC 2014. Data availability USAID/Senegal will provide key strategic documents, project contracts, IP AWPs, budgets, indicators/metrics data, and other necessary information to meet the evaluation objectives and assess performance against expected standards/results. Coordination assistance USAID/Senegal will provide contact information of key stakeholders for in-person interviews. If Team EY experiences difficulty in scheduling interviews/meetings, USAID/Senegal will provide support as stated in the Request for Proposal (RFP). Table 3: Project Assumptions 5.2 Constraints Constraints Description Access to health sites On-site access is required to conduct observations and interviews. Team EY is dependent on USAID/Senegal for expediting the protocol to gain access to hospital, clinics, and community-based sites. Team EY will work with the COR in advance to Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 193 Constraints Description gain approval for site visits. Regulations and requirements EY is a public accounting firm subject to certain regulations and requirements issued by the American Institute of Certified Public Accountants (AICPA), Public Company Accounting Oversight Board (PCAOB) and other regulatory and professional bodies and its professional industry. EY will provide its services in accordance with the Standards for Consulting Services established by the AICPA. Data point/site selection criteria To accurately develop data points/site selection criteria, an existing mapping of interventions is required. Based on documentation requested and received from USAID/Senegal to date, a mapping was not available. Team EY is currently dedicating resources to develop the mapping in order to appropriately select data points/sites. Therefore data point/site selection will be provided after the submission of the Evaluation Work Plan. Table 4: Project Constraints 6.0 PROJECT PLAN Team EY developed a detailed project plan that lists tasks to be performed to meet objectives. These functions are broken down into groups of sub-tasks and activities, which are further refined into smaller components so that each element can is linked to the required resources. This approach results in a more manageable and understandable break down for each activity/task. The full project plan for the evaluation is depicted in Figure #4 below. Once site selection is complete and approved by USAID/Senegal, we will update the project plan to reflect the regions each of the field teams will be going to collect data. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 194 Figure 4: Project Plan Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 195 Annex E: Alignment of Principal Evaluation Questions and Component Specific Questions Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment Program Design and Integration How effective has the structure of USAID/Senegal’s overall health program and the division of the program into five components been in helping achieve the health DO? All: • Component 1: HSS (Abt) • Component 2: HSI (IntraHealth) • Component 3: CH (ChildFund Consortium) • Component 4: HIV/AIDS (FHI 360) • Component 5: HCP (ADEMAS) All: • IR 1: Increased availability of an integrated package of quality health services • IR 2: Improved health seeking and healthy behaviors • IR 3: Improved performance of the health system All: • Sub IR 1.1: Increased access to quality clinical services • Sub IR 1.2: Increased access to quality CH services • Sub IR 1.3: Increased access to key health products • Sub IR 2.1: Improved attitudes toward healthy behaviors • Sub IR 2.2: Improved promotion of key health products and services • Sub IR 3.1: Improved management of district and regional health teams • Sub IR 3.2: Improved health system performance through implementation of national level policies AOR/IP: a1_1-_3; a4_1; b1_1- 2; b3_1- b5_4 Regional Coordination Offices: brc_1-11;14- 23; 28-33 CHW: asc_1-8 How have interventions been coordinated and implemented in synergy across components, with other USAID/Senegal programs, and with other development partners? All All All AOR/IP: a2_1; a4_2- 3; b2_1- 4 CHW: asc_2 To what extent has DF to the three regions been implemented successfully and what could be All All All AOR/IP: a3_1-a3_5 MOH Component 1: Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 196 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment improved? moh_com1_3a; 5b; 6a Regional and District Health Offices: rds_com1_5 Regional Coordination Offices: brc_12; 13; 24-27 To what extent has the system of RB and integrated AWPs improved coordination among the five components? All All All AOR/IP: a4_4 MOH Component 1: moh_com1_3e; 3f; 5f; 5g; 6e; 6f Regional and District Health Offices: rds_com1_7; 11; 12 PLWHIV: plwha_com4_4a- 4b To what extent has the program strengthened government ownership and demonstrated sustainability? AOR/IP: a5_1- 3 Regional and District Health Offices: rds_com3_2- 2a Component Specific Questions To what extent have the components achieved their objectives? Component 1 IR3 Sub IRs: 3.1 & 3.2 MOH Component 1: moh_com1_1 – 3; 3b￾3d; 4; 5c- 5e; 6; 6b￾6d; 7b- 7e Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 197 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment Health Structures: ss_com1_1- 4; 6- 12; 15; 17- 18 Health Huts: cs_com3_29- 30 National/Regional Pharmacies: pn_com1_1- 2; 4; 6-12 Regional and District Health Offices: rds_com1_1-3; 6; 8; 10; 13 –14 Community Health Insurance (Committee): csm_com1_2- 8; 10- 11; 13-15; 17 Community Health Insurance (Manager): gm_com1_1- 6; 8-11; 13 -15; 19 -24a Community Health Insurance (Beneficiaries): bm_com1_1- 4; 7- 10 Component 2 IR1 Sub IRs: 1.1, 1.2, & 1.3 MOH Component 2: moh_com2_1- 6e; 8- Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 198 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment 9e Health Structures: ss_com2_26-27; 31 Regional and District Health Offices: rds_com2_1-7; 9- 15 Component 3 IRs:1, 2, & 3 Sub IRs: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, & 3.2 MOH Component 3: moh_com3_1- 5e; 7- 7d Regional and District Health Offices: rds_com3_1-8; 11 Community Health Management committee: cgcs_com3_1- 5; 8-14; 16-19; 22-24; 26 Component 4 IRs:1 & 2 Sub IRs: 1.1, 1.2, 1.3, 2.1 & 2.2 MOH Component 4: moh_com4_1- 6; 8- 9 Regional and District Health Offices: rds_com4_1- 4; 4d -5; 5b-5c; 6; 8- 8c PLWHIV: plwha_com4_1- plwha_com4_11 Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 199 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment Component 5 IR2 Sub IRs: 2.1 & 2.2 MOH Component 5: moh_com5_1- 3 National/Regional Pharmacies: pn_com5_15- 20; 26- 26a Private Pharmacies: pp_com5_1- 2; 3a; 4- 7; 9- 10 Regional and District Health Offices: rds_com5_1- 8a PPP: ppp_com5_1-10; 12; 14 To what extent has each sub￾component been successfully implemented? What are the factors contributing to the achievement of each sub-component? Component 1 IR3 Sub IRs: 3.1 & 3.2 Health Structures: ss_com1_5 Health Huts: cs_com3_1- 4; 7- 13;16- 21; 28; 31- 32; 32b- 32d; 36 Component 2 IR1 Sub IRs: 1.1, 1.2, & 1.3 Health Structures: ss_com2_18- 24;28- 30; 32 -33 Component 3 IRs:1, 2, & 3 Sub IRs: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, & 3.2 N/A Component 4 IRs:1 & 2 Sub IRs: 1.1, 1.2, 1.3, 2.1 & 2.2 Health Structures: ss_com4_1- 13; 15- Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 200 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment 16; 19 Component 5 IR2 Sub IRs: 2.1 & 2.2 N/A What are the constraints and challenges that have hindered successful implementation of each sub-component, and how has the implementing partner dealt with those challenges? Component 1 IR3 Sub IRs: 3.1 & 3.2 MOH Component 1: moh_com1_5-5a; 7-7a Health Structures: ss_com1_13- 14; 16 Health Huts: cs_com3_5- 6; 22- 27; 32b; National/Regional Pharmacies: pn_com1_3- 3a; 5; 13; 14 Regional and District Health Offices: rds_com1_4 Community Health Insurance (Committee): csm_com1_9; 12; 16 Community Health Insurance (Manager): gm_com1_12; 16- 18 Community Health Insurance Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 201 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment (Beneficiaries): bm_com1_5-6a Component 2 IR1 Sub IRs: 1.1, 1.2, & 1.3 MOH Component 2: moh_com2_7-7f; 10; 11 Regional and District Health Offices: rds_com2_8 Component 3 IRs:1, 2, & 3 Sub IRs: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, & 3.2 MOH Component 3: moh_com3_6- 6e; 8- 9 Regional and District Health Offices: rds_com1_9 Community Health Management Committee: cgcs_com3_15-16; 20- 21- 21a; 25 Component 4 IRs:1 & 2 Sub IRs: 1.1, 1.2, 1.3, 2.1 & 2.2 MOH Component 4: moh_com4_7; 7a; 10- 12 Health Structures: ss_com4_14; 17; 18 Regional and District Health Office: rds_com3_10-10a Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 202 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment PLWHIV: plwha_com4_4c; 5a; 5c_i; 7 Component 5 IR2 Sub IRs: 2.1 & 2.2 MOH Component 5: moh_com5_4; 11 National/Regional Pharmacies: pn_com5_21; 23 Private Pharmacies: pp_com5_3; 3b; 8 Regional and District Health Office: rds_com5_9-10 PPP: ppp_com5_11; 13 Are there interventions that should be added or removed? Are there changes that could be made to improve performance? Component 1 IR3 Sub IRs: 3.1 & 3.2 Health Huts: cs_com3_14; 15; 37 Regional and District Health Offices: rds_com1_15 Community Health Insurance (Committee): csm_com1_18- 19 Community Health Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 203 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment Insurance (Beneficiaries): bm_com1_11 Community Health Insurance (Manager): gm_com1_25 Component 2 IR1 Sub IRs: 1.1, 1.2, & 1.3 MOH Component 2: moh_com1_12 Health Structures: ss_com2_1- 17; 18-25; 28-30; 32-33 Component 3 IRs:1, 2, & 3 Sub IRs: 1.1, 1.2, 1.3, 2.1, 2.2, 3.1, & 3.2 MOH Component 3: moh_com3_10 Regional and District Health Office: rds_com3_9 Community Health Management Committee: cgcs_com3_27 Component 4 IRs:1 & 2 Sub IRs: 1.1, 1.2, 1.3, 2.1 & 2.2 MOH Component 4: moh_com4_13 Health Structures: ss_com4_20 Regional and District Health Office: Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 204 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment rds_com4_12 PLWHIV: plwha_com4_10- 11 Component 5 IR2 Sub IRs: 2.1 & 2.2 MOH Component 5: moh_com5_12 National/Regional Pharmacies: pn_com5_24- 25 Regional and District Health Office: rds_com5_11 PPP: ppp_com5_15 To what extent have gender considerations been integrated into the design and implementation of each component? Have women and men benefitted from each component equitably or differently, and how? Component 1 IR3 Sub IRs: 3.1 & 3.2 Health Huts: cs_com3_32d- 35 Community Health (Staff): asc_8 Community Health Insurance (Committee): csm_com1_1 Community Health Insurance (Manager): gm_com1_7 Component 2 IR1 Sub IRs: 1.1, 1.2, & 1.3 N/A Component 3 IRs:1, 2, & 3 Sub IRs: 1.1, 1.2, 1.3, 2.1, 2.2, Community Health Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 205 Evaluation Question Health Component Alignment IR Alignment Sub-IR Alignment Tool Alignment 3.1, & 3.2 Management Committee: cgcs_com3_6- 7b Component 4 IRs:1 & 2 Sub IRs: 1.1, 1.2, 1.3, 2.1 & 2.2 PLWHIV: plwha_com4_9 Component 5 IR2 Sub IRs: 2.1 & 2.2 N/A Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 206 Annex F: Literature Review Annotated Bibliography HSS Jutting J., “Do Community-based Health Insurance Schemes Improve Poor People’s Access to Healthcare? Evidence From Rural Senegal,” World Development Vol. 32, Number 2, 2003. The study of mutuelle explored whether rural Senegal members of a health insurance scheme are actually better-off than non-members. The results show that in poor environments, insurance programs can work. Members of les MHO de sante (mutual health organizations) have a higher probability of using hospitalization services than non-members and pay substantially less when they need care. Furthermore, the analysis revealed that while the schemes attracted poor people, the most vulnerable are not reached or not participating. The ability of the community-based health insurance schemes to target those chronically poor within the region of coverage will take innovation and further investigation. Three debated approaches to address this gap in the poorest communities’ access to and use of these social insurance schemes in Senegal are the introduction of well-targeted subsidies, flexibility in the payment procedure of the premium, and the strengthening of the management capacity of the organizations running the MHO. Take away: More research is needed how to reach the poorest families and encourage them to take part in mutuelle organizations. Keugoung, B., Macq, J., Buve A., Meli, J. and Criel, B. “The interface between health systems and vertical programmes in Francophone Africa: the managers’ perceptions,” Volume 16 Number 4, 2011. Semi-structured interviews were conducted in the Democratic Republic of Congo with health mid-level managers from 11 Francophone countries. Based on these interviews, this paper summarizes the positive and negative effects of vertical programs, integration, and general health services. Some of the challenges summarized among the respondents in the areas of leadership and governance are, multiple coordination bodies in decision-making; loss of a macro vision that guides activities and performance, and insufficient management capacity of the district health teams. Lastly, lagging health policies, ill-conceived implementation plans, and weak pharmaceutical supply chains, arose as neglected items, pending solutions. In the area of HIS, the multiplicity of health information tools and indicators is equally problematic when implementation plans are not synced by an essential set of indicators. For health service delivery, the hypothesis put forth is that routine monitoring by the national and local health authorities, development partners and donors can contribute to timely identification of any disruptive effects of vertical programs and any unintended effects of integrated programs. Take away: Integrated consultations, program planning and shared indicators, although instructive, are only as effective as how managers perceive and implement them. In competitive service delivery environments, incentives that foster ways to collaborate and integrate services should be clearly established to help avoid vertical programming. Lemiere, C., “Senegal Health & Social Financing: P129472 - Implementation Status Results Report: Sequence 02.” Washington, DC: The World Bank, 2014. The Health and Social Financing Project, initiated in July 2014, aims to improve health and nutritional outcomes among women and children in regions among the poorest populations. The proposed Project Development Objective (PDO) is to increase utilization and quality of maternal, neonatal and child health care and nutritional services, especially among the poorest households in targeted Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 207 areas of Senegal. The social financing approach, particularly use of MHO in Senegal is thriving and reporting positive targets such as the creation of new MHO reaching communities, handicapped clients and other vulnerable groups. Take away: It is an opportune time to learn from the implementation experiences of this new The World Bank Project and look at differences reported in the future by geographical location, gender, age, and community profiles. These can inform existing programs and areas where mid-course corrections may be needed. Massoud, R., Mensah-Abrampah, N., Barker, P., Letherman, S., Kelley, E., Agins, B., Sax, Sylvia, and Heiby, J. “Improving the delivery of safe and effective health care in low and middle income countries” Research is needed into creating workable systems that can deliver and sustain interventions,” BMC, 2012. Africa lags behind in attainment of the millennium development goals despite progress. The gap between knowing components of an effective intervention and implementing these activities requires involves implementation of practices that are simple, evidence-based, and highly impactful for rural and urban populations means. For example, there is still room for improvement in broader adoption of active management of the third stage of labor intervention, skilled attendance at birth and adherence to infection prevention standards (i.e. proper use of gloves, hand washing and hygiene and sanitation practices) which work in settings wherever patients are located. Consideration of a systems approach is discussed as is a call for research in resource-constrained countries, such as an investigation of characteristics of leadership and provider action that explain consistency in implementation of best practices (rather than episodically). Knowledge management and dissemination of learnings, standards for how to maintain safe and effective health care practices, and affordability indices are explored. Take away: Leadership engagement matters (e.g. SSP centers, regional hospitals and tertiary centers) as well as interventions’ simplicity and capacity for replicability and transferability in different service sites. “Abuja Declaration: Ten Years On,” WHO, 2011. Since the 2001 Abuja Declaration target which states that at least 15% of a country’s national budget should be allocated for health, only South Africa and Rwanda have achieved the recommended threshold. The report asserts why as long as per capita health expenditure is low, Senegal and other countries will not achieve the health Millennium Development Goals. Take Away: A number of opportunities to overturn this trajectory were recommended: increase of donor contribution to development countries, enhanced governance and better controls of the funds flowing into the country; better reporting by implementing partners to governments to facilitate better health expenditure tracking by disease or health system priority. HSI Daff B., Seck C., Belkhayat H., Sutton P. “Informed push distribution of contraceptives in Senegal reduces stock-outs and improves quality of FP services,” Global Health: Science and Practice, Volume 2, Number 2,” 2014. Contraceptive stock-outs are a major issue in Senegal and occurred widely, despite stock availability at the national level. This pilot study of the informed push distribution model demonstrated feasibility as an appropriate and effective solution to address the problem of stock-outs in Senegal. Audits of public sector facilities supply chain data and monthly replenishment by dedicated logisticians were core interventions implemented. These generated timely and accurate details on contraceptive consumption by facility, in addition to maintaining a minimum stock level. Under the model, logisticians perform their duties set by Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 208 fixed-fee contracts based on binding requirements and penalties related to stock-out rates and data availability, before receiving payment. Take away: Access to and use of supply chains data by district and regional health managers can be a transformational public health management practice where they are trained in its content or skilled logisticians are on staff to utilize informatics. A positive outcome reported from the pilot is the growing culture of data-driven performance improvement in Senegal for supply chain managers. A clear implication from the pilot is an opportunity to invest in partners with expertise in health commodity procurement and security, to address supply chain challenges in collaboration with in-country stakeholders. “Evidence to Action Compendium of Best Practices,” 2013. The website includes a description of key factors related to quality, access and overall health service improvement recommendations, using the perspectives of what constitutes an enabling environment what constitutes an environment that facilities health-seeking behaviors. Examples of elements of an enabling environment for the provision of health services include: performance-based management; supportive policies and guidelines; evidence-based decision-making; contraceptive security; availability of human and financial resources; community mobilization to address socio-cultural barriers to seeking health services; youth-friendly services; and strategies to address gender norms. Examples of demand side elements for health posts and health centers to take into account in the provision of a well-coordinated and high quality services include: use of different models of BCC, targeted messages appropriate to the different needs of each phase of the reproductive life cycle; a rights-based approach to service provision; community mobilization for participation; social marketing and whole market approaches; and reduction in the cost of essentials medicines where possible. Take away: Both demand side and environment elements require attention and the right policy, implementation plan and oversight to roll out activities appropriate for health service improvements to be felt. Interventions must be tested and retested to gauge if what works in one district is transferrable in a different setting. Heiby, J. “The use of modern quality improvement approaches to strengthen Africa Health systems: a 5-year agenda,” Journal for Quality in Health care, 2014. Malaria, AIDS, pneumonia, diarrhea, obstetrical complications, and low skilled birth attendance remain dominate basic health problems in West Africa. At the heart of addressing low performance in African health systems, is a phenomena referred to as the relative neglect of health-care processes. The assertion and data in the article explores how to harness the impressive potential of broad-based improvements through approaches like regional hubs of excellence, additive health worker trainings in quality improvement methodology, well-documented policy developments, and changes in regulations/legislation. In addition, regional and district health committees must strive for partners to apply care models that link information from patient-provider interactions across organizational levels and across administrative operations at health facilities, community service delivery networks, and private sector entities. Take away: Investigation of planning, execution and learning processes should be built into performance measures of programs in Senegal. Relationships between decentralized management structures, implementing partners and communities should be based on two-way communication, functional procedures and take advantage of regular feedback channels such as meetings, informal on site visits, and formal referrals that help track performance and how well integrated systems are working. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 209 Rusa, L. Schneidman M., Fritsche G., and Musango L. “Rwanda: Performance-Based Financing in the Public Sector,” Center for Global Development. Three PBF schemes in Rwanda provide evidence of the feasibility and utility of the approach in both public and private nonprofit health facilities. Positive results from Butar, Cyangugu and Kigali, showed increased coverage in curative care, deliveries, FP, and measles services as compared to the non-PBF provinces. Performance was affected by incentives developed such as creative provider remuneration, recruitment, community engagement, and health facility investment approaches. Distinct strategies and activities related to facility-based and civil society monitoring also factored into some of the strengths and limitations for the effective implementation of PBF in Rwanda and lessons for other low-income countries. Take away: The appropriate mix of national and sub￾national policies, incentive systems, and supportive supervision and monitoring, are essential ingredients for a high functioning PBF model. Incentives have the potential to reward providers, patients, communities, health systems writ large through increases in quantity and higher quality of select services. Walshe, K. “Psuedoinnovation: the development and spread of health care quality improvement methodologies,” International Journal for Quality in Healthcare Volume 21, Number 3, 2009. This analysis of quality improvement methodologies on health care organizations covers a review from 1988 to 2007 that is useful for stakeholders in Senegal operating in the public health and policy spaces. Key results demonstrate a need for clear metrics by process and disease￾specific indicators linked to service intervention. In addition long-term investment and support is required to implement and achieve significant improvements in health program integration and improvement. Three critical terms and methodologies that remain relevant in health services improvement highlighted were accreditation schemes, patient safety, and clinical governance – the author voiced a strong caution for health practitioners to avoid switching from one quality improvement strategy to another. Takeaway: There is more to be gained by adoption of the most robust evidence-based quality improvement methodology in Senegal, deemed applicable to the health systems context and patient orientation rather than opting for new QI tools and methodologies which can be lost or diminished with turnover of clinical teams and local leadership. Understanding how each partner identifies what service improvements look like (e.g. quality regulation function at hospitals or nongovernmental sites; composite indicators as proxies for quality; socio-political awareness; and engagement levels) and manages its complexity, will call for a collaborative strategic planning, evaluation, and research cycle. CH Amanyeiwe, U., Leclerc-Madlala, S and Gardi, H., “Do Community-Based Programs Help to Improve HIV Treatment and Health Outcomes? A Review of the Literature,” World Journal of AIDS, 2014. The article through a review of literature presents positive results of the impact that community programs have on various dimensions of HIV treatment and care plus their contribution to health and HIV outcomes. Research focused on low-middle income countries with innovative strategies to address the HIV epidemic based on various resource constraints in Africa, Asia, Latin America, and studies that included multiple countries. Although community￾based programs remain an area for further investigation, authors assert that there is significant existing evidence on how community platforms and interventions address issues of stigma and disclosure, increase viral load of patients, improve adherence (which in turn decreases loss to follow-up and mortality), show remarkable health outcomes beyond HIV, and represent the most Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 210 cost-effective service delivery model in resource-limited settings. Take away: The role of community support, community-based activities and resources are integral part of effective HIV and health programs, especially when integrated from the planning, implementation, and evaluation phases of a project. Figuring out what is the right type and degree of collaboration between CBOs and HIV treatment facilities or health SDPs requires better study and documentation. Questions that capture how community-based systems work, such as aspects of the continuum of care (referrals and linkages), capacity development, supportive supervision, and M&E practices that complement the traditional health and HIV sector are noteworthy. Hodgins, S., Pullum, T. and Dougherty, L., “Understanding where parents take their sick children and why it matters: a multi-country analysis,” Global Health: Science and Practice, 2013. To effectively reach children with potentially life-threatening illness with needed treatment, it is important to understand where parents seek care. Data from 42 Demographic Health Surveys and Multiple Indicator Cluster Surveys conducted since 2005 were reviewed. A prominent finding is that a majority of care in Africa is sought from the public sector. For Senegal, specifically, analysis reveals that the private sector is not playing a prominent role in service provision, which implies a need to focus program efforts and health management information systems on interactions with public-sector providers, regional and district health committees and structures. Take away: The authors suggest that, due to the continued overall low level of care seeking and the very infrequent systematic utilization of CHWs as a source of care, investigation of implementation problems with community case management is a first step. Public health practitioners and national and subnational health management teams are urged to understand the factors that have contributed to poor performance of this program. Two known challenges, the availability of program commodities and acceptability of the providers, continue as barriers to obtaining care from public-sector health facilities. Although under-five mortality is low in Senegal compared neighboring countries in West Africa, malaria, pneumonia, and diarrhea persist as the leading cause of death for children. The relatively low rate of ORS dispensing also needs to be addressed. “Integrated Community Case Management of Childhood Illness: Documentation of Best Practices and Bottlenecks to Program Implementation in Senegal,” MCHIP, 2012. The integrated community base management approach is being implemented on a national scale in Senegal as in the Democratic Republic of the Congo, Rwanda, Madagascar, and Niger. Three promising practices were highlighted from the implementation experience: the use of global and local evidence and success stories to affect decision-making on seeking care for childhood illness, the use of credible local leaders as champions for policy change, and involvement of development partners like UNICEF and USAID. In terms of bottlenecks and select issues requiring attention, two financing issues were cited: the high costs associated with producing and broadcasting a successful television series during the expansion phase, and the cost of training CHWs and other community agents. Lastly, the creation of a framework on how to identify sick children and assist parents with accessing care through local support, and the sustained commitment of financial and technical partners, impacted the success of this model. Take away: Integrated community case management, specifically for diagnosis and treatment of diarrhea, pneumonia, and malaria can work especially where partnerships with communities reinforce household level responsibilities and imperatives that families invest in their own health. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 211 Stone, L., “Cultural influences in community participation in health.” Comparative Study Review, Soc Sci Med, 1992. Different ideologies and models of participation are explored in the literature, from community involvement and mobilization to transformational and empowerment models. A look at recent perspectives shows that the fate of programmatic considerations for CH programs tend to rely more on structural factors in health care systems than on cultural factors within local communities. Increased emphasis on power relationships within and between health agencies, decentralized government hierarchies, and various levels of national health care systems are critical for understanding and planning processes. These perspectives raise new questions for CH program design and evaluation. Fine tuning strategies for sustaining community participation by implementing partners and within integrated AWPs is a long term pursuit. Take away: Moving from community-based to community-driven health programs requires more than general consultation and must proactively seek out client groups from marginalized demographics and key populations to understand what meaningful and long lasting engagement means. Balancing issues of culture, confidentially, and socio-economic status calls for sensitivity and skills sets in implementing partner staff to navigate these issues in the heavily Muslim and geographically diverse Senegal context. Underwood, C., Boulay, M., Snetro-Plewman, G., Macwan'gi, M., Vijayaraghavan, J., Namfukwe, M., & Marsh, D., “Community Capacity as Means to Improved Health Practices and an End in Itself: Evidence from a Multi-Stage Study,” International quarterly of CH education, 2012. Community Capacity as Means to Improved Health Practices and an End in Itself: Evidence from a Multi-Stage Study. International quarterly of CH education, 33(2), pp 105-127. Community dialogue and collective action are change processes in which community members take action together to solve a problem, leading not only to a reduction in the community prevalence of a disease but also to a social change that increases the collective capacity of the community to solve new problems. These change processes are not necessarily linear. Because every community and circumstance is different, this process can skip, reorder, or reverse certain aspects of community dialogue or collective action. Finally, cohesion within communities varies greatly. Take away: Adoption of healthy behaviors and positive health-seeking practices can take effect in one on one interaction, support groups or expert patient interventions rather than an overreliance on saturating one community with information, education and communication activities on a single health topic. HIV/AIDS Coutinho, A., Roxo, U., Epino H., Muganzi, A., Dorward, E., and Pick, B., “The Expanding Role of Civil Society in the Global HIV/AIDS Response: What Has President’s Emergency Program For AIDS Relief’s Role Been?” Special Supplement Volume 60:S152 Journal of Acquired Immune Deficiency Syndromes, 2012. Advocacy, activism, serving as government watchdog, and acting as community caretaker, traditional roles of civil society, have been critical to the HIV/AIDS response. In addition, CSOs play an integral part in providing HIV prevention and treatment services and helping to validate continuity of care. The PEPFAR has increased the global scale-up of combination ART reaching for more than 5 million people in developing countries, as well as implementation of effective evidence-based combination prevention approaches. PEPFAR databases in five countries and annual reports from a centrally managed initiative were mined and analyzed to determine the numbers and types of CSOs funded by Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 212 PEPFAR over a 5-year period (2006–2011). Data are also presented from Uganda showing the overall resource growth in CSO working for HIV. Case studies document the evolution of three indigenous CSOs that increased the capacity to implement activities with PEPFAR funding. The growth of civil society to address social and health issues as well as recognition by governments that partnerships with facility and community-based clients and civil society result in better outcomes, is necessary but not sufficient alone for achieving an AIDS-free generation. Take away: Scale-up of the global response can be accelerated by capacity development interventions for civil society (groups as well as influential individuals such as religious leaders, women’s associations, retired nurse midwives, youth activists) and PLWHIV, formation of formal or registered associations, mobilization of funding from diverse sources, and adoption of referrals schemes linking facility and community-based care. Dramé, F., Crawford, E., Diouf, D., Beyrer, C., Baral, S. “A pilot cohort study to assess the feasibility of HIV prevention science research among men who have sex with men in Dakar, Senegal” Journal of the International AIDS Society, Special Supplement 16:18753, 2013. A pilot cohort study was conducted to assess the feasibility of HIV prevention science research among men who have sex with men in Dakar, Senegal. Authors collectively emphasize the idea that providing services specific to MSM can directly benefit the larger public welfare, and provide this promising example led by a community-based partner that targets a young population of Senegalese MSM. These clients not only have a high burden of HIV and rate of incident HIV infections, this population appears to be appropriate for the evaluation of new HIV prevention, treatment and care approaches. Interventions highlighted address three areas: the multiple levels of HIV risk in this setting coupled with risk reduction strategies, sensitization and social protection programming that combat stigma and discrimination, and lastly, the low retention of clients across different phases of the program. Lessons learned from this pilot intervention flag how these issues call for special consideration, and if addressed, will have a higher likelihood of program success. The feasibility of research of larger-scale efficacy trials hinges on overcoming stigma and discrimination and offering a comprehensive package of services that includes targeted structural, behavioral and biomedical components affecting MSM. Kranzer K et al. “Quantifying and addressing losses along the continuum of care for people living with HIV infection in sub-Saharan Africa: a systematic review” Journal of the International AIDS Society, 15:17383, 2012. The authors conducted a systematic review of data from studies in sub-Saharan Africa and published between 2000 and June 2011 on patients’ rates and risk factors for loss to care. A framework of where the break in interaction with the SDP shows where HIV positive persons seek HIV testing, assessment for ART eligibility, ART initiation, retention in pre-ART care until eligible, continuous lifelong ART services in addition to processes explaining where many default or reengage with a service. These observations and findings inform how HIV programs in Senegal can take into account referrals, tracers and other activities that affect the patient’s ability to follow through with health services in the facility and the community. Take Away: Losses occur throughout the care pathway, especially prior to ART initiation, and for some patients this is a transient event, as they may re-engage in care at a later time. However, data regarding interventions to address this issue are scarce. Research is urgently needed to identify effective solutions so that a far greater proportion of infected individuals can benefit from long-term ART. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 213 Meda, N., Ndoye, I., Boup, S., Wade, A., Ndiayee, S., Niang, C., Sarr, F., Diop, I., Caraël, M., “Low and stable HIV infection rates in Senegal: natural course of the epidemic or evidence for success of prevention?” AID, Volume 13, Number 11, 1999. Low and stable HIV infection rates in Senegal: natural course of the epidemic or evidence for success of prevention?” AIDS, Volume 13, Number 11 pp 1397-1405. From a historical analysis of relevant data from 1989- 1996 on HIV and sexually transmission diseases (STDs) epidemiology, sexual behavior, and the efforts in prevention, the review shows how the levels of HIV infection remained stable at around 1.2% in the population of pregnant women, and at 3% in male STD patients. Noted, in contrast, were the increased rates of up to 19% found in female sex workers. A strong political and community commitment led to an early response to the HIV/AIDS epidemic that has been extended since 1986. Blood transfusion safety was established at the start of the HIV epidemic. The level of knowledge of preventive practices relating to HIV/AIDS among the general population exceeded 90% in the early 1990s. From 1991 to 1996, a 66% to 30% decrease of sexually transmitted diseases’ prevalence rates were observed in pregnant women and sex workers in Dakar. In 1997, 33% of men aged 15–49 years in Dakar reported having had sex with non-regular partners. Among them 67% reported condom use. Data from a number of sources do reveal the successfulness of efforts in prevention. From available data, Senegal can rightfully claim to have contained the spread of HIV by intervening early and comprehensively to increase knowledge and awareness of HIV/AIDS and to promote safe sexual behavior in the general population. Further, longitudinal studies of progress to date are encouraged to substantiate or refute the historical perspective on the current HIV/AIDS response, especially taking into account HIV infection rates that are higher in sex workers, men who have sex with men and Senegalese in high risk categories due to factors of poverty and poor geographic access to health centers. PEPFAR. 2013. “Technical Considerations Provided by PEPFAR Technical Working Groups for Fiscal Year 2014 Country Operational Plans and Regional Operational Plans.” Of the 20 different programmatic areas covered in the technical considerations, the guidance flags the importance three updates moving forward in the next phase of the PEPFAR initiative: the Continuum of HIV Response, reinvigorated focus on adolescent girls and key populations, and integration/linkages/wraparounds. The Continuum of HIV Response Framework depicts key features of HIV testing and counseling as the entry point for the HIV response, suggesting a set of pathways to interventions for HIV negative clients in addition to HIV positive clients. Guidance calls for USAID Missions and field teams to prioritize services that represent a mix of combination prevention (population-specific condoms, post-exposure prophylaxis, voluntary male circumcision, blood/injection safety) and core or essential care, support and treatment interventions that reach and retain PLWHA. PMTCT activities and pediatric treatment should be implemented with renewed efforts given the underperformance in respective service delivery indicators globally. A similar mandate for the collection of complete epidemiological data and expansion of interventions targeting the most marginalized and those often missed in the service cascade: girls, men who have sex with men, and children living with HIV. Take away: Overcoming the complex barriers to HIV care calls for a concerted effort to leverage other health and non-health platforms to integrate HIV/AIDS services. Specifically the 2014 technical considerations name HIV and TB, FP, nutrition, and maternal child health as vital opportunities for linkages and wraparound services, health infrastructure, and human resources for health leveraging to better serve clients. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 214 “Tuberculosis and Malaria. Senegal Grant Scorecard,” The Global Fund to Fight AIDS, 2011. Senegal is characterized by a concentrated HIV epidemic with a low prevalence within the general population. The program supported by this grant aims to improve economic and psychosocial care and support for orphans and other vulnerable children and PLWHIV who have been receiving free treatment. Funded activities include the strengthening of HIV and AIDS communication; accelerating implementation of a PMTCT of HIV program; strengthening prevention of HIV sexual transmission within vulnerable groups; improving coordination and management mechanisms for a more effective national response; and strengthening access to voluntary CT services for the general population and vulnerable groups in the regions of Kaolack, Tambacounda, and Louga. The program includes a significant component for strengthening the cross-cutting health system. Take away: As Senegal is classified as a lower middle income country with sex workers, homosexual and bisexual persons disproportionately infected, the HIV response implemented nationally should focus on underserved MARPs and or highest impact interventions. As of 2011, around 54% of the HIV proposal focused on these categories and there is a continual need for improvement in implementation, synergistic donor and implementing partner coordination. HCP Diop, O., “Connectivity in Africa: Use, Benefits and Constraints of Electronic Communications: The Case of Senegal,” United Nations Economic Commission for Africa, 1998. As Part of the Capacity Building for Electronic Communication in Africa (CABECA) project, research was conducted on connectivity issues in West Africa and other regions. The poor telecommunications infrastructure and the prohibitive costs of access for private/individual users were considered to be the main problems in Senegal. The findings point to real limitations in the use of new media that requires electronic technologies. Radio, social events, and word of mouth tend to serve as the most opportune venues for communication in peri-urban and rural settings based on the implications of this analysis, until power issues and connectivity are resolved in Senegal. “Johns Hopkins/ Center for Communication Programs “A Field Guide: Designing and Health Communication Strategy,” 2010. The resource provides guidance for implementers, managers and lay persons planning or executing information education and communication in the conceptualization through strategy implementation phases. Six areas of evidence-based communication programs used in international and domestic public health interventions receive attention: targeted mass media, community-level sensitization and demand-creation activities, interpersonal communication (counselling in clinical settings and in patients’ homes), information and communication technologies, and new media. BCC rooted in contextual factors such as demographics, has proven effective in several health areas, such as increasing the use of FP methods, HIV and AIDS prevention messaging, reducing the spread of malaria and other infectious diseases, and improving newborn and maternal health. Kerr, R. Dakishoni, L., Shumba, L., Msachi R., Chirwa, M., “We Grandmothers Know Plenty: Breastfeeding, Complementary Feeding and the Multi-faceted Role of Grandmothers in Malawi,” Social Science & Medicine Volume 66, Issue 5, 2008. Findings demonstrate that, to address child feeding practices which have an effect on nutrition, attention must be paid to the broader context that influences child nutrition, including extended family relations. Similar too many African countries, paternal grandmothers have a powerful and multifaceted role within the Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 215 extended family in Malawi, both in terms of childcare and in other arenas such as agricultural practices and marital relations. Grandmothers often differ in their ideas about early child feeding from conventional Western medicine. Despite the important integrated role older women have within households and communities, they are underutilized programmatically in design and implementation schemes and attitudes toward ‘grannies’ and their knowledge can be negative. Health education and promotion practitioners who involve grandmothers have the potential to move efforts further when these perspectives are taken into consideration. Take away: Authors caution those that view ‘traditional knowledge’ as backward, using their research findings to show how grandmothers view current child health conditions within a broader context of changing livelihood conditions and a high prevalence of HIV/AIDS. The paper concludes by discussing the challenges of involving grandmothers in health education, and the difficulties of incorporating local knowledge into a medical system that largely rejects it. Inferred from the findings are the positive contributions of influential persons in the community and households such as grandmothers, religious leaders, traditional healers which need to be better utilized in health program design and implementation. Meekers D., Rahaim S., “The importance of socio-economic context for social marketing models for improving reproductive health: Evidence from 555 years of program experience.” Bio Med Central Public Health, 2005. For effectiveness of future social marketing programs, it is essential that more effort is devoted to ensuring that such programs consider the local context. The three different management structures common in social marketing programs were compared: management by an affiliate of an international NGO, management by local clinic based or non￾clinic-based organizations, and partnerships with a commercial organization. Findings from the review explain that NGOs remain the dominate implementer of social marketing interventions in highly urban settings. Take away: Program maturity and the size of the target population appear equally important, to guide implementation decisions for social marketing programming. In some cases, tailoring the sales of the health product to the needs of clients depending on key characteristics of the population and commercial infrastructure are possibly more impactful than the role of the social marketing program management approaches dominated by NGOs. There is a need for deeper involvement by the public and for profit sectors to diversify and compete in this area. This in turn gives communities a richer landscape of reproductive health information, pricing options and quality of products across different socio-economic demographics. “Mhealth New Horizons for Health through Mobile Technology,” WHO, 2011. Short message service (SMS) sent with health information as interventions have proven effective in disseminating information such as appointment reminders, information about mobile or free services, and health events (e.g. national immunization days, HIV/AIDS counselling and testing days and community mobilization activities). There are few studies that provide evidence of the impact of such programs on behavior change which implies an opportunity for further study of the most appropriate use of mobile phones for health promotion. Take away: A few areas of concern requiring special consideration in the use of text messaging for health communication are the potential challenges with protecting confidential health information and the potential for unintended disclosure or discrimination when messages are viewed by the wrong person. This is particularly a possibility in the case of conditions like HIV/AIDS, which remain highly stigmatized. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 216 Annex G: Bibliography # Title of Document Component #1 - Abt 1 "Composante Renforcement des Systèmes de Sante, Plan D’Action Annuel Octobre 2011- Septembre 2012 De la Composante Renforcement des Systèmes de Santé," Abt, December 30, 2011. 2 "Composante Renforcement du Système de Sante, Draft – Rapport Annuel de la Période Octobre 2012- Septembre 2013," Abt, November 7, 2013. 3 "Composante Renforcement du Système de Sante, Plan D’Action Annuel Octobre 2012- Septembre 2013, De la composante renforcement du système de sante," Abt, August 30, 2012. 4 "Composante Renforcement du Système de Sante, Plan D’Action Annuel Octobre 2013, Septembre 2014, De la composante renforcement du système de sante," Abt, September 24, 2013. 5 "Composante Renforcement du Système de Sante, Rapport Annuel de la Période Octobre 2011-Septembre 2012, " Abt, December 2012. 6 "Health System Strengthening Component, December 2012, Annual Report-October 2011 to September 2012, " Abt. 7 "Health System Strengthening Component, Draft of November 7, 2013, Annual Report￾October 2012 to September," Abt, November 7, 2013. 8 "Health System Strengthening Program Component (HSS) Cooperative Agreement, " Abt, September 27, 2011. 9 "Programme Sante 2011-2016 Rapport Trimestriel de la Composante Renforcement du Système de Sante: Octobre-Décembre 2013," Abt, January 27, 2014. 10 "Programme Sante 2011- 2016 Rapport Trimestriel de la Composante Renforcement de Système de Sante: Janvier – Mars 2014," Abt, April 2014. 11 "Programme Sante 2011- 2016 Rapport Trimestriel de la Composante Renforcement de Système de Sante: Avril- Juin 2014," Abt, April 2014. Component #2 – IntraHealth 12 "Composante Renforcement Des Prestations de Services, Plan D’Action Annuel 1er Octobre 2011- 30 Septembre 2012," IntraHealth. 13 "Composante Renforcement Des Prestations de Services, Plan D’Action Annuel 1er Octobre 2012- 30 Septembre 2013," IntraHealth. 14 "Composante Renforcement Des Prestations de Services, Plan D’Action Annuel 1er Octobre 2013- 30 Septembre 2014," IntraHealth. 15 "Composante Renforcement Des Prestations de Services, Plan de Monitoring des Performances 1er Octobre 2011-30 Septembre 2012," IntraHealth. 16 "Composante Renforcement Des Prestations de Services, Rapport Annuel D’Activités et Financier 1er Octobre 2011 – 30 Septembre 2012," IntraHealth. 17 "Composante Renforcement Des Prestations de Services, Rapport Annuel D’Activités 1er Octobre 2012-30 Septembre 2013," IntraHealth. 18 "Composante Renforcement Des Prestations de Services, Rapport Trimestriel D’Activités 1er Octobre – 31 Décembre 2013," IntraHealth. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 217 19 "Composante Renforcement Des Prestations de Services, Rapport Trimestriel D’Activités 1er Janvier – 31 Mars 2014," IntraHealth. 20 "Composante Renforcement des Prestations de Services, Rapport Trimestriel D’Activités 1er Avril -30 Juin 2014," IntraHealth. 21 "Health Services Strengthening Project Annual Work Plan, October 1, 2011- September 30, 2012,” IntraHealth. 22 "IntraHealth Health Service Improvement Program Component (HSI) Cooperative Agreement, " USAID/Senegal, September 19, 2011. Component #3 - ChildFund consortium 23 "CH Program Component Cooperative Agreement, " USAID/Senegal, September 28, 2011. 24 "CHP Work Plan Year 1,” ChildFund consortium. 25 "CHP Work Plan Year 2,” ChildFund consortium. 26 "Draft Performance Management Plan (Revised PMP) Health Outcomes, " ChildFund consortium, October 25, 2011. 27 "Expérience Acquise de Programme de Sante Communautaire ChildFund/USAID Senegal- Evaluation Final Programme de Sante Communautaire du Sénégal/ChildFund 2006-2011," USAID/Senegal, May 2011. 28 "Fiche de synthèse de suivi du transfert des cases. " 29 "Grants Financial Management Guide,” ChildFund consortium, March 2014. 30 "Grants Management and Compliance Guide,” ChildFund consortium, March 2014. 31 "Lessons Learned from the USAID/Senegal Community Health Program Final Evaluation of CHP, as implemented by ChildFund 2006-2011,” USAID /Senegal, May 2011. 32 "Matrice D’Autorisation D’Achat De Biens et services," ChildFund consortium. 33 "Procedure, " ChildFund consortium. 34 "Programme Sante USAID, Sante Communautaire Phrase II, Plan D’Action An 3 Octobre 2013- Septembre 2014," ChildFund consortium, September 1, 2013. 35 "Programme Santé USAID, Sante Communautaire Phase II, Plan D’Action An 4, Octobre 2014-Septembre 2015, " ChildFund consortium, October 2014. 36 "Programme Sante USAID, Sante Communautaire Phase II, Rapport Annuel des activités Octobre 2013-Septembre 2014," ChildFund consortium, October 2014. 37 "Programme Sante USAID, Sante Communautaire Phrase II, Rapport Annuel des Activités Octobre 2012- Septembre 2013 (An 2)," ChildFund consortium, November 2013. 38 "Programme Sante USAID, Sante Communautaire Phrase II- Rapport Trimestriel des Activités Octobre-Novembre-Décembre 2013," ChildFund consortium, January 2014. 39 "Programme Sante USAID, Sante Communautaire Phrase II, Rapport Trimestriel des Activités An 1 Oct. 2011-Sept. 2012,” ChildFund consortium. 40 "Programme Sante USAID, Sante Communautaire Phrase II, Rapport Trimestriel des Activités (Janvier – Février – Mars 2014)," ChildFund consortium, April 2014. 41 "Programme Sante USAID, Sante Communautaire Phrase II, Rapport Trimestriel des Activités (Avril-Mai-Juin 2014)," ChildFund consortium, July 2014. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 218 42 "Stratégie de Transfert de L’Encadrement des Cases de Sante aux Communautés et Districts Sanitaires: Guide Méthodologique," ChildFund consortium, September 2012. 43 "Sub Award Management Guidelines and Tools," ChildFund consortium, March 2014. 44 "USAID Health Program, Annual Activities report, October 2011-September 2012, Final Version," ChildFund consortium. 45 “USAID Health Program, CH Phase II, Annual activities report, October 12-September 2013(year2)," ChildFund consortium, November 2013. Component #4 - FHI360 46 “Composante Lutte contre le VIH et la Tuberculose, Rapport Annuel, FY 2014 Oct 2013- Sept 2014," FHI360. 47 "Composante Lutte contre le VIH et la Tuberculose, Rapport Trimestriel Janvier-Mars 2014," FHI 360. 48 "Composante Lutte contre le VIH et la Tuberculose, Rapport Trimestriel Avril-Juin 2014,”FHI 360. 49 "Composante Lutte contre le VIH et la Tuberculose Rapport Trimestriel Spécifique, Janvier-Mars 2014," FHI 360. 50 "Composante Lutte contre le VIH et la Tuberculose Rapport Trimestriel Spécifique, Avril- Juin 2014," FHI 360. 51 "Evaluation à Mi-Parcours des Programmes VHI/SIDA et TB de L’USAID/Sénégal, " USAID/Senegal, June 2010. 52 "FHI HIV PMP," FHI 360. 53 "FHI HIV Reported Indicators and Targets Year 2," FHI 360. 54 "Health HIV/AIDS and TB Program Component Cooperative Agreement," USAID/Senegal, September 27, 2011. 55 "HIV-AIDS & TB Component- Jan-March 2013 Quarterly Report," FHI 360. 56 "Programme de Sante de L’USAID Composante VIH/SIDA & Tuberculose, Plan D’Action FY 2012," FHI 360. 57 "Programme de Sante de L’USAID Composante VIH/SIDA & Tuberculose, Plan D’Action FY 2013," FHI 360, January 15, 2013. 58 "Programme de Sante de L’USAID Composante VIH/SIDA & Tuberculose, Plan D’Action FY 2014," FHI 360, Octobre 30, 2014. 59 "Programme de Sante de L’USAID Composante VIH/SIDA & Tuberculose, Rapport Annuel FY 2012 Annexes," FHI 360, November 2012. 60 "Programme de Sante de L’USAID Composante VIH/SIDA & Tuberculose, Rapport Annuel FY 2012," FHI 360, November 2012. 61 "Programme De Sante De L'USAID Composante VIH/SIDA et Tuberculose, Rapport Annuel FY 2013, " FHI 360, January 2014. 62 "Quarterly Report. Q1FY2012," FHI 360, January 2012. 63 "Rapport Trimestriel Q1-FY14 Octobre - Novembre-Décembre 2013," FHI 360. 64 "USAID HIV-AIDS & TB Component - FY2012 Annual Report," FHI 360. 65 "USAID HIV-AIDS TB COMPONENT - April-June 2012 Quarterly Report," FHI 360. 66 "USAID HIV-AIDS TB COMPONENT - Jan-Mar 2012 Quarterly Report," FHI 360. 67 "USAID HIV-AIDS TB COMPONENT - Oct-Dec 2012 Quarterly Report," FHI 360. 68 "USAID Indicators and Target FY2014," FHI 360. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 219 69 "USAID/Senegal HIV/AIDS and Tuberculosis Programs Interim Assessment," USAID/Senegal, June 2010. Component #5 – ADEMAS 70 "ADEMAS Plan Stratégique 2014-2019," ADEMAS, July 30, 2014. 71 "Agence pour le Développement du Marketing Social," ADEMAS. 72 "Health Communication and Promotion Program Component (HCP) Cooperative Agreement," USAID/Senegal, March 1, 2012. 73 "Performance Monitoring Plan (PMP)/ Results Framework for Health Communication and Promotion Program in Senegal," ADEMAS, February 4, 2013. 74 "Plan D’Action Annuel," ADEMAS. 75 "Programme Sante USAID 2011-2016 Composante Communication et Promotion de la Sante, Plan D’Actions 2012," ADEMAS. 76 "Programme Sante USAID 2011-2016 Composante Communication et Promotion de la Sante, Plan D’Actions 2012-2013," ADEMAS. 77 "Programme Sante USAID 2011-2016 Composante Communication et Promotion de la Sante, Plan D’Actions 2013-2014," ADEMAS. 78 "Programme Sante USAID Senegal Composante communication et promotion de la santé, Rapport Annuel D’Activités Mars-Septembre 2012," ADEMAS, October 2012. 79 "Programme Sante USAID Senegal Composante communication et promotion de la santé Rapport Annuel D’Activités, Octobre 2012-Septembre 2013," ADEMAS. 80 "Programme Sante USAID Senegal Composante communication et promotion de la santé, Rapport Trimestriel D’Activités Octobre-Décembre 2013," ADEMAS, January 2014. 81 "Programme Sante USAID Senegal 2011-2016, Composante Communication et Promotion de Sante, Rapport Annuel D’Activités Octobre 2013-Septembre 2014," ADEMAS, February 11, 2014. 82 "Programme Sante USAID Senegal 2011-2016, Composante Communication et Promotion de la Sante, Rapport Trimestriel D’Activités Janvier – Mars 2014," ADEMAS, April 2014. 83 "Programme Sante USAID Senegal 2011-2016, Composante Communication et Promotion de la Sante, Rapport Trimestriel D’Activités Avril- Juin 2014," ADEMAS, July 31, 2014. 84 "Résumé des réalisations majeurs en 2013 et des actions prioritaires en 2014," ADEMAS. 85 "Tableau de Suivi PMP Juillet 2014-Septembre 2014,” ADEMAS, 2014. 86 "USAID Senegal Health Program, Health Communication and Promotion Component, Annual Activity Report, March-September 2012,” ADEMAS, October 19, 2012. 87 "USAID Senegal Health Program, Health Communication and Promotion Component, Annual Activity Report, October 2012-September 2013," ADEMAS. Additional Sources 88 "Abuja Declaration: Ten Years On," WHO, 2011. 89 Amanyeiwe, U., Leclerc-Madlala, S and Gardi, H., "Do Community-Based Programs Help to Improve HIV Treatment and Health Outcomes? A Review of the Literature," World Journal of AIDS, 2014. 90 "Cadre Stratégique National de Lutte Contre le Paludisme au Sénégal 2014-2018," Ministère de la Sante et de L’Action Sociale, March 2014. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 220 91 "Contracting Officer’s Representative Designation – Contract Administration, " USAID/Senegal, September 9, 2012. 92 "Copie de PAI FY 2014.” 93 "Copie de Plan d’action Intègre Programme Sante." 94 Coutinho, A., Roxo, U., Epino H., Muganzi, A., Dorward, E., and Pick, B., "The Expanding Role of Civil Society in the Global HIV/AIDS Response: What Has President’s Emergency Program For AIDS Relief’s Role Been?" Special Supplement Volume 60:S152 Journal of Acquired Immune Deficiency Syndromes, 2012. 95 Daff B., Seck C., Belkhayat H., Sutton P. "‘Informed push distribution of contraceptives in Senegal reduces stockouts and improves quality of FP services,’ Global Health: Science and Practice, Volume 2, Number 2," 2014. 96 Diop, O., "Connectivity in Africa: Use, Benefits and Constraints of Electronic Communications: The Case of Senegal," United Nations Economic Commission for Africa, 1998. 97 "Direct Financing Procedures Manual, USAID," USAID/Senegal, February 2013. 98 Dramé, F., Crawford, E., Diouf, D., Beyrer, C., Baral, S. Journal of the International AIDS Society, Special Supplement 16:18753, 2013. 99 "Epidemiological Fact Sheets on HIV/AIDS and Sexually Transmitted Infection," WHO, 2004. 100 "Evaluation de l’impact de la mise a échelle des interventions de lutte contre le paludisme au Sénégal 2005-2010," Programme National de lutte le Paludisme, December 10, 2013. 101 "Evidence to Action Compendium of Best Practices," 2013. 102 "Finance Questions," October 27. 103 Fritsche, G., and Rusa, L. "Rwanda: Performance-Based Financing in Health," Sourcebook: Second Edition. 104 Heiby, J. "The use of modern quality improvement approaches to strengthen Africa Health systems: a 5-year agenda," Journal for Quality in Healthcare, 2014. 105 Hodgins, S., Pullum, T. and Dougherty, L., "Understanding where parents take their sick children and why it matters: a multi-country analysis," Global Health: Science and Practice, 2013. 106 "Integrated Community Case Management of Childhood Illness: Documentation of Best Practices and Bottlenecks to Program Implementation in Senegal," Maternal and Child Health Integrated Program, 2012. 107 "Johns Hopkins/ Center for Communication Programs A Field Guide: Design and Health Communication Strategy," John Hopkins, 2010. 108 Jutting J., "Do Community-based Health Insurance Schemes Improve Poor People’s Access to Healthcare? Evidence From Rural Senegal," World Development Vol. 32, Number 2, 2003. 109 Kerr, R. Dakishoni, L., Shumba, L., Msachi R., Chirwa, M., "We Grandmothers Know Plenty: Breastfeeding, Complementary Feeding and the Mulit-faceted Role of Grandmothers in Malawi," Social Science & Medicine Volume 66, Issue 5, 2008. 110 Keugoung, B., Macq, J., Buve A., Meli, J. and Criel, B. "The interface between health systems and vertical programmes in Francophone Africa: the managers’ perceptions," Volume 16 Number 4, 2011. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 221 111 Kranzer K et al. " Quantifying and addressing losses along the continuum of care for people living with HIV infection in sub-Saharan Africa: a systematic review" Journal of the International AIDS Society, 15:17383, 2012. 112 Lemiere, C., "Senegal Health & Social Financing: P129472 - Implementation Status Results Report: Sequence 02." Washington, DC: The World Bank, 2014. 113 Lillie, Tiffany, "USAID/Senegal: Social Marketing Program Performance Evaluation," Global Technical Assistance Bridge Project, June 2012. 114 "Malaria Operational Plan Fiscal Year 2013," President’s Malaria Initiative. 115 "Malaria Operational Plan Fiscal Year 2014," President’s Malaria Initiative. 116 "Malaria Operational Plan Year Five - Fiscal Year 2011," President’s Malaria Initiative. 117 "Malaria Operational Plan Year Six - Fiscal Year 2012," President’s Malaria Initiative, November 11, 2011. 118 Massoud, R., Mensah-Abrampah, N., Barker, P., Letherman, S., Kelley, E., Agins, B., Sax, Sylvia, and Heiby, J. "Improving the delivery of safe and effective healthcare in low and middle income countries: Research is needed into creating workable systems that can deliver and sustain interventions, " BMJ 344: e 981, 2012. 119 Meda, N., Ndoye, I., Boup, S., Wade, A., Ndiayee, S., Niang, C., Sarr, F., Diop, I., Caraël, M., "Low and stable HIV infection rates in Senegal: natural course of the epidemic or evidence for success of prevention?" AID, Volume 13, Number 11, 1999. 120 Meekers D., Rahaim S., "The importance of socio-economic context for social marketing models for improving Reproductive Health: Evidence from 555 years of program experience," Bio Med Central Public Health, 2005. 121 "Mhealth New Horizons for Health through Mobile Technology," WHO, 2011. 122 "Pilot Project: Performance-Based Financing (PBF) in the health sector," Republic of Senegal, March 2012. 123 "Plan d’Activités Intègre – Résumé des Activités de FD." 124 "Plan National de Développement Sanitaire (2009-2018)," Ministry of Health. 125 "Plan National Stratégique de Sante Communautaire 2014-2018," République du Sénégal, February 2014. 126 "Plan Stratégique de Lutte Contre le Sida 2014-2017," République de Sénégal. 127 "Program Description for the Senegal Health Communication and Promotion Component," USAID/Senegal. 128 Rusa, L. Schneidman M., Fritsche G., and Musango L. "Rwanda: Performance-Based Financing in the Public Sector," Center for Global Development. 129 "Senegal, Continuous Survey Year One : 2012-2013, Key Findings," Agence Nationale de la Statistique et de la Démographie (ANSD) and ICF International, 2014. 130 "Senegal Country Development Cooperation Strategy 2012-2016," USAID/Senegal, February 2012. 131 "Senegal Demographic and Health Survey and Multiple Indicator Cluster Survey (EDS￾MICS) 2010-2011," Agence Nationale de la Statistique et de la Démographie (ANSD) and ICF International, 2012. 132 "Senegal Health Profile," WHO, 2012. 133 "Senegal Programme Sante 2011-2016, Plan D’Action Intègre FY 2013," USAID/Senegal, February 18, 2013. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 222 134 "Senegal Programme Sante 2011-2016, Plan D’Action Intègre FY 2014," USAID/Senegal, December 9, 2013. 135 Stone, L., "Cultural influences in Community Participation in Health." Comparative Study Review, Social Science and Medicine, 1992. 136 "Technical Considerations Provided by PEPFAR Technical Working Groups for Fiscal Year 2014 Country Operational Plans and Regional Operational Plans," PEPFAR, 2013. 137 "The Bamako Initiative," UNICEF. 138 "The United States Health Strategy Senegal Global Health Initiative Strategy," USAID/Senegal. 139 "Tuberculosis and Malaria. Senegal Grant Scorecard," The Global Fund to Fight AIDS, 2011. 140 Underwood, C., Boulay, M., Snetro-Plewman, G., Macwan'gi, M., Vijayaraghavan, J., Namfukwe, M., & Marsh, D., "Community Capacity as Means to Improved Health Practices and an End in Itself: Evidence from a Multi-Stage Study," International Quarterly of CH Education, 2012. 141 "USAID/Senegal 2011-2016 Health Program, Inter-Agency Working Group on Financing," USAID/Senegal. 142 "USAID/Senegal Healthcare Financing and Policy Project- Midterm Evaluation," USAID/Senegal, December 2009. 143 Walshe, K. "Psuedoinnovation: the development and spread of healthcare quality improvement methodologies," International Journal for Quality in Healthcare Volume 21, Number 3, 2009. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 223 Annex H: Implementing Partner Survey IP POC information POC name: POC job position: POC contact information (email, telephone number) Survey submission date: IP Survey # Question Scale 1-5 Program Design & Implementation 1 USAID’s regional decentralized health program implementation structure has been effective in helping achieve your specific project goals and objectives. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 2 Program interventions are coordinated and integrated in synergy across components, with other USAID/Senegal health programs. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 3 Program interventions are coordinated and implemented in synergy with other non-U.S Government donors and partners (e.g., The World Bank, GFATM). 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 4 Project interventions are functioning effectively to strengthen the GOS ownership at the national level. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 5 Project interventions are functioning effectively to strengthen the GOS ownership at the decentralized level (i.e., regional and district). 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: Component Specific Questions 6 Your project components have achieved their objectives according to the planned timeline. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 7 M&E activities support the use of data for decision-making of your component. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 224 IP Survey # Question Scale 1-5 8 Gender considerations are integrated in the design and implementation of your component. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 9 Your component interventions have been effective. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: Financial Management 10 Cost-savings are being achieved through the use of USAID’s integrated programming. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 11 Cost-savings were achieved through the most efficient (e.g., timing, resources used) and effective (e.g., impact, outcome achieved) means. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: 12 DF to the three regions has improved your project’s ability to successfully implement interventions. 1. Strongly agree 2. Agree 3. Neutral 4. Disagree 5. Strongly disagree Additional notes on scale: Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 225 Annex I: In-Depth Interview Guides AOR/IP Code Question Answer 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Interviewee 4a) Function 4b) Telephone 4c) Email address 5) Team EY Team A/Team B/Team C 6) Region 7) Month/Year the project started 8) Analysis period: Starting (M/Y) to End (M/Y) 9) COR (if different from AOR) 10) Location of Head Office of IP 11) Key stakeholders 12) Other notes 13) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinics, Beneficiaries (Individual), Hospitals, Pharmacies, Private Firms) 14) Component (1, 2, 3, 4, 5, N/A) Code Question Answer a1_1 How effective has the structure of USAID/Senegal’s overall health program and the division of the program into five components been in helping achieve the health DO? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 226 Code Question Answer a1_2 Did it help the project reach Senegal's health objectives? Yes/No a1_2a Please explain. a1_3 Strengths? Weaknesses? Ideas to improve the project? a2_1 What work synergy and monitoring plan were put in place in each component of the USAID/Senegal Health Program? a3_1 What was the USAID/Senegal program trying to improve by setting up three Regional Offices for DF? a3_2 What about the finance based results? a3_3 What do you think of DF? Of finance based results? Of the social health insurance (with state financing) in the first period of the program? a3_4 Strengths? Weaknesses? a3_5 What would you suggest to improve the funding mode? a4_1 What mechanisms were used to set up the three USAID/Senegal Regional Offices? a4_2 Did they succeed in integrating, supervising and coordinating the five components? a4_3 Strengths? Weaknesses? a4_4 What are the proposed improvement systems in the coordination? a5_1 Does the technical and financial support provided support recipient ownership? a5_2 In terms of sustainability what are strengths and weaknesses? a5_3 What is the proposed improvement in this transfer of ownership? b1_1 Do you think that the objectives assigned to the component contribute to the achievement of the objectives of the USAID health program in Senegal? Yes/No b1_1a Please explain. b1_2 On the basis of which advantage (in terms of objectives, mission and/or programs) your organization was chosen for the component (HSS, HSI, CH, HIV/AIDS, CHP) of the Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 227 Code Question Answer USAID/Senegal Health Program? b2_1 Which component have you collaborated with? How successful has it been so far? b2_2 What support (e.g., capacity development, equipment) have you received from the component? b2_3 Which activities have you implemented in collaborating with the program component? b2_4 What is the main reason for this success (in terms of services, integration of AWPs, financing system, supervision, monitoring and coordination)? b3_1 Which weaknesses/challenges (in terms of capacity building, organization, integration of AWPs, financing system, supervision, monitoring and coordination) have you noticed in the first period of the program management? b3_2 Did the implementing partners provide specific answers to the components and/or the program? Yes/No b3_2a Please explain. b4_1 Do you think that one or more components should be added? b4_2 What would justify those changes in order to boost the program? b5_1 Were gender issues considered during the program component development? b5_2 What about during the implementation, monitoring, supervision of the component? b5_3 How have men and women benefited equitably to the services provided by the program component? b5_4 How could the integration of gender equity be improved in the program components? MOH – Component #1: HSS Code Question Answer Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 228 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Person Interviewed 4a) Function 4b) Telephone 4c) Email Address 5) EY Team Team A/Team B/Team C 6) Date of Submission of the Survey 7) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinique, Beneficiary(Individual), Hospital, Pharmacy, Private Firm) 8) Component 1 Code Question Answer moh_com1_1 What was your role in the USAID/Senegal Health Program regarding the HSS component managed by Abt? moh_com1_2 How useful was the USAID HSS Program for your department? moh_com1_3 In your opinion, how has the USAID/Senegal Health Program contributed to medical staffs training at the District and Regional level? moh_com1_3a Tell us about DF? Or PBF? moh_com1_3b Tell us about UHC? And social health insurance? moh_com1_3c Tell us about the National health policy committee and the policy reforms. moh_com1_3d Tell us about the capacity building in pharmaceutical management? moh_com1_3e Tell us about the integration health program action plans? moh_com1_3f Tell us about the support to reinforce the collaboration between USAID/Senegal Regional Offices? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 229 Code Question Answer moh_com1_4 To what extent does the USAID/Senegal Health Program contribute to medical staff training at the District and Regional level? moh_com1_5 Were there any challenges in the implementation of medical staff training programs at the District and Regional level? Yes/No moh_com1_5a What did you do to overcome that? moh_com1_5b DF? PBF? moh_com1_5c UHC? Social health insurance? moh_com1_5d The national health policy committee? The health sector reform? moh_com1_5e The reinforcement of the pharmaceutical management? moh_com1_5f The integration of health program action plans? moh_com1_5g The support for the collaboration between regional office? moh_com1_6 How has the USAID Health Program contributed to the reinforcement of monitoring, supervising and coordinating of the medical staff training program at district and regional level? moh_com1_6a DF? Finance Based Result? moh_com1_6b UHC? Social Health Insurance? moh_com1_6c The National Health Policy Committee? The Health Sector reform? moh_com1_6d The reinforcement of the pharmaceutical management? moh_com1_6e The integration of health program action plans? moh_com1_6f Support of the collaboration between regional office? moh_com1_7 What were the main challenges and constraints during the implementation of this component? moh_com1_7a Which regions were most challenging for implementing the HSS? moh_com1_7b Do you have any other suggestions? moh_com1_7c What changes has the program brought to your department? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 230 Code Question Answer moh_com1_7d What did you do to overcome them? moh_com1_7e Do you have comments or suggestions? MOH – Component #2: HSI Code Question Answer 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Person Interviewed 4a) Function 4b) Telephone 4c) Email Address 5) EY Team Team A/Team B/Team C 6) Date of Interview 7) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinique, Beneficiaries (Individuals), Hospital, Pharmacy, Private Firm) 8) Component 2 Code Question Answer moh_com2_1 What is your role in the implementation of the USAID/Senegal Health Program regarding the HSI component by IntraHealth? moh_com2_2 Please tell us how useful the USAID/Senegal Health Program was for your department? moh_com2_3 In your opinion, how has the USAID/Senegal Health Program contributed to the integrated package of service quality (e.g., maternal and infant health/full EMOC, malaria, child survival and nutrition, and vaccination campaigns)? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 231 Code Question Answer moh_com2_3a To put in place the equipment and supplies of health posts/centers and health huts, provision in contraceptives? moh_com2_3b The reinforcement of referral systems, evaluation, and performance techniques? moh_com2_3c The establishment of public-private partnerships, TutoratPlus, private agents, and private pharmacies? moh_com2_3d The management and the development of District Leadership Team (ECD) human resources? moh_com2_3e The mapping of RH and the development of recruitment procedures? moh_com2_4 To what extent has the USAID/Senegal Health Program contributed to building the capacity of actors at national level for the integrated package of services, malaria, child survival, nutrition, and vaccination campaigns? moh_com2_4a To put in place the equipment and supplies for health posts, health huts, health centers, and provision with contraceptives? moh_com2_4b To the reinforcement of referral systems and performance evaluation techniques? moh_com2_4c To the public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_4d To the human resource management and capacity building of ECD? moh_com2_4e To the mapping of human resources and the development of recruitment procedures? moh_com2_5 Please explain the progress made in the reinforcement of the integrated package of services, malaria, child survival, nutrition, and vaccination campaigns? moh_com2_5a Please explain. moh_com2_5b In your opinion, what are the results of the quality of the integrated health services package? moh_com2_5c To put in place the equipment and supplies for the health posts, health huts, health centers, and provision with contraceptives? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 232 Code Question Answer moh_com2_5d To the reinforcement of referral systems, performance evaluation techniques? moh_com2_5e To establish public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_5f To manage human resource and build the capacity of ECD? moh_com2_5g To organize the mapping of human resources and the development of recruitment procedures? moh_com2_6 How has the USAID/Senegal Health Program contributed to the reinforcement of staff capacity at a national level to provide quality services in the integrated health package? moh_com2_6a To put in place the equipment and supplies for the health posts, health huts, health centers, and provision with contraceptives? moh_com2_6b To reinforce referral systems and performance evaluation techniques? moh_com2_6c To establish public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_6d To manage human resource and build the capacity of ECD? moh_com2_6e To organize the mapping of human resources and the development of recruitment procedures? moh_com2_7 Have you had any challenges in putting into place the capacity building program for the integrated health package for quality services? Yes/No moh_com2_7a What did you do to overcome the challenges? moh_com2_7b To put in place the equipment and supplies for the health posts, health huts, health centers, and provision with contraceptives? moh_com2_7c To reinforce referral systems and performance evaluation techniques? moh_com2_7d To establish public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_7e To manage human resource and build the capacity of ECD? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 233 Code Question Answer moh_com2_7f To organize the mapping of human resources and the development of recruitment procedures? moh_com2_8 How has the USAID health program contributed to the monitoring, the supervision, and the coordination of the integrated PAQ? moh_com2_8a To put in place the equipment and supplies for the health posts, health huts, health centers, and provision with contraceptives? moh_com2_8b To reinforce referral systems and performance evaluation techniques? moh_com2_8c To establish public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_8d To manage human resource and build the capacity of ECD? moh_com2_8e To organize the mapping of human resources and the development of recruitment procedures? moh_com2_9 Did the equipment and furniture received from the USAID health program contribute to the reinforcement of services in the public health centers in charge of the integrated package of health services quality? moh_com2_9a To put in place the equipment and supplies for the health posts, health huts, health centers, and provision with contraceptives? moh_com2_9b To reinforce referral systems and performance evaluation techniques? moh_com2_9c To establish public-private linkages, TutoratPlus to agents from private sector and pharmacists? moh_com2_9d To manage human resource and build the capacity of ECD? moh_com2_9e To organize the mapping of human resources and the development of recruitment procedures? moh_com2_10 What were the main challenges you were confronted with in the implementation of this component? How did you overcome them? moh_com2_11 In the regions where the component health service Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 234 Code Question Answer improvement were implemented, which regions have been the most challenging? moh_com2_12 Do you have any other comments and suggestions? MOH – Component #3: CH Code Question Answer 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Interview 4a) Function/Position/Occupation 4b) Telephone: 4c) Email Address: 5) Names of Interviewer teams Team A/Team B/Team C 6) Date of submission of the survey 7) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinic, Beneficiaries (Individual), Hospital, Pharmacy, Private Firm) 8) Component 3 Code Question Answer moh_com3_1 What was your role in the implementation of the USAID/Senegal Health Program on the CH component, managed by the consortium led by ChildFund consortium performance organizations? moh_com3_2 Why was the USAID/Senegal Health Program useful for your department? moh_com3_3 In your opinion, how has the USAID Health Program contributed to the extension of the integrated quality services in the health hut package? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 235 Code Question Answer moh_com3_3a Child survival and nutrition? moh_com3_3b Capacity development of CBOs? moh_com3_3c The provision of oral and injectable contraceptives to the health huts? moh_com3_3d The plan to transfer the management of health hut in the MOH? moh_com3_4 To what extent has the USAID/Senegal Health Program helped to build capacity at the national level to the extension of the integrated quality services in the health hut package? moh_com3_4a Child survival and nutrition? moh_com3_4b Capacity development of CBOs? moh_com3_4c The provision of oral and injectable contraceptives to the health case? moh_com3_4d The plan to transfer the management of health case in MSAS? moh_com3_5 Please focus on the progress made in the programming of the extension of the integrated quality health services at health hut package? Yes/No moh_com3_5a Explain. moh_com3_5b Child survival and nutrition? moh_com3_5c Capacity development of the CBOs? moh_com3_5d The provision of oral and injectable contraceptives to the health case? moh_com3_5e The plan to transfer the management of health case in MOH? moh_com3_6 Have you encountered challenges in the implementation of the extension of integrated quality health services in the health hut package? Yes/No moh_com3_6a What did you do to overcome them? moh_com3_6b Child survival and nutrition? moh_com3_6c Capacity development of the CBOs? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 236 Code Question Answer moh_com3_6d The provision of oral and injectable contraceptives to the health case? moh_com3_6e The plan to transfer the management of health case in MOH? moh_com3_7 How has the USAID Health Program helped to strengthen the monitoring, supervision, and coordination of the extension of integrated quality health services at health hut package? moh_com3_7a Child survival and nutrition? moh_com3_7b Capacity development of the CBOs? moh_com3_7c Supply of oral and injectable contraceptives to the health case? moh_com3_7d The plan to transfer the management of health hut in MOH? moh_com3_8 What were the main challenges and constraints that you faced in the implementation of health community component? What did you do to overcome them? moh_com3_9 Which regions was the development work of the CH component causing the most problems? moh_com3_10 Do you have any comments or suggestions? MOH – Component #4: HIV/AIDS Code Question Answer 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Person Interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email Address 5) Names of EY Interviewer Teams Team A/Team B/Team C Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 237 Code Question Answer 6) Date of submission of the Survey 7) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinic, Beneficiaries (Individual), Hospital, Pharmacy, Private Firm) 8) Component 4 Code Question Answer moh_com4_1 What was your role in the implementation of the USAID/Senegal Health Program on HIV/AIDS and STIs, managed by FHI 360? moh_com4_2 How was the USAID Health Program useful for your department? moh_com4_3 In your opinion, how has the USAID Health Program helped the Communication for Behavior Change (CBC) for HIV prevention, diagnosis and treatment of STIs, and ongoing support in the form of advice, screening, and prevention in groups at high risk? moh_com4_4 To what extent has the USAID/Senegal Health Program helped to strengthen the entire processing services, health care, and economic and nutritional support to PLWHIV? moh_com4_5 Please focus on the progress made in the programming of CT among high-risk groups and treatment of STIs/HIV? Yes/No moh_com4_5a Please explain. moh_com4_6 How has the USAID/Senegal Health Program helped to strengthen national capacity to program CT among high-risk groups and treatment of STIs/HIV? moh_com4_7 Have you encountered challenges in the implementation of STI and HIV programs? Yes/No moh_com4_7a What did you do to overcome them? moh_com4_8 How has the USAID/Senegal Health Program helped to strengthen the monitoring, supervising, and coordinating of the National Program for HIV and STIs? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 238 Code Question Answer moh_com4_9 In your opinion, what are the advances in the monitoring, supervising, and coordinating of the integration of HIV and STI made in the offer of integrated services quality health package? moh_com4_10 In your opinion, what are the challenges of integrating HIV and STIs in the offer of integrated services quality health package? What did you do to overcome them? moh_com4_11 What were the main challenges and constraints that you faced in the implementation of this component? What did you do to overcome them? moh_com4_12 In what areas are the implementation of the HIV and STI component causing the most problems? moh_com4_13 Do you have other comments or suggestions? MOH – Component #5: HCP Code Question Answer 1) Date of Interview 2) Duration of Interview 3) Place of Interview 4) Person Interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email Address 5) Names of EY Interviewer teams Team A/Team B/Team C 6) Date of the field research (mission) 7) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal, MOH, Health Hut, Clinic, Beneficiary (Individual), Hospitals, Pharmacies, Private Organizations) 8) Component 5 Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 239 Code Question Answer moh_com5_1 What was your role in the USAID/Senegal Health Program regarding the communication and health promotion component managed by ADEMAS? moh_com5_2 How has the USAID/Senegal Health Program been useful for your department? moh_com5_3 What was the added-value component in communication and health promotion in your area? moh_com5_4 What have been the challenges? moh_com5_5 In your opinion, how has the USAID/Senegal Health Program helped train a critical mass of public agencies? To create a network of CBOs? And support the development of a national framework for communication? Please explain. moh_com5_6 How has the USAID/Senegal Health Program promoted social and political commitment to health programs at the central level? moh_com5_7 Please indicate whether national advocacy strategies have been developed in priority areas (e.g., FP, prevention and treatment of malaria, the treatment of water-related diseases, child malnutrition, infant hygiene and sanitation)? moh_com5_8 What was the effect of the use of social marketing in the introduction of new products and services? moh_com5_9 To what extent has the USAID/Senegal Health Program helped to strengthen the supervision, coordination and management of the National Program for the Fight against HIV/AIDS and STIs? moh_com5_10 What do you think of the PPPs to support communication activities for behavior change and social marketing? moh_com5_11 What were the main challenges and constraints that you have faced in the implementation of this component? moh_com5_12 Do you have any comments or suggestions? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 240 Health structures (i.e., hospitals, centers, and posts) Code Question Answer 1) Date of interview 2) Duration of interview 2a) Start 2b) End 3) Place of interview 4) Person interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email address 5) EY team Team A/Team B/Team C 6) Region 7) District Code Question Answer Component 1: HSS Sub-component A: Improving management and system performance at regional district levels ss_com1_1 Can you describe your system of PBF? What was the process of setting up the PBF system? Mutuelle? Community/CDS? How does the system work? ss_com1_2 Who are the stakeholders (management at the facility)? ss_com1_3 What is your role within the system (How do you participate in this system)? ss_com1_4 What are the advantages of PBF for your facility? Explain. ss_com1_5 What was the contribution of the USAID/Senegal Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 241 Code Question Answer Health Program in the adoption, implementation and the operation of the PBF? ss_com1_6 Can you explain the mechanism of resource allocation? The use of the USAID/Senegal Health Program funding? ss_com1_7 How did you find funding to improve the infrastructure of the health facility? ss_com1_8 On what basis of decisions/criteria are such funds used? ss_com1_9 Describe the monitoring mechanisms of the use of funds? ss_com1_10 Explain how the bonus system works. Describe the criteria to award bonuses to agents? ss_com1_11 How do you find the PBF on the health program in your area? Yes/No ss_com1_11a Please explain. ss_com1_12 What aspects of the PBF system are working well? ss_com1_13 What are the challenges in the health structures? At the mutuelles? ss_com1_14 What are the challenges at the community level? ss_com1_15 What are the benefits to your organization? Please explain. ss_com1_16 Please explain what are limitations of this system of PBF? ss_com1_17 Do you want to continue with the PBF systems? Yes/No ss_com1_17a Please explain. ss_com1_18 Do you have any suggestions to enhance learning and improve the PBF system? Component 2: Health Services Improvement Sub-component A: Increase access to an integrated package of package health services (Thiès, Diourbel, Kaolack, Kaffrine, Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 242 Code Question Answer Fatick, Kolda, Sédhiou, Ziguinchor, Saint-Louis, Kédougou, Dakar, Pikine et Rufisque). Integrated malaria package in all 14 regions. ss_com2_1 What is the integrated services package? Malaria, nutrition, HIV? How does it work? ss_com2_2 What support have you received from USAID/Senegal to allow you to deliver services of the integrated services package? ss_com2_3 What topics do the training providers covered: FP, child survival, pre-eclampsia and eclampsia, obstetric and neonatal emergencies, nutrition, diarrhea diseases, and malaria? ss_com2_4 What are the other areas are covered in the training of staff? ss_com2_5 To what extent does this complement the knowledge of the staff? Explain. ss_com2_6 How has the training changed skills and professional practice of the personnel? ss_com2_7 How is the staff training monitored? ss_com2_8 How is the staff supervised to ensure positive changes? ss_com2_9 How has your structure using TutoratPlus? ss_com2_10 What do you think of TutoratPlus? ss_com2_10a Please explain. ss_com2_11 What benefit has the staff gained from TutoratPlus? ss_com2_12 What suggestions do you have for improving TutoratPlus? ss_com2_13 Are the benefits of training received reflected in the effective provision of the integrated package of services by your organization? Yes/No ss_com2_13a Please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 243 Code Question Answer ss_com2_14 What support has your structure received from IntraHealth for the immunization campaign? ss_com2_15 To what extent has it been helpful? ss_com2_16 To what extent has it improved immunization campaigns? ss_com2_17 Do you have any suggestions to improve the support for immunization campaigns? Describe. Sub-component B: Improve functioning of health services in public health posts and health centers and regional hospitals for related priority services provided in the Integrated Package ss_com2_18 What kind of support have you received to reinforce services in health facilities? ss_com2_19 What comments do you have on the equipment or furniture received by the health huts? ss_com2_20 How will equipment be maintained in the future? ss_com2_20a Please explain. ss_com2_21 What remarks do you have about the products received from the USAID/Senegal Health Program (IntraHealth) to strengthen the services of the integrated health package? ss_com2_22 How do your structure and its partners, including IntraHealth, purchase contraceptives and other health products to avoid shortages? ss_com2_23 What system of integrated supervision of service reinforcement have you set for the region? The district? The facility? The post? ss_com2_24 How does the new system compare to the previous one? ss_com2_25 What are the improvements? ss_com2_26 What are the benefits of implementing the system? ss_com2_27 What are the challenges of implementing the system? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 244 Code Question Answer ss_com2_28 Describe the changes made by the USAID/Senegal Health Program to strengthen the referral system between the regional hospital, health center, health station, the health put, and CH services? ss_com2_29 Can we see referral sheets or evidence of a referral system? ss_com2_30 How do you rate the functioning of the referral system between the regional hospital, health center, health huts, health post, and CH services? ss_com2_31 What aspects of the referral system and counter￾reference work properly? Explain. ss_com2_32 What system (tools) of collection and monitoring have you put in place to track progress toward access to the full range of care services? ss_com2_33 Do you have any suggestions to improve the system? Component 4: HIV/AIDS Sub-component B: Reinforce comprehensive package of treatment, care, and support for PLWHIV (Kédougou, Kolda, Sédhiou, Ziguinchor, Thiès (Mbour), Louga, Kaffrine, Kaolack) ss_com4_1 Can you tell us what the treatment package, care and support for PLWHIV is all about? ss_com4_2 What innovations or major changes have you seen over time in policies and strategies against HIV and support for PLWHIV? ss_com4_3 Could you tell us about the treatment package, care, and support offered to PLWHIV? ss_com4_4 What are the important contributions of the USAID/Senegal Health Program in the fight against HIV and support for PLWHIV? ss_com4_5 What have been the outcomes of the USAID/Senegal Health Program (e.g., attitudes and performance, supply and quality of services, cooperation, Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 245 Code Question Answer coordination) ss_com4_6 Are gender issues better integrated into policies and strategies? How? ss_com4_7 Are the needs of teenagers or other specific groups (MSM) taken into account? Yes/No ss_com4_7a How? ss_com4_8 Have your personnel been trained in the care for PLWHIV (HIV continuum of care)? Yes/No ss_com4_9 Describe the components in this training? ss_com4_10 How is knowledge gained through this training being used in this health facility? ss_com4_11 What changes have you observed in HIV care as a result of this training? ss_com4_12 What system has your structure had in place to ensure access to the full range of care for PLWHA and to establish links between patients and community services for PLWHA? ss_com4_13 What successes do you attribute to this system? ss_com4_14 What challenges have you faced? What solutions were employed? ss_com4_15 What antiretrovirals have your structure received from the program? ss_com4_16 What other sources of funding or technical support do you have access to? ss_com4_17 What are the challenges? ss_com4_18 How do you think you will overcome the challenges? ss_com4_19 What aspects (including assistance, supervision, coordination, etc.) are used to improve, maintain, strengthen, or help the program to achieve its goals? ss_com4_20 Do you have any suggestions to improve program Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 246 Code Question Answer results and to achieve your goals and the goals of this component? Health huts Code Question Answer 1) Date of interview 2) Duration of interview 3) Place of interview 4) Person interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email address 5) EY team Team A/Team B/Team C 6) Region 7) Month/Year the project started 8) Period of analysis- Start(M/Y) End (M/Y) 9) District 9a) Local Health Station 10) Location of the IP headquarters 11) Main stakeholders 12) Other notes Code Question Answer Component 3: CH Subcomponent A: Improving the quality of access to information, products and services at health huts and outreach sites. cs_com3_1 What changes have you noticed in the operation of the health huts? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 247 Code Question Answer cs_com3_1a Explain how the agendas are posted at the health hut work. cs_com3_1b What has been the support of the USAID/Senegal Health Program to your local health hut? Please explain. cs_com3_2 What services are provided at health stations? Describe. cs_com3_3 What are the services provided at the community-level (in terms of advanced strategy)? Describe. cs_com3_4 What is working well? Describe. cs_com3_5 What are the challenges you are facing? Explain. cs_com3_6 How do you do to tackle these issues? Explain. cs_com3_7 What services were offered by the health hut to the community before the introduction of the USAID/Senegal Health Program? cs_com3_8 Did this package cover the needs of the community? Yes/No cs_com3_8a Explain. cs_com3_9 What is the content of the integrated services package currently available at the health hut? Describe. cs_com3_9a What is the availability of contraception (all types), obstetric equipment, basic drugs stock, ORS sachets, tablets for headaches, physical care (bandages and alcohol), medication for respiratory infections, treating malaria, treated mosquito nets, and IEC materials? cs_com3_10 Does this package cover the current needs of the community? Yes/No cs_com3_10a Please explain. cs_com3_11 Have you received training to administer the package? Yes/No cs_com3_11a Please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 248 Code Question Answer cs_com3_12 Describe the way it is offered at community level (fixed or outside)? cs_com3_13 What assistance did you receive in terms of monitoring and supervision? From whom and how did you receive it? (describe the activities of monitoring and supervision and the note the frequency of the activities) cs_com3_14 Since the project began, what components have you added to the integrated care package? cs_com3_15 How are these new services supplied? cs_com3_16 How do the health post personnel provide services to the health hut? cs_com3_17 How often do the agents of the CBOs visit? cs_com3_18 When were the last two visits of the head of the health post? cs_com3_19 Please describe what happens when the health post head visits the health hut. cs_com3_20 How does the community around the health hut receive information about the visit of the head of the health post? cs_com3_21 Is the community satisfied with services performed by the head of the health post? Explain. cs_com3_22 Generally, what are the unsatisfied needs in your community? cs_com3_23 What efforts are being made to overcome these issues? cs_com3_24 Do you have any suggestions for the improvement of this situation? Yes/No cs_com3_24a Please explain. cs_com3_25 How does the monitoring and supervising of the head Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 249 Code Question Answer of the station allow you to overcome problems related to the services provided with the interrelated package? cs_com3_26 How are you finding this supervision? What are its strengths and weaknesses? cs_com3_27 How did you do overcome these challenges? cs_com3_28 What other suggestions do you have to improve your supervision? cs_com3_29 What reference system is established between the health hut, health post, health center and the hospital? cs_com3_30 How does ASC work in collaboration with volunteers, facilitators, matrons? cs_com3_31 What are the activities performed on a daily basis by everyone? cs_com3_32 What training have you received? cs_com3_32a What other training do you think is needed for better service delivery of the integrated package? cs_com3_32b What difficulties have you encountered? What have you done to overcome them? cs_com3_32c To what extent has the USAID/Senegal Health Program assisted you in the renovation, equipment, drug supplies, developments of skills, monitoring and supervising of CBO staffs? cs_com3_32d How did you answer to the various needs of health concerns of men and women around the health hut? cs_com3_32e To what extent are men and women involved in the various fields of health care, FP, maternal and child health, and malaria? cs_com3_32f Do you promote the involvement of men in the integrated health care package? (e.g., FP, maternal and child health, immunization, childbirth, prevention and treatment of malaria, and diarrhea diseases). Yes/No Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 250 Code Question Answer cs_com3_32g Please explain. cs_com3_33 In which activities are men usually involved? cs_com3_34 What are the difficulties of involving men? cs_com3_35 What do you do to involve them? cs_com3_36 For what do you use the data collected in the field at the health hut? cs_com3_37 Do you have any suggestions for improving the program outcomes? National/Regional pharmacies (GOS) Code Question Answer 1) Date of interview 2) Duration of interview 3) Place of interview 4) Person interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email address 5) EY team Team A/Team B/Team C 6) Region 7) Month/Year the project locally started 8) Period of analysis: Start(M/Y) End (M/Y) 9) Location of the IP headquarters. 10) Main stakeholders 11) Other notes Code Question Answer Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 251 Code Question Answer I: Strengthening pharmaceutical management pn_com1_1 Describe the system of national and regional pharmacy management. pn_com1_2 How does the mobile system of products distribute work at regional level? pn_com1_3 Have you encountered any problems? Yes/No pn_com1_3a If so, what efforts have been made to solve these problems? pn_com1_4 What aspects of pharmaceutical management do Abt and the National Pharmacy work to strengthen? pn_com1_5 What specific problems or weaknesses has Abt helped you solve? Explain. pn_com1_6 Please explain each field of capacity building. pn_com1_7 Exactly what is being done? pn_com1_8 How have these changes been useful? pn_com1_9 Have these activities involved changes in the pharmaceutical supply system of the national pharmacy? Yes/No pn_com1_9a If yes, what are the changes? pn_com1_10 What do you think of the effects of these changes? pn_com1_11 How is the improvement of the pharmaceutical management translated? pn_com1_12 What other building capacity needs do you have? pn_com1_13 How often are there shortages of oral and injectable contraceptives, equipment obstetrics, mosquito nets, and condoms? pn_com1_14 How was the problem solved? II: Social Marketing pn_com5_15 Explain how the national pharmacy system Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 252 Code Question Answer contributes to social marketing? pn_com5_16 How do you get the products to be marketed under the social marketing? pn_com5_17 Please explain the system used for distributing products to private pharmacies? pn_com5_18 How are the private pharmacies participating in social marketing selected? pn_com5_19 How do you help the USAID/Senegal Health Program increase the percentage of participating pharmacies and remain sustainable in this activity? pn_com5_20 What areas of the system work well? pn_com5_21 What are the problems you have with the system? What has been done to solve them? Describe if nothing was done. Why? pn_com5_22 What are the benefits of the social marketing of health products? pn_com5_23 What are the disadvantages of the social marketing of health products? pn_com5_24 Do you have any suggestions to improve the system of the social marketing of health products? pn_com5_25 What health products would you like to add or remove from social marketing of health products? pn_com5_26 Do you think the social marketing of health products should continue as planned? Yes/No pn_com5_26a Please explain. Private pharmacies Code Question Answer 1) Date of interview 2) Duration of interview Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 253 Code Question Answer 3) Place of interview 4) Person interviewed 4a) Function/Position/Occupation 4b) Telephone 4c) Email Address 5) EY team Team A/Team B/Team C 6) Region 7) Month/Year the project locally started 8) Period of analysis: Start(M/Y) End (M/Y) 9) Name of the pharmacy 10) Location of the IP headquarters 11) Main stakeholders 12) Other notes Code Question Answer pp_com5_1 Describe the social marketing of health products? pp_com5_2 What products do you buy at the national pharmacy for the purpose of social marketing? pp_com5_3 What problems do you face with the delivery of the products to your pharmacy? pp_com5_3a Do you receive them in a reasonable time? Explain. pp_com5_3b What shortages of condoms, oral contraceptives, injectable, treated mosquito nets, and others do you have? pp_com5_4 What is the demand (percent) for this product compared to the demand of the same product marketed outside social marketing? pp_com5_5 Compare the sales flow of the products sold by social Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 254 Code Question Answer marketing and the same product sold outside social marketing? pp_com5_6 What products do you need to add into social marketing program? pp_com5_7 What are the benefits of social marketing? pp_com5_8 What are the disadvantages of social marketing? pp_com5_9 Do you think the program should allow you to continue participating in social marketing? pp_com5_10 Do you have any suggestions to improvement the system? Regional and district health offices (GOS) Code Question Answer 1) Date of interview 2) Duration of interview Beginning: End: 3) Place of interview 4) Person interviewed 4a) Position/Occupation/Function 4b) Telephone 4c) Email address 5) EY team Team A/Team B/Team C 6) Region 7) Month/year the project locally started 8) Period of analysis: Start(M/YEAR) End (M/YEAR) 9) COR (if Different from AOR) 10) Location of the IP headquarters 11) Main stakeholders 12) Other notes Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 255 Code Question Answer 13) Organization (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID/Senegal Health Program, MOH, Health Hut, Clinic, Beneficiary (Individual), Hospital, Pharmacy, Private Firm) Code Question Answer Component #1: Health Strengthening System rds_com1_1 How have regional and district teams been involved in the formation of health governance and financial management? rds_com1_2 How did it contribute to the improvement of their performance? Please explain. rds_com1_3 Describe the system of direct funding in your region/district as part of the USAID/Senegal Health Program. rds_com1_4 What were the Success, challenges, advantages and disadvantages? Please explain. rds_com1_5 What effects has direct funding had on the implementation of the program in the region or district? rds_com1_6 What is the most recent implementation of health policy, supported by the USAID/Senegal Health Program that you have seen? rds_com1_7 What impact have they had on the implementation of your health program? rds_com1_8 Do you think these new health policies will contribute in the long run to the improvement of the health system? Yes/No Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 256 Code Question Answer rds_com1_9 How have they helped the regional and district teams develop an integrated action plan that takes into account the components of USAID/Senegal Health Program? rds_com1_10 How are the objectives of the USAID/Senegal Health Program in line with the health objectives at the regional and district level? rds_com1_11 How is the action plan used to coordinate all the activities of each component? And the entire program? rds_com1_12 How would you further enhance the implementation of the program? rds_com1_13 How does the Pavilion Outpatient Treatment (PTA) of the USAID/Senegal Health Program help improve the development in health programs at the regional and district level? Explain. rds_com1_14 What other activities were supported by Abt to improve the health care system? Explain each activity. rds_com1_15 What positive changes or improvements has it brought in the activities of the health system? Component #2: Health Services Improvement rds_com2_1 What contribution has the USAID/Senegal Health Program given to the improvement of increased access to an integrated quality health services package (maternal and child health, reproductive health, malaria, clinical and vaccination campaigns)? rds_com2_2 Has the USAID/Senegal Health Program helped improve the functioning of health services in the health posts and public health centers in relation to priority services within the integrated package? Yes/No rds_com2_2a Please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 257 Code Question Answer rds_com2_3 Has the USAID/Senegal Health Program helped improve the human resource management in public health facilities? Yes/No rds_com2_3a Please explain. rds_com2_4 Is there a referral system between the community level and other levels of the health pyramid? Yes/No rds_com2_4a Please explain. rds_com2_5 Is there a referral system between regional hospitals and community services? Yes/No rds_com2_5a Please explain. rds_com2_6 What state is it in? rds_com2_7 What aspects are functioning properly? rds_com2_8 What aspects are not functioning properly? rds_com2_9 To what extent does the USAID/Senegal Health Program respond to the needs of health facilities in equipment, products? Please explain. Can you show it to us? rds_com2_10 What are the other needs? rds_com2_11 Has the USAID/Senegal Health Program helped strengthen the relationship between local and regional district as part of a joint effort to support programs and better health result? Yes/No rds_com2_11a Please explain. rds_com2_12 What activities are supported by IntraHealth to improve the management of human resources at the public health facilities? rds_com2_13 For each activity, please indicate whether and to what extent it has been helpful. rds_com2_14 If they were not useful, why not? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 258 Code Question Answer rds_com2_15 What are the other needs? Component #3: CH rds_com3_1 Please indicate to what extent the USAID/Senegal Health Program has contributed to the improvement of the quality of access of information, products, and services at health posts and outreach services. rds_com3_2 Has the USAID/Senegal Health Program facilitated community ownership of the project? Yes/No rds_com3_2a If yes, please explain how. If not, please explain the constraints. rds_com3_3 Has the USAID/Senegal Health Program helped improve linkages and collaboration between stakeholders at the regional level, medical district teams, development partners, and the community? Yes/No rds_com3_3a If yes, please explain how. If not, please explain the constraints. rds_com3_4 Has the number of beneficiaries using options and seeking health care (services and health information) increased? Yes/No rds_com3_4a Please explain. rds_com3_5 What support did the ChildFund consortium provide you? Please explain. rds_com3_6 Was it useful? Yes/No rds_com3_6a Please explain. rds_com3_7 What major changes in the project were brought to the health program at the regional level? Explain. rds_com3_8 Which activities of the ChildFund consortium project have worked particularly well? Please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 259 Code Question Answer rds_com3_9 What were the main challenges and constraints encountered during the implementation of this component? rds_com3_10 Has the USAID/Senegal Health Program helped improve the relationship between the MOH, regional entities, and districts and facilities at the local level? Yes/No Component #4: HIV/AIDS rds_com4_1 Please indicate to what extent the USAID/Senegal Health Program has responded to STI/HIV/AIDS and TB at national, regional and district level? rds_com4_2 Has the USAID Health Program helped strengthen the package of treatment, care, and support for PLWHA? Yes/No rds_com4_2a If yes, please explain how. If not, please explain the constraints. rds_com4_3 Has the USAID/Senegal Health Program helped strengthen the capacity and performance of regions and districts to diagnose and treat TB? Yes/No rds_com4_3a If yes, please explain how. If not, please explain the constraints. rds_com4_4 Do you think that the interventions have contributed to the reduction of HIV infections? Yes/No rds_com4_4a Please explain. rds_com4_5 Do you think that the interventions have contributed to the reduction and the decline of stigma? Yes/No rds_com4_5a Please explain. rds_com4_6 Is the USAID/Senegal Health Program contributing to the implementation of the project "Border and Vulnerability" in the region of Ziguinchor? Yes/No rds_com4_6a If yes, please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 260 Code Question Answer rds_com4_7 What major changes has the project brought in the intervention of HIV/AIDS in Senegal? rds_com4_8 What major changes has the project made in the interventions of HIV/AIDS at a community level? rds_com4_9 What were the main challenges and constraints encountered during the implementation of the HIV/AIDS component? rds_com4_10 What solutions have been recommended? Were there difficulties addressed? Please explain. rds_com4_11 Has the USAID/Senegal Health Program helped strengthen the planning, coordination, and management of HIV activities in your region/district? Yes/No rds_com4_11a If yes, please explain how. rds_com4_12 Do you have any comments or suggestions to help achieve the objectives of this component? Component #5: Health Communication and Promotion rds_com5_1 Please indicate to what extent the USAID/Senegal Health Program helps build capacity to promote effective CBC in your region/district? rds_com5_2 Has the USAID/Senegal Health Program helped improve the quality of interventions in terms of BCC at regional and district levels? Yes/No rds_com5_2a If yes, please explain how. If not, please explain the constraints. rds_com5_3 Did it lead to the adoption of healthy behaviors and the increased use of health services? Yes/No rds_com5_3a If yes, please explain how. If not, please explain the constraints. rds_com5_4 Has the MHSA (SNEIPS or BREIPS) contributed to capacity building of health workers at regional and district levels? Yes/No Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 261 Code Question Answer rds_com5_4a If yes, please indicate the type of training received. rds_com5_5 Did the training meet your needs? Yes/No rds_com5_5a Please explain. rds_com5_6 Have you noticed an improvement in the work of health workers after this training? Yes/No rds_com5_6a Please explain. rds_com5_7 Has the USAID/Senegal Health Program helped to strengthen the planning, coordination, and management at the central and at the local level? Yes/No rds_com5_8 Has the USAID/Senegal Health Program helped strengthen the regional staff's technical capacity and organizational management? Yes/No rds_com5_8a Please explain. rds_com5_9 What are the main challenges and constraints encountered during the implementation of this component at regional and district levels? rds_com5_10 What are the main challenges and constraints encountered during the implementation of this component at the community level? rds_com5_11 Do you have any comments or suggestions to help achieve the objectives of this component? Regional and district coordinating bureaus (USAID/Senegal) Code Question Answer 1) Date of interview 2) Duration of interview 3) Place of interview 4) Person interviewed 4a) Position Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 262 Code Question Answer 4b) Telephone 4c) Email address 5) EY team Team A/Team B/Team C 6) Region 7) Month/year the project locally started 8) Period of analysis: Start(M/Y) End (M/Y) 9) COR (if Different from AOR) 10) Location of the IP headquarters of the agency. 11) Main stakeholders 12) Other notes Code Question Answer Organization/Firm brc_1 What is the role of the RBs in the USAID/Senegal Health Program? brc_2 How are the RBs structured? Can you describe the structure of the office? brc_3 What are your relationships with the advisors of each component of the program like? Please explain. brc_4 What are the reporting lines and collaboration between components? brc_5 What are your assessments of the flowchart of the functions of the bureau? brc_8 What are your duties and responsibilities as regional coordinator? Please explain. brc_9 What is your relationship with the region and district management teams ? What is your relationship with managers of each component team of the programs? brc_10 What problems arise in these relationships? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 263 Code Question Answer brc_11 What did you do to overcome them? Financial Management brc_12 Please explain how the decentralized financing system (DF) is implemented through your office. brc_13 How does the DF system function? brc_14 How are the funds allocated to the various components of the USAID/Senegal Health Program? brc_15 Describe how decisions are made in regard to funds allocated. Who decides? How? brc_16 What criteria is used to determine the allocation of funds? brc_17 Are the staff members of the various components satisfied with the terms of allocation of funds? Yes/No brc_17a Explain. brc_18 Are arrangements made to ensure that the IP of each component disburses the funds allocated to activities in a timely manner? brc_19 If not, how are you trying to correct this situation? brc_20 What internal systems have you put in place to allow the IPs to provide funds for the component's integrated AWP activities? brc_21 What problems have you encountered with the implementation of expenditures with the IPs and what recourse do you have? brc_22 Do you think that the funding provided to the Regional Office is enough to cover its operations and activities in the AWP? Yes/No brc_23 What financial constraints that tend to delay the project have you encountered? brc_24 What is DF all about? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 264 Code Question Answer brc_25 What are the advantages of DF of program activities? brc_26 What are the inconveniences of DF? brc_27 What suggestions do you have to overcome problems of the DF of program activities? Program Management brc_28 To what extent does the Regional Office contribute to the achievement of the goals of USAID/Senegal Health Program? brc_29 How are the regional implementation and monitoring plan for the activities of the program performed? brc_30 How does the planning of service delivery work? brc_31 How do the control supply agents function? brc_32 How does the coordination of activities function? brc_33 What suggestions do you want to make to improve the planning, monitoring, supervision, coordination, and management of health service delivery? PPP Code Question Answer 1 Date of interview 2 Duration of the interview 3 Place of the interview 4 Person interviewed 4a Position 4b Telephone 4c Email addresses 5 EY team Team A/Team B/Team C 6 Region Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 265 Code Question Answer 7 Month/year the project locally started 8 Period of Analysis: Starting (M/Y) End (M/Y) 9 COR (if different from AOR) 10 Location of the IP headquarters 11 Main stakeholders 12 Other notes 13 Organizations (Abt, IntraHealth, ChildFund consortium, FHI 360, ADEMAS, USAID, MOH, Health Hut, Clinic, Beneficiaries (Individual), Hospital, Pharmacy, Private Firm) 14 Component (1, 2, 3, 4, 5, N/A) Code Question Answer Component 5: HCP ppp_com5_1 How did you get involved in the USAID/Senegal Health Program? ppp_com5_2 What is your role within the USAID/Senegal Health Program? ppp_com5_3 What services do you provide to the community through the USAID/Senegal Health Program? ppp_com5_4 Why did you agree to collaborate with a non-profit program like USAID/Senegal Health Program? ppp_com5_5 Is it for purely commercial purposes or is it part of your social responsibility? ppp_com5_6 Explain how social marketing functions? ppp_com5_7 What are the most sold pharmaceutical products with the social marketing program? ppp_com5_8 Describe the distribution system of health products? ppp_com5_9 How do you compare the product sales of social Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 266 Code Question Answer marketing and the product sales of the same products sold in the normal profit system? Could you provide statistics/status of sales of products offered in social marketing? ppp_com5_10 What are the advantages of social marketing of health products? ppp_com5_11 What are the downsides that could be faced with social marketing? ppp_com5_12 What works well in the social marketing program? Explain. ppp_com5_13 What are the main challenges of collaboration with the IP agency? ppp_com5_14 How are you considering working with social marketing programs in the future? ppp_com5_15 Do you have suggestions about the improvement of the PPP? Explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 267 Annex J: Focus Group Interview Guides CH workers Number Question Answer 1) Date of interview 2) Duration of interview 2a) Starting time 2b) Ending time 3) Place of interview 4) Name of interviewee 5) Number of participants 6) Interviewer team 7) Region 8) District 9) Health hut Code Question Answer Information on implementing partner pmo_1 Name of Focal Point pmo_2 Title of Focal Point pmo_3 Address (E-mail, telephone number) pmo_4 Date of interview Code Question Answer asc_1 Do you agree that to reach the objectives of the health huts, collaboration between CH agents, relais (community outreach workers), the BG, matrones (community worker for maternal health), and volunteers is necessary? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 268 Code Question Answer asc_2 Do the CH agents, the community relais, and the BG collaborate on all components? asc_3 Does the distribution of tasks allow you to work efficiently? Yes/No asc_4 What support is provided to the CH agents, community relais, BG, matrones, and volunteers received in terms of capacity building, equipment, finance, supervision, coordination, etc.? asc_5 Do the different funding systems help you to fully contribute to the integrated health system package? Yes/No asc_6 Do you believe the multiplicity of actors intervening in the health hut is beneficial? Yes/No asc_7 Should the number of community agents be reduced? Yes/No asc_8 Do you think there is gender equity in the selection of CH agents, community relais, BG, the matrones, and the volunteers? CH insurance (Committee) Number Question Answer 1) Date of interview 2) Duration of interview 2a) Starting time 2b) Ending time 3) Place of interview 4) Name of interviewee 5) Number of participants 6) EY team 7) Region 8) District 9) Health post Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 269 Number Question Answer 10) Health hut Code Question Answer Health Committee csm_com1_1 Can you please describe the structure of the health committee (including gender aspects)? csm_com1_2 Can you describe how social health insurance operates? csm_com1_3 What is the added value of social health insurance and how is it affecting the quality of health? csm_com1_4 Are the health structures reaching people in need? Yes/No csm_com1_5 Have you received technical or financial support for the operations and the management of the social health insurance system? Yes/No csm_com1_5a If yes, who did you receive it from and what kind of support was it? csm_com1_6 What is the role of each stakeholder (beneficiaries, managers, health hut, health post, hospital, private and public...) in the social health insurance system? csm_com1_7 How would you assess the collaboration between all stakeholders of social health insurance? csm_com1_8 How is the health committee structured? csm_com1_9 What are the problems you have with social health insurance management? Please describe. csm_com1_10 What are the problems you encounter with the health structures? Please describe. csm_com1_11 What are the collaboration mechanisms between your two institutions? csm_com1_12 How do you manage the challenges arising in your collaboration with social health insurance system? How do you manage the challenges arising in your collaboration with health structures (health hut, health post, private clinic, hospital...)? Please explain. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 270 Code Question Answer csm_com1_13 Please describe the social health insurance system in your structure. csm_com1_14 What is the role of each stakeholder in social health insurance? csm_com1_15 What are the collaboration mechanisms between different entities? Please explain. csm_com1_16 How are any problems managed? csm_com1_17 What is the impact of social health insurance on the usage of health structures (health hut, health center, private clinic, and hospital)? Please explain. csm_com1_18 Do you have suggestions on how to improve the operations of the social health insurance system? csm_com1_19 Do you have suggestions on how to improve the quality of health services? CH insurance (Leader/Manager) Number Question Answer 1 Date of interview 2 Duration of interview 2a Starting time 2b Ending time 3 Place of interview 4 Name of interviewee 5 Number of Participants 6 EY Team 7 Region 8 District 9 Health Post 10 Health Hut Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 271 Code Question Answer Information on Implementing Partners pmo_1 Name of Focal Point pmo_2 Title of Focal Point pmo_3 Address (E-mail, Telephone number) pmo_4 Date of interview Question Answer Mutual Managers gm_com1_1 What led to the establishment of your social health insurance program? gm_com1_2 Can you explain how your social health insurance program is organized in your area of intervention? (e.g., geographic coverage, number of members) gm_com1_3 How is your social health insurance program operating? gm_com1_4 Do you have a memorandum of agreement/instruments/formal work mechanism/cooperation and collaboration between stakeholders of the social health insurance program (health huts, private and public health centers)? Yes/No gm_com1_4a Please explain. gm_com1_5 Does cooperation/collaboration exist between stakeholders of the social health insurance and health structure (health huts, health posts, hospitals, etc.)? Yes/No gm_com1_5a Please explain. gm_com1_6 What is working well in this collaboration? gm_com1_7 Please describe the profile of your members (e.g. gender, informal sector, rural sector) gm_com1_8 What are the medical services covered by the social health insurance program (medication for mothers, children, vaccinations, consultations, hospital complementary Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 272 Question Answer package, caesarian section, etc.) gm_com1_9 Have you received support from the USAID/Senegal Health Program/HSS for the implementation and management of the social health insurance program? Yes/No gm_com1_9a What type of support is this? (e.g., training, legal support, financial) gm_com1_10 Please describe how the poorest populations are served by the health structure? gm_com1_11 What aspects of the health structure work well? Please explain (community adherence, management of social health insurance and its relation to the health structure)? gm_com1_12 What are the challenges that impede the operation of your social health insurance? gm_com1_13 What aspects of the social health insurance system are least effective (e.g., membership, social health insurance management, coordination within the health structure gm_com1_14 What aspects of the collaboration, between social health insurance and the community, works well? gm_com1_15 How does the social health insurance program encourage community membership? gm_com1_16 What are the challenges in the collaboration between the social health insurance and the health structure? gm_com1_17 How do you resolve problems between the two structures? gm_com1_18 Has an appeal system been established in case of conflict between social health insurance and the health structure? Please give examples of issues that were resolved. gm_com1_19 How is access to the health services and the quality of the services for the members of social health insurance program? If the service is good, please explain. If not, what would you do to improve the situation? gm_com1_20 Have you received any support to establish or manage social health insurance? (training or capacity building in this project) Yes/No Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 273 Question Answer gm_com1_21 If yes, please describe the nature of the support received and if it was useful. gm_com1_22 Is anything done to ensure the sustainability of social health insurance (Government support) ? Please explain. gm_com1_23 What is the benefit of being a member of the social health insurance program? gm_com1_24 Is there a benefit of not being a member of a social health insurance? Yes/No gm_com1_24a Please explain. gm_com1_25 Do you have suggestions to improve the operations of social health insurance, to encourage membership, to improve the quality of services provided to members? CH insurance (Beneficiaries) Number Question Answer 1 Date of interview 2 Duration of interview 2a Starting time 2b Ending time 3 Place of interview 4 Name of interviewee 5 Number of participants 6 EY team 7 Region 8 District 9 Health Post 10 Health Hut Code Question Answer Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 274 Code Question Answer Beneficiaries/Members of the Social Health Insurance bm_com1_1 How does social health insurance operate? What type of services is covered by social health insurance? bm_com1_2 What do you think of social health insurance? What are the benefits for members and their families? Please explain. bm_com1_3 Why would people in the community refuse to become members of the social health insurance program? bm_com1_4 What type of services is provided to members? bm_com1_5 What challenges are you facing in social health insurance? Please give a few examples. bm_com1_6 Do you have mechanisms to resolve conflicts? Yes/No bm_com1_6a If yes, give examples. bm_com1_7 What is your understanding of the services offered by the social health insurance program to other members of the family? bm_com1_8 What is the impact of social health insurance on the quality of life of members and their families in your community? bm_com1_9 Does social health insurance help you to give better access and better services? bm_com1_10 What is the impact of social health insurance on the poor, children, women, PLWHIV, MSM, and ChildFund-led Community Health Program (PSC), the quality of services/treatment, the health centers, and on the f lives of the community? bm_com1_11 Do you have suggestions to improve the operations of the social health insurance program and the USAID/Senegal Health Program? CH management committee Number Question Answer 1 Date of interview Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 275 Number Question Answer 2 Duration of interview 2a Starting time 2b Ending time 3 Place of interview 4 Name of interviewee 5 Number of participants 6 EY Team 7 Region 8 District 9 Health Post 10 Health Hut Code Question Answer Beneficiaries/Members of the Social Health Insurance cgcs_com3_1 What is the relationship between the health committee, the health post and the health hut? cgcs_com3_2 How was the health committee set up? cgcs_com3_3 What are the roles and responsibilities of the health committees? cgcs_com3_4 What is the composition of the committee and who are the members? cgcs_com3_5 How are members chosen? cgcs_com3_6 How is the health committee organized? How many women and how many men are there in the committee? cgcs_com3_7 How are decisions made? cgcs_com3_7a Do the women participate in the decision-making? Yes/No cgcs_com3_7b Please explain. cgcs_com3_8 What are the activities of the health committee? Please describe. Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 276 Code Question Answer cgcs_com3_9 How do you make decisions for the financing of each activity? cgcs_com3_10 Do you have enough funding for your activities? Yes/No cgcs_com3_10a Please explain. cgcs_com3_11 Who identifies each activity and its execution? Please explain. cgcs_com3_12 Have you benefited from someone's help? Yes/No cgcs_com3_12a If yes, what support have you received (individual and organization)? Who have you received it from? Was it useful? Please indicate for each activity. cgcs_com3_13 What support do you need to improve the capacity of the committee to manage its activities? cgcs_com3_14 Of these activities, which ones were implemented by the health committee? cgcs_com3_15 Have you encountered any challenges? Yes/No cgcs_com3_15a Please explain. cgcs_com3_16 Have you overcome these challenges? Yes/No cgcs_com3_16a If yes, how have you overcome them? If not, why? cgcs_com3_17 Which of those activities have succeeded? Please explain. cgcs_com3_18 Which activities have you started and not completed? cgcs_com3_19 Which activities are you putting in place? cgcs_com3_20 Have you encountered any difficulties in the implementation of activities? Please explain for each activity. cgcs_com3_21 Have you overcome the challenges linked to the implementation of these activities? Yes/No cgcs_com3_21a If yes, how? If not, why? cgcs_com3_22 Do these activities have an impact on the community? cgcs_com3_23 What do you have to do for the community to benefit from the activities of the health committee? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 277 Code Question Answer cgcs_com3_24 What is working in the health committee? cgcs_com3_25 What are the main challenges of the health committee? cgcs_com3_26 What are you doing to overcome those challenges? If not, why not? cgcs_com3_27 Do you have suggestions? Associations of PLWHA Number Question Answer 1 Date of interview 2 Duration of interview 3 place of interview 4 Number of participants 5 EY Team 6 Region 7 District 8 Health Post Code Question Answer plwha_com4_1 Please describe the structure of your organization. How was it established? What are its objectives and how does it function? plwha_com4_1a Do you have enough members in your organization? Yes/No plwha_com4_1b Do you look for a quorum whenever it is necessary to make a decision? Yes/No plwha_com4_2 What are the procedures to recruit new members? plwha_com4_3 What support do members receive from the organization? plwha_com4_4 What are the activities of the organization? Please describe. plwha_com4_4a Does the organization collaborate with partners for its Yes/No Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 278 Code Question Answer activities? plwha_com4_4a_i If yes, explain who the organization collaborates with for each activity and how it works? What are the advantages of that collaboration? What are the challenges? Please explain. plwha_com4_4b What works well in the implementation of the activities? plwha_com4_4c What are the challenges? plwha_com4_4d How does the organization resolve its problems and how successful are they in resolving problems? plwha_com4_5 Please tell us how the members of the organization access services, testing, counseling and treatment. plwha_com4_5a Where do you get your ARVs? Do you have difficulty getting to them? Please explain. plwha_com4_5b Are you satisfied with the ATU? Yes/No plwha_com4_5b_i Please explain. plwha_com4_5c Regarding your other medical needs, are you satisfied with services? Yes/No plwha_com4_5c_i If not, what are you not satisfied about? Please explain. plwha_com4_6 Transmission of HIV is a very important aspect of the health program. Do you practice any preventive activities? If yes, please explain. Yes/No plwha_com4_7 Please explain the behavior of the community regarding PLWHA. Has your organization conducted any advocacy? If yes, are you satisfied? If no, what are the problems? Yes/No plwha_com4_8 Are you informed of the other aspects of the health program financed by USAID/Senegal and its partners who also work with the community to change community behaviors toward PLWHIV? Yes/No plwha_com4_8a Please describe. plwha_com4_8b Were there any changes since the beginning of this program? Mid-Term Evaluation of USAID/Senegal Health Programs Ernst & Young LLP Final Evaluation Report Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this document. 279 Code Question Answer plwha_com4_8c Please explain. plwha_com4_9 Men and women don't have the same needs. Does your organization take into consideration these differences to satisfy the needs of men and women? Please explain. plwha_com4_10 Please give suggestions for the improvement of the health program for PLWHIV. plwha_com4_11 Are there any other questions we should have asked or any other important information we should know? EY | Assurance | Tax | Transactions | Advisory About EY EY is a global leader in assurance, tax, transaction and advisory services. The insights and quality services we deliver help build trust and confidence in the capital markets and in economies the world over. We develop outstanding leaders who team to deliver on our promises to all of our stakeholders. In so doing, we play a critical role in building a better working world for our people, for our clients and for our communities. EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients. For more information about our organization, please visit ey.com. © 2014 Ernst & Young LLP All Rights Reserved. 1411-1356360 ED none This material has been prepared for general informational purposes only and is not intended to be relied upon as accounting, tax, or other professional advice. Please refer to your advisors for specific advice. ey.com