USAID/West Africa Evidence for Development AID 624-C-15-00001 Supply Chain Management System (SCMS) Final Performance Evaluation Côte d’Ivoire Final Report USAID/West Africa Evidence for Development February 2017 This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The content of this report is the sole responsibility of the authors and does not necessarily reflect the views of USAID or the United States Government. 1 USAID/West Africa Evidence for Development AID 624-C-15-00001 Supply Chain Management System (SCMS) Final Performance Evaluation Côte d’Ivoire Final Report February 2017 DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents of this report are the sole responsibility of International Business and Technical Consultants, Inc. (IBTCI) and do not necessarily reflect the views of USAID or the United States Government. 2 ACKNOWLEDGMENTS The Final Performance Evaluation Team of the Supply Chain Management Systems (SCMS) Côte D’Ivoire Project wishes to acknowledge the relentless support received in this assignment from the following: ● The Ministry of Health and Public Hygiene (MOHPH)1 in Côte d’Ivoire, which showed a lot of support for the SCMS project and provided very useful answers to questions, as well as insight from the government’s perspective. ● The United States Agency for International Development (USAID)/West Africa Office but also the mission in Côte d’Ivoire teams who provided insight and guidance from the donor’s perspective. ● The SCMS Directorate in Côte d’Ivoire and its team who gave valuable time to answer questions and provide program support documents. ● The Nouvelle Pharmacie de Santé Publique (NPSP) representative who graciously responded to the team’s questions and provided insight on the support received from SCMS. The Evidence For Development (E4D) team wishes to thank all the members of local health facilities visited, and NGOs interviewed, who graciously gave their time to respond to interview questions, share experiences and concerns, and make suggestions for future improvements. The E4D team is also grateful to the IBCTI headquarters and field teams for their long-distance assistance and direct support during the preparation and field implementation of this evaluation, as well as the final editing of this report. Finally, the E4D team would like to express its deepest appreciation to all those who generously reviewed the conclusions and recommendations of this report and provided feedback. 1 Previously known as the Ministry of Health and the Fight against AIDS (MSLS). 3 ACRONYMS AOR ASCM ART BCC CA CEPREF CHR CHU CCM COP DPML DHS DIEM ECAP ECOWAS ENSEA ESPC FP FHI FSW GFATM GIZ HIV/AIDS HTC IBBS IBTCI ICAP ICASA IDU INS IP IR IT JHU JSI KII KP KPI LMIS LNME MAC M&E MARP MICS MOHPH Agreement Officer’s Representative Approved Supply Chain Management Guide Anti-Retroviral Treatment Behavioral Change Communication Cooperative Agreement Centre de Prise en Charge, de Recherche et de Formation Centre Hospitalier Régional Centre Hospitalier Universitaire Country Coordinating Mechanism Country Operational Plans Direction de la Pharmacie, du Médicament et des Laboratoires Demographic and Health Survey Direction des Infrastructures de l’Equipement et de la Maintenance -Decentralized Supply Chain Management System Enquête Connaissances, Attitudes et Pratiques Economic Community of West African States Ecole Nationale Supérieure de Statistique et d’Economie Appliquée Etablissements Sanitaires de Premier Contact Family Planning Family Health International Female Sex Workers Global Fund to Fight AIDS, Tuberculosis and Malaria Deutsche Gesellschaft für Internationale Zusammenarbeit Human Immunodeficiency Virus and Acquired Immune Deficiency Syndrome HIV Testing and Counseling Integrated Biological and Behavioral Surveillance Survey International Business & Technical Consultants Incorporated International Center for AIDS Program of Columbia University International Conference on AIDS and Sexually Transmitted Infections Diseases in Africa Injection Drug Users Institut National de la Statistique Implementing Partner Intermediate Results Information Technology Johns Hopkins University John Snow International Key Informant Interview Key Populations Key Performance Indicators Logistic Management Information System Liste Nationale de Médicaments Essentiels Monthly Average Consumption Monitoring and Evaluation Most at Risk Populations Multiple Indicators Cluster Survey Ministry of Health and Public Hygiene 4 MOU MSH MSM NACC NACP NEML NGO NPSP NSCA NSP PEPFAR PLHIV PNDAP PNLS PMP PMTCT PSAMAO PSI PFSCM RDC SCM SCMS SDP SHARM SME SO SOP SOW SSA STG STI STOP T&C TA TPM TWG UIC UNAIDS UNDP UNFPA URC USAID USAID/CI USAID/WA USG VCT WAHO WHO Memorandum of Understanding Management Sciences for Health Men Who Have Sex with Men National AIDS Control Council National AIDS Control Program National Essential Medicines List Non-governmental Organization Nouvelle Pharmacie de la Santé Publique National Supply Chain Assessment National Strategic Plan President's Emergency Plan for AIDS Relief People Living with HIV/AIDS Programme National pour le Développement des Activités Pharmaceutiques Programme National de Lutte contre le Sida Performance Monitoring Plan Prevention of Mother to Child Transmission Projet de lutte contre le Sida sur les Axes Migratoires de l’Afrique de l’Ouest Population Service International Partnership for Supply Chain Management Regional Distribution Center Supply Chain Management Supply Chain Management Services Service Delivery Point Survey of HIV and Associated Risk factors among MSM Subject Matter Expert Strategic Objectives Standard Operating Procedures Scope of Work Sub-Saharan Africa Standard Treatment Guidelines Sexually Transmitted Infections Stock Out Project Testing and Counseling Technical Assistance Team Planning Meeting Technical Working Group Unique Identifying Code Joint United Nations Program on HIV/AIDS United Nations Development Program United Nations Population Fund University Research Co. United States Agency for International Development USAID Côte d’Ivoire USAID West Africa United States Government Voluntary Counseling and Testing West African Health Organization World Health Organization 5 Content ACKNOWLEDGMENTS ............................................................................................................. ................................... 3 ACRONYMS ....................................................................................................................................................................... 4 EXECUTIVE SUMMARY .................................................................................................................................................. 8 I. INTRODUCTION AND BACKGROUND .......................................................................................................... 21 1.1 LOGISTICS CYCLE.............................................................................. ................................................................ 21 1.2 PROJECT OVERVIEW ........................................................................................ ................................................ 24 II. EVALUATION PURPOSE AND QUESTIONS ........................................................................................ ...........25 2.1 EVALUATION PURPOSE .......................................................................................................... ........................ 25 2.2 EVALUATION PERIOD AND GEOGRAPHIC FOCUS ............................................................................25 2.3 EVALUATION QUESTIONS.............................................................................................................................26 III. EVALUATION DESIGN AND METHODS ......................................................................... ................................ 26 3.1 EVALUATION DESIGN ........................................................................................................... ..........................26 3.2 SAMPLING .................................................................................................................... .........................................27 3.3 DATA SOURCES .................................................................................................................................................28 3.3.1 Qualitative Data ............................................................................................................ ................................. 28 2.4 QUANTITATIVE DATA .....................................................................................................................................30 2.5. DATA ANALYSIS ................................................................................................................................................ 32 IV. EVALUATION FINDINGS ...................................................................................................................................... 33 3.1 CONTRIBUTION TO LOGISTICS CYCLE (EFFECTIVENESS): .............................................................. 33 3.1.1 Product Selection ........................................................................................................... ............................... 33 3.1.2 Forecasting and Supply Planning ............................................................................................. ................... 34 3.1.3 Procurement ................................................................................................................. ................................. 37 3.1.4 Warehousing and Inventory ................................................................................................... .................... 38 3.1.5 Transportation Measures ..................................................................................................... ....................... 44 3.1.6 Quality Management .......................................................................................................... ........................... 45 6 3.1.7 Laboratory ..................................................................................................................................................... 46 3.1.8 Project efficiency .......................................................................................................................................... 46 3.2 RESPONSES TO THE OTHER EVALUATION QUESTIONS ................................................................. 50 3.2.1 Institutional Capacity Building and TA to Local Institutions.............................................................. 50 3.2.2 Technical Assistance to the NPSP............................................................................................................ 50 3.2.3 Current Status of the Supply Chain at the National, Regional and District Levels...................... 51 3.2.4 Technical Assistance to Health Districts................................................................................................ 52 3.2.5 Decentralized Supply Chain Management.............................................................................................. 53 3.2.6 Contribution to Reduction of Breaks in ARV and Lab Product Supply.......................................... 54 3.2.7 Contribution to the reduction of unusable products.......................................................................... 56 3.2.8 Work in Synergy and Communication with other Actors in the Chain......................................... 56 IV-CONCLUSIONS....................................................................................................................................................... 57 RECOMMENDATIONS ........................................................................................................................................... 59 V- ANNEXES................................................................................................................................................................... 61 Annex-1 Statement of Work................................................................................................................................... 61 Annex-II: List of ART Sites Selected.................................................................................................................83 Annex III: Documents Reviewed ............................................................................................................................ 86 Annex IV: List of institutions Visited...................................................................................................................... 87 Annex V: Evaluation Team ....................................................................................................................................... 88 ANNEX V: Capability Maturity Model and Rankings......................................................................................... 90 Annex VI: Questionnaire .......................................................................................................................................... 92 Annex VII: Post-Evaluation Action Review Table……………………………………………………..101 7 EXECUTIVE SUMMARY USAID West Africa (WA) and USAID Côte d’Ivoire (CI) commissioned the Evidence for Development (E4D) project to conduct an end-of-the project performance evaluation of the Supply Chain Management Systems (SCMS) project in Côte d’Ivoire from September 15 to November 30, 2016. This evaluation focused primarily on assessing the achievements of SCMS intended results and the contribution to the improvement of the country’s overall supply chain performance. The purpose of the Performance Evaluation for the Supply Chain Management System (SCMS) Project is to increase learning about the performance of SCMS interventions in Côte d’Ivoire. It serves as an end￾of-project evaluation of SCMS interventions in Côte d'Ivoire. USAID health office is interested to know whether the SCMS project has achieved its intended results. Furthermore, this evaluation will help in identifying and addressing critical gaps in evidence for supply chain strengthening activities. Likewise, it will inform the Government of Côte d'Ivoire’s National Health Plan 2016-2020 and future program design and implementation for supply chain management in the country. The evaluation will also serve to document key contributions by the Government of the United States of America to supply chain management in Côte d'Ivoire. This evaluation complements and builds on the 2015 National Supply Chain Assessment (NSCA) for pharmaceutical products, including HIV commodities conducted by the Ministère de la Santé Publique et de Lutte contre le Sida (MSLS) in collaboration with SCMS and other donors. Therefore, it does not repeat the measurement of capability maturity of key supply chain management functions. Rather, the current evaluation seeks to assess the achievements compared to the goals; and determine the Supply Chain Management (SCMS) Project’ contribution to the achievements identified. The results and findings will guide and inform future evidence-based programming and decision making by the Ministry of Health of Côte d’Ivoire, the US President’s Emergency Plan for AIDS Relief (PEPFAR) and other donors to prioritize their continued support to strengthening the supply chain system. 8 BACKGROUND Côte d’Ivoire has one of the highest Human Immunodeficiency Virus and Acquired Immune Deficiency Syndrome (HIV/AIDS) infection rates in West Africa with a prevalence of 3.7% (Institut National de la Statistique (INS) et ICF International, 2012). The estimated number of people living with HIV is 450,000 of which 190,000 would be eligible for treatment (Joint United Nations Programme on HIV/AIDS (UNAIDS), 2013) . This prevalence is higher among women (4.6%) compared to men (2.7%). The HIV epidemic in Côte d’Ivoire is mixed (HIV1 and HIV2) and generalized across the country. In late 2005, USAID awarded the SCMS project as part of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR)’s support to the Government of Côte d’Ivoire. The project’s mission is to: • Establish and operate a safe, secure and reliable supply chain; • Buy and distribute high-quality, antiretroviral (ARVs) low-cost essential medicines, HIV test kits, laboratory supplies and other products; • Strengthen the capacity of national supply chains to ensure long-term sustainability; and • Support supply chain collaboration and information sharing among global and local partners in the HIV/AIDS community. Over the past 10 years, SCMS has been working closely with the Côte d’Ivoire MOHPH to maintain a supply chain that ensures reliable supply of quality HIV/AIDS medicines for People Living with HIV (PLHIV). The project supports the central pharmacy (PSP) in procuring, importing, storing, and distributing a variety of HIV commodities, including antiretroviral drugs, rapid test kits, opportunistic infection drugs, and other medical supplies. SCMS has provided technical assistance (TA) to the MOHPH through the Central Medical Stores-Nouvelle Pharmacie de la Santé Publique (NPSP) and other departments of the Ministry (including Direction des Infrastructures de l’Equipement et de la Maintenance (DIEM); Programme National pour le Développement des Activités Pharmaceutiques (PNDAP); Regional Health Office and Districts Health Offices). In collaboration with in -country and international partners, SCMS adopted a three-fold approach to strengthen the in-country supply chain system. This includes : ● Provision of quality, best-value health care products to those who need them; ● Deployment of innovative solutions to help programs enhance their supply chain capacity; and ● Ensuring accurate supply chain information is collected, shared, and used. SCMS activities at the sub-national level covered the 82 districts of Côte d’Ivoire where PEPFAR worked, supporting about 400 HIV/AIDS care and treatment service sites. Furthermore, SCMS provided training support to regional/district Directors and RIP+ representatives on supply chain; trained 40 stock managers on Logistic Management Information System (LMIS); upgraded 15 districts depots and 5 pharmacies of Service Delivery Points (SDPs); provided District depots in need with AC, thermometers and pallets; provided some equipment to regional and district offices, and to RIP+ zonal representatives; provided 20 4x4 pickup to health districts; reimbursed distribution costs to health centers served by the districts without vehicle for distribution; updated supervision tools being developed; implemented supervision visits at all levels; and organized quarterly coordination meetings at central, regional and district levels, including data validation at regional level. Evaluation Questions This evaluation aimed to answer the following questions suggested by USAID/WA: 1. What was accomplished and what were the challenges encountered during the implementation of the project at the national, regional and district levels regarding: a. Computerized commodity management system and reporting systems? Technical Area: Logistics Management Information System 9 b. Integrated electronic inventory management tool? Technical Area: Logistics Management Information System c. Integrated HIV/AIDS product management into the broader pharmaceutical supply chain? Technical Area: Warehousing and Inventory Management d. Collection and destruction of expired HIV/AIDS commodities? Technical Area: Warehousing and Inventory Management e. Prevention of stock out of tracer HIV/AIDS commodities at service delivery points? Technical Area: Warehousing and Inventory Management and Logistics Management Information System 2. As a technical assistance provider, did SCMS offer multiple services i.e., were beneficiaries able to access all the supply chain technical assistance they needed through a single project SCMS (aka “One-stop-shop”) for the Government of Côte d’Ivoire MOH and AIDS for HIV/AIDS supplies and supply-related services? 3. What measures did SCMS assist to put in place at the Central Medical Stores- NPSP to improve management (including risk management and waste reduction) of antiretroviral and other commodities for HIV/AIDS programs? What remains to be done? 4. How is the supply chain actually performing at the national, regional and district levels? What are the maximum and minimum stock levels for each aspect of the supply chain management system (central, district, local)? Evaluation Objectives The scope of this evaluation covers SCMS activities at the central level (NPSP) and at the sub-national level where PEPFAR supports about 400 HIV/AIDS care and treatment service sites in all the 82 health districts of Côte d’Ivoire. This evaluation focuses on the effectiveness and efficiency of SCMS interventions on key management functions of the logistics cycle, exploring, per USAID’s request, 1) the changes catalyzed by SCMS interventions and 2) other factors that contributed to the observed change. The evaluation includes four specific objectives: 1. To identify key achievements and major challenges encountered during the implementation of the project at the national, regional and district levels; 2. To describe SCMS’ technical assistance support for HIV/AIDS supplies and supply-related needs provided to institutions involved in the supply chain management in Côte d’Ivoire; 3. To describe SCMS’ technical assistance to the Central Medical Stores-NPSP to improve the management of antiretroviral and other commodities for HIV/AIDS programs; and 4. To assess the performance of supply chain at the national, regional and district levels. Evaluation Design and Methods This evaluation uses a cross- sectional and non-experimental design. The study relies on primary and secondary data using a mixed approach. Secondary data include quantitative indicators from the NSCA study, the Performance Management Plan (PMP) and other program reports, whereas primary data encompass information from 14 key informants Interviews (KII), 10 Focus Group Discussions (FGDs) as well as quantitative data (based on structured questionnaire) from 60 service delivery points in four regions: Abidjan (45 sites), San Pedro (5 sites), Bouake (5 sites) and Korhogo (5 sites) . The evaluation used a three-stage sampling approach to select the 60 sites. First, the team selected 4 districts among PEPFAR-supported districts with the highest percentage of PLHIV. At the second stage, medical stores were selected randomly. Last, the team selected health facilities associated with those stores. As a result, the evaluation team visited a total of 60 sites. 10 The evaluation team collected data at the central and sub-national levels, encompassing the medical store, district hospital pharmacies and ‘first contact’ service delivery points (Etablissement Sanitaire de Premier Contact - ESPC). Data analysis methods included quantitative and qualitative techniques. The quantitative analysis relied on descriptive statistical methods, including trend analysis and interpretation of proportions using STATA software. The qualitative analysis, on the other hand, comprised content and thematic analyses using Atlas Ti software. Key Findings This report organizes the evaluation results and findings under two major sections. The first section, “Contributions to the Logistics Cycle, ” seeks to identify the effect of SCMS interventions on key Logistics Cycle functions such as 1) product selection; 2) quantification; 3) procurement; 4) warehousing and inventory management; 5) distribution; 6) waste management; 7) laboratory; and 8) quality monitoring at each step. This section of the report responds to the first evaluation question on progress achieved toward project’s objectives (effectiveness) listed in section on Evaluation Objectives above. The second section, “Responses to the Other Evaluation Questions, ” provides responses to the remaining three evaluation questions also listed above: capacity-building and TA provided to local institutions; support provided to NPSP; and current status of the supply chain at the national, regional and district levels. Contributions to the Logistics Cycle: 1. Product selection is the entry point to the logistics cycle and a key element of the National Essential Medicine List (LNME) and Standard Treatment Guidelines (STGs) validated by the MSLS. As a best practice, the LNME should align with the STGs. Using the rankings established by the Capability Maturity Model (CMM) presented in Annex V, the NSCA found product selection to be at the “advanced practices” stage, which means there are well-defined processes and integrated technologies. Per the NSCA, while the purchase of essential medicines and consumables usually conforms to national policy on essential medicines, there is evidence that LNME are not uniformly, or even widely, used. The NSCA found that only 18% of 325 facilities visited had an updated LNME. The E4D evaluation, which used a different sampling approach, linked PEPFAR support to the use of LNME in 48.3% of the facilities visited. Going beyond what was evaluated by NSCA, the E4D evaluation also assessed the presence of supply-chain management protocols, manuals, and approved documented guidelines, in addition to LNME. The E4D assessment found compliance among 26.7% of facilities. 2. Quantification (forecasting and supply planning) determines the quantities of products required for priority disease programs to ensure continued availability. In Côte d’Ivoire, the NPSP does forecasting and supply planning for essential medicines and consumables through national committees on quantification that are established for the different types of products. These committees are incorporated within the National Committee for Supply Coordination of Essential Medicines and Strategic Commodities (Commission Nationale pour la Coordination des Approvisionnements en Médicaments essentiels et produits stratégiques en Côte d’Ivoire) (CNCAM-CI) under Ministerial Decree No 134/MSLS/CAB of March 20, 2015. CNCAM-CI is charged with the “coordination and monitoring of logistics activities related to essential medicines 11 and strategic products, for the diagnosis, prevention, and treatment of targeted health conditions for priority diseases programs”. In the NSCA evaluation, forecasting and supply planning performance was evaluated in terms of the indicator, “forecast accuracy,” as well as indicators related to the LMIS. The composite maturity for essential medicines, including HIV commodities scored at 61%, which corresponds to a maturity level of the “qualified” stage. This reflects generally well-defined and documented practices that incorporate certain automated systems The E4D assessment explored indicators related to LMIS and found that 68.3% of facilities visited submitted a complete LMIS report on time. These data corroborate the conclusion of NSCA survey. Forecasting and supply planning was a priority focus of SCMS activities pursuant to the goal to reduce the stock-out rate to 0%. 3. Procurement: Procurement at the central level is based on standard product specifications and a reference list of items. The NSCA’s findings on procurement indicate that the NPSP procurement process achieved an overall 74% maturity rating, meaning the processes are well defined and documented, and some technologies are in place to support operations. The functional areas, “Processes and tools” and “Strategic planning and monitoring” achieved 85% and 60%, respectively. Using the CMM ranking approach, the NSCA found procurement maturity to be at the “qualified” stage, which means processes are well defined and documented, with some use of technology. The E4D assessment indicates that 12 KII out of 14 reported an electronic procurement system provided through collaboration with SCMS. The system contains list of all essential drugs and other applications facilitating the procurement, inventory and monitoring processes. This system is also used to monitor requests for bids, unfilled orders and awarded contracts, fulfilled orders, and payments to suppliers. 4. Warehousing and inventory management: To improve pharmaceutical-product storage conditions, SCMS embedded a long-term consultant within NPSP during Fiscal Years ’14 and ’15 to support implementation of warehousing standard-operating procedures (SOPs) and key performance indicators (KPIs). In FY 15, SCMS began renovating district pharmacies. Using the CMM ranking, the NSCA found warehousing and inventory practices to be at the “qualified” stage, meaning that processes were well-defined and documented with expected automated systems in use. However, the E4D evaluation determined that warehousing conditions and inventory management functions are not performing at the same level between the central medical stores (central level) and down to the services delivery points (peripheral level of the healthcare pyramid). Indeed, the SCMS Project started its activities at the central level with substantial support for the construction of an ultra-modern warehouse at NPSP, so called Warehouse-in-a Box or "WiB". The WiB almost doubled the storage capacity at NPSP. Likewise, SCMS upgraded the storage conditions for warehouse "M" including cooling and racking. At the intermediary level, SCMS upgraded 15 districts depots and 5 pharmacies of SDPs; provided District depots in need with AC, thermometers and pallets; and provided some equipment to regional and district offices, and to RIP+ zonal representatives. Unfortunately, many health facilities are still experiencing challenges with correctly warehousing and stocking the ARVs and other associated furniture. Indeed, according to the SCMS FY 15 report, only 37% of the SDP were managing the ARV and associated furniture according to the national guidelines (SCMS CI FY15, Country PMP Report_02162015_final). 12 According to the SCMS Country PMP Report from 20152, the stock-out rate declined continuously from 91% in October-December 2013 to 6% in July-September 2015. The E4D Evaluation found the stock-out rate the day of the survey at 1.9% in a sample of 60 facilities. Over the six months prior to the evaluation day, the stock-out rate was reported at 4.8%. Also, all the KII participants stated that SCMS has significantly contributed in improvement of the storage and dispatching conditions. These conditions include electronic inventory systems, monitoring of the room temperature, hygiene, electricity at the central and intermediary levels. But the SCMS project started the decentralization of its activities to the health facility level during the late 2015 and 2016. According to the SCMS management, interviewed during the assessment, the project was planning to intensify these activities at the health facility level during the following years. 5. Distribution: Best practices require all transportation processes to be clearly defined. The NSCA found the distribution function, for essential medicines, requires overall strengthening. Per the CMM ranking, it remains at the “qualified” stage for essential medicines and vaccines. Per different secondary data sources, the evaluation team confirmed that the on-time delivery rate from central level to lower levels evolved between the end of 2013 and the end of 2014, rising from 28.0% to 78.0%. It later dropped four points, going down to 74.0% in July-September 2015, still far from the projected target of 90%. The recurrent statements derived from the KIIs interviews clearly mention the lack of computerized system at the health facility level (computer, internet, software for the management of the drugs and trainings). Many health facilities are still in need of computers but also a specific software to support the management system. Indeed, the existing software (eSIGL ) is mostly for the procurement and not for the daily management of drugs at the facility level. Therefore, in almost all the health facilities, the management is mainly manual based on the “fiches de stocks”. 6. Waste management: Unusable pharmaceutical products should be disposed-off in accordance with national guidelines, if available, or WHO standards. Based on the CMM ranking, the NSCA found waste management to be one of the lowest ranking among all functional areas with a maturity score at the “marginal” stage, which is characterized by incoherent basic processes that are mostly manual. The E4D evaluation found that SCMS interventions in waste management helped Côte d’Ivoire collect and destroy a significant amount of expired products that were occupying shelves and storage space in warehouses. Approximately, 50 tons of expired ARVs have been collected from district health depots and NPSP. The central level is responsible to destroy all out of date drugs and materials, whereas SCMS ensures the promptness and effectiveness of that activity. SCMS also provides materials (cartons, for example) to facilitate collecting and transporting obsolete drugs. 7. Laboratory: Using the CMM ranking, the NSCA found the laboratory component to be at “marginal” stage, meaning processes are not used in a coherent manner and are principally manual. Most of the time, ongoing laboratory stock outs are attributed to misuse of reagents. This ranking was attributed to the laboratory component even though SCMS provided considerable assistance to laboratory commodity management as this was identified as one of the weakest area in the country. This assistance was provided via the development of pertinent tools and training of main stakeholders. That is, SCMS assistance focused on: 1) supporting Côte d’Ivoire’s Direction des 2 Rapport SCMS CI FY15, Country PMP Report, February 16, 2015. 13 Infrastructures de l’Equipement et de la Maintenance (DIEM) in the development of a national, standardized list of laboratory equipment; 2) supporting main agencies and counterparts on laboratory equipment and its use; 3) training regional pharmacists on laboratory optimization (e.g. equipment maintenance, instrument use. Findings from the KII showed that NPSP roles include estimation of the quantity needed, supervision and coordination of orders as well as providing products to facilities. Regarding laboratory related aspects, findings from KIIs showed that the SCMS trainings have enabled staff to understand different laboratory products but also how these laboratory products should be used and managed. All the interviewed staff reported understanding of the mechanism of “entry and exit” of these products. Nevertheless, this area still faces numerous challenges. Indeed, from the KII conducted as part of the E4D assessment, the main themes emerging that could explain the poor performance of the laboratory component include the following: 1- The SOPs related to basic laboratory processes were not always available at facility level; 2- Personnel training on the SOP and compliance with procedures was not conducted systematically; 3- There was no separate location or guidelines for the management of hazardous chemical products; 4- Some SDPs as well as hospitals and district pharmacies do not have computer or software applications for managing laboratory products and rely on informal systems or paper forms to keep track of the expiration dates of laboratory products. 8. Monitoring the implementation of supply chain activities The SCMS project aimed to provide assistance to Regional and District health offices in the project area to adequately monitor implementation of supply chain activities. The activity aimed also to pilot and document innovative approaches, which have potential for positive impact on the supply chain system. One of the weaknesses observed by the NSCA evaluators in the overall Côte d’Ivoire supply chain system was the absence of an audit system for activities related to waste management mainly for the DPML which does not engage in any annual audits activity with regard to health facilities at peripheral level. The NSCA found the Reverse Logistics capacity to be high at central level (80%) and low at peripheral level (29%) where 41% of facilities (72 SDPs, 11 district pharmacies, and 16 hospitals) did not have reverse logistics processes in place. The E4D evaluation found from KII that SCMS worked with the NPSP and relevant divisions within the Ministry of Health (DPM, LNPS) to progressively establish quality control processes and improve implementation of quality insurance policy. Other key achievements included: • Ensuring routine mechanism is in place for expired HIV/AIDS related products to be destroyed as per SOP to support safe consumption of products by reducing availability of unusable products at all levels; • Providing support for collection and return of unusable HIV commodities to Central level; and • Assisting relevant institutions to develop waste management plan for unusable health commodities according to existing guidelines; 14 Responses to the Other Evaluation Questions: This section addresses the remaining three evaluation questions posed by USAID, which were not covered by the NSCA, yet addressed through the E4D evaluation. The questions are listed under Evaluation Objectives above and repeated and discussed sequentially below. Question 2. How did SCMS, as a technical assistance provider, offer multiple services for HIV/AIDS supplies and supply-related needs to institutions involved in the supply chain management in Côte d’Ivoire? All respondents interviewed at the central level, including Programme National pour le Développement des Activités Pharmaceutiques (PNDAP), the CNCAM and PNLS during the E4D evaluation recognized SCMS’s assistance and achievements. The SCMS TA began at the central level before progressing to health regions and districts, where efforts included the introduction of SOPs for SCM. Technical assistance included trainings on supervision, management (financial, logistic and supply chain), software and computerization, etc. Furthermore, SCMS has contributed in restoring and improving the storage conditions (hygiene and air conditioning). The next phase consisted of strengthening delivery capacities at the service delivery points. This process was still underway during the evaluation, involving SCMS regional offices, supervisory meetings, and monitoring and evaluation at numerous points in the supply chain. Question 3: What measures did SCMS assist to put in place at the Central Medical Stores-NPSP to improve management of antiretroviral and other commodities for HIV/AIDS programs?; All the respondents interviewed during the E4D assessment recognized SCMS assistance and achievements. SCMS support began in 2007 with two main activities: 1) the implementation of a formal quantification process and 2) the implementation of a Logistics Management Information System (LMIS). SCMS has strengthened the NPSP through training on quantification, development of tools and applications for computerized management, technical assistance for storage upgrades, assembling a WIB and renovating numerous pharmacies, and support in the transport of products at the peripheral level in cases of expressed need. SCMS support led to doubling storage capacity at NPSP with the construction of an ultra-modern warehouse built on 4000 square meters. SCMS also advocated for a structural transformation at NPSP, turning it from a public organization to a not-for-profit operated on business principles. Other planned support, including development of an electronic inventory system, skill transfer in procurement and accountability, and training for NPSP laboratory specialists, had not yet begun at the time of the evaluation. In addition, SCMS collaborated with other agencies to improve availability of drugs and commodities. For instance, SCMS collaborates with CDC to supply laboratory products. Likewise, SCMS collaborates with MEASURE Evaluation in the development of the computerized management system and quantification. Question 4: How is the supply chain actually performing at the national, regional and district levels? Performance of the chain varies according to the nature of the products and location. Inventories are well managed centrally and computerized management tools are operational. The NPSP receives products and distributes them to its direct clients (CHU, CHR, HG, district pharmacies, and health facilities in Abidjan) through a requisition process, except for HIV and tuberculosis control products, 15 which are distributed through an allocation system. For all other service delivery points and peripheral￾level health, the supply is allocated based on the monthly average consumption (MAC) and other clinical and epidemiological data at the level of the service delivery point and surrounding area. Performance is better at the central level when compared to the periphery level. KIIs reported low stock out at the central level compared to the periphery level. Furthermore, staff at the central level benefited from more trainings than the periphery level. Conclusion and Recommendations The purpose of this Evaluation for Supply Chain Management System (SCMS) Project was to increase learning about the performance of SCMS interventions in Côte d’Ivoire. The activity aimed to identify key achievements and major challenges encountered during the implementation of the project at the national, regional and district levels. The overall findings of this report are that SCMS has helped the MOHPH make significant progress in strengthening the public health supply chain system at the central level. SCMS Support has also helped transform the Nouvelle Pharmacie de la Santé Publique (NPSP) from a state-owned enterprise into a not-for-profit organization run on the basis of sound business principles. The SCMS project successfully improved the reporting systems, set up the electronic and computerized systems and consequently reduced the stock out rates. The activity also improved the disposal of outdated products and storage systems. Much more efforts are still to be put on the storage and distribution systems at service delivery points level. Key recommendations from the Evaluation are presented in the following tables. 16 Objective 1: Contributions to Logistics Cycle (Effectiveness) Specific Supply Indicator Key Finding(s) Areas of Improvement Recommendation(s) Chain (if any) /Achievement Management Function Warehousing and Stock out rates at 1. Stock out rates at health facilities Electronic systems allow for Inventory central and site levels decreased from 91% in 2013 to 6% in 2015 on time quantification of 1. Train regional and district-pharmacists on Management and to a further 5.3 % at the time of the products and orders. It also computerized inventory management and survey. allows for identification of provide adequate supervision thereof. 2. Despite the low stock out rate, SCMS Districts and sites with could not achieve a 0% stock out rate partly overstock and those 2. Complete the upgrading of all district due to poor inventory and a lack of experiencing stock out pharmacies (82) to the standards of proactivity among pharmacy managers as regularly. Consequently, the organization and storage of health products. revealed by some key informants during the stock-out rate for ARVs has survey. Furthermore, the evaluation team dropped dramatically since 3. Strengthen the operational capacities of detected weak order fulfillment rate at the the start of SCMS. pharmacy departments in health districts by site level. allocating more resources to improve the 3. It was generally observed that stock out performance of LMIS. problems are common in the periphery compared to center of the system. 4. Implement Integrated Management Software at the NPSP level. Warehousing and % of total stock that The FY 15 SCMS report estimated the Continue training of staff for better planning Inventory expired in previous percentage of total stock that expired in the and management of stock. management reporting period previous reporting period declined over the year going from 0.26% July-September 2014 to 0.49% for the quarter of October￾December reaching 0.23% for the quarter of July September 2015. SCMS thus met the target of 1% by the end of 2016. Warehousing and Order fulfilment E4D survey revealed that order fulfilment SCMS collaborated with 1. Review the various Logistic Management Inventory rate. rate was at 73.2% at PEPFAR supported MEASURE Evaluation to Information System (LMIS) currently in use by Management facilities with only 26.8% of facilities indicating develop a computerized different programs and implement integrated order fulfilment problems product management management software at the NPSP level. E4D survey revealed that 20% of health system, e-SIG or e-LMIS. 2. Training on reporting and access to facilities visited reported that their suppliers Deployment of the e-SIG or computers as well as internet and other had experienced transport and logistics e-LMIS began in April 2016 logistics. difficulties. with the training of 317 staff 3. As much as possible, provide The survey also revealed that replenishment from the NPSP direct client transportation support to the districts and 17 process between warehouses is inefficient sites (2 per site including: health facilities. due to the way data is set up in the SAGE district pharmacy, CHR, system; which does not adequately capture CHU, general hospitals, data relevant to fulfillment analysis. peripheral pharmacies CSU, Also, not all facilities have moved to the pharmacies operated by electronic order systems and some staff NGOs and specialized received training on e-SIGL software only institutions, and specialized recently. laboratories). All logistical factors (inventory status, inputs, and outputs) were to be recorded in LMIS. Nonetheless, some challenges exist. Distribution On-time delivery The on-time delivery (OTD) rate from 1. Communicate a delivery schedule to all rate from central to central level to lower levels improved stakeholders to ensure full knowledge of lower levels. between the end of 2013 and the end of delivery times and optimize transport costs. 2014, rising from 28.0% to 78.0%. It later 2 Conduct root-cause analysis of substandard dropped to 4% in July-September 2015. E4D delivery periods to identify bottlenecks. survey reported an on-time delivery of 73.2% 3. Strengthen the distribution capacities of which is below the 90% target. the districts using tools and levers such as The inability of SCMS to meet its on-time distribution plans, follow-up protocols, delivery rate may be explained by the poor reverse logistics, cold-chain management, transportation infrastructure and logistics vehicle maintenance, fuel allocation, etc. from the central level to the periphery. 4. Establish a standardized drug distribution system for health districts that could be funded by the 8% reimbursement of NPSP transportation costs. Quantification Percentage of 1. The SCMS project 1. Review the various LMIS currently in use (Forcasting and facilities submitting E4D survey reported that 62.7% of facilities supports the NPSP health by different programs to establish an Supply Planning) timely and timely submitted a complete LMIS report. Per program management unit integrated, national LMIS. completed LMIS the SCMS report, the timely submission of to collect, analyze and 2. Integrate peripheral stock management reports rate. LMIS reports from lower levels has increased generate on-time LMIS into this automated national ISIGL. significantly, from 16% in October-December feedback reports to clients 3. Strengthen the coordination of 2013 to 54% in October-December 2014, and donors/implementing procurement plans for different products and and then to 82% in July-September 2015. partners. from different sources (NPSP-CIs, programs SCMS was effective for this indicator, as the 2. SCMS also supports and donors). target was 80%. PNDAP to monitor key One cause of delay identified is that the performance indicators on a intervals for ordering and reporting differ quarterly basis in 18 (monthly versus weekly). The misalignment of conjunction with the the LMIS report deadlines and distribution decentralized team, as well schedule deadlines create inconsistency in as to prepare and submit order. quarterly PMP reports for USAID and quarterly PEPFAR reports for the PEPFAR team in Côte d’Ivoire. 3. 115 users have completed e-LMIS training. 114 were deemed competent. Product Selection Percentage/Number Facility-level data collected by E4D showed Availability of LNME, supply Review and periodically disseminate the of projects assisted that approximately half (43.37%) of PEPFAR chain management LNME and at all levels of the health pyramid in country supported health centers visited had an protocols and manuals. organization that LNME. have documented About 71.7% of facilities visited had the and approved presence of supply-chain management protocols/procedure protocols, manuals, and approved s/guidelines for documented guidelines, in addition to LNME. supply chain functions Objectives 2, 3,4: Responses to Other Evaluation Questions Theme Summary of Key Findings Capacity Building and Technical 1. Technical assistance included trainings on supervision, management (financial, logistic and supply chain), software and Assistance (for institutions involved in computerization, etc. It is recommended that these trainings continue especially at the peripheral level where some facilities SCM in Côte D’Ivoire) are still getting used to the software and others have logistics challenges. 2. SCMS has contributed in restoring and improving the storage conditions (hygiene and air conditioning) in many sites. Efforts should be made to maintain standards of storage conditions. 3. Strengthening delivery capacities at the service delivery points. This process was still underway during the evaluation, involving SCMS regional offices, supervisory meetings, and monitoring and evaluation at numerous points in the supply chain. Support provided to NPSP (to 1. Transformation of NPSP from an inefficient organization into an autonomous and results-driven organization. SCMS has improve management of provided management training to the new advisory board and has assisted NPSP in procuring, importing, storing, and antiretroviral and other commodities distributing all PEPFAR-supported commodities for HIV/AIDS programs. for HIV/AIDS programs) 2. Establishment of a sound computerized system. The e-SIG, an automated LMIS put in place generates activity reports combined with purchase orders in paper and electronic formats. Staff can provide an activity report when submitting a purchase order. This should improve forecasting and supply planning at the NPSP. 4. SCMS embedded a long-term consultant within NPSP to support the implementation of various warehousing SOPs and KPIs. Performance in all key functions improved, from goods receipt, to order processing, to product delivery. Among the 19 underperformance issues identified are problems entering product into MACS, irregular stock counts, and delays in transferring stock from the central warehouse to the agency. It is recommended that with regular trainings and improvement in infrastructure and logistics, issues of underperformance will be solved. 5. SCMS collaborated with other agencies to improve availability of drugs and commodities. For instance, SCMS collaborates with CDC to supply laboratory products. Likewise, SCMS collaborates with MEASURE Evaluation in the development of the computerized management system and quantification. Efforts should be made to strengthen such collaborations to further improve management of ARVs and other commodities along the supply chain. 6. SCMS has strengthened the NPSP through training on quantification, development of tools and applications for computerized management, technical assistance for storage upgrades, assembling a WIB and renovating numerous pharmacies, and support in the transport of products at the peripheral level in cases of expressed need. 7. SCMS support led to doubling storage capacity at NPSP with the construction of an ultra-modern warehouse built on 4000 square meters; an initiative which should improve storage for ARVs and other commodities. Status of supply chain at the National, 1. Inventories along the supply chain are well managed centrally and computerized management tools are operational. SCMS Regional and District levels. set up a real- time CMM reporting system for use with inventory and supply orders of PEPFAR- sponsored products from the intermediary level to the central level. It is recommended that these reporting systems are strengthened, with the implementation of an electronic inventory management system at all warehousing sites when it is fully developed. 2. Allocation and distribution of products within the supply chain has improved and is better at the central level compared to the periphery. However, for Laboratory products, central level components and pharmacists in the laboratory have not been adequately strengthened resulting in repeated shortages of laboratory products. It is recommended that a specialist pharmacist is put on Laboratory products at the NPSP to strengthen and ensure that supply and distribution of laboratory products improves. Furthermore, all SOPs related to laboratory supply chain should be in place and should be aligned to local and national guidelines. 3. Capacity building in logistics management, computerized management of the supply chain by the districts. This has resulted in better monitoring and distribution of ARVs at the District levels. However, in terms of trainings, staff at the central level benefit more than those at the periphery. Thus, priority should be given to peripheral areas in future. 4. SCMS has helped Districts collect and destroy obsolete products. Waste destruction of products is done centrally and SCMS ensures this is done promptly and in accordance with national guidelines. However, commodity expiration is problematic. SCMS is collaborating with MEASURE to support the MOHPH to implement an effective electronic logistics management system to improve traceability of commodities. 20 I. INTRODUCTION AND BACKGROUND The Supply Chain Management System (SCMS) project in Côte d’Ivoire is a Cooperative Agreement (CA) between the United States Agency for International Development/ Côte d’Ivoire (USAID/CI) and PFSCM, a nonprofit organization established by the JSI Research & Training Institute and Management Sciences for Health (MSH). The project Award Number is GPO-I-00- 05-00032-00, and the total budget for the 12-year implementation period (2005-2017) is $300,000,000. Established in 2005 as part of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) to deliver HIV/AIDS commodities and to help strengthen supply chains in Côte d’Ivoire, the SCMS project is closed at the end of 2016. The USAID/West Africa’s (USAID/WA’s) Evidence for Development (E4D) project conducted an independent, external performance evaluation from September 15 to November 30, 2016, with the aim to increase learning about the performance of SCMS interventions in Côte d’Ivoire. It serves as an end￾of-project evaluation of SCMS interventions in Côte d'Ivoire. USAID Health Office is interested to know whether the SCMS project has achieved its intended results. Jointly with SCMS and other donors, the Ministry of Health and Public Hygiene (MOHPH) conducted an assessment of the national supply chain (NSCA) for pharmaceutical products, including HIV commodities in 2015. That report examined the capacity and maturity of each management function of the logistic cycle and describes the status of the supply chain at national and sub-national (region and district) levels. The SCMS final performance evaluation, the focus of this report, complements and builds on the NSCA assessment and does not repeat the measurement of key supply chain management functions’ capability maturity, which is not expected to have changed significantly since March 2015.3 These results and findings will inform future evidence- based programming and decision making by the MOHPH, PEPFAR, and other donors and help prioritize their continued support to strengthening the supply chain system. This report organizes the evaluation findings under two major sections. The first section, “Contributions to the Logistics Cycle, ” seeks to identify the effect of SCMS interventions on key logistics cycle functions. This responds to the first of four questions posed by USAID to guide this evaluation, which are outlined in Section 2.3 (page 20). The second section, “Responses to the Remaining Evaluation Questions” responds to the three other evaluation questions, which address technical assistance (TA), commodity management at the Central Medical Stores-NPSP, and to assess actual supply-chain performance at national, regional, and district levels. 1.1 LOGISTICS CYCLE The provision of high-quality, affordable, anti-retrovirals (ARVs) and related products at service delivery points (SDP) is essential to successful HIV/AIDS programs. Thus, logistics systems, which facilitate product selection, forecast demand, inform the mobilization of necessary financing, procure commodities in a timely manner, and deliver products to clients on a reliable basis, are an essential element of HIV/AIDS programs everywhere. To support these programs, USAID has invested PEPFAR 3 Using the cluster sampling method, NSCA selected a representative sample of 6 regions out of 20, 12 districts out of 82, and 325 primary care health centers out of 2,116. To achieve a statistically significant 95% confidence interval at the primary health care level, 177 primary health care facilities were randomly selected, in proportion to the variety of health facilities within the districts (for example, rural health centers, urban health centers, or urban specialized health centers). 21 resources in improvement of countries’ logistics systems for HIV/AIDS commodities. In Côte d’Ivoire, the logistics cycle for HIV/AIDS commodities comprises a range of supply chain management (SCM) functions as shown in figure 1, below. Those management functions are organized as follows: 1) Product selection: led by the La Direction de la Pharmacie, du Médicament et des Laboratoires (DPML) with the participation of other actors in the system. 2) Quantification (forecasting & supply planning) and procurement, which are initiated by Programme National de Lutte contre le Sida (PNLS) and NPSP and carried out by a quantification committee in collaboration with DPML and PNDAP; 3) Inventory management (storage, distribution) under the responsibility of NPSP that received, stored and distributed the products; 4) Logistics management information systems (LMIS) covering pipeline monitoring, organization and staffing, budgeting, supervision, and evaluation, and used to reliably gather and analyze key information essential to delivering quality services for all HIV/AIDS product users. 5) Serving customers: carried out by service delivering points such as hospitals, clinics and health facilities; 6) Quality monitoring at each step, supervised by NPSP and PNLS. Figure 1: The logistics cycle. Source: JSI/DELIVER 2004 The complexity of HIV/AIDS programs’ logistics systems arises from the involvement of multiple stakeholders from different system components. ARV and other HIV/AIDS-related products are often provided by international manufacturers and are usually financed through multiple sources, including governments, donors, and international development organizations. The procurement process often requires international procurement agencies to comply with local policies as well as procedures set forth by the donors. Warehouse and transportation system managers manage the products storage and 22 the distribution, whereas health workers provide ARV- and HIV/AIDS-related products to clients. For the supply chain to function successfully, all of the involved organizations and managers must share information and closely coordinate their activities. In Côte d’Ivoire, SCMS supported all supply chain functions except the provision of direct services to patients (the “serving customers” function). SCMS did not create a parallel, independent system but rather provided its support through government channels and systems. The NPSP receives products and distributes them to its direct customers: hospitals - Centre Hospitalier Universitaire (University Hospital Center, CHU), Centre Hospitalier Régional (Regional Hospital Center, CHR), Hôpital Général (General Hospital, HG)), district pharmacies, and health facilities in Abidjan. NPSP supplies these customers via a requisition process for all goods except HIV and tuberculosis control and treatment products, which are distributed through an allocation system. Figure 2 illustrates the product and information flow from the central procurement agency to health facilities at the lowest level of the supply chain. Figure 2: The national distribution channel of ARV- and HIV/AIDS related laboratory products and flow of logistics information. Source: JSI/DELIVER 2004 23 1.2 PROJECT OVERVIEW The 2014 census indicates that Côte d’Ivoire has an estimated total population of 22,848,945, of whom 51.7% (11,716,826) are males and 48.3% (10,954,505) are females. According to UNAIDS4 in 2014, Côte d’Ivoire had approximately 470,000 people living with HIV (PLHIV), including 250,000 female adults, 30,000 children aged 0 to 14 years, plus 400,000 orphans and vulnerable children (OVC), and 50,937 pregnant women needing ARVs. The same source indicated that approximately 26,000 new HIV infections and 25,000 deaths from AIDS are reported every year. Mother-to-child transmission contributes significantly to the HIV epidemic with 11.6% of infants born to HIV-infected mothers infected (GARP report, 2011). In 2005, the government of Côte d’Ivoire sought support from PEPFAR, through the SCMS project, to ensure a reliable, cost-effective, and secure supply of products for HIV/AIDS programs. Before SCMS, shortages and stock-outs of commodities caused dangerous breaks in treatment among patients and inflated the cost of providing timely, effective treatment. Emergency orders wasted money on rush fees and high freight costs. Lack of inventory control wasted valuable commodities due to product expiration, improper storage, and theft. Poor coordination led to redundancies and gaps in service. Since its inception in 2005, SCMS worked closely with the Côte d’Ivoire MOHPH to maintain a global supply chain to ensure a reliable supply of quality HIV/AIDS medicines for people living with HIV/AIDS (PLHIV). The project supported the central pharmacy (NPSP) in procuring, importing, storing, and distributing all PEPFAR-supported commodities for HIV/AIDS programs. In Côte d’Ivoire, SCMS provided technical assistance to the MOHPH through the Central Medical Stores- NPSP and other departments of the Ministry. Overall Project Goal and Components Goal: The primary goal of SCMS was to provide a reliable, cost-effective, and secure supply of products for HIV/AIDS programs in PEPFAR-supported countries, including Côte d’Ivoire. This included : ● Establishing and operating a safe, secure, and reliable supply chain; ● Buying and distributing high-quality ARVs, low-cost essential medicines, HIV test kits, laboratory supplies, and other products; ● Strengthening the capacity of national supply chains to ensure long-term sustainability; and ● Supporting supply-chain collaboration and information sharing among global and local partners in the HIV/AIDS community. Components: SCMS helped build the supply chain capacity in two key areas: infrastructure and human resources. 1- Infrastructure: SCMS helped equip warehouses with modern racking, security, forklifts, cold rooms, and computerized inventory systems. The project also installed innovative modular systems—called warehouse-in-a-box, storage-in-a-box and clinic-in-a-box—in several countries, including Côte d’Ivoire. 2- Human Resources: SCMS provided traditional training programs and partnered with universities to provide pre-service training and to build human-resource capacity. Increasingly, SCMS and ministries engaged private-sector firms to store and distribute commodities. 4 http://aidsinfo.unaids.org/ 24 Management Structure The Partnership for Supply Chain Management (PFSCM) is a non-profit organization established in 2005 by two of the leading international health consultancy organizations in the U.S.: John Snow Incorporated (JSI) and Management Sciences for Health (MSH), both of which are also in Côte d’Ivoire. To deliver its services, PFSCM draws on the capabilities and experience of 13 organizations5 that are among the most trusted names in international public health and development, each of which offers unique capabilities, including procurement, freight forwarding, and technical assistance. II. EVALUATION PURPOSE AND QUESTIONS 2.1 EVALUATION PURPOSE The purpose of the Performance Evaluation for Supply Chain Management System (SCMS) Project is to increase learning about the performance of SCMS interventions in Côte d’Ivoire. It serves as an end-of￾project performance evaluation of SCMS interventions in Côte d'Ivoire. USAID/WA Health Office is interested to know whether the SCMS project has achieved its intended results. Furthermore, this evaluation will help in identifying and addressing critical gaps in evidence for supply chain strengthening activities. Likewise, it will inform the Government of Côte d'Ivoire’s National Health Plan 2016-2020 and future program design and implementation for supply chain management in the country. The evaluation will also serve to document key contributions by the Government of the United States of America to supply chain management in Côte d’Ivoire. The target audiences for the SCMS performance evaluation are the U.S. Embassy in Abidjan’s Front Office; the USAID Côte d’Ivoire Health Office, the PEPFAR Team and Program Office; the Government of Côte d’Ivoire Ministry of Health (Nouvelle Pharmacy of Public Health (NPSP)); USAID/West Africa (USAID/WA), USAID/Washington (USAID/W), the Global Fund and other donors in the health sector. 2.2 EVALUATION PERIOD AND GEOGRAPHIC FOCUS This final project performance evaluation covered activities implemented from September 2005 to December 2015. The team conducted the evaluation from September 15 through November 30, 2016. A final presentation of findings is planned for December 2016. Data collection was carried out in the regions of Abidjan, San Pedro, Bouake and Korhogo, all of which have a high HIV burden and are supported by USAID, as specified in the Statement of Work (SOW) found in Annex I. 5 JSI; MSH; Booz Allen Hamilton; Crown Agents (USA and UK); i+solutions; Imperial Health Sciences; The Manoff Group; MAP International; North-West University; Northrop Grumman; UPS Supply; Chain Solutions; Voxiva; 3i Infotech 25 2.3 EVALUATION QUESTIONS At USAID’s request, this evaluation focused on performance (effectiveness and efficiency) rather than population-level impact (long-term changes or benefits) and used the standard evaluation criteria including: effectiveness, efficiency, relevance, impact, and sustainability. USAID suggested four guiding evaluation questions intended to identify changes that occurred due to SCMS as well as to identify other factors that contributed to desired changes. The evaluation questions were, in order of priority: 1- What was accomplished and what were the challenges encountered during the implementation of the project at the national, regional and district levels regarding: a. Computerized commodity management system and reporting systems? Technical Area: Logistics Management Information System b. Integrated electronic inventory management tool? c. Integrated HIV/AIDS product management into the broader pharmaceutical supply chain? Technical Area: Warehousing and Inventory Management d. Collection and destruction of expired HIV/AIDS commodities? Technical Area: Warehousing and Inventory Management e. Prevention of stock out of tracer HIV/AIDS commodities at service delivery points? Technical Area: Warehousing and Inventory Management and Logistics Management Information System 2- As a technical assistance (TA) provider, did SCMS offer multiple services i.e., were beneficiaries able to access all the supply chain technical assistance they needed through a single project (aka “one-stop-shop”) for the Government of Côte d’Ivoire MOHPH and AIDS for HIV/AIDS supplies and supply-related services? 3- What measures did SCMS help put in place at the Central Medical Stores-NPSP to improve management (including risk management and waste reduction) of antiretroviral and other commodities for HIV/AIDS programs? What remains to be done? 4- How is the supply chain actually performing at the national, regional and district level? What are the maximum and minimum stock levels for each level of the supply chain management system (central, district, local)? III. EVALUATION DESIGN AND METHODS 3.1 EVALUATION DESIGN This evaluation uses a cross- sectional and non-experimental design. The study relies on primary and secondary data using a mixed approach. Secondary data include quantitative indicators from the NSCA study, the Performance Management Plan (PMP) and other programs reports, whereas primary data encompass information from 14 key informants Interviews (KII), 10 Focus Group Discussions (FGDs) as well as quantitative data (based on structured questionnaire) from 60 service delivery points in four regions: Abidjan (45 sites), San Pedro (5 sites), Bouake (5 sites) and Korhogo (5 sites). The evaluation process was divided into three major phases: preparation, fieldwork, and reporting. Preparation phase: During this phase, the team reviewed systematically the project-related documentation provided by USAID/CI and by SCMS (see Annex II). The process aimed to obtain insight into the program’s goals, objectives, key indicators, progress made each year, managerial and managerial processes, and program 26 strategies and approaches. In addition, the team met with USAID/CI in order to discuss project background and major developments as well as to align on expectations, roles, and responsibilities for the evaluation. Consequently, the team developed semi-structured interview guides and proposed an agenda for the evaluation, which was refined based on information provided by USAID/WA and USAID/Côte d’Ivoire. Findings of the documents’ review provided responses to questions related to effectiveness, efficiency and alternative strategies. Fieldwork phase: The second phase focused on fieldwork in Côte d’Ivoire: KIIs (see list in Annex III), visits and surveys at selected PEPFAR-supported sites, and FGD with PLWHIV. The sites visited were selected based on criteria for representative geographical, epidemiological, and technical coverage of the project. Reporting phase: During the reporting phase, the team analyzed the quantitative and qualitative data collected and met with USAID/CI to share preliminary findings and discuss the development of the final report, submission schedule, and dissemination plan. 3.2 SAMPLING The evaluation used a three-stage sampling approach to select the 60 sites. First, the team selected seven districts among PEPFAR-supported districts with the highest percentage of PLHIV in four regions: San Pedro (4.3%), Bouake (3.0%), Korhogo (2.5%)—and Abidjan (5.1%). Table 1 reports the selected districts in the four considered regions. Table 1 – List of selected district N° District Region 1 Yopougon-Est Abidjan 2 Abobo Ouest Abidjan 3 Marcory-Treichville Abidjan 4 Koumassi-Port Bouet-Vridi Abidjan 5 Grand Lahou Abidjan 6 Bouaké Bouaké 7 Korhogo Korhogo 8 San-Pedro San Pedro Source: E4D SCMS Evaluation, 2016 Within the four regions, at the second stage, USAID/CI randomly selected 60 sites (45 in Abidjan and 5 in each of the 3 other regions). It should be noted here that the prevalence is for the Regions. Within the 4 regions, the SCMS intervention were in 82 districts including 3 districts with the same name as those regions (the same name for the capital towns of those 3 regions). In each selected district, the body of service delivery points sampled included regional and district hospitals, public health facilities, and private clinics. In each sampled district, the team interviewed health-department directors and district pharmacists. 27 Three strata were considered: “aggressive scale- up,” “scale-up to saturation,” and “sustained” PEPFAR support for the Project’s fiscal year 2016. Last, the team selected medical stores and health facilities associated with the selected stores randomly taking into account gamut of options as services are provided by the public sector, by the private sector and NGOs. The evaluation team used unequal probability sampling to select districts, and simple probability sampling to select health facilities within districts. The evaluation team collected data at the central and sub-national levels, encompassing the medical store, district hospital pharmacies and Service delivery points (Etablissement Sanitaire de Premier Contact - ESPC). 3.3 DATA SOURCES Data sources included secondary data and primary data collected in four regions: San Pedro (4.3%), Bouake (3.0%), Korhogo (2.5%) — and Abidjan (5.1%). Primary data consisted of qualitative data from 14 KIIs and 10 FGDs and quantitative data from 60 health facilities. 3.3.1 Qualitative Data In Total, the team conducted 14 KIIs and 10 FGDs. To bring structure and consistency to KIIs, the team developed and used specific KII guides to probe informants’ perceptions of the supply chain’s evolution and SCMS’s technical-assistance contributions. At the central level, informants were persons with supply-chain expertise and SCMS experience, affiliated with institutions such as PNLS, PNDAP, Measure JSI, NPSP, ICAP, RETROCI, CCM-Global Fund and SCMS. At the peripheral level, interviews targeted district medical officers and the regional and district pharmacists. Additionally, the team conducted 10 FGD with PLWHIV (4 in Abidjan and 6 in the 3 regions of Bouake, Korhogo and San Pedro). To ensure data quality, the evaluation team employed three strategies. First, the team trained fieldworkers to systematically record all interviews both in writing and with tape recorders; and interviewers transcribed KIIs and FGDs immediately following the interviews. Secondly, the evaluation team triangulated KII and FGD notes with interview recordings. The team members found that transcriptions followed the sequence of questions defined in the interview guide. Lastly, the team applied content- analysis methodology to organize KIIs and FGDs by topic and isolated each participant’s opinions thereof. In total, ten (10) KIIs were completed before conducting FGDs (the remaining KIIs were conducted later) so as to inform and guide the discussions. Institutions interviewed or collaborating with SCMS (as KIIs) Table 2 presents SCMS partners visited by the E4D Evaluation team during the assessment. Overall, the evaluation team visited nine SCMS partners. The team conducted KIIs with the leadership staffs from these institutions. Table 2 – Institutions Visited Name Description National Created by decree in December 2008, PNDAP leads pharmaceutical activities on behalf of the Program for the MSLS, as described in the pharmaceutical policy. PNDAP coordinates, monitors, and evaluates Development of implementation of the national program intended to make quality, affordable drugs available at Pharmaceutical all levels. SCMS works with PNDAP on strategic plans, which are renewed every four years, Activities to train public pharmacists in quantification, refurbish deteriorated offices and pharmacies, and 28 Name Description (PNDAP) to collect supply chain performance indicators. SCMS and PNDAP collaborate with other partners, such as UNFPA and MEASURE on training of regional and district pharmacists in quantification and supply chain and logistics management. PSP – Transformed into NPSP in November 2013, this is the country’s central store in charge of Transformed into receiving, warehousing and distributing all pharmaceutical products in Côte d'Ivoire, including NPSP ARVs. NPSP serves two types of clients: the CHUs, Specialized National Institutes, HG and FSUCOM of Abidjan; and rural facilities across the 82 districts, which are operated by CS, NGOs, and faith-based organizations. SCMS work with NPSP focused on quantification and procurement. MEASURE An American NGO, Measure Evaluation develops software applications for monitoring and Evaluation evaluation of health programs. Measure supported SCMS by introducing supply chain information systems for quantification. In 2014, Measure was tasked with incorporating electronic signatures (e-SIG) in NPSP reform. This led to the development of e-LMIS with SCMS. In this joint venture, Measure was responsible for IT development and financing side while SCMS oversaw technical SCM. The partnership with Measure stood out as one of the most seamless among SCMS PNLS As the MOHPH implementing body for HIV/AIDS policy and programs, PNLS has the mandate to: a) coordinate the activities of the national response to HIV/AIDS; b) propose a strategic policy and specific approaches to the fight against HIV/AIDS; and c) provide technical assistance to the promotional, preventive, and curative plans to reduce the mortality and morbidity associated with AIDS. Retro-CI Retro-CI was established in collaboration between the MOHPH and the US Centers for Disease Control and Prevention (CDC) to conduct operational research on the response to HIV-2 and coinfection. Retro-CI is also the CDC reference center for molecular tests (viral load, PCR, genotyping) and supports indoor laboratories through procurement of supplies beyond the scope of SCMS and pursuant to biosecurity (e.g., gowns, hygiene, and work environment). With the establishment of SCMS, Retro-CI narrowed its focus to quantification, although, in 2008, Retro-CI regained responsibility for design, development, and implementation of its laboratory management software, which was integrated into the LMIS. ICAP A division of the Columbia University Mailman School of Public Health, active in Côte d'Ivoire since 2007, ICAP provides implementation support to the holistic care of PLHIV. From October 2013, ICAP has worked in the Loh Djiboua and Worodougou districts out of Abidjan, six districts of Agnebi Tiassalé, the CHU of Cocody, and the IPCI (Clinical unit based at the CHU). ICAP has helped establish networks of pharmacists and has organized the supply systems for regional and district pharmacies. In order to solidify collaboration between ICAP and SCMS, SCMS engaged with ICAP at all relevant operational levels, ensuring effective connections and reliable ARV access among private pharmacists and health facilities. CCM-Global The Country Coordination Mechanism (CCM) is the national coordinating body for the Global Fund: Fund, responsible for coordinating proposal development and other funding mechanisms for malaria, tuberculosis, and HIV/AIDS. 29 Name Description CDC The PEPFAR-supported leader in the fight against HIV/AIDS, the CDC, began operations in Côte d'Ivoire in 1998 with a research station and the establishment of the Côte d’Ivoire Retrovirus Project (CDC Retro-CI). As described above, CDC Retro-CI produces evidence to help shape the response to HIV/AIDS. SCMS has also held joint procurement-planning meetings with the CDC to organize the supply of laboratory products. The SCMS-CDC collaboration was initially impeded by miscommunication and resistance to feedback, but later became successful after the parties involved had clarified roles and responsibilities. SCMS The subject of this evaluation, this USAID-funded activity was designed to help the MOHPH to strengthen commodity logistics management and to build the capacity of MSLS staff in order to increase the availability of pharmaceutical products at service delivery points. Skills transfer was a critical activity, with the objective of MOHPH staff gaining the capacity to implement activities independently, without the support of MSLS staff. Source: E4D SCMS evaluation, 2016 Focus Group Discussions (FGDs) The evaluation team conducted ten FGDs through Réseau Ivoirien des Organisations de PVVIH (RIP+), a network of organizations of PLHIV. RIP+ undertakes integrated, comprehensive programs to build the capacity of local organizations to provide care and support to PLHIV in Côte d’Ivoire. In 2013, together with UNAIDS and the International Foundation for Therapeutic Solidarity, RIP+ initiated the establishment of an early alert system, Système d’Alerte Précoce du RIP+ (SAR), and developed a set of reporting tools. SAR enables fast reporting on drug stock-outs found at SDPs and enables district pharmacists, regional pharmacists, and NPSP to identify challenges and to take corrective actions to avoid recurrence. Through a memorandum of understanding signed in March 2014 with RIP+, D-SCMS utilizes SAR to improve the availability of HIV medicines and other essential goods at SDPs. Regional SCMS offices worked closely with RIP+ zonal delegates to collect data and feedback, which allowed for a more efficient and rapid response in the case of stock-outs or technical challenges, as well as identification of fundamental weaknesses in the supply chain. The SCMS project provided for zonal delegates and members of RIP+, training on supply chain management, implementation, and usage of SAR reporting tools . Additionally, SCMS procured IT and communication equipment and hosted regional coordination meetings for RIP+ delegates. The collaboration between SCMS and RIP+ is a good example of how close cooperation between the public health system and civil society can work together to benefit patients. Making SAR an integral part of the LMIS optimized the management of antiretroviral drugs and other HIV medications and ensured continuous access to HIV medicines and services at all levels of the public health supply chain. 2.4 QUANTITATIVE DATA The evaluation team developed a questionnaire for administration at the 60 selected PEPFAR-supported service delivery points. The evaluation team worked with two E4D’s Recipient Groups (RG), INSP and ASAPSU, to collect data. Prior to data collection, the Evaluation team trained fieldworkers and conducted pretests of the questionnaire from October 18 to 19, 2016. Training of Data Collectors The Evaluation Team Leader, Senior Evaluation Expert and Subject Matter Expert co -facilitated field staff training. Trainees were experienced researchers and in some cases they had formal training as statisticians. Topics addressed included the background presentation of SCMS Project as well as on E4D 30 and the context of the evaluation. It included discussions on the evaluation objectives and methods. The facilitators described the questionnaire; explained the field manual and how to use it. There were also data collection simulation exercises. The evaluation team developed an itinerary for data collection and discussed data collection logistics. Annex VI reports the questionnaire used to collect quantitative data at the facility level. Table 3 describes the training agenda. There were three evaluation teams, including one team leader, three data collectors and one data transcriber. Data collectors were assigned to teams based on three criteria: 1) good knowledge of the region and districts to be surveyed; 2) background experience in health surveys and field data collection, and 3) gender balance. Each team had at least two females. Table 3 – Training Agenda October 2016 Time Activity Presenters Tuesday, October 18th 9:00 – 9 :30 - Introduction of Consultants and Participants Dr. Pierre Marie 9 :30 – 10 :30 General information on evaluation (quantitative and qualitative Dr. Esso data collection activities) 10:30 – 11 :00 Coffee Break 11:00 – 11 :30 - SCMS Project Presentation Dr. Pierre Marie 11:30 – 12 :30 Review and Discussion of the Interview Guide Dr. Coulibaly 12:30 – 2:30 Lunch Break 2:30 – 3 :30 Review and Discussion of the Focus Group Discussion Guide Dr. Coulibaly 3:30 – 4 :00 Coffee Break 4:00 – 5 :30 Explanation of the interviewer's manual and Dr. Esso / Dr. Coulibaly review of the questionnaire 5 :45 End of Day 1 Wednesday, October 19th Time Activity Presenter 9 :00 – 10 :30 Review of the Questionnaire Dr Esso / Dr Coulibaly 10:30 – 10 :45 Coffee Break 10 :45 – 12 :30 Interview Simulation Dr Esso 12:30 – 2:30 Lunch Break 2:.30 – 4:30 Focus Group Discussion Simulation Dr Esso 4:30 – 4:45 Coffee Break Pilot Survey Debriefing Dr Pierre Marie 4:45 – 5 :45 Review of Logistics Dr Esso Review of Data Collection Agenda Dr Coulibaly 5:30 End of the Training Source: E4D SCMS Evaluation, 2016 All the three teams visited the 45 sites in Abidjan, including Grand Lahou, and collected data (using questionnaires, KII, and FGD) on October 20, 21, and 22, 2016. Each team then traveled to one of the three regions on Sunday, October 23. A small group of data collectors stayed behind and continued to collect data in Abidjan during this period. The Research Assistant cross-checked all completed 31 questionnaires to ensure data quality. In total, 60 health centers were visited and surveyed; nine district pharmacists and seven health district directors were interviewed; and 10 FGD were conducted (4 in Abidjan and 6 in the 3 regions). All questionnaires addressed the site laboratory, where applicable. Each team received supervision from one of the three evaluation experts. At the end of every data collection day, experts debriefed data collectors in order to provide guidance, as needed. To ensure data quality, in addition to the training of field staff and close supervision, the team crosschecked and validated all filled questionnaires. Furthermore, research assistants performed systematic checks and the team used double data entry. In addition, the team conducted descriptive analyses to identify outliers, assess the completeness of the question as well as proportion of missing values. 2.5. DATA ANALYSES Qualitative data analysis The transcription of focus group discussions and individual interviews began immediately after the interview was completed. The team leader supervised this process. Qualitative method comprised content and thematic analyses using Atlas Ti software. Data analysis included the following activities: • Using verbatim remarks and having sentences as the unit of analysis • Grouping results by key areas of interest; • Identifying different positions in relation to each important topic. • Summarizing each position and assessing its strength or degree of importance.; Quantitative data analysis: Health Facilities survey The team conducted data entry on CSPRO. Quantitative analysis relied on descriptive statistical methods, including trend analysis and interpretation of proportions using STATA software. Data analysis was consistent with the evaluation framework. The database includes 45 health centers in Abidjan and 5 facilities in each of the 3 regions of Bouake, San Pedro and Korhogo. Of the sites visited, 61.7% were public health facilities, 28.3% were private clinics, and 10.0% were NGOs-run facilities. Access to electricity, running water, telephone and paved roads are useful for effective functioning of the supply chain system. At the time of the visits, almost all the centers visited had functional electricity (96.7%) and running water (93.3%). The team observed existence of landline telephone in 61.7% of facilities. The majority of surveyed facilities (68.3%) had paved road access as shown in Figure 3. Storage of some drugs requires constant supply of electricity and regular attention to empty the water containers. Electricity is also required for computers. In addition, paved road and telephone might contribute to prevent stock out by facilitating communication and transportation. 100 96.7 93.3 Figure 3: 90 Percentage of Health Centers visited with specified characteristics. E4D SCMS evaluation, 2016 a c c e s s 80 68.3 70 61.7 60 ha vi ng 50 % 40 30 20 10 0 Electricity Clean water Paved road Phone 32 Considering respondent’s characteristics (average 6.7 years of experience), the number of years of experience ranges from 0.5 year to 16 years. The large majority of respondents (70.0%) were pharmacy specialists (pharmacist, assistant pharmacist, preparatory manager in pharmacy). IV. EVALUATION FINDINGS This section presents findings on the effectiveness of SCMS support to MOHPH and local organizations involved in the management of the HIV-commodities supply chain. This report organizes the evaluation results and findings under two major sections. The first section, “Contributions to the Logistics Cycle,” seeks to identify the effect of SCMS interventions on key logistics cycle functions listed earlier. It responds to the first evaluation question on progress achieved toward the project’s objectives (effectiveness). The second section, “Responses to the Other Evaluation Questions,” responds to the remaining three evaluation questions: capacity- building and TA provided to local institutions; support provided to NPSP; and the current status of the supply chain at the national, regional and district levels. Although the data available are discontinuous, preventing efficiency-ratio calculation for the entire duration of the project (2005-2015), it is possible to assess the achievement of selected targets within the period of 2013-2015. This presentation of findings first addressed the national-representative facilities sampled, where both PEPFAR and non-PEPFAR activities operate together. Subsequent discussion may address findings of the E4D evaluation sample. This report uses the common supply chain-maturity terminology to classify the maturity of functional practices as “advanced,” “qualified,” or “marginal.” This terminology is derived from the national supply chain assessment (NSCA) tool developed by SCMS, the USAID|DELIVER project and the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) project. The tool has two components – one quantitative and one qualitative - which allow the maturity and the performance of the supply chain to be assessed separately and then compared to each other. 3.1 CONTRIBUTION TO LOGISTICS CYCLE (EFFECTIVENESS): A well-performing logistics cycle is critical to the success of HIV/AIDS programs. SCMS interventions focused primarily on strengthening the logistics cycle in Côte d’Ivoire in order to ensure consistent availability of high-quality, affordable HIV/AIDS commodities at service delivery points. From 2005 to 2013, SCMS focused on strengthening NPSP at the central level, which oversees product selection, quantification, procurement, and warehousing. SCMS concentrated training and capacity building activities on four priority areas: a) general management of medicine supply chain including coordination, and policy development; b) ARV supply forecasting, quantification, and procurement; c) management of pharmaceuticals, including laboratory testing kits and; d) development and installment of LMIS (BIOS, MACS). 3.1.1 Product Selection Product Selection is the entry point to the logistic cycle. Key tools for product selection include the National Essential Medicine List (LNME) and Standard Treatment Guidelines (STGs), which are validated by the MOHPH. Best practices call for the LNME to align with the STG, for the relevant national 33 pharmaceutical and therapeutic committees to update both documents periodically, and for dissemination of both across all levels and stakeholders, including supply chain managers. The NSCA study found product selection to be at the “advanced practices” stage, with well￾defined processes and integrated technologies. The E4D KII confirmed this score. In the last NPSP international request for proposals, launched in 2014 and related to the purchase of essential medicines and consumables, 95% of purchased commodities were listed on the LNME. This shows that purchases conform to national policy on essential medicines. It is important to note that the development of LNME is not SCMS mandate. However, SCMS should ensure those lists are present at supported facilities and properly used for product selection. At the peripheral level, however, only 18% out of the 325 facilities visited by the NSCA study had an updated LNME. Facility-level data collected by E4D show that approximately half of PEPFAR-supported health centers (48.37%) visited had an LNME. Sampling methods may explain the discrepancy among these reports, as the E4D sample represents only PEPFAR-supported facilities, rather than a national sample. One interpretation of the disparity (18% at the periphery in the NSCA study vs 48.37% in the periphery through the E4D study) is that PEPFAR support may have contributed to the increased use of LNME. The E4D evaluation also explored the extent to which PEPFAR-supported facilities had access to other protocols, including the Standard Treatment Guidelines (STG) and the Approved Supply Chain Management Guide (ASCM). This evaluation found that, 71.7%% of PEPFAR-supported facilities had LNME, STG, and ASCM. 3.1.2 Forecasting and Supply Planning Quantification (Forecasting and Supply Planning): Forecasting calculates the quantities of products that priority disease programs will require to ensure continuous availability. Forecasting is conducted annually for the following 24-month period. A supply plan documents the delivery schedule necessary to: 1) ensure that adequate stocks are available to satisfy consumption needs, and 2) maintain continuity of the distribution system. Best practices require an efficient mechanism for archiving, reporting, collecting, reviewing, and analyzing data in order to improve forecasting and supply planning. The NSCA found the forecasting and supply planning maturity to be at the “qualified” stage for essential medicines, including HIV commodities. This reflects generally well-defined and documented practices that incorporate certain automated systems. Forecasting and supply planning was a priority focus of SCMS activities pursuant to the goal to reduce the stock-out rate to 0% The E4D team evaluated forecasting and supply planning performance in terms of the indicators related to the LMIS. Facilities submitting timely and complete LMIS reports: Per the SCMS report, the timely submission of LMIS reports from lower levels has increased significantly, from 16% in October-December 2013 to 54% in October-December 2014, and then to 82% in July-September 2015. SCMS was effective for this indicator, as the target was 80%. However, the SCMS report noted a decrease from 87% in April-June 2015 to 82% in July-September 2015 (figure 4 below). Efforts should be made to reverse this trend. 34 One cause of delay identified is that the intervals for ordering and reporting differ (monthly versus weekly). NPSP customers follow the ordering schedule received from NPSP, which is set on a weekly cycle, but the reporting cycle for these customers is monthly. Therefore, up to half of the reports (ordering and reporting) need to be submitted before the end of the month, which creates confusion and delays in reporting. In Côte d’Ivoire, LMIS reports and orders are submitted to NPSP in a single document, but each component has its own deadline. LMIS reports have a fixed deadline of the 5th of each month, while order deadlines vary within a 28-day period based on client location (in one of five zones) and the timing of deliveries to that location, according to NPSP’s distribution schedule. The misalignment of the LMIS report deadlines and distribution schedule deadlines create inconsistency and puts NPSP in a disadvantaged position relative to its own benchmarks. The process is further constrained by the absence of a validation mechanism at both NPSP and the regional level for orders that are placed. submitted reports % of facilities that 100 80 60 40 20 0 80 87 82 65 49 54 Target Jul-Sep 2014 Oct-Dec 2014 Jan-Mar 2015 Apr-Jun 2015 Jul-Sep 2015 Quarter Figure 4: Percentage of Facilities Submitting Timely and Complete LMIS Reports to the Central Level. Source: Rapport SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_final The SCMS project supports the NPSP health program management unit to collect, analyze and generate on-time LMIS feedback reports to clients and donors/implementing partners. SCMS also supports PNDAP to monitor key performance indicators on a quarterly basis in conjunction with the decentralized team, as well as to prepare and submit quarterly PMP reports for USAID and quarterly PEPFAR reports for the PEPFAR team in Côte d’Ivoire. Percentage of monthly logistical data reports with a passing audit score Overall, the percentage of monthly logistics data reports with a passing audit score remains stable but still very far from the target of 85.0%. The E4D assessment explored a related issue on which health facilities submitted a completed and timely monthly supply chain report. Data from facilities’ survey show that though 68.3% submitted a report on time, only 62.7% of facilities visited met standards (submitted the completed report on time). These data corroborate the conclusion of the national survey. 35 % submitted 100 90 80 68.3 70 62.7 60 50 40 30 20 10 0 Submitted report on time Submitted complete report on time Submission of reports Figure 5: Health Center Report Submission: on Time and Complete Non-SCMS staff trained in supply-chain functions Findings from KII revealed that the general supply-chain challenges found primarily at the regional and district levels were due to the lack of adequately trained staff that could effectively manage medications at local-level health facilities. Only a small number of health facilities had trained pharmacists at the management level, but staff managing ARVs did not have sufficient training. The Figure 6 below shows that at the end of 2015, SCMS had trained 156 non-SCMS staff, compared to 787 staff targeted initially. This high gap might have negative effect on the SCMS functions for these staff. 36 Number of staff trained 800 787 750 700 650 600 550 500 450 400 350 300 250 200 156 150 100 50 0 Target FY2015 Oct-Dec 2015 Figure 6: Number of Non-SCMS Staff Trained and Deemed Competent in Supply Chain Functions. Source: E4D SCMS Evaluation, 2016 Country-counterpart ownership demonstrated in quantification and supply planning Per the SCMS report CI FY 15, the percentage of activities under the “Shared Counterpart Leads” and “Wholly Counterpart” categories completed by local entities remains stable, at 86%, over the last nine months considered, and is above the target, which was 75%. All the KII participants mentioned that although members of the committee responsible for the quantification of laboratory products understand the software used, the committee needs additional practice using the software. This will increase their proficiency. All key informants reported the need for laboratory data quality improvement. Furthermore, according to KII, majority of laboratory staff are not trained. In addition, there are lack of data culture and appropriate equipment for data storage and reporting (computers). 3.1.3 Procurement The SCMS offers a full range of items necessary for HIV/AIDS prevention, care and treatment programs, including antiretroviral (ARVs), medicines for opportunistic infections such as tuberculosis, rapid HIV test kits, laboratory commodities, and a host of other products. To meet the needs across the country and leverage economies of scale, the activity has created a consolidated procurement mechanism that feeds into local supply chains. The result is lower prices for products of assured quality, along with increased cooperation between suppliers and recipients. Procurement at the central level is based on standard product specifications and a reference list of items. Ideally, qualified personnel should update these two elements periodically. An electronic procurement system should be used to monitor requests for bids, orders/awarded contracts, fulfilled orders, and payments made to suppliers. The NSCA found the procurement of essential medicines to be at the “qualified” stage, which means processes are well defined and documented and some technology is used. SCMS worked intensely on procurement and shares credit for the success in this area. In Côte d’Ivoire, findings from the KII report are: (1) SCMS is using a computer-based procurement system, which includes a list of all essential drugs and commodities; (2) all requests and orders are submitted monthly online; (3) pharmacists are responsible for procurement planning; (4) orders and requests are done for four months of provision. 37 3.1.4 Warehousing and Inventory Pharmaceutical warehousing or warehouse management is “the physical movement of stock into, through, and out of a medical store warehouse. Warehousing is a key element of pharmaceutical supply chain management. It ensures the constant availability and flow of essential quality health commodities, in appropriate quantities, in a timely and cost- efficient manner, through the supply chain system. Key warehousing functions include receiving and storing stock, inventory management, and distribution management. Maximum (max) and minimum (min) stock levels should be established for each level of the supply chain. To maintain stock between the min and max, reordering should be done systematically by the forecasting and supply-planning service or through a stock management system, which uses pre-defined values for min and max levels. Product management should follow the principle of ‘first expiry, first out’ (FEFO) to avoid expiries. Best practices in infrastructure consist of making sure that there are no expired or unusable products taking up storage space in the warehouse or storeroom and that the arrangement of existing zones and/or the flow of products is evaluated for efficiency and security. Products should be stored on shelves or pallets and be correctly labeled. To ensure best storage practices, an organized system for monitoring products is critical. The NSCA study found warehousing and inventory practices to be at the “qualified” stage, meaning that processes were well defined and documented and certain automated systems were in use. However, the NSCA study also found that warehousing conditions and inventory management functions deteriorated between the central medical stores and the service delivery points at the peripheral level of the healthcare pyramid. SCMS efforts to improve storage conditions for pharmaceutical products included renovating district pharmacies (a project that began in FY 15) and supporting the destruction of expired products, in order to prevent clutter in existing storage rooms or the unsafe consumption by the public. In FY 14 and FY 15, SCMS embedded a long-term consultant within NPSP to support the implementation of various warehousing SOPs and key performance indicators. The E4D evaluation investigated the extent to which facilities documented the different supply chain functions. Figure 8 below illustrates key findings from the 60 health centers surveyed, including the crucial observation that 57% of facilities face problems with supply transport. This was the most frequent problem reported by facilities visited. Purchasing plans also emerged as a critical area of concern: 52.5% of the facilities surveyed by E4D did not have purchasing plans. In an effort to capture the state of supply-chain documentation beyond warehousing infrastructure, E4D went further than the NSCA assessments. Documentation on those topics (also graphed in Figure 7), including waste management and inventory management, shows an average performance, and scores on warehouse management and procurement are even higher. These findings from quantitative survey are consistent with KII reports which indicated that not all facilities experienced stock out, and reported good inventory system. However, about 4 key informants reported transportation challenges. 38 Waste management 59.3 Distribution/ Transportation 42.4 LMIS Inventory management 62.7 Storage stock 69.5 Supply 81.4 Quantification and purchase prediction 47.5 0 10 20 30 40 50 60 70 80 90 Figure 7: Percentage of Facilities with Documentation of Warehousing and Inventory Practices. Source: E4D SCMS Evaluation, 2016. Order fulfillment rate The SCMS FY 15 Report concluded that the rate of order fulfillment rose from 74.0% to 80.0% between 2013 and 2014, moving in the expected direction to reach the target of 90%. However, in the five quarters for which data are presented in Figure 8, below, there are variations from quarter to quarter that show a downward trend. This is particularly apparent in the last quarter, when order fulfillment decreased from 73% in April-June to 57% in July-September 2015. At the end of the period considered, the order-fulfillment rate is more than 30 percentage points below the target of 90%. These data reflect inefficiency partly due to weaknesses in the replenishment process between warehouses. Notably, data collection is impeded by the way that data is set up in the SAGE system: maximum value, updated average month of stock (CMM), which does not adequately capture data relevant to fulfillment analysis. 39 Order fullfiment rate % 100 80 60 40 20 0 90 82 83 68 73 57 Target Jul-Sep 2014 Oct-Dec 2014 Jan-Mar 2015 Apr-Jun 2015 Jul-Sep 2015 Quarter Figure 8: Order Fulfillment Rate. Source: Rapport SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_final. The results from the E4D survey are different, and show an improvement when comparing national data to data obtained at PEPFAR-supported facilities. In the latter, the order fulfillment rate was 73.2%; with only 26.8% of facilities indicating order fulfillment problems. The E4D evaluation team explored potential delays in fulfillment, finding that one-fifth of the health centers (20%) visited reported that their suppliers had experienced transport and logistics difficulties (vehicles, fuel and maintenance) getting products to facilities on time. Furthermore, it is likely that all facilities have not yet moved totally to the electronic order systems. Indeed, one Key Informant reported that their staff received training on E-SIGL software only recently. Another stated that the electronic system is not yet active in their institution. Percentage of total stock that expired in previous reporting period The FY 15 SCMS report determined the percentage of total stock that expired in the previous reporting period, finding that the rate of expiry declined over the year going from 0,26% July- September 2014 to 0.49% for the quarter October-December 2014 and then was stable at about 0.05% over the last three quarters of the fiscal year, reaching only 0.23% for the quarter July-September 2015. Because the E4D survey could not measure the total stock of each product at each health center, it was not possible to calculate the overall rate of expired stock. Instead, E4D assessed expiry rates for 15 products and combinations of products at 60 sites, and findings revealed that 6.6% of those products had expired. Even if the denominators are different, this finding is relevant as it shows higher percentage of expired stock compared to what was reported in the SCMS FY 15 Report. Because the E4D team only focused on 15 tracer products, and given the attention and efforts consented in this area, the E4D detected percentage of expired stock should have been lower than that presented by SCMS. Percentage of sites managing ARV commodities that meet acceptable storage conditions According to the SCMS report, the percentage of facilities managing ARV commodities with acceptable storage conditions, as defined by national standards, increased from 36 % to 48% in December 2015. The end-of-the-project target was 85%. Table 4 shows reports of overall good storage conditions. However, efforts should be made to improve fire safety and to make sufficient space available. Findings from KIIs support the contribution of SCMS in improving storage conditions (rehabilitation of rooms and air-conditioning). 40 E4D evaluation findings on storage conditions: Table 4 – Conditions of Storage Conditions % Products ready-to-distribute are arranged in such a way that identification labels and expiry and / or manufacturing dates are visible 95.0 Cartons and products are in good condition and are not damaged. If cartons are opened, the products are not wet or cracked due to heat 93.3 The center always separates damaged and / or outdated products from good products and removes them from stock 98.3 Cartons and products are protected from water and moisture at all seasons. 98.3 The storage area is free of insects and rodents 81.7 The storage area is secured (locked with key) but is accessible during normal working hours, with limited access to authorized personnel 95.0 Space and organization are sufficient for existing products and for possible expansion 55.0 Fire safety equipment is available and accessible 45.0 The products are stored and ranged to facility identification counting of out of date products as well as general management 95.0 Products are protected from direct sunlight 91.7 The roof is maintained in good condition to prevent the penetration of sunlight and water 98.3 The storage room is kept in good condition 90.0 Products are stored at least 10 cm above the ground 95.0 The products are stored at least 30 cm from the walls and other storage piles 71.7 Source: E4D SCMS Evaluation, 2016 Stock-out: The stock-out rate declined continuously from 91% in October- December 2013 to 6% in July￾September 2015. This is a drastic reduction, but in terms of project effectiveness, it is still below the target of 0.0%. From July-September 2014 to July-September 2015, data show a slow trend toward reduction of stock-outs, from 8% to 6%. The period Oct-Dec 2014 stands out with an even lower stock-out rate of 4%, while for the quarter January-March 2015 the stock-out rate reaches 8% and then 7% for the quarter April-June 2015. Analysis of the quantitative data collected by the E4D evaluation team found the stock-out rate the day of the survey at 2.1% in a sample of 60 facilities. For a more representative perspective, the evaluation team also assessed stock outs over the period of 6 months prior to the evaluation day, finding a stock￾out rate of 5.3%. The team observed that a longer recall period increases the likelihood of finding reported stock outs and, thus, discourages comparison between a single-day stock-out report and one spanning six months. 41 Findings from qualitative surveys report that stock-out was uncommon over the most recent period. One out of 14 KII reported no stock-out over the last 5 to 10 years. This is also consistent with findings from FGDs. Low rate of stock-out is due to the electronic inventory systems and collaboration between districts. The electronic systems allow on-time quantification of products and orders. It also allows to identify districts with overstock. Nevertheless, stock-out problems are more common in the periphery than at the center of the system and that they may emerge at the periphery even when no similar problem may exist at the central level. Notably, the central-level findings come from qualitative data while a quantitative survey informed peripheral-level findings. The SCMS target for stock-out was 0%. While the reported stock-out rates vary, no data source suggests that the 0% target was achieved. In interviews with the evaluation team, key informants alluded to poor inventory and a lack of proactivity among pharmacy managers as causal factors for the persistence of stock-outs. In some cases, the evaluation team also detected weak order fulfillment rates at the site level. Average Duration of stock-outs of tracer commodities The SCMS project collected data on the duration of stock outs of tracer commodities, which indicate considerable variation across the quarters measured, from July-September 2014 through July-September 2015. Figure 9 below reports average stock-out duration, in number of days, across this period, indicating a decrease, from 3.5 to 1.7 days, during the first two quarters, before increasing to 10.4 days in April- June 2015. Although the final quarter assessed, July-September 2015, had very low stock-out duration (1.4 days), this remains above the project target of zero days. 42 12 10.4 10 8 5.5 6 3.5 4 1.7 1.4 2 0 Target Jul-Sep 2014 Oct-Dec 2014 Jan-Mar 2015 Apr-Jun 2015 Jul-Sep 2015 Quarter Figure 9: Average Duration (Days) of Stock Outs of Tracer Commodities at the Site Level. Source: SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_final. Percentage of facilities stocked according to plan According to the SCMS FY 15 report, there is a very low proportion of facilities that stocked according to the plan of 60% of stock, using tracer commodity-dispensing. The results presented below in Figure 10, show also many inconsistencies and variations of the indicator by quarter. According to the respondents from the KIIs, this situation is essentially due to the “manual” management at the health facilities. Indeed, the Health Facilities place the orders based on their “fiches de stocks” and therefore do not have in place a system of alert. Furthermore, the physical conditions described for most of the health facilities do not enable them to stock large quantities of products. 43 70 60 60 50 40 30 16 14 19 20 13 8 10 0 Target Jul-Sep 2014 Oct-Dec 2014 Jan-Mar 2015 Apr-Jun 2015 Jul-Sep 2015 Quarter Figure 10: Percent of Facilities Stocked According to Plan. Source: SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_final. 3.1.5 Transportation Measures Distribution: Best practices require all transportation process elements to be clearly defined. To ensure that all parties are aware of expected delivery times, a clearly written, detailed delivery schedule must be disseminated regularly. Vehicles used to distribute products should be customized to maintain product stability and packaging integrity. All options for the “last kilometer” distribution should be considered (own fleet, vehicle rental, and/or outsourcing distribution). NSCA found the distribution function for essential medicines requires overall strengthening. It remains at the “qualified” stage for essential medicines and vaccines. On-time delivery rate (from central level to lower levels) The on-time delivery (OTD) rate from central level to lower levels improved between the end of 2013 and the end of 2014, rising from 28.0% to 78.0%. It later dropped to 4% in July-September 2015. Even during better-performing periods, this indicator remained below the projected target of 90% (Figure 11). This can be explained by the transportation infrastructure from the central level to the periphery, which is not properly developed. According to the E4D survey, a significant percentage (31.7%) of health facilities reported not having access to paved roads which constitutes an impediment for on-time delivery of essential medicines and vaccines. Some KIIs reported transportation problems and expressed need of having a pick up or a truck for logistic purposes. 44 100 90 90 78 80 70 70 60 50 40 30 20 8 10 4 0 Target Oct-Dec 2014 Jan-Mar 2015 Apr-Jun 2015 Jul-Sep 2015 Quarter Figure 11: On-Time Delivery (OTD) Rates. Source: SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_final. 3.1.6 Quality Management Waste Management Unusable pharmaceutical products should be disposed-off in accordance with national guidelines, if available, or WHO standards. A dedicated location for unusable pharmaceutical products should have limited access, be secured, clearly identified, and marked. Although the percentage of total stock that has expired is consistently low, the NSCA study found waste management to be one of the lowest￾performing functional areas, with a maturity score at the “marginal” stage, which is characterized by incoherent basic processes that are mostly manual. A review of SCMS Quarter 4 Country Performance Monitoring Report (2015) shows that the teams have continued to focus their efforts on the transfer of close-to-expired products and reducing the quantity of expired stock; yet commodity expiration remains problematic. Furthermore, the SCMS report indicates that approximately 50 tons of expired ARVs have been collected from district health depots and NPSP. SCMS support for destruction of expired products has primarily consisted of collecting and transporting expired products to incineration sites identified by the project. The Government of Côte d’Ivoire, particularly the MOH, appreciates this activity because it has helped the country collect and destroy a significant number of expired products that were occupying shelves and storage space in warehouses. Other donors, including the European Union, also contributed finances to this very important waste management activity. In an interview with key informants from MEASURE, they acknowledged that SCMS also helped the DPML develop a process for managing unusable products. The project developed, printed, and distributed a manual; defined circuits for the deposit of unusable products; and conducted an assessment to identify a structure capable of destroying large molecules. In addition, SCMS has been tasked to partner with MEASURE in supporting the MOHPH to implement an effective, electronic logistics management system (eLMIS) to improve traceability of commodities. Monitoring and Benchmarking 45 SCMS assisted regional and district health offices in the project area to adequately monitor the implementation of supply chain activities. Data from KIIs revealed that the project also worked with health officers to pilot and document innovative approaches that had potential to improve the supply chain system. As this document describes, supply-chain performance has varied over time, across levels, and among the various SCM functions. Of particular note, the percentage of facilities stocked according to plan has been consistently low, according to the SCMS Quarter 4 country Performance Monitoring Plan Report (2015). Despite the identification and implementation of corrective actions throughout the project, this indicator has underperformed, remaining far below its target of 60%. KIIs revealed also that SCMS facilitated the supervision of the health facilities and warehouses but was not conducting parallel (external) supervision, which remains the role of districts. 3.1.7 Laboratory SCMS provided considerable assistance to laboratory commodity management as this was identified as one of the weakest areas in the country. SCMS assistance focused on instructing NPSP and counterparts on laboratory equipment and its use. The project also trained regional pharmacists on laboratory optimization (e.g. equipment maintenance and instrument use). More importantly, in 2012, SCMS supported Côte d’Ivoire’s Direction des Infrastructures de l’Equipement et de la Maintenance (DIEM) in the development of a national, standardized list of laboratory equipment. In spite of these efforts, an assessment of laboratory storage security showed continuous weakness. Criteria included doors with locks, windows with iron bars, a limited number of individuals with access keys, and a policy requiring that one of the key-holders be present at all times. SOPs aligned to local and national guidelines, for all processes related to the laboratory supply chain, were not always in place. The NSCA found the laboratory component to be at “marginal” stage, meaning processes are not used in a coherent manner and are principally manual. The NSCA team indicated that most of the time, ongoing laboratory stock outs are attributed to misuse of reagents. Themes emerging from KII that could explain the poor performance of laboratory include the following: 1. The SOPs related to basic laboratory processes were not always available at facility level; 2. Personnel training on the SOP and compliance with procedures was not conducted systematically; 3. There was no separate location or guideline for the management of hazardous chemical products; 4. Some SDPs as well as hospitals and district pharmacies do not have any computer or software applications for managing laboratory products and rely on informal systems or paper forms to keep track of the expiration dates of laboratory products. 3.1.8 Project efficiency Efficiency measures cost effectiveness: it indicates whether implementation of a project’s activities used budgeted funds reasonably. An objective analysis of efficiency requires access to both the budget and the actual expenditures per activity, in order to track changes in expenditures over time. A lack of these data has limited the ability to conduct this analysis, however, what little financial information the evaluation team could access showed a very low budget consummation rate, generally below the 85% forecasted (primarily for product purchase). For 2012, total expenditure on products was 72.1% of the total budget allocated (Table 5.) 46 Table 5 Summary of Purchases COP 11 - SCMS CI in 2012 in $US Name of Programmatic domain Expended Allocated Expenditure rate (%) 01-MTCT Prevention: PMTCT Consumables $2,872,765 $4,797,239 59.9% HMIN, Injection Safety $155,252 $212,152 73.2% 08-HBHC Care: Adult Care and Support CD4 $2,624,746 $14,389,657 18.2% 09-HTXS Treatment: Adult Treatment CD4 $1,597,654 $8,240,453 19.4% 10-PDCS Care: Pediatric Care & Support $588,123 $1,493,896 39.4% Cotrimoxazole 14-HVCT Care: Counseling and Testing $7,864,216 $12,670,100 82.2% Consumables 15-HTXD ARV Drugs ^ ARVs $58,133,140 $59,255,094 98.1% 16-HLAB Laboratory Infrastructure $1,458,190 $3,085,168 47.3% Equipment HVOP, STIs $335,114 $636,157 52.7% HVSI $75,026 $218,650 34.3% HBHC - Food by Prescription $40,548 $100,000 40.5% TOTAL $75,744,774 $105,098,566 72.1% Source: SCMS-CI-July-sept 2012_sk 5Nov2012 The above table shows that some budget lines are severely underused while other expenditure rates exceeded 90.0%. Paradoxically, laboratory-product purchases have not reached 50% of their budget, yet one quarterly report states: (Unfortunately, information from the 2015 COP could not be compared here). [When] it comes to needs in CD4 equipment bio-chemistry, hematology and other general equipment (Centrifuges, refrigerators, freezers, microscopes, and micropipettes), the total amount of the PMOs was $2,548,900 USD, compared to $1,396,630 USD available to SCMS for this purchase on the HLAB line. To close the gap of $1,152,370 shortage in the budget, discussions are under way with headquarters to consider the possibility of a budget readjustment using surplus funds of reagents and consumables available on other lines such as HTXS and HBHC. (SCMS-report Quarterly PEPFAR FY2013) Similarly, the 2007 SCMS quarterly report stated that: “There is an urgent need to prioritize activities and allocate funds to support them" (SCMS Côte d'Ivoire _quarterly_report _01_12_2007_v2) suggesting that uneven progress toward budget targets has been a pattern throughout the project. Highlights of the SCMS project performance at the end of 2015 The following summarizes key findings from the evaluation team with reference to the first evaluation question, which states that: 1- What was accomplished and what were the challenges encountered during the implementation of the project at the national, regional and district levels regarding: a. Computerized commodity management system and reporting systems? Technical Area: Logistics Management Information System b. Integrated electronic inventory management tool? 47 Technical Area: Logistics Management Information System c. Integrated HIV/AIDS product management into the broader pharmaceutical supply chain? Technical Area: Warehousing and Inventory Management d. Collection and destruction of expired HIV/AIDS commodities? Technical Area: Warehousing and Inventory Management e. Prevention of stock out of tracer HIV/AIDS commodities at service delivery points? Technical Area: Warehousing and Inventory Management and Logistics Management Information System ➢ Computerized commodity management system and reporting systems SCMS set up a real-time CMM reporting system for use with inventory and supply orders of PEPFAR-sponsored products from the intermediary level to the central level. In addition to the CMM toll, SCMS collaborated with MEASURE Evaluation to develop a computerized product management system, e-SIG or e-LMIS. Deployment began in April 2016 with the training of 317 staff from the NPSP direct client sites (2 per site including: district pharmacy, CHR, CHU, general hospitals, peripheral pharmacies CSU, pharmacies operated by NGOs and specialized institutions, and specialized laboratories). All logistical factors (inventory status, inputs, and outputs) were to be recorded in LMIS. However, challenges with data reliability persist. Specifically, there are issues with consumption reports, inventories (primarily MAP-generated stock status), and the consistency of CMM with clinical and epidemiological data. ➢ Integrated electronic inventory management tool There is no inventory management tool yet. SCMS has equipped all the PPS involved in the management of HIV activities with computers and Internet connections and has provided stock management training. An inventory management tool is under development by other donors, including UNFPA and SIDEP2, in collaboration with SCMS. According to SCMS, the next step and a key priority for the next project is to implement an electronic inventory management system at all warehousing sites. ➢ Integrated HIV/AIDS-product management into the broader pharmaceutical supply chain HIV/AIDS-product management is integrated into the broader supply chain of pharmaceuticals at both central and peripheral levels. SCMS built a modern warehouse for NPSP; rehabilitated 15 pharmacies; equipped 73 public facilities with air conditioners, metal shelves, wall thermometers, and refrigerators; and trained warehouse managers. This support to the supply chain facilitated the integration of HIV/AIDS products management into the broader system. ➢ Collection and destruction of expired HIV/AIDS commodities Prior to SCMS, expired ARV products occupied a problematic volume of storage space and their presence posed a threat to patient safety. Today, the proportion of stocked ARVs that are expired is very small. SCMS helped the DPMN develop a process for managing unusable products and developed, reproduced, and disseminated a manual outlining that process. SCMS defined circuits where unusable products must be deposited and provided clear instructions on reducing the risk of product expiration by avoiding over-storage. One review identified the firm RMG as the only one capable of destroying large molecules. To date, under the leadership of SCMS, three destructions of expired products have taken place. However, destruction of expired products places a huge financial burden on the country; thus it has become problematic. Currently, there is no mechanism in place to collect expired products on a regular basis; this is done in an ad-hoc manner. 48 Prevention of stock out of tracer HIV/AIDS commodities at service delivery points SCMS supported MSLCS in setting up the CNCAM to coordinate forecasting activities; develop, monitor, and validate procurement plans; mobilize financial resources; and align use of ARVs and strategic inputs with NAC guidelines. SCMS has also improved the reliability of data collection (through LMIS) to allow better analysis and decision support. Issues related to the supply chain security of ARVs and strategic inputs were also addressed. Consequently, the stock-out rate for ARVs has dropped dramatically since SCMS start. This unfortunately is not yet the case for laboratory products, as this is beyond the scope of SCMS. Laboratory products are managed by a different entity of the MOH. Table 6: Selected Performance Indicators N Indicators 2014 2015 Survey Target (2016) 1 Stock out rate (central level and site level) (%) 91.0 6.0 5.3 0.0 2 % of facilities submitting timely and complete LMIS 6.0 82.0 62.7 80.0 reports to the central level (%) 3 % of total stock that expired in previous reporting 0.019 0.023 NA 1.0 period (%) 4 Order fulfillment rate (%) 82.0 57.0 NA 90.0 5 On-time delivery rate from central level to lower 78.0 4.0 73.2 90.0 levels (%) % of supply chain functions documented in standard 6 operating procedures (SOPs) at SCMS-supported 43.0 75.0 70.2 75.0 facilities (%) 7 % of SCMS-managed product categories with NA 100.0 47.4 100.0 coordinated procurement plans (%). %/# of project assisted in country organizations that 8 have documented and approved 75.0 75.0 26.7 TBD protocols/procedures/guidelines for supply chain functions Source: E4D SCMS Evaluation, 2016 As indicated in Table 6 above and in the Q4 country PMP for FY 15, key selected achievements of the SCMS project at the end of 2015 were: - Stock out rates at health facilities decreased from 91% in 2013 to 6% in 2015. - 115 users have completed e-LMIS training, 114 of whom were deemed competent. - The percentage of total stock that expired in the previous reporting period is still very low, 0.023%. - The duration of stock-outs of tracer commodities decreased from 13 days in 2013 to 1.4 days in 2015. 49 - 89% of district and 85% regions completed their supervision plans. - The percentage of sites managing ARV commodities that meet acceptable storage conditions increased from 24% in 2013 to 48% in 2015. - The percentage of monthly logistical data reports with a passing audit score increased from 52% in 2014 to 62% in 2015. Despite these accomplishments, the expected targets were not achieved for all indicators. The E4D team assessment, using a sample of facilities, did not necessarily find the same level of accomplishments as those detected by previous data collection efforts. To date, SCMS has successfully strengthened the national system in many of the key functional areas, including forecasting and supply planning, procurement, warehousing (infrastructure, processes, people, and systems), capacity building, and LMIS. However, it should be noted that during the same period, between 2013 and 2015: - Client satisfaction, measured by order fulfillment rate, decreased from 74% to 57%; - The OTD rate between NPSP and the lower level drastically decreased from 70% to 4% between the 3rd and 4th quarters of 2015; - The percentage of facilities stocked according to plan decreased from 19% to 8% between the 3rd and 4th quarters of 2015; - COTD for FOMP decreased from 91% to 83% between the 3rd and 4th quarters of 2015; and - Vendor on-time delivery (VOTD) for FOMP decreased from 93% to 67% in the same period. 3.2 RESPONSES TO THE OTHER EVALUATION QUESTIONS 3.2.1 Institutional Capacity Building and TA to Local Institutions All persons interviewed recognized SCMS assistance and achievements. SCMS first strengthened the technical capacity of the NPSP at the central level before extending to health regions and districts that have now integrated SOP for SCM. The third phase of TA strengthened delivery capacities at the point of service delivery. This process is still underway through the regional offices of SCMS, involving supervisory meetings and monitoring and evaluation of the various functions of the supply chain. According to one of the Key informant: “ . . . the fight against HIV became effective in Côte d'Ivoire, starting in 2004 with major NGOs. Each had its own management style, overseeing and managing products themselves. In 2008, they were asked to hand everything to the NPSP. It is precisely at that time that the support of SCMS benefited NPSP. They have highlighted quantification as a process, which begins with data control - product specification - hypotheses - hypotheses review. The second part was the development of the procurement plan. Ever since Cote d’Ivoire adopted the common basket, these quantities are allocated by the donor and by period ". 3.2.2 Technical Assistance to the NPSP In 2013, SCMS supported NPSP’s transformation from a complex, bureaucratic, and highly inefficient operation (under PSP) to an autonomous and results- driven organization (renamed NPSP). SCMS provided management training to the new advisory board that was created to oversee the organization following the transformation and also assisted NPSP in procuring, importing, storing, and distributing all PEPFAR-supported commodities for HIV/AIDS programs. Likewise, SCMS worked with NPSP to develop and implement a system that provides timely information about inventory and consumption of commodities at PEPFAR-supported districts and sites. 50 The SCMS provided technical assistance to Côte d’Ivoire’s National Quantification Committee (NQC) with the focus on reinforcing the Committee’s knowledge of the Quantimed®, PipeLine®, and ForLab tools to perform quantification of ARVs and lab commodities. Additionally, SCMS worked with DGS and PNLS to develop and disseminate a simplified document with quantification and hypothesis guidelines for both ARVs and lab products, so that stakeholders have clear instructions for collecting the data required for quantification using existing tools. Key informants interviewed confirmed that SCMS helped the NPSP establish a sound computerized management system, starting with MACS and later replaced by SAGE in 2014, with an option for warehouse management. SCMS put in place the e-SIG, an automated LMIS that generates activity reports combined with purchase orders, in paper and electronic formats. This means that staff must provide an activity report when submitting a purchase order. Expanded support to NPSP included increasing of storage capacity and improving computer equipment. Specifically, SCMS renovated two stores, including L and M buildings, and supplied pallets and air conditioners. As stated by informant MO, "We now have an integrated management system.” Interviews showed that NPSP was fully satisfied with SCMS support. In FY 14 and FY 15, SCMS embedded a long-term consultant within NPSP to support the implementation of various warehousing SOPs and KPIs. Performance in all key functions improved, from goods receipt, to order processing, to product delivery6. Among the underperformance issues identified are problems entering product into MACS, irregular stock counts, and delays in transferring stock from the central warehouse to the agency. 3.2.3 Current Status of the Supply Chain at the National, Regional and District Levels Inventories are well managed centrally or computerized management tools are operational. The NPSP receives products and distributes them to its direct customers (CHU, CHR, HG, district pharmacies, and health facilities in Abidjan) through a requisition process, except for HIV and tuberculosis control products, which are distributed based on an allocation system. For all other service delivery points and all health districts at the peripheral level, supply allocation is calculated based on the CMM and clinical and epidemiological data at the level of the service delivery point. Supply-chain performance varies according to the nature of the products: • For ARVs that are used exclusively for the treatment of HIV/AIDS, funding is available through multi-year agreements between technical and financial partners; gaps that require additional resources are quantified and stock-out and expiry rates are limited. • For ARVs that are used both for HIV/AIDS and Hepatitis B treatment, quantification takes into account both pathologies in order to minimize stock outs. However, it is challenging to monitor the use of the specific portion of products allocated by PEPFAR. • For rapid testing and laboratory consumables (TRC), numerous discrepancies between users and supply mechanisms persist. For hematology and biology reagents, reporting is poor. It is important to integrate this reporting in the LMISL. In addition to consumption reporting, MAPs must be equipped with a SAGE-type ERP for 6 For example, the previously clogged NPSP, which displayed service levels below 30% and delays of four weeks for order processing had, by January 2014, improved OTD to 95%. SOP were rolled out for 184 best practices. 51 inventory management. The technical assistance and general support to regional pharmacies and districts can be found not only in training, but also in warehousing, storage, supervision and coordination. As PBN, one of the persons interviewed pointed out: “The needs covered by SCMS include: training, coaching, and the reproduction of management tools. At the supervisory level, SCMS has provided financial resources for supervision per diems, fuel, improved storage conditions including shelving, installing air conditioning and the rearrangement of the storage room. The SCMS project has created a platform for coordination meetings…” According to PAO: “It should be noted that the SCMS project has addressed several challenges which for us are (now) more or less covered, such as putting in place good coordination and a better follow -up of ARV distribution. Also, support needs for reporting in terms of promptness and completeness of reports that has been well covered. Without forgetting the provision of specific tools and the training of agents in the use of these tools” Despite the achievements of the SCMS project, interviewees identified some needs. These are usually at the level of computer hardware: computer tools, Internet connection, cold chain, transport vehicles. In this regard, PYE pointed out for example that: “Talking about the remaining challenges, there are many in this case: the purchase of computers to computerize our service, printers and a refrigerator to keep cold products. It should be noted that the expansion of our building is critical for storing products and the sound management of the needs of the peripheral centers. The lack of coverage for these needs is mainly due to the fact that most part is out of SCMS responsibility and decisions are taken by superior hierarchy” According to PAO: “We are in the public service and every year there are new people who are assigned to our services and others who leave and we will have to continue to support the capacity building sessions even internally. Also, the need for a computerized management system is not fully covered in our service because even if some agents of the sites have been trained, computers will have to be available so that they can use them”. When questioned about the needs not yet covered, PSA stated: “We need computers, because even the use of LMIS requires a computer, and we have not received any SCMS computers. It's my own computer that I use. We did not have an Internet connection while LMIS requests a connection. SCMS has given us a fleet of communication without cellular devices to facilitate communication between districts. We received a monthly phone credit of 35 000 francs to communicate with our peripheral sites because they do not benefit from the fleet. Moreover, the days of supervision are insufficient, normally it takes one day to supervise a site, but SCMS provides only 11 days for all the sites and it is really difficult”. 3.2.4 Technical Assistance to Health Districts In 2013, following a series of technical discussions with the MOHPH of Côte d’Ivoire, USAID, and PEPFAR, SCMS offered to assist the MSLS in strengthening the logistics management of health products at the local level. The Decentralized Supply Chain Management project (D-SCM) provides technical assistance to reduce stock-out rates, delivery delays, logistical shortcomings, and improves the system of product distribution from district depots to service delivery points (SDPs). 52 To improve the effectiveness of the management system, D- SCM introduced an intermediary level of supervision – the position of regional pharmacists. This role bridges the gap between the 82 district pharmacies and the central level. SCMS positioned five sub-offices in Abidjan, Bouaké, Abengourou, Man, and Gagnoa for implementation of the D-SCM. These offices work closely with regional and district pharmacists to increase the availability of HIV/AIDS products and essential drugs at SDPs by focusing on capacity building of regional pharmacists, health district pharmacy managers, regional and general hospitals, and health centers. SCMS provided assistance to the Health Districts in various areas, including training, risk management and waste management. The vast majority of facilities visited confirmed that they received assistance from SCMS, including training, vehicles for product transportation when critically needed, office space refurbishment, air conditioner repair, collection and destruction of expired drugs, and provision of computer when needed. For example, SCMS financed the rehabilitation of the Divo Pharmacy, which suffered extensive fire damage in 2015, and the Daloa Pharmacy, which required significant attention. SCMS put a special emphasis on building staff capacity in the areas of LMIS, communication, and coordination to avoid duplicated efforts. The project also worked to harmonize and standardize treatment protocols at SDPs. In FY13, SCMS supported the redesign of a manually-integrated ARV and lab LMIS for HIV commodities, trained more than 350 users of the LMIS systems for lab supplies and ARV/OI drugs, and upgraded the existing BIOS tool to include new information captured on the updated forms. Together, the LMIS and BIOS form the functioning paper-based system. SCMS also financed the MACS warehouse management system at NPSP’s central warehouse (this support will continue in FY 16). In FY 15, SCMS supported MEASURE in its implementation of an e-LMIS at 50 pilot sites, all NPSP clients. Even though SCMS was not directly tasked with transporting and distributing ARVs, the project purchased vehicles for districts in critical need and provided electronic equipment for better monitoring and follow- up of drug distribution and consumption. All people interviewed acknowledged improvements in the information system as well as among the distribution mechanisms. SCMS provided assistance to the Health Districts in various areas, including training, risk management, and waste management, as stated by DBN: “SCMS has also made available to us the funding of trainings, coaching, accommodation, shelving, air conditioning, transport of medicines, the collection and destruction of obsolete products”. DAO indicated that as a result of SCMS: “There is capacity building in logistics management, computerized management of the supply chain by the health district. It should be noted that the SCMS project addressed several challenges. Today, there is better monitoring of the distribution of ARVs”. DK added: “I have to say that they helped us to rehabilitate our pharmacy. This has enabled us enough to bring comfort to our structure. Secondly, they supported us in technical and financial aspects, in supervision, in monitoring and in coordination meetings…”. DS also said: “Adding to that, it has become easier to validate data, capacity building of new providers so that every year we have new agents arriving.” 3.2.5 Decentralized Supply Chain Management 53 SCMS supported regional and district pharmacies through its five, decentralized sub- offices. SCMS worked with the decentralized supply chain technical committee (DSCTC) to establish a planning and coordination platform aimed at planning and monitoring implementation of decentralized supply chain activities. Such support included funding for workshops, and meetings (per diems, and meals and incidentals), and communication. Support to MSHP Regional Health Offices (RHOs) and Districts Health Offices (DHOs) resulted in the inclusion of supply chain activities and cost in their respective annual work plans. Prior to the existence of the SCMS program, data management, analyses, and validation were carried out using Microsoft Excel. In collaboration with MEASURE, SCMS supported RHOs and DHOs in the roll- out of e-LMIS to collect more accurate HMIS and LMIS data for quantification of HIV commodities and for decision making in supply chain management. Similarly, SCMS provided financial support for RHOs and DHOs to organize data analysis and validation meetings and to perform data quality assessments (as part of routine supervision). Finally, SCMS trained pharmacists in forecasting and supply planning for ARVs as well as in monitoring and evaluation methodologies. Supervision remains an area of great need at the peripheral level. SCMS supported DSCTC to perform supervision of RHOs and central level facilities (university hospitals and national institutes of health); and supported RHOs and DHOs to perform supervision of supply chain activities according to their respective supervision plans. DHOs received support to implement their distribution plans of HIV/AIDS commodities from district pharmacies to health centers while RHO pharmacists received assistance in training, coaching, and initiating performance incentives for peripheral-level LMIS users. Despite the above-mentioned achievements at regional and district levels, KII revealed a number of unmet needs. Improving the overall management of HIV/AIDS commodities, including ordering, storage, and rational use requires access to quality and timely data from its health facilities, clinics, and other SDPs. Therefore, it is important to ensure proper logistic and software equipment as well as sound training of health workers at all levels. Adequate internet access in the field also poses a significant challenge. 3.2.6 Contribution to Reduction of Breaks in ARV and Lab Product Supply The different actors in the supply chain visited agree that as far as drugs are concerned, the work of SCMS has been remarkable. "We started at a considerable breaking rate and are at almost nothing today," expressed almost all key informants. In the event of ARV stock shortages, health facilities first turn to peers who may be over-stocked, to request product transfer. In the event peers cannot rectify the shortage, health facilities solicit the assistance of Centre de Prise en Charge, de Recherche et de Formation (CEPREF), which generally has adequate stock of ARVs. In the unlikely event products are not found at CEPREF, health facilities then turn to SCMS with an emergency order, which is quickly loaded and delivered. Where stock-out problems have been identified during supervision visits, SCMS has sought to find solutions. According to pharmacist A: "SCMS project is in the cutting edge. In case of a break up, SCMS is in charge of replenishment in collaboration with the other centers or the NPSP. Therefore, ruptures do not last. It should be noted that the orders are monthly (once a month), and each order is valid for a maximum of four months and a minimum of two months.” K supported the same idea when he indicated that: “We turn to SCMS when facing an urgent order which takes care of it immediately.” 54 In contrast to the ARV supply, almost all actors interviewed recognized the weakness of laboratory￾product supply. According to HB Moi: “One of the mandates assigned to SCMS was to strengthen the system, capacity building, and see if the objectives on the component were reached. Pharmacies are focused on drugs but the laboratory ... SCMS has not strengthened the central level components and pharmacists in the laboratory, which explains repeated breaks up that are observed. The SIGL lab was put in place - SCMS was asked to put a specialist pharmacist on laboratory products at the NPSP to help, but they did not follow our instructions”. TY shares the same view when he points out that: “For laboratory products, there are substantial shortages, SCMS must have a service dedicated to the laboratory because they do not fully understand the functioning of lab products. At the quantification meetings, one speaks of ARV at 95% and 5% Lab. There must be constant communication on the stock levels; there are places overstocked and others that are understocked.” Study participants interviewed acknowledge that there had been a moment of progress but that the situation had regressed. RT comments on the same issue to illustrate the situation: "When I arrived in 2008, SCMS had already worked to set up the LMIS. It was functional but the lab was not in place. We mounted some pressure to get the LMISSIGL Lab because there were enormous concerns about the management of Lab products after all the system was put in place. There was an improvement throughout that period. 3-4 years later, SCMS, NPSP staff left.” Findings from FGDs supported almost absence of stock-out in ARV except one lady in Bouake who experienced once stock- out long time ago. Nevertheless, she agreed with other participants on the improvement in that area. SCMS set up strategies to mitigate stock out: 1. Electronic inventory system provides information on availability of different products in all facilities; 2. The system allows identifying facilities with over stock and those experiencing stock-out regularly. Therefore, districts are informed where the products are available. SCMS also handles emergency transfer of products from one district or facility to another. One respondent decried the quantification process at the Laboratory level: “Formerly, we had the data from the major laboratories. Today, quantification is done over the phone and now we call the laboratories.” In contrast, some study participants think that numbers of individuals have been overcritical vis-à-vis SCMS. They seem to want more than the project objectives and target activities. According to the SCMS management, the project had been providing logistic support to NPSP. Unfortunately, some lab technicians misused, for lack of training, the lab containers and were responsible for some challenges faced such as lab items stocked out. This is consistent with findings from the NSCA, which indicated that most of the time, ongoing laboratory stock-outs were attributed to misuse of reagents by the lab technicians. Findings from KIIs support this argument. About five key informants stated that lack of training was one of the leading factors of stock-out in laboratory. Nevertheless, recent survey in Côte d'Ivoire (NSCA, 2015) revealed only minor breakdowns of HIV laboratory reagents, with minimal interruption of HIV-specific services (CD4) at all sites visited. According to the study, 85% of the sites visited had CD4 machines in good condition and did not report 55 any interruption of service due to a CD4 equipment failure. Hematology and biochemistry tests were available at all sites; evidence that correctly functioning machines were also available. About 71% of the sites visited had consistently functioning biochemical equipment throughout the study period. Stockpiles of hematology and biochemical reagents (thinner for NFS, glucose, creatinine) were reported in most of the sites visited, and to a lesser extent, these have been linked to CD4 products (main reagent CD4) . While only two of the 17 sites reported at least one episode of CD4 reagent failure, five and 11 sites reported breaks in biochemistry and hematology, respectively. Shortages of health commodities, whether for HIV or not, are the main causes of disruption in service in resource-limited countries. Consequently, the high rate of reagent stock-outs may lead to an interruption of service, even if the instruments are functional. 3.2.7 Contribution to the reduction of unusable products SCMS's actions to dispose of expired products include routing products to incineration sites set up by the project. According to PTM: "SCMS helps to transport obsolete products to the place of incineration". This action is valued because it allowed the MSHP of Côte d'Ivoire to destroy a significant number of obsolete products. In this regard, DBN argued that: “SCMS participates in the collection, transport and destruction of obsolete medicines in Abidjan. And it should be noted that SCMS has helped the district collect and destroy obsolete products stored for many years. SCMS, together with other partners such as the European Union, has financed the whole process” (Pharmacist, Treichville-Marcory health District). PYE stated: “SCMS visits all health facilities every month to recover the waste.” For PBS "Waste destruction is carried out centrally and SCMS ensures the promptness of this service." PSA says: "Yes, they did it for ARVs. They gave us material (boxes, etc. ...) so that the agents who make the inventory of the outdated products can dress properly and be able to store the outdated ARVs, then SCMS comes pick them up.“ According to a SCMS staff, and confirmed by the evaluation team, they helped the DPMN to develop a process for managing unusable products. A manual has been developed, reproduced and disseminated. SCMS defined the circuits where unusable products should be deposited and conducted an evaluation to identify a structure capable of destroying the largest molecules. All KIIs confirmed the contribution of SCMS in destroying the out of date products, though facilitating collection and transportation. KII participants also reported systematic destruction of all out of date products. 3.2.8 Work in Synergy and Communication with other Actors in the Chain For supply chain management, SCMS collaborates successfully with stakeholders. With CDC, for example, SCMS works to supply laboratory products. As the RF indicated: “Supply chain, we are part of the technical committee where the NPSP provides the secretariat. In order to have a good procurement plan, meetings have been held.” SCMS works in synergy with NPSP for quantification needs. As suggested by DP: 56 “Quantification is a process. It begins with data collection . . . now a less cumbersome process . . . PNLS meets with SCMS and Measure JSI to collect the data.” PNDAP describes its collaboration with SCMS in the development of strategic plans and in the training of pharmacists in quantification. In the words of DPN: “With the support of SCMS, every 4 years, the strategic plan is drawn up; each year, they accompany this with annual plans, rehabilitate the services of public pharmacies, and also training. SCMS supported the collection of supply chain performance indicators. PNDAP assisted the regions with partners including, UNFPA, Measure and SCMS, to train its pharmacists in quantification. SCMS has done a lot in the region in terms of supervision. We have asked SCMS for the training of the first managers of the districts in logistics management…” With Retroci, collaboration is also at the level of quantification. DRI indicated that: “From 2007 onwards, SCMS gradually took over Retroci's prerogatives in terms of supply ... Retroci's assistance was reduced to quantification. In 2008, the Retroci laboratory management software was integrated into the LMIS, which has thus reclaimed all the responsibilities for design, development and implementation”. MEASURE Evaluation essentially intervened at the quantification level and worked in close collaboration with SCMS in the development of the computerized management system. In 2014, MEASURE took over LMIS in the context of NPSP reform and developed it in collaboration with SCMS. The development of the IT component was done by MEASURE, but the technical aspects of supply chain management were done by SCMS. IV-CONCLUSIONS The purpose of this Evaluation for Supply Chain Management System (SCMS) Project was to increase learning about the performance of SCMS interventions in Côte d’Ivoire. The activity aimed to identify key achievements and major challenges encountered during the implementation of the project at the national, regional and district levels. The overall findings of this report are that SCMS has helped the MOHPH make significant progress, strengthening the public health supply chain system at the central level. SCMS support has also helped transform the Nouvelle Pharmacie de la Santé Publique (NPSP) from a state-owned enterprise into a not￾for-profit, apolitical organization run on the basis of sound business principles. The project successfully improved the reporting systems, set up the electronic and computerized systems and consequently reduced the stock-out rates. The activity also improved destruction of out-of-date products and storage systems. However, while the performance at the central level improved (due to SCMS support since 2005), performance at the local level remained weak (benefiting from SCMS support only since 2013). Health facilities in the districts continued to experience stock-outs of HIV products and sizable volumes of expired products still occupied valuable storage space. Corrective actions had begun, aided by the change of program focus in 2013 that expanded support beyond the central level to the sub-national (regional and district) levels as well, where initiatives had targeted increasing human resource capacity and improving inventory-management systems. Table 7 summarizes key achievements, areas of improvement and recommendations. Table 7 – Summary of key findings and recommendations 57 Key Achievement Areas of improvement Recommendations Timely report Logistic and lab related data report Training on reporting and access to computer as well as internet. Electronic inventory and Access to internet and availability The government and partners to procurement system of computer facilitate access to computers and Computerization of the systems Regular order fulfillment; internet. quantification and distribution Reduction of stock out Quantification, Procurement, Continue supporting the NPSP and Transportation and on-time training of staff for better planning delivery rate and management of stock. Widely install management software and train providers. Storage system and conditions Stocked according to the plan; The government and partners to storage space. assess needs and increase the capacity of rooms accordingly, Training on supervision Training of non SCMS staff, lab Strengthen the capacity of lab staff staff, etc. and other non-SCMS staff in logistic management. Good waste management Quality of care and prevention of Training service providers and Lab infections agents on “good waste management practices”. Specific Recommendations for Supply Chain Functions: Product Selection • Review and periodically disseminate the LNME at all levels of the health pyramid and ensure supply-chain management protocols, manuals, and approved documented guidelines are available in facilities and structures at each level. Quantification: Supply Planning and Forecasting • Review the various Logistic Management Information System (LMIS) currently in use by different programs in order to establish an integrated, national LMIS. • Integrate peripheral stock management into this automated national electronic LMIS. • Ensure a sustainable funding mechanism for maintenance and renewal of management tools, as needed. • Strengthen the coordination of procurement plans for different products and from different sources (NPSP-CIs, programs and donors). Warehousing and Inventory Management • Train region- and district-pharmacists on computerized inventory management and provide adequate supervision thereof. • Complete the upgrading of all (82) district pharmacies to the standards of organization and storage of health products. • Strengthen the operational capacities of pharmacy departments in health districts by allocating more resources to improve the performance of first point of contact Service Delivery Points (SDP). • Implement integrated management software at the NPSP level. Improve the order management and delivery process. Key recommendations of this evaluation are: 1. Staff needs to be empowered and clearly motivated to take responsibility for managing the quality and security of the supplies under their control. 58 2. Advocate with key stakeholders to ensure that the change in status of the NPSP, from government-run to independent, does not make it lose its public-health and public-service perspective in favor of a narrow focus on autonomy and profitability. 3. Ensure that SCMS and PEPFAR clinical partners operate in a complementary manner, avoiding both competition and duplication of efforts. 4. Clarify that laboratory test supply is outside of the scope of ARV SCM responsibilities, as there are other mechanisms in the country for laboratory test supply. 5. Strengthen the laboratory-competence capacity of stock managers at the central level. 6. Improve communication between the next initiative and other partners involved in ARV SCM, taking into account this area of improvement. RECOMMENDATIONS 1. It is important to ensure that the follow up project to SCMS has an anchor of a certain level with the authorities like the DGS at the MOH. 2. The private sector should be better involved. 3. A culture of accountability and responsibility, with clear consequences for decisions, actions, and inaction, should be promoted. Staff need to be empowered and clearly motivated to take responsibility for managing the quality and security of the supplies under their control. 4. Advocate with key stakeholders to ensure that the change in status of the NPSP, from government-run to independent, does not make it lose its public-health and public-service perspective in favor of a narrow focus on autonomy and profitability. 5. Ensure that SCMS and PEPFAR clinical partners operate in a complementary manner, avoiding both competition and duplication of efforts. 6. Clarify that laboratory test supply is outside of the scope of ARV SCM responsibilities, as there are other mechanisms in the country for laboratory test supply. 7. Strengthen the laboratory-competence capacity of stock managers at the central level. 8. Improve communication between the next initiative and other partners involved in ARV supply chain management. Specific Recommendations for supply chain functions: Product Selection • Review and periodically disseminate the LNME at all levels of the health pyramid. Quantification : Supply Planning and Forecasting • Review the various LMIS currently in use by different programs in order to establish an integrated, national LMIS. • Integrate peripheral stock management into this automated national ISIGL. • Ensure a sustainable funding mechanism for maintenance and renewal of management tools, as needed. • Strengthen the coordination of procurement plans for different products and from different sources (NPSP-CIs, programs and donors). • Improve the order management and delivery process. Warehousing and Inventory Management • At the peripheral level, conduct inventories routinely and ensure the financial department is notified of discrepancies, in order to make accounting adjustments. • Record all items (including no-cost items) in enterprise resource planning (ERP) software and inventory records. 59 • Validate all items received against delivery vouchers to ensure that all products ordered and received are placed in stock. • Train region and district pharmacists on computerized inventory management and provide adequate supervision thereof. • Complete the upgrading of all (82) district pharmacies to the standards of organization and storage of health products. • Strengthen the operational capacities of pharmacy departments in health districts by allocating more resources to improve the performance of LMIS. • Implement integrated management software at the NPSP level. Distribution (Transportation) • Communicate a delivery schedule to all stakeholders, in order to ensure full knowledge of delivery times and optimize transport costs. • Conduct root-cause analysis of substandard delivery periods in order to identify bottlenecks. • Strengthen the distribution capacities of the districts, using tools and levers such as distribution plans, follow-up protocols, reverse logistics, cold-chain management, vehicle maintenance, fuel allocation, etc. • Establish a standardized drug distribution system for health districts that could be funded by the 8% reimbursement of NPSP transportation costs. Waste Management • Quarantine and dispose of unusable pharmaceuticals, in accordance with existing SOPs and guidelines. • Disseminate the "National Procedure Manual for the Management of Unsafe Pharmaceuticals" at the peripheral level of the supply chain. • Implement a process of decentralization of the destruction of expired pharmaceuticals • Ensure the traceability of pharmaceutical expiry information. • Collect and routinely destroy expired pharmaceuticals to prevent accumulation that clutters and limits storage. Laboratory • Integrate the laboratory with the LMIS or a laboratory information management system and generate monthly reports to determine which products are expiring. • Develop and disseminate SOP for logistics management of laboratory products, inventory management, risk management and safety at all levels of the health system. • Establish a mechanism for laboratory-staff skills assessment and training. • Assess the ability of lower levels to store and handle hazardous products; develop a risk management plan for those lower-level facilities. • Improve laboratory supervision. • Improve the storage and management of hazardous products. • Develop an effective maintenance strategy and remove obsolete equipment. 60 V- ANNEXES Annex-1 Statement of Work PURPOSE OF THE EVALUATION The purpose of the Performance Evaluation for Supply Chain Management System (SCMS) Project is to increase learning about the performance of SCMS interventions in Côte d’Ivoire. It serves as an end-of-project evaluation of SCMS interventions in Côte d'Ivoire. USAID Health Office is interested to know whether the SCMS project has achieved its intended results. Furthermore, this evaluation will help in identifying and addressing critical gaps in evidence for supply chain strengthening activities. Likewise, it will inform the Government of Côte d'Ivoire’s National Health Plan 2016-2020 and future program design and implementation for supply chain management in the country. The evaluation will also serve to document key contributions by the Government of the United States of America to supply chain management in Côte d'Ivoire. The target audiences for the SCMS performance evaluation are the U.S. Embassy in Abidjan’s Front Office; the USAID Côte d’Ivoire Health Office, PEPFAR Team and Program Office; the Government of Côte d’Ivoire Ministry of Health (Nouvelle Pharmacy of Public Health (NPSP)); USAID/West Africa (WA), USAID/Washington (W), the Global Fund and other donors in the health sector. SUMMARY INFORMATION The period of performance to be evaluated is September 2005 to December 2015. To ensure that data is collected before the activity begins its final stage, field data collection must be completed no later than October 30, 2016. Box 1 – Summary of the Supply Chain Management Systems project in Côte d’Ivoire Activity/Project Name Supply Chain Management System (SCMS) Implementer Partnership for Supply Chain Management (PfSCM) Cooperative Contract # GPO-I-00-05-00032-00 Agreement/Contract # Total Estimated Ceiling of the $300,000,000 Evaluated Project/Activity(TEC) Life of Project/Activity October 2005 – February 2017 Active Geographic Regions Central level interventions and Peripheral units of health system. The intervention area covers 82 health districts, 71 general hospitals, 17 regional hospitals, 4 university teaching hospitals and 1500 + Development Objective(s) primary health care facilities. Côte d’Ivoire reduces its HIV infection rate through prevention, care and (DOs) treatment by working with and strengthening the Ivorian health care USAID Office system. USAID/Côte d’Ivoire 61 BACKGROUND SCMS was established in late 2005 as part of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), to deliver HIV/AIDS commodities and to help strengthen supply chains in 21 PEPFAR focus7 countries throughout the developing world, including Côte d’Ivoire. SCMS works to strengthen supply chains to enable the scale-up of HIV/AIDS care and treatment. The project’s mission is to: a) Establish and operate a safe, secure and reliable supply chain; b) Buy and distribute high-quality, anti-retrovirals (ARVs) low-cost essential medicines, HIV test kits, laboratory supplies and other products; c) Strengthen the capacity of national supply chains to ensure long-term sustainability; and d) Support supply chain collaboration and information sharing among global and local partners in the HIV/AIDS community. In most countries, SCMS delivers commodities to central medical stores from international suppliers of antiretroviral medicines and other commodities. However, when possible, SCMS procures commodities from local suppliers of laboratory commodities and, in limited cases, essential medicines. Central and regional medical stores, often under the Ministry of Health (MOH), then distribute to testing and treatment sites. The system looks to curtail the treatment shortages; money wasted on the high freight costs of emergency orders, and the lack of inventory control and improper storage all of which lead to redundancies and gaps in the HIV/AIDS service. The innovative mechanisms used by SCMS include state￾of-the-art commercial warehousing facilities that act as Regional Distribution Centers (RDCs) . These RDCs have attracted private sector clients such as GlaxoSmithKline, HTC telecom products and Pfizer. SCMS has helped build the supply chain capacity in two key areas: infrastructure and human resources. SCMS has helped equip warehouses with modern racking, security, forklifts, cold rooms and computerized inventory systems. The project also installed innovative modular systems — called warehouse-in-a-box, storage-in-a-box and clinic-in-a-box — in several countries. SCMS has provided traditional training programs, partnered with universities to provide pre-service training and build human resource capacity. Increasingly, SCMS and Ministries are engaging private-sector firms to store and distribute commodities. Currently, SCMS interventions are aimed at strengthening the institutional capacity of the Central Medical Stores, district/hospital pharmacies and other national institutions involved in the management of ARVs and other commodities for HIV/AIDS programs. Description of the Problem, Development Hypothesis (es), and Theory of Change HIV treatment is a unique tool in the AIDS response preventing illness and death, averting new infections, and saving money. Until about 10 years ago, out of 30 million infected with HIV in sub￾Saharan Africa (SSA), only 50,000 had access to treatment. Against this backdrop, the President George W. Bush announced in his 2003 State of the Union address the establishment of U.S. President's Emergency Plan for the Fight against HIV/AIDS Relief (PEPFAR). This is the biggest international health initiative ever funded by one nation to address a single disease. In 2005, PEPFAR established the Supply 7 Other PEPFAR focus countries include Botswana, Burma, Burundi, the Democratic Republic of Congo, El Salvador, Ethiopia, Guatemala, Guyana, Haiti, Mozambique, Namibia, Nigeria, Panama, Rwanda, South Africa, Tanzania, Uganda, Vietnam, Zambia and Zimbabwe. 62 Chain Management System (SCMS) project with the goal of procuring a reliable, cost-effective, and secure supply of products for HIV/AIDS programs in PEPFAR-supported countries. Before the initiation of the SCMS project, shortages and stock-outs of commodities caused dangerous “treatment holidays” for patients. Emergency orders wasted money on rush fees and high freight costs. Lack of inventory control wasted valuable commodities due to product expirations, improper storage and theft. Poor coordination led to redundancies and gaps in service. Since 2005, SCMS has been working closely with the Ivorian Ministry of Health to maintain a global supply chain that ensures reliable supply of quality HIV/AIDS medicines for people living with HIV/AIDS (PLHIV). The project supports the central pharmacy (PSP) in procuring, importing, storing, and distributing all PEPFAR-supported commodities for HIV/AIDS programs. SCMS has been providing technical assistance to the MOHPH through the Central Medical Stores-Nouvelle Pharmacie de la Santé Publique (NPSP), and other departments of the ministry as part of PEPFAR activities in Côte d’Ivoire. In collaboration with in-country and international partners, SCMS adopted a three-fold approach to strengthen the in- country supply chain system which includes: a) provision of quality, best-value health care products to those who need them; b) deployment of innovative solutions to help programs enhance their supply chain capacity; and c) ensuring accurate supply chain information is collected, shared and used. Overall, SCMS has virtually eliminated stock outs at the central level in PEPFAR-supported countries. Furthermore, the annual cost of ARVs dropped from about $1,500 per patient to between $100 and $200. The project met its mandate to provide a safe, reliable and secure supply. The expiration rate of ARVs in SCMS’s RDCs is 0.64 percent compared to an international pharmaceutical industry standard of between three and seven percent. SCMS’s robust quality assurance program has created a credible threat to potential theft and to suppliers of substandard or counterfeit commodities. With the launch of the Decentralized Supply Chain Activity in October 2013, USAID has been able to provide technical assistance at district and facility levels of the supply chain. The immediate goal of the Decentralized Supply Chain Activity under SCMS is “…to ensure the consistent availability of HIV commodities at PEPFAR-supported service delivery points throughout Côte d’Ivoire. The project lays the foundations for a longer-term goal of strengthening the logistics management information system, supply chain staff capacity, streamlining the distribution system, and enhancing inventory controls, transparency and efficiency.” Despite these achievements, evidence is needed regarding the effectiveness of the SCMS project in Côte d’Ivoire at the regional and the district level. USAID/WA’s Evidence for Development (E4D) program was tasked to conduct this independent, external end-of-the-project performance evaluation to measure achievements of the activity. This evaluation will help identify and address critical gaps in evidence for supply chain strengthening activities. In addition, the evaluation will inform the Government of Côte d'Ivoire’s National Health Plan 2016-2020 and future program design and implementation for supply chain management in the country. The evaluation will also serve to document key contributions by the United States government (USG) to the management of Côte d'Ivoire’s supply chain management system. Lessons learned from this evaluation have the potential to inform SCMS programs, and HIV/AIDS care implementation policies in other countries of the West African region, as well as other regions in Sub-Saharan Africa (SSA). Results Framework Below is the results framework for activities carried out by the SCMS project delineated at the central (national) and sub-national levels. There was a change in scope for the SCMS activity in 2013. Prior to 63 2013, SCMS support was limited to the central level of the health system. In 2013, support began at the lower levels of the health system. Figures 1 and 2 summarize the results framework respectively at the national (central level) and at the sub-national level. Figure 1 – Supply Chain Management Systems: Result framework at Central Level Support (2005- Present) Objective: Establish and operate a safe, secure, reliable and sustainable supply chain to procure and distribute pharmaceuticals and other commodities needed to provide care for persons with HIV/AIDS and related infections IR 1: Strengthened institutional capacity of the Central Medical Store (Nouvelle Pharmacie de la Santé Publique de Cote d’Ivoire) IR 2: Improved IR 3: Improved commodities procurement management management Figure 2 – Supply Chain Management Systems: Result framework at Sub-national Level Support (2013-2016) Objective: Ensure consistent availability of HIV commodities at PEPFAR-supported service delivery points throughout Côte d’Ivoire IR 1: Strengthened IR 2: District depots and institutional capacity at the pharmacies upgraded regional and district levels IR 3: Equipment for regional and district offices, and to RIP+ organization zonal representatives provided Note : RIP+ is Réseau Ivoirien des personnes vivant avec le VIH - Ivorian network of people living with HIV/AIDS. Summary Description of the SCMS Activity The immediate goal of the SCMS activity is to ensure the consistent availability of HIV commodities at PEPFAR-supported service delivery points throughout Côte d’Ivoire. This activity also lays the foundation for a longer-term goal of strengthening the logistics management information system, streamlining the distribution system, and enhancing inventory control, transparency and efficiency. 64 Through the SCMS consortium, PEPFAR provides assistance to the MOHPH and AIDS (MSLS) to strengthen logistics management of HIV commodities at the intermediate and peripheral levels of the health system (Regional Health Offices, District Health Offices, Hospitals and Health Centers). These efforts strengthen the overall essential commodities supply chain and business processes and increase the impact of PEPFAR’s investments in Côte d’Ivoire’s health sector. SCMS provides technical assistance to regional and district health management teams to plan, coordinate and carry out supply chain activities. The Côte d’Ivoire peripheral level of the health system in the public sector includes 82 health districts, 71 general hospitals, 17 regional hospitals, 4 university teaching hospitals and 1500 + primary health care facilities. The evaluation will cover SCMS supply chain interventions at the central level and at the sub-national level. Activities at the sub-national level cover the 82 districts of Côte d’Ivoire where PEPFAR works and currently supports about 400 HIV/AIDS care and treatment service sites. The activity will include data collection at each type at the intermediate and peripheral levels of the health system mentioned above, assuming there is one week of data collection allocated per district. SCMS provides support in the following components of the health supply chain system: a. Warehousing, Inventory Management and Distribution ● Supports introduction of a computerized management system for health commodities in 76 districts and at 458 sites that provide HIV services. These sites are assisted by SCMS to provide ARVs and stock other HIV commodities. ● Provides technical assistance (TA) to improve planning, forecasting and inventory management of medical supplies, including HIV supplies. ● Conducts training for commodities managers in logistics management to strengthen their ability to effectively manage medical supplies. ● Provides TA to management teams to improve reporting on logistics data using national tools (stock cards, consumption reports). b. Logistics Management Information System ● Provides TA for logistics system design and review. ● Supports the deployment and effective use of pharmaceutical management software at the selected treatment sites. ● Assists with the integration of an electronic inventory management tool in district pharmacies. ● Supports periodic end-user verification surveys for HIV/AIDS commodities. c. Capacity Building ● Supports regional and district health teams with the design and implementation of training plans with a focus on logistics and inventory management for medical supplies. ● Facilitates greater integration of HIV product management with the broader pharmaceutical supply chain. ● Builds workforce at regional and district levels to effectively manage commodities. d. Waste Management ● Provides TA to ensure implementation of guidelines on expired drugs. ● Strengthens risk management and waste reduction at the decentralized level of the health system. ● Coordinates collection and destruction of any expired PEPFAR-procured HIV/AIDS commodities. e. Monitoring and Benchmarking 65 ● Provides assistance to Regional and District Health Offices in the project area to adequately monitor implementation of supply chain activities. ● Works with health officers to pilot and document innovative approaches, which have potential for positive impact on the supply chain system. Summary of the SCMS Activity M&E Plan The SCMS intervention targets technical areas, has expected results, and monitors performance indicators for both the central and sub-national levels. Tables 1 and 2 give these details for the central level and sub-national levels, respectively. Existing Data All work plans, national supply chain assessment (NSCA), quarterly reports, updated M&E Plan, and annual reports from the project start date to present will be made available to evaluators. Quarterly performance reports, indicator definitions and targets have been provided to the evaluation team through Google drive. The indicators in Tables 1 and 2 are used to measure performance under this project. Table 1: Central Level (Started in 2005) Technical Area Expected Results Measure Life of Project Target C-1. Improved product availability at Stock-out Rate Overarching health facilities and service Measures delivery points C-2. National Medical Stores have a On-time delivery Warehousing robust warehouse management rate (from central and Inventory system level to lower Management levels) (OTD) Measures and distribution C-3 Logistics Routine generation of reliable Percentage of Management data to inform logistics decision facilities submitting Information making timely and System complete LMIS Data and reports Information Measure (LMIS) C-4 Capacity Supply chain and logistics Number of project Building /Human professional serving at various assisted in country Resources levels of the health system organizations that Measures have documented and approved protocols/ procedures/guidelin es for supply chain functions C-5 Strategic Supply plans are well Forecast Accuracy 66 Planning and Coordination /Forecasting and Supply Planning coordinated. Approved national supply chain strategic plans available and regularly reviewed with stakeholders Table 2: Sub-national Units (Started in 2013) Technical Area Expected Results Measures Life of Project Targets Sub-1 Warehousing Improved product availability at and Inventory health facilities and service Stock-out rate at sites Management delivery points Measures and distribution Increased number of health Order fulfillment rate facilities adopting proper (from health districts storage and inventory to health facilities) management of HIV/AIDS commodities Percent of facilities Guidelines on expired drugs are complying with implemented. minimum Reduced waste at the decentralized requirements level of the health system Sub-2 Logistics Routine generation of reliable Percentage of facilities Management and data to inform logistics decision submitting timely and Information making complete LMIS Data and information reports measure Sub-3 Capacity Supply chain and logistics Number of project Building Human professionals serving at various assisted in country Resources Measures levels of the health system. organizations that have documented and approved protocols/procedures/ guidelines for supply chain functions Percentage of health personnel with assigned supply chain responsibilities who received adequate training 67 Technical Area Expected Results Measures Life of Project Targets (disaggregate by gender) Number (and proportion) of host country staff trained and deemed competent in supply chain functions (disaggregate by gender) Number of staff trained in logistics management of medical supplies (disaggregate by gender) Sub-4 Waste Guidelines on expired drugs are Management implemented. Percentage of waste Reduced waste at the (unusable health decentralized level of the health commodities, either system expired or damaged) (adequately) properly disposed of Costs incurred for waste management Sub-5 Regional and District Health Monitoring and Offices are able to conduct Number of staff trained Benchmarking regular monitoring of supply in logistics management chain activities of medical supplies? (disaggregate by gender) Percentage of health personnel with assigned supply chain responsibilities who received adequate training (disaggregate by gender) EVALUATION QUESTIONS The proposed evaluations questions are as follows and in order of priority. 1. What was accomplished and what were the challenges encountered during the implementation of the project at the national, regional and district levels regarding: 68 a. Computerized commodity management system and reporting systems? [Computerized system] Technical Area: Logistics Management Information System b. Integrated electronic inventory management tool? [Integrated electronic inventory tool] Technical Area: Logistics Management Information System c. Integrated HIV/AIDS product management into the broader pharmaceutical supply chain? [Integrated product management] Technical Area: Warehousing and Inventory Management d. Collection and destruction of expired HIV/AIDS commodities?[Waste Management]Technical Area: Warehousing and Inventory Management e. Prevention of stock-out of tracer HIV/AIDS commodities at service delivery points? [Stock-out] Technical Area: Warehousing and Inventory Management and Logistics Management Information System 2. As a technical assistance provider, did SCMS offer multiple services i.e., were beneficiaries able to access all the supply chain technical assistance they needed through a single project SCMS (a k a “One-stop-shop”) for the Government of Côte d’Ivoire MOHPH and AIDS for HIV/AIDS supplies and supply-related services? [Comprehensive Services] 3. What measures have SCMS assisted the Central Medical Stores-Nouvelle Pharmacie de la Santé Publique (NPSP) to put into place to improve management (including risk management and waste reduction) of antiretroviral and other commodities for HIV/AIDS programs? What remains to be done? [Policy-Procedures-Guidelines] 4. How is the supply chain actually performing at the national, regional and district level? What are the maximum and minimum stock levels for each level of the supply chain management system (central, district, local)? EVALUATION DESIGN AND METHODOLOGY Geographic Focus Data collection will be carried out in four regions with high HIV burden – Abidjan, San Pedro, Bouake and Korhogo. Within those regions, service delivery points in health districts with the highest number of PLHIVs and the highest number ART patients will be sampled. During one day of data collection, a total of 3- 4 key informant interviews (KII) can be conducted per senior team member (team leader, evaluation specialist, or SME). The team will also review and analyze supply chain management data at the selected sites, medical stores that supply the selected sites and central stores. Technical Requirements Proposed Evaluation Design and Methodology The evaluation is a performance evaluation using a non-experimental design because the SCMS intervention does not have a comparison/control group. The activity will rely on primary and secondary data, combining quantitative and qualitative methods. Quantitative data will include secondary data from the SCMS project, whereas qualitative data will contain primary information from KIIs with key stakeholders. Specifically, performance information sources include: 1. Baseline assessments for program implementation (SCMS) if any; 2. Country Operational Plan FY 2010 - FY 2010 (SCMS); 3. Project Workplans and project management plan; 4. Quarterly, semi-annual, and annual progress reports; 5. Financial report and pipelines; 6. Ministry of Health reports on SCMS activities; 7. Any signed agreement with local partners; 69 8. Key informant interviews. The evaluation will use a plausibility study design to assess changes in primary outcome measures included in the SCMS M&E Plan for a period before and after the start of the SCMS project in Côte d’Ivoire. The hypothesis is that no change in primary evaluation outcomes would have occurred without the changes in Central Medical Stores-Nouvelle Pharmacie de la Santé Publique systems. The evaluation will use complementary methods, including the National Supply Chain Assessment tool kit, to measure the capability maturity of the supply chain systems as well as the Key Performance Indicators. The indicators that are to be used to judge performance for the evaluation at the central and site levels are as follows: 1. Stock-out rate (central level and site level) 2. % of facilities submitting timely and complete LMIS reports to the central level 3. % of total stock that expired in previous reporting period 4. Order fulfillment rate 5. On-time delivery rate from central level to lower levels 6. % of supply chain functions documented in standard operating procedures (SOPs) at SCMS￾supported facilities 7. % of SCMS-managed product categories with coordinated procurement plans 8. %/# of project assisted in country organizations that have documented and approved protocols/procedures/guidelines for supply chain functions (Definition: Numerator: Number count of medical stores and ART sites sampled that have documented and approved protocols/procedures/guidelines for supply chain functions. Denominator: Total sites sampled.) Indicator definitions and targets are found in Annex 1 (see document titled “CI_ HSS_PWS_Data_Collection_Form_Annual Measures (1) _Final_Indicator DEF”). Data on tracer commodities, namely ARVs and HIV Test Kits where appropriate, will be collected and analyzed. The trend of performance against each indicator, over the life of the project, will be used to judge whether the project has performed acceptably or not. The evaluation must meet the criteria of a quality evaluation as defined by USAID. Once the SOW is approved and the consultants’ evaluation team is under contract (and approved by USAID/WA), the evaluation team will develop a more detailed and refined evaluation design, as well as complete the design matrix provided below in Table 3. The team will conduct structured interviews with the project staff and key partners, including the MOHPH (national, regional and district directors of SCMS) and relevant managers at the site level, using interview guide and the Capability Maturity Model (CMM) questionnaires. The interview guide will include, (but not limited to), the following themes: Support Received, Capacity Building, duplicity, quality of Implementation, sustainability and comments as well as recommendations. The CMM tool covers the key functional areas of the supply chain as well as measuring key “enablers” (Figure 3) that impact all functions across the supply chain. For each functional area, scores will be assigned for each capability, aggregated to understand the functional area as a whole as well as the enabling elements impacting the functional area which include; processes and tools, infrastructure, oversight, human resources and management information systems (MIS) . An overall maturity scale guides the definitions within the CMM tool. Capability level varies from one to five for each component. The CMM tool will be implemented at the central level and interviews will be conducted for each of the functional areas. At the site level, the data collection tool will be the Key Performance Indicator (KPI) tool. 70 Table 3: Evaluation Design Matrix Questions Suggested Data Sources Suggested Data Data Analysis Methods (*) Collection Methods What was accomplished and what were the challenges encountered during the Documents (including. Desk review, Descriptive methods, (including implementation of the project at the national, regional and district levels regarding : performance monitoring data, Questionnaires or proportions), content and thematic Computerized commodity management system, electronic inventory management tool , previous evaluations, reports, surveys, analyses. HIV/AIDS product management into the broader pharmaceutical supply chain , collection etc. Location: district, region and destruction of expired HIV/AIDS commodities , Prevention of stock out oftracer Facilities Interviews Type of sector: public versus private HIV/AIDS commodities? As a technical assistance provider, did SCMS offer multiple services i.e., were beneficiaries Documents (including. Desk reviews, Content and thematic analyses. able to access all the supply chain technical assistance they needed through a single project performance monitoring data, questionnaires, SCMS (a k a “One-stop-shop”) for the Government of Côte d’Ivoire MOHPH and AIDS previous evaluations, reports, for HIV/AIDS supplies and supply-related services? etc. KII; What measures have SCMS assisted the Central Medical Stores-Nouvelle Pharmacie de la Key Informant interview Interviews using Content and thematic analyses. Santé Publique (NPSP) to put into place to improve management (including risk questionnaires management and waste reduction) of antiretroviral and other commodities for HIV/AIDS programs? What remains to be done? How is the supply chain actually performing at the national, regional and district level? What are the maximum and minimum stock levels for each level of the supply chain Documents (including. Desk review, Descriptive methods, content and management system (central, district, local)? performance monitoring data, Questionnaires or thematic analyses. previous evaluations, etc. surveys, Location: district, region Facilities Interviews Type of sector: public versus private What is the capability of the system to perform at the national, the regional and the Documents (including. Desk reviews, Descriptive methods content and district level? performance monitoring data, questionnaires, thematic analyses. previous evaluations, reports Location: district, region and KII. Type of sector: public versus private 71 Implementation of Evaluation Activities The evaluation team will be composed of: ● One Team Leader (International); ● One Senior Evaluation Specialist who is from Côte d’Ivoire; ● One Subject Matter Expert from Côte d’Ivoire; ● The fields Evaluation teams (data collectors, transcribers, data entry clerks); ● The logistics team. The evaluation team (Team Leader, Evaluation Specialist and the Subject Matter Expert) will commence the study as soon as they are all approved by USAID/WA and conduct document review for one week and submit a Desk Review Report for USAID/WA review. The evaluation team will start developing the Inception Report in parallel in collaboration with USAID/CIV and the PEPFAR team. The entire team, supported by the E4D COP and the E4D Sr Research and Evaluation Advisor will be present in Abidjan/Côte d’Ivoire to meet with USAID/CIV, convene the Team Planning Meetings and finalize the Inception Report. The Inception Report will include the final methodology, the final sampling and the final evaluation data collection tools. Once the Inception Report is approved, the Evaluation Team will train the data collectors, then pilot test all instruments and tools prior to implementation in the field. The Evaluation Team will be organized in four sub-teams led each by the E4D Sr Research and Evaluation Advisor, the Team Leader, the Sr Evaluation Specialist and the Subject Matter Expert plus the data collectors and transcribers per sub-team. In Abidjan, the team will be led by the E4D Sr Research and Evaluation Advisor plus 9 data collectors and transcribers. In the remaining regions, other evaluation experts (Team leader, Sr Evaluation Expert and SME) plus 2 data collectors/transcribers will lead the team. A research assistant will join the sub -team team to supervise data collectors and conduct KIIs. Indeed, the Subject Matter Expert is not an evaluator, and may not have required skills to conduct KII and/or to supervisor data collection. All sub-teams will work in parallel to cover the selected sites, their associated medical stores and central stores for the data collection. The data collection will be completed within 10-15 days. The evaluation team will then convene in Abidjan for a final debrief with USAID. The international Team members will return to their home base and work remotely with the local team on data analysis and report writing to prepare for the first draft of the report, which will be submitted to USAID for comments. After the report is finalized, a dissemination meeting among stakeholders will be organized to disseminate findings of the evaluation. Sampling Methodology The sampling framework for the evaluation consists of the three intervention districts with the highest percentage of people living with HIV - San Pedro, Bouake, Korhogo, plus Abidjan. Within these four districts, Fiscal Year (FY) 2016 ART sites that are targeted by the PEPFAR team in Côte d’Ivoire for “Aggressive Scale-up”, “Scale-up to Saturation” and “Sustained” will be randomly selected. A total of 60 sites will be visited: 5 sites in San Pedro, Bouake and Korhogo each, and 45 sites in Abidjan, the Central Stores and medical stores that supply the selected ART sites. See Annex 2 for the list of ART sites selected. The proposed sample of 60 sites were selected using a Simple Random Sampling (SRS) method for the medical stores first by district category including Aggressive Scale up, Scale up to Saturation, Sustained. Then, the sites covered by these selected medical stores were SRS by the district category as well. They 72 include hospitals (regional and district), health facilities and pharmacies. The inception report will describe the sample per function. Data Collection Methodology and Instruments Prior to the start of data collection, the evaluation team will present data collection instruments and sources to USAID team (both West Africa Mission and Côte d’Ivoire Team) for review and comment. The draft instruments must be included as part of the Inception Report. All final instruments used in conducting the evaluation and raw data will be included in an annex of the final report. Data Analysis Plan Prior to the start of data collection and as part of the Inception Report, the evaluation team will present a data analysis plan to USAID/WA for review. The plan shall describe how the data will be transcribed and analyzed. It shall also describe how qualitative and quantitative data will be triangulated to reach final conclusions and recommendations. The evaluation team will have its own professional quantitative and qualitative data analysis software and any other analytical tools required to meet project deliverables. The data analysis procedures shall be included in the final report. Nevertheless, the team could use the Logistics Performance Gap analysis (see figures 3 and 4). In these figures, the inside blue line represents the performance of the supply chain being measured, whereas the red line on the outside represents the score of a similar, but highly performing, supply chain. Therefore, the team could use the gaps to assess strengths and weaknesses, and indicate areas that require attention. The spider graphs included below may also have a line representing the targets that SCMS Côte d’Ivoire was expected to achieve during its life at the central and peripheral warehouses. Some of the topics only apply to the central warehouses (customs clearance for example) and some others all across the supply chain system. Figure 3 - Warehousing/Storage Performance Indicators Figure 4 - Supplier/Sourcing Performance Measures. This is a generic system for NPSP to periphery, including other orders. At the central level, for example, custom clearance cycle is expected to be for a max of 15.5 days when intermediary and periphery sites are not expected to conduct those efforts. 73 Data Disaggregation Programmatic data analysis will be disaggregated by district. Furthermore, if possible, the capacity building data will be disaggregated by age and gender. Data Quality Data quality must meet USAID’s five standards: Validity, Integrity, Precision, Reliability and Timeliness Evaluation Team Qualification The proposed evaluation team must have skills and experience in leading and conducting evaluations, supply chain management programming, surveying and sampling, qualitative and quantitative analysis, writing, oral presentation, as well as cultural competencies. Methodological Strengths and Limitations The evaluation methodology must state all methodological strengths and limitations explicitly in the evaluation Inception Report, presentations and draft and final reports. DELIVERABLES AND REPORTING REQUIREMENTS A. Evaluation Work plan: Within two weeks of the start date of the evaluation (approval of the SOW by USAID/WA), a draft work plan that includes all phases and deliverables for the evaluation shall be completed and presented to the E4D’s Contracting Officer’s Representative (COR) and USAID Côte d’Ivoire Health Team. The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the skills of the evaluation team, delineated by roles and responsibilities. B. Desk Review Report: A written report summarizing what is known from performance monitoring reports and other project documents is due no later than two weeks after the SCMS evaluation team has been contracted and approved by USAID/WA (and E4D receives project reports and other documents). The Desk Review Report is also due before finalization of the evaluation design and should be used to inform data collection instruments and include findings and recommendations 74 as to how, if at all, the Inception Report should be modified. The evaluation team should meet virtually and communicate via email at home to complete the Desk Review Report, before travelling to Côte d’Ivoire (for the international team members). Inception Report including the Evaluation Design: Within seven business days of approval of the Desk Review Report, E4D must submit to the COR and the USAID Côte d’Ivoire Health Team an evaluation Inception Report, including the evaluation design (which will become an annex to the Evaluation Report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. The Evaluation team will organize a Team Planning Meeting with USAID/CIV and relevant stakeholders, consisting of a three-day working session to discuss and work with the team as they start developing the different parts of the Inception Report. USAID and stakeholders will be providing inputs during the Team Planning Meetings in Abidjan, as the evaluation team develops the Inception Report. USAID offices and relevant stakeholders will require three business days to review and consolidate comments on the Inception Report through the COR for E4D. Once E4D receives the consolidated comments on the initial evaluation design and work plan, IBTCI is expected to return with a revised evaluation design and work plan within two business days, and before field data collection begins. In-briefing: Within two (2) days of arrival in Côte d’Ivoire, the evaluation team will have an in-briefing with E4D, the COR and the USAID Côte d’Ivoire Health Team for introduction and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to make any final adjustments to the SOW (if necessary). C. Mid-term Briefing and Interim Meetings: The evaluation team is expected to hold a mid-term briefing with E4D, the COR and the USAID Côte d’Ivoire Health Team on the status of the evaluation, including potential challenges, emerging opportunities and data quality. The team will also provide the COR/evaluation/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. D. Final Exit Briefing: The evaluation team is expected to hold a final exit briefing in Abidjan prior to leaving the country. This presentation will be scheduled as agreed upon during the in-briefing. The exit briefing should include E4D, the COR, the USAID Côte d’Ivoire Health Team and the SCMS team. The evaluation team shall prepare and share, at least one day in advance of the exit briefing, a 10 slide (or less) presentation describing the status of data collection and analysis, and any preliminary findings. E. Draft Evaluation Report: A draft evaluation report should be submitted within 21 business days after the final exit briefing, with the main findings. This should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. 75 Once the initial draft evaluation report is submitted, the COR and the USAID Côte d’Ivoire Health Team will have 10 business days within which to review and provide comments on the initial draft, after which point the E4D’s COR will submit the consolidated comments to the evaluation team. The evaluation team will then submit a revised final draft report within five business days for review and final comments by USAID. USAID provides its comments on this final draft report within five business days of receipt from IBTCI. F. Final Evaluation Report: The evaluation team will submit final report within five business days of receiving final comments from the USAID Côte d’Ivoire Health Team and E4D’s COR. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITION The evaluation team will consist of: 1. Team Leader: an evaluation and/or supply chain management expert with demonstrated experience leading evaluation teams for international public health interventions and who is not employed by USAID. The evaluation team leader must have excellent organization, writing and oral presentation skills, as well as cultural competencies. 2. Evaluation Specialist: A local evaluation expert with demonstrated experience designing and carrying out evaluations of international public health interventions. The evaluation expert must have demonstrated experience in surveying and sampling, statistical analysis, as well as quantitative and qualitative methods and analysis. He will lead data management and data analysis activities. 3. Senior Local Subject Matter Expert (s): a senior local supply chain management expert with demonstrated experience in developing and managing supply chain management assistance activities to public and/or private entities. 4. Research Assistant: a researcher will be contracted during the data collection period to coordinate data collection in one of the districts. He will perform data management and data analysis under the Evaluation Specialist’ supervision. 5. Data Collector(s): Twelve local data collectors with demonstrated experience assisting with surveys and other methods of research, as well as assisting with data quality control and analysis or data management. Three data collectors will remain in Abidjan. 6. Data transcribers(s): Three local data transcribers will assist to transcript KIIs. The roles, responsibilities and qualifications of the senior evaluation team members are defined below: Evaluation Team Leader Responsibilities The Evaluation Team Leader will be responsible for: • Overall management of the evaluation team; • Desk review of documents, development of Inception Report, consisting of draft methodology and detailed work plan; • Coordination of evaluation activities, including training of data collectors, data collection, implementation, data management and quality assurance and other related tasks; • Conduct debriefing on the methodology; • Conduct debriefing with implementers on evaluation findings; 76 • Conduct field visits to the pilot project site and interviews with stakeholders (Key Informants, Key populations etc.; • Throughout the evaluation period, exercise strong communication, organizational, team leadership and interpersonal skills; periodically coordinate/update E4D’s Senior Research and Evaluation Advisor and as requested. • Development and submission of the evaluation draft report; • Finalization and submission of the final evaluation report after incorporating feedback received on the draft report; • Disseminate the evaluation findings. Qualifications • A Master’s degree in social sciences, public health, statistics, or a related area from an accredited institution is required; • At least seven (7) years’ experience conducting public health program evaluations with both quantitative and qualitative methods for data collection and analysis; (highly desired) • Previous experience leading evaluation teams is required; • Prior evaluation experience in Sub-Saharan Africa is required; • Excellent oral and written skills in French and English are required; • Previous experience preparing high-quality evaluation reports; • Previous experience with USG-funded projects and knowledge of USAID’s ADS2013 policy, standards, guidance and protocols (highly desired). Senior Evaluation Specialist Responsibilities The Local Evaluation Expert's responsibilities include, but are not limited to, the following: • Develop evaluation design, methodology, sampling strategy, and data collection instruments; • Coordinate evaluation activities, including training of data collectors, data collection, implementation, data management and quality assurance and other related tasks; • Develop data analysis plan and conduct qualitative and /or quantitative data analysis, as required; • Actively participate with other team members during data triangulation, presentations and report writing; • Assist the Team Leader in completion of the Inception Report and the writing of the evaluation report in conformity with the scope of work; • Develop final evaluation report with quality assurance and timeliness of all deliverables to USAID and be responsive to all comments. Qualifications • A Master’s degree in social sciences, public health, statistics, or a related area from an accredited institution is required; • At least seven (7) years of evaluation experience with both qualitative and quantitative methods for data collection and analysis; (highly desired) • Prior evaluation experience in West Africa is required; • Previous experience with USG-funded projects and knowledge of USAID’s ADS 2013 policy, standards, guidance and protocols (highly desired); • Experience in using SPSS, STATA and/or other analytical software packages including qualitative analytical software packages such as NVIVO; • Strong oral and written communication skills in French and English is required; • Ability to effectively work in teams and embrace participatory approaches. 77 Senior Local Subject Matter Expert (Cote d’Ivoire) Responsibilities The Local Subject Matter Expert Expert's responsibilities include, but are not limited to, the following: • Provide insight and knowledge with respect to the common practices and activities for delivering supply chain management assistance to public and/or private entities; • Assist in developing appropriate evaluation design, methodology, sampling strategy, and data collection instruments for evaluation of a supply chain management assistance intervention; • Assist in coordinating evaluation activities, including training of data collectors, data collection, implementation, data management and quality assurance and other related tasks; • Assist in developing data analysis plan and conduct qualitative and /or quantitative data analysis, as required • Actively participate with other team members during data triangulation, presentations and report writing. • Assist the Team Leader in completion of the Inception Report and the writing of the evaluation report in conformance with the statement of work; Qualifications • A Master’s Degree from an accredited institution in public health, or similar discipline is required. Formal training and experience in supply chain management is required. • A minimum of seven (7) years of progressive responsibilities in program management for supply chain management programs is required; • Previous experience evaluating international public health programs is highly desired; • Knowledge of West and Central African health institutions as well as familiarity with and sensitivity to socio-cultural factors affecting development in the region is required; • Previous experience with USG-funded projects and knowledge of USAID Evaluation Policy (highly desired); • Strong oral and written communication skills in French and English is required; • Ability to effectively work in teams and embrace participatory approaches; and • Local resident of Côte d’Ivoire is required. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. EVALUATION SCHEDULE Please see Annex A: Evaluation Timeline. Also refer to the MS Excel Budget Worksheet: LOE Worksheet to get a breakdown of the total home days and total field days for each of the evaluation team members. FINAL REPORT FORMAT The evaluation final report should include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or 78 methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report should be formatted according to the evaluation report template. The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: ● The Evaluation SOW; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; ● Sources of information, properly identified and listed; and ● Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of interest. In accordance with AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 203, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report.8 ● The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. ● Evaluation reports shall address all evaluation questions included in the SOW. ● The evaluation report should include the SOW as an annex. All modifications to the SOW — whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline — need to be agreed upon in writing by the AOR/COR. ● The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation — questionnaires, checklists, and discussion guides — will be included in an annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 8 See Appendix I of the Evaluation Policy and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 79 ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. ● Sources of information need to be properly identified and listed in an annex. ● Recommendations need to be supported by a specific set of findings. ● Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. 80 Annex 1. SCMS Performance Indicators with definitions CI_ HSS_PWS_Data_Collection_Form__Annual Measures (1)_Final_Indicator DEF.pdf 81 82 Annex-II: List of ART Sites Selected No 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 Health District Region: Abidjan Yopougon-Est Yopougon￾Est Yopougon-Est Yopougon-Est Yopougon￾Est Yopougon-Est Yopougon-Est Yopougon￾Est Yopougon-Est Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Koumassi-Port Bouet-Vridi Abobo-Ouest Abobo￾Ouest Abobo-Ouest Abobo-Ouest Abobo￾Ouest Abobo-Ouest Treichville-Marcory Treichville-Marcory District category Site Name Scale-up to saturation Cliniques les Oliviers Scale-up to saturation Cliniques Ste Rosa de CASCIA Scale-up to saturation Centre de Santé Urbain Communautaire de Andokoi Scale-up to saturation Formation Sanitaire Urbaine Communautaire de Koweit Scale-up to saturation CIP/CAMES Yopougon Scale-up to saturation Formation Sanitaire Urbaine Communautaire de Ouassakara Scale-up to saturation Centre Nazareen Scale-up to saturation Formation Sanitaire Urbaine Communautaire de Toits Rouges Scale-up to saturation Formation Sanitaire Urbaine Communautaire de Gesco Aggressive scale-up Cabinet Médical de Koumassi Aggressive scale-up Espace Médical les Ruches Aggressive scale-up Centre de Santé Municipal Akwaba Aggressive scale-up Hopital Municipal Vridi Cite Aggressive scale-up Centre Médico-Social CARITAS Aggressive scale-up Centre de Sante Urbain Communautaire de Pangolin Aggressive scale-up Centre AntiTuberculeux de Koumassi Aggressive scale-up Centre de Santé Urbain Communautaire de Gonzagueville Aggressive scale-up Hôpital Général de Koumassi Aggressive scale-up Centre de Santé Rural de Ako Brakre Aggressive scale-up Centre de Santé Urbain Communautaire de Zoe Bruno Scale-up to saturation Grande Clinique du Dokoui Scale-up to saturation Centre de Santé Urbain Communautaire de Agoueto Scale-up to saturation Centre de Santé Urbain Communautaire de Anonkoua 3 Scale-up to saturation Formation Sanitaire Urbaine Communautaire de Abobo-Sagbe Scale-up to saturation Hôpital Général de Abobo-Sud Scale-up to saturation Centre de Santé El Rapha Scale-up to saturation Clinique Nanan Yamoussou Scale-up to saturation Polycliniques Avicennes 83 No Health District 29 Treichville-Marcory 30 Treichville-Marcory 31 Treichville-Marcory 32 Treichville-Marcory 33 Treichville-Marcory 34 Treichville-Marcory 35 Treichville-Marcory 36 Treichville-Marcory 37 Treichville-Marcory 38 Treichville-Marcory 39 Treichville-Marcory 40 Treichville-Marcory 41 Treichville-Marcory 42 Treichville-Marcory 43 Treichville-Marcory 44 Treichville-Marcory 45 Grand-Lahou Region: Bouaké/Gbèkè District category Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Scale-up to saturation Sustained Site Name Polycliniques Hotel-Dieu Clinique Polymed Centre de Santé Urbain Communautaire de Aliodan Formation Sanitaire Urbaine Communautaire de Anomabo Hôpital Général de Treichville Dispensaire Anti Venerien INHP Centre Médical La Pierre Angulaire Service de Gyneco-obstetrique Centre Hospitalier Universitaire Treichville Centre de Santé Espace Confiance Service de Dermatologie du Centre Hospitalier Universitaire Treichville Hôpital Général de Marcory Hope Worldwide Cote d'Ivoire de Treichville KO'KHOUA Centre National de Transfusion Sanguine Unité de Soins Ambulatoires et Conseils (USAC) Centre Integré de Recherches Biocliniques d’Abidjan (CIRBA) Service de Maladies Infectueuses Tropicales du CHU Treichville Hôpital Général de Grand Lahou 46 Bouake-Nord-Est Aggressive scale-up 47 Bouake-Nord-Est Aggressive scale-up 48 Bouake-Sud Scale-up to saturation 49 Bouake-Sud Scale-up to saturation 50 Bouake-Sud Scale-up to saturation Region: Korogo/ Poro-Tchologo-Bagoue 51 Korhogo Scale-up to saturation 52 Korhogo Scale-up to saturation 53 Korhogo Scale-up to saturation 54 Korhogo Scale-up to saturation 55 Korhogo Scale-up to saturation Region: Gbokle-Nawa-San Pedro 56 san-pedro Scale-up to saturation Centre de Santé Communautaire de Kottiakoffikro Hôpital Général HMI de ATTIENKRO Centre de Santé Catholique de Djebonoua Centre de Santé Urbain de Koko Renaissance Santé Bouake Centre de Santé Urbain de Tioroniaradougou Centre de Santé Urbain de KOKOTON Centre de Santé Urbain de Napie Protection Maternelle Infantile de KORHOGO Dispensaire Rural de Torgokaha Société Africaine de Plantation d’Hevea (SAPH) de San Pedro 84 No Health District 57 san-pedro 58 san-pedro 59 San-Pedro 60 San-Pedro District category Site Name Scale-up to saturation AIBEF San Pedro Scale-up to saturation Protection Maternelle Infantile de Bardot San Pedro Scale-up to saturation Centre de Santé Rural de Moussadougou Scale-up to saturation Maternité Terre Rouge 85 Annex III: Documents Reviewed 1. CI_Chenin_ECOWAS Stock Assessment_Final report Edited by Ana de Paiva 2. PNSCA 2016-2020 3. LLamasoft_CI_Final Report 4. Task Order 3 Objectives 5. SCMS CI STTA June 2013- Technical Report Final Draft 6. ART sites and medical stores 28Apr2016.xlsx 7. Cote d’Ivoire - PFSCM Comments on Concept Note 8. PEPFAR Geographic Coverage-Jan2016 9. 2013APR11 Courrier MSLS designation de points focaux pour le projet pilote d'appui a la gestion decentralisée 10. Decentralized_Supply_Chain_CNote-sk-hj-26Feb2013 11. Note Technique_D-SCM_23sept 2013 12. One Pager on Decentralized Supply Chain Support-db-sk-3June 2014 13. Data Collection tool for Supply Chain Site Visit db-sk- 062514 14. FAQ on SCMS Project Nov2015 15. Letter to MoH_re D-SCM implementation-sk-30Sept2013 16. From SCMS to GHSC -21Oct2015 17. Cote d'Ivoire_One-pager_FINAL 18. CI_ HSS_PWS_Data_Collection_Form__Annual Measures (1)_Final 19. Results Framework 20. Note Technique_D-SCM_23sept 2013 21. SCMS-CdI_Activity Data Sheet_PIR FY2015-12Nov2015 22. SoW National Supply Chain Assessment-Draft-Sk-28Jan2015 23. SCMS CI FY15_Cote d'Ivoire_Q3_Country PMP Report_05082015 24. SCMS-CdI_Activity Data Sheet_PIR FY2015-12Nov2015 25. Evaluation de la chaine d'approvisionnement des médicaments en CI NSCA 2015- Final Report French 26. Presentation resultats quantification_2015_Lab 27. SCMS CI FY16_Cote d'Ivoire_Q1_Country PMP Final Report version 28. SCMS CI FY15_Cote d'Ivoire_Q4_Country PMP Report_13112015_f 29. PEPFAR PPR FY 2014 for SCMS Dft Sk-REVISED Jan 8th 2015 30. Recommendations support to decentralized level CI Draft 3 (2) 31. SCMS CI FY13Q4 Country PMP Report Nov22Revision 32. SCMS CI - MFS _Sept2015 - Summary 33. SCMS PMP REPORT_JAN-MARCH_2012-Skamdem-26avril2012 (1) 34. Task Order 1 35. SCMS CI FY15_Cote d'Ivoire_Q2_Country PMP Report_12.05.2015 36. SCMS_Rapport Trimestiel_oct-nov-dec 2014 37. SCMS_Rapport Trimestiel_Jan-Mars 2015_20- 04- 2015 sans les annexes 38. SCMS_Rapport Octobre-Decembre 2015 39. SCMS_Rapport d'activités Janv-Fev-Mars 2016 86 Annex-IV: List of institutions Visited N° Name Structure Fonction Emails 1 Dr Koudoufum Noel C. SCMS/MSH DSCM ckoudougnon@ci.psfcm.org 2 Dr Dje Kouakou SCMS/MSH Senior Lab Advisor dkouakou@msh.org 3 Yapi Achou Sabin SCMS/MSH M et C specialist syapi@msh.org 4 Svendsen Pete SCMS Procurement +supply psvendsen@ci.psfcm.org 5 Dr Elloh Severin SCMS/MSH Directeur pays selloh@ci.psfcm.org 6 Dr Bosso Edwije Measure evaluation Deputy Director edwije.bosso@ci.jsi.com 7 Akaffou Ange Measure evaluation IT Advisor angeakaffou@ci.jsi.com 8 Dr Adou Marie Appoline PNDAP-CNCAM Médecin marieapolline@gmail.com 9 Dr Oga Eulalie Benie PNDAP-CNCAM Pharmacienne ogaeulalie@yahoo.fr PNDAD-RE 10 Berthe Karidjatou PNDAP Pharmacienne bktou@yahoo.fr 11 Yayo Sagou Olivier PNDAP DC Yayooli07@yahoo.fr 12 Hodjo Danielle PNDAP Cellule d’Appui dhoddjo@yahoo.fr 13 Omono Martin PNDAP ASP martomonoz@yahoo.fr 14 Dr Tehe André CDC Biologiste Hpis@cdc.gov Chef de bureau 15 Dr Ekra Alexandre CDC prévention, soins et Hpo7@cd.gov traitement 16 Dr Kohemon CDC Pharmacien biologiste hpmu@cdc.gov 17 Adjé Touré Christiane CDC Directrice Labo CIA96cdc.gov 18 Dr Zoundi Ouattara Odile ICAP Coordinatrice soutien oz2120@cumc.columbia-edu aux sites 19 Dr Makaila Oyewole ICAP Site support manager om2218@cumc.colombia-edu 20 Koné Sylvestre ICAP CT Labpham KP2603@cumu.colombia-edu Dr Ouattara Ouattara Chef Service 21 PNLS Médicament et huguette.likane@gmail.com Dieneba Laboratoire Chef Service 22 Dr Likane Liliane Huguette PNLS Médicament et huguette.likane@gmail.com Laboratoire 23 Dr Abo Kouamé PNLS Directeur kwagny@gmail.com Coordonnateur 87 Annex V: Evaluation Team Dr. Pierre-Marie Metangmo, Evaluation Team Leader Dr. Pierre-Marie Metangmo is a seasoned health system and supply chain management evaluator and manager with extensive working expertise in disease surveillance, epidemic response, reproductive health, institutional capacity building, and health system strengthening. Dr. Metangmo is fluent in French and English and has over 25 years’ experience in and a proven track record of providing consulting services, leading technical and management evaluations, and implementing projects and transformational initiatives in complex settings with organizations including ministries of health, international NGOs, graduate schools, rural communities, and regional health organizations. He possesses a solid foundation in the best practices of reproductive, newborn, child and maternal health; project evaluation, including baselines studies, mid- term, and final evaluations; capacity building, training, leadership and health systems strengthening; and is well acquainted with USAID, UNICEF and the World Bank funding procedures. His past professional experiences include work with Management Sciences for Health (MSH), where he backstopped leadership management and sustainability (LMS) projects in 4 countries (DRC, Ethiopia, South Sudan and Tanzania). In this assessment, he was responsible for: ● Overall management of the evaluation team; ● Desk review of documents, development of the inception report, consisting of draft methodology, detailed work plan; ● Coordination of evaluation activities, including training of data collectors, data collection, implementation, data management and quality assurance, and other, related tasks; ● Conducting debriefing on the methodology; ● Conducting debriefing with implementers on evaluation findings; ● Conducting field visits to the pilot project site and interviews with stakeholders (key informants, key populations etc.); ● Throughout the evaluation period, exercise strong communication, organizational, team leadership, and interpersonal skills; periodically coordinate/update E4D’s senior research and evaluation advisor and as requested. ● Development and submission of the evaluation draft report; ● Finalization and submission of the final evaluation report after incorporating feedback received on the draft report; and ● Dissemination of evaluation findings. Dr. Emmanuel Esso, Senior Evaluation Specialist A statistician, demographer, and expert in reproductive and community health, Dr. Esso is well suited to serve as the Evaluation Specialist on the SCMS performance evaluation. He has 15 years of experience developing health information systems and conducting supply chain evaluations. He held a long-term position as the HIV program coordinator at Alliance Côte d’Ivoire and is a specialist in planning, research, capacity building and conducting evaluations of organizational progress. In fact, Dr. Esso developed the first health and demographic surveillance site in Côte d’Ivoire, which is based in Taabo. Dr. Esso is proficient in a number of data analysis software applications and teaches Epi-Info, Epi-Data, Epi 2000, SPSS, SAS, STATA, SPAD, Eviews, Access, PowerPoint, Excel, and Word software. In this evaluation, he was responsible for the following tasks: ● Developing evaluation design, methodology, sampling strategy, and data-collection instruments; ● Coordinating evaluation activities, including training data collectors, data collection, implementation, data management and quality assurance, and other, related tasks; ● Developing the data analysis plan and conducting qualitative and /or quantitative data analysis, as required; 88 ● Actively participating in activities with other team members during data triangulation, presentations, and report writing; ● Assisting the team leader in the completion of the inception report and writing the evaluation report in conformity with the scope of work; and ● Developing the final evaluation report, with quality assurance and timeliness of all deliverables, to USAID, and responding to all comments. Dr. Assane Coulibaly, Senior Local Subject Matter Expert Dr. Coulibaly earned a PHD in pharmaceutical sciences from Rouen University and has since gained 25 years of experience in the pharmaceutical industry. He served as the Deputy CEO of CIPHARM SA, where he coordinated supply chain management in the private and public sectors. He was also Vice President of the West African Pharmaceutical Manufacturers Association (WAPMA), which includes Côte d’Ivoire. Dr. Coulibaly has carried out consultancy assignments for UNIDO on the PACIR project (EU/ECOWAS) and for the Islamic Corporation for the Development of the Private Sector, an affiliate of the Islamic Development Bank. He also owns the pharmacy LES TULIPES in Abidjan. In this evaluation, his tasks included: ● Providing insight and knowledge with respect to the common practices and activities for delivering supply chain management assistance to public and/or private entities; ● Assisting in developing appropriate evaluation design, methodology, sampling strategy, and data collection instruments for evaluation of a supply chain management assistance intervention; ● Assisting in coordinating evaluation activities, including training data collectors, data collection, implementation, data management and quality assurance, and other related, tasks; ● Assisting in developing the data analysis plan and conducting qualitative and /or quantitative data analysis, as required ● Actively participating with other team members during data triangulation, presentations, and report writing; and ● Assisting the team leader in the completion of the inception report and writing the evaluation report in conformity with the scope of work. Team members were required to provide a written disclosure of conflicts of interest. 89 ANNEX V: Capability Maturity Model and Rankings The CMM tool encompasses the main supply chain functional areas (product selection, forecasting and supply planning, procurement, storage and stock management, distribution, waste management, and laboratories), and key measures or “catalysts” or “enablers,” which impact all functions of the supply chain. As shown in figure 4, the five enablers are: ● Processes and tools ● Infrastructure ● Management information system (MIS) ● Strategic planning and oversight ● Human resources Figure 1. Functional areas and catalysts/enablers covered by the CMM tool* *Note that the Dispensing function was not included in the NSCA for Côte d’Ivoire. For a particular functional area, each catalyst can be further broken down into a certain number of capabilities. For example, the catalyst, “Processes and tools” within the “laboratory” functional area includes, among other capabilities, standard operating procedures (SOP) for laboratory activities, clear procedures for the management of hazardous and flammable products, and a well-defined schedule for external quality assurance (QA) audits. The CMM rating for each of the capabilities is on a 1-5 maturity scale, based on a group of specific and well-defined criteria. Figure 5 illustrates this maturity scale as a staged representation, with criteria ranging from lowest to highest level for each capability area (1-5). 90 Figure 2. Maturity scale description 5 Best 4 Advanced Practices: Continuous 3 Qualified: Practices: improvements Processes are 2 Marginal: Processes are well defined well defined 1 Minimal: Basic and processes are Informal not used processes are followed with For example, as shown in table 1, for a warehouse’s capability in terms of SOPs, a maturity at the minimal level (1) means that “procedures are lacking,” whereas the best practices level (5) signifies that “detailed SOPs are available for all related storage procedures, that all employees are required to read them, and that the SOPs are in line with national and local guidelines.” 91 Annex VI - Questionnaire REPUBLIQUE DE COTE D’IVOIRE ______________ UNION – DISCIPLINE – TRAVAIL ____________ MINISTERE DE LA SANTE ET DE L’HYGIENE PUBLIQUE EVALUATION DE PERFORMANCE DU PROJET SUPPLY CHAIN MANAGEMENT SYSTEM EN COTE D’IVOIRE Octobre 2016 QUESTIONNAIRE ETABLISSEMENT SANITAIRE SECTION A : IDENTIFICATION N0 Questions et filtres Codes Abidjan 1 Q01 DEPARTEMENT Bouaké 2 Korhogo 3 San Pedro 4 Q02 Nom du centre ………………………………………………… |__|__] Q03 District |__|__] 92 N0 Questions et filtres Codes Q04 Autorité responsable 1=MS ; 2=ONG |__] Q05 Identifiant (sera créé automatiquement) SECTION B : INTRODUCTION ET CONSENTEMENT Bonjour. Je m’appelle ________________. Mes collègues et moi-même réalisons une étude sur le système de gestion logistique des ARV et intrants financée par l’USAID. Nous recueillons des données sur la disponibilité de ces produits ainsi que la manière dont vous les commandez. Nous visitons des centres de santé choisis sur l’ensemble du pays de manière aléatoire. Cette étude a pour objet de réunir des informations à jour sur la performance du système logistique et l’état de stock des produits et fournitures sanitaires essentiels relatifs au VIH/sida. Il ne s’agit pas d’une visite de supervision et les performances individuelles des membres de l’équipe ne sont pas évaluées. Les résultats de cette étude fourniront des informations visant à prendre des décisions et à encourager des améliorations. Nous aimerions poser une série de questions au responsable des approvisionnements sur les produits et fournitures relatifs au VIH/sida disponibles dans ce centre. De plus, nous aimerions regarder certains produits traceurs que vous avez en stock aujourd'hui ainsi que leurs conditions générales de stockage. Avez-vous des questions ? 1……………………………………………………………………………. 2……………………………………………………………………………. 3……………………………………………………………………………. Pouvons-nous commencer l’interview à présent ? Non 0 Oui 1 Merci. 93 Date Heure de début de l’enquête [__|__] h[__|__]mn [__|__][__|__][__|__] Nom de Durée de l’entretien : [__|__] mn l’enquêteur…………………………………………………………… ……. SECTION 1 : FONCTIONNEMENT DU CENTRE N° Questions et filtres Codages Passer à Caractéristiques du centre : Non Oui A. Route goudronnée au centre de 0 1 santé ? Q101 B. Électricité fonctionnant le jour de 0 1 la visite ? C. Eau courante dans le centre de 0 1 santé le jour de la visite ? D. Téléphone (ligne fixe ou mobile) 0 1 fonctionnant le jour de la visite ? Q102 Nombre d’années et de mois que vous travaillez dans ce A. Année [__|__] centre B. Mois [__|__] Quelle est la principale personne responsable de la 1. Infirmier gestion des produits sanitaires dans ce centre ? 2. Responsable clinique 3. Technicien de laboratoire 4. Préparateur Gestionnaire en Q103 pharmacie [__| 5. Pharmacien 6. Assistant médical 7. Assistant Pharmacien 8. Autre………………………….. Q104 La gestion des produits/du stock est-elle la fonction 1. Non [__| principale de cette personne dans ce centre ? 2. Oui Est-ce que vous utilisez les formulaires logistiques suivants Non Oui pour gérer les produits sanitaires dans ce centre ? A. Fiche de stock/fiche de contrôle 0 1 Q105 d’inventaire B. Journal de stock 0 1 C. Autre………………………….. 0 1 Quels sont les formulaires SIGL que vous utilisez pour les Non Oui rapports /les commandes ? A. Registre de dispensation ARV 0 1 Adulte B. Registre de dispensation ARV 0 1 Q106 Enfant C. Bordereau commande ARV 0 1 D. Bordereau commande intrant 0 1 E. Rapport mensuel ARV 0 1 94 F. Rapport mensuel intrant 0 1 G. Rapport mensuel 0 1 Attrition/inclusion H. Autre………………………… 0 1 Est-ce que les formulaires de rapport SIGL comprennent Non Oui les éléments suivants ? A. Le stock disponible 0 1 Q107 B. Quantités utilisées 0 1 C. Les pertes et les ajustements 0 1 Est-ce que le dernier rapport SIGL complété comprend N O NV les éléments suivants ? (vérifier avec le rapport rempli) A. Stock disponible 0 1 9 Q108 B. Quantités utilisées 0 1 9 C. Pertes et ajustements 0 1 9 Selon quelle fréquence ces rapports SIGL sont-ils envoyés au niveau hiérarchique supérieur ? 1. Une fois par mois 2. Une fois par trimestre Q109 |__] 3. Une fois par semestre 4. Une fois par an 5. Autre 1. Jamais Quelle est la dernière fois où vous avez envoyé une 2. Au cours du mois dernier Q110 commande/un rapport pour des produits à ce centre ? 3. Il y a 2 mois |__] 4. Il y a 3 mois 5. Il y a plus de 3 mois Comment avez-vous appris à remplir les Non Oui formulaires/dossiers utilisés dans ce centre ? A. Lors d’un atelier de logistique 0 1 Q111 B. Par une formation sur le tas 0 1 C. Jamais reçu de formation 0 1 D. Autre…………………………… 0 1 95 Q112 Combien de commandes d’urgence pour ARV avez-vous ………………………. commandes |__]|__] passées au cours des 3 derniers mois ? Qui détermine les quantités de réapprovisionnement de Non Oui ce centre ? A. Le centre lui-même 0 1 Q113 B. Un centre au niveau supérieur 0 1 C. Autre…………………………… 0 1 Comment sont déterminées les quantités de 1. Formule (tout calcul) Q114 2. Ne sait Pas |__] réapprovisionnement du centre ? 3. Autres moyens……………………. Qui est responsable du transport des produits vers votre Non Oui centre ? A. Fournisseurs locaux 0 1 Q115 B. Niveau supérieur 0 1 C. Ce centre les collecte 0 1 D. Autre…………………………. 0 1 Q116 Vos fournisseurs ont-ils eu des difficultés de transport 1. Non [__| pour vous acheminer les produits à temps ? 2. Oui 1. Moins de 2 semaines Q117 En moyenne, combien de temps environ s’écoule entre la 2. De 2 semaines à 1 mois [__| commande et la réception des produits ? 3. Entre 1 et 2 mois 4. Plus de 2 mois Quand avez-vous reçu votre dernière visite de 1. Jamais reçue 2. Au cours du mois dernier supervision ? 3. Entre 1 et 3 mois Si 1, allez Q118 |__] Vérifier le registre de supervision 4. Entre 3 et 6 mois à Q201 5. Il y a plus de 6 mois 6. Autre (spécifier) Votre dernière visite de supervision reçue, incluait-elle la 1. Non Si 0, allez Q119 2. Oui |__] gestion des médicaments à Q201 3. Ne sais pas Si Oui, incluait-elle ? N O NSP D. Vérification des fiches de stock 0 1 9 Q120 E. Vérification des rapports 0 1 9 F. Retrait du stock périmé 96 G. Vérification des conditions de 0 1 9 stockage des produits SECTION 2: SYSTEME LOGISTIQUE DE CHAINE DE FROID N° Questions et filtres Codages Passer à Disposez-vous d’un réfrigérateur en bon 1. Non Q201 état de fonctionnement pour stocker les kits 2. Oui |__] de test de dépistage du VIH ? 3. Non applicable Pour noter la température réelle, regardez le thermomètre qui se trouve à l’intérieur Q202 du réfrigérateur- la température idéale doit Température (en centigrades)_______________ |__]__]__] être comprise entre 0 et +8 degrés centigrades. (Prière de noter si le thermomètre est cassé ou absent.) Les réfrigérateurs sont-ils situés à une 1 Non Q203 distance d’au moins 50 centimètres de tout 2 Oui |___| objet environnant ? SECTION 3 : MAGASIN DE STOCKAGE Est-ce que les fiches de stock et les rapports 1 Non Q301 sont complétés en utilisant la plus petite 2 Oui |___| unité de comptage Q 302 Tableau 1. État du stock (spécifier une période de six mois complète et le jour de la visite) D. E. F. G. B. C. Fiche de Rupture de Nombre Nombre Rupture de stock Quantité de stock à stock de total de A. aujourd’hui ? produits jour ? derniers 6 ruptures jours de Produit (O/N) périmés (O/N) mois (O/N) de stock rupture SECTION 4 : CONDITIONS DE STOCKAGE N° Questions et filtres Réponses Codes Passer à Les produits prêts à être distribués sont disposés de 1. Non Q401 telle manière que les étiquettes d’identification et les 2. Oui |__] dates de péremption et/ou de fabrication sont visibles. 97 Les produits sont stockés et rangés permettant un 1. Non Q402 comptage de premier-périmé, premier-sorti et une 2. Oui |__] gestion générale. Les cartons et les produits sont en bon état et ne sont 1. Non pas endommagés. Si les cartons sont ouverts, les Q403 2. Oui |__] produits ne sont pas humides ou craquelés par suite de la chaleur ou du rayonnement Le centre sépare toujours les produits endommagés 1. Non Q404 et/ou périmés des bons produits et les supprime du 2. Oui |__] stock. 1. Non Q405 Les produits sont à l’abri de la lumière directe du soleil. 2. Oui |__] Les cartons et les produits sont protégés de l’eau et de 1. Non Q406 2. Oui |__] l’humidité en toute saison. La zone de stockage est exempte d’insectes et de 1. Non rongeurs. (Vérifiez visuellement les traces de chauve- Q407 2. Oui |__] souris et/ou rongeurs [déjections ou insectes] dans la zone de stockage.) La zone de stockage est sécurisée par un verrou et une 1. Non Q408 clé, mais est accessible pendant les heures de travail 2. Oui |__] normales, avec un accès limité au personnel autorisé. Le toit est maintenu en bon état pour éviter la 1. Non Q410 2. Oui |__] pénétration de la lumière, du soleil et de l’eau. Le local de stockage est maintenu en bon état (propre, 1. Non Q411 sans déchets, les étagères sont nettoyées et les boîtes 2. Oui |__] correctement disposées). L’espace et l’organisation sont suffisants pour les 10 Non produits existants et pour une éventuelle expansion Q412 11 Oui |__] (par exemple, réception de produits attendus dans un avenir proche). Le matériel de sécurité-incendie est disponible et 1. Non Q413 accessible (tout article permettant de lutter contre le 2. Oui |__] feu doit être pris en compte). Les produits sont rangés à 10 cm au moins au-dessus du 1. Non Q416 2. Oui |__] sol. Les produits sont rangés à 30 cm au moins des parois et 1. Non Q417 2. Oui |__] des autres piles de rangement. Q418 Qualité des données SIGL : Stock disponible au moment du rapport SIGL le plus récent Stock disponible (au moment du rapport SIGL le plus récent) B. C. Depuis le journal de D. E. Selon le rapport stock ou les fiches de % différence A. Motifs de la différence SIGL le plus récent stock au moment du (C–B) /B *100 Produit rapport SIGL 98 Q419 Différence, en pourcentage, entre la quantité commandée et la quantité reçue B. C. D. E. A. Quantité commandée Quantité reçue lors de la pour la dernière Date de passation dernière Date de réception de la Méthode/marque/produit période de commande de la commande commande/acquisition commande Q420 Taux de satisfaction des commandes calculé dans les entrepôts de distribution Nom du centre passant la commande :_______________________ Mois 1 Mois 2 Mois 3 A. B. C. D. E. F. Produit Quantité Quantité Quantité Quantité commandée Quantité commandée Quantité commandée reçue reçue reçue 99 Commentaires, notes, raisons pour lesquelles les commandes n’ont pas été honorées A. Nom de la personne interrogée ……………………………………………………………… B. Fonction …………………………………………………………… C. Numéro de téléphone mobile pour cette section [__|__][__|__][__|__][__|__] Notes/Commentaires : 100 1 Post-Evaluation Action Review Table for SCMS Cote d’Ivoire Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status 1.It is important to ensure that the follow up project to SCMS has an anchor of a certain level with the authorities like the DGS at the MOH. Rejected The DGS, Directorate General of within Ministry of Health oversees implementation of MoH policies. The PNDAP is a technical unit representing the DGS on supply chain issues. Implementing partner interacts with the technical with PNDAP. It is not expected for USAID Implementing partners to be holding technical discussion at the level of Director General of Health (DGS) n/a n/a m/a 2.The private sector should be better involved. Accepted USAID had awarded a new supply chain technical assistance contract under the name of Integrated supply chain technical assistance activity (IHSC-TA). The IP will to explore collaboration with Association of Private Clinics of Côte d'Ivoire (ACPCI) in the management of the supply chain in view of the specificity of private healthcare establishments. April 2018 To be included in IP’s FY18 work plan 3.A culture of accountability and responsibility, with clear consequences for decisions, actions, and inaction, should be promoted. Staff needs to Accepted To be included in IHSC￾TA’s work plan 2019 To be included in IP’s FY18 2 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status be empowered and clearly motivated to take responsibility for managing the quality and security of the supplies under their control. work plan 4.Advocate with key stakeholders to ensure that the change in status of the NPSP, from government-run to independent, does not make it lose its public-health and public-service perspective in favor of a narrow focus on autonomy and profitability. Rejected NPSP is set up as a non-for-profit local organization. A memorandum of understanding is established between the GoCI and NPSP to clarify expectations, roles and responsibility of each of the parties. A list of performance indicators is included in the MoU. The MoU also sets the service levels expected from NPSP n/a n/a n/a 5.Ensure that the new mechanism and PEPFAR clinical partners operate in a complementary manner, avoiding both competition and duplication of efforts. Rejected The design of the new supply chain technical assistance is based on the principle of no duplication with other interventions. n/a n/a n/a 6.Clarify that laboratory test supply is outside of the scope of ARV SCM responsibilities, as there are other mechanisms in the country for laboratory test supply. Rejected The division of procurement and supply chain roles between PEPFAR implementing partners is clearly defined n/a n/a n/a 7.Strengthen the laboratory-competence capacity of stock managers at the central level and peripheral or facility level. Accepted IHSCTA will include Technical assistance to NPSP to strengthen the Laboratory management components. By December 2018 Included in IHSCTA’s work plan 8.Improve communication between the next initiative and other partners involved in ARV supply chain management. Rejected The scope of the new Supply Chain TA mechanism is to work primarily with PNDAP to strengthen the capacity of n/a n/a n/a 3 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status CNCAM, the supply planning coordination platform of the ministry of Health. All stakeholders involved in supply chain strengthening activities participate in CNCAM meetings. Product Selection • Review and periodically disseminate the LNME and at all levels of the health pyramid. Confirmation of findings during in￾country discussions n/a The scope of work for new USAID’s Supply Chain Technical assistance activity includes a comprehensive list of interventions to help address weaknesses observed across the various supply chain functions n/a IHSC-TA is a 4 year activity Quantification: Supply Planning and Forecasting • Review the various LMIS currently in use by different programs in order to establish an integrated, national LMIS. • Integrate peripheral stock management into this automated national eSIGL. • Ensure a sustainable funding mechanism for maintenance and renewal of management tools, as needed. • Strengthen the coordination of procurement plans for different products and from different sources (NPSP-CIs, programs and donors). • Improve the order management and delivery process. Confirmation of findings during in￾country discussions n/a Same as above n/a n/a 4 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status Warehousing and Inventory Management • At the peripheral level, conduct inventories routinely and ensure the financial department is notified of discrepancies, in order to make accounting adjustments. • Register all items (including no-cost items) in enterprise resource planning (ERP) software and inventory records. • Validate all items received against delivery vouchers to ensure that all products ordered and received are placed in stock. • Train region- and district-pharmacists on computerized inventory management and provide adequate supervision thereof. • Complete the upgrading of all district pharmacies (82) to the standards of organization and storage of health products. • Strengthen the operational capacities of pharmacy departments in health districts by allocating more resources to improve the performance of LMIS. • Implement integrated management software at the NPSP level. Confirmation of findings during in￾country discussions n/a Same as above n/a n/a Distribution (Transportation) • Communicate a delivery schedule to all stakeholders, in order to ensure full knowledge of delivery times and optimize transport costs. • Conduct root-cause analysis of substandard delivery periods in order to identify and, consequently, bottlenecks. • Strengthen the distribution capacities of the districts using tools and levers such as distribution plans, follow-up protocols, reverse Confirmation of findings during in￾country discussions n/a Same as above n/a n/a 5 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status logistics, cold-chain management, vehicle maintenance, fuel allocation, etc. • Establish a standardized drug distribution system for health districts that could be funded by the 8% reimbursement of NPSP transportation costs. Waste management • Quarantine and dispose of unusable pharmaceuticals, in accordance with existing SOPs and guidelines. • Disseminate the "National Procedure Manual for the Management of Unsafe Pharmaceuticals" at the peripheral level of the supply chain. • Implement a process of decentralization of the destruction of expired pharmaceuticals • Ensure the traceability of pharmaceutical expiry information. • Collect and routinely destroy expired pharmaceuticals to prevent accumulation that clutters and limits storage. Confirmation of findings during in￾country discussions n/a Same as above n/a n/a Laboratory • Integrate the laboratory with the LMIS or a laboratory information management system and generate monthly reports to determine which products are expiring. • Develop and disseminate SOP for logistics management of laboratory products, inventory management, risk management and safety at all levels of the health system. • Establish a mechanism for laboratory-staff skills assessment and training. • Assess the ability of lower levels to store and handle hazardous products; develop a risk management plan for those lower-level facilities. Confirmation of findings during in￾country discussions n/a Same as above n/a n/a 6 Evaluation Recommendation Acceptance Status If not accepted, reason(s) for rejection Responsibility for Action Deadline for Impleme ntation Implemen tation Status • Improve laboratory supervision. • Improve the storage and management of hazardous products.