This publication is made possible by the generous support of the American people through the U.S. Agency for International Development (USAID) under the terms of Cooperative Agreement AID-OAA-14-000046. The contents are the responsibility of AIDSFree and do not necessarily reflect the views of USAID, PEPFAR, or the U.S. Government. AIDSFREE ZAMBIA ELECTRONIC LOGISTICS MANAGEMENT INFORMATION SYSTEM ENDLINE EVALUATION REPORT December 2019 This publication is made possible by the generous support of the American people through the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) with the U.S. Agency for International Development (USAID) under the Cooperative Agreement Strengthening High Impact Interventions for an AIDS-free Generation, number AID-OAA-A-14-00046. The information provided does not necessarily reflect the views of USAID, PEPFAR, or the U.S. Government. AIDSFREE ZAMBIA ELECTRONIC LOGISTICS MANAGEMENT INFORMATION SYSTEM END LINE EVALUATION REPORT December 2019 iv | Page AIDSFree The Strengthening High Impact Interventions for an AIDS-free Generation (AIDSFree) Project is a five-year cooperative agreement funded by the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) with the United States Agency for International Development (USAID) under Cooperative Agreement AID-OAA-A-14-00046. AIDSFree is implemented by JSI Research & Training Institute, Inc. with partners Abt Associates Inc., Elizabeth Glaser Pediatric AIDS Foundation, EnCompass LLC, IMA World Health, Frontline AIDS (formerly the International HIV/AIDS Alliance), Jhpiego Corporation, and PATH. AIDSFree supports and advances implementation of the U.S. President’s Emergency Plan for AIDS Relief by providing capacity development and technical support to USAID missions, host-country governments, and HIV implementers at local, regional, and national levels. Recommended Citation Strengthening High Impact Interventions for an AIDS-free Generation (AIDSFree) Project. 2019. AIDSFree Zambia eLMIS Endline Evaluation Report. Arlington, VA: AIDSFree. JSI Research & Training Institute, Inc. Plot 8658 Kudu Road PO Box 320087 Lusaka, Zambia Email: info@aids-free.org Web: aidsfree.usaid.go TABLE OF CONTENTS AIDSFREE ZAMBIA 1 Acronyms 7 EXECUTIVE SUMMARY 8 Introduction 10 Background 11 Overview of Zambia Supply Chain and eLMIS 12 Supply Chain Limiting Factors 15 Evaluation Objectives 18 Conceptual Framework 19 Methodology 20 Study Sample 20 Data Collection 21 Data Management and Analysis 23 Key Findings 25 Research Question 1: 25 To what extent has the eLMIS improved timeliness, frequency, and accuracy of reporting? 25 Research Question 2 27 To what extent has eLMIS improved data accessibility, visibility and quality? 27 Research Question 3 31 To what extent has eLMIS led to increased data use and/or data driven decision-making? 31 Research Question 4 32 To what extent is the eLMIS FE usable and acceptable among different users? 32 Research Question 5 34 To what extent has the eLMIS influenced supply chain performance through improved commodity availability? 34 Research Question 6 41 Research Question 7 46 What is the cost benefit of scaling to additional sites not yet covered by the eLMIS FE? 46 Discussion and Recommendations 48 Research Question 1 48 Timeliness, frequency, and accuracy of reporting 48 6 | Page Research Question 2 48 Data accessibility, visibility, and quality 48 Research Question 3 49 Data use and data-driven decision-making 49 Research Question 4 50 To what extent is the eLMIS usable and acceptable among different users? 50 Research Question 5 51 Commodity availability, stock status, and expiries 51 Research Questions 6&7 51 Costing: return on investment and cost benefit 51 Conclusion 53 References 54 Appendix 1. Phase i Evaluation Facility List 55 Appendix 2. Key Evaluation Indicators 58 Appendix 3. Questionnaires 62 Endline 18th February – 30th April, 2019 62 Zambia eLMIS Evaluation: DISTRICT HEALTH OFFICE Interview Guide 62 Name of Facilitator: ________________ 62 Endline 18th February – 30th April, 2019 67 Zambia eLMIS Evaluation: HEALTH CENTER AND HOSPITAL Interview Guide 67 Name of Facilitator: ________________ 67 Appendix 4. Realighment of supply chain logistics management information systems 71 ACRONYMS AIDSFree Strengthening High Impact Interventions for an AIDS-free Generation ARV ART antiretroviral antiretroviral therapy CE Central Edition CHAZ Churches Health Association of Zambia DHO District Health Office eLMIS electronic Logistics Management Information System EM Essential Medicine FE Facility Edition GHSC-PSM Global Health Supply Chain Procurement Supply Management HF health facility LMIS Logistics Management Information System MOH Ministry of Health MOS months of stock MSL Medical Stores Limited PEPFAR U.S. President’s Emergency Plan for AIDS Relief R&R report and requisition SATP stocked according to plan SCC stock control card SCMgr Supply Chain Manager (software) USAID United States Agency for International Development 8 | Page EXECUTIVE SUMMARY The implementation of the electronic logistics management information system (eLMIS) in Zambia started in 2014 with a six-month pilot followed by a nationwide rollout. The initial evaluation of the eLMIS was conducted in 2014. A second evaluation, an eLMIS midline and the Strengthening High Impact Interventions for an AIDS-free Generation (AIDSFree) Project baseline, took place in May 2017. The midline evaluation assessed the impact of the eLMIS on supply chain performance following its rollout of the Facility Edition (FE) to 100 high volume health facilities. As part of AIDSFree Project close-out, an endline evaluation was conducted from April to May 2019. The overall objective of this evaluation was to determine the benefits and impact of the eLMIS system on the supply chain of health commodities in Zambia. The study focused on both the eLMIS Central Edition (CE) and Facility Edition or FE. For the FE, the study looked at all the 580+ sites that were deployed at the time of this evaluation and the 100 Phase I facilities that were visited during the midline evaluation conducted in May of 2017. The CE focus was on the overall influence of the eLMIS on the supply chain of over 2,600 health facilities, inclusive of the 100 Phase I and other FE sites. The endline evaluation also included a costing model that addressed the total upfront and ongoing cost of ownership. The study attempted to answer the following research questions: 1. To what extent has the eLMIS improved frequency, timeliness, and accuracy of reporting? 2. To what extent has the eLMIS improved data accessibility, visibility, and quality? 3. To what extent has the availability of eLMIS data led to increased data use and/or data-driven decision-making? 4. To what extent is the eLMIS FE usable and acceptable among different users? 5. To what extent has the eLMIS contributed to improved overall supply chain performance? 6. Based on the eLMIS’s current scale of implementation, what is the return on investment? 7. What are the costs and benefits of expanding to additional sites not yet covered by the eLMIS? This evaluation used a mixed methods approach. The data collection included Medical Stores, District Health Offices, and health facility surveys using quantitative analysis and qualitative interviews, a district survey, a desk review of secondary data involving performance indicators, and a cost benefit analysis. Key findings Timeliness and frequency of reporting increased at endline. There was an overall average improvement in reporting of 12% from Pre-eLMIS to endline. Facilities with FE reported on time 2% more frequently. 1. User session in CE have increased by 37% from midline to endline. 2. District supervisors are using eLMIS to improve stock status at facilities by reviewing stock levels and transferring stocks between facilities to avoid stock￾outs and expiries. 3. There was a 95 percent user acceptance rate of the eLMIS software suite. 4. Facilities with FE experienced 15.5 percent fewer stock-outs than similar facilities without FE. 5. Although eLMIS FE has increased supply chain costs, as of 2018, the eLMIS’ program implementation and support over five years coincided with a 15.5 percent increase in commodity availability. 6. Given fixed overheads for maintaining the overall eLMIS, further expansion to an individual primary health facility represents a good opportunity for economies of scale. A consistent reduction in the percentage of implementation and support costs per health facility is a sign of positive adoption, maturity and sustainability. The findings of this evaluation show that despite challenges in the supply chain that led to low order fill rates during the endline, eLMIS has contributed to general improvement in efficiency, cost and commodity security. Automation of inventory control at health facilities using with eLMIS Facility Edition resulted in better performance of the supply chain than in those without. Based on the indicators on commodity availability, stock status, and reporting, a greater proportion of health facilities that have the eLMIS FE were better stocked, reported on time and achieved a higher frequency of reporting with the two periods of review. The eLMIS Facility Edition sites recorded reporting rates averaging 99.5 percent, an indication that eLMIS FE has improved the ability of health facility staff to generate and submit their reports and requisitions as required by a pull type of logistics system. 10 | Page INTRODUCTION The World Health Organization identifies information systems as a part of the six building blocks essential for a strong client centered health system (WHO 2010). Recognizing the importance of a robust supply chain management information system, the United States Agency for International Development (USAID), with funding from the President’s Emergency Plan for AIDS Relief (PEPFAR) supported the Government of the Republic of Zambia in (GRZ) to automate logistics management information systems. The implementation of an electronic Logistics Management Information System (eLMIS) at central medical stores and selected high- to medium￾volume health facilities countrywide began with a pilot phase in 2014 and national rollout starting 2015. In line with the Joint United Nations Programme on HIV/AIDS (UNAIDS) 95-95-95 HIV global initiative, automation of data capture and reporting are aimed at complementing actions to improve commodity availability and ensure uninterrupted service provision at health facilities. One of the pillars of achieving epidemic control is a strong, data-driven logistics management information system. The Zambian National Health Strategic Plan 2017–2021 supports the achievement of the global goal to end HIV/AIDS by 2030. The USAID/PEPFAR and GRZ initiative to roll out eLMIS since 2014 has enabled over 2,600 health facilities to submit requisitions within one day, reducing lead time for report submission by over 10 days. The eLMIS Facility Edition (FE), a scalable enterprise-level inventory management, dispensing, and reporting system is being used in more than 600 high￾and medium-volume health facilities. There have been a number of phased evaluations of the eLMIS. The primary focus of these phased evaluations is to determine whether the eLMIS has contributed to anticipated improvements in performance of the supply chain and assess the effectiveness of eLMIS adoption. The other aspects of this evaluation are to examine anticipated improvement in data quality and conduct cost benefit and return on investment assessment. The 100 Phase I eLMIS FE facilities were used to assess the performance at both the midline and endline periods of this evaluation. These Phase I facilities have used eLMIS FE for at least two years and have been subjected to a range of interventions over an extended duration. BACKGROUND The Zambia National Health Commodity Logistics Systems are designed with the flexibility to manage all health commodities. The categories currently managed include HIV, select TB commodities, cancer, laboratory commodities, malaria products, reproductive health, and a range of essential medicines and supplies. The logistics system uses a pull requisition system that reports on essential data (consumption/usage, stock on hand, and losses and adjustments) on a monthly requisitioning cycle. These reports are submitted to the central medical stores, Medical Stores Limited (MSL), which then supplies commodities based reported data and forecasted need for facilities. Whereas reports from hospitals levels 1 to 3 are reviewed and approved by MSL, reports from health centers and health posts are reviewed and approved by district health office (DHO) supervisors before MSL creates orders. Throughout its years of implementation, eLMIS was supported by various stakeholders. The user-centered design principles and the collaborative development methodologies implemented through the life of the project have been critical in keeping stakeholders constructively engaged in the activities. Figure 1 below shows some of the partners who have actively supported the implementation of eLMIS in Zambia. The late Hon. Minister of Health, Dr. Joseph Kasonde, launched the implementation of eLMIS on February 20, 2014. This was followed by a pilot phase involving 45 health facilities and subsequent scale-up to more than 600 high- and medium-volume facilities to￾date. In recognition of the critical need to improve and sustain efficiency of the supply chain, stakeholders have continually participated in the implementation process. USAID PEPFAR (AIDSFree) SOFTWARE: MOH / ICT, OPENLMIS, CDC / BROADREACH, PCI, CLINICAL: SAFE, EQUIP, DISCOVER-H, UMB, CIDRZ, CRS SUPPLY CHAIN PARTNERS GHSC-PSM, CHAZ, UNFPA, PCI GRZ: MOH, MSL, UNZA, SZI, AUDITOR GENERAL Figure 1: eLMIS Partners 12 | Page Apart from the 45 pilot facilities that were selected through stratified sampling, the rollout sites were followed using a purposive selection method that prioritized high￾volume consumers of health commodities, availability of critical antiretroviral therapy (ART) services, and availability of electricity and cellular networks. As shown in Figure 2, eLMIS implementation includes site assessment, network deployment, user training, on-site initiation, and technical support and supervision. Figure 2: eLMIS Implementation Process Overview of Zambia Supply Chain and eLMIS The eLMIS comprises a suite of open-source applications that automate the national logistics business processes, depicted in Figure 3. Its implementation in Zambia began in 2014, replacing the previous desktop application (Supply Chain Manager or SCMgr). Using the eLMIS Central Edition (CE), all health facilities submit their reports either through web-based data entry by district supervisors or electronic facility-level transmission from eLMIS FE, reducing the reporting lead-time by over 10 days. Churches Health Association of Zambia (CHAZ) is a partner of the Ministry of Health. Its Pharmaceutical Warehouse, as depicted in Figure 3, complements the government-owned MSL in storage and distribution of essential medicines for faith￾based affiliated health facilities, more than 90 percent of which are using eLMIS FE. Rollout of eLMIS FE began with a pilot phase in 2014, targeting high- to medium-volume facilities. It automates inventory management, dispensing of medicines and generation of requisitions at health facilities. Now deployed in over 600 facilities, eLMIS FE has facilitated standardization of workflows and business processes, inevitably leading to improved compliance with logistics management best practices, easier skills transfer, and adoption among users. The 600+ high- to medium- volume facilities that use eLMIS FE consume over 80 percent of antiretrovirals (ARVs) and laboratory commodities in the country, as shown on the map in Figure 4. This implementation coverage led to an early realization of anticipated outcomes and may continue to influence commodity security beyond what this evaluation can reveal. Figure 3: Overview of commodity and Information Flow in the Zambia Commodity Logistics Systems 14 | Page Figure 4: Facilities with FE as of 30.Sept.2019 Figure 5 below shows the eLMIS implementation roadmap. There has been increased buy-in for adoption of eLMIS by all key stakeholders. In 2018, the Ministry of Health instituted the use of eLMIS across all health facilities in the country and directed all cooperating partners to collaborate on this initiative. 42 eLMIS FE sites (73% ART Clients) 66 eLMIS FE sites (67% ART Clients) 111 eLMIS FE sites (90% ART Clients) 82 eLMIS FE sites (73% ART Clients) 51 eLMIS FE sites (84% ART Clients) 97 eLMIS FE sites (95% ART Clients) 42 eLMIS FE sites (88% ART Clients) 48 eLMIS FE sites (71% Clients) 35 eLMIS sites (84% ART Clients) 90 eLMIS FE sites (77% ART Clients) 664 facilities with FE • ~25% of all HFs • 30% of all ART sites • 83% of ART Clients captured at FE sites Figure 5: eLMIS Implementation Roadmap Supply Chain Limiting Factors The supply chain in Zambia experienced a number of changes during the periods under evaluation. As some of these changes directly or indirectly influence the evaluation findings, we have taken time to identify and outline them. 1. Change in commodity distribution/replenishment schedule To cope with internal operational constraints, the central-level warehouse had to change the facility replenishment schedule from monthly to bi-monthly. However, the maximum stock levels at health facilities could not be similarly increased, since this would likely lead to shortage of storage capacity, requiring a higher-capacity transportation system and provisions for short-shelf-life and cold storage commodities. This change happened at midline and as shown in Figure 7, between midline and endline, it was observed that order processing frequency became erratic and declined by more than 10 percent. Furthermore, to cope with erratic re-supply, the rate of commodity transfers between facilities increased, as shown in Figure 6, continuously increased between 2016 and 2019. 16 | Page Figure 6: Trend of Commodity Transfer Transactions at Health Facilities (2016 – 2019) Consequently, the increase in facility commodity transfers coincides with a decrease in the rate of orders processed or fulfilled. Figure 7. Percentage of Requisitions Converted to Order, March 2017 –February 2019 2. Central-level stock-outs of some essential medicines During the endline evaluation, the MOH announced that there was a shortage of Health Centre Essential Medicines kits. Although the incidence of stock-outs was not 0 1000 2000 3000 4000 5000 6000 7000 TRANSFER_IN TRANSFER_OUT 0% 10% 20% 30% 40% 50% 60% 70% 80% Mar-17 Apr-17 May-17 Jun-17 Jul-17 Aug-17 Sep-17 Oct-17 Nov-17 Dec-17 Jan-18 Feb-18 Mar-18 Apr-18 May-18 Jun-18 Jul-18 Aug-18 Sep-18 Oct-18 Nov-18 Dec-18 Jan-19 Feb-19 very high, they may have caused a slight decline in stock availability at between midline and endline. 3. ART satellite sites ART logistics system is designed to link “satellite” sites to ART parent facilities. The parent sites provide supervision and support re-supply of commodities to the satellite sites. However, as the parent facility acts as a hub, it aggregates requisition data for satellite sites. This limits the visibility of individual satellite facility data at the central level. The effect of “invisibility” is further aggravated by the systematic increase in the number of ART satellite sites in the past four years. 4. Parallel information systems in health facilities As implementation of health information systems continues to be scaled up, the need to re-align workflows, implement data interchange, and ultimately minimize duplication increases. This need has affected timeliness in capturing inventory control data. 18 | Page EVALUATION OBJECTIVES The overall objective of this evaluation is to determine the benefits and impact of the eLMIS system on the supply chain of health commodities in Zambia. The study focuses on both the eLMIS Central and Facility Edition. For the FE, the study looked at all the 580+ sites that were deployed at the time of this evaluation and the 100 Phase I health facilities that were visited during the Midline evaluation conducted in May of 2017. The CE focus was on the overall influence of the eLMIS on the supply chain of over 2600 health facilities, inclusive of the 100 Phase I and other FE sites. The Endline also included a costing model that addressed the total upfront and ongoing cost of ownership, (questions 6&7). These questions were added to the cost-benefit analysis conducted during the eLMIS Midline. The study attempted to answer the following research questions: 1. To what extent has the eLMIS improved frequency, timeliness, and accuracy of reporting? 2. To what extent has the eLMIS improved data accessibility, visibility, and quality? 3. To what extent has the availability of eLMIS data led to increased data use and/or data-driven decision-making? 4. To what extent is the eLMIS FE usable and acceptable among different users? 5. To what extent has the eLMIS contributed to improved overall supply chain performance? 6. Based on the eLMIS’s current scale of implementation, what is the return on investment? This portion builds on the analysis in the Midline evaluation by including additional outcome measurements and updated figures to capture total cost of ownership for the system. This question compares the initial investment and ongoing costs to financial and supply chain performance benefits. 7. What is the cost-benefit of expanding to additional sites not yet covered by the eLMIS? The Government of Zambia and its partners currently have the option of expanding the eLMIS FE to additional sites. This question considers the specific projected costs and benefits of expansion to new sites given the current level of investment. The costs and savings considered in the model are only those attributable to eLMIS and relevant points of comparison for operation and maintenance of the paper-based LMIS. Conceptual Framework The conceptual framework described in Figure 8 outlines the mechanisms through which the eLMIS improves the supply chain by generating and rendering for use, ultimately enhancing supply chain performance, supporting improved access to health care services and contributing to improved health outcomes. Additionally, the conceptual framework illustrates the integral role of data visibility and quality in influencing data-driven decision-making to improve services. Figure 8: eLMIS Conceptual Framework Better Health Outcomes Increased Use of health Services Improved Supply Chain Outcomes • Quantification • Transportation • Inventory Management • Capacity management • Data management • Design and planning • Monitoring and evaluation eLMIS intervention: Web-based access of central level; facility edition on/offline computer-based system at selected HFs. Platform Utilization: Satisfaction with usability of platform Accessibility Improved reporting: • Frequency of reporting • Timeliness of reporting • Accuracy of reporting Improved data: • Increased accessibility • Improved visibility • Improved data quality Improved data use: • Increased data utilization • Improved data-driven decision making 20 | Page METHODOLOGY This evaluation used a mixed methods approach. The data collection included MSL, DHO, and health facility surveys using quantitative analysis and qualitative interviews; a district survey; a desk review of secondary data involving performance indicators; and a cost-benefit analysis. The data collection included field visits to complete two types of surveys: health facility and DHO surveys. The health facility surveys included a quantitative and qualitative questionnaire. For the DHO survey, only a qualitative interview was administered to either the laboratory or the pharmacy personnel. Study Sample The study sample was based on three main components. • Assessing the impact of two versions of eLMIS- Facility and Central Editions (FE & CE), on the performance of the supply chain o The phase I (100) facilities using FE o All the 2,600+ health facilities using CE • Assessing eLMIS user’s perceptions • The cost-benefit analysis As was done for the midline, examination of the FE’s effects only looked at the 100 Phase I health facilities, whereas for the CE assessment, all 2,600+ health facilities were included and analyzed by ARVs, HIV test kits, essential medicines, and laboratory program areas to understand the overall supply chain impact of eLMIS Central Edition. A qualitative assessment was conducted to determine eLMIS users’ perceptions using key informant interviews. Respondents included either the laboratory or the pharmacy department at 34-selected health facilities and 16-selected DHOs. In addition to the 34 health facilities selected from 100 eLMIS FE Phase I, and 16 DHOs, MSL was included in the qualitative key informant interviews. The cost-benefit analysis entailed a review of the cost of eLMIS implementation and support, in relation to the resultant operational savings generated. Table 1 outlines the breakdown by level of the 98 phase I eLMIS FE facilities visited at endline. A full list of health facilities visited for field data collection is in Appendix 1. Table 2 outlines the distribution of respondents across departments at targeted facilities. Table 1. Health Facility Selection by Level and Location Facility Province Number of Facilities by Level Level 3 Level 2 Level 1 Health Center Total Central 0 2 5 4 11 Copperbelt 1 9 3 9 22 Eastern 0 2 5 1 8 Luapula 0 2 2 1 5 Lusaka 1 2 3 13 19 Muchinga 0 1 2 1 4 Northern 0 2 1 2 5 North Western 0 2 0 1 3 Southern 1 6 3 4 14 Western 0 0 7 0 7 Total 3 28 31 36 981 Table 2: Health Facility Respondent Distribution by Department Province No. of Health Facilities Pharmacy Numbers Laboratory Number Total Central 11 11 11 22 Copperbelt 22 21 22 43 Eastern 8 7 8 15 Luapula 5 5 5 10 Lusaka 19 19 15 34 Muchinga 4 4 4 8 North 5 5 5 10 North Western 3 3 3 6 Southern 14 14 14 28 Western 7 7 7 14 Total 98 96 94 190 Data Collection The same methods were used to collect and analyze data at midline and endline. Three teams consisting, of MOH and AIDSFree staff, collected data in all 10 provinces, for a duration of five weeks. Primary data were collected through structured interviewer administered questionnaires. The data collection teams 1 Data were not collected in two of the 100 Phase I facilities due to lack of suitable respondents. 22 | Page included three personnel; two AIDSFree and one MOH District and/or Provincial Health Office staff member. The number of eLMIS Facility Edition sites visited represents 17 percent (100) of the total implementation coverage at endline (580 facilities). Respondents included key staff at MSL, DHOs and health facilities (i.e., hospitals and health centers). The quantitative interviews were completed at 96 of the 100 Phase I eLMIS FE facilities at both the pharmacy and laboratory departments (See Appendix 3 for questionnaires). These interviews focused on reporting, internet connectivity, computer hardware and software, stock-outs of selected tracer commodities, and inventory management practices at the health facilities. None of the DHOs had quantitative interviews completed, as they do not manage stock for their use, but rather as a pass-through to health facilities within their jurisdiction. All quantitative data were collected using mobile devices and immediately uploaded to a web-based application called Magpi. For the qualitative interviews, staff from randomly selected DHOs and health facilities from the 100 Phase I eLMIS sites were interviewed at either the laboratory or the pharmacy departments. The interview’s focus was on understanding the user’s perspective of eLMIS and its impact on their day-to-day reporting responsibilities at district and health facility levels. Qualitative interviews were captured using mobile phone recording and later transcribed. For interviewees who did not consent to recording, the evaluation team used paper-based tools to capture the interview. The team used routine data from eLMIS Central Edition for comparative analysis of performance indicators at pre-eLMIS, midline and endline. The comparison between Pre-eLMIS and eLMIS (midline and endline) periods in the third component of the study looked at supply chain performance in all 2,600+ health facilities as influenced by the implementation of eLMIS. This data included Phase I eLMIS FE facilities, compared as a whole, to the pre-eLMIS period. An indicator list based on data derived from eLMIS CE and SCMgr databases is included in Appendix 2. Financial data detailing eLMIS implementation and support costs were largely drawn from AIDSFree project accounting records. Project staff supplied tracked costs in line with eLMIS implementation and operation activities that were complemented by non-accounting management records to provide additional detail on volumes and timing. The team used routine data from eLMIS Central Edition for comparative analysis of performance indicators at pre-eLMIS, Midline and Endline. The comparison between pre-eLMIS and eLMIS (Midline and Endline) periods in the third component of the study looked at supply chain performance in all 2600+ health facilities as influenced by the implementation of eLMIS. This data included Phase I eLMIS FE facilities, compared as a whole, to the pre-eLMIS period. An indicator list based on data derived from eLMIS CE and SCMgr databases is included in Appendix 2. Data Management and Analysis 1. Quantitative Data Key indicator data were extracted in excel format from the eLMIS CE for the endline period using Metabase BI tool. Quantitative indicators that were not available in eLMIS were collected using the facility and district level questionnaires. The data were captured using the Magpi mobile application. A secondary data repository was created to store data electronically on the AIDSFree servers. Access to these data was limited to the research team members from AIDSFree Zambia offices. A total of 190 interviews were completed and uploaded to the Magpi server. To assess change in supply chain performance between pre-eLMIS, midline and endline, descriptive statistics on key quantitative indicators. Using Excel, PowerBI, and metabase analytics system, cross tabulations were performed and data outputs used during data validation and development of a data framework. Subsequently, specific indicators were evaluated for patterns, connections, and themes that provide insights to the research questions and indicators. 2. Qualitative Data Interviews were conducted at health facilities, DHOs, and MSL. Each data collection team had a smartphone to record the interviews, but in cases where the interviewee was not comfortable with the recording, the interviewer and the MOH representative transcribed word-for-word responses on hard-copy questionnaires. This process was followed to ensure the accuracy and completeness of the transcripts and to share understanding of the respondents’ views and experiences. Following reconciliation of the recordings and notes, the research team compiled a comprehensive final transcript of the interview and saved an electronic copy. The completed interviews did not include any personal identification data. Members of the research team reviewed all qualitative transcripts for themes and patterns. The themes were derived from the questions that the qualitative data intended to answer. An initial set of predefined coding categories, based on the research questions and the conceptual framework, were used to guide analysis of the transcripts. Additional codes were drawn directly and inductively throughout the data analysis process. References were categorized into codes, and the study team 24 | Page analyzed the findings to provide an explanation of the detail, variation, meaning, and nuance. 3. Costing Data Project staff supplied tracked costs in line with eLMIS implementation and operation activities that were complemented by non-accounting management records to provide additional detail on volumes and timing. Additional cost-related assumptions, including resource requirements for project operations, come from interviews with project management staff and review of existing records. Human resource labor value and hours of effort required for eLMIS and paper LMIS completion come from a survey executed during the eLMIS midline evaluation (detailed further in the midline evaluation report). Finally, relevant performance data included cost-effectiveness and cost-benefit analyses which came from the same sources used for commodity availability of the evaluation: historical eLMIS reports (descriptive statistics), covering March 2018 – February 2019. In some cases, these analyses directly compare eLMIS FE to paper LMIS performance within these datasets. The three main components of this evaluation are summarized in Table 3. Table 3: Summary of Methodology Impact on Supply Chain User Perceptions Cost Benefit Analysis Research Question 1,2,3,5 2,3,4,5 6,7 Data Type Quantitative Qualitative Quantitative Sample -100 facilities using eLMIS FE -All facilities using CE Randomly selected 16 DHOs and 34 HFs from the 100 eLMIS FE sites Implementation and support costs from accounting and management records Focus -Reporting rates -Stock status -Software -Infrastructure Understanding user perceptions of eLMIS Total cost of ownership Data Analysis Microsoft excel spreadsheets Reviewed for themes and patterns using coding categories Cost benefit analysis Data Management Local AIDSFree servers Local AIDSFree servers Local AIDSFree servers Tools Used -Magpi App -eLMIS CE -Metabase Interview transcripts Desk review KEY FINDINGS Research Question 1: To what extent has the eLMIS improved timeliness, frequency, and accuracy of reporting? Reporting Timeliness: Figure 9 shows the average on-time reporting rates for the midline and endline periods by program area for eLMIS FE and all facilities reporting in CE (inclusive of FE and non-FE facilities). Both facility types (eLMIS FE and all facilities in CE) recorded significant percentage increase in all program areas with FE facilities recording slightly higher on time reporting at an average of 72 percent across program areas, while all facilities in CE averaged 70 percent across program areas. Therefore, both FE and CE have improved reporting timeliness, but facilities with FE have a slightly higher timeliness statistic. Figure 9: Average On-Time Reporting by Program Area - Midline and Endline periods Reporting Frequency: Figure 10 shows the average reporting rates for all facilities in CE by program area for different periods. On average, there was a 6 percent increase in reporting rates across all program areas from the pre-eLMIS period to midline. furthermore, between the midline and endline, a 5 percent average increase across program areas was recorded, an indication of continued improvement. 60% 76% 40% 75% 40% 62% 21% 60% 50% 72% 33% 70% 52% 77% 31% 73% ELMIS FE AT MID￾LINE ELMIS FE AT END￾LINE ELMIS CE AT MID￾LINE ELMIS CE AT END￾LINE Antiretroviral Drugs Essential Medicine HIV Test Kits Laboratory Commodities eLMIS FE eLMIS CE 26 | Page Figure 10: Average Reporting Rates: Pre-eLMIS, Midline, and Endline Periods Further analysis on reporting rates was done between Phase 1 FE facilities and all facilities at Endline. Figure 11 shows that there was a difference of 1 percent in average reporting rates between Phase 1 eLMIS FE facilities and all other health facilities in the country. Although not signification, there is a consistent difference in overall performance between eLMIS FE and sites with non-eLMIS FE sites. Figure 11: Average Reporting Rates for eLMIS Phase I Facility Edition – Midline and Endline Reporting Accuracy: As stated in the baseline/midline eLMIS Evaluation report, “eLMIS was found to produce more accurate reports than SCMgr. This is because FE and CE pre-generate beginning balances based on the previous month’s closing balance and flag discrepancies in entries. (A report was considered to be accurate if the closing balance for the previous month matched the beginning balance for the reporting month.)” Unfortunately, given the data available, it was not possible to analyze reporting accuracy any further. As noted in the next section, there were 93% 88% 78% 87% 93% 94% 94% 92% 99% 99% 98% 96% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ARV EM HIV Test Kits Lab Pre-eLMIS (March 12 - Feb 13) Mid-line (March 16 - Feb 17) End-line (March 18 - Feb 19) 99% 100% 99% 100% 98% 99% 96% 98% 0% 20% 40% 60% 80% 100% ARV EM HIV Test Kits Lab All Sites End-line Phase I FE Sites End-line concerns about data quality, but staff mentioned in the qualitative interviews that reporting accuracy has improved. In the recommendations, we suggest further study to ensure the quality of data reported. Research Question 2 To what extent has eLMIS improved data accessibility, visibility and quality? Data Accessibility: To assess data accessibility, the evaluation team used two sets of data; quantitative analysis of the user sessions on eLMIS CE, and qualitative interviews with system users at selected health facilities, DHOs and MSL. Figure 12: eLMIS Central Edition Average User Sessions Figure 12 shows an increase in average user sessions on eLMIS CE of 58.1 percent between midline and endline. An indication of increased adoption and use of data in the eLMIS CE for supply chain decisions. Similarly, respondents to qualitative interviews pointed out the impact of improvements in data accessibility (Box 1). The outcomes are also influenced by user trainings, as observed over the life of project. 4964 7846 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Mid-line (March 16 - Feb 17) End-line (March 18 - Feb 19) User Sessions Period 28 | Page Data Visibility: One of the strategies for transitioning to data-driven decision￾making was to enhance the data visibility that supports all levels of the supply chain. The eLMIS CE is web-based and is accessible to all key stakeholders and operators of the supply chain. The user-friendly reports address strategic and operational data requirements. Automated messaging and specialized data extracts further strengthen workflows and foster system efficiency. Responses from the interviews (Box 2) indicated appreciation of data visibility. According to interview respondents, eLMIS has enhanced visibility as all data can be easily accessed by anyone with appropriate user rights, for supply chain monitoring and analysis. Box 2 “eLMIS is a very powerful tool, I can see logistics data for any part of the country. Since its introduction it has impacted the efficiency of service delivery in Zambian Health logistics.” - Former Permanent Secretary Ministry of Health “It’s quite interesting how we manage reagents in the eLMIS. We have different types of adjustments, missing, lost, computer generated and so on. So every time I see these I want to know what happened to the reagents for such an adjustment type to be used. I think for me that is the most important thing, it’s not about how much we have or don’t have but why is this missing. I want to find out what happened.” -Hospital Biomedical Scientist “With this system available, no one can cheat me with what is happening with stock availability and stock imbalances for the facilities within my district. It is a very good tool and I will continue using it” - District Health Director Box 1 “The thing that I like about eLMIS is being able to have all the information in one place, instead of having to go through the files” – District Biomedical Scientist. “Yes. I can easily get any information I want concerning the drugs that we have been using even like last time I managed to check from 2016 how the drugs where coming in and how they were being used. And I also discovered the ones they have been receiving and the ones they have not been receiving using the eLMIS.” – Health Center Pharmacy Technologist Data Quality: The respondents at both DHOs and health facilities reported an improvement in the quality of data in the eLMIS as compared to the paper-based system. One of the elements that define data quality is timely updates of records within a health facility. An eLMIS user at the facility is expected to update records in real time, meaning at the time of executing each transaction. A key objective of implementing electronic systems is to reduce use of parallel manual (paper-based) records as regulations permit. Paper records could then only be maintained as backup in case of system outage due to some technical challenge. Timeliness of updating electronic stock control cards (eSCCs) at the 100 Phase I FE facilities dropped by 15 percent, from an average of 63 percent to 48 percent, between midline and endline, as depicted in Table 4. This may be attributed to recent electricity outages that affects facilities that do not have solar UPS (Uninterruptible Power Supply) systems. Collaboration with stakeholders to improve power backup capacity at health facilities is critical in addressing this challenge. Box 3 “…Before we started using central edition you find that facilities were reporting direct to DHO then you report for them to MSL and the quality of data was compromised but this time if ..if for example they are reporting this month for the previous month if something is wrong on the beginning balance we will be able to know that here quality is compromised then you tell them to change to rectify their problem. Before yes, after like what I have mentioned the beginning balance you find that this month they recording on a different quantity the next month just like that. And some facilities up to now are not conversant with the REMMS especially the new members of staff they have just deployed.” - District Pharmacy Technologist “I think it has changed, because if you have made an error, it will be able to tell you to say; No this does not match with what we have in the system so that you can go back and check” - Hospital Biomedical Scientist 30 | Page Table 4: Completeness of Electronic Stock Control Cards for eLMIS Phase I Health Facilities at Midline and Endline Products Midline Endline Product Category Product Name No. of Facilities % Updated on Time No. of Facilities % Updated on Time ARV Tenofovir 300 mg/Lamivudine 150 mg/Efavirenz 400 mg (TLE) 94 60.6 93 49.5 ARV Abacavir 30 mg/Lamivudine 60 mg Lopinavir 80 mg/Ritonavir 20 mg- (ABC/3TC) 93 63.4 92 52.2 ARV Lopinavir 80 mg/Ritonavir 20 mg ( LPV/r) 88 63.6 93 47.3 EM￾Antibiotic Cotrimoxazole Tablets 480mg 93 59.1 96 46.9 Malaria Artemether 120 mg/Lumefantrine 20 mg (ALs) 1*6 tabs 90 61.1 93 53.8 Malaria Malaria rapid diagnostic test (RDT) 60 63.3 81 39.5 RH Depo-Provera 68 57.4 82 51.2 RH Oral Combined pill 71 62.0 81 38.3 RH Male condoms 77 55.8 86 36.0 Lab BD Facs Count CD4% reagent 71 60.6 85 41.2 Lab EDTA Vacutainer (4 ml) 78 60.3 92 35.9 Lab Rapid test kit for syphilis (RPR) 76 59.2 90 56.7 Lab DBS Bundles for 50 tests 74 70.3 85 63.5 Lab ABX Minoton (Minidil) 55 83.6 75 65.3 HIV Tests Determine HIV test kits 83 59.0 93 43.0 Although eLMIS has been commended for reducing workload and users’ perception is that quality has improved, there are still other data requirements that health workers must attend to in parallel. At the 98 Phase I facilities visited, 68 percent of respondents indicated that they are tasked with updating other electronic information systems, such as SmartCare, and that made it difficult update eLMIS at the time of transaction. These same respondents noted that the eSCCs were typically updated weekly, but not always at the time of the transaction, meaning that the data reported could still be accurate. This calls for expedited implementation of integrated information systems, an initiative that is underway. At the time of this evaluation, the team was not able to validate the quality of the data reported verses the actual data at the site. Research Question 3 To what extent has eLMIS led to increased data use and/or data driven decision￾making? Data Use for Decision-Making: To determine how eLMIS has strengthened the use of data for supply chain decision-making, the team investigated dependence on data to mitigate stock-outs. Supervisors and program managers at various levels have the knowledge and skills to interpret eLMIS data and act on risks to mitigate stock-outs. Data visibility can be said to have facilitated redistribution of products at the facility level to counter the effect of decreasing rate of order fulfillment. In qualitative interviews (Box 4), district supervisors or health facility staff noted that using relevant stock status reports in eLMIS CE enabled them to improve commodity management across facilities. To explore the validity of these statements, eLMIS CE data were extracted on commodity movement between health facilities (“transfer in” and “transfer out”). These transfers are done by a facility or district supervisor, whenever there is a need to redistribute/share limited stocks and avoid costly disruption of health services. The data extract from eLMIS CE shown in Figure 13 validates supervisors’ statement that they are using the data in eLMIS to complete these transactions. Figure 13 shows the Box 4 “I run the reports after finishing entering the report so that you have idea of which facility is understocked or overstocked with particular product so that you can use that as a decision making tool to know that this facility has a lot it can share with other facilities or we have this product as the district an overstock will share with other districts.” – Facility staff “Of course we also use it in terms of redistributions then apart from that one if you see that maybe there is a problem where there maybe need of in terms of TS (Technical Support), like even at the point of receiving copies because eLMIS will refuse if the report is not tallying with the there. So it will tell you something maybe about the one who has reported just at the point of entering the report. So even that one again you also use it as a basis to offer some TS in terms of how to come up with the reports”. – District pharmacist 32 | Page volume of commodities transferred at endline and 6 above shows the number commodity transfer transactions increasing over time at health facilities. Figure 13: Quantity of Commodities Transferred at Endline - All Programs by Province Research Question 4 To what extent is the eLMIS FE usable and acceptable among different users? eLMIS Usability and Acceptance: User-driven enhancements of eLMIS FE continue to provide a unique user experience. Key to user experience is continuous improvement of user satisfaction is better accessibility, usability and efficiency in interaction with the system. The data below verifies that the system is usable and accepted by staff using the system. As noted on the eLMIS Implementation Roadmap (Figure 5), the eLMIS was officially adopted as the national electronic logistics system in January 2019. Appendix 4 shows the letter announcing that this system has been accepted and adopted by the MOH. Figures 14 and 14a illustrates the level of user satisfaction based on qualitative survey responses and physical verification of system functionality assessed on the day of visit. The number of facilities where the evaluation team had no access to the eLMIS server on the day of visit is noted as unverified in the figures below. The number of facilities with functional software at endline increased relative to midline by 3.6 percent. 6819 8755 8626 5391 6223 2239 3264 3809 8220 5292 6046 6096 8462 4645 6570 1993 3080 2849 7111 5218 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 Central Copperbelt Eastern Luapula Lusaka Muchinga North Western Northern Southern Western Number of Commodities Transfer In Transfer Out Figure 14: eLMIS FE Software Functionality at Midline (n=186) and Figure 14a: Endline (n=188) As the findings in Figure 15 indicate, user satisfaction is consistently high. Additionally, as noted in figure 14, user access sessions to eLMIS CE between midline and endline increased by 58 percent. Figure 15: eLMIS User’s Satisfaction at Midline March 2016 to February 2017 and Endline (n=188); March 2018 to February 2019 90.3% 6.5% Fig. 14. Midline 3.2% Functional Non Functional Unverified 93.9% 5.1% Fig. 14a: Endline 1.0% Functional Non Functional Unverified 94% 95% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Mid-line End-line 34 | Page Research Question 5 To what extent has the eLMIS influenced supply chain performance through improved commodity availability? To show the system’s influence on the performance of the supply chain, the key indicators used during the midline, such as commodity availability, commodity stock status (stocked according to plan), rate of expiries, and emergency order rates were also used for the endline assessment with users of both the Central and Facility Editions. Commodity Availability: The strategic objective of the eLMIS system implementation, in collaborative effort with other key supply chain players, is to improve commodity availability in health facilities in Zambia to ensure commodities are available for clients when and where they are needed. At midline, with the exception of HIV test kits that experienced a central-level stock-out during the period reviewed, three commodity areas (ARVs, laboratory, and Essential Medicines or EM) recorded an increase in the percentage of commodities available following the implementation of eLMIS. Figure 16 shows commodity availability findings of pre￾eLMIS, midline, and endline evaluations. The average increase in commodity availability from pre-eLMIS to endline was 17.3 percent across all commodity areas. Figure 16. Product Availability by Program – Pre-eLMIS, Midline and Endline Periods: Average Percentage of Facilities with Stock (National) 31% 69% 88% 70% 78% 89% 84% 79% 78% 91% 92% 65% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Lab ARVs HIV Test Kits EM Average Percentage of Facilities Pre-eLMIS (March 12 - Feb 13) Mid-line (March 16 - Feb 17) End-line (March 18 - Feb 19) To determine specific impact of eLMIS FE on selected facilities, analysis of stock availability at pre-eLMIS and at Endline was performed for all programs. As indicated in Figure 16a, these facilities posted much higher rates of commodity availability at Endline that the combined facilities. Furthermore, the EM program, which showed a decline between pre-eLMIS and Endline in figure 16, had a remarkable improvement in eLMIS FE facilities 67 percent to 75 percent. All programs had much higher stock availability (between 75 percent and 95 percent) at endline than shown in the combined facilities. Facilities with eLMIS FE have a clear capacity to support a robust supply chain in resource-limited settings. Figure 16a. Product Availability by Program – Pre-eLMIS and Endline Periods: Average Percentage of eLMIS FE Facilities with Stock Further analysis was performed to compare commodity availability at facilities with eLMIS FE with non-eLMIS FE. The site selection for this analysis was limited to health centers and health posts to avoid operational-level bias, since more than 90 percent of hospitals have eLMIS FE. The period of data analysis was the same as all endline data analysis, March 2018 – February 2019. In all cases, the facilities with eLMIS FE demonstrated higher average availability than those with paper-based LMIS, for the same time period and facility types. Given the relatively large number of line stock items reported for Essential Medicines, the weighted average availability is influenced 69% 86% 37% 67% 95% 95% 82% 75% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ARV HIV LAB EM % of stock Availability Pre-eLMIS eLMIS 36 | Page by this factor. These differences are likely driven by improved inventory management between facilities using eLMIS FE and timely data visibility at DHOs and MSL. Figure 17 shows the data comparison; sites with FE had 15.5 percent fewer stock-outs than sites using paper-based system. Figure 17: Stock Availability Comparison Sites with FE vs. Non-FE The data indicate that the introduction of eLMIS CE reduced stock-outs at health facilities by approximately 17 percent from pre-eLMIS to endline. This is impressive given the supply chain changes and challenges noted earlier. Even in a time of distribution challenges, with data visibility and access, supervisors and facilities have used the data in the system to ensure commodities remain available to clients through transfers between facilities. Additionally, it is noted that introduction of FE at facilities further improves commodity availability by up to 15 percent at health facilities. Commodity Stock Status: The design of the logistics system ensures stock availability within the desired maximum, the minimum and emergency order point (EOP). The stock level is measured in months of stock (MOS), the duration that usable inventory can last based on a moving-average consumption. This provides a rational and standardized interpretation of stock levels to health workers and supply chain operators. Apart from the Essential Medicines program at health centers whose maximum stock level was set at 4.0 MOS, all other programs at all health facilities have the maximum at 3.0 MOS. 95% 96% 84% 77% 81% 87% 93% 65% 64% 65% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% ARVs HIV Kits Lab EM All Groups (Weighted) eLMIS Paper (Non FE) In the Zambian logistics systems design, two alert levels require immediate intervention: the EOP (0.5 MOS), and more than 1 month above the maximum stock level. Though the ideal stock level is between minimum and maximum, for the purpose of this evaluation, stocked according to plan (SATP) is redefined as the stock status at which a supply chain intervention is not required. Stock availability is defined as when a product is not stocked out. Figures 18 and 18a show average SATP by product area at pre-eLMIS, midline, and endline. The ARVs and laboratory commodities programs showed consistent improvement; HIV tests and EM commodities had slight declines at endline. Figure 19 compares SATP and stock availability of sites with and without the eLMIS FE. Figure 18: Health Facility SATP Pre-eLMIS, Midline, and Endline 17% 46% 50% 41% 35% 48% 51% 20% 39% 54% 46% 40% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Lab ARVs HIV Test Kits EM Average Percentage of Facilities Pre-eLMIS (March 12 - Feb 13) Mid-line (March 16 - Feb 17) End-line (March 18 - Feb 19) 38 | Page Figure 18a: Average Stock Availability and SATP for All Programs at Pre-eLMIS, Midline, and Endline Figure 19: Stock Availability and SATP for eLMIS FE and Non-eLMIS FE Sites at Endline Further analysis on stock availability at endline shows sites with FE having a much higher rate of stock availability. 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 90.00% Pre-eLMIS Mid-line End-Line 64.50% 82.25% 81.75% 38.50% 38.50% 44.75% Stock Availability Stocked According to Plan 58% 82% 86% 95% 91% 95% 62% 73% 29% 40% 54% 56% 43% 53% 39% 43% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Non FE FE Non FE FE Non FE FE Non FE FE Lab ARVs HIV Test Kits EM Stock Availability Stocked According to Plan These results continue to show that with supply chain automation to the last mile, high resilience of the supply chain is achievable. Data driven supervision keeps the dynamic challenges in control as was implied by interviewed respondents during key informant interviews. Commodity Expiries: Commodity expiries are another important indicator for measuring both supply chain performance and the impact of systems on commodity security. The team measured the rates of expiries for HIV and HIV programs. Due to the influence of expiries on products available for use, they applied the percentage ratio of expiries to consumption in the assessment. Figures 20 and 21 show the percentage ratio of expiries to consumption for HIV test kits and ARVs at midline and endline, respectively. There is a consistent reduction in the ratio of expiries over both periods of evaluation. Findings indicate a high reduction in the percentage ratio of expired commodities to those consumed between midline and endline, this can be attributed to the positive influence of the eLMIS on reduction of product wastage. Figure 20: Percentage Ratio of Expiries to Consumption for HIV test kits and ARVs at Midline 0% 2% 4% 6% 8% 10% 12% 14% 16% % RATIO OF EXPIRIES TO CONSUMPTION HIV ARVs 40 | Page Figure 21: Percentage Ratio of Expiries to Consumption for HIV Test Kits and ARVs at Endline Emergency Orders: The rate of emergency orders is among the measures of supply chain efficiency that supervisors rely on as an indicator for monitoring operational challenges. Given the distribution challenges highlighted earlier, a slight increase in the rate of emergency orders (0.34% ) was recorded between midline and endline. Figure 22 shows the rate of emergency orders for the two evaluation periods. The rate of emergency orders remained below 2.5 percent at both periods of evaluation. However, the data reveal a seasonal pattern that seems to mimic the resupply schedules. Figure 22. Trend of Emergency Requisitions at Midline and Endline 0% 2% 4% 6% 8% 10% 12% 14% 16% % RATIO OF EXPIRIES TO CONSUMPTION HIV ARV 0.00% 0.50% 1.00% 1.50% 2.00% 2.50% 1 2 3 4 5 6 7 8 9 10 11 12 % EMERGENCY ORDERS EVAL PERIOD (%)Mid-line (Mar 16 - Feb 17) (%) End-line (Mar 18 - Feb 19) Research Question 6 Based on the eLMIS’s current scale of implementation, what is the return on investment? “Return on investment,” in commercial settings, reflects an assessment of the financial gains relative to the required upfront investment and related upkeep. These assessments provide a sense of whether the endeavor will be worthwhile, especially when compared to other opportunities. In a public health setting, these returns will not come from revenue, but rather from reductions in operating costs, required resources, or opportunities missed. It is assumed in these cases that reductions in direct expenses or resource requirements enable these saved costs to be deployed to other valuable applications. For example, time saved from switching from paper￾based to electronic reporting can be dedicated to other health service delivery needs within the clinic. The eLMIS, from its initial development through deployment to DHOs and health facilities, has incurred investment costs that can be justified through the accrued benefits to these institutions receive. This evaluation of the return on investment for the eLMIS includes an assessment of the total cost of ownership, the cost of operating the co-existing paper LMIS, reduced costs of facility labor for LMIS, and stock-outs averted. Total Cost of Ownership: For the eLMIS, the total cost of ownership reflects the full amount of funding required to implement, support, and maintain it in a health facility. Building on estimates developed for the midline evaluation, this calculation reflects line items and key assumptions, as identified by project managers, with cost rates and total expenditures validated by project accounting records. This assessment does not include any expenses or resources related to actions for improving supply chain performance that fall outside the scope of eLMIS. This assessment also does not include costs of solar installations by the donor, government, and other implementing partners, except for project-implemented electricity back-up systems. For confidentiality purposes, actual costs are not included in this report. Instead, percentage comparisons and trends are used to illustrate the results. The actual costs are shared with the USAID and the Ministry of Health Zambia. Line item cost categories for the eLMIS included in this analysis: • Software development (initial investment) • Initial and replacement hardware (which vary by facility type) • Facility installation • Initial training and ongoing facility support • Software support and dedicated project staff 42 | Page The costs cover both fixed overhead related to maintaining the eLMIS at any scale, and direct expenses incurred on a per-facility basis. Figure 23 shows the percentage by cost component over the first five years of the eLMIS, as CE was implemented and supported and FE was deployed to 580 health facilities. Note that the initial software development costs were allocated to Zambia from a shared Zambia-Tanzania development initiative, with estimated annual total costs of ownership based on volume of installations. Although the aggregation of costs involved converting between Zambian Kwacha and United States Dollars (USD$), a conversion factor was used to anonymize the actual cost. The resultant values are therefore not associated with any known currency. Table 6 shows the number of facilities implemented annually and associated deployment and support costs. Figure 23: Distribution of Total Cost of Ownership by Cost Component 2014 – 2018 Software Development, 13% Initial and Replacement Hardware, 31% Facility Installation, 9% Initial Training and Ongoing Facility Support, 9% Software Support and Dedicated Project Staff, 38% Table 6. Total Number of Facilities on eLMIS FE with Associated Costs of Implementation and Support (Currency Units) 2014 2015 2016 2017 2018 TOTAL Cumulative Total Implementations 45 132 244 389 580 580 Number of New Implementations 45 87 112 145 191 Costs Replacement Hardware (Direct Variable) 19 257 1,318 1,586 2,073 5,253 Site Support (Direct Variable) 566 1,056 1,723 2,587 2,398 8,330 System Support and Dedicated Staffing 17,970 19,976 19,976 21,982 23,988 103,892 Total 18,555 21,289 23,017 26,155 28,458 117,474 Total Operating Costs - Per Facility 412 161 94 67 49 Per Facility Operating Costs (Direct Variable Only) 13 10 12 11 8 Though Figure 23 above captures total cost of ownership for the first five years, a large portion of these costs are one-time, driven by initial software development, hardware installation and training expenses. The increase in total annual expenses largely reflects a growing number of new and cumulative installations per year, which require ongoing support and replacement equipment. System operating costs include system support costs and a portion of initial training and hardware acquisition costs. Analysis of the proportion of annual costs per facility to the total cost of ownership over five years shows that as more facilities deployed eLMIS FE, the percentage of the total cost per facility greatly decreased. It is our observation that economies of scale, improvements in the design of eLMIS, improvement in the deployment strategy, adoption of remote support and automated upgrades, and overall system maturity are some of the factors that have influenced the reduction in unit cost of ownership, as indicated in Figure 24. This reduction is therefore expected to continue with subsequent implementation of eLMIS FE. We anticipate that future implementation and support costs per facility will be much lower. 44 | Page Figure 24: Proportion (%) of Annual Cost Per Facility to Total Cost of Ownership over 5 Years and Number of eLMIS FE Facilities To understand the total cost of the investment for deployment of eLMIS FE to health facilities, costs directly attributed to FE installations were identified. Following the initial one-time investment of computer hardware, network infrastructure installation, and recurrent operating costs of eLMIS FE include: • Replacement hardware – computer hardware is replaced gradually; as it wears out, is damaged or lost. Given historical experience, there is not anticipation that there will be a major overhaul of the equipment in the future. • Site support. • System support and dedicated staffing. Return on Investment: To answer the question on return on investment, the evaluation team analyzed costs of FE deployment against direct facility cost savings accrued from eLMIS inventory control and data transmission efficiencies. This includes indirect facility savings from stock-outs averted. 1. Paper-Based Logistics System Savings 1 2 3 4 5 # Facilities 45 132 244 389 580 % Cost per Facility 0.32 0.13 0.11 0.05 0.04 0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0 100 200 300 400 500 600 700 % Cost Per Facility Cummulative Number of facilities Period (Year) P I L O T Over time, the eLMIS FE was systematically implemented to replace the paper-based LMIS. The costs of original development and implementation of the paper LMIS are not factored into this assessment, but the scale-up of the eLMIS FE deployment enabled reduced expenditure on printing of paper forms and extensive LMIS training. These two activities are managed by the GHSC-PSM (Global Health Supply Chain Program - Procurement and Supply Management) project, which provided expenses for these two line items for calendar year 2018 for this endline assessment. Each expansion of the eLMIS can therefore be assumed to approximately reduce existing expenses by these cost values. As the eLMIS implementation is scaled to more facilities, paper-based LMIS operating costs can be assumed to gradually decrease. Although the detailed costs are not shown here, these costs were used to determine a return on investment for deployment of eLMIS FE. Responses to a research question during the eLMIS midline indicated that adoption of eLMIS reduced workload and enabled facility and DHO staff to spend less total time completing monthly reports. An in-person survey captured estimated time requirements and staff salaries, and analysis demonstrated that in comparison to paper-based LMIS, eLMIS saved approximately $400 per facility per year, with variations depending on facility size—secondary facilities saving $492 per year and primary facilities saving $265 per year. 2. Stock Availability Improvement As indicated in Table 7 below and Figure 17 above, during 2018 the eLMIS FE achieved higher overall availability (lower stock-outs) for all four product groups when compared to facilities still using the paper-based LMIS. This comparison was done for all products managed at health centers and health posts. To minimize bias, hospitals, which by 2018 were predominantly on eLMIS FE, were not included in this comparison. In total, this comparison includes approximately 1.85 million records, with about 24% from eLMIS FE facilities, and the other 76 percent from paper-based facility reporting. Overall, sites with FE experienced 15.5 percent fewer stock-outs compared to those using the paper-based inventory management. Table 7. Stock Availability (% of Reported Products Not Stocked Out) at Endline ARVs HIV Kits Lab EM All Programs (Weighted by number of LMIS report records) eLMIS 95.0% 95.9% 84.1% 77.2% 80.6% Paper 86.8% 93.3% 64.7% 64.0% 65.1% Although there was an initial cost for setup of a paper-based system, we assume it to be a derived benefit to the eLMIS, and did not include that cost in the analysis. 46 | Page Moreover, the eLMIS implementation is built on the success of the paper-based system. Though the scale-up of eLMIS FE has likely led to cost reductions over the initial five years of its adoption, these alone will not have offset the eLMIS’s investment and operating costs. From a purely financial sense, the eLMIS’s return on investment cannot be fully accurately determined using the cost components outlined above. However, the eLMIS was not introduced to provide financial benefits only. Other benefits could include, among others, patient treatment outcomes, reduction in patient waiting time, reduction in storage space for paper-based files, time saved by the inventory audit teams, time saved by the MOH monitoring and evaluation teams, and potential saving from cost of admission or treatment upgrade for patients who relapse due to missed treatment. Other demonstrated benefits were not monetized in this evaluation, and should be further investigated, including actual cost savings on reduction of expiries that can be accurately attributed to eLMIS. Research Question 7 What is the cost benefit of scaling to additional sites not yet covered by the eLMIS FE? As of February 2019, approximately 580 of an estimated 3,130 primary and secondary health facilities were equipped and trained to use eLMIS FE, leaving another 2,550 facilities as potential candidates (although some of these locations are not yet operational and would not currently work as candidates for eLMIS). Local partners now have the opportunity to continue implementation of eLMIS FE, with the benefit of cost and performance assessment data to inform their plans. This section attempts to identify the cost effectiveness of scaling implementation of eLMIS FE to additional sites, in relation to associated benefits. The costs included in this section include installation and direct support of the eLMIS system at a given facility, and do not include costs of related interventions that support product availability. Additionally, this section focuses on establishing eLMIS in new locations, and not expansion of eLMIS FE to additional programs within facilities that already use eLMIS. Finally, at this stage, the eLMIS has been installed in most secondary facilities already, meaning that the majority of remaining facilities are rural health centers (RHCs) and health posts (HPs). For the purpose of this section, it will be assumed that the additional scaling will occur at an RHC (in terms of costs and benefits). The pay-back period for the initial and ongoing investments in the implementation of eLMIS FE is not fully ascertained; but eLMIS installation clearly yields improvements in supply chain performance. The exact improvement might depend on which programs the facility operates, but on average, the facility could expect product availability to improve by 15.5 percent over five years. Cost-effectiveness Assessment: A cost-effectiveness analysis compares the costs of two supply chain initiatives: the one that involves eLMIS implementation, and the paper￾based/non-eLMIS approach. These two-supply chain strengthening approaches are analyzed against stock availability as a common measure of effectiveness and a key supply chain objective. We have attempted to highlight the effectiveness of PEPFAR focus programs, HIV test kits, and ARVs. Availability of these products guarantees HIV prevention and treatment. Globally, there are two principal measures of cost effectiveness, the average cost￾effectiveness ratio (ACER) and the incremental cost-effectiveness ratio (ICER). • ACER is derived by dividing eLMIS implementation and support cost by the total change in commodity availability (effectiveness of an intervention). ACER = (eLMIS total cost of implementation and support) ÷ (total effect on % availability). • ICER compares the differences in costs and health outcomes of two alternative interventions competing for the same resources. The ICER is calculated as: ICER = (cost of intervention A – cost of intervention B) ÷ (effectiveness of intervention A – effectiveness of intervention B). The cost effectiveness of implementing and supporting eLMIS FE in a health center will use the implementation and support costs as the investment, and stock availability as a measure of the effect. Overall, the average cost effectiveness of implementing eLMIS FE in a typical RHC is the total cost of 137.2 units of currency over five years, during which time the health programs could expect to see a 15.5 percent improvement in product availability, resulting in ACER of 8.85 units of currency per percentage point of availability. This is likely to be cheaper than the cost of the product itself (when considering the actual currency), indicating basic efficiency gains, including economies of scale, continuous digital improvement, and increased system maturity with scale-up of the eLMIS. 48 | Page DISCUSSION AND RECOMMENDATIONS Research Question 1 Timeliness, frequency, and accuracy of reporting Frequency of Reporting: The endline reporting rate was higher, with an average of 98 percent, whereas midline was at 93 percent and pre-eLMIS at 86 percent. This improvement is consistent with increase in eLMIS FE implementation coverage that also supports positive user opinions on ease of reporting and supervision. Timeliness of Reporting: An improvement in timeliness of reporting occurred between midline and endline for all facilities. Furthermore, facilities with eLMIS FE showed more improvement in timely reporting than non-eLMIS FE facilities, a finding attributable to the ease of reporting introduced by the deployment of eLMIS Facility Edition. Reporting Accuracy: Respondents in the facility qualitative questionnaires also indicated that eLMIS has brought about improvement in report accuracy. By virtue of its design, the eLMIS can be said to have improved influenced data accuracy. Multiple validations are integrated into the user interfaces to ensure the consistency and accuracy of data. For example, in the past, paper-based reports, it was common to find that the beginning balances in the current report would be different from the closing balance of the previous month. With eLMIS FE, data validation occurs with each transaction, and the report generated is based on data that conforms to inventory control rules. Reports entered directly on eLMIS CE from paper-based requisitions also benefit from validation rules applied to aggregate data and historical entries. The other advantage with eLMIS CE is the provision for three levels of reviews and approvals performed by supervisors on requisitions before they are qualified as orders. Research Question 2 Data accessibility, visibility, and quality Data Accessibility: As with the midline, the endline evaluation confirms that the web-enabled eLMIS CE provides a desired level of data accessibility. Although internet was a challenge in the past, continued improvement of mobile telecommunications coverage has facilitated access to supply chain data that can be used in decision-making. Comparing user sessions, the endline period shows a higher number of user sessions, with an average of 7,846 than the midline with 4,964. The 58 percent increase in the number of sessions can be associated with improved demand and use of eLMIS generated data for planning such as forecasting and quantification, and operational management like in product replenishment and redistribution, as seen with increase in the rate of product transfers between health facilities by province. Data Visibility: The eLMIS system enhances data visibility at health facility, district, provincial, and central levels. Respondents at the central and district levels indicated how eLMIS enables them to view health facility data and use it to make decisions on product redistribution and/or planning for supportive supervision visits. At the facility level, respondents using eLMIS FE said that they can verify transactions on the system and account for their commodities appropriately. Data Quality: It is now possible to monitor and enhance data quality at facilities that use eLMIS FE. The validation rules in eLMIS FE ensure adherence to standard inventory control procedures. The eLMIS maintains an audit trail of all transactions completed at the health facility for enhanced accountability. However, timeliness of data capture needs to be further improved. The number of stock control cards diminished by 15 percent between midline and endline. This is mostly attributed to the parallel data entry requirements caused by silo health information systems at the facility level. Although data are reported updated by the end of each week or reporting schedule, it is important to minimize duplication of data entry at health facilities. Research Question 3 Data use and data-driven decision-making Data Use: Use of eLMIS data at health facility and district levels is evident in the progressive increase in user sessions by more than 58 percent between midline and endline. The increase in use of data to monitor and redistribute inventory between health facilities is also apparent. In a number of interviews, staff at the district level said that they used the stock imbalance reports from the system for commodity redistribution to avert stock-outs and product expiries. Ease of data access also enables program managers and policy-makers to take timely corrective action and ensure accountability, especially at facilities. An enhanced upgrade of eLMIS FE (Version 4.04) is designed to enable efficient tracking and tracing of commodities to the last mile. This entails use of batch/lot numbers to 50 | Page identify products through the supply chain including during dispensation of medicines to patients. The other initiative is the GS1 (global standard) serialization that is being adopted in Zambia. This will further strengthen product tracking. Continuous improvement will remain a driving factor for strengthening a data use culture. The Zambian public health supply chain and the eLMIS are designed to use easy to interpret stock measurement techniques such as “Months of Stock” that are based on the rate of consumption. Users are therefore able to use the stock status reports to determine how much stock can be transferred out. Use of machine learning and artificial intelligence for forecasting inventory needs at health facilities will also further strengthen supply chain management as the system continues to mature. Research Question 4 To what extent is the eLMIS usable and acceptable among different users? System Usability: Findings from the qualitative interviews showed that 100 percent of the users interviewed had accessed eLMIS at least once in 30 days prior to the interview. Further, the users were able to outline specific activities or modules they had accessed in their recent interactions with the eLMIS. Along with the continued increase in the rate of user satisfaction levels (>95%), increased user sessions and ability to use data to inform operational and strategic decisions provide evidence of the usability of the eLMIS. System Acceptability: As more users become dependent on eLMIS as a source of data and a tool that improves routine operations, we observed a consistent increase in the rate of use at all levels of the supply chain. Overall, users of eLMIS FE and CE have demonstrated their acceptance of the system, as indicated by increased use, consistent improvement in reporting rates, and a high user rate of satisfaction. This testifies to continuous improvement in the provision of effective computer infrastructure and user support, including user-focused trainings, and spotlight sessions on interpreting supply chain data. Research Question 5 Commodity availability, stock status, and expiries Commodity Availability: With the exception of Essential Medicines, all program areas recorded an increase in commodity availability. Between midline and endline, HIV test kits had the highest increase in availability of 8 percent, whereas ARVs and laboratory commodities increased by 2 percent and 1 percent, respectively. The late rollout of the paper-based Essential Medicines Improvement Logistics program may have been the main reason for limited evidence of early impact by eLMIS. Further investigation is required to ascertain the compounding impact of changes in resupply schedule from one to two months, and the decrease in the rate of order processing. Commodity Stock Status: The stock status of a program area is an indication of the stability of inventory management for the logistics system at health facilities. Ideally, facilities will be stocked according to plan to minimize risk of stock-outs and expiries. As shown in Figure 18a, there was an improvement of 6.3 percent of items stocked according to plan between pre-eLMIS and endline. Additionally, as shown in Figure 19, facilities using eLMIS FE showed a better improvement of 7 percent between pre￾eLMIS and endline. This attests to the role of an eLMIS at facility level in improving inventory management and mitigating service interruption arising from stock-outs. Expiries: We used the ratio of expiries to consumption to determine the percentage of commodities going to waste through expiration. As shown in Figures 26 and 27, ARVs and HIV test kits performed better between midline and endline, maintaining the rate of expiries below 2 percent. Research Questions 6 & 7 Costing: return on investment and cost benefit Overall, investment in the eLMIS accrues key supply chain and health service benefits, some of which are not easily monetized. Key investment costs include computer and network infrastructure hardware, dedicated project staff for development, deployment, training, and support. The impact of eLMIS on supply chain efficiency results in cost savings in key areas such as improvements in stock availability, reduction in wastage or expiries, transition from paper-based costs like printing of forms and courier fees, and facility staff time saved. The reduction in expiries between midline and endline of 79 percent for ARVs and 75 percent for HIV test kits was equivalent to $2,003,091 and $3,83,521, respectively. This evaluation did not 52 | Page examine the cost benefit of the prevention of expiry itself, which would have required facilities and/or clients to use their budgets to purchase the drugs or kits from private pharmacies at higher prices. Given a fixed, pre-existing overhead (for maintaining the overall eLMIS), expansion to an individual primary health facility represents a good opportunity. After offsetting operating and initial investment costs over five years, the remaining investment cost can yield availability improvement at 8.85 units of currency per percentage point over five years. This cost is expected to decrease further with continued scale-up of eLMIS FE implementation. CONCLUSION The findings of this evaluation show that despite challenges experienced in the supply chain at the periods of evaluation, eLMIS has contributed to a general improvement in efficiency, cost, and commodity availability. Automation of inventory control at health facilities using eLMIS FE resulted in better performance of the supply chain than in non-eLMIS FE facilities. Based on key supply chain indicators (commodity availability, stock status, and reporting rates), a greater proportion of health facilities that have the eLMIS FE were better stocked, more often reported on time, and achieved a higher frequency of reporting at midline and endline. The eLMIS Facility Edition sites recorded average reporting rates of 99.5 percent, an indication of the capability of eLMIS FE to influence improvement in the ability of health facility staff to generate and submit their reports and requisitions. Although other logistics operational constraints may have influenced improvements in commodity availability, the growing resilience of the supply chain was evident at midline and endline, and even more apparent at eLMIS FE facilities. The common confounding factors may include stock replenishment lead times, distribution intervals, and item fill rates. Between midline and endline, the supply chain had to adjust to change in the distribution cycles from monthly to bi-monthly. This would have been more complex without a reliable electronic system. However, a separate evaluation may be necessary to understand the full impact of such challenges and events on the overall supply chain performance. With continued consistency in efficiency and good supply chain practices at health facilities with eLMIS Facility Edition, a higher percentage of commodity availability and fewer stock outs can be achieved. Furthermore, the data visibility and accountability introduced by eLMIS Central Edition will improve the quality and accuracy of reports while minimizing wastage through expiries, since facility staff will have access to data for decision-making. 54 | Page REFERENCES Strengthening High Impact Interventions for an AIDS-free Generation (AIDSFree) Project. 2018. AIDSFree Zambia eLMIS Evaluation Report. Arlington, VA: AIDSFree. UNAIDS. 2014. 90-90-90 An Ambitious Treatment Target to Help End the AIDS Epidemic . Geneva : UNAIDS. World Health Organization. 2010. Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies (https://www.who.int/healthinfo/systems/WHO_MBHSS_2010_full_web.pdf). APPENDIX 1. PHASE I EVALUATION FACILITY LIST Province District Facility Code Facility Name Facility Type Central Kabwe 102015 Mahatma Gandhi Urban Health Center HC Central Kabwe 102001 Kabwe Mine Hospital HC Central Kabwe 102016 Makululu Urban Health Center HC Central Kapiri-Mposhi 103014 Mpunde Mission Rural Health Center HC Central Chibombo 101029 Mwachisompola Demo Rural Health Center HC Central Mumbwa 105001 Mumbwa District Hospital LVL1 Central Itezhi-tezhi 803001 Itezhi-tezhi District Hospital LVL1 Central Mkushi 104001 Mkushi District Hospital LVL1 Central Chibombo 101001 Liteta District Hospital LVL1 Central Mumbwa 105001 Nangoma Mission Hospital LVL1 Central Kabwe 102002 Kabwe General Hospital LVL2 Copperbelt Kitwe 204016 Chimwemwe Urban Health Center HC Copperbelt Chingola 202012 Chiwempala Urban Health Center HC Copperbelt Kitwe 204021 Ipusukilo Urban Health Center HC Copperbelt Kalulushi 203014 Kalulushi Government Urban Health Center HC Copperbelt Kitwe 204036 Ndeke Urban Health Center HC Copperbelt Chingola 202013 Kabundi East Urban Health Center HC Copperbelt Kitwe 204027 Luangwa Urban Health Center HC Copperbelt Chililabombwe 201010 Kakoso Urban Health Center HC Copperbelt Ndola 210045 Twapia Urban Health Center HC Copperbelt Ndola 2100Q9 Catholic Diocese of Ndola HC Copperbelt Ndola 210031 Kalewa Barracks Urban Health Center HC Copperbelt Luanshya 205003 Thomson District Hospital LVL1 Copperbelt Mufulira 209002 Malcom Watson Hospital LVL1 Copperbelt Mufulira 209001 Kamuchanga District Hospital LVL1 Copperbelt Mpongwe 208002 Mpongwe Mission Hospital LVL1 Copperbelt Mpongwe 208003 St. Theresa Mission Hospital LVL1 Copperbelt Chililabombwe 201001 Konkola Mine Hospital LVL1 Copperbelt Kitwe 204003 Wusakile Mine Hospital LVL2 56 | Page Copperbelt Luanshya 205002 Roan General Hospital LVL2 Copperbelt Chingola 202001 Nchanga North General Hospital LVL2 Copperbelt Mufulira 209003 Ronald Ross General Hospital LVL2 Copperbelt Chingola 202002 Nchanga South General Hospital LVL2 Copperbelt Kitwe 204001 Kitwe Central Hospital LVL3 Eastern Petauke 308001 Petauke District Hospital LVL1 Eastern Chipata 303002 Mwami Mission Hospital LVL1 Eastern Lundazi 305032 Lundazi District Hospital LVL1 Eastern Chadiza 301001 Chadiza District Hospital LVL1 Eastern Nyimba 307001 Nyimba District Hospital LVL1 Eastern Mambwe 306001 Kamoto Mission Hospital LVL1 Eastern Chipata 303097 Kapata Urban Health Center HC Eastern Chipata 303001 Chipata General Hospital LVL2 Luapula Samfya 407028 Samfya Stage II Rural Health Center HC Luapula Nchelenge 406001 St. Pauls Mission Hospital LVL1 Luapula Kawambwa 402001 Kawambwa District Hospital LVL1 Luapula Mwense 405021 Mambilima Mission Hospital LVL1 Luapula Mansa 403001 Mansa General Hospital LVL2 Lusaka Lusaka 504021 Kanyama Urban Health Center HC Lusaka Lusaka 504017 George Urban Health Center HC Lusaka Lusaka 504012 Chawama Urban Health Center HC Lusaka Lusaka 504014 Chilenje Urban Health Center HC Lusaka Lusaka 504013 Chelstone Urban Health Center HC Lusaka Lusaka 504059 Chreso Ministries HC Lusaka Lusaka 504028 Mtendere Urban Health Center HC Lusaka Lusaka 504026 Matero Main Urban Health Center HC Lusaka Chongwe 501013 Chongwe Rural Health Center HC Lusaka Lusaka 504058 Circle of Hope HC Lusaka Lusaka 5040HG Kara Clinic HC Lusaka Lusaka 504018 Kabwata Urban Health Center HC Lusaka Lusaka 504029 Ng'ombe Urban Health Center HC Lusaka Lusaka 504034 Chazanga Urban Health Center HC Lusaka Lusaka 504010 Bauleni Urban Health Center HC Lusaka Lusaka 504057 UNZA Health Center HC Lusaka Luangwa 503001 Katondwe Mission Hospital LVL1 Lusaka Rufunsa 501001 St. Luke Mission Hospital LVL1 Lusaka Lusaka 5040T9 Levy Mwanawasa Hospital LVL2 Lusaka Chirundu 811002 Mtendere Mission Hospital LVL2 Lusaka Lusaka 504004 Maina Soko Military Hospital LVL2 Lusaka Lusaka 504002 University Teaching Hospital LVL3 Muchinga Mpika 608002 Mpika District Hospital LVL1 Muchinga Isoka 603001 Isoka District Hospital LVL1 Muchinga Chama 302001 Chama District Hospital LVL1 Northern Kasama 605014 Kasama Urban Health Center HC Northern Mpulungu 611016 Mpulungu Urban Health Center HC Northern Luwingu 606001 Luwingu District Hospital LVL1 Northern Kasama 605001 Kasama General Hospital LVL2 Northern Mbala 607001 Mbala General Hospital LVL2 NorthWestern Solwezi 706038 Solwezi Urban Health Center HC NorthWestern Kasempa 703001 Mukinge Mission Hospital LVL1 NorthWestern Solwezi 706001 Solwezi General Hospital LVL2 Southern Livingstone 806010 Maramba Urban Health Center HC Southern Livingstone 806008 Mahatma Gandhi Urban Health Center HC Southern Choma 801028 Shampande Urban Health Centre HC Southern Namwala 809011 Chitongo Rural Health Center HC Southern Kalomo 804002 Kalomo District Hospital LVL1 Southern Namwala 809001 Namwala District Hospital LVL1 Southern Monze 808031 Chikuni Mission Hospital LVL1 Southern Siavonga 811001 Siavonga District Hospital LVL1 Southern Sinazongwe 812001 Maamba District Hospital LVL1 Southern Choma 801001 Choma General Hospital LVL2 Southern Monze 808001 Monze Mission Hospital LVL2 Southern Livingstone 806001 Livingstone General Hospital LVL2 Southern Choma 801002 Macha Mission Hospital LVL2 Western Sesheke 906002 Yeta District Hospital LVL1 Western Kaoma 902001 Kaoma District Hospital LVL1 Western Mwandi 906001 Mwandi Mission Hospital LVL1 Western Kalabo 901001 Kalabo District Hospital LVL1 Western Lukulu 903001 Lukulu District Hosptial LVL1 Western Kalabo 901002 Yuka Mission Hospital LVL1 Western Senanga 905001 Senanga District Hospital LVL1 Note: Facility type HC refers to health center; LVL1, 2, or 3 refers to hospital level 1, 2, or 3. 58 | Page APPENDIX 2. KEY EVALUATION INDICATORS Research Question Category Indicator Indicator Definition Target Respondent Data Source To what extent has the eLMIS improved reporting frequency, timeliness, and accuracy? Reporting Frequency Average annual reporting rates by commodity area Number of reports submitted in a 12-month period/ 12 months N/A eLMIS; SCMgr Reporting Frequency % of facilities submitting reports Number of facilities submitting reports per month /Total facilities expected to report per month N/A eLMIS; SCMgr Reporting Frequency % of facilities using FE that submitted reports Number of facilities submitting reports per month /Total facilities expected to report per month N/A eLMIS Reporting Timeliness User perception on reporting time User perceptions of ease of reporting with FE Pharmacists and lab techs (HF) In-depth interviews Reporting Timeliness % of facilities submitting timely reports (CE and FE) Number of facilities submitting reports before cutoff date in a 12-month period/Total facilities reporting N/A eLMIS ONLY Reporting Accuracy User perception on reporting accuracy User perceptions of reporting accuracy with FE and CE Pharmacists and lab techs (HF and In-depth interviews Reporting Accuracy Average % of facilities per month whose beginning balance is equivalent to end balance of previous (Number of facilities whose beginning balance is equivalent to end balance of previous month/Total facilities) Average over 1 year N/A eLMIS; SCMgr To what extent has the eLMIS improved data accessibility, Data Accessibility User perception on data accessibility User perceptions of accessibility of data (Do they have access to different types of data, e.g., inventory, issues, management, etc.?) Pharmacists and lab techs (HF and district levels), IT unit (central level) In-depth interviews Research Question Category Indicator Indicator Definition Target Respondent Data Source visibility, and quality? Data Visibility User perception on data visibility User perception of visibility of data Pharmacists and lab techs (HF and district levels), IT Unit (central level) In-depth interviews Data Quality User perception on data quality User perception on data quality Pharmacists and lab techs (HF and district levels), IT Unit In-depth interviews Data Quality % of facilities with SCC balance matching physical count The number of facilities that have a SCC balance of +/- 5% / Total facilities N/A Facility survey Has the availability of eLMIS data led to increased data use and/or data￾driven decision￾Data usage User perception on data usage Do they have access to the data that they need for decision-making?) Pharmacists and lab techs (HF and district levels), IT Unit In-depth interviews Data usage Number of user sessions Number of user sessions per month from 2015–17 N/A eLMIS To what extent is the eLMIS FE usable and acceptable among different users? eLMIS satisfaction % Satisfaction with eLMIS FE Number of interviewees satisfied with the software installation/Total number of people interviewed Pharmacists and lab techs (HF) In-depth interviews Functional software % of facilities with functional eLMIS software Number of facilities with functional software/Number of facilities with software installed Pharmacists and lab techs (HF) Facility survey Functional hardware % of facilities with working computer and reliable internet Number of facilities with functional hardware/Number of facilities with hardware installed Pharmacists and lab techs (HF) Facility survey 60 | Page Research Question Category Indicator Indicator Definition Target Respondent Data Source Competence Number of facilities with staff trained in eLMIS who are competent in the software Number of facilities with staff trained in eLMIS who are competent in the software Pharmacists and lab techs (HF) Facility survey Usability and acceptance User perception on system usability and acceptance Do they find the system usable and acceptable? Pharmacists and lab techs (HF and district levels), IT unit (central level) In-depth interviews Last R&R submission mode Mode of submission of last R&R No. of R&Rs submitted by different modes of submission/No. of facilities submitting R&Rs Pharmacists and lab techs (HF) Facility survey; eLMIS FE To what extent has the eLMIS contributed to improved overall supply chain perfor￾mance? Product availability % of facilities stocked out of one or more of the tracer commodities within the 12- month reporting period No. of times facilities reported a stockout within the i d f 12 Percentage of facilities stocked out of one or more of the tracer commodities within the 12-month reporting period Number of times facilities reported a stock-out within the period of 12 months None eLMIS; SCMgr Stock status % of facilities SATP for one or more of the tracer commodities within the 12-month reporting period Percentage of facilities stocked according to plan for one or more of the tracer commodities within the 12- month reporting period None eLMIS; SCMgr Order fill rates % of items ordered received with correct products and quantities Percentage of items ordered received with correct products and quantities None eLMIS; SCMgr Research Question Category Indicator Indicator Definition Target Respondent Data Source Expiry % of facilities with expiries of tracer commodities in last 6 months Percentage of facilities with expiries of tracer commodities in last six months None Facility survey Lead time Average lead time from submission of R&R form to delivery of commodities to HFs Average time from submission of report to delivery of commodities to health facility None Facility survey 62 | Page APPENDIX 3. QUESTIONNAIRES Endline 18th February – 30th April, 2019 Zambia eLMIS Evaluation: DISTRICT HEALTH OFFICE Interview Guide (QUALITATIVE TOOL) Instructions to facilitators: The following questions are a guide. An in-depth interview should feel like a conversation (where the respondent does most of the talking). It is best to begin with easy, open ended questions so the respondent feels comfortable and it allows them to convey in their own words their experience. Focus on the respondent’s experience and weave the topics and subtopics into the conversation (rather than worrying about asking each question as written). Try not to ask them to generalize or summarize their opinions on the eLMIS until the very end. Try not to ask Yes/No questions or leading questions. Ask respondents to illustrate their opinions with examples or use their examples to draw out their feelings and perceptions. You should probe and ask follow up questions only where appropriate. Name of Facilitator: ________________ Name of Note taker: _____________ Date: _____________ Province: ________________________ District: ________________________ Facility Name: _____________________________________________________ Facility Code: ____________________ Respondent’s sex (Please circle one): Male Female Respondent’s job title: ________________________________________ Interview Start Time: End Time: Instructions: Please introduce yourself to the respondent and thank him or her for their time. Explain the objectives of the mid-term data collection: the purpose of this evaluation is to assess the extent to which the eLMIS has improved supply chain processes compared to the previous paper-based system. The evaluation will focus on supply chain performance, data quality and use, and acceptability of the system. We would like to find out a little more about your use of the eLMIS. Let’s begin by talking a little bit about your job and your interaction with the eLMIS 1. How long have you been working in your position? 2. What are some of your activities as a……….. (Use their Job title)? {Probe for descriptions or examples.} 3. What are some of the challenges that you face as part of your job? {Probe for examples.} ------------------------------------------------ eLMIS Questions -------------------------- ----------------------- Let’s talk about the eLMIS and how you use it in your job. 4. How do you use the eLMIS as part of your job? {Probe for: a. How often b. How long they have been using it } 5. Can you describe your most recent experience of using the eLMIS? {Probe for: a. last time used b. what they used it for c. What they like or don’t like about using eLMIS d. What modules they find useful and which one they feel are not useful e. advantages and disadvantages } ------------------------------------Information questions-------------------------- --------------------- Let’s talk about the information that’s in the eLMIS. 6. Has using logistics data changed since the introduction of the eLMIS in 64 | Page your health facility? {Probe for: How it has changed} 7. Has the quality of data changed since the introduction of the eLMIS? {Probe for: How Data quality before (get example) Issues/challenges with data quality before (get example)} 8. Before eLMIS, was the information that you needed for reporting or decision-making complete or was some of it missing? 9. (follow up to question 8) With the eLMIS, has this remained the same or has it changed? {Probe for: How it has changed} 10.Before the eLMIS, was the information that you needed for past months available? {Probe for: Accessibility of past reports} 11.(follow up question to 10) With the eLMIS, has this remained the same or has it changed? {Probe for: How has it changed if it has } 12.After data is entered in the eLMIS, what do you do with the information? {Probe for: Data availability What data is missing in the eLMIS How the missing data be useful } 13.Do you review data in the eLMIS? {Probe for: How What they check for when reviewing What actions they take after reviewing the data } 14.Do you use data visualization features of the eLMIS to analyze and summarize data from the system if yes how do you use it? {Probe for examples} 15.Do you use eLMIS information to produce any other reports? {Probe for: What reports (e.g. TB reports) What the reports are for } 16.Are there any other decisions you make using data from eLMIS and can you give an example? -------------------------------------Supervision questions------------------------- ---------------------- 17.Supervision: a. As a district supervisor, what does your supervisory role involve? b. How has your supervision changed since the introduction of eLMIS and can you give specific examples? c. Do you use data from the eLMIS to help with supervision if so what data do you use and how do you use it? 18. How you support emergency orders (EO)? Probe for: Do you consider product redistribution within the district before authorizing EO? Do you support facilities with transport to collect emergency orders? On Average how many emergency orders do you authorize per quarter ----------------------------------------------Impact question ----------------------------- ------------------------ Let’s talk about the eLMIS and its impact on your job. 19.Has using the eLMIS had an impact on your overall job if so How? {Probe: a. Daily impact 20.Are you spending more or less time reporting data with eLMIS? If less i. What they are doing with the extra time If more ii. What activities in eLMIS takes more time 21.Since the introduction of the eLMIS, have your reporting responsibilities become more or less time consuming? 22.Based on your experience with the eLMIS so far, would you recommend that other health facilities have it or not? Why or why not? 23.With your experience working with the system, what limitation do feel 66 | Page the system has and what solutions would you recommend for those limitations? 24.What suggestions for improvement do you have for the eLMIS? You have now reached the end of the questionnaire Endline 18th February – 30th April, 2019 Zambia eLMIS Evaluation: HEALTH CENTER AND HOSPITAL Interview Guide (QUALITATIVE TOOL) Instructions to facilitators: The following questions are a guide. An in-depth interview should feel like a conversation (where the respondent does most of the talking). It is best to begin with easy, open ended questions so the respondent feels comfortable and it allows them to convey in their own words their experience. Focus on the respondent’s experience and weave the topics and subtopics into the conversation (rather than worrying about asking each question as written). Try not to ask them to generalize or summarize their opinions on the eLMIS until the very end. Try not to ask Yes/No questions or leading questions. Ask respondents to illustrate their opinions with examples or use their examples to draw out their feelings and perceptions. You should probe and ask follow up questions only where appropriate. Name of Facilitator: ________________ Name of Note taker: _____________ Date: _____________ Province: ________________________ District: ________________________ Facility Name: _____________________________________________________ Facility Code: ____________________ Respondent’s sex (Please circle one): Male Female Respondent’s job title: ________________________________________ Interview Start Time: End Time: Instructions: Please introduce yourself to the respondent and thank him or her for their time. Explain the objectives of the endline data collection: the purpose of this evaluation is to assess the extent to which the eLMIS has improved supply chain processes compared to the midline evaluation findings. The evaluation will focus on supply chain performance, data quality and use, and acceptability of the system. We would like to find out a little more about your use of the eLMIS. Let’s begin by talking a little bit about your job and your interaction with the eLMIS 68 | Page 1. How long have you been working in your position? 2. What are some of your activities as a……….. (Use their Job title)? {Probe for descriptions or examples.} 3. What are some of the challenges that you face as part of your work? {Probe for examples.} ------------------------------------------------ eLMIS Questions -------------------------- ----------------------- Let’s talk about the eLMIS and how you use it in your work. 4. How do you use the eLMIS as part of your work? {Probe for: a. How often b. How long they have been using it } 5. Can you describe your most recent experience of using the eLMIS? {Probe for: a. last time used f. what they used it for g. What they like or don’t like about using eLMIS h. What modules they find useful and which one they feel are not useful i. advantages and disadvantages ------------------------------------Information questions-------------------------- --------------------- Let’s talk about the information that’s in the eLMIS. 6. Has using logistics data changed since the introduction of the eLMIS in your health facility? {Probe for: How it has changed} 7. Has the quality of data changed since the introduction of the eLMIS? {Probe for: How Data quality before (get example) Issues/challenges with data quality before (get example)} 8. After data is entered in the eLMIS, how do you use the information? {Probe for: Data availability What data is missing in the eLMIS How the missing data be useful Do you review data in the eLMIS? What they check for when reviewing What actions they take after reviewing the data Do you use eLMIS information to produce any other reports? What reports (e.g. HMIS, HIA2) What the reports are for } ------------------------------Hospital/Health Centre Supervision questions------ ------------------------------------ 9. Supervision: a. Do you do any type of supervision in your role (If no, skip to question 17) If yes, b. What kind of supervision do you do? c. How has your supervision changed since the introduction of eLMIS and can you give specific examples? d. Do you use data from the eLMIS to help with supervision? What data do you use? How do you use this data? ----------------------------------------------Impact question ----------------------------- ------------------------ Let’s talk about the eLMIS and its impact on your work. 10.Has using the eLMIS had an impact on your overall work if so How? {Probe: a. Daily impact 11.Are you spending more or less time reporting data with eLMIS FE? If less i. What they are doing with the extra time 70 | Page If more ii. What activities in eLMIS takes more time 12.Since the introduction of the eLMIS, have your reporting responsibilities become more or less time consuming? 13. What are some of the factors that cause you to place emergency orders? Probe for: Last emergency order Why the facility reached EOP 14.With your experience working with the system, what limitations do you feel the system has and what solutions would you recommend for those limitations? 15.What suggestions for improvement do you have for the eLMIS? You have now reached the end of the questionnaire APPENDIX 4. REALIGHMENT OF SUPPLY CHAIN LOGISTICS MANAGEMENT INFORMATION SYSTEMS 72 | Page JSI Research & Training Institute, Inc. Plot 8658 Kudu Road PO Box 320087 Lusaka, Zambia JSI Research & Training Institute, Inc. 2733 Crystal Drive, 4th Floor Arlington, VA 22202 USA Phone: 703-528-7474 Fax: 703-528-7480 Web: aidsfree.usaid.gov