DELIVER LOGISTICS MANAGEMENT INFORMATION SYSTEM FINAL EVALUATION REPORT DECEMBER 1, 2016 This publication was produced for review by the United States Agency for International Development. It was prepared by (in alphabetical order) Misbah Aman, Robert Bernstein, Hammad Habib, Muhammad Khalid, Atif Rao, and Abid Ali Soomro, Management Systems International, a Tetra Tech Company. DELIVER LOGISTICS MANAGEMENT INFORMATION SYSTEM FINAL EVALUATION REPORT Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. DELIVER LMIS: Final Evaluation Report iii ACKNOWLEDGMENTS The evaluation team would like to thank all government personnel, NGOs, donors, and other stakeholders interviewed, who generously gave their valuable time and input to the evaluation process despite busy schedules. The team is also grateful for the support and cooperation of USAID/Pakistan staff. DELIVER LMIS: Final Evaluation Report iv CONTENTS Acknowledgments............................................................................................................................................................ iii Acronyms............................................................................................................................................................................ vi Project Summary................................................................................................................................................................1 Executive Summary ...........................................................................................................................................................2 Evaluation Purpose and Questions................................................................................................................................4 Project Background...........................................................................................................................................................4 The DELIVER Project ..............................................................................................................................................4 Theory of Change and Intended Results.............................................................................................................5 Evaluation Methods and Limitations..............................................................................................................................6 Data Collection Methods and Sources................................................................................................................6 Data Analysis..............................................................................................................................................................7 Limitations..................................................................................................................................................................9 Findings.................................................................................................................................................................................9 Findings for Question 1: Effectiveness.................................................................................................................9 Objective 1: Improve and Strengthen In-Country Supply Chains.......................................................9 Objective 2: Strengthen Environments for Commodity Security .....................................................14 Objective 3: Increase Knowledge Management and Dissemination .................................................17 Conclusions for Question 1 .......................................................................................................................19 Findings for Question 2: Sustainability of cLMIS .............................................................................................19 Perceptions of Usefulness...........................................................................................................................20 Knowledge and Skills....................................................................................................................................20 Staff Turnover................................................................................................................................................21 Maintenance of Equipment and Supplies .................................................................................................21 Coordination Between DoH and PWD..................................................................................................22 Factors Influencing Institutionalization of the cLMIS ............................................................................23 Conclusions for Question 2 .......................................................................................................................24 Findings for Question 3: Scale-up of vLMIS......................................................................................................24 vLMIS Scale-up...............................................................................................................................................24 Programmatic Efficiency ..............................................................................................................................25 vLMIS Reporting............................................................................................................................................25 Vaccine Coverage and Wastage ................................................................................................................25 Conclusions for Question 3 .......................................................................................................................26 Findings for Question 4: Best Practices ............................................................................................................27 Cross-Cutting Findings: Gender .........................................................................................................................28 DELIVER LMIS: Final Evaluation Report v Overarching Conclusions..............................................................................................................................................28 Recommendations...........................................................................................................................................................29 Lessons Learned...............................................................................................................................................................30 Annexes.............................................................................................................................................................................31 Annex 1: Evaluation Scope of Work..................................................................................................................31 Annex 2: Assignment Work Plan........................................................................................................................35 Annex 3: Data Collection Instruments..............................................................................................................53 Annex 4: List of Interviews................................................................................................................................ 155 Annex 5: List of Documents Reviewed.......................................................................................................... 156 Annex 6: Desk Research on LMIS ................................................................................................................... 157 Annex 7: Qualitative and Quantitative Interviews....................................................................................... 177 Annex 8: Trend Analysis.................................................................................................................................... 183 Annex 9: Conflict of Interest Statements...................................................................................................... 194 List of Tables Table 1: Project Summary................................................................................................................................................1 Table 2: Distribution of Sample by Districts...............................................................................................................8 Table 3: Provincial/District Managers’ and Data Entry Operators’ Knowledge, Skills, and Abilities to Use cLMIS ...................................................................................................................................................................21 Table 4: Factors Influencing Sustainability of cLMIS.................................................................................................24 List of Figures Figure 1: Trends in CYPs from Three-month Injections........................................................................................10 Figure 2: How the cLMIS Improved Supply Chain Management..........................................................................11 Figure 3: Reporting Rates for Measles Vaccine ........................................................................................................13 Figure 4: How the vLMIS Improved Supply Chain Management..........................................................................13 Figure 5: How the cLMIS Improved Commodity Security.....................................................................................15 Figure 6: How the vLMIS Improved Vaccine Security.............................................................................................16 Figure 7: Challenges to Commodity Security ...........................................................................................................16 Figure 8: Challenges to Vaccine Security ...................................................................................................................17 Figure 9: Data-Driven Decisions..................................................................................................................................17 Figure 10: Demonstrated Skills of Managers and DEOs........................................................................................19 Figure 11: How Managers and DEOs Access the cLMIS........................................................................................20 Figure 12: Availability of Functional Hardware at Facilities...................................................................................22 Figure 13: Coordination Between DoH and PWD.................................................................................................23 Figure 14: Measles Coverage and Wastage...............................................................................................................26 DELIVER LMIS: Final Evaluation Report vi ACRONYMS AJK Azad Jammu and Kashmir AOR Agreement Officer’s Representative ASV Assistant Superintendent Vaccination BCC Behavior Change Communications BCG Bacillus Calmette –Guérin (vaccine) cLMIS Contraceptives Logistics Management Information System COR Contracting Officer’s Representative CYP Couple Years of Protection DEO Data Entry Operator DHS Demographic and Health Survey DoH Department of Health DSV District Superintendent Vaccination EPI Expanded Program on Immunization FEFO First Expiry, First Out FGD Focus Group Discussion (used only in annex tables) FIFO First In, First Out FP/RH Family Planning and Reproductive Health FWW Family Welfare Worker GoP Government of Pakistan HSS Health Systems Strengthening ICT Islamabad Capital Territory IUCD Intrauterine Contraceptive Device JSI John Snow, Inc. KII Key Informant Interview (used only in annex tables) KP Khyber Pakhtunkhwa LHS Lady Health Supervisor LHW Lady Health Worker LMIS Logistics Management Information System LQAS Lot Quality Assurance Sampling MCH Maternal and Child Health MIS Management Information System MNCH Maternal, Newborn, and Child Health MSI Management Systems International PPHI People’s Primary Healthcare Initiative PWD Population Welfare Department SCM Supply Chain Management SDP Service Delivery Point TSV Tehsil Superintendent Vaccination UNFPA United Nations Population Fund UNICEF United Nations Children’s Emergency Fund UPS Uninterruptable Power Supply USAID United States Agency for International Development USB Universal Serial Bus vLMIS Vaccines Logistics Management Information System WHO World Health Organization DELIVER LMIS: Final Evaluation Report 1 PROJECT SUMMARY Table 1summarizes basic information about the DELIVER Logistics Management Information System (LMIS) project. TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers Contract No. GPO-I-00-06-00007-00 Contracting/Agreement Officer’s Representative (COR/AOR) - Start date 2009 Completion date 2016 Location Nationwide Implementing partner(s) John Snow, Inc. USAID/Pakistan Mission Strategic Framework objectives addressed IR 5.1: Increased utilization of quality family planning and maternal and child health (MCH) services Budget - DELIVER LMIS: Final Evaluation Report 2 EXECUTIVE SUMMARY Evaluation Purpose and Questions The final evaluation of the logistics management information system (LMIS) focuses on assessing the effectiveness of the LMIS component of the DELIVER project. The USAID/Pakistan Health Office expects to use the best practices, innovations, and lessons learned to guide the implementation of existing projects and the design of future projects. The evaluation focuses exclusively on the LMIS component of the DELIVER project and answers four questions: 1. To what extent has the project been successful in meeting its three major objectives for the LMIS activity? In particular, to what extent has trained staff used training to address supply chain gaps or issues? To what extent is the staff using data for decision-making? 2. What changes could be made to ensure sustainability of the cLMIS and to strengthen data driven decisions? 3. What change could be made to increase programmatic and cost efficiencies of vLMIS scale-up? 4. What best practices, innovations, and lessons learned can be applied to future programming in supply chain systems strengthening? Project Background The DELIVER project aimed to strengthen the Government of Pakistan’s (GoP’s) supply chains for family planning commodities and, later, vaccines to improve commodity security and increase knowledge management and dissemination. The project designed, developed, and deployed two LMIS applications for the public sector—the contraceptives LMIS (cLMIS) and the vaccines LMIS (vLMIS). The project also provided technical support to the GoP in contraceptive forecasting, procurement planning, warehouse management, supply chain strengthening, and automating warehousing. The cLMIS has been scaled up to all districts of Pakistan, while the vLMIS has been scaled up to 83 districts, mostly in Punjab and Sindh. Project activities focus on training, and the project reports having trained 6,071 provincial and district managers, lady health workers (LHWs), family welfare workers (FWWs), and vaccination supervisors. The evaluation relied largely on qualitative data obtained from interviews with supply chain actors, project staff, stakeholders, and experts and group discussions with data entry operators, lady health supervisors (LHSs), and FWWs. It also used quantitative data from the LMIS to explore trends in indicators of supply chain operation and performance. Key Findings and Conclusions DELIVER has improved supply chain performance: The cLMIS and vLMIS have improved the operation and performance of their respective supply chains. Trends in indicators of supply chain performance, i.e., reporting rates, consumption, wastage rates, and vaccine coverage, have increased significantly in project-supported provinces relative to other provinces. For cLMIS, managers, data entry operators (DEOs), and other supply chain actors explained that better record keeping improved the timeliness and accuracy of data on stocks and consumption. This information allowed them to make and fill resupply requests based on inventory and demand instead of requesting or supplying the same fixed amount each time as had been the practice in the past. Aligning stocks more closely with demand reduced stock-outs (improving commodity security) and overstocks (reducing wastage). Better record DELIVER LMIS: Final Evaluation Report 3 keeping also improved transparency and reduced pilferage. For vaccines in particular, better stock rotation practices—such as first expiry, first out (FEFO) and first in, first out (FIFO)—and cold chain facilities reduced wastage. DELIVER has facilitated data-driven decision-making: Although few managers and DEOs described specific decisions for which they relied on LMIS data, the most common explanation of how the LMIS had improved supply chains rested on using more timely and accurate inventory data to align resupply orders with demand. This is an important decision-making function which has substantially improved many aspects of supply chain performance. Sustainability of cLMIS: Prospects for sustainability of the cLMIS are promising. Managers and DEOs are using the skills they have learned to improve supply chain performance and see value in the LMIS. Prospects for sustainability are probably higher in Punjab and Sindh, where results have been more pronounced and collaboration between the two government stakeholders, the Department of Health (DoH) and Population Welfare Department (PWD), is more advanced. The fact that many indicators of supply chain performance have started to decline since the project started scaling back implementation in September 2015 emphasizes the need for additional training for managers (provincial and district) on using LMIS data, dedicated staff to enter data, and more complete and functional hardware and internet access to ensure sustainability. Scale-up of vLMIS: The vLMIS has proven useful and effective, and this has garnered substantial support for scaling it up to at least the district level. However, scaling up will require additional investment in infrastructure and human capital, especially in Khyber Pakhtunkhwa (KP), where vLMIS was implemented in only five districts, and Balochistan, where it was implemented in nine. Best practices: Respondents identified the LMIS itself to be a best supply chain management practice, as it not only shifted the traditional manual reporting system to convenient and timely online reporting, but at the same time introduced the much needed LMIS for vaccines and contraceptives. The warehousing practices promoted by the project—e.g., FEFO, FIFO, tracking expiry dates, and more closely aligning inventory to demand—are also best practices in supply chain management. Summary Recommendations  Future projects should explore the possibility of integrating the multiple vaccine and commodity MISs maintained at the provincial and national levels to improve the efficiency of supply chain management, e.g., integrating the cLMIS with the LHW program MIS.  To promote cLMIS sustainability and vLMIS programmatic efficiency, future projects should continue to train managers and DEOs on how to access and use the LMIS. Supportive supervision and post-training follow-up visits may be effective methods. It is particularly important to train managers (provincial and district) on additional ways to use LMIS data for decision-making.  Future projects to support the LMIS might consider advocating for separating the duties of the DEO from store management to relieve potential staffing constraints and improve transparency.  To support prospects for sustainability and scale-up, future projects should consider advocating with the government and potential donors to institutionalize the LMIS, improve collaboration between DoH and PWD, and consolidate around a single LMIS. In the context of the 18th Amendment, advocacy may also be required to allocate the budgets at the provincial level necessary to provide adequate storage, cold chain facilities, and transportation. DELIVER LMIS: Final Evaluation Report 4 EVALUATION PURPOSE AND QUESTIONS The final evaluation of the logistics management information system (LMIS) focuses on assessing the effectiveness of the LMIS component of the DELIVER project. The project supports the fourth component of the USAID/Pakistan maternal and child health (MCH) program. The USAID/Pakistan Health Office expects to use the best practices, innovations, and lessons learned identified in the evaluation to guide the implementation of existing projects and the design of future projects. Audiences for the evaluation include USAID/Pakistan, John Snow, Inc. (JSI), implementing partners leading other USAID/Pakistan MCH projects, and government and other external stakeholders. Evaluation Questions The scope of work (Annex 1) posed three evaluation questions that the assignment work plan (Annex 2) describes in more detail. The specific questions are: 1. To what extent has the project been successful in meeting its three major objectives for the LMIS activity? In particular, to what extent has trained staff used training to address supply chain gaps or issues? To what extent is the staff using data for decision-making? 2. What changes could be made to ensure sustainability of the cLMIS and to strengthen data-driven decisions? 3. What change could be made to increase programmatic and cost efficiencies of vLMIS scale-up? 4. What best practices, innovations, and lessons learned can be applied to future programming in supply chain systems strengthening? PROJECT BACKGROUND Prior to passage of the 18th Amendment by Pakistan’s National Assembly in 2010, the Government of Pakistan (GoP) procured family planning commodities through the United Nations Population Fund (UNFPA) and relied on a paper-based tracking system to manage the commodity supply chain. The GoP also used the World Health Organization’s (WHO’s) vaccine storage and supply management software to track vaccines at the federal level only. After passage of the 18th Amendment, USAID/Pakistan began procuring all family planning commodities for the public sector and storing them at a central warehouse that directly distributes commodities to surrounding districts. For vaccines, the United Nations Children’s Emergency Fund (UNICEF) manages international procurement, while federal and provincial expanded programs on immunization (EPIs) handle local procurement. The federal EPI is responsible for vaccine storage at the federal EPI warehouses and distribution for the entire country, except for vaccines procured by the provincial EPIs. The DELIVER Project The DELIVER project is one component of a five-component initiative for strengthening MCH programs. The components are family planning and reproductive health (FP/RH); maternal, newborn, and child health (MNCH); behavior change communications (BCC); health commodities and supply chain management (SCM); and health systems strengthening (HSS). DELIVER LMIS: Final Evaluation Report 5 The DELIVER project was initially a five-year contract managed out of USAID’s Bureau of Global Health and implemented by JSI; later it was extended for two more years. DELIVER was one of the first supply chain management interventions implemented in Pakistan. The project aimed to improve and enhance the GoP’s in-country distribution of health commodities and strengthen the supply chain systems. Its three major objectives were to improve and strengthen in-country supply chains, strengthen environments for commodity security, and increase knowledge management and dissemination. The DELIVER project designed, developed, and deployed two LMIS applications for the public sector— the contraceptives LMIS (cLMIS) and the vaccines LMIS (vLMIS)—that captured multiple levels of storage, consumption, and wastage data from the union council, district, provincial, and national levels for vaccines, contraceptives, and tuberculosis commodities, ensuring visibility and accountability of these public-sector commodities. The project also provided technical support to the GoP in contraceptive forecasting, procurement planning, warehouse management, supply chain strengthening, and automating warehousing. After the cLMIS launched in July 2011, it was expanded to report contraceptive and tuberculosis logistics data from the country’s 143 districts. In May 2013, USAID/Pakistan asked DELIVER to expand the web￾based LMIS to cover and improve the vaccine and cold chain logistics management system in Pakistan. Based on strategic meetings with all stakeholders, DELIVER designed a comprehensive, sustainable, and automated vLMIS. DELIVER initially implemented the vLMIS in 54 districts prioritized because of their high incidence of polio, including 9 districts and 3 towns of Sindh. In February 2015, responding to a request from the Sindh government’s Department of Health (DoH), USAID/Pakistan supported the scaling up of the vLMIS in all districts and towns of the province. Implementation Project activities focus on training. The project reports having trained 1,047 individuals on operating and using the cLMIS. Of the 1,008 trainees on which the evaluation team had information, the majority (66 percent) were from the DoHs (397 from DoH, 161 from the lady health worker [LHW] program, 106 from the Integrated Reproductive Maternal Newborn Child Health and Nutrition Program, and 25 from the People’s Primary Health Care Initiative—the entity responsible for managing the DoH’s Basic Health Units in Sindh), 29 were from the Capital Development Authority, 10 were from the Family Planning Association of Pakistan, 2 were from GreenStar Social Marketing, and 278 were from the Population Welfare Department (PWD). The project also reports having trained 5,024 participants on the vLMIS, the majority of whom (98 percent) were from the DoHs (4,829 from DoH, 58 from the Global Alliance for Vaccines and Immunization, and 26 from the People’s Primary Health Care Initiative); 13 from UNICEF; 3 from the federal EPI; 2 from the LHW program; 1 from the Federal Ministry of National Health Services, Regulations, and Coordination; and 92 from other departments. Among the 5,024 participants, 50 were trained as master trainers on vLMIS, most (36) of whom were from the DoH. This evaluation is a follow-up to the midterm evaluation conducted in 2013, but unlike the midterm, it focuses exclusively on the LMIS component. It examines implementation of the LMIS, sustainability of the cLMIS, scale-up of the vLMIS, and strengthening of the commodity supply chains, and covers project activities from September 2012 to May 2016. Theory of Change and Intended Results According to the 2013 Pakistan Demographic and Health Survey (DHS), the contraceptive prevalence rate in the country was only 35.4 percent, which means that nearly two-thirds (64.6 percent) of married DELIVER LMIS: Final Evaluation Report 6 women aged 15–49 did not use any contraceptive methods. One in five married women had an unmet need for family planning services, and the total fertility rate was 3.8 children per woman. An avoidable unwanted pregnancy can be costly for both the mother and child’s health in addition to the direct healthcare costs of a pregnancy. By directly supporting activities that strengthen the supply chains for family planning commodities and vaccines, the DELIVER project expected to improve supply chain management and performance. Improved performance would improve distribution and storage, reduce wastage and pilferage, provide data for forecasting and procurement, and ultimately improve access to family planning commodities and vaccines. Improved access to family planning commodities and vaccines would contribute to the primary goal of USAID/Pakistan’s health programming—to reduce maternal and child mortality. EVALUATION METHODS AND LIMITATIONS The evaluation employed a mixed-methods approach that included collecting and analyzing quantitative and qualitative data from multiple sources (i.e., project documents, GoP line departments, stakeholders, project and USAID staff, and experts). The mixed-methods approach ensured multiple levels of triangulation to help answer the evaluation questions. In total, the team developed six data collection instruments (Annex 3) tailored for different audiences and methods. The quantitative data documented what happened, while the qualitative data helped explain how and why. Annex 4 contains the list interviews. Data Collection Methods and Sources Prior to beginning fieldwork, the evaluation team conducted a team planning workshop during which it developed a data analysis plan, designed data collection instruments, planned the fieldwork, and presented the evaluation plan to the Mission. During the workshop, the team identified the five data collection methods and a variety of sources.  Document review—The evaluation team reviewed available project documents and reports, including the cooperative agreement, annual work plans and reports, the performance management plan, and procurement and training manuals. The document review helped the team develop a thorough understanding of project goals and objectives and planned and actual activities, outputs, and results. Annex 5 lists the documents the team reviewed, and Annex 6 contains a detailed review of selected documents.  LMIS online dashboard—The team downloaded data from the dashboard to analyze trends in key indicators of supply chain performance (couple years of protection [CYP], reporting rate, vaccine coverage, and wastage rates) in all provinces for key contraceptive and vaccine commodities.  In-depth interviews—The team conducted individual, in-depth interviews with district and provincial DoH and PWD managers and data entry operators (DEOs), relevant public and private health service providers, donors, and experts. The team also interviewed USAID and JSI/DELIVER staff and individuals from the project implementation organizations to develop a thorough understanding of project objectives, implementation mechanisms, and the evaluation purpose and context.  Group discussions—The team also conducted group discussions with lady health supervisors (LHSs), family welfare workers (FWWs), and vaccination supervisors including district, tehsil, and assistant superintendents vaccination (DSV, TSV, and ASVs) in each province. DELIVER LMIS: Final Evaluation Report 7  Direct observation—The team also collected qualitative data from direct observation of health department staff. Sampling The project piloted the cLMIS in 19 districts and rolled it out in 143 districts nationally; it piloted the vLMIS in 54 priority districts and later scaled up to 65 districts and 18 towns in Karachi. The evaluation team used a mix of two-stage purposive and random sampling to select districts and stores for site visits and data collection. At the first stage, the team purposively selected 11 of Pakistan’s 143 districts across all four provinces, Azad Jammu and Kashmir (AJK), Gilgit-Baltistan, and Islamabad Capital Territory (ICT). The team used clustering to keep field work practical, ensure adequate geographic coverage, and increase efficiency by covering cLMIS and vLMIS in a single location. Of the 11 selected districts, 7 were also covered in the midterm evaluation. At the second stage, health facilities for cLMIS and stores for vLMIS were treated as secondary sampling units. The team randomly selected at least four facilities and two stores in each sampled district. The evaluation team selected a convenience sample of DoH and PWD managers and DEOs1 and purposively selected stakeholders and experts to capture LMIS-specific expertise, experience, or perspectives. The team conducted 94 semi-structured individual interviews with key informants associated with the project and external sector stakeholders who could provide feedback on the LMIS’s design, implementation, and results. The team designed three instruments for these interviews, one for provincial/district level managers, one for provincial/district level DEOs, and one for national level stakeholders. The evaluation team conducted 11 group discussions with LHSs, FWWs, and ASVs. The discussions were conducted in the four sampled districts (Karachi, Muzaffargarh, Peshawar, and Quetta). Table 2 summarizes the data collection by province and district. Annex 7 provides more detail on the distribution of interviews by source and location. Data Analysis The quantitative data provided information regarding relevance, effectiveness, and sustainability of LMIS. The team used the Statistical Package for the Social Sciences to produce frequencies and cross￾tabulations for the quantitative analysis. The qualitative data provided detail to answer the questions of how and why various aspects of the LMIS design and implementation worked well or did not work well. The analysis used MAXQDA, a software package, to analyze the qualitative data. The package facilitates coding, organizing, and extracting patterns from the qualitative data. The team employed both deductive and inductive coding systems. 2 To integrate quantitative and qualitative data, the evaluation team used an explanatory approach. In this analysis, qualitative findings helped explain trends and findings in the quantitative data. 1 The team could not obtain a list of relevant managers or DEOs from which to draw a sample. Therefore, the team called provincial and district DoH, PWD, and People’s Primary Healthcare Initiative (PPHI) officials, and asked for their recommendations of individuals who were most knowledgeable of the LMIS. 2 A deductive code is a provisional code list based upon the background documents, evaluation questions, and data collection instruments. As the team coded interview transcripts against this provisional list, other (inductive) codes emerged progressively from the data itself. DELIVER LMIS: Final Evaluation Report 8 TABLE 2: DISTRIBUTION OF SAMPLE BY DISTRICTS Province or Territory Key Informant Interviews Group Discussions Total Interviews/ Group District DEOs Discussions District Managers National Managers Provincial Managers Donors/ NGOs/ Others ASV/LHS/ FWW AJK Muzaffarabad 2 2 - 1 - - 5 Balochistan Pishin 2 2 - - - - 4 Quetta 5 1 - 4 - 3 13 KP Abbottabad 3 2 - - - - 5 Peshawar 5 2 - 3 - 2 12 Punjab Muzaffargarh 2 2 - - - 3 7 Lahore 4 3 - 4 - - 11 Sindh Hyderabad 4 4 - 1 - - 9 Karachi 5 3 - 3 - 3 14 ICT Islamabad 3 1 3 - 18 - 25 Total 35 22 3 16 18 11 105 DELIVER LMIS: Final Evaluation Report 9 Limitations The mixed methods approach utilizes a wide variety of quantitative and qualitative data from multiple sources to ensure validity and reliability. However, the evaluation has the following limitations.  The DELIVER project’s operations ended by the time this evaluation began, so no members of the direct field implementation team were available for interviews. However, the evaluation team was able to interview DELIVER’s ex chief of party and monitoring and evaluation specialist to understand aspects of project implementation.  Questions pertaining to the trainings required recall of one year or more. The team mitigated potential recall bias by giving respondents enough time before answering the questions involving recall, and structuring questions and probing to assist with accurate recall.  Flight cancellations thwarted the team’s planned visit to Gilgit-Baltistan. Therefore, evaluation findings may not be valid in the context of Gilgit-Baltistan. The team conducted additional interviews in Quetta and Karachi to achieve the targeted sample size.  Findings cannot be generalized to the entire project because sample selection was not entirely random. Furthermore, the LMIS was just one component of the DELIVER project and the other components may also have affected outcomes. FINDINGS Findings for Question 1: Effectiveness Evaluation Question: To what extent has the project been successful in meeting its three major objectives for the LMIS activity? In particular, to what extent has trained staff used training to address supply chain gaps or issues? To what extent is the staff using data for decision-making? The three major objectives of the LMIS activity are: 1) improving and strengthening in-country supply chains, 2) strengthening environments for commodity security, and 3) increasing knowledge management and dissemination; this section addresses each of these objectives separately. Objective 1: Improve and Strengthen In-Country Supply Chains The analysis of this section first uses quantitative data from the online cLMIS and vLMIS to examine trends in indicators of supply chain operation (i.e., reporting rates) and performance (i.e., availability) for selected family planning commodities and vaccines during the 33-month period covered by the evaluation (September 2013–May 2016). The team did not visit the warehouses or service delivery points (SDPs) to verify that the information in the database was correct but did draw on secondary verification exercises. 3 The analysis also presents qualitative data from interviews with managers and others involved in supply chain operation and management to gain a nuanced understanding of how, if at all, the LMIS affected supply chain management and the challenges that remain. 3 Between December 2015 and January 2016, Apex Consulting conducted a rapid stock assessment of the DELIVER project and found that a majority of SDPs (more than 80 percent) and stores (more than 70 percent) had accurate stock records (i.e., +/- 10 percent discrepancy between physical inventory and stock register balance) for family planning commodities. The assessment covered a total of 1,991 facilities (71 stores and 1,920 SDPs) over seven weeks; 952 facilities in 10 Punjab districts (30 stores and 922 SDPs), and 1,039 in 11 Sindh districts (41 stores and 998 SDPs). DELIVER LMIS: Final Evaluation Report 10 cLMIS Trend Analysis Quantitative data downloaded from the cLMIS dashboard show significant positive trends in consumption4 and CYP for three-month contraceptive injections (short-term) and the Copper-T (long￾term). These are common commodities that flow through the system and thus provide a good test of system performance. Time series regression analysis of consumption and CYP for both methods over the 33-month period found a significantly positive trend (improvement) in both measures for both methods in the two provinces on which the project focused (Punjab and Sindh) relative to other areas. Finally, visual inspection of the trends shows an initial steep increase followed by a flattening, and eventual decline, in the trend, much of it after September 2015 when the project was scaling back its support, particularly in the non-focus regions, and USAID stopped procuring commodities. The regression analysis found a significant correlation between the declining trend and scaled back implementation. Error! Reference source not found. illustrates the trend in CYPs associated with three-month injections. Annex 8 provides the full results of the trend analysis. The improvement in stock situations cannot be solely attributed to the implementation of cLMIS, since during the period of analysis, USAID also started procuring contraceptives, which may have affected the availability of family planning supplies. FIGURE 1: TRENDS IN CYPs FROM THREE-MONTH INJECTIONS cLMIS Qualitative Analysis To examine the causes of the observed trends, the evaluation team conducted interviews with district and provincial managers and DEOs and group interviews with LHSs and FWWs to explore if and how the cLMIS had affected supply chain management and performance. In individual interviews, 97 percent of 32 provincial and district level managers said that they believed the cLMIS had improved supply chain management. The team asked those who said supply chain management had improved to describe how, and the 31 managers provided 50 separate responses. 5 The various individuals the evaluation team interviewed had different roles in and perspectives on the supply chain. The story that emerged from examining the responses as a whole is that the cLMIS facilitates convenient (from anywhere) and timely access to information on stocks/inventory and consumption (as estimated from distribution) at the 4 Consumption is estimated from quantities distributed to users by SDPs, the LHW program, basic health units, rural health centers, and tehsil headquarters hospitals. 5 The analysis identifies a “response” as a separate coded theme in what may be a broader response. DELIVER LMIS: Final Evaluation Report 11 district and facility levels. Easy access to accurate data on stocks and demand helped managers maintain adequate months of supply at each point in the supply chain (either through improved forecasting or by facilitating transfers from surplus to deficit SDPs) and thus helps prevent stock-outs. One respondent noted that maintaining adequate supply maintains the contraceptive prevalence rate. One also implied that the cLMIS helped reduce pilferage. As a district manager for PWD explained: “cLMIS has [a] very positive effect on supply chain management. Before cLMIS manual reporting was done, which was not [an] accurate and reliable system, [the] storekeeper was managing [the] stock register manually and it was easy to change any record (enter any bogus entry) at any time. Now as we are entering data on [an] online system, it helps us in maintaining [the] quality and accuracy of the data. Facility staff submits their monthly reports, we check and verify opening, closing balance, and consumption of family planning commodities from monthly reports, and then send contraceptives stock to facilities or facility staff self-pick their stock from [the] district store.”—District manager, PWD The left side of Figure 2 summarizes the coded themes that support this overall explanation. The associated percentages—for Figure 2 and subsequent similar figures—represent the percentage of multiple responses that reflected the theme. FIGURE 2: HOW THE cLMIS IMPROVED SUPPLY CHAIN MANAGEMENT Managers (32 respondents, 50 responses) LHSs and FWWs (51 respondents, 16 responses) Monitor consumption and demand 39% Improve reporting and record keeping 50% Track inventory 34% Helps predict requirements 19% Manage stock between SDPs 12% Feedback improves reporting accuracy 13% Prevent stock-outs 10% The evaluation team also asked 51 FWWs and LHSs in group discussions how the cLMIS had changed the supply chain. These individuals did not use the cLMIS directly; they reported consumption data to others who entered it into the system and received commodities from SDPs based on forecasted requirements generated by the cLMIS. The right side of Figure 2 summarizes their coded responses. The conclusion that emerged from the discussions is that the cLMIS has improved the accuracy of data, largely due to timely feedback from the DEOs on potential data errors. Better record keeping has improved forecasting, and LHSs and FWWs now get the quantities of commodities they request more often than before. Three specifically noted that they now get an itemized and accurate list of commodities they receive, while before they had to sign receiving sheets that were not itemized and did not always correspond to actual quantities received. Additionally, 6 of 10 donors, implementers, and technical experts interviewed said that the cLMIS had improved the supply chain for family planning commodities. However, they gave few concrete explanations for their responses. DELIVER LMIS: Final Evaluation Report 12 Summary Conclusions: The evidence suggests that the cLMIS has improved performance of the commodities supply chain. Consumption of commodities has increased significantly in the two provinces where the project focused its support relative to other areas. Because other factors contributed to increasing consumption, the positive trends are weak evidence that the improvement is attributable to the project. However, convincing qualitative evidence points to the role of the cLMIS in strengthening the supply chain to better manage stocks, prevent stock-outs, and reduce wastage, all of which can contribute to increased availability of commodities. vLMIS Trend Analysis For vaccines, the evaluation team used time series regression analysis to examine trends in reporting rates (an indicator of supply chain operation) for Bacillus Calmette–Guérin (BCG), Pentavalent (routine immunization), and measles vaccine for all districts of Pakistan where the vLMIS was implemented. It also analyzed trends in coverage for BCG, Pentavalent, and measles and wastage rates for measles.6 The team selected these vaccines to examine supply chain performance for routine vaccines and less common vaccines. The analysis revealed an increase in reporting rates for all three vaccines but no difference in reporting rate trends between the project’s focus provinces of Punjab and Sindh and other provinces and regions. It found a significant decline in reporting rates for all vaccinations as implementation tapered off from September 2015 onward, but with a steeper decline among areas with less project support. Consumption of BCG and Pentavalent vaccines increased substantially during the period, and even more so in Punjab and Sindh than elsewhere.7 The slowdown in implementation after September 2015 reduced consumption of BCG but did not significantly affect the positive trend in Pentavalent consumption. The analysis also examined trends in measles vaccine wastage and found rates to be lower in project focus provinces than elsewhere. Figure 3 illustrates trends in the reporting rate for measles vaccine. Annex 8 provides additional detail about trends in vaccine coverage, reporting rates, and wastage. 6 Because of the time required to download detailed data from the online database, the evaluation team obtained and analyzed data from only a handful of key products and indicators. 7 The team did not have data on consumption for the measles vaccine. The cLMIS Improved Distribution of Commodities “This system is good as it provides district-wise data, with which we can identify regions where specific contraceptive commodities are not available despite unmet needs as measured by survey data.”—Greenstar Social Marketing cLMIS focal person “Rechecking the stock balance and record keeping has improved. The data entry operator gives us timely feedback if there are any errors in the data. We get to know about our requirement of family planning commodities for a three-month period.”—FWW “It [the cLMIS] has improved the accuracy of data; issues and errors in reporting are highlighted and addressed in a timely manner.”—LHS DELIVER LMIS: Final Evaluation Report 13 FIGURE 3: REPORTING RATES FOR MEASLES VACCINE vLMIS Qualitative Analysis Consistent with the results of the quantitative data analysis, 8 of 10 managers the team interviewed said that the vLMIS had improved the vaccine supply chain. When asked to describe how, they explained that the vLMIS facilitates timely and reliable inventory monitoring, which helps prevent stock-outs. They also noted that their stock calculations can now incorporate data on vaccines that have passed, or are close to, their expiry dates. The left side of Figure 4 presents the frequency of themes in the coded data that support this conclusion. Managers also noted that additional capacity building of the facility and field staff could further improve the supply chain (1 response) and suggested that the vLMIS needs to be scaled up to all SDPs in each district (1 response). FIGURE 4: HOW THE vLMIS IMPROVED SUPPLY CHAIN MANAGEMENT Managers (10 respondents, 6 responses) Vaccination Supervisors (30 respondents, 34 responses) Prevent stock-outs 50% Improve reporting and record keeping 32% Improved inventory monitoring 33% Reduce wastage/pilferage of vaccines 15% Reduce wastage/pilferage of vaccines 17% Improved inventory monitoring 12% Improve reporting accuracy 9% Improve accessibility of data 9% The team also asked 30 vaccination supervisors (ASVs, DSVs, and TSVs) in focus group discussions about their perceptions of whether the vLMIS had changed the vaccine supply chain. The right side of Figure 4 presents the coded segments that describe their perceptions of how the vLMIS affected the vaccine supply chain. The overall explanation that emerged from the responses is that the vLMIS improved record keeping and reporting, which facilitated timely and accurate inventory monitoring. This in turn reduced wastage (by making it easier to identify older vaccines and either distributing them before they passed their expiry date or removing them from the inventory) and made it easier to identify and control pilferage. DELIVER LMIS: Final Evaluation Report 14 In unstructured interviews, only one of four donors, implementers, and technical experts knowledgeable of the vLMIS said that it had improved the supply chain of vaccines, while the other three reported no change. A national level manager explained that this may be because national government agencies are still using the older reporting systems as well as the vLMIS, so they now must maintain an additional reporting system, which is time-consuming and has delayed reporting. Summary Conclusions: The evidence suggests that the vLMIS contributed to improving the performance of the vaccine supply chain. Although external factors, such as measles outbreaks, have undoubtedly influenced supply chain performance indicators and the positive trends in these indicators, the qualitative evidence provides a level of plausible attribution to project activities. As with the cLMIS, the vLMIS has improved decision-making with respect to maintaining sufficient stocks without overstocking. This has improved access and reduced wastage. Objective 2: Strengthen Environments for Commodity Security The project’s performance management plan and annual reports define commodity security in terms of availability, a definition consistent with the MEASURE evaluation’s: “Commodity security exists when every person is able to choose, obtain, and use quality contraceptives and other reproductive health products whenever he or she needs them.” 8 Extending this definition to vaccines also implies a focus on availability. This evaluation addresses this question by examining if and how the LMIS has affected the availability of family planning commodities and vaccines. cLMIS The previous section presented evidence that the cLMIS has improved supply chain management, increased consumption and CYPs, and helped managers prevent stock-outs—all indicators of improved access to contraceptive commodities. The evidence also suggests that the cLMIS helped service providers—i.e., LHSs, FWWs, and SDPs—obtain the commodities they needed for distribution. This section presents qualitative evidence of two other aspects of commodity security—storage and distribution practices. To explore these dimensions of security, the evaluation team used qualitative data from interviews with managers, DEOs, LHSs, and FWWs to examine if, and how, the cLMIS contributed to better storage and distribution of family planning commodities to ensure continuous availability. In individual interviews, 72 percent of 32 managers and DEOs said that the cLMIS has contributed to safe storage of family planning commodities.9 An open-ended follow-up question asked respondents to explain how the cLMIS contributed to safe storage. Respondents largely interpreted security in terms of availability and explained that the timely and accurate stock and consumption information maintained in the cLMIS helped them forecast future consumption and order quantities consistent with the forecast, thereby contributing to preventing stock-outs and overstocking (a contributor to wastage). When asked a similar question in group discussions, 61 percent of 41 LHSs and FWWs said that their reporting into the cLMIS had not changed conditions for commodity security, and 39 percent reported a 8 MEASURE Evaluation. Accessed from: https://www.measureevaluation.org/prh/rh_indicators/crosscutting/commodity-security-and-logistics￾1/contraceptive-security-or-reproductive-health-commodity-security-strategy-is-being-implemented 9 The remaining 28 percent said the cLMIS had not changed storage practices but did not give reasons why. vLMIS Contributes to Improved Accountability “vLMIS is an important management component because for money coming into the system for immunization, it is essential to be able to monitor the procurement, storage, distribution, and use of vaccines and cold chain equipment and supplies that are purchased with these funds.”—World Bank official DELIVER LMIS: Final Evaluation Report 15 positive change. Those who reported no change either had not been trained in the LMIS and were not directly involved in entering data into the LMIS or could not differentiate between the effects of multiple interventions such as the LHW program MIS, improved monitoring and supervision, and the cLMIS. In Balochistan, DoH officials reported that the cLMIS had not been fully implemented and they were still resupplying fixed quantities that were not based on estimated consumption. Those who reported a positive change explained that better record keeping and reporting have enabled better matching of the quantity required (calculated based on previous consumption/distribution) to received quantities, and allowed them to receive three months of stock at a time. Better record keeping and reporting have also facilitated better inventory tracking, which has reduced wastage and pilferage. Twenty-five percent also noted that a cooler has reduced wastage of temperature-sensitive stock such as progesterone. Figure 5 presents the frequency of coded themes that emerged during the interviews to support the explanations given above. FIGURE 5: HOW THE cLMIS IMPROVED COMMODITY SECURITY Managers (32 respondents, 28 responses) LHSs and FWWs (51 respondents, 16 responses) Provides timely and accurate inventory data 43% Improved record keeping 25% Improved storage practices (coolers) 25% Helps match inventory to demand and storage capacity 43% Reduced wastage/pilferage 12% Six of 10 donors, NGOs, and technical experts also indicated that the cLMIS has improved the environment for security of family planning commodities. These informants’ understanding of, and exposure to, the cLMIS were very different from the LHSs’ and FWWs’, so it is understandable that their perceptions of its performance are quite different. Summary Conclusions: Respondents interpreted commodity security largely in terms of reliable access. Their explanations for if and how the cLMIS improved security therefore mirrored their responses to how the cLMIS affected supply chain performance in terms of managing stocks and reducing waste. While facility managers, DEOs, LHSs, and FWWs all noted improvements in the performance of the supply chain, managers and DEOs—i.e., those who interact most directly with the cLMIS—were more likely than LHSs and FWWs to attribute improved performance to the cLMIS. vLMIS When asked whether the vLMIS had contributed to the safe storage of vaccines, 80 percent of 10 managers and DEOs answered in the affirmative. In response to an open-ended follow-up question, they explained that the vLMIS allows them to track batch numbers, expiry dates, and the shelf location of vaccines. Using this information, and practices such as first in, first out (FIFO) and first expiry, first out (FEFO), they can better manage stock to prevent overstocking, ensure quality, and reduce wastage. vLMIS Facilitated Safe Storage of Vaccines “The basic reason behind Pentavalent wastage case was that it was heavily over-stocked. What we did through vLMIS was, we applied effective vaccine management and calculated the capacities of cold rooms and how many vials of which antigens we can place in [them]. We incorporated those capacities into vLMIS and planned our shipments according to that capacity.”—National EPI manager DELIVER LMIS: Final Evaluation Report 16 In group discussions, 53 percent of 30 vaccination supervisors provided detailed descriptions of how the vLMIS improved vaccine storage. Overall, their explanations were similar to those related by managers and DEOs, i.e., that access to more timely and accurate data in the vLMIS has facilitated better monitoring of expiry dates and inventory, which has reduced wastage and pilferage. Figure 6 illustrates the main coded themes from the qualitative data that support these explanations. FIGURE 6: HOW THE vLMIS IMPROVED VACCINE SECURITY Managers (10 respondents, 11 responses) Vaccination Supervisors (30 respondents, 26 responses) Prevents overstocking (wastage) 49% Improved accuracy of inventory data 21% Cold chain and storage reduce wastage 30% Prevented wastage 15% Improved record keeping 21% Improved inventory monitoring 12% Challenges in Strengthening Commodity Security In response to an open-ended question about how the cLMIS had, or had not, contributed to commodity security, 32 managers and 51 LHSs and FWWs mentioned several challenges. Managers explained that they had little or no budget for transportation, either from the central warehouse to the provinces or within the provinces. They described an inefficient and costly (bilty) transportation system to move commodities from the central warehouse to the provincial. Seventeen percent noted that the central warehouse did not deliver requested commodities in a timely manner and often did not communicate with the provinces regarding the status or delivery date for an order. These issues made the delivery of commodities unpredictable and difficult to track and affected availability. Finally, they mentioned that providing contraceptives was not a high priority for the DoH, which led to limited coordination between the DoH and PWD to improve the commodity supply chain. Group discussions with 51 FWWs and LHSs revealed similar issues related to transportation and storage. Respondents explained that they had to arrange to collect commodities on their own with no budget and few operational vehicles. Issues with storage centered on the lack of storage space and the inability to control temperature and humidity. Figure 7 presents the qualitative data on challenges to ensuring family planning commodity security. FIGURE 7: CHALLENGES TO COMMODITY SECURITY Managers (32 respondents, 33 responses) LHSs and FWWs (51 respondents, 19 responses) Transportation from the central warehouse to the SDPs 19% Transportation from the district stores to the SDPs 32% Delayed communication with the central warehouse regarding supply 17% Storage at the SDPs 26% Lack of coordination between DoH and PWD 12% Maintaining the cold chain for injections 21% In interviews with managers and vaccinators, issues related to transportation, storage, and maintaining the cold chain also emerged as key factors affecting the security of vaccines. Managers further mentioned that a shortage of skilled DEOs at the facility level constrained the overall performance of the vLMIS and that they did not have dedicated staff to unload vaccines that were delivered to their stores (Figure 8). DELIVER LMIS: Final Evaluation Report 17 FIGURE 8: CHALLENGES TO VACCINE SECURITY Managers (10 respondents, 8 responses) Vaccination Supervisors (30 respondents, 14 responses) Maintaining the cold chain 26% Transportation 29% Shortage of vaccines at the central warehouse 21% Inadequate storage 21% Limited staff (human resources) 16% Maintaining the cold chain at SDPs 21% Objective 3: Increase Knowledge Management and Dissemination The evidence to address this objective comes largely from the previous sections, which provide ample evidence that managers and others are using the cLMIS and vLMIS to make decisions, largely related to stocking. Additionally, the evaluation team asked district and provincial managers and DEOs directly whether the training they had received had improved their knowledge and skills, and if and how they were using the information in decision-making. Most managers (52 percent of 31 cLMIS and 100 percent of 7 vLMIS) reported that the trainings had improved their knowledge and skills relevant to their work. Similarly, of the 80 percent of DEOs who reported participating in training (78 percent of 27 cLMIS DEOs and 88 percent of 8 vLMIS DEOs), 96 percent (95 percent of 21 cLMIS DEOs and 100 percent of 7 vLMIS DEOs) reported gaining knowledge or skills relevant to their work. Data-Driven Decision-Making When asked the extent to which the LMIS had affected data-driven decision-making, 83 percent of provincial and district managers and DEOs said that the LMIS had improved data-driven decision￾making.10 In response to an open-ended follow-up question that asked them to give examples, most mentioned characteristics of the information that alluded to decision-making without providing complete descriptions. The conclusion that emerged from the responses was that timely, accurate, and accessible data (easy-to-understand graphs) facilitated close monitoring of stocks and demand; managers used the data to decide the quantities of commodities to distribute to warehouses, and to and between stores. The ability to distribute quantities consistent with demand helped prevent stock-outs and wastage related to expiration and overstocks. They also mentioned that the data allowed them to identify discrepancies between reported and actual stock and thus control pilferage. Figure 9 summarizes the most prevalent coded themes from the analysis of the qualitative data. Because many responses provided only part of the explanation, the coded data may not appear to provide strong evidence. However, the conclusion that emerged from the responses was consistent and strong. FIGURE 9: DATA-DRIVEN DECISIONS Managers and DEOs (77 respondents, 216 responses) Monitoring and supervising facilities 18% Decisions regarding resupply 17% Accuracy of reporting and forecasting 13% Monitoring and supervising facilities 12% 10 The question used a five-point Likert scale with responses of greatly improved, somewhat improved, stayed the same, somewhat worsened, and greatly worsened. The analysis aggregated greatly improved and somewhat improved into one category—improved. DELIVER LMIS: Final Evaluation Report 18 In general, DEOs use cLMIS data to determine family planning commodity requirements and distribute commodities within a district based on consumption trends and patterns. Managers, on the other hand, use the cLMIS data to plan monitoring visits, present data in coordination meetings at the district level, identify stock-outs, and make requisition requests. Ten percent of 42 LMIS provincial and district managers and 16 percent of 25 LMIS DEOs said that the LMIS had no effect on decision-making. They explained their responses by saying that they did not use c/vLMIS (6 of 8 responses), used other MISs (1 of 8 responses), and lacked sufficient staff to operate the cLMIS (1 of 8 responses). Demonstrated LMIS Skills To validate self-reported data on using the LMIS for decision-making, the evaluation team also asked provincial and district managers and DEOs to demonstrate their ability to use the LMIS. The two groups demonstrated similar levels of competency with the LMIS, except that DEOs were more likely than mangers to be able to generate reports and inventory balances (Figure 10). To some extent, this reflects the different roles of managers and DEOs, i.e., DEOs are more fluent with the online system because they enter data, while managers focus more on using the data. How Managers and DEOs Use the cLMIS and vLMIS “Since 2011, I have been using cLMIS for monthly reporting on family planning services and forecasting for quarterly requisition to the central warehouse to avoid any interruption in the supply of commodities to the district. [It makes] decision-making based on the supply, demand, and consumption pattern at district level easy.”—DEO, cLMIS “[I use the vLMIS] to generate forms and graphs which can be used for decision-making and monitoring performance of field staff and for analyzing trends of use of [family planning] commodities.”—DEO, vLMIS “[The] vLMIS dashboard view is presented in monthly video conference for reporting on vaccines consumption. We have been having regular meetings at the provincial level for discussing the progress on vaccines coverage and reporting through the vLMIS. We have been using the reports and graphs from the vLMIS for many meetings at the provincial level. It was helpful [to] know the vaccine coverage and requirements for the next period. ”— DOH manager, vLMIS “At the district level, the district officials cooperate with each other to share or redistribute stock in case of any over- or understock situation; one district gets support from other district to immediately fill that gap.”— DOH manager, cLMIS DELIVER LMIS: Final Evaluation Report 19 FIGURE 10: DEMONSTRATED SKILLS OF MANAGERS AND DEOs Summary Conclusions: Training on data entry (imparted to DEOs) has been somewhat more effective in building skills than training on how to use LMIS data for decision-making (imparted to managers). However, most managers and DEOs are using the LMIS for decision-making, although they differ in the way they access the information. The LMIS facilitated forecasting and timely decisions about stocking which more closely aligned stocks to demand and reduced wastage, particularly of vaccines. Conclusions for Question 1 The LMIS and associated trainings have improved supply chain management for family planning commodities and vaccines by improving managers’ ability to monitor and supervise district stores (and in a few cases, facilitate stock transfers between SDPs in districts), and helping DEOs better manage stock, which prevents stock-outs and reduces wastage due to overstocks. The LMIS has strengthened commodity security by improving inventory management and storage practices (cold chain and stock rotation), which helped prevent overstocking, spoilage and waste, and pilferage. The LMIS has also increased knowledge management by enhancing staff access to, understanding of, and use of data for decision-making related to supply chain management. Although the LMIS has improved supply chain management, challenges remain that affect performance. These include a lack of dedicated staff, inadequate storage facilities (cold chain), transportation issues (cold chain, reliance on private resources), and inadequate communication that delays the receipt of orders from the central warehouse. Moreover, because the vLMIS is not scaled up in all provinces, national and some provincial managers must operate multiple supply chain management systems for vaccines, which is time-consuming and inefficient. Findings for Question 2: Sustainability of cLMIS Evaluation Question: What changes could be made to ensure sustainability of the cLMIS and to strengthen data-driven decisions? The evaluation team examined factors that could impact (positively or negatively) the sustainability of cLMIS, including perceptions of usefulness, human capacity, staff turnover, availability and functionality of equipment and supplies, and coordination between departments that use the database. They also DELIVER LMIS: Final Evaluation Report 20 explored whether and how the cLMIS has improved decision-making related to supply chain management. Perceptions of Usefulness A system is more likely to be sustainable if it is useful. In key informant interviews, the evaluation team asked national, provincial, and district cLMIS managers and DOEs directly whether the cLMIS was useful enough to be sustainably used as a supply chain management tool for reporting data and/or making other supply chain management decisions. Eighty-eight percent of 33 managers and 100 percent of 27 cLMIS DEOs said they believed it was useful enough to be sustainable. In response to an open-ended follow-up question that asked them to explain their response, 89 percent of 54 responses noted that the cLMIS facilitates monitoring and supervision of facilities and stocks and enhances reporting and forecasting accuracy. Twenty-two percent of the 54 responses cited online availability and easy accessibility as reasons for sustainable use of cLMIS as a supply chain management tool. Knowledge and Skills The knowledge and skills necessary to use the cLMIS are essential to its sustainability. To ascertain the skill level of the managers and DEOs who were trained to use the cLMIS, the evaluation team conducted an observation-based skills assessment by asking respondents to perform various cLMIS-related tasks while the evaluators observed. Access to cLMIS The data are difficult to interpret and probably reflect how individuals access the cLMIS rather than their ability. For example, managers reported much lower rates of online access to the cLMIS and much higher rates of hard copy access than DEOs. However, this probably reflects the fact that managers would ask DEOs, or others, to access the cLMIS and give them a report, chart, or table. The “no access” responses are more likely to reflect actual inability to access the cLMIS. Figure 11 illustrates how managers and DEOs accessed the cLMIS. FIGURE 11: HOW MANAGERS AND DEOs ACCESS THE cLMIS DELIVER LMIS: Final Evaluation Report 21 Competencies to Use cLMIS For the managers and DEOs who were able to demonstrate online access or provide hard copies of cLMIS reports, the evaluation team conducted an observation-based skills assessment. Overall, managers and DEOs demonstrated a good understanding of how to use the cLMIS (from the system. Table 3). Consistent with their roles and findings reported above, the cLMIS DEOs showed a slightly higher capacity than managers to download tables and graphs from the cLMIS. When cLMIS DEOs were not able to download tables or graphs, the evaluation team observed that they printed the tables directly from the cLMIS. This solution, while not ideal, at least demonstrated the DEOs’ ability to obtain information from the system. TABLE 3: PROVINCIAL/DISTRICT MANAGERS’ AND DATA ENTRY OPERATORS’ KNOWLEDGE, SKILLS, AND ABILITIES TO USE cLMIS Skills cLMIS DEOs (n=24) cLMIS Managers (n=21) Knowledge about different performance reports on LMIS NA 71% Identification of opening balance from online LMIS 96% NA Ability to calculate projected commodity requirements NA 76% Ability to calculate closing balance 96% NA Understanding of “month of stock” 62% 86% Ability to download tables from online LMIS 71% 67% Ability to download graphs from online LMIS 71% 67% NA stands for not asked. Staff Turnover Staff turnover is one of the important factors that determines the sustainability of cLMIS since skills to use the LMIS may disappear with transferred staff. The evaluation findings found very low staff turnover rates in the 45-month period between September 2012 and May 2016. Fifty-six percent of 36 managers and 83 percent of 35 DEOs reported that they had been in their positions since September 2012. Those who had not been in their positions since September 2012 reported an average of 2.2 turnovers during the period, with no difference in turnover rates between managers and DEOs. Maintenance of Equipment and Supplies Maintenance is another important factor in sustainability; government offices at every level need functioning machines, internet connections, and power supplies to continue using the cLMIS after the project ends. The project supplied computers (either desktop or laptop), uninterruptable power supplies (UPSs), and printers. At each of the 35 sites they visited, the evaluation team checked the availability and functionality of equipment and supplies. All sites had a functional computer, most had a functional printer, and just over half had a UPS (Figure 12). 11 The project did not supply barcode scanners, which would have helped facilities track inventory. None of the facilities had a scanner. 11 In all but one case (i.e., the Muzaffarabad District DoH in AJK), the equipment was not missing; it was merely non-functional. DELIVER LMIS: Final Evaluation Report 22 FIGURE 12: AVAILABILITY OF FUNCTIONAL HARDWARE AT FACILITIES Ten DoH and PWD respondents from Balochistan reported that they did not receive hardware until well after training. Furthermore, three of the DoH respondents reported that they had still not received passwords to access the cLMIS. Consequently, none of the DoH respondents in Balochistan had online access to the cLMIS. Coordination Between DoH and PWD Coordination between the DoH and PWD is important to prospects for sustainability since both departments deliver services pertaining to family planning and reproductive health and are responsible for providing contraceptives to the population. The evaluation explored the current status of coordination between the PWD and DoH and found that it has significantly improved in Punjab and Sindh—the provinces on which the project focused—over the course of the project, while other regions experienced minimal improvement. In individual interviews, the evaluators asked managers directly how, if at all, cooperation, coordination, and/or collaboration between the DoH and PWD changed because of the work the DELIVER project did on the LMIS. 12 Respondents reported the greatest improvement in Punjab and Sindh and some improvement in Khyber Pakhtunkhwa (KP) and AJK (Figure 13). Overall, however, most respondents reported no change. None of the respondents said that coordination had worsened, but considering that improving coordination was an identified need, these results suggest room for improvement. 12 The question used a five-point Likert scale with responses of greatly improved, somewhat improved, stayed the same, somewhat worsened, and greatly worsened. DELIVER LMIS: Final Evaluation Report 23 FIGURE 13: COORDINATION BETWEEN DOH AND PWD A follow-up question asked the 14 respondents who reported that coordination had improved to identify, from a list of choices, the areas in which the LMIS had improved cooperation, coordination, and/or collaboration between the DoH and PWD. The 31 responses from these respondents indicated that the LMIS had improved coordination in service delivery (52 percent of responses), data reporting (42 percent of responses), generating demand (39 percent of responses), and preventing duplicating services counting (26 percent of responses). Specific examples include:  Service delivery—coordinating on conducting family health days (IUCD camps) and LHSs’ referring family planning clients to family welfare centers;  Generating demand—coordinating on conducting health melas (fairs) to promote family planning and generate demand;  Data reporting—sharing data in the district technical committee meetings; and  Duplicating services—coordinating on a joint mapping exercise to demarcate service areas and avoid duplication of services. In unstructured interviews, representatives of donors and NGOs that use cLMIS data (such as UNFPA, Marie Stopes Society, and Jhpiego) also indicated that coordination had improved. Factors Influencing Institutionalization of the cLMIS Institutionalizing the cLMIS in government ministries and departments is a key step towards potential sustainability. In an open-ended question during individual interviews, the evaluation team asked 33 national, provincial, and district managers to identify the changes or resources required in the current system to make the cLMIS more useful and sustainable. The interviewers prompted the respondents to explore sustained availability of inputs including human resources, hardware, and technical assistance for maintaining software. They also probed for evidence that the cLMIS was useful for reporting requirements, financial allocations, and other data-driven decisions on supply chain management. The factors respondents mentioned most frequently included additional technical assistance and training, additional staff (human resources) to operate the cLMIS, and hardware (Table 4). DELIVER LMIS: Final Evaluation Report 24 TABLE 4: FACTORS INFLUENCING SUSTAINABILITY OF cLMIS Factors Influencing Sustainability % of 65 Responses Technical assistance, trainings, and refresher trainings 40% Additional staff to enter data 14% Hardware 14% Electricity backup 11% Financial allocation 9% Internet connections/EVO (USB modem) 8% Inclusion of capability to track orders/deliveries 2% Consolidated provincial report in cLMIS 2% The training itself may need to become sustainable as well. In semi-structured interviews, PWD and PPHI officials in Quetta suggested training government officials as master trainers. Additionally, scaling up may influence sustainability. In unstructured interviews, 60 percent of 10 donors, implementers, NGOs, and technical experts suggested that cLMIS needs to be scaled up to the sub￾district or SDP level to ensure sustainability. Conclusions for Question 2 The cLMIS shows some promising signs of sustainability. All DEOs and almost 90 percent of managers interviewed found the cLMIS useful, and more than three-quarters indicated that they thought it had improved decision-making related to supply chain management. Moreover, coordination and collaboration between the two major stakeholders, PWD and DoH, has improved in Punjab and Sindh, where the project focused most of its effort. However, more training for managers (provincial and district) on additional ways to use cLMIS data to manage the supply chain would enhance its usefulness and prospects for sustainability. Further, but less important, additional staff to enter data and more complete and functional hardware would help promote sustainability. Findings for Question 3: Scale-up of vLMIS Evaluation question: What change could be made to increase programmatic and cost efficiencies of the vLMIS scale-up? Rather than addressing the efficiencies of the scale-up, this section addresses both support for vLMIS scale-up and how vLMIS affected programmatic and cost efficiencies. USAID/Pakistan and the evaluation team agreed to take this approach based on the available data. A financial analysis was beyond the scope of the evaluation. Consequently, findings on programmatic efficiency rely on the perceptions, and explanations, of individuals knowledgeable of the vLMIS. vLMIS Scale-up The evaluation first examined whether respondents believed the vLMIS should be scaled up and, if so, how. In response to a direct yes/no question, all eight vLMIS national, provincial, and district managers DELIVER LMIS: Final Evaluation Report 25 and all 15 DEOs said that vLMIS should be scaled up. When asked in an open-ended question about how they believed the vLMIS should be scaled up, five of eight managers said that vLMIS should be scaled up to all districts, while the remaining three suggested that the scale-up should be to the sub-district (i.e., facility and tehsil) level. The evaluation team also asked the eight managers an open-ended question about the types of support that would be necessary for scale-up. Three of seven respondents mentioned the need for hardware support, and the other four each mentioned one of the following: adequate human resources, proper planning, provision of administrative rights to the system, and capacity building. The team also posed an open-ended question to the eight managers about the constraints to vLMIS scale-up. Three of nine respondents suggested that budgetary constraints posed challenges for scale-up, and three mentioned a shortage of dedicated staff at the provincial and district levels. Additional constraints, each mentioned by one individual, included a lack of political will and commitment, partners’ lack of support for the government, and internet and connectivity issues. Programmatic Efficiency Managers expressed generally positive views about the effect of the vLMIS on programmatic efficiency. When asked to identify the extent to which the vLMIS affected programmatic efficiency, seven of nine national, provincial, and district managers said that vLMIS improved programmatic efficiency, including five who said it somewhat improved efficiency, and two who said it greatly improved efficiency. The remaining two respondents said that programmatic efficiency stayed the same. When the evaluation team asked the managers an open-ended follow-up question about reasons for improvement, three of eight responses noted that the vLMIS facilitated timely reporting, and two responses identified reduction of vaccine wastage and timely distribution of vaccines to district stores. Managers were less positive about the effect of vLMIS on the cost of managing the vaccine supply chain. In response to a multiple choice, single-response question, five of nine national, provincial, and district managers said that vLMIS had no effect on the cost of managing the vaccine supply chain. In response to an open-ended follow-up question, the provincial focal person for vLMIS in KP (where vLMIS is implemented in only 5 of 26 districts) noted that the lack of implementation in all districts was the reason that vLMIS did not have any effect on costs. Still, three of nine managers indicated that online reporting, data management, and accurate inventory tracking decreased the cost of managing the vaccine supply chain. vLMIS Reporting Reporting rates13 for all vaccines have increased across Pakistan since 2013. By 2016, overall reporting rates were above 80 percent for all districts of Sindh and selected districts of Punjab where vLMIS was implemented. Annex 8 presents data on reporting rates for BCG, Pentavalent, and measles vaccines. Vaccine Coverage and Wastage One objective of the vLMIS was to improve the efficiency of the logistics management system by increasing vaccine coverage and reducing wastage. Examination of the vLMIS data indicates that the vLMIS contributed to these goals. Measles vaccine coverage, for example, showed a slight upward trend from 2013 to 2016 (Figure 14). Data on measles wastage is mixed but shows substantial improvements since the vLMIS scale-up in May 2015, demonstrating the ability of vLMIS to identify instances of vaccine wastage and facilitate corrective action in a timely manner. Coverage has also increased in the project 13 The reporting rate is the ratio of facilities that report on vLMIS to the total number of facilities, expressed as a percentage. DELIVER LMIS: Final Evaluation Report 26 focus districts of Punjab and Sindh relative to other areas. The qualitative data presented previously in this report also demonstrated the importance of improved inventory tracking, stock rotation practices, and cold chain equipment to reducing wastage. FIGURE 14: MEASLES COVERAGE AND WASTAGE Conclusions for Question 3 DEOs and managers overwhelmingly support scaling up vLMIS. All the DEOs and managers interviewed supported scaling up vLMIS to all districts. Scaling up to additional districts should take priority over expanding the database to the sub-district level since getting all districts on the same system will improve efficiency at the provincial and national levels. However, effective scale-up will require more investment in infrastructure and human capital, especially in KP, where the vLMIS was implemented in only five districts, and Balochistan, where it was implemented in nine. DELIVER LMIS: Final Evaluation Report 27 In general, vLMIS has helped to improve programmatic efficiencies. It successfully contributed to ensuring the availability of vaccines, reducing vaccine wastage and pilferage, and improving coverage and reporting rates for vaccinations in targeted districts. The availability of information has improved substantially, and the current programmatic approach is efficient in achieving the intended results. However, vLMIS has not yet improved cost efficiencies across all areas where it is implemented. This is because unless all districts within a province implement vLMIS, government agencies still must use the older systems of tracking and managing vaccine supplies to meet the needs of the districts that have not implemented vLMIS. Findings for Question 4: Best Practices Evaluation question: What best practices, innovations, and lessons learned can be applied to future programming in supply chain systems strengthening? To answer this question, the evaluation team asked provincial and district managers for examples of promising or proven practices that the DELIVER project has developed or strengthened in Pakistan. The interviewers prompted the respondents for practices in tracking inventory, data management, procurement, warehousing, distribution, service delivery, and utilization of health commodities, as well as supply chain management and human resource management in supply chain systems. Respondents identified the following best practices:  Use of innovative, web-based technology rather than the old manual systems of reporting and recording. Seventy-six percent of 41 managers considered the LMIS itself a best practice that needs to be sustained and scaled up.  Best warehouse practices. Seventy-three percent of 41 managers mentioned best warehousing practices including systematic tracking, cold chain management, and FEFO stock management practices.  Online requisition of commodities. Fifty-one percent of 41 managers said that automated, formula-based quantification for ordering vaccines and family planning commodities is a best practice that needs to be taken forward.  Use of LMIS for monitoring. Twenty-seven percent of 41 managers said that use of the LMIS for monitoring is a practice that should be taken forward.  Consumption-based distribution of commodities instead of the earlier practice of quota-based distribution to provinces, districts, and SDPs. Twenty-two percent of 41 managers believed that using information on consumption (as measured by distribution in the cLMIS) should replace the previous practice of providing a fixed quantity of commodities regardless of demand or stock situation. Twenty-seven percent suggested that the distribution of commodities across SDPs within a district should be based on consumption.  Provincial budgeting and procurement of contraceptives. Twenty-five percent of 12 provincial cLMIS managers considered the provincial procurement of family planning commodities a best practice. In addition to the best practices that the respondents identified, the evaluation team observed that active and meaningful government involvement is key to sustainability, since government agencies are the primary users of the database and will determine if and how to use it going forward. DELIVER LMIS: Final Evaluation Report 28 Cross-Cutting Findings: Gender Overall, 62 percent of the study respondents were men and 38 percent were women. Respondents who were directly related to LMIS (i.e., national, provincial, and district managers and DEOs) were mostly men: only 10 (or 13 percent) of 76 respondents were women. The national, provincial, and district governments do not have any gender preference for these positions; however, women in Pakistan have fewer opportunities for education and jobs, and governments might consider ensuring equitable rather than equal opportunities to women. OVERARCHING CONCLUSIONS DELIVER has improved supply chain performance: The cLMIS and vLMIS appear to have improved the performance of their respective supply chains, particularly in Punjab and Sindh, the provinces on which the project focused its support. Indicators of supply chain operation and performance—including reporting rates, consumption of family planning commodities, vaccine coverage, and vaccine wastage—have moved in the right direction in these two provinces. Although it is difficult to attribute the quantitative results to DELIVER interventions, the qualitative evidence provides a level of plausible attribution. Managers and DEOs explained that better record keeping improved the timeliness and accuracy of data on stocks and consumption. Better information allowed managers to align stocks more closely with demand, which reduced stock-outs (improving commodity security) and overstocks (reducing wastage). Better record keeping also improved transparency and reduced pilferage. For vaccines in particular, better stock rotation practices (FEFO and FIFO) and the cold chain reduced wastage. Managers and DEOs for the cLMIS mentioned that they used stock and consumption information to base resupply requests on inventory and demand instead of on the fixed quotas as they had in the past. DELIVER has facilitated data-driven decision-making: Although few managers and DEOs described specific decisions for which they relied on LMIS data, the most common explanation of how the LMIS had improved supply chains rested on using more timely and accurate inventory data to align resupply orders with demand. This is an important decision-making function which has substantially improved many aspects of supply chain performance. Sustainability of cLMIS: Managers and DEOs have learned and are applying new skills, and staff turnover has been low; these are good signs for the will and skills to sustain the system. Prospects for sustainability are probably higher in Punjab and Sindh, where the project focused its support. These provinces showed substantial improvement in supply chain performance over the life of the project; this improves prospects for sustainability based on perceived usefulness and implies some level of institutionalization. Collaboration between DoH and PWD has also improved in these provinces. However, even in Punjab and Sindh, many indicators of performance have started to decline since the project started scaling back implementation in September 2015, a trend that calls into question the sustainability of results. The interviews suggest that managers’ and DOEs’ limited decision-making and data interpretation skills may constrain sustainability. Additional training for managers (provincial and district) on using LMIS data, training for master trainers, dedicated staff to enter data, and more complete and functional hardware and internet access may help promote sustainability. Training is particularly important in the context of the devolution of responsibility to the provinces, where capacities to manage the supply chain may be lower than at the national level. Scale-up of vLMIS: The vLMIS has proven useful and effective, and this has garnered substantial support for scaling it up to the district level, and perhaps the sub-district level. However, scaling up will DELIVER LMIS: Final Evaluation Report 29 require additional investment in infrastructure and human capital, especially in KP, where the vLMIS was implemented in only five districts, and Balochistan, where it was implemented in nine. Although the evidence is thin, the scale-up to date appears to have been relatively low-cost. Best practices: Respondents identified the LMIS itself to be a best supply chain management practice, as it not only shifted the traditional manual reporting system to convenient and timely online reporting, but at the same time introduced the much needed LMIS for vaccines and contraceptives. The warehousing practices promoted by the project—e.g., FEFO, FIFO, tracking expiry dates, and more closely aligning inventory to demand—are also best practices in supply chain management. RECOMMENDATIONS Based on the findings, the evaluation team proposed the following key recommendations:  Future projects should explore the possibility of integrating the multiple vaccine and commodity MISs maintained at the provincial and national levels to improve the efficiency of supply chain management and the potential sustainability of the overall system. One example would be integrating the cLMIS with the LHW-MIS to reduce LHSs’ burden of reporting into two separate systems. Collaborating with other donors that support various alternative MISs may be advisable.  To promote cLMIS sustainability and vLMIS programmatic efficiency, future projects should continue to train managers and DEOs on how to access and use the LMIS. Supportive supervision and post-training follow-up visits may be effective methods. Creating a pool of master trainers within relevant government institutions may also be an effective approach to sustaining the skills of managers and DEOs. It is particularly important to train managers (provincial and district) on additional ways to use LMIS data for decision-making to more effectively manage the supply chain.  Future USAID/Pakistan programming should consider scaling up the vLMIS to all districts of Pakistan with priority given to KP and Balochistan.  If future projects support scaling up LMIS at the SDP level, they will need to address internet connectivity and hardware deficiencies, especially the need for barcode scanners.  To support prospects for sustainability and scale-up, future projects should consider advocating with the government and potential donors to institutionalize the LMIS, improve collaboration between DoH and PWD, and consolidate around a single LMIS. In the context of the 18th Amendment, advocacy may also be required to allocate the budgets at the provincial level necessary to provide adequate storage, cold chain facilities, and transportation. DELIVER LMIS: Final Evaluation Report 30 LESSONS LEARNED  Select appropriate/relevant personnel for training; for example, DEOs should have at least some basic understanding of how to operate computer-based systems.  Train multiple individuals to enter data at the district level. The project trained only one DEO in each district, a limitation that may disrupt LMIS operation if the DEO is absent due to leave, emergency, transfer, or retirement.  Follow-up supportive supervision of the trainings is essential for ensuring sustainability and refreshing or enhancing skills.  Necessary hardware and access to LMIS (internet connectivity and username/password) should be provided immediately after trainings so trainees can practice and cement skills while lessons are fresh. DELIVER LMIS: Final Evaluation Report 31 ANNEXES Annex 1: Evaluation Scope of Work DELIVER LMIS: Final Evaluation Report 32 DELIVER LMIS: Final Evaluation Report 33 DELIVER LMIS: Final Evaluation Report 34 DELIVER LMIS: Final Evaluation Report 35 Annex 2: Assignment Work Plan DELIVER LMIS: Final Evaluation Report 36 DELIVER LMIS: Final Evaluation Report 37 DELIVER LMIS: Final Evaluation Report 38 DELIVER LMIS: Final Evaluation Report 39 DELIVER LMIS: Final Evaluation Report 40 DELIVER LMIS: Final Evaluation Report 41 DELIVER LMIS: Final Evaluation Report 42 DELIVER LMIS: Final Evaluation Report 43 DELIVER LMIS: Final Evaluation Report 44 DELIVER LMIS: Final Evaluation Report 45 DELIVER LMIS: Final Evaluation Report 46 DELIVER LMIS: Final Evaluation Report 47 DELIVER LMIS: Final Evaluation Report 48 DELIVER LMIS: Final Evaluation Report 49 DELIVER LMIS: Final Evaluation Report 50 DELIVER LMIS: Final Evaluation Report 51 DELIVER LMIS: Final Evaluation Report 52 DELIVER LMIS: Final Evaluation Report 53 Annex 3: Data Collection Instruments Provincial and District Data Operators INSTRUMENT FOR DELIVER LMIS EVALUATION: For use at provincial and district levels with LMIS data entry operators انٹرویو کے کوائف : :Interview of Demographics تاریخ (دن-ماہ-سال( : ________________________________________:(Day-Month-Year (Date انٹرویو لینے والے کا نام: _______________________________________ :name s’Interviewer نوٹس لینے والے کا نام: _______________________________________ :name s’taker Note انٹرویو لینےکی جگہ/مقام _______________________________________ :location Interview صوبہ: ___________________________________________ :Province ضلع: ____________________________________________ :District مرکز صحت: _________________________________ :facility Health انٹرویو دینے والے کا نام: _____________________________________________ :name Interviewee انٹرویو دینے والے کے ادارے کا نام: ________________________________ :organization Interviewee انٹرویو دینے والے کا عہدہ: _____________________________________________ :title Interviewee Interviewee’s phone number: _________________________________ :تفصیالت کی رابطے Introduction: My name is ______________________. I work for a research organization called Management Systems International (MSI) which is based in Islamabad. As explained in the official letters from the Department of Health (DoH), the Population Welfare Department (PWD), and MSI, we are conducting a final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan (GOP) continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain for health commodities (family planning commodities and vaccines). We will ask a few questions related to: 1. The effectiveness of LMIS for managing the supply chain of medicines, such as family planning commodities and vaccines. 2. The sustainability of the Logistics Management Information System for contraceptives (cLMIS) and strengthening of data-driven decisions on supply chain management. 3. The programmatic and cost-efficiencies of scaling up the vaccines’ LMIS (vLMIS). 4. Best practices, innovations, and lessons learned in supply chain management, cLMIS, and vLMIS. This interview will take approximately 45minutes to 1 hour. We will treat the information we collect as confidential and will never associate it with your name. میرا نام ۔۔۔۔۔۔۔۔۔۔۔۔۔ ہے، میں اسالم آباد میں قائم ایک تحقیقی ادارے (MSI (International systems Managementکے ساتھ کام کر رہا ہوں۔ جیسے کے اجازت نامے میں بتایا گیا ہے کہ ہم JSI کی طرف سے کیے جانے والے project Deliverکے LMIS کے حوالے سے تحقیق کر رہے ہیں۔ اس تحقیق کے نتائج سے حاصل شدہ معلومات حکوم ِت پاکستان کو فیملی پالننگ کے اشیاء کی عوام تک ترسیل کے طریقہ کار کو مضبوط اور بہتر بنانے اور اس کے نتیجے میں ، زچہ اور بچہ کی صحت کی بہتر ی کے لیے مددگار ثابت ہونگی۔ میں آپ سے نیچے بیان کردہ نکات کے متعلق کچھ سوال کرنا چاہونگا۔ .1 ویکسین کی ترسیل اور انتظامی امور پر LMISکے اثر ات ۔ .2 cLMIS سے حاصل کردہ معلومات کی بنیاد پر فیملی پالننگ کے اشیاء کی ترسیل کے نظام کو باقی عالقوں میں پھیالنا ۔ 3 ان اشیاء کی ترسیل سے متعلق فیصلہ سازی میں معاونت۔ .4cLMISاور vLMISکے استعمال کے دوران management chain supplyمیں بہترین عمل، جدت اور حاصل شدہ اسباق کے متعلق جاننا۔ DELIVER LMIS: Final Evaluation Report 54 یہ انٹرویو تقریبا 45 منٹ سے 01 گھنٹے تک جاری رہےگا۔ اس سے حاصل کردہ معلومات مکمل طور پر صیضۂ راز میں رکھا جائیگا اور یہ معلومات کہیں بھی آپ کے نام سے منسوب نہیں کیا جائیگا۔ May I have your permission to proceed with the interview? کیا مجھے انٹرویو کا آغاز کرنے کی اجازت ہے؟ Yes No (STOP THE INTERVIEW) May I have your permission to record the interview to ensure the completeness and accuracy of your opinions? ا قیمتی معلومات سے مکمل طور پ کا انٹرویو ٓپ سے حاصل کردہ پر مستفید ہونے کے لیئے ہم ا ریکارڈکرنا چاہتے ہیں۔ کیا ہمیں ٓ انٹرویو ریکارڈ کرنے کی اجازت ہے؟ Yes (SWITCH ON THE RECORDER AND START THE INTERVIEW) No (START THE INTERVIEW) INSTRUMENT FOR PROVINCIAL AND DISTRICT LMIS DATA ENTRY OPERATORS Detailed questions: Questions for all interviewees – the interviewer will explain to the interviewee that “I will be asking you some questions about the following issues, and I hope that you will provide your opinions based on your experience during the implementation of the DELIVER project.” یہ وضاحت کرے گا کے "میں ا پ مجھ ٓ انٹرویور جواب دہندہ کو پ سے مندرجہ ذیل امور کے متعلق کچھ سوال کرونگا اور مجھے امید ہے کے آ سے ڈلیور پروجیکٹ کے عملدرآمد کے دوران ہونے والے تجربات کی بنیاد پر اپنے راۓ سے آگاہ کرینگے۔" Increase knowledge management and dissemination 1. Have you received any training from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا آپ / JSI DELIVER کی طرف سے کوئی تربیت حاصل کر ُچکے ہیں؟ ہاں، تاریخ بتائیے (_________________________ :date approximate (Yes No (GO TO Q17) نہیں 2. On which of the following topics, if any, did you receive training? (CIRCLE ALL THAT APPLY) مندرجہ ذیل موضوعات میں سے کس پر آپ نے تربیت حاصل کی ہوئی ہے؟ 1 Principles of supply chain management ترسیل کے نظام کے بنیادی اصول 2 Purpose of the cLMIS in supply chain management فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق cLMISکے مقاصد 3 Purpose of the vLMIS in supply chain management: ویکسین کی ترسیل کے نظام کے متعلق vLMIS کے مقاصد 4 cLMIS Training-of-Trainers cLMIS پر تربیت دینے والوں کی تربیت 5 vLMIS Training-of-Trainers vLMIS پر تربیت دینے والوں کی تربیت DELIVER LMIS: Final Evaluation Report 55 6 Usage of the cLMIS for entering data اعداد و شمار کے اندراج کے لیئے cLMIS کا استعمال 7 Usage of the vLMIS for entering data اعداد و شمار کے اندراج کے لیئے vLMIS کا استعمال 8 Usage of cLMIS for supply chain management decision-making: فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی میں cLMIS کا استعمال IF 8 IS CIRCLED, PROBE REGARDING DECISION-MAKING IN SUPPLY CHAIN MANAGEMENT FOR FAMILY PLANNING SERVICES—CAN THE INTERVIEWEE DESCRIBE AN EXAMPLE OF HOW S/HE HAS USED CLMIS DATA? 9 Usage of vLMIS for supply chain management decision-making: فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی میں vLMIS کا استعمال IF 9 IS CIRCLED, PROBE REGARDING DECISION-MAKING IN SUPPLY CHAIN MANAGEMENT FOR FAMILY PLANNING SERVICES—CAN THE INTERVIEWEE DESCRIBE AN EXAMPLE OF HOW S/HE HAS USED VLMIS DATA? 10 Maintenance of the cLMIS cLMIS کو بحال رکھنے کی تربیت 11 Maintenance of the vLMIS Yes/No نہیں/ہاں vLMIS کو بحال رکھنے کی تربیت 12 Other cLMIS training (describe below): Yes/No نہیں/ہاں cLMIS کے متعلق کوئی اور تربیت (PROBE ON WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE cLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM) Describe: ______________________________________________________ ______________________________________________________ 13 Other vLMIS training (describe below) Yes/No نہیں/ ہاں vLMIS کے متعلق کوئی اور تربیت (PROBE ON WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE vLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM) Describe: DELIVER LMIS: Final Evaluation Report 56 14 Other supply chain management training (describe below) Yes/No نہیں/ ہاں فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق کوئی اور تربیت Describe: 3. Did the training that you received provide you with new information? (CIRCLE ONE NUMBER) حاصل کردہ تربیت سے آپ کو کوئی نئی معلومات حاصل ہوئیں؟ ہاں Yes 1 نہیں No 2 4. Did you find the training received to be useful in your work? (CIRCLE ONE NUMBER) کیا حاصل کردہ تربیت آپ کے روز مرہ کے کام میں مددگار ثابت ہوئی؟ ہاں Yes 1 نہیں No 2 5. Did you gain some knowledge or skills relevant to your work? (CIRCLE ONE NUMBER. IF “YES”, PROBE ON LIST OF TOPICS IN Q2) کیا تربیت نے آپ کے روزمرہ کے کام کے متعلق معلومات اور مہارت حاصل کرنے میں معاونت کی؟ 1 Yes (WHICH OF Q2 TOPICS): ہاں _________________________________________________________________ _________________________________________________________________ 2 No نہیں) GO TO Q8) 6. Please give me an example of how you applied this new knowledge or new skills in your work. (WRITE AN ANSWER) برائے مہربانی ایک ایسی مثال بیاں کریں کے آپ نے تربیت سے حاصل کردہ معلومات اور مہارت کو اپنے روزمرہ کے کام میں استعمال کیا؟ 7. In your experience, which training topics, if any, contributed the most to enhancing your skills at work? (WRITE AN ANSWER) ا روزمرہ کے کام میں مہارت کو بڑھانے میں معاون ثابت ہوا ؟ ٓپ کے خیال میں تربیت کا وہ کون سا اہم پہلو تھا جو آپ کے DELIVER LMIS: Final Evaluation Report 57 8. What kinds of testing or assessment, if any, did you receive during the training? (CIRCLE تربیت کے دوران ا لیئے گئے ٓپ سے کس قسم کے امتحانات (NUMBER ONE 1 Pre-training only 2 Post-training only 3 Both pre-training and post-training 4 None 9. Following the training, have there been any follow up visits at your workplace to assess the usefulness of the trainings for the performance of your job? (CIRCLE ONE NUMBER) تربیت کے بعد یہ جانچنے کے لیئے کہ تربیت ا معاونت کے لیئے کتنی حد پ کے ٓپ کے روز مرہ کے کام میں تک ُمفید رہی ہے۔ کیا آ کام کرنے کی جگہ پر کسی نے معائنے کے لیئے دورہ کیا؟ ہاں Yes 1. 2. No نہیں) GO TO Q13) 10. If YES, who did the follow-up visit after the training? (CIRCLE ONE NUMBER) اگر ہاں تو تربیت کے بعد معائنے کے لیئے کس نے دورہ کیا؟ 1 JSI DELIVER team 2 Government officials 3 Others (specify) _______________________________________ 11. During the follow up visit, were you asked how the trainings improved your skills? (CIRCLE ONE NUMBER) تربیت کے بعد معائنے کے لیئے کیے جانے والے دوروں میں کیا آپ سے پوچھا گیا کے تربیت کس حد تک آپ کے روز مرہ کے کام میں مہارت کو بڑھانے کے لیئے معاون ثابت ہوئی ہے؟ ہاں Yes 1. 2. No نہیں) GO TO Q13) 12. If yes, how did you respond? (PROBE ON: HOW THE TRAINEE’S KNOWLEDGE IMPROVED, HOW SKILLS IMPROVED, WHAT WAS MISSING FROM THE TRAINING, AND HOW THE TRAININGS CAN BE IMPROVED IN THE FUTURE.) (WRITE AN ANSWER) اِس سوال پر آ اگر ہاں تو پ کا کیا جواب تھا؟ 13. In your opinion, how could the training be improved? (PROBE: CONTENT OF CURRICULUM, TRAINING METHOD, AND FOLLOW-UP—IS THERE A NEED FOR A “REFRESHER COURSE”; CAN THE LMIS OPERATOR TRAIN HIS/HER SUCCESSOR; WHO IS CURRENTLY PROVIDING THE IT TECHNICAL SUPPORT TO cLMIS / vLMIS; HOW FREQUENTLY DOES THE LMIS OPERATOR SEEK SUPPORT; IS HE/SHE SATISFIED WITH THE IT TECHNICAL SUPPORT THAT HE/SHE IS CURRENTLY RECEIVING?) (WRITE AN ANSWER) DELIVER LMIS: Final Evaluation Report 58 آپ کے خیال میں تربیت میں کس طرح کی بہتری الئی جاسکتی ہے؟ )نصاب کے مواد، ٹریننگ کا طریقہ کار، ریفریشر کورس کی ضرورت کے بارے میں جانئیے۔ مزید یہ کہ کیاایل ایم آئی ایس آپریٹر دوسروں کو تربیت دے سکتا/سکتی ہے؟ سی ایل ایم آئی ایس اور وی ایل ایم آئی ایس میں آئی ٹی سے متعلق معاونت کون کرتا ہے؟ آپریٹر عموعا کب معاونت حاصل کرتا ہے؟ کیا وہ اس معاونت سے مطمئن ہوتا/ہوتی ہے؟ ( 14. How did you think the trainer / facilitator was in terms of the characteristics below? (WRITE AN ANSWER BY ASKING ABOUT EACH TYPE OF TRAINING TAKEN AND WRITING A SCORE ABOUT THE TRAINER’S CHARACTERISTICS—WAS THE TRAINER (1) VERY GOOD, (2) MODERATELY GOOD, (3) VERY BAD, (4) MODERATELY BAD, OR (5) THE RESONDENT HAS NO OPINION) مندرجہ ذیل پر آپ نے تربیت دینے والے کو کیسا پایا؟ )ہر حاصل کردہ ٹریننگ میں ٹرینر کے خصوصیات سےمتعلق پوچھ کر اسکور لکھیں: کیا ٹرینر )1( بہت اچھا، )2( کچھ حد تک اچھا، )3( بہت برا، )4( کچھ حد تک برا، )5( کوئی رائے نہیں( Characteristics of Trainer /Facilitator and Curriculum تربیت دینے والے / سہولت کار اور مواد کی خصوصیات 1) Trainer well prepared تربیت دینے والے کی تیاری 2) Time management وقت کی پابندی 3) Methodology used (brainstorming, group discussion, and audio-visual aids) استعمال کیاجانے واال طریقہ کار 4) Use of training aids (eg, hand-outs) and technology آ تربیت کے لیئے ٹیکنالوجی اور دوسرے مددگار الت کا استعمال 5) Knowledge of the subject موضوع کے متعلق معلومات 6) Content easily understood مواد کا آسان فہم ہونا 7) New concept(s) introduced نئے تصورات کا تعارف 15. Do you think that the training on the use of the LMIS could be scaled up? کیا آپ کے خیال میں LMIS کے استعمال سے متعلق تربیت کو بڑھایا جا سکتا ہے؟ ہاں Yes 1. نہیں No 2. DELIVER LMIS: Final Evaluation Report 59 Why or why not? (WRITE AN ANSWER—PROBE ON QUALITY OF TRAINING CONTENT, METHODS, FOLLOWUP MENTORING AND SUPPORTIVE SUPERVISION, GEOGRAPHIC COVERAGE, AND SCALING UP) ہاں اور نہ دونوں جواب کی صورت میں وضاحت کیجئے۔ 16. Is there anything you would like to add regarding scaling up or improving the trainings? تربیت کو اور زیادہ بڑھانے اور بہتری النے کے لیئے ا اور کہنا چاہینگے؟ ٓپ کچھ 17. In your opinion, can a LMIS like the one developed by DELIVER improve decision-making related to supply chain management? (CIRCLE ONE NUMBER) ا DELIVERکے تیار کردہ LMIS جیسا پروگرام فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی ٓپ کی نظر میں کیا کو بہتر کرنے میں مددگار ثابت ہو سکتا ہے؟ ہاں Yes 1 نہیں No 2 Why or why not? (PROBE ON THE DATA ENTRY OPERATOR’S EXPERIENCE – ARE ACTIONS TAKEN BY DECISIONMAKERS CONSISTENT WITH THE DATA S/HE CAN GENERATE FROM THE DASHBOARD) ہاں اور نہیں دونوں جواب کی صورت میں وضاحت کیجئے 18. In your experience, to what extent, if any, has the LMIS improved decision-making related to supply chain management? Would you say it has greatly improved, somewhat improved, not changed, somewhat worsened, or greatly worsened decision-making? (CIRCLE ONE NUMBER) ا LMIS فیملی پالننگ اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی کو بہتر کرنے میں کس حد تک ٓپ کے تجربات کی بنیاد میں مددگار ثابت ہوا ہے؟ 1 Greatly improved....................................(GO TO Q19) ہے ئی بہت بہتر تبدیلی آ 2 Somewhat improved..............................(GO TO Q19) ہے ئی کسی حد تک بہتر ی آ 3 Stayed the same ......................................(GO TO Q19) ئی کوئی بہتر ی یا خرابی نہیں آ 4 Somewhat worsened .............................(GO TO Q19) آئی خرابی 5 Greatly worsened...................................(GO TO Q19) آئی خرابی بہت 6 Don’t know .............................................(GO TO Q20)نہیں معلوم DELIVER LMIS: Final Evaluation Report 60 7 Refused to answer .................................(GO TO Q20)انکار سے دینے جواب 19. Why do you think so? Please use examples to support your response? (WRITE AN ANSWER) آپ کے ایسا سوچنے کی کیا وجہ ہے؟ برائے مہربانی کوئی مثا ل دے کر وضاحت کیجئے۔ Evaluation Question 2: “What are the changes that could be made to ensure sustainability of the cLMIS and to strengthen data-driven decisions?” cLMIS سسٹم سے حاصل شدہ اعداد و شمار کی بنیاد پر لیئے جانے والے فیصلوں کی پائیداری کے لیئے کن تبدیلیوں کی ضرورت ہے۔ 20. Please tell me your job title, the date of your appointment to LMIS data management, and describe your responsibility for using the cLMIS. برائے مہربانی مجھے اپنے عہدہ کا نام، LMIS کے متعلق ذمہ داریوں پر تقرری اور سی ایل ایم آئی ایس کے متعلق آپ کی ذمہ داریوں کے متعلق آگاہ کیجئے۔ (WRITE DATE OF APPOINTMENT AND DURATION IN COMPLETED YEARS AND MONTHS—ASK FOR A DETAILED DESCRIPTION OF RESPONSIBILITIES REGARDING USE OF THE LMIS: E.G., WAS IT FOR DATA ENTRY? DATA ANALYSIS? DATA INTERPRETATION? DATA REPORTING? SYSTEM MANAGEMENT? SYSTEM MAINTENANCE?) عہدہ کا نام title Job ذ مہ داریاں Responsibilities Is the position dedicated to the LMIS? کیا یہ پوزیشن ایل ایم آء ایس کے لئے وقف ہے؟ ہاں Yes 1. نہیں No 2. Date of appointment YYYY-MM-DD اس عہدہ پر تقرری کی تاریخ Duration appointed in this position (Years) (Months) اس عہدہ پر تقرری کی مدت 21. For the position you are holding currently, do you know how many transfers or postings have taken place in the period between September 2012–May 2016? (ENTER ONE NUMBER AND, IF YES, THE NUMBER OF INDIVIDUALS IN THE POST) جس عہدے پر آپ ابھی فائز ہیں، کیا آپ جانتے ہیں کہ ستمبر 2012 سے مئی 2016 کے دوران کتنی تقرریاں اور تبادلے ہوچکے ہیں؟ Yes: _____ ہاں) NOTE NUMBER OF TRANSFERS/POSTINGS) نہیں No Number of transfers postings DELIVER LMIS: Final Evaluation Report 61 ASK QUESTIONS ONLY OF A DATA ENTRY OPERATOR WHO IS RESPONSIBLE FOR USING THE cLMIS FOR GENERATING REPORTS AND/OR FOR DECISION-MAKING ON SUPPLY CHAIN MANAGEMENT: 22. Please share with us the last monthly cLMIS report. (PREFERABLY BY USING THE ONLINE cLMIS SYSTEM. IF NOT AVAILABLE, OR IF THE RESPONDENT DOES NOT KNOW HOW TO DO SO ONLINE, OR IF THE ONLINE SYSTEM IS NOT OPERATING, THEN ASK FOR A COMPUTER-GENERATED HARD COPY) (CIRCLE ONE NUMBER) ہمارے ساتھ گزشتہ ماہانہ cLMIS رپورٹ شیئر کریں . 1. Yes—online access 2. Yes—hard copy access 3. No نہیں) EXPLAIN WHY NOT POSSIBLE AND GO TO Q28) (OBSERVE THE FOLLOWING AND SCORE RESPONDENT’S UNDERSTANDING, KNOWLEDGE, SKILLS, AND ABILITIES) Questions for data entry operators معلومات کا اندراج کرنے والے کے لئے سواالت Response (ENTER ONE NUMBER IN EACH ROW) Incorrect Partially Correct Correct 23. Kindly inform us about the opening balance of the condom supply from the last monthly report. برائے مہربانی مجھے گذشتہ مہینے کے کونڈم سپالئے کی اوپننگ بیلنس سے آگاہ کیجئے۔ 0 1 2 24. Please tell us how the closing balance is calculated at the end of each month. برائے مہربانی ہمیں بتائیے کہ مہینے کے آخر میں کلوزنگ بیلنس کے اعداد و شمار کس طرح اکٹھے کیئے جاتے ہیں۔ 0 1 2 25. Please tell us what is meant by “months of stock.” منتھس آف اسٹاک سے کیا مراد ہے، برائے مہربانی بیان کیجئے 0 1 2 26. Please show us how to download data from the online cLMIS. برائے مہربانی مجھے دکھائیے کے آن الئن cLMISسے کس طرح اعداد و شمار ڈاؤن لوڈ کیئے جاتے ہیں 0 1 2 27. Please show us how graphs can be obtained from the online cLMIS. برائے مہربانی مجھے دکھائیے کے آن الئن cLMISگرافس کیسے حاصل کیئے جاتے ہیں۔ 0 1 2 DELIVER LMIS: Final Evaluation Report 62 28. Do you strongly agree, agree, disagree, strongly disagree, or have no opinion about the following statement: “cLMIS is essential for appropriate data-driven decisions about supply chain management?” (CIRCLE ONE NUMBER) فیملی پالننگ اشیاء کی ترسیل کے نظام کے بارے میں حاصل شدہ اعداد و شمار کے متعلق فیصلوں کے لیئے cLMISبہت الزمی/ضروری ہے۔ آپ اس بیان پر کیا رائے رکھتے ہیں؟ 1 Strongly agree .............(GO TO Q29) 2 Agree ............................(GO TO Q29) 3 No opinion ...................(GO TO Q29) 4 Disagree ........................(GO TO Q29) 5 Strongly disagree.........(GO TO Q29) 6 Refuse ...........................(GO TO Q30) 29. Why do you think so? Please describe an example to support your response. آپ ایسا کیوں سوچتے ہیں؟ برائے مہربانی اپنی اس راۓ کی تائید میں کوئی مثال دیں۔ 30. In your experience, do you think the cLMIS is useful enough to be sustainably used as a supply chain management tool for reporting data and/or making other supply chain management decisions? (CIRCLE ONE NUMBER) فیملی پالننگ اشیاء کی ضرورت ذخیرہ کے تخمینے کے لیئے آپ cLMISسے حاصل شدہ کون سے اعداد و شمار استعمال کرتے ہیں؟ ہاں Yes 1. نہیں ... No 2. 31. Why, in your experience, is cLMIS useful or not useful? (WRITE AN ANSWER) ِل آپ کے خیال میں cLMIS استعمال ہے یا نہیں ہے؟ کس وجہ سے قاب 32. In your experience, how can the current cLMIS system be made sustainable? (EXPLORE WITH REGARDS TO SUSTAINED AVAILABILITY OF INPUTS INCLUDING HUMAN RESOURCES, HARDWARE, TECHNICAL ASSISTANCE FOR MAINTAINENCE OF SOFTWARE, EVIDENCE THAT THE cLMIS IS USEFUL FOR REPORTING REQUIREMENTS, FOR FINANCIAL ALLOCATIONS, AND FOR OTHER DATA-DRIVEN DECISIONS ON MANAGEMENT OF THE SUPPLY CHAIN FOR CONTRACEPTIVE COMMODITIES) آپ کے خیال میں، موجودہ cLMISسسٹم کو کس طرح مزید کار آمد اور پائیدار بنایا جا سکتا ہے ؟ DELIVER LMIS: Final Evaluation Report 63 کیا آپ کے پاس الیکٹرانک ایل ایم آء ایس سسٹم موجود ہے؟ ?use in LMIS electronic an there Is 33. ہاں Yes 1 نہیں No 2 Why or why not? (WRITE AN ANSWER) ہاں اور نہیں دونوں جواب کی صورت میں وضاحت کیجئے 34. Is eLMIS fully functional? کیا الیکٹرانک سی ایل ایم آء ایس سسٹم فعال ہے؟ ہاں Yes 1. نہیں No 2. Comments: i. Computer exists ii. Availability of power and internet connection iii. Software installed and stable iv. Warehouse personnel trained and using software 35. Since when has the eLMIS been operational? الیکٹرانک سی ایل ایم آء ایس سسٹم کتنے عرصے سے فعال ہے؟ 36. What is the approximate percentage of LMIS reports that are received in time to be used for logistics decisions (ordering, distribution, etc.) at the district store? اندازا کتنے فیصد ایل ایم آء ایس رپورٹس وقت پر موصول ہو جاتی ہیں، تاکہ ان کی بنیاد پر الجسٹکس فیصلے کئے جا سکیں؟ 37. If the percentage is below 100%, please explain why some service delivery points (SDPs) or sub￾districts report don’t report on time? کی طرف سےوقت پر رپورٹ نہ کرنے کی کیا وجوہات ہیں؟ SDP اگر ٪100 سے کم تو بتائیں کے DELIVER LMIS: Final Evaluation Report 64 38. What feedback mechanisms are in place to channel logistics information back to SDPs or sub￾district stores? (CIRCLE ALL THAT APPLY AND WRITE ANY COMMENTS) 1. Telephone 2. Reports 3. Meetings 4. Supervisory visit 5. Other 6. None Comments: 39. Is the LMIS used to monitor and evaluate the Family Planning and/or Immunization program’s performance? کیا سی ایل ایم آء ایس نگرانی اور پروگرام کی کارکردگی کا اندازہ کرنے کے لئے استعمال کیا جاتا ہے؟ ہاں Yes 1. نہیں No 2. Why or why not? (WRITE AN ANSWER) کیوں اور کیوں نہیں 40. How are logistics data recorded, managed, analyzed, and used at district stores? ضلعی اسٹور پر الجسٹکس ڈیٹا کا کس طرح اندراج، تجزیہ اور استعمال کیا جاتا ہے؟ How recorded: ___________________________________________________________ How managed: ___________________________________________________________ How analyzed: ___________________________________________________________ How used at district stores: _________________________________________________ DELIVER LMIS: Final Evaluation Report 65 41. What indicators related to logistics and/or product availability does the LMIS track (e.g., stock￾out rate, percentage of reporting, rational prescribing practices, etc.)? Who tracks these indicators? How often? ان انڈیکیٹرز کو کون دیکھتا ہے؟ اور کتنی دفعہ؟ 42. What decisions are based on LMIS reports and who makes these decisions? سی ایل ایم آء ایس رپورٹ کی بنیاد پر کس طرح کہ فیصلے کیے جاتے ہیں؟ 1. Forecasting 2. Procurement 3. Transport/delivery 4. Scheduling supervisory visits 5. Inventory management 6. How much to resupply 7. Other 43. Who is responsible for assuring the quality of LMIS data (accuracy, completeness, and timeliness) as standalone data or when compared to other sources of data on commodities (e.g., service statistics, etc.)? (PROBE ON: IS THERE A DATA VALIDATION PROCESS IN PLACE TO CAPTURE ANY ERROR IN DATA ENTRY? IF YES DESCRIBE THE PROCESS.) کے اعداد و شمارکے معیار کو جانچنے اور برقرار رکھنے کی ذمہ داری کس کی ہے؟ )کیا ڈیٹا کی توثیق کے لیئے کوئی طریقہ موجود LMISہے؟ جو ڈیٹا کے اندراج کے وقت غلطی کا بتا سکے( ANSWER: _______________________________________ 44. a. Is logistics information provided to the appropriate decision makers for logistics planning (e.g., CWH, SDPs/NGOs)? کیا الجسٹکس معلومات الجسٹکس منصوبہ بندی کے لئے مناسب فیصلہ سازوں کو فراہم کی جاتی ہے؟ ہاں Yes 1 نہیں No 2 Why or why not? (WRITE AN ANSWER) نہیں کیوں یا کیوں b. What information is provided? کس قسم کی معلومات فراہم کی جاتی ہیں؟ DELIVER LMIS: Final Evaluation Report 66 c. Who provides the information? معلومات کون فراہم کرتا ہے؟ d. Who receives the information? معلومات کون وصول کرتا ہے؟ e. How often? (ENCIRCLE RELEVANT OPTIONS) کتنی دفعہ؟ i. Monthly ii. Quarterly iii. Semi-annually iv. Annually v. Other__________________________________ f. How is the information used? (PROBE ON FEEDBACK TO THE FACILITY LEVEL) معلومات کیسےاستعمال کی جاتی ہیں؟ 45. How does the district store ascertain quality of data for these essential data elements? (PROBE ON: IS THERE A DATA VALIDATION PROCESS IN PLACE TO CAPTURE ANY ERROR IN DATA ENTRY? IF YES, DESCRIBE THE PROCESS.) ضلع سٹور کس طرح ان ضروری اعداد و شمار کے معیار کو یقینی بناتا ہے؟ )کیا ڈیٹا کی توثیق کے لیئے کوئی طریقہ موجود ہے؟ جو ڈیٹا کے اندراج کے وقت غلطی کا بتا سکے( 46. Does the district store staff compare the eLMIS outputs to the paper-based system? کیاای ایل ایم آء ایس سے حاصل شدہ معلومات کا کاغذی معلومات کے ساتھ موازنہ کیاجاتا ہے؟ 1 Yes 2 No 47. Does the district store use barcoding technology to track: کیا ضلعی اسٹور مین barcodingٹیکنالوجی استعمال کی جاتی ہے i. Receipts ii. Issues of commodities iii. Balance of commodities DELIVER LMIS: Final Evaluation Report 67 48. Is there a built-in trigger mechanism or reminder functionality within the eLMIS to alert staff about stock levels and requisitions? کیا آپ کے eLMIS سسٹم میں ایسا کوئی خودکار نظام موجود ہے، جو اسٹاف کو اسٹاک کی سطح اور درخواستوں کے بارے میں آگاہ کرے؟ 1. Yes (describe:_________________________________) 2. No 49. Does your LMIS data entry interface have plausibility checks at entry points (WHICH PREVENTS OR PROMPTS YOU FROM ENTERING WRONG DATA)? کیاآپ کے LMIS ڈیٹا انٹری انٹرفیس کے انٹری پوائنٹس پر چیک موجود ہیں؟ ہاں Yes 1. نہیں No 2. Why or why not? (WRITE AN ANSWER) نہیں کیوں اور کیوں DELIVER LMIS: Final Evaluation Report 68 OBSERVATION CEHCKLIST (PUT A TICK ‘√’ AGAINST THE RIGHT OPTION) Sr. # Description Present and functional Present and non-functional Not present 1. Computer equipment i. Desktop ii. Laptop iii. UPS iv. Printers v. Scanners vi. Bar code scanners 2. Is computer network wired or wireless? 3. Describe the type of internet (either in office or service provider internet packages) (ENCIRCLE THE RELEVANT TYPES) i. Dial-up ii. ISDN iii. DSL iv. Wireless DSL 4. Electricity backup generator? i. Entire facility ii. Just for running computers iii. For non-computer uses 5. Back-up system or server in times of repair or maintenance of the original system or when the original system develops fault? DELIVER LMIS: Final Evaluation Report 69 Provincial and District Facility Managers INSTRUMENT FOR DELIVER LMIS EVALUATION: For use at provincial and district levels with facility managers انٹرویو کے کوائف : :Interview of Demographics تاریخ (دن-ماہ-سال( : ___________________________________:(Day-Month-Year (Date انٹرویو لینے والے کا نام: ___________________________________ :name s’Interviewer نوٹس لینے والے کا نام: ___________________________________ :name s’taker Note انٹرویو لینےکی جگہ/مقام: ___________________________________ :location Interview صوبہ: ___________________________________________ :Province ضلع: ____________________________________________ :District مرکز صحت: _______________________________ :facility Health انٹرویو دینے والے کا نام:___________________________________________ :name Interviewee انٹرویو دینے والے کے ادارے کا نام: ______________________________ :organization Interviewee انٹرویو دینے والے کا عہدہ : _______________________________________________ :title Interviewee Interviewee’s phone number: ________________________________ :تفصیالت کی رابطے Introduction: My name is ______________________. I work for a research organization called Management Systems International (MSI) which is based in Islamabad. As explained in the official letters from the Department of Health (DoH), the Population Welfare Department (PWD), and MSI, we are conducting a final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan (GOP) continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain for health commodities (family planning commodities and vaccines). We will ask a few questions related to: 1. The effectiveness of LMIS for managing the supply chain of medicines, such as family planning commodities and vaccines. 2. The sustainability of the Logistics Management Information System for contraceptives (cLMIS) and strengthening of data-driven decisions on supply chain management. 3. Programmatic and cost-efficiencies of scaling up the vaccines’ LMIS (vLMIS). 4. Best practices, innovations, and lessons learned in supply chain management, cLMIS, and vLMIS. This interview will take approximately 45 minutes to 1 hour. We will treat the information we collect as confidential and will never associate it with your name. میرا نام ۔۔۔۔۔۔۔۔۔۔۔۔۔ ہے، میں اسالم آباد میں قائم ایک تحقیقی ادارے (MSI (International systems Managementکے ساتھ کام کر رہا ہوں۔ جیسے کے اجازت نامے میں بتایا گیا ہے کہ ہم JSI کی طرف سے کیے جانے والے project Deliverکے LMIS کے حوالے سے تحقیق کر رہے ہیں۔ اس تحقیق کے نتائج سے حاصل شدہ معلومات حکوم ِت پاکستان کو فیملی پالننگ کے اشیاء کی عوام تک ترسیل کے طریقہ کار کو مضبوط اور بہتر بنانے اور اس کے نتیجے میں ، زچہ اور بچہ کی صحت کی بہتر ی کے لیے مددگار ثابت ہونگی۔ میں آپ سے نیچے بیان کردہ نکات کے متعلق کچھ سوال کرنا چاہونگا۔ .1 ویکسین کی ترسیل اور انتظامی امور پر LMISکے اثر ات ۔ .2 cLMIS سے حاصل کردہ معلومات کی بنیاد پر فیملی پالننگ کے اشیاء کی ترسیل کے نظام کو باقی عالقوں میں پھیالنا ۔ 4 ان اشیاء کی ترسیل سے متعلق فیصلہ سازی میں معاونت۔ .4cLMISاور vLMISکے استعمال کے دوران management chain supplyمیں بہترین عمل، جدت اور حاصل شدہ اسباق کے متعلق جاننا۔ DELIVER LMIS: Final Evaluation Report 70 یہ انٹرویو تقریبا 45 منٹ سے 01 گھنٹے تک جاری رہےگا۔ اس سے حاصل کردہ معلومات مکمل طور پر صیضۂ راز میں رکھا جائیگا اور ان معلومات کوکہیں بھی آپ کے نام سے منسوب نہیں کیا جائیگا۔ May I have your permission to proceed with the interview? کیا مجھے انٹرویو کا آغاز کرنے کی اجازت ہے؟ ہاں Yes نہیں (انٹرویو کو روک دیں( (INTERVIEW THE STOP (No May I have your permission to record the interview to ensure the completeness and accuracy of your opinions? ا قیمتی معلومات سے مکمل طور پ کا انٹرویو ریکارڈ کرنا چاہتے ہیں۔ کیا ہمیں ٓپ سے حاصل کردہ پر مستفید ہونے کے لیئے ہم آ انٹرویو ریکارڈ کرنے کی اجازت ہے؟ Yes (SWITCH ON THE RECORDER AND START THE INTERVIEW) No (START THE INTERVIEW) DELIVER LMIS: Final Evaluation Report 71 Provincial and District Stakeholders INSTRUMENT FOR PROVINCIAL AND DISTRICT STAKEHOLDERS Detailed questions: Questions for all interviewees – the interviewer will explain to the interviewee that “I will be asking you some questions about the following issues, and I hope that you will provide your opinions based on your experience during implementation of the DELIVER project.” یہ وضاحت کرے گا کہ "میں ا پ مجھ ٓ انٹرویور جواب دہندہ کو پ سے مندرجہ ذیل امور کے متعلق کچھ سوال کرونگا اور مجھے امید ہے کے آ سے ڈلیور پروجیکٹ کے عملدرآمد کے دوران ہونے والے تجربات کی بنیاد پر اپنے راۓ سے آگاہ کرینگے"۔ Evaluation Question 1 seeks your opinions about the three objectives of the project:  What is the extent to which the project has been successful in meeting its three major objectives for the LMIS activity?  What is the extent to which trained staff has used the training they received to address supply chain gaps or issues?  To what extent is staff using LMIS data for decision-making? Objective No. 1: Improve and strengthen in-country supply chains 1. Are you aware of the Logistics Management Information System for contraceptives (cLMIS), and if so, have you had training on the cLMIS from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا ا cLMIS کے بارے میں جانتے ہیں؟ اگر ہاں، تو کیا آپ نے DELIVER/JSIکی طرف سے ٹریننگ حاصل کی ہے؟ ٓپ 1 Yes, and I have received cLMIS training from JSI/DELIVER میں نے ہاں، اور DELIVER/JSI کی طرف سے ٹریننگ حاصل کی ہے 2 Yes, but I have not received training on the cLMIS میں نے ہاں، مگر DELIVER/JSI کی طرف سے ٹریننگ حاصل نہیں کی ہے 3 No (GO TO Q5) نہیں 2. In order to ensure the availability of family planning commodities (e.g., condoms, pills, contraceptives injections and Copper-T), a supply chain system is required, and there are many factors involved. In your experience, what are the three most important factors that can adversely affect the supply chain of family planning commodities from the central warehouse in Karachi to the providers of family planning services at the district level? (ASK FOR THE TOP THREE FACTORS) ُس کے بہت سے عناصر فیملی پالننگ کے اشیاء کی موجودگی کو یقینی بنانے کے لیئے ایک ترسیلی نظام کا ہونا ضروری ہے اور ا ہو سکتے ہیں، اپنے تجربات کی بنیاد پر بتائیں کے ایسے کون سے 03 اہم عناصر ہیں جو ویکسین کے سینٹرل ویئرہاؤس سے ضلعی سطح تک ترسیل پر منفی اثرات ُمرتب کر سکتے ہیں ۔ 1) _____________________________________________________________ 2) _____________________________________________________________ 3) _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 72 3. In your experience, to what extent, if any, has the cLMIS changed the management or operation of the supply chain of family planning commodities? (CIRCLE ONE NUMBER) ا کے تجربات کی بنیاد پر cLMIS کس حد تک فیملی پالننگ کے اشیاء کی ترسیل کے نظام میں تبدیلی کا سبب بنا ہے۔؟ ٓپ ئی ہے improved Greatly 1 بہت بہتر تبدیلی آ ئی ہے improved Somewhat 2 کسی حد تک بہتر تبدیلی آ 3 Stayed the same ہے ئی کوئی بہتر ی یا خرابی نہیں آ بہت خراب ہوئی ہے worse Got 4 5 Don’t know نہیں معلوم جواب دینے سے انکار answer to Refused 6 4. Why do you think so? Please describe an example to support your response. (PROBE THE FACTORS THAT WERE MENTIONED IN Q2: do you think the training you received on using the cLMIS was sufficient? Was there a field-based follow-up to the cLMIS training you received for mentoring and supportive supervision of your use of the system at your workplace? Does the cLMIS Dashboard include options you needed? Does the cLMIS provide the data that you needed to make routine decisions about managing the supply chain for contraceptives such as identifying weaknesses or gaps in meeting targets, possible pilfering, etc.?) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے ۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا 5. Are you aware of the Logistics Management Information System for vaccines (vLMIS) and, if so have you had training on the vLMIS by JSI/DELIVER? (CIRCLE ONE NUMBER) کیا ا vLMIS کے با رے میں جانتے ہیں؟ اگر ہاں، تو کیا آپ نے DELIVER/JSIکی طرف سے ٹریننگ حاصل کی ہے؟ ٓپ 1. Yes, and I have had training on the vLMIS by JSI/DELIVER میں نے ہاں، اور DELIVER/JSI کی طرف سے vLMIS پر ٹریننگ حاصل کی ہے۔ 2. Yes, but I have not had training on vLMIS from JSI/DELIVER میں نے ہاں، مگر DELIVER/JSI کی طرف سے vLMIS پر ٹریننگ حاصل نہیں کی ہے۔ 3. No (GO TO Q9) نہیں 6. In order to ensure the availability of vaccines (.g. BCG, polio, measles, etc.) for infants, children, and adults, a supply chain system is required, and there are many factors involved. In your experience, what are the three most important factors that can adversely affect the supply chain of vaccines from the central warehouse in Karachi to the end-beneficiary at the district level? (ASK FOR THE TOP THREE FACTORS) ویکسین کی موجودگی کو یقینی بنانے کے لیے ایک ترسیلی نظام کا ہونا ضروری ہے۔ اور اس کے بہت سے عناصر ہو سکتے ہیں، اپنے ٰی تجربات کی بنیاد پر بتائیں کے ایسی کون سی تین انتہائ اہم عناصر ہیں جو ویکسین کے سینٹرل ویئر ہاؤس سے ضلعی سطح تک کی ترسیل پر منفی اثرات مرتب کر سکتے ہیں۔ 1) _____________________________________________________________ 2) _____________________________________________________________ 3) _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 73 7. In your opinion or experience, to what extent, if any, has the vLMIS affected the supply chain of vaccines? (CIRCLE ONE NUMBER) اپنے کے تجربات کی بنیاد پر بتائیے کہ vLMISکس حد تک ویکسین کی ترسیل کے نظام میں تبدیلی کا سبب بنا ہے؟ ئی ہے improved Greatly 1 بہت بہتر تبدیلی آ ئی ہے improved Somewhat 2 کسی حد تک بہتر تبدیلی آ 3 Stayed the same ہے ئی کوئی بہتر ی یا خرابی نہیں آ بہت خراب ہوئی ہے worse Got 4 5 Don’t know نہیں معلوم جواب دینے سے انکار answer to Refused 6 8. Why do you think so? Please use examples to support your response. (PROBE THE FACTORS THAT WERE MENTIONED IN Q6) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا Objective No. 2: Strengthen environment for commodity security 9. What is your understanding of the term “commodity security”? (PROBE ABOUT SPECIFIC EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK-OUTS) “security commodity“کے بارے میں آپ کیا جانتے ہیں؟ )جس میں ان اشیاء کی دستیابی، صفائی کے ساتھ حفاظت ، چوری اور اسٹاک میں موجود نہ ہونے سے متعلق مثالوں کے بارے میں جانیئے( 10. In your experience, has the cLMIS system facilitated the safe storage of family planning commodities in the district stores? (CIRCLE ONE NUMBER) ا ۓ میں cLMIS ویکسین کی ضلعی اسٹورز میں بحفاظت ذخیرہ کرنے کو یقینی بنانے میں مددگار ثابت ہوا ہے؟ ٓپ کی را 1 Yes 2 No 11. Why do you think so? Please use examples to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا DELIVER LMIS: Final Evaluation Report 74 12. In your experience, has the cLMIS system facilitated the safe distribution of family planning commodities (preventing spoilage and pilferage to the market)? (CIRCLE ONE NUMBER) ا ۓ میں کیا cLMIS فیملی پالننگ کے اشیاء کے ضلعی اسٹورز میں بحفاظت ذخیرہ کرنے کو یقینی بنانے میں مددگار ٓپ کی را ثابت ہوا ہے؟ 1 Yes 2 No 13. Why do you think so? Please use examples to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا 14. In your experience, has the vLMIS system helped ensure the safe storage of vaccines (adequate temperatures and space) in the district stores? (CIRCLE ONE NUMBER) ا ۓ میں vLMIS ویکسین کی ضلعی اسٹورز میں بحفاظت ذخیرہ کرنے کو یقینی بنانے میں مددگار ثابت ہوا ہے؟ ٓپ کی را 1 Yes 2 No 15. Why do you think so? Please use examples to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا 16. In your experience, has the vLMIS system facilitated the safe distribution of vaccines (preventing spoilage and pilferage to the market)? (CIRCLE ONE NUMBER) ا ۓ میں کیا vLMIS فیملی پالننگ کے اشیاء کی ضلعی اسٹورز میں بحفاظت ترسیل کو یقینی بنانے میں مددگار ثابت ہوا ٓپ کی را ہے؟ 1 Yes 2 No 17. Why do you think so? Please use examples to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا DELIVER LMIS: Final Evaluation Report 75 (ASK ONLY OF DoH AND PWD RESPONDENTS) 18. What sort of participation (cooperation, coordination, and/or collaboration) do you have with your counterpart (DoH or PWD)? Please provide an example. کے ساتھ )تعاون ، ہم ا ہیں؟ ٓ آپ اپنے ہم منصب (PWD/DoH ( ہنگی اور اشتراک ( کے لیئے کس قسم کی شمولیت رکھتے (PROBE WITH RESPECT TO FORECASTING AND ENSURING THE AVAILABILITY AND ACCESSIBILITY OF COMMODITIES FOR SERVICE DELIVERY, ADVOCACY/DEMAND GENERATION, DATA REPORTING, DOUBLE COUNTING ETC. FURTHER, PROBE ABOUT FACILITATION OR HINDERING FACTORS. PROBE ON ISSUES LIKE: WHETHER THERE ARE MEASURES TO REDUCE DISPARITIES BY COST-SHARING, ETC.; AND ON ISSUES LIKE WHETHER THERE ARE REGULARLY SCHEDULED JOINT DoH-PWD MEETINGS AT THE PROVINCE AND/OR DISTRICT LEVELS) (ASK ONLY OF DoH AND PWD RESPONDENTS) 19. In your experience, has cooperation, coordination, and/or collaboration between DOH and PWD changed because of work the DELIVER project has done on the LMIS? Would you say that it has greatly improved, somewhat improved, stayed the same, become somewhat worse, or become much worse? (ENCIRCLE ONE NUMBER) اپنے تجربات کی بنیاد پر بتائیے کہ کیا project Deliverکے LMIS پر ہونے والے کام کی وجہ سے DoHاور PWD کے ہنگی اور ئی ہیں؟ ٓ درمیان باہمی تعاون ، ہم ا اشتراک میں تبدیلیاں آ 1 Greatly improved……………….(GO TO Q20) ہے ئی بہت بہتر تبدیلی آ کسی حد تک بہتر تبدیلی آئی (20Q TO GO……………(improved Somewhat 2 3 Stayed the same………………....(GO TO Q20) ئی کوئی بہتر ی یا خرابی نہیں آ کس حد تک خراب صورتحال رہی (20Q TO GO………(worse somewhat Became 4 بہت حد تک خراب صورتحال رہی (23Q TO GO……………(worse much Became 5 6 Don’t know……………………...(GO TO Q23) نہیں معلوم 7 Refused to answer………………(GO TO Q20) انکار سے دینے جواب 20. Why do you think so? Please describe any example to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا DELIVER LMIS: Final Evaluation Report 76 (ASK ONLY FROM RESPONDENTS WHO CHOSE OPTION 1, 2, 4, or 5 IN QUESTION 19) 21. In which areas, if any, did the LMIS have an effect on cooperation, coordination, and/or collaboration between DoH and PWD? ان میں سے کن حصوں میں ا سمجھتے ہیں کہ LMIS اور PWD ی ٓپ ٰ ئ ہنگی اور اشتراک میں بہتری آ کے درمیان باہمی تعاون ، ہم آ ہے؟ 1 Service delivery 2 Demand generation 3 Data reporting 4 Double counting 5 Other (specify) __________________ 22. Why do you think so? Please use examples to support your response? (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا Objective No. 3: Increase knowledge management and dissemination 23. Have you received any training from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا آپ / JSI DELIVER کی طرف سے کوئی ٹریننگ حاصل کر ُچکے ہیں؟ 1 Yes 2 No .........(GO TO Q41) 24. On which of the following topics, if any, did you receive training? (CIRCLE ALL THAT APPLY) مندرجہ ذیل موضوعات میں سے آپ نے کس پر ٹریننگ حاصل کی ہوئی ہے؟ 1 Principles of supply chain management ترسیل کے نظام کے بنیادی اصول 2 Purpose of the cLMIS in supply chain management کے مقاصدcLMISفیملی پالننگ کے اشیاء کی ترسیل کے نظام متعلق ے 3 Purpose of the vLMIS in supply chain management کے مقاصدvLMISویکسین کی ترسیل کے نظام کے متعلق 4 cLMIS training-of-trainers cLMISپر ٹریننگ دینے والوں کی ٹریننگ 5 vLMIS training-of-trainers vLMISپر ٹریننگ دینے والوں کی ٹریننگ 6 Usage of the cLMIS for entering data اعداد و شمار کے اندراج کے لیئے cLMIS کا استعمال 7 Usage of the vLMIS for entering data اعداد و شمار کے اندراج کے لیئے vLMIS کا استعمال DELIVER LMIS: Final Evaluation Report 77 8 Usage of cLMIS for supply chain management decision-making: فیملی پالننگ کے اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی میں cLMIS کا استعمال. IF 8 IS CIRCLED, PROBE REGARDING DECISION-MAKING IN SUPPLY CHAIN MANAGEMENT FOR FAMILY PLANNING SERVICES, WHETHER THEY HAVE QUARTERLY REVIEW MEETINGS AND HOW USEFUL ARE THOSE—CAN THE INTERVIEWEE DESCRIBE AN EXAMPLE OF HOW S/HE HAS USED cLMIS DATA? __________________________________________________________ 9 Usage of vLMIS for supply chain management decision-making: فیملی پالننگ کے اشیاء کی ترسیل کے نظام کے متعلق فیصلہ سازی میں vLMIS کا استعمال IF 9 IS CIRCLED, PROBE REGARDING DECISION-MAKING IN SUPPLY CHAIN MANAGEMENT FOR FAMILY PLANNING SERVICES— WHETHER THEY HAVE QUARTERLY REVIEW MEETINGS AND HOW USEFUL ARE THOSE. CAN THE INTERVIEWEE DESCRIBE AN EXAMPLE OF HOW S/HE HAS USED vLMIS DATA? __________________________________________________________ 10 Maintenance of the cLMIS Yes/No cLMISکو بحال رکھنے کی ٹریننگ 11 Maintenance of the vLMIS Yes/No vLMISکو بحال رکھنے کی ٹریننگ 12 Other cLMIS training (describe below): Yes/No Describe: (PROBE ON WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE cLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM) cLMISسے متعلق کوئی اور ٹریننگ، وضاحت: ______________________________________________________ 13 Other vLMIS training (describe below) Yes/No Describe: (PROBE ON WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE vLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM) vLMISسے متعلق کوئی اور ٹریننگ، وضاحت: ______________________________________________________ DELIVER LMIS: Final Evaluation Report 78 14 Other supply chain management training (describe below) Yes/No i. Describe: commodities کی ترسیل کے نظام سے متعلق کوئی اور ٹریننگ، وضاحت: ______________________________________________________ 25. Did the training received provide you with new information? (CIRCLE ONE NUMBER) حاصل کردہ ٹریننگ سے آپ کو کوئی نئی معلومات حاصل ہوئی؟ 1 Yes 2 No 26. Did you find the training received to be useful in your work? (CIRCLE ONE NUMBER) کیا حاصل کردہ ٹریننگ آپ کے روز مرہ کے کام میں مددگار ثابت ہوئی؟ 1 Yes 2 No 27. Did you gain some knowledge or skills relevant to your work? (CIRCLE ONE NUMBER) کیا ٹریننگ نے آپ کے روزمرہ کے کام کے متعلق معلومات اور مہارت حاصل کرنے میں معاونت کی؟ 1 Yes 2 No .............(GO TO Q30) 28. Please give me an example of how you applied this new knowledge or new skills in your work. (WRITE AN ANSWER) ا مہارت کو اپنے روزمرہ کے کام میں استعمال کیا ٓ براۓ مہربانی ایک ایسی مثال بیان کریں کےکہ پ نے ٹریننگ سے حاصل کردہ معلومات اور ہو؟ 29. In your experience, what was the most important aspect of the training that has contributed to enhancing your skills at work? (WRITE AN ANSWER) ا روزمرہ کے کام میں مہارت کو بڑھانے میں معاون ثابت ہوا ؟ ٓپ کے خیال میں ٹریننگ کا وہ کون سا اہم پہلو تھا جو آپ کے 30. What kinds of testing/assessment, if any, did you have during the training? (CIRCLE ONE NUMBER) ٹریننگ کے دوران آپ سے کس قسم کی امتحانات لیئے گئے/ ٹریننگ کے حوالے سے آپ سے کس قسم کا جائزہ لیا گیا؟ قبل از ٹریننگ جائزہ only test-Pre 1 بعد از ٹریننگ جائزہ only test-Post 2 3 Pre- and post-, both دونوں کوئی نہیں None 4 DELIVER LMIS: Final Evaluation Report 79 31. Following the training, have there been any follow up visits at your workplace to assess the usefulness of the trainings on the performance of your job? (CIRCLE ONE NUMBER) ٹریننگ کے بعد یہ جانچنے کے لیئے کے ٹریننگ ا معاونت کے لیئے کتنی حد پ ٓپ کے روز مرہ کے کام میں تک ُمفید رہی ہے۔ کیا آ کے کام کرنے کی جگہ پر کسی نے معائنے کے لیئے دورہ کیا؟ 1. Yes 2. No (GO TO Q35) 32. If yes, who did the follow-up visit of after the training? (CIRCLE ONE NUMBER) اگر ہاں تو ٹریننگ کے بعد معائنے کے لیئے کس نے دورہ کیا؟ 1 JSI DELIVER team 2 Government officials 3 Others (specify) _______________________________________ 33. During the follow-up visit, were you asked about how the trainings have improved your skills? (CIRCLE ONE NUMBER) ٹریننگ کے بعد معائنے کے لیئے کیے جانے والے دوروں میں کیا آپ سے پوچھا گیا کے ٹریننگ کس حد تک آپ کے روز مرہ کے کام میں مہارت کو بڑھانے میں معاون ثابت ہوئی ہے؟ 1. Yes 2. No .............(GO TO Q35) 34. If yes, how did you respond? (PROBE ON: HOW THE TRAINEE’S KNOWLEDGE IMPROVED, HOW SKILLS IMPROVED, WHAT WAS MISSING FROM TRAINING, AND HOW THE TRAININGS CAN BE IMPROVED IN THE FUTURE.) (WRITE AN ANSWER) اگر ہاں تو اِس سوال پر آپ کا کیا جواب تھا؟ 35. In your opinion, how could the training be improved? (PROBE: CONTENT OF CURRICULUM, TRAINING METHOD, AND FOLLOW-UP) (WRITE AN ANSWER) آپ کے خیال میں ٹریننگ میں کس طرح کی بہتری الئی جاسکتی ہے؟ 36. How did you think the trainer/facilitator was in terms of the characteristics below? (WRITE AN ANSWER BY ASKING ABOUT EACH TYPE OF TRAINING TAKEN AND WRITING A SCORE ABOUT THE TRAINER’S CHARACTERISTICS. WAS THE TRAINER (1) VERY GOOD, (2) MODERATELY GOOD, (3) VERY BAD, (4) MODERATELY BAD, OR (5) THE RESONDENT HAS NO OPINION) DELIVER LMIS: Final Evaluation Report 80 مندرجہ ذیل پر آپ نے تربیت دینے والے کو کیسا پایا۔ Characteristics of Trainer/Facilitator and Curriculum تربیت دینے والے / سہولت کار اور مواد کی خصوصیات 1. Trainer well prepared تربیت دینے والے کی تیاری 2. Time management وقت کی پابندی 3. Methodology used (brainstorming, group discussion, and audio-visual aids) استعمال کیاجانے واال طریقہ کار 4. Use of training aids (eg, handouts) and technology آ تربیت کے لیئے ٹیکنالوجی اور دوسرے مددگار الت کا استعمال 5. Knowledge of the subject موضوع کے متعلق معلومات 6. Content easily understood مواد کا آسان فہم ھونا 7. New concept(s) introduced نئے تصورات کا تعارف 37. In your experience, to what extent, if any, has the DELIVER project’s training on the use of LMIS affected your supply chain management skills (i.e. in reporting, requisition, forecasting, procurement, delivery, and availability). Would you say that your skills greatly improved, somewhat improved, stayed the same, got worse, or you don’t know? (ENCIRCLE ONE) یں کے ٰ اپنے تجربہ کی بنیاد پر بتائ Project DELIVERکی ٹریننگ فیملی پالننگ کے اشیاء کی ترسیل کے نظام کی مہارت میں ئی ؟ کتنی بہتری آ 1 Greatly improved …………………(GO TO Q38) ئی بہت بہتر تبدیلی آ 2 Somewhat improved ……………....(GO TO Q38) ئی کسی حد تک بہتر تبدیلی آ 3 Stayed the same …………………...(GO TO Q38) ائی کوئی بہتر ی یا خرابی نہیں ٓ 4 Got worse ………………………....(GO TO Q38) ہوئی خراب بہت 5 Don’t know ………………………..(GO TO Q39) نہیں معلوم 6 Refused to answer ………………....(GO TO Q39) انکار سے دینے جواب 38. Why do you think so? Please describe any example to support your response. (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا DELIVER LMIS: Final Evaluation Report 81 39. Do you think that the training on the use of the LMIS should be scaled up? کے استعمال کے متعلق ٹریننگ کو بڑھانا چاہۓ؟ LMISکیا آپ سمجھتے ہیں کہ 1. Yes 2. No Why or why not? (WRITE AN ANSWER AND PROBE ON THE QUALITY OF TRAINING CONTENT, METHODS, FOLLOWUP MENTORING AND SUPPORTIVE SUPERVISION, GEOGRAPHIC COVERAGE, AND SCALING UP) ہاں اور نہیں دونوں جواب کی صورت میں وضاحت کیجئے۔ 40. Is there anything you would like to add regarding scaling up or improving the trainings? ٹریننگ کو مزید بڑھانے اور بہتری النے کے لیئے ا ۓ گاہ کیجئے۔ ٓپ کچھ کہنا چاہتے ہیں، تو برا مہربانی ہمیں آ 41. In your opinion, can an LMIS like the one developed by the DELIVER improve decision￾making related to supply chain management? (CIRCLE ONE NUMBER) ا DELIVERکے تیار کردہ LMIS جیسا پروگرام commodities کی ترسیل کے نظام کے متعلق فیصلہ سازی کو بہتر ٓپ کی نظر میں کیا کرنے میں مددگار ثابت ہو سکتا ہے؟ 1 Yes 2 No Why or why not? (WRITE AN ANSWER) ہاں اور نہیں دونوں جواب کی صورت میں وضاحت کیجئے۔ 42. In your experience, to what extent if any, has the LMIS had an effect on decision-making related to supply chain management? Would you say it has greatly improved, somewhat improved, not changed, somewhat worsened, or greatly worsened decision-making? (CIRCLE ONE NUMBER) ا LMIS ,commodities کی ترسیل کے نظام کے متعلق فیصلہ سازی کو بہتر کرنے میں کس حد ٓپ کے تجربات کی بنیاد پر تک مددگار ثابت ہوا ہے؟ 1 Greatly improved.............(GO TO Q43) ئی بہت بہتر تبدیلی آ 2 Somewhat improved........(GO TO Q43) ئی کسی حد تک بہتر تبدیلی آ 3 Stayed the same................(GO TO Q43) ئی کوئی بہتر ی یا خرابی نہیں آ 4 Somewhat worsened.......(GO TO Q43) ئی کسی حد تک خرابی آ 5 Greatly worsened ............(GO TO Q43) ئی بہت حد تک خرابی آ 6 Don’t know ......................(GO TO Q44) نہیں معلوم 7 Refused to answer ..........(GO TO Q44) انکار سے دینے جواب DELIVER LMIS: Final Evaluation Report 82 43. Why do you think so? Please use examples to support your response? (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا Evaluation Question 2: “What are the changes that could be made to ensure sustainability of the cLMIS and strengthen data-driven decisions?” cLMIS سسٹم سے حاصل شدہ اعداد و شمار کی بنیاد پر لیئے جانے والے فیصلوں کی پائیداری کے لیئے کن تبدیلیوں کی ضرورت ہے۔ Questions for all interviewees: 44. Please tell me your job title, the date of your appointment to this job, and describe your responsibility for using the cLMIS. برائے مہربانی مجھے اپنے عہدہ کا نام، تقرری کی تاریخ اور cLMIS کے متعلق آپ کی زمہ داریوں کے متعلق آگاہ کیجئے۔ (WRITE DATE OF APPOINTMENT AND DURATION IN COMPLETED YEARS AND MONTHS—ASK FOR A DETAILED DESCRIPTION OF RESPONSIBILITIES REGARDING USE OF THE LMIS: E.G., WAS IT FOR DATA ENTRY? DATA ANALYSIS? DATA INTERPRETATION? DATA REPORTING? SYSTEM MANAGEMENT? SYSTEM MAINTENANCE?) Job title عہدہ کا نام Responsibilities زمہ داریاں Date of appointment YYYY-MM-DD تقرری کی تاریخ Duration appointed in this position (Years) (Months) اس عہدے پر فائز ہونے کی مدت 45. For the position you are holding currently, do you know how many transfers and postings have taken place in the period between September 2012 andMay 2016? (ENTER ONE NUMBER AND, IF YES, NUMBER OF INDIVIDUALS IN POST) جس عہدے پر آپ ابھی فائز ہیں، کیا آپ جانتے ہیں کہ ستمبر 2012 سے مئی 2016 کے دوران کتنی تقرریاں اور تبادلے ہوچکے ہیں؟ 1 Yes 2 No Number of transfers and postings تقرری اور تبادلےکی تعداد جانئے DELIVER LMIS: Final Evaluation Report 83 ASK QUESTIONS ONLY OF A PROVINCIAL/DISTRICT MANAGER WHO IS RESPONSIBLE FOR DECISION-MAKING ON SUPPLY CHAIN MANAGEMENT: 46. Please share with us the last monthly cLMIS report. (PREFERABLY BY USING THE ONLINE CLMIS SYSTEM, IF NOT AVAILABLE, OR IF THE RESPONDENT DOES NOT KNOW HOW TO DO SO ONLINE, OR IF THE ONLINE SYSTEM IS NOT OPERATING, THEN ASK FOR A COMPUTER-GENERATED HARD COPY). (CIRCLE ONE NUMBER) 1. Yes – online access 2. Yes – hard copy access 3. No – PLEASE EXPLAIN WHY NOT AVAILABLE/NOT POSSIBLE _______________________________________ (GO TO Q47) (OBSERVE THE FOLLOWING AND SCORE RESPONDENT’S UNDERSTANDING, KNOWLEDGE, SKILLS, AND ABILITIES) ASK ONLY OF DISTRICT AND PROVINCIAL MANAGERS Questions for provincial or district manager Response (ENTER ONE NUMBER IN EACH ROW) Incorrect Partially correct Correct 47. Kindly inform us the type of supply chain management performance reports that are available on the cLMIS. برائے مہربانی ہمیں آگاہ کیجئے کہ کس طرح کی سپالئے رپورٹسcLMISچین مینیجمینٹ پرفارمنس پر موجود ہیں؟ 0 1 2 48. Please show us how to get the projected contraceptive requirements (forecasts) from the cLMIS. براۓ مہربانی بتائےی کہ درکار اشیاء کیسے حاصل کی جا سکتی ہیں؟ 0 1 2 49. Please tell us what is meant by “months of stock”. برائے مہربانی مجھے یہ بتائیے کہ منتھس آف اسٹاک سے کیا مراد ہے؟ 0 1 2 50. Please show us how to download data from the online cLMIS. برائے مہربانی مجھے دکھائیے کے آن الئن سے کس طرح اعداد و شمار ڈاؤن لوڈ کیئے جاتے ہیں؟ cLMIS 0 1 2 DELIVER LMIS: Final Evaluation Report 84 Questions for provincial or district manager Response (ENTER ONE NUMBER IN EACH ROW) Incorrect Partially correct Correct 51. Please show us how graphs can be obtained from the online cLMIS. برائے مہربانی ہمیں دکھائیے کہ آن الئن گرافس کیسے حاصل کیئے جاسکتے ہی؟۔ 0 1 2 52. Do you strongly agree, agree, disagree, strongly disagree, or have no opinion about the following statement: “the cLMIS is essential for appropriate data-driven decisions about supply chain management?” (CIRCLE ONE NUMBER) فیملی پالننگ کے اشیاء کی ترسیلی نظام کے بارے میں حاصل شدہ اعداد و شمار کے مطابق فیصلوں کے لیئے cLMISبہت الزمی/ضروری ہے۔ آپ اس بیان پر کیا رائے رکھتے ہیں؟ 1 Strongly agree ............................(GO TO Q53) اتفاق زیادہ بہت 2 Agree ...........................................(GO TO Q53) اتفاق 3 No opinion ..................................(GO TO Q53) نہیں رائے کوئی 4 Disagree .......................................(GO TO Q53) اختالف 5 Strongly disagree........................(GO TO Q53) اختالف زیادہ بہت 6 Refuse ..........................................(GO TO Q54) انکار 53. Why do you think so? Please describe any example to support your response? ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا 54. Which data from the cLMIS do you use to determine the quantity of family planning commodities you need to stock? (CIRCLE ALL THAT APPLY) 1 Average monthly consumption 2 Previous month’s demand 3 Projected Contraceptive Requirement module of cLMIS دیگر )براۓ مہربانی بتائیے( ____________________ (specify please (Other 4 55. In your experience, do you think the cLMIS is useful enough to be sustainably used as a supply chain management tool for reporting data and/or making other supply chain management decisions? (CIRCLE ONE NUMBER) اپنے تجربے کے بنیاد پر کیا آپ سمجھتے ہیں کہ اشیاء کی ترسیلی نظام اور عداد و شمار کی رپورٹنگ کے لیئے سی ایل ایم آئی ایس ایک بہترین اور قابل بھروسہ سسٹم کے طور پر جاری رکھا جاسکتا ہے؟ 1. Yes 2. No DELIVER LMIS: Final Evaluation Report 85 56. Why in your experience is cLMIS useful or not useful? (WRITE AN ANSWER) ِل استعمال ہے یا نہیں ہے؟ آپ کے خیال میں cLMISکس وجہ سے قاب 57. In your experience, what changes or resources are required in the current system to make the cLMIS more useful and sustainable? آپ کے خیال میں cLMISموجودہ ،سسٹم کو مزید کارآمد بنانے کے لئے کیا تبدیلیاں کی جا سکتی ہیں؟ (EXPLORE WITH REGARDS TO SUSTAINED AVAILABILITY OF INPUTS INCLUDING HUMAN RESOURCES, HARDWARE, AND TECHNICAL ASSISTANCE FOR MAINTAINENCE OF SOFTWARE, EVIDENCE THAT THE cLMIS IS USEFUL FOR REPORTING REQUIREMENTS, FOR FINANCIAL ALLOCATIONS, AND FOR OTHER DATA-DRIVEN DECISIONS ON MANAGEMENT OF THE SUPPLY CHAIN FOR CONTRACEPTIVE COMMODITIES) Evaluation Question 3: “Now, I will ask you some questions about the kind of changes which you think could be made to scale up the Vaccine Logistics Management Information System (vLMIS) in order to increase the “programmatic efficiency” of Vaccine Supply Chain Management,” سب سے پہلے میں آپ سے ان تبدیلیوں سے متعلق سواالت پوچھوں گا جو آپ سمجھتے ہیں کہ وی ایل آئی ایم ایس کے استعمال کو بڑھانے میں کی جا سکتی ہیں تاکہ management chain supply vaccine کی پروگرامی کارکردگی میں اضافہ کیا جا سکے 58. Which of the following levels of the vaccine supply chain are you responsible for? (CIRCLE ALL THAT APPLY) آپ ان میں سے کس زمہ داری پر فائز ہیں؟ صوبائی سطح level Provincial 1 ضلعی سطح level District 2 3 Neither……………………………………………………….(GO TO Q59) 59. Which of the following kinds of vaccine supply chain management programmatic decision￾making are you responsible for, including decisions about supporting the implementation, training, scale up, and other aspects of supply chain management tools like vLMIS? (ENCIRCLE ONE OPTION AGAINST EACH) 59.1 Forecasting vaccine supply requirements: Yes No مطلوبہ ویکسین فراہمی کی پیشن گوئی/ پیش بینی 59.2 Assessing the accuracy of forecasts: Yes No پیش بینی کی درستگی کی تشخیص 59.3 Reducing the costs of forecasting errors: Yes No پیش بینی / پیش گوئی میں ہونے والی غلطیوں کی وجہ سے ہونے والی الگت میں کمی النا DELIVER LMIS: Final Evaluation Report 86 59.4 Selecting sources for vaccine purchasing: Yes No ویکسین کی خریداری کے ذرائع کا انتخاب کرنا 59.5 Procuring/purchasing selected vaccines: Yes No منتخب شدہ ویکسینز کی خریداری کرنا 59.6 Storing vaccines: Yes No ویکسینز کا ذخیرہ کرنا 59.7 Distributing vaccines to health facilities: Yes No ِز ویکسینز صحت تک تقسیم کی مرک 59.8 Monitoring vaccine quality: Yes No ویکسین کے معیار کی جانچ پڑتال کرنا 59.9 Monitoring vaccine coverage: Yes No ؤ کی جانچ پڑتال کرناویکسین کے معیار کے پھیال 59.10 Reporting on vaccine distribution: Yes No ویکسینز کی تقسیم کی رپورٹنگ کرنا 59.11 Reporting on vaccine supplies: Yes No ویکسینز کی فراہمی کی رپورٹنگ کرنا 59.12 Preventing stock-outs: Yes No اسٹاک آؤٹ کی روک تھام 59.13 Financially supporting training on supply chain management: Yes No اشیاء کی ترسیل کے نظام کی تربیت کے لیئے مالی امداد فراہم کرنا 59.14 Financially supporting vLMIS implementation: Yes No vLMISپر عمل دار آمد کے لیئے مالی امداد فراہم کرنا 59.15 Vaccination service delivery: Yes No ویکسنیشن کی سروس کی فراہمی 59.16 Other vaccine supply chain management decisions (describe below): Yes No ترسیل کے نظام کے متعلق فیصلہ سازی کے متعلق کوئی اور زمہ داری؟ اگر ہاں تو برائے مہربانی وضاحت سے بیان کریں۔ Describe: _________________________________________ 60. In your opinion, or based on your experience, to what extent has the implementation of the vLMIS had the following effects on the efficiency of managing the Vaccine Supply Chain? Would you say it has greatly increased, somewhat increased, or not affected, the efficiency of managing the Vaccine Supply Chain? (SELECT ONLY ONE ANSWER AND PROBE TO FIND OUT HOW THE RESPONDENT DEFINES “EFFICIENCY” IN TERMS OF THE OPERATION OF THE SUPPLY CHAIN MANAGEMENT PROGRAM, AND HOW THEY THINK THAT THE vLMIS TOOL HAS, OR HAS NOT, AFFECTED VACCINE SUPPLY CHAIN MANAGEMENT PROGRAMMATIC EFFICIENCY) (CIRCLE ONE NUMBER) 1 Greatly improved ............(ONLY RESPOND TO Q60.1 AND Q60.2) بہت حد تک بہتری 2 Somewhat improved .......(RESPOND TO Q60.1, Q60.2 AND Q60.3) کسی حد تک بہتری 3 Stayed the same ............... (RESPOND TO Q60.1, Q60.2 AND Q60.3) کوئی بہتری یا خرابی نہیں 4 Don’t know .......................(GO TO Q61) معلوم نہیں 5 Refused ...............................(GO TO Q62) جواب دینے سے انکار DELIVER LMIS: Final Evaluation Report 87 60.1 Why do you think so? Please describe any example to support your response? (WRITE AN ANSWER) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا 60.2 Do you think that the vLMIS should be scaled up? کیا آپ کو لگتا ہے کہ vLMISکے استعمال کو بڑھانا چاہئیے؟ 1 Yes……...(GO TO Q65) 2 No……...(GO TO Q64.3) (PROBE ON WHY, WHERE, AND HOW TO SCALE UP) 60.3 What are the problems of scaling up the vLMIS and how should these be addressed? (WRITE AN ANSWER) کیا مسائل درپیش آئے ؟ وی ایل آئی ایم ایس کے استعمال کو بڑھانے سے پہلے ان مسائل کو کس طرح حل کیا جانا چاہئیے؟ 61. Based on your experience, to what extent has the implementation of the vLMIS had the following effect on the costs of managing the vaccine supply chain—would you say it has greatly reduced, somewhat reduced, greatly increased, somewhat increased, or not affected, the cost of managing an effective vaccine supply chain? (SELECT ONLY ONE ANSWER AND PROBE TO FIND OUT HOW THE RESPONDENT DEFINES “COSTS” IN TERMS OF THE COST OF EFFECTIVELY OPERATING THE vLMIS AND THE COST OF EFFECTIVELY MANAGING THE VACCINE SUPPLY CHAIN PROGRAM, AND HOW THEY THINK THAT THE vLMIS TOOL HAS, OR HAS NOT, AFFECTED PROGRAMMATIC COSTS OF VACCINE SUPPLY CHAIN MANAGEMENT ) 1 Greatly reduced ..........................(RESPOND TO Q61.1 to Q61.4) بہت حد تک اضافہ 2 Somewhat reduced ....................(RESPOND TO Q61.1 to Q61.4) کسی حد تک اضافہ 3 Greatly increased ........................(RESPOND TO Q61.1 to Q61.4) بہت حد تک کمی 4 Somewhat increased ..................(RESPOND TO Q61.1 to Q61.4) کسی حد تک کمی 5 Not affected .................................(RESPOND TO Q61.1 to Q61.4) کوئی اثر نہیں 6 Don’t know ..................................(RESPOND TO Q61.3 & Q61.4) معلوم نہیں 7 Refused to answer ......................(GO TO Q62) جواب دینے سے انکار DELIVER LMIS: Final Evaluation Report 88 61.1 Please tell me if and how and why the vLMIS has increased or decreased the cost of managing the vaccine supply chain. manage کو chain supply vaccine کرنے میں آنے والے اخراجات میں وی ایل آئی ایم ایس نے کیوں اور کیسے اضافہ کیا؟ براۓمہربانی بتائیےکہ If: _______________________________________________________________ How: ____________________________________________________________ Why: ____________________________________________________________ 61.2 If there are higher costs, how would you reduce them before scaling up the vLMIS? vLMISکےاستعمال کو بڑھانے سے پہلے آپ ان اخراجات میں کمی کیسے الیئں گے؟ 61.3 If you think the vLMIS should be scaled up, how and where would you recommend that scale-up be done? vLMISکےاستعمال کو کیسے اور کہاں بڑھانا چاہئیے؟ How: ___________________________________________________________ Where:___________________________________________________________ 61.4 If you have experienced problems with the implementation of the vLMIS, how should these be addressed before scaling up the vLMIS? کیا مسائل درپیش آئے ؟ وی ایل آئی ایم ایس کے استعمال کو بڑھانے سے پہلے ان مسائل کو کس طرح حل کیا جانا چاہئیے؟ Evaluation Question 4: “What best practices, innovations, and lessons learned from the DELIVER project can be applied to future programming for strengthening supply chain systems?” ڈلیور innovations ,practices اور learned lessons کو مستقبل میں ترسیلی نظام کی مضبوطی کے لیے استعمال کیا جا سکتا ہے؟ پراجیکٹ سے حاصل شدہ کونسی 62. Do you have responsibility for making decisions about the overall management of the supply chain for health system-related commodities at: (CIRLCE ALL THAT APPLY) کیا ا سطح پر نظام صحت کے ترسیل کے نظام سے متعلق فیصلہ سازی کے زمہ دار ہیں؟ ٓپ درجہ ذیل Provincial level: Yes No سطح صوبائی District level: Yes No سطح ضلعی DELIVER LMIS: Final Evaluation Report 89 63. Which of the following kinds of health system supply chain management programmatic decision-making are you responsible for: (CIRLCE ALL THAT APPLY) مندرجہ ذیل میں سے آپ کس قسم کے سپالئے چین مینیجمینٹ کی پروگرامی فیصلہ سازی کے زمہ دار ہیں: 63.1 Forecasting commodity supply requirements: Yes No مطلوبہ اشیاء کی فراہمی کی پیشن گوئی 63.2 Assessing the accuracy of forecasts: Yes No پیش بینی کی درستگی کی تشخیص 63.3 Reducing the costs of forecasting errors: Yes No پیش بینی/ پیشن گوئی میں ہونے والی غلطیوں کی وجہ سے ہونے والی الگت میں کمی النا 63.4 Selecting sources for commodity purchasing: Yes No commodityکی خرید کے لیئے ذرائع کا انتخاب کرنا 63.5 Procuring/purchasing selected commodities: Yes No منتخب شدہ Commoditiesکو خریدنا 63.6 Storing commodities: Yes No commoditiesکا ذخیرہ کرنا 63.7 Distributing commodities to health facilities: Yes No صحتی مراکز میں اشیاء کی تقسیم 63.8 Monitoring the quality of commodities: Yes No commodities کے معیار کی جانچ پڑتال 63.9 Monitoring the coverage of commodities: Yes No commodities کی پہلو کی جانچ پڑتال/نگرانی 63.10 Reporting on vaccine distribution: Yes No ویکسین کے تقسیم کی رپورٹنگ 63.11 Reporting on commodity supplies: Yes No commoditiesکی فراہمی کی رپورٹنگ 63.12 Preventing stock-outs: Yes No اسٹاک آؤٹ کی روک تھام 63.13 Financially supporting training on supply chain management: Yes No سپالئے چین مینجمینٹ کی ٹریننگ کے لیئے مالی معاونت کرنا 63.14 Financially supporting supply chain management implementation: Yes No سپالئے چین مینجمینٹ کے عملدرآمد کے لیئے مالی معاونت کرنا 63.15 Service delivery for health commodities: Yes No صحت کے commodities سے متعلق سروس کی فراہمی 63.16 Supply chain management Human Resource Management: Yes No انسانی وسائل کے انتظام کی سپالئے چین 63.17 Other supply chain management decisions (describe below): Yes No دیگر سپالئے چین مینجمینٹ کی فیصلہ سازی )وضاحت کریں( Describe: ________________________________________________ INTERVIEWERS FIRST PLEASE READ THE FOLLOWING DEFINITIONS TO RESPONDENTS (NO NEED TO PROVIDE THE URLs): “Promising practices are defined as interventions showing progress toward improving health commodity supply chains.” (http://siapsprogram.org/wp￾content/uploads/2014/07/Intro_Acknowledgements-format.pdf and http://siapsprogram.org/publication/promising-practices-in-supply-chain￾management/) اس سے مراد وہ اقدامات ہیں جو کم اور اوسط آمدن والے ممالک میں صحت سے متعلق اشیاء کے ترسیلی نظام میں بہتری النے کا سبب بنتے ہیں۔ DELIVER LMIS: Final Evaluation Report 90 “Proven practices are defined as interventions with proven outcomes in improving health commodity supply chains.” (http://siapsprogram.org/wp￾content/uploads/2014/07/8_Proven-Practices-final.pdf) تحقیق کے ذریعے آزمائے گئے وہ ثابت شدہ نتائج جو کم اور اوسط آمدن والے ممالک میں صحت سے متعلق اشیاء کے ترسیلی نظام میں بہتری النے کا سبب بنتے ہیں۔ ڈلیور پروجیکٹ کے ساتھ اپنے تجربے کی بنیاد پر مندرجہ ذیل سواالت کے جواب دیں۔ 64. Using the following definitions for promising or proven practices in supply chain management, please describe one or more examples of the following supply chain management practices that have been developed or strengthened by the DELIVER project in Pakistan? ِر ترسیل کے نظام میں آپکی promising غور رکھتے ہوئے، کیا آپ درج ذیل ، بہترین اور ثابت شدہ پریکٹسز کی تعریف کو زی ترسیل کے نظام کی پریکٹس جو ڈلیور پروجیکٹ نے پاکستان میں بنائی اور مستحکم کی، ایک یا ایک سے زائد مثال دے کر وضاحت کریں۔ 64.1 Please describe a promising or proven practice for the quantification of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی صحت کے commoditiesکی مقداری تعین کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کی ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ 64.2 Please describe a promising or proven practice for the procurement of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی صحت کے commoditiesکی خرید کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ 64.3 Please describe a promising or proven practice for the warehousing of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی صحت کے commoditiesکے ذخیرے کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ 64.4 Please describe a promising or proven practice for the distribution of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی صحت کے commoditiesکی تقسیم کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ DELIVER LMIS: Final Evaluation Report 91 64.5 Please describe a promising or proven practice for the service delivery and utilization of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی سروس کی ترسیل اور صحت کے commoditiesکے استعمال کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ 64.6 Please describe a promising or proven practice for the data management of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی صحت کے commoditiesکی ڈیٹا مینجمینٹ کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ 64.7 Please describe a promising or proven practice for human resource management in supply chain systems that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: برائے مہربانی سپالئے چین سسٹم میں انسانی وسائل کے انتظام کے لیئے promising، بہترین اور ثابت شدہ پریکٹس کی وضاحت کریں، جو آپ نے ڈلیور پروجیکٹ کے ذریعے سیکھیں اور آپ سمجھتے ہیں کہ پاکستان میں ان کے پیمانے کو بڑھانے کی ضرورت ہے۔ DELIVER LMIS: Final Evaluation Report 92 National Level INSTRUMENT FOR DELIVER LMIS EVALUATION For use at the national level Demographics of Interview: Date (Year-Month-Day):___________________________________________ Interviewer’s name: _______________________________________________ Note taker’s name: _______________________________________________ Interview location: Province:________________________________________ District:_________________________________________ Health Facility:_______________________________ Interviewee name: ________________________________________________ Interviewee organization: ____________________________________________ Interviewee title: _________________________________________________ Interviewee’s phone number: ________________________________________ Introduction: My name is ______________________. I work for a research organization called Management Systems International (MSI) which is based in Islamabad. We are conducting the final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan to continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain of health commodities (family planning commodities and vaccines). We will ask questions related to:  The effectiveness of LMIS for medicines such as family planning commodities and vaccines.  The sustainability of cLMIS and strengthening of data-driven decisions on supply chain management.  The programmatic and cost efficiencies of scaling up the vaccines LMIS (vLMIS).  Best practices, innovations, and lessons learnt in supply chain management, cLMIS, and vLMIS. This interview will take approximately 45–60 minutes. We will treat the information you give us as confidential and will never associate it with your name. May I have your permission to proceed with the interview? Yes No (STOP INTERVIEW) DELIVER LMIS: Final Evaluation Report 93 May I have your permission to record the interview to ensure the completeness and accuracy of your opinions? Yes…(SWITCH ON RECORDER AND START THE INTERVIEW) No…(START THE INTERVIEW) DELIVER LMIS: Final Evaluation Report 94 INSTRUMENT FOR NATIONAL STAKEHOLDERS Detailed questions: Questions for all interviewees – the interviewer will explain to the interviewee “I will be asking you some questions about the following issues, and I hope that you will provide your opinions based on your experience during implementation of the DELIVER project.” Evaluation question 1 has three parts and seeks your opinions about the following:  Part 1: what is the extent to which the project has been successful in meeting its three major objectives for the Logistics Management Information System (LMIS):  to improve and strengthen in-country supply chains,  to strengthen the environment for commodity security, and  to increase knowledge management and dissemination?  Part 2: what is the extent to which trained staff have used the training that they received to address supply chain gaps or issues?  Part 3: to what extent are trained staff using LMIS data for decision-making? Questions about objective 1: Improve and strengthen in-country supply chains 1. Are you aware of the Logistics Management Information System for contraceptives cLMIS, and have you had any training on the cLMIS from the DELIVER Project? (CIRCLE ONE NUMBER) 1. Yes, and I have had training on the cLMIS 2. Yes, but I have not had training on the cLMIS 3. No (GO TO Q5) If yes, what do you know about the cLMIS? (PROBE ON THE CONTRACEPTIVE SUPPLY CHAIN MANAGEMENT-RELATED POLICIES, PROGRAMS, AND STRATEGIC ACTIVITIES OF THE NATIONAL/INTERNATIONAL ORGANIZATION; IS THE ORGANIZATION CONTRIBUTING FUNDING, SUPPORTING TRAINING ON cLMIS, ADVOCATING FOR USE OR SUSTAINABILITY OF THE cLMIS, OR SUPPORTING OTHER ASPECTS OF THE OPERATION OF THE cLMIS?) DELIVER LMIS: Final Evaluation Report 95 2. In order to ensure the availability of family planning commodities (e.g. condoms, pills, contraceptives injections, and Copper-T), a supply chain system is required, and there are many factors involved. In your experience, what are the three most important factors that can adversely affect the supply chain of family planning commodities from the central warehouse in Karachi to the providers of family planning services at the district level? (ASK FOR THE TOP THREE FACTORS) 1) ____________________________________________________________________ 2) ____________________________________________________________________ 3) ____________________________________________________________________ 3. In your experience, to what extent, if any, has the cLMIS affected the operation of the supply chain of these family planning commodities? Would you say that since implementation of cLMIS, the supply chain operates much better, somewhat better, somewhat worse, much worse, or has there been no change? (CIRCLE ONE NUMBER) 1. Much better 2. Somewhat better 3. No change 4. Somewhat worse 5. Much worse 6. Don’t know (GO TO Q5) 7. Refused to answer (GO TO Q5) 4. Why do you think so? Please use examples to support your response? (PROBE ON THE FOLLOWING FACTORS: WAS TRAINING ON USING THE cLMIS SUFFICIENT OR GOOD ENOUGH? WAS THERE A FIELD-BASED FOLLOW-UP TO cLMIS TRAINING FOR MENTORING AND SUPPORTIVE SUPERVISION OF YOUR USE OF THE SYSTEM AT YOUR WORKPLACE? DID THE cLMIS DASHBOARD INCLUDE OPTIONS THAT YOU NEEDED? DID THE cLMIS PROVIDE THE DATA THAT YOU NEEDED TO MAKE ROUTINE DECISIONS ABOUT MANAGING THE SUPPLY CHAIN FOR CONTRACEPTIVES, SUCH AS IDENTIFYING WEAKNESSES OR GAPS IN MEETING TARGETS, POSSIBLE PILFERING, ETC.?) DELIVER LMIS: Final Evaluation Report 96 5. Are you aware of the Logistics Management Information System for vaccines (vLMIS) and, if so have you had training on the vLMIS from the DELIVER Project? (CIRCLE ONE NUMBER) 1 Yes, and I have had training on the vLMIS 2 Yes, but I have not had training on the vLMIS 3 No……………………..(GO TO Q8) If yes, what do you know about the vLMIS? (PROBE ON THE VACCINE SUPPLY CHAIN MANAGEMENT-RELATED POLICIES, PROGRAMS, AND STRATEGIC ACTIVITIES OF THE NATIONAL/INTERNATIONAL ORGANIZATION; IS THE ORGANIZATION CONTRIBUTING FUNDING, SUPPORTING TRAINING ON vLMIS, ADVOCATING FOR USE OR SUSTAINABILITY OF THE vLMIS, OR SUPPORTING OTHER ASPECTS OF THE OPERATION OF THE vLMIS?) 6. In your experience, to what extent, if any, has the vLMIS affected the overall performance of the supply chain of vaccines? Would you say that since implementation of the vLMIS the supply chain for vaccines performs much better, somewhat better, somewhat worse, much worse, or has there been no change? (CIRCLE ONE NUMBER) 1 Much better 2 Somewhat better 3 No change 4 Somewhat worse 5 Much worse 6 Don’t know …………(GO TO Q8) 7 Refused ……………...(GO TO Q8) 7. Why do you think so? Please use examples to support your response? (PROBE ON THE FOLLOWING FACTORS: WAS TRAINING ON USING THE vLMIS SUFFICIENT OR GOOD ENOUGH? WAS THERE A FIELD-BASED FOLLOW-UP TO vLMIS TRAINING FOR MENTORING AND SUPPORTIVE SUPERVISION OF YOUR USE OF THE SYSTEM AT YOUR WORKPLACE? DID THE vLMIS DASHBOARD INCLUDE OPTIONS THAT YOU NEEDED? DID THE vLMIS PROVIDE THE DATA THAT YOU NEEDED TO MAKE ROUTINE DECISIONS ABOUT MANAGING THE SUPPLY CHAIN FOR CONTRACEPTIVES, SUCH AS DELIVER LMIS: Final Evaluation Report 97 IDENTIFYING WEAKNESSES OR GAPS IN MEETING TARGETS, POSSIBLE PILFERING, ETC.?) Objective 2: Strengthen the environment for commodity security 8. What is your understanding of the term “commodity security”? (PROBE ABOUT SPECIFIC EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK￾OUTS) 9. In your experience, has the cLMIS system facilitated the safe storage of family planning commodities? (CIRCLE ONE NUMBER) 1. Yes 2. No Why do you think so? Please use example to support your response: 10. How in your experience, has the cLMIS system facilitated the safe storage of family planning commodities in the district stores? (GIVE AN EXAMPLE) 11. In your experience, has the cLMIS system facilitated the safe distribution of family planning commodities (preventing spoilage and pilferage to the market)? (CIRCLE ONE NUMBER) 1 Yes 2 No Why do you think so? Please use example to support your response: DELIVER LMIS: Final Evaluation Report 98 12. How in your experience, has the cLMIS system facilitated the safe distribution of family planning commodities (preventing spoilage and pilferage to the market)? (GIVE AN EXAMPLE) 13. In your experience, has the vLMIS system facilitated the safe storage of vaccines (adequate temperatures and space, preventing spoilage and pilferage to the market)? (CIRCLE ONE NUMBER) 1 Yes 2 No Why do you think so? Please use example to support your response: 14. In your experience, how has the vLMIS system facilitated safe storage of vaccines (adequate temperatures and space)? (GIVE AN EXAMPLE) 15. In your experience, has the vLMIS system facilitated the safe distribution of vaccines (preventing spoilage and pilferage to the market)? (CIRCLE ONE NUMBER) 1 Yes 2 No Why do you think so? Please use example to support your response: 16. How, in your experience, has the vLMIS system facilitated the safe distribution of vaccines (preventing spoilage and pilferage to the market)? (GIVE EXAMPLE) (ASK ONLY FROM MINISTRY OF NHS&RC AND POPULATION WELFARE WING RESPONDENTS) DELIVER LMIS: Final Evaluation Report 99 17. What sort of participation (cooperation, coordination, or collaboration) do you have on the cLMIS with your counterpart at the Ministry of National Health Services Research and Coordination (NHSR&C) or at the Population Welfare Wing (PWW) of the Ministry of Planning, Development and Reforms? Please describe an example. (PROBE WITH RESPECT TO SERVICE DELIVERY, ADVOCACY/DEMAND GENERATION, DATA REPORTING, DOUBLE COUNTING, ETC. FURTHER, PROBE ABOUT FACILITATION OR HINDERING FACTORS, AND THE SOURCE(S) OF SUCH FACTORS. PROBE ON ISSUES LIKE: WHETHER THERE ARE MEASURES TO REDUCE DISPARITIES BY COST-SHARING, ETC.; AND ON ISSUES LIKE WHETHER THERE ARE REGULARLY SCHEDULED JOINT DoH-PWD MEETINGS AT THE PROVINCE AND/OR DISTRICT LEVELS) (ASK ONLY OF MINISTRY OF NHS&RC AND POPULATION WELFARE WING RESPONDENTS) 18. In your experience, to what extent, if any, has the LMIS affected cooperation, coordination and collaboration between NHS&RC and PWW in service delivery, forecasting, advocacy/demand generation, data reporting, double counting, etc.? Would you say it has greatly improved, somewhat improved, stayed the same, somewhat worsened, or greatly worsened? (CIRCLE ONE NUMBER) 1 Greatly improved 2 Somewhat improved 3 Stayed the same 4 Somewhat worsened 5 Greatly worsened 6 Don’t know........................................................(GO TO Q22) 7 Refused to answer............................................(GO TO Q22) 19. Why do you think so? Please use examples to support your response. (WRITE AN ANSWER) DELIVER LMIS: Final Evaluation Report 100 (ASK Q20 ONLY OF RESPONDENTS WHO SELECTED OPTIONS 1, 2, 4, OR 5 IN Q18) 20. Which areas of participation (cooperation, coordination, and collaboration), if any, improved between NHS&RC and Population Welfare Wing (PWW) due to implementation of the LMIS? (CIRCLE ONE NUMBER) 1 Service delivery 2 Advocacy/demand generation 3 Data reporting 4 Double counting 5 Other (specify) _____________________________________________ 21. Why do you think so? Please use an example to support your response. (WRITE AN ANSWER) Objective 3: Increase knowledge management and dissemination 22. What steps, if any, do you believe need to be taken to scale up vLMIS and/or cLMIS trainings in the future? (WRITE AN ANSWER) 23. Does the LMIS play any role in your decision-making related to supply chain management? (CIRCLE ONE NUMBER) 1 Yes......................................(GO TO Q24) 2 No......................................(GO TO Q25) 24. Why and how does the LMIS have a role in your decision-making related to supply chain management? (WRITE AN ANSWER) DELIVER LMIS: Final Evaluation Report 101 25. In your experience, to what extent, if any, has LMIS affected decision-making related to supply chain management? Would you say decision-making related to supply chain management is much better, somewhat better, about the same, somewhat worse, or much worse because of the LMIS? (CIRCLE ONE NUMBER) 1 Much better.....................(GO TO Q26) 2 Somewhat better ...........(GO TO Q26) 3 About the same...............(GO TO Q26) 4 Somewhat worse............(GO TO Q26) 5 Much worse .....................(GO TO Q26) 6 Don’t know......................(GO TO Q27) 7 Refused to answer..........(GO TO Q27) 26. Why do you think so? Please use examples to support your response? (WRITE AN ANSWER) Evaluation Question No. 2: “What are the changes that could be made to ensure sustainability of the cLMIS and to strengthen data-driven decisions?” 27. About how many different people have held your current post between September 2012 and May 2016? (ENTER NUMBER OF PEOPLE OR CIRCLE -99 FOR DON’T KNOW) ________ (ENTER NUMBER) -99 Don’t know 28. Do you strongly agree, somewhat agree, somewhat disagree, or strongly disagree with the statement: “the cLMIS is essential for appropriate data-driven decisions about supply chain management”? (CIRCLE ONE NUMBER) 1 Strongly agree 2 Somewhat agree 3 Don’t know 4 Somewhat disagree 5 Strongly disagree DELIVER LMIS: Final Evaluation Report 102 29. Why do you think cLMIS does or does not play an essential role in facilitating data-driven decisions about supply chain management? (WRITE AN ANSWER) 30. In your experience, are the following kinds of data from cLMIS useful for understanding the quantity of family planning commodities in stock and the need, if any, for modifying procurement of contraceptives? (CIRCLE ALL THAT APPLY) 1 Average monthly consumption 2 Previous month’s demand 3 Projected contraceptive requirement module of cLMIS 4 Other cLMIS data (PLEASE SPECIFY) ______________________ 5 Other data which is not available in the cLMIS (PLEASE SPECIFY) ______________________ 31. In your opinion, is the cLMIS useful enough that it should be sustained and continued to be used as a data reporting tool? (CIRCLE ONE NUMBER) 1 Yes 2 No 32. Why, in your opinion, is the cLMIS useful/not useful enough to be sustained/continued as a data reporting tool? If it is not useful, what kind of reporting tool, if any, do you think should replace the cLMIS? (WRITE AN ANSWER) Useful – continue/sustain: Not useful – what tool should replace it: 33. In your opinion, what changes, if any, are required in the current system to improve the prospects that cLMIS will be sustainable? (WRITE AN ANSWER) (EXPLORE WITH REGARDS TO SUSTAINED AVAILABILITY OF INPUTS INCLUDING BUDGETARY RESOURCES, HUMAN RESOURCES, HARDWARE, AND TECHNICAL ASSISTANCE FOR MAINTAINENCE OF SOFTWARE, EVIDENCE THAT THE cLMIS IS DELIVER LMIS: Final Evaluation Report 103 USEFUL FOR REPORTING REQUIREMENTS, FOR FINANCIAL ALLOCATIONS, AND FOR OTHER DATA-DRIVEN DECISIONS ON MANAGEMENT OF THE SUPPLY CHAIN FOR CONTRACEPTIVE COMMODITIES) Evaluation Question No. 3: “Now, I will ask you some questions about the kind of changes which you think could be made to scale up the Vaccine Logistics Management Information System (vLMIS) in order to increase the “programmatic efficiency” of vaccine supply chain management.” 34. Do you have responsibility for making decisions about the management of the vaccine supply chain? (CIRCLE ONE NUMBER) 1 Yes 2 No (GO TO Q 36) 35. Which of the following kinds of vaccine supply chain management programmatic decision-making are you responsible for, including decisions about supporting the implementation, training, scale￾up, and other aspects of supply chain management tools like vLMIS? (READ AND CIRCLE ALL THAT APPLY) 1 Forecasting vaccine supply requirements 2 Assessing the accuracy of forecasts 3 Reducing the costs of forecasting errors 4 Selecting sources for vaccine purchasing 5 Procuring or purchasing selected vaccines 6 Storing vaccines 7 Distributing vaccines to health facilities 8 Monitoring vaccine quality 9 Monitoring vaccine coverage 10 Reporting on vaccine distribution 11 Reporting on vaccine supplies 12 Preventing stock-outs 13 Financially supporting training on supply chain management 14 Financially supporting vLMIS implementation 15 Vaccination service delivery 16 Other vaccine supply chain management decisions (describe): (PROBE ON WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE DELIVER LMIS: Final Evaluation Report 104 cLMIS/vLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM) 36. In your opinion, or based on your experience, to what extent has implementation of the vLMIS affected the programmatic efficiency of the vaccine supply chain? Would you say the vLMIS has greatly improved, somewhat improved, not affected, somewhat reduced, or greatly reduced the programmatic efficiency of the vaccine supply chain? (SELECT ONLY ONE ANSWER— PROBE TO FIND OUT HOW THE RESPONDENT DEFINES “EFFICIENCY” IN TERMS OF THE OPERATION OF THE SCM PROGRAM, AND HOW THEY THINK THAT THE vLMIS TOOL HAS, OR HAS NOT, AFFECTED VACCINE SCM PROGRAMMATIC EFFICIENCY) 1 Greatly improved 2 Somewhat improved 3 Stayed the same 4 Somewhat reduced 5 Greatly reduced 6 Don’t know (GO TO Q36.2) 7 Refused (GO TO Q36.2) 36.1 Why do you think so? Please use examples to support your response? (PROBE ON WHETHER THE COSTS OF OPERATING THE SYSTEM OF SUPPLY CHAIN MANAGEMENT (SCM) FOR VACCINES HAVE BEEN AFFECTED BY IMPLEMENTING THE vLMIS—HAVE SCM COSTS GONE UP, OR DOWN, OR STAYED THE SAME) 36.2 Do you think that the vLMIS should be scaled up? (WRITE AN ANSWER) 1 Yes 2 No………………... (GO TO Q36.3) (PROBE ON WHY, WHERE, AND HOW TO SCALE UP) Why: __________________________________________________________ DELIVER LMIS: Final Evaluation Report 105 Where: ________________________________________________________ How: __________________________________________________________ 36.3 Why do you think the vLMIS should not be scaled up? (PROBE FOR SPECIFIC WEAKNESSES WHICH, IF ADDRESSED, WOULD MAKE THE vLMIS WORTH SCALING UP) (WRITE AN ANSWER) Why not: ______________________________________________________ How can these problems be overcome: ________________________________ 37. In your opinion, or based on your experience, do you believe the vLMIS has greatly increased, somewhat increased, greatly decreased, somewhat decreased, or not affected the cost of effectively managing vaccine supply? (SELECT ONLY ONE ANSWER—PROBE TO FIND OUT HOW THE RESPONDENT DEFINES THE “COST OF EFFECTIVELY OPERATING THE vLMIS" AND THE “COST OF EFFECTIVELY MANAGING THE VACCINE SUPPLY CHAIN PROGRAM”, AND HOW THEY THINK THAT THE vLMIS TOOL HAS, OR HAS NOT, AFFECTED VACCINE SUPPLY CHAIN MANAGEMENT PROGRAMMATIC COSTS) 1 Greatly increased 2 Somewhat increased 3 Greatly decreased 4 Somewhat decreased 5 Not affected (GO TO Q37.2) 6 Don’t know (GO TO Q37.2) 7 Refused to answer (GO TO Q37.2) 37.1 Please tell me how and why you think that the vLMIS has increased or decreased the cost-efficiency of the vaccine supply chain. (WRITE AN ANSWER) DELIVER LMIS: Final Evaluation Report 106 37.2 How would you further improve the programmatic efficiency of the vaccine supply chain? (WRITE AN ANSWER) (ASK ONLY OF RESPONDENTS THAT ARE REPRESENTATIVES OF DONOR AGENCIES) 37.3 Without making any commitment at this time, what is your opinion about joining with the GOP and USAID in supporting further development and scale-up of the Logistics Management Information Systems for contraceptive commodities or vaccines? (WRITE AN ANSWER) Evaluation Question No. 4: “What best practices, innovations, and lessons learned can be applied to future programming for strengthening supply chain systems?” 38. Do you have responsibility for making decisions about the management of the supply chain for health system-related commodities? 2. Yes 3. No…………………..(GO TO Q40) 39. Are you responsible for any one or more of the following supply chain management programmatic decision-making? (READ AND CIRCLE ALL FOR WHICH THE RESPONDENT HAS RESPONSIBILITY) 1 Forecasting commodity supply requirements 2 Assessing the accuracy of forecasts 3 Reducing the costs of forecasting errors 4 Selecting sources for purchasing commodities 5 Procuring/purchasing selected commodities 6 Storing commodities 7 Distributing commodities to health facilities 8 Monitoring the quality of commodities 9 Monitoring the coverage of commodities 10 Reporting on the distribution of commodities 11 Reporting on the supplies of commodities DELIVER LMIS: Final Evaluation Report 107 12 Preventing stock-outs 13 Financially supporting training on supply chain management 14 Financially supporting SCM implementation 15 Service delivery for health commodities 16 Other supply chain management decisions (Describe)___________________________________________ INTERVIEWERS FIRST PLEASE READ THE FOLLOWING DEFINITIONS TO RESPONDENTS (NO NEED TO PROVIDE THE URLs): “Promising practices are defined as interventions showing progress toward improving health commodity supply chains.” (http://siapsprogram.org/wp￾content/uploads/2014/07/Intro_Acknowledgements-format.pdf and http://siapsprogram.org/publication/promising-practices-in-supply-chain￾management/) “Proven practices are defined as interventions with proven outcomes in improving health commodity supply chains.” (http://siapsprogram.org/wp￾content/uploads/2014/07/8_Proven-Practices-final.pdf) 40. Using the following definitions for promising or proven practices in supply chain management, please describe one or more examples of the following supply chain management practices that have been developed or strengthened by the DELIVER project in Pakistan? 40.1 Please describe a promising or proven practice for the quantification of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: 40.2 Please describe a promising or proven practice for the procurement of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: 40.3 Please describe a promising or proven practice for the warehousing of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: DELIVER LMIS: Final Evaluation Report 108 40.4 Please describe a promising or proven practice for the distribution of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: 40.5 Please describe a promising or proven practice for the service delivery and utilization of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: 40.6 Please describe a promising or proven practice for the data management of health commodities that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: 40.7 Please describe a promising or proven practice for human resource management in supply chain systems that you learned about through the DELIVER project and that you think should be scaled up in Pakistan: DELIVER LMIS: Final Evaluation Report 109 District Level Group Discussions with Supervisors of Vaccinators/Assistant Superintendent Vaccination (ASV) INSTRUMENT FOR DELIVER LMIS EVALUATION: For use in FGDs at district levels with supervisors of vaccinators/assistant superintendent vaccination (ASV) Demographics of Group Discussion: تاریخ (دن-ماہ-سال( : ______________________________________:(Day-Month-Year (Date انٹرویو لینے والے کا نام: :name s’Moderator نوٹس لینے والے کا نام: :name s’taker Note Some Guidelines for Arranging Participants 1. In all districts, one group of ASVs should be organized. 2. The Focus Group Discussion (FGD) should be conducted at the district level and the ASVs should be engaged for FGDs through the concerned district program manager. 3. It is requested not to make the framework/guidelines too rigid as a FGD is best conducted with some flexibility and spontaneity. 4. Depending upon the situation during the FGD, the sector specialist may decide to add on the spot questions, and/or decline from asking some. 5. Some questions below may seem like repetition, but this is deliberate. Based on our experience, some critical questions asked in the beginning may not generate satisfactory responses. However, as the discussion moves on, the same questions repeated later are received with more enthusiasm, due to rapport created between the FGD researcher and the group. 6. If participants are sitting on the floor, all team members should also sit on the floor. Chairs for only team members or sector specialist should be avoided. 7. The ASVs are taking time out of their daily routine to participate in FGDs. It is only common courtesy that some refreshment should be offered. It does not mean an elaborate lunch or high tea. Transportation should also be provided to the participants. A per-diem should also be provided to each participant if this is consistent with the policies of GOP and USAID. FGD Moderation Guidelines The FGDs should follow the below guidelines (FIRST, ASK ALL PARTICIPANTS TO KINDLY PUT THEIR CELL PHONES ON “SILENT”, AND ALSO REQUEST THAT PARTICIPANTS KINDLY NOT HAVE “SIDE-BAR” CONVERSATIONS BECAUSE IT IS IMPORTANT FOR EVERYONE TO HEAR EVERYONE ELSE’S OPINIONS) 1. The participants should be briefed about the project and the purpose of conducting the FGD. 2. They should be informed about the confidentiality and an informed consent should be obtained. 3. Participation of all the intended participants should be ensured. 4. The moderator should broadly follow the below sequence of questions a. Opening questions b. Follow-up questions DELIVER LMIS: Final Evaluation Report 110 c. Probing questions d. Prompting questions 5. The moderator should ensure that note taking and recording are done. 6. The moderator should effectively use silence and note/record non-verbal communication. 7. The moderator should take a note of time. 8. The moderator should thank the participants in the end. 9. After the FGD, the team members should meet for completion/finalization of notes. Introduction: My name is ______________________. I work for a research organization based in Islamabad. As explained in the official letters from Government (DoH/DPW), we are conducting a final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan (GOP) continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain for health commodities (vaccines). We will ask a few questions about your experience and your recommendations regarding:  Effectiveness of the LMIS for managing the supply chain of medicines, such as and vaccines.  Sustainability of the Logistics Management Information System for vaccine (vLMIS) and strengthening of data-driven decisions on supply chain management. This group discussion will take approximately 45 minutes to 1 hour. We will treat the information we collect as confidential and will never associate the information with your name. میرا نام ۔۔۔۔۔۔۔۔۔۔۔۔۔ ہے، میں اسالم آباد میں قائم ایک تحقیقی ادارے (MSI (International systems Managementکے ساتھ کام کر رہا ہوں۔ جیسے کے اجازت نامے میں بتایا گیا ہے کہ ہم JSI کی طرف سے کیے جانے والے project DELIVERکے LMIS کے حوالے سے تحقیق کر رہے ہیں۔ اس تحقیق کے نتائج سے حاصل شدہ معلومات حکوم ِت پاکستان کو فیملی پالننگ کے اشیاء کی عوام تک ترسیل کے طریقہ کار کو مضبوط اور بہتر بنانے اور اس کے نتیجے میں ، زچہ اور بچہ کی صحت کی بہتر ی کے لیے مددگار ثابت ہونگی۔ میں آپ سے نیچے بیان کردہ نکات کے متعلق کچھ سوال کرنا چاہونگا۔ .1 ویکسین کی ترسیل اور انتظامی امور پر LMISکے اثر ات ۔ .2 vLMIS سے حاصل کردہ معلومات کی بنیاد پر ویکسین کے ترسیل کے نظام کو باقی عالقوں میں پھیالنا۔ یہ گروپ مباحثہ تقریبا 45 منٹ سے 01 گھنٹے تک جاری رہےگا۔ اس سے حاصل کردہ معلومات مکمل طور پر صیغہ راز میں رکھا جائیگا اور یہ معلومات کہیں بھی آپ کے نام سے منسوب نہیں کیا جائیگا۔ May I have your permission to proceed with the group discussion? کیا مجھے گروپ مباحثہ کا آغاز کرنے کی اجازت ہے؟ ہاں Yes No (THOSE WHO DO NOT WANT TO PARTICIPATE MAY LEAVE. STOP THE GROUP DISCUSSION IF ALL PARTICIPANTS DECIDE TO نہیں (گروپ مباحثہ روک دیں اگرشرکاء میں سے کوئی بھی اجازت نہ دے( (GO May I have your permission to record the group discussion to ensure the completeness and accuracy of your opinions? The recording will remain confidential and kept at our office. ا قیمتی معلومات سے مکمل طور پر مستفید ہونے کے لیے ہم گروپ مباحثہ ریکارڈ کرنا چاہتے ہیں۔ کیا ہمیں ٓپ سے حاصل کردہ گروپ مباحثہ ریکارڈ کرنے کی اجازت ہے؟ DELIVER LMIS: Final Evaluation Report 111 Yes (SWITCH ON RECORDER AND START THE GROUP DISCUSSION) No (START THE GROUP DISCUSSION) DELIVER LMIS: Final Evaluation Report 112 FOCUS GROUP DISCUSSION (PRINT IN ADVANCE AND CIRCULATE THE FOLLOWING TABLE, ENSURING THAT ALL FGD PARTICIPANTS FILL IN THE INFORMATION CLEARLY) BEFORE LEAVING (BRING PENS AND A CLIPBOARD IN CASE PARTICIPANTS SIT ON THE FLOOR) District Date RESP Name Years of Service as ASV Tehsil Union Council 1 2 3 4 5 6 7 8 9 10 DELIVER LMIS: Final Evaluation Report 113 INSTRUMENT FOR DISTRICT LEVEL VACCINATION SERVICE VACCINATORS’ SUPERVISORS/ASSISTANT SUPERINTENDENT VACCINATION (ASV) 1. Are you aware of the Logistics Management Information System for Vaccines (vLMIS)? کیا ا vLMIS کے بارے میں جانتے ہیں؟ ٓپ (ASK PARTICIPANTS TO RAISE HANDS IF THEY KNOW ABOUT vLMIS. ENTER THE COUNT BELOW) Number of hands raised = ________________ Total number of participants = ________________ 2. In order to ensure the availability of vaccines that are requested by the clients of your vaccinators, a supply chain system is required, and there are many factors involved. In your experience, what are the most important factors that affect the supply of quality vaccines to vaccinators who provide vaccination services at the community level? (ASK FOR THE TOP TWO OR THREE FACTORS THAT HELP ENSURE THAT ALL OF THE REQUIRED/REQUESTED VACCINE COMMODITIES ARE AVAILABLE FOR YOUR VACCINATORS TO PROVIDE TO THEIR CLIENTS, AND THE TOP 2 OR 3 FACTORS THAT ADVERSELY AFFECT THE AVAILABILITY OF VACCINES) ُس کے بہت سے Vaccinatorکے پاس طلب شدا ویکسین کی موجودگی کو یقینی بنانے کے لیے ایک ترسیل کے نظام کا ہونا ضروری ہے اور ا عناصر ہو سکتے ہیں، آپ اپنے تجربات کی بنیاد پر بتائیں کہ ایسے کون سے اہم عناصر ہیں جو ویکسین کے ضلعی سٹورسےآپ تک اورآپ سے vaccinator تک ترسیل پر اثرات ُمرتب کر سکتے ہیں۔ ۔ )سب سے اہم دو یا تین عوامل کے بارے میں معلوم کریں جو اس بات کو یقینی بناتے ہیں کہ مطلوبہ ویکسینز کالئنٹس کو فراہم کرنے کے لئے ویکسینیٹرز کے پاس موجود/دستیاب ہیں، اور ان اہم دو یا تین عناصر کے بارے میں معلوم کریں جو ان ویکسینز کی موجودگی/دستیابی پر اثر انداز ہوتے ہیں(۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 114 3. In your experience, how, if at all, has the vLMIS changed the management or operation of the supply chain of the vaccines required by the clients of your vaccinators? (PROBE ON WHETHER THE REPORTING IN TO THE vLMIS BY VACCINATORS AND FEEDBACK ON THESE REPORTS HAS CONTRIBUTED TO IMPROVEMENTS IN THE SUPPLY OF VACCINES) ا میں vLMIS کس حد تک ویکسین کے ترسیل کے نظام میں تبدیلی کا سبب بنا ہے؟ )اس بارے میں جانئیے کہ آیا ، LMIS ٓپ کے تجربے پر vaccinatorsکی پورٹنگ اور ان رپورٹس پر دئیے گئے آراء کی وجہ سے یہ ویکسینزکی سپالئی میں بہتری کا سبب بنے ہیں؟( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 4. Why do you think so? Please describe an example to support your response. (PROBE ON THE FACTORS THAT WERE MENTIONED IN Q2, AND THE POSITIVE/NEGATIVE RESPONSES OBTAINED IN Q3) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے ۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 115 RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 5. We know from the LMIS system that sometimes vaccinators in the districts have problems in being able to submit monthly reports that are complete or on time. In your experience, what are the factors that enable or that prevent or constrain you from being able to report on time, every month, with completeness and accuracy? (PROBE FOR THE FACTORS THAT AFFECT TIMELINESS, COMPLETENESS AND ACCURACY OF THE REPORTS; WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE vLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM. FOR EVERY FACTOR—POSITIVE OR NEGATIVE— PROBE FOR EXPERIENCE ON/IDEAS FOR EXPANDING POSITIVE/OVERCOMING NEGATIVE FACTORS) LMISکے نظام سے موصول شدہ معلومات اس طرف اشارہ کرتی ہیں کہ کچھ ضلعوں میں vaccinators کو وقت پر ماہانہ رپورٹ جمع کرانے میں دشواری/مسائل کا سامنا کرنا پڑتا ہے۔کیا آپ اپنے تجربے کی روشنی میں ان عناصر کے بارے میں بتا سکتی ہیں جو آپ کو ماہانہ رپورٹ)مکمل اور غلطیوں کے بغیر( وقت پر جمع کرانے /نہ کرانے میں اثر انداز ہوتے ہیں۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 116 6. What is your understanding of the term “commodity security”? (IF NO REASONABLE ANSWERS, PROVIDE A DEFINITION AND THEN ویکسینز کی حفاظت کے بارے میں آپ کیا جانتے ہیں؟ PROBE ABOUT SPECIFIC EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, EXPIRATION, SPOILAGE, AND STOCK-OUTS) )جس میں ان کی دستیابی، صفائی کے ساتھ حفاظت ، چوری، تنسیخ، خرابی اور اسٹاک میں موجود نہ ہونے سے متعلق مثالوں کے بارے میں جانیئے( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 7. In your experience, during the past 3-5 years, what effect, if any, has your reporting in to the vLMIS system had on the security of vaccines in the facilities and the district stores? (PROBE ABOUT SPECIFIC POSITIVE/NEGATIVE EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, EXPIRATION, SPOILAGE, AND STOCK-OUTS) ا ۓ میں کیا گزشتہ3 سے 5 سال کے دوران آپ کی vLMISرپورٹنگ ویکسین کی ضلعی اسٹورز میں بحفاظت ذخیرہ کرنے میں ٓپ کی را مددگار ثابت ہوئی ہے؟ ؟ )مخصوص مثبت/منفی مثالوں کے بارے میں جانئیے: دستیابی/موجودگی، صفائی کے ساتھ حفاظت، چوری، تنسیخ، خرابی اور اسٹاک میں موجود نہ ہونا( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 117 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 8. Why do you think so? Please use examples to support your response. (RECORD AND ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا (ANSWERS THE WRITE میں وضاحت کیجئے۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 118 9. In your experience, what effect, if any, have your reports in to the vLMIS system had on the timeliness, accuracy, and safety of the supply and distribution of vaccine? (PROBE ON WHETHER ASVs RECEIVE USEFUL FEEDBACK AND APPROPRIATE (RE-) SUPPLIES OF REQUESTED COMMODITIES FROM THE DISTRICT SUPERVISOR, AND ASK ABOUT SPOILAGE AND PILFERAGE) ا ۓ میں کیا vLMIS ، ویکسین کی فراہمی اور vaccinatorsکو بروقت تقسیم کو یقینی بنانے میں مددگار ثابت ہوا ہے؟ )جانئیے کہ ٓپ کی را آیا ASVsضلعی Supervisorسےمفید آراء وصول کرتے ہیں ، اور مطلوبہ ویکسین کی فراہمی کرتے ہیں؟ مزید یہ کہ خرابی اور چوری سے متعلق پوچھ گچھ کرتے ہیں یا نہیں؟( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 10. Why do you think so? Please use examples to support your response. (PROBE ON WHETHER AND HOW THE vLMIS SYSTEM IS CONTRIBUTING TO EFFECTIVE AND EFFICIENT BACK-AND-FORTH COMMUNICATION FORVACCINE SUPPLY CHAIN MANAGEMENT) ٓٓ پ کے ایسا سوچنے کی کیا وجہ ہے ؟ براۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 119 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 11. Have you received any training from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا آپ / JSI DELIVER کی طرف سے کوئی ٹریننگ حاصل کر ُچکے ہیں؟ (ASK PARTICIPANTS TO RAISE HANDS IF THEY HAD RECEIVED TRAINING. NOTE DOWN THE COUNT) Number of hands raised = ________________ Total number of participants =________________ 12. On which of the following topics, if any, did you receive training? (READ THE NAME OF EACH TRAINING AND ASK PARTICIPANTS TO RAISE THEIR HANDS IF THEY RECEIVED THE TRAINING. ENTER THE COUNT FOR EACH TRAINING IN THE TABLE BELOW) مندرجہ ذیل موضوعات میں سے آپ نے کس پر ٹریننگ حاصل کی ہوئی ہے؟ o Principles of supply chain management ترسیل کے نظام کے بنیادی اصول o Purpose of the vLMIS in supply chain management ویکسین کی ترسیل کے نظام متعلق ےLMIS vکے مقاصد o Entering data in the vLMIS vLMIS پر اعداد و شمار کے اندراج کے لیے o Other vLMIS training (describe below) vLMISکےمتعلق کوئی اور ٹریننگ Describe: 13. Did the training provide you with new information? If yes, please provide an example. (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کیا ٹریننگ سےآپکو کوئی نئی معلومات حاصل ہوئی؟ RESP No.1 o Yes Example: __________________________________________________________ o No DELIVER LMIS: Final Evaluation Report 120 RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No RESP No.9 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No DELIVER LMIS: Final Evaluation Report 121 Was the training useful in your work? If yes, please provide an example. (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کیا ٹریننگ آپ کے کام میں معاون ثابت ہوئی ؟ RESP No.1 o Yes Example: __________________________________________________________ o No RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No DELIVER LMIS: Final Evaluation Report 122 RESP No.9 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No 14. Did the DELIVER project’s training on the use of the vLMIS improve your skills in supply chain management (i.e. reporting, requisition, and availability)? Why do you think so? Please use examples to support your response. کیا Project Deliverکے LMISکی ٹریننگ کی وجہ سے ویکسینز کے ترسیل کے نظام )رپورٹننگ، درخواست اور موجودگی( سے اپ کے ایسا سوچنے کی کیا وجہ ہے ؟ براۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت ئی ؟ ٓ متعلق آپکی مہارت میں بہتری آ کیجئے ۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 15. Which information, if any, do you use to determine the quantity of vaccines commodities you ویکسین کی درکار تعدادکا تعین کرنےکے لیئے آپ کن معلومات کا استعمال کرتے ہیں؟?need RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 123 RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 16. Why in your experience is vLMIS useful or not useful? (WRITE AN ANSWER) ِل استعمال ہے / نہیں ہے؟ آپ کے خیال میں vLMISکس وجہ سے قاب RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 124 District Level Group Discussions with Supervisors of Family Planning Service Providers/Lady Health Supervisors (LHS) INSTRUMENT FOR DELIVER LMIS EVALUATION: For use in FGDs at district levels with supervisors of family planning service providers/lady health supervisors (LHS) Demographics of Group Discussion: تاریخ (دن-ماہ-سال(: ______________________________________:(Day-Month-Year (Date انٹرویو لینے والے کا نام: :name s’Moderator نوٹس لینے والے کا نام: :name s’taker Note Some Guidelines for Arranging Focus Group Discussions (FGDs) for LHSs 1. In all districts, one group of LHSs should be organized. 2. The Focus Group Discussion (FGD) should be conducted at the EDO Health office and the LHSs in the district should be engaged for FGDs through the concerned District Program Manager. 3. It is requested not to make the framework/guidelines too rigid as a FGD is best conducted with some flexibility and spontaneity. 4. Depending upon the situation during the FGD, the sector specialist may decide to add on the spot questions, and/or decline from asking some. 5. Some questions below may seem like repetition, but this is deliberate. Based on our experience, some critical questions asked in the beginning may not generate satisfactory responses. However, as the discussion moves on, the same questions repeated later are received with more enthusiasm, due to rapport created between the FGD researcher and the group. 6. If participants are sitting on the floor, all team members should also sit on the floor. Chairs for only team members or sector specialist should be avoided. 7. The LHSs are taking time out of their daily routine to participate in FGDs. It is only common courtesy that some refreshment should be offered. It does not mean an elaborate lunch or high tea. Transportation should also be provided to the LHSs. A per diem should also be provided to each LHS if this is consistent with the policies of GOP and USAID. FGD Moderation Guidelines The FGDs should follow the below guidelines (FIRST, ASK ALL PARTICIPANTS TO KINDLY PUT THEIR CELL PHONES ON “SILENT”, AND ALSO REQUEST THAT PARTICIPANTS KINDLY NOT HAVE “SIDE-BAR” CONVERSATIONS BECAUSE IT IS IMPORTANT FOR EVERYONE TO HEAR EVERYONE ELSE’S OPINIONS) 1. The participants should be briefed about the project and the purpose of conducting the FGD. 2. They should be informed about the confidentiality and an informed consent should be obtained. 3. Participation of all the intended participants should be ensured. 4. The moderator should broadly follow the below sequence of questions a. Opening questions b. Follow-up questions DELIVER LMIS: Final Evaluation Report 125 c. Probing questions d. Prompting questions 5. The moderator should ensure that note taking and recording are done. 6. The moderator should effectively use silence and note/record non-verbal communication. 7. The moderator should take a note of time. 8. The moderator should thank the participants in the end. 9. After the FGD, the team members should meet for completion/finalization of notes. Introduction: My name is ______________________. I work for a research organization based in Islamabad. As explained in the official letters from Government (DoH/PWD), we are conducting a final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan (GOP) continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain for health commodities (family planning commodities). We will ask a few questions about your experience and your recommendations regarding:  Effectiveness of the LMIS for managing the supply chain of medicines, such as family planning contraceptive commodities.  Sustainability of the Logistics Management Information System for contraceptives (cLMIS) and strengthening of data-driven decisions on supply chain management. This group discussion will take approximately 45 minutes to 1 hour. We will treat the information we collect as confidential and will never associate the information with your name. میرا نام ۔۔۔۔۔۔۔۔۔۔۔۔۔ ہے، میں اسالم آباد میں قائم ایک تحقیقی ادارے (MSI (International systems Managementکے ساتھ کام کر رہا ہوں۔ جیسے کے اجازت نامے میں بتایا گیا ہے کہ ہم JSI کی طرف سے کیے جانے والے project DELIVERکے LMIS کے حوالے سے تحقیق کر رہے ہیں۔ اس تحقیق کے نتائج سے حاصل شدہ معلومات حکوم ِت پاکستان کو فیملی پالننگ کے اشیاء کی عوام تک ترسیل کے طریقہ کار کو مضبوط اور بہتر بنانے اور اس کے نتیجے میں ، زچہ اور بچہ کی صحت کی بہتر ی کے لیے مددگار ثابت ہونگی۔ میں آپ سے نیچے بیان کردہ نکات کے متعلق کچھ سوال کرنا چاہونگا۔ .1فیملی پالننگ سے متعلق اشیاء کی ترسیل اور انتظامی امور پر LMIS کے اثر ات ۔ .2 cLMIS سے حاصل کردہ معلومات کی بنیاد پر فیملی پالننگ کے اشیاء کی ترسیل کے نظام کو باقی عالقوں میں پھیالنا ۔ یہ گروپ مباحثہ تقریبا 45 منٹ سے 01 گھنٹے تک جاری رہےگا۔ اس سے حاصل کردہ معلومات مکمل طور پر صیغہ راز میں رکھا جائیگا اور یہ معلومات کہیں بھی آپ کے نام سے منسوب نہیں کیا جائیگا۔ May I have your permission to proceed with the group discussion? کیا مجھے گروپ مباحثہ کا آغاز کرنے کی اجازت ہے؟ ہاں Yes No (THOSE WHO DO NOT WANT TO PARTICIPATE MAY LEAVE. STOP THE GROUP DISCUSSION IF ALL PARTICIPANTS DECIDE TO نہیں (گروپ مباحثہ روک دیں اگرشرکاء میں سے کوئی بھی اجازت نہ دے( (GO DELIVER LMIS: Final Evaluation Report 126 May I have your permission to record the group discussion to ensure the completeness and accuracy of your opinions? The recording will remain confidential and kept at our office. ا قیمتی معلومات سے مکمل طور پر مستفید ہونے کے لیے ہم گروپ مباحثہ ریکارڈ کرنا چاہتے ہیں۔ کیا ہمیں ٓپ سے حاصل کردہ گروپ مباحثہ ریکارڈ کرنے کی اجازت ہے؟ Yes (SWITCH ON RECORDER AND START THE GROUP DISCUSSION) No (START THE GROUP DISCUSSION) DELIVER LMIS: Final Evaluation Report 127 FOCUS GROUP DISCUSSION (PRINT IN ADVANCE AND CIRCULATE THE FOLLOWING TABLE, ENSURING THAT ALL FGD PARTICIPANTS FILL IN THE INFORMATION CLEARLY) BEFORE LEAVING (BRING PENS AND A CLIPBOARD IN CASE PARTICIPANTS SIT ON THE FLOOR) District Date RESP Name Years of Service as LHS Tehsil Union Council 1 2 3 4 5 6 7 8 9 10 DELIVER LMIS: Final Evaluation Report 128 INSTRUMENT FOR DISTRICT LEVEL FAMILY PLANNING SERVICE LADY HEALTH SUPERVISORS (LHS) – DEPARTMENT OF HEALTH (DOH) 1. Are you aware of the Logistics Management Information System for contraceptives (cLMIS)? کیا ا cLMISکے بارے میں جانتے ہیں؟ ٓپ (ASK PARTICIPANTS TO RAISE HANDS IF THEY KNOW ABOUT cLMIS. ENTER THE COUNT BELOW) Number of hands raised = ________________ Total number of participants = ________________ 2. In order to ensure the availability of modern family planning contraceptive commodities that are requested by the clients of your LHWs (e.g. condoms, pills, contraceptives injections and Copper￾T), a supply chain system is required, and there are many factors involved. In your experience, what are the most important factors that affect the supply of modern family planning contraceptives to you and the LHWs who provide family planning services at the community level? (ASK FOR THE TOP TWO OR THREE FACTORS THAT HELP ENSURE THAT ALL THE REQUIRED/REQUESTED MODERN CONTRACEPTIVE COMMODITIES ARE AVAILABLE FOR YOUR LHWs TO PROVIDE TO THEIR CLIENTS, AND THE TOP TWO OR THREE FACTORS THAT ADVERSELY AFFECT THE AVAILABILITY OF THESE MODERN CONTRACEPTIVES) ُس کے بہت سے عناصر ہو فیملی پالننگ کی جدید اجناس کی موجودگی کو یقینی بنانے کے لیے ایک ترسیل کے نظام کا ہونا ضروری ہے اور ا سکتے ہیں، اپنے تجربات کی بنیاد پر بتائیں کہ ایسے کون سے اہم عناصر ہیں جو ان اشیاء کے آپ تک اورآپ سے LHWs تک ترسیل پر اثرات ُمرتب کر سکتے ہیں۔ )سب سے اہم دو یا تین عوامل کے بارے میں معلوم کریں جو اس بات کو یقینی بناتے ہیں کہ مطلوبہ جدید اشیاء کالئنٹس کو فراہم کرنے کے لئے LHWsکے پاس موجود/دستیاب ہیں، اور ان اہم دو یا تین عناصر کے بارے میں معلوم کریں جو ان اشیاء کی موجودگی/دستیابی پر اثر انداز ہوتے ہیں(۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 129 3. In your experience, how, if at all, has the cLMIS changed the management or operation of the supply chain of the modern family planning contraceptives requested by the clients of your LHWs? (PROBE ON WHETHER THE LHSs’ REPORTING IN TO THE ASSISTANT DISTRICT COORDINATOR (ADC) HAS CONTRIBUTED TO IMPROVEMENTS IN THE SUPPLY OF MODERN CONTRACEPTIVE COMMODITIES…NB re DATA FLOW: AFTER REVIEWING ALL OF THE LHSs’ REPORTS FOR QUALITY, THE ADC SENDS THE REPORTS TO THE DATA-ENTRY OPERATOR AND S/HE AND THE ADC ARE RESPONSIBLE FOR FEEDBACK & ACTIONS BACK TO THE LHSs ON THESE REPORTS) ا میں cLIMS کیسے فیملی پالننگ کی اشیاء کے ترسیل کے نظام میں تبدیلی کا سبب بنا ہے؟ ٓپ کے تجربے RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 4. Why do you think so? Please describe an example to support your response. (PROBE ON THE FACTORS THAT WERE MENTIONED IN Q2, AND THE POSITIVE/NEGATIVE RESPONSES OBTAINED IN Q3) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے ۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 130 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 5. We know from the LMIS system that sometimes LHSs in the districts have problems in being able to submit monthly reports that are complete or on time. In your experience, what are the factors that enable, or that prevent, or that constrain you from being able to report on time, every month, with completeness and accuracy? (PROBE FOR THE FACTORS THAT AFFECT TIMELINESS, COMPLETENESS AND ACCURACY OF THE REPORTS; WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE cLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM. FOR EVERY FACTOR—POSITIVE OR NEGATIVE— PROBE FOR EXPERIENCE ONIDEAS FOR EXPANDING POSITIVE/OVERCOMING NEGATIVE FACTORS) LMISکے نظام سے موصول شدہ معلومات اس طرف اشارہ کرتی ہیں کہ کچھ ضلعوں میں LHSs کو وقت پر ماہانہ رپورٹ جمع کرانے میں دشواری/مسائل کا سامنا کرنا پڑتا ہے۔کیا آپ اپنے تجربے کی روشنی میں ان عناصر کے بارے میں بتا سکتی ہیں جو آپ کو ماہانہ رپورٹ)مکمل اور غلطیوں کے بغیر( وقت پر جمع کرانے /نہ کرانے میں اثر انداز ہوتے ہیں۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 131 RESP No.10 _____________________________________________________________ 6. What is your understanding of the term “commodity security”? (IF NO REASONABLE ANSWERS, PROVIDE A DEFINITION AND THEN فیملی پالننگ کی اشیاء کی حفاظت کے بارے میں آپ کیا جانتے ہیں؟ PROBE ABOUT SPECIFIC EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK-OUTS) )جس میں ان اشیاء کی دستیابی، صفائی کے ساتھ حفاظت ، چوری اور اسٹاک میں موجود نہ ہونے سے متعلق مثالوں کے بارے میں جانیئے( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 7. In your experience, during the past 3-5 years, what effect, if any, has your reporting in to the cLMIS system had on the security of modern family planning contraceptive commodities in the health facility stores? (PROBE ABOUT SPECIFIC POSITIVE/NEGATIVE EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK-OUTS) ا ۓ میں کیا گزشتہ3 سے 5 سال کے دوران آپ کی cLMISپر رپورٹننگفیملی پالننگ کی جدید اشیاء کی ضلعی اسٹورز ٓپ کی را میں بحفاظت ذخیرہ کرنے میں مددگار ثابت ہوئی ہے؟ )مخصوص مثبت/منفی مثالوں کے بارے میں جانئیے: دستیابی/موجودگی، صفائی کے ساتھ حفاظت، چوری، اور اسٹاک میں موجود نہ ہونا( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 132 RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 8. Why do you think so? Please use examples to support your response. (RECORD AND ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا (ANSWERS THE WRITE میں وضاحت کیجئے۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 133 9. In your experience, what effect, if any, have your reports in to the cLMIS system had on the timeliness, accuracy, and safety of the supply and distribution of modern family planning contraceptive commodities? (PROBE ON WHETHER LHSs RECEIVE USEFUL FEEDBACK AND APPROPRIATE (RE-) SUPPLIES OF REQUESTED COMMODITIES FROM THE ASSISTANT DISTRICT COORDINATOR, AND ASK ABOUT SPOILAGE AND PILFERAGE) ا ۓ میں کیا cLMIS پر رپورٹنگ فیملی پالننگ کی جدید اشیاء کی فراہمی اور LHWsکو بروقت تقسیم کو یقینی بنانے میں مددگار ٓپ کی را ثابت ہو ئی ہے؟ )جانئیے کہ آیا ,Coordinator District Assistant LHSsسےمفید آراء وصول کرتے ہیں ، اور مطلوبہ اشیاء کی فراہمی کرتے ہیں؟ مزید یہ کہ خرابی اور چوری سے متعلق پوچھ گچھ کرتے ہیں یا نہیں؟( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 10. Why do you think so? Please use examples to support your response. (PROBE ON WHETHER AND HOW THE cLMIS SYSTEM IS CONTRIBUTING TO EFFECTIVE AND EFFICIENT BACK-AND-FORTH COMMUNICATION FOR CONTRACEPTIVE SUPPLY CHAIN MANAGEMENT) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 134 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 11. Have you received any training from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا آپ / JSI DELIVER کی طرف سے کوئی ٹریننگ حاصل کر ُچکے ہیں؟ (ASK PARTICIPANTS TO RAISE HANDS IF THEY HAD RECEIVED TRAINING. NOTE DOWN THE COUNT) Number of hands raised = ________________ Total number of participants =________________ 12. On which of the following topics, if any, did you receive training? (TELL EACH TRAINING NAME AND ASK PARTICIPANTS TO RAISE HANDS IF THEY HAD RECEIVED THAT TRAINING. NOTE DOWN THE COUNT FOR EACH TRAINING) مندرجہ ذیل موضوعات میں سے آپ نے کس پر ٹریننگ حاصل کی ہوئی ہے؟ o Principles of supply chain management ترسیل کے نظام کے بنیادی اصول o Purpose of the cLMIS in supply chain management فیملی پالننگ کی اشیاء کے ترسیل کے نظام کے متعلق cLMIS کے مقاصد o Entering data in the cLMIS اعداد و شمار کے اندراج کے لیئے cLMIS کا استعمال o Other cLMIS training (describe below) cLMIS کےمتعلق کوئی اور ٹریننگ Describe: 13. Did the training provide you with NEW INFORMATION? If yes, please provide an example: (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کوئی نئی معلومات حاصل ہوئی؟ کیا حاصل کردہ ٹریننگ سے آپکو RESP No.1 o Yes Example: __________________________________________________________ o No DELIVER LMIS: Final Evaluation Report 135 RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No RESP No.9 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No DELIVER LMIS: Final Evaluation Report 136 Was the training USEFUL IN YOUR WORK? If yes, please provide an example: (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کیا حاصل کردہ ٹریننگ آپ کے کام میں معاون ثابت ہوئی؟ RESP No.1 o Yes Example: __________________________________________________________ o No RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No RESP No.9 DELIVER LMIS: Final Evaluation Report 137 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No 14. Did the DELIVER project’s training on the use of the cLMIS improve your skills in supply chain management (i.e. reporting, requisition, and availability of modern contraceptives)? Why do you think so? Please use examples to support your response. یں کے ٰ اپنے تجربہ کی بنیاد پر بتائ Project DELIVERکے LMISکی ٹریننگ کی وجہ سے فیملی پالننگ کے اشیاءکی ترسیل کے نظام ئی ؟ آپ کے ایسا سوچنے کی کیا وجہ ہے ؟ براۓ مہربانی اپنے )رپورٹننگ، درخواست اور موجودگی( سے متعلق آپکی مہارت میں بہتری آ دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 15. Which information, if any, do you use to determine the quantity of modern family planning contraceptive commodities you need? فیملی پالننگ کی اشیاء کی درکار تعدادکا تعین کرنےکے لیئے آپ کس معلومات کا استعمال کرتے ہیں؟ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 138 RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 16. Why in your experience is cLMIS useful or not useful? (WRITE AN ANSWER) ِل استعمال ہے / نہیں ہے؟ آپ کے خیال میں cLMISکس وجہ سے قاب RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 139 District Level Group Discussions with Family Welfare Workers (FWWs) – Population Welfare Department (PWD) INSTRUMENT FOR DELIVER LMIS EVALUATION: For use in FGDs at district levels with Family Welfare Workers (FWWs)-Population Welfare Department (PWD) Demographics of Group Discussion: تاریخ (دن-ماہ-سال(: ______________________________________:(Day-Month-Year (Date انٹرویو لینے والے کا نام: :name s’Moderator نوٹس لینے والے کا نام: :name s’taker Note Some Guidelines for Arranging Focus Group Discussions (FGDs) for FWW Participants 1. In all districts, one group of FWWs should be organized. 2. The Focus Group Discussion (FGD) should be conducted at the office of the District Population Welfare Officer (DPWO), and the FWWs in the district should be engaged for FGDs through the concerned DPWO. 3. It is requested not to make the framework/guidelines too rigid as a FGD is best conducted with some flexibility and spontaneity. 4. Depending upon the situation during the FGD, the sector specialist may decide to add on the spot questions, and/or decline from asking some. 5. Some questions below may seem like repetition, but this is deliberate. Based on our experience, some critical questions asked in the beginning may not generate satisfactory responses. However, as the discussion moves on, the same questions repeated later are received with more enthusiasm, due to rapport created between the FGD researcher and the group. 6. If participants are sitting on the floor, all team members should also sit on the floor. Chairs for only team members or sector specialist should be avoided. 7. The FWWs are taking time out of their daily routine to participate in FGDs. It is only common courtesy that some refreshment should be offered. It does not mean an elaborate lunch or high tea. Transportation should also be provided to the FGD participants. Per-diem should also be provided to each FWW participant consistent with the policies of GOP and USAID. FGD Moderation guidelines The FGDs should follow the below guidelines (FIRST, ASK ALL PARTICIPANTS TO KINDLY PUT THEIR CELL PHONES ON “SILENT”, AND ALSO REQUEST THAT PARTICIPANTS KINDLY NOT HAVE “SIDE-BAR” CONVERSATIONS BECAUSE IT IS IMPORTANT FOR EVERYONE TO HEAR EVERYONE ELSE’S OPINIONS) 1. The participants should be briefed about the project and the purpose of conducting the FGD. 2. They should be informed about the confidentiality and an informed consent should be obtained. 3. Participation of all the intended participants should be ensured. 4. The moderator should broadly follow the below sequence of questions a. Opening questions DELIVER LMIS: Final Evaluation Report 140 b. Follow-up questions c. Probing questions d. Prompting questions 5. The moderator should ensure that note-taking and recording are done. 6. The moderator should effectively use silence and note/record non-verbal communication. 7. The moderator should take a note of time. 8. The moderator should thank the participants in the end. 9. After the FGD, the team members should meet for completion/finalization of notes. Introduction: My name is ______________________. I work for a research organization based in Islamabad. As explained in the official letters from Government (DPW/DOH), we are conducting a final evaluation of the Logistics Management Information System (LMIS) of the DELIVER project which was implemented by John Snow International (JSI). This evaluation will help the Government of Pakistan (GOP) continue to improve the health of mothers and children in Pakistan by strengthening and improving the public supply chain for health commodities (family planning commodities). We will ask a few questions about your experience and your recommendations regarding:  Effectiveness of the LMIS for managing the supply chain of medicines, such as family planning contraceptive commodities.  Sustainability of the Logistics Management Information System for contraceptives (cLMIS) and strengthening of data-driven decisions on supply chain management. This group discussion will take approximately 45 minutes to 1 hour. We will treat the information we collect as confidential and will never associate the information with your name. میرا نام ۔۔۔۔۔۔۔۔۔۔۔۔۔ ہے، میں اسالم آباد میں قائم ایک تحقیقی ادارے (MSI (International systems Managementکے ساتھ کام کر رہا ہوں۔ جیسے کے اجازت نامے میں بتایا گیا ہے کہ ہم JSI کی طرف سے کیے جانے والے project Deliverکے LMIS کے حوالے سے تحقیق کر رہے ہیں۔ اس تحقیق کے نتائج سے حاصل شدہ معلومات حکوم ِت پاکستان کو فیملی پالننگ کے اشیاء کی عوام تک ترسیل کے طریقہ کار کو مضبوط اور بہتر بنانے اور اس کے نتیجے میں ، زچہ اور بچہ کی صحت کی بہتر ی کے لیے مددگار ثابت ہونگی۔ میں آپ سے نیچے بیان کردہ نکات کے متعلق کچھ سوال کرنا چاہونگا۔ .1 فیملی پالننگ اشیاء کی ترسیل اور انتظامی امور پر LMISکے اثر ات ۔ .2 cLMIS سے حاصل کردہ معلومات کی بنیاد پر فیملی پالننگ کے اشیاء کی ترسیل کے نظام کو باقی عالقوں میں پھیالنا ۔ یہ گروپ مباحثہ تقریبا 45 منٹ سے 01 گھنٹے تک جاری رہےگا۔ اس سے حاصل کردہ معلومات مکمل طور پر صیغہ راز میں رکھی جا ئیں گی اور کہیں بھی آپ کے نام سے منسوب نہیں کی جائیں گی۔ May I have your permission to proceed with the group discussion? اجازت کی کرنے غازٓا کا مباحثہ گروپ مجھے کیا ہے؟ ہاں Yes No (THOSE WHO DO NOT WANT TO PARTICIPATE MAY LEAVE. STOP THE GROUP DISCUSSION IF ALL PARTICIPANTS DECIDE TO نہیں (اگرشرکاء میں سے کوئی بھی اجازت نہ دے تو گروپ مباحثہ روک دیں( (GO DELIVER LMIS: Final Evaluation Report 141 May I have your permission to record the group discussion to ensure the completeness and accuracy of your opinions? The recording will remain confidential and kept at our office. ا قیمتی معلومات سے مکمل طور پر مستفید ہونے کے لیے ہم گروپ مباحثہ ریکارڈ کرنا چاہتے ہیں۔ کیا ہمیں ٓپ سے حاصل کردہ گروپ مباحثہ ریکارڈ کرنے کی اجازت ہے؟ Yes (SWITCH ON THE RECORDER AND START THE GROUP DISCUSSION) No (START THE GROUP DISCUSSION) DELIVER LMIS: Final Evaluation Report 142 FOCUS GROUP DISCUSSION )PRINT IN ADVANCE AND CIRCULATE THE FOLLOWING TABLE, ENSURING THAT ALL FGD PARTICIPANTS FILL IN THE INFORMATION CLEARLY) BEFORE LEAVING (BRING PENS AND A CLIPBOARD IN CASE PARTICIPANTS SIT ON THE FLOOR) District Date RESP Name Years of Service as FWW Tehsil Union Council 1 2 3 4 5 6 7 8 9 10 DELIVER LMIS: Final Evaluation Report 143 INSTRUMENT FOR DISTRICT LEVEL FAMILY PLANNING SERVICE FAMILY WELFARE WORKERS (FWW) OF PWD 1. Are you aware of the Logistics Management Information System for contraceptives (cLMIS)? کیا ا cLMIS کے بارے میں جانتے ہیں؟ ٓپ (ASK PARTICIPANTS TO RAISE HANDS IF THEY KNEW ABOUT cLMIS. NOTE DOWN THE COUNT) Number of hands raised = ________________ Total number of participants =________________ 2. In order to ensure the availability of modern family planning contraceptive commodities that are requested by your clients (e.g. condoms, pills, contraceptives injections and Copper-T), a supply chain system is required, and there are many factors involved. In your experience, what are the most important factors that affect the supply of modern family planning contraceptives to you as a provider of family planning services at the community level? (ASK FOR THE TOP TWO OR THREE FACTORS THAT HELP ENSURE THAT ALL OF THE REQUIRED/REQUESTED MODERN CONTRACEPTIVE COMMODITIES ARE AVAILABLE FOR YOU TO PROVIDE TO YOUR CLIENTS, AND THE TOP TWO OR THREE FACTORS THAT ADVERSELY AFFECT THE AVAILABILITY OF THESE MODERN CONTRACEPTIVES) ُ فیملی پالننگ کی اشیاء کی موجودگی کو یقین س کے بہت سے عناصر ہو ی بنانے کے لیے ایک ترسیل کے نظام کا ہونا ضروری ہے اور ا سکتے ہیں، اپنے تجربات کی بنیاد پر بتائیں کہ ایسے کون سے اہم عناصر ہیں جو ان اشیاء کے ضلعی سٹورسےآپ تک ترسیل پر اثرات ُمرتب کر سکتے ہیں۔ )سب سے اہم دو یا تین عوامل کے بارے میں معلوم کریں جو اس بات کو یقینی بناتے ہیں کہ مطلوبہ اشیاء کالئنٹس کو فراہم کرنے کے لئے موجود/دستیاب ہیں، اور ان اہم دو یا تین عناصر کے بارے میں معلوم کریں جو ان اشیاء کی موجودگی/دستیابی پر اثر انداز ہوتے ہیں(۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 144 3. In your experience, how, if at all, has the cLMIS changed the management or operation of the supply chain of the modern family planning contraceptives requested by your clients? (PROBE ON WHETHER THE FWWs’ REPORTING IN TO THE LMIS DATA-ENTRY OPERATOR, AND FEEDBACK FROM HIM/HER BACK TO THE FWWs ON THESE REPORTS, HAS CONTRIBUTED TO IMPROVEMENTS IN THE SUPPLY OF MODERN CONTRACEPTIVE COMMODITIES) ا میں cLMIS فیملی پالننگ کی اشیاء کے ترسیل کے نظام میں تبدیلی کا سبب بنا ہے؟ )اس بارے میں جانئیے کہ آیا FWWs ٓپ کے تجربے ، Operators Entry Data LMISکو رپورٹنگ کرتے ہیں یا نہیں، اور کیا انکو ان رپورٹس پر آراء ملتی ہیں۔ مزید یہ کہ کیا یہ فیملی پالننگ کے جدید اشیاء کے management chain supplyمیں بہتری کا سبب بنے ہیں؟( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 4. Why do you think so? Please describe an example to support your response. (PROBE ON THE FACTORS THAT WERE MENTIONED IN Q2, AND THE POSITIVE/NEGATIVE RESPONSES OBTAINED IN Q3) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے ۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 145 RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 5. We know from the LMIS system that sometimes FWWs in the districts have problems in being able to submit monthly reports that are complete or on time. In your experience, what are the factors that enable, or that prevent, or that constrain you from being able to report on time, every month, with completeness and accuracy? (PROBE FOR THE FACTORS THAT AFFECT TIMELINESS, COMPLETENESS AND ACCURACY OF THE REPORTS; WHAT INSTRUMENTS CARE PROVIDERS WERE USING BEFORE cLMIS AND HOW THE NEW TOOLS ARE DIFFERENT AND HELPFUL TO THEM. FOR EVERY FACTOR—POSITIVE OR NEGATIVE— PROBE FOR EXPERIENCE ON/IDEAS FOR EXPANDING POSITIVE/OVERCOMING NEGATIVE FACTORS) LMISکے نظام سے موصول شدہ معلومات اس طرف اشارہ کرتی ہیں کہ کچھ ضلعوں میں FWWs کو وقت پر ماہانہ رپورٹ جمع کرانے میں دشواری/مسائل کا سامنا کرنا پڑتا ہے۔کیا آپ اپنے تجربے کی روشنی میں ان عناصر کے بارے میں بتا سکتی ہیں جو آپ کو ماہانہ رپورٹ)مکمل اور غلطیوں کے بغیر( وقت پر جمع کرانے /نہ کرانے میں اثر انداز ہوتے ہیں۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 146 6. What is your understanding of the term “commodity security”? (IF NO REASONABLE ANSWERS, PROVIDE A DEFINITION AND THEN فیملی پالننگ کی اشیاء کی حفاظت کے بارے میں آپ کیا جانتے ہیں؟ PROBE ABOUT SPECIFIC EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK-OUTS) )جس میں ان اشیاء کی دستیابی، صفائی کے ساتھ حفاظت ، چوری اور اسٹاک میں موجود نہ ہونے سے متعلق مثالوں کے بارے میں جانیئے( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 7. In your experience, during the past 3-5 years, what effect, if any, has your reporting in to the cLMIS system had on the security of modern family planning contraceptive commodities in the health facility stores? (PROBE ABOUT SPECIFIC POSITIVE/NEGATIVE EXAMPLES: AVAILABILITY, CLEAN STORAGE, PILFERAGE, AND STOCK-OUTS) ا ۓ میں کیا گزشتہ3 سے 5 سال کے دوران آپ کی cLMISپر رپورٹنگ ویکسین کی ضلعی اسٹورز میں بحفاظت ذخیرہ ٓپ کی را کرنے میں مددگار ثابت ہوئی ہے؟ )مخصوص مثبت/منفی مثالوں کے بارے میں جانئیے: دستیابی/موجودگی، صفائی کے ساتھ حفاظت، چوری، اور اسٹاک میں موجود نہ ہونا( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 147 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 8. Why do you think so? Please use examples to support your response. (RECORD AND WRITE THE ANSWERS) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 148 9. In your experience, what effect, if any, have your reports in to the cLMIS system had on the timeliness, accuracy, and safety of the supply and distribution of modern family planning contraceptive commodities? (PROBE ON WHETHER FWWs RECEIVE USEFUL FEEDBACK AND APPROPRIATE (RE-) SUPPLIES OF REQUESTED CONTRACEPTIVE COMMODITIES FROM THE DISTRICT DPW SUPERVISOR, AND ASK ABOUT SPOILAGE AND PILFERAGE) ا ۓ میں کیا cLMIS پر رپورٹنگ فیملی پالننگ کی اشیاء کی فراہمی اور FWWsکو بروقت تقسیم کو یقینی بنانے میں مددگار ثابت ٓپ کی را ہو ئی ہے؟ )جانئیے کہ آیا FWWsضلعی Supervisor DPWسےمفید آراء وصول کرتے ہیں ، اور مطلوبہ اشیاء کی فراہمی کرتے ہیں؟ مزید یہ کہ خرابی اور چوری سے متعلق پوچھ گچھ کرتے ہیں یا نہیں؟( RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 10. Why do you think so? Please use examples to support your response. (PROBE ON WHETHER AND HOW THE cLMIS SYSTEM IS CONTRIBUTING TO EFFECTIVE AND EFFICIENT BACK-AND-FORTH COMMUNICATION FOR CONTRACEPTIVE SUPPLY CHAIN MANAGEMENT) ا ۓ مہربانی اپنے دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ ٓپ کے ایسا سوچنے کی کیا وجہ ہے ؟ برا RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 149 RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 11. Have you received any training from JSI/DELIVER? (CIRCLE ONE NUMBER) کیا آپ / JSI DELIVER کی طرف سے کوئی ٹریننگ حاصل کر ُچکے ہیں؟ (ASK PARTICIPANTS TO RAISE HANDS IF THEY HAD RECEIVED TRAINING. NOTE DOWN THE COUNT) Number of hands raised = ________________ Total number of participants =________________ 12. On which of the following topics, if any, did you receive training? (READ THE NAME OF EACH TRAINING TYPE, AND ASK PARTICIPANTS TO RAISE THEIR HANDS IF THEY HAD RECEIVED THAT TRAINING. ENTER THE COUNT FOR EACH TRAINING IN THE TABLE BELOW) مندرجہ ذیل موضوعات میں سے ا کس پر ٹریننگ حاصل کی ہوئی ہے؟ ٓپ نے o Principles of supply chain management ترسیل کے نظام کے بنیادی اصول o Purpose of the cLMIS in supply chain management فیملی پالننگ کے اشیاء کی ترسیل کے نظام سے متعلق cLMIS کے مقاصد o Entering data in the cLMIS اعداد و شمار کے اندراج کے لیئے cLMIS کا استعمال o Other cLMIS training (describe below) cLMIS کےمتعلق کوئی اور ٹریننگ Describe: DELIVER LMIS: Final Evaluation Report 150 13. Did the training provide you with NEW INFORMATION? If yes, please provide an example: (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کیا حاصل کردہ ٹریننگ سے آپکو نئی معلومات حاصل ہوئی؟ RESP No.1 o Yes Example: __________________________________________________________ o No RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No RESP No.9 DELIVER LMIS: Final Evaluation Report 151 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No Was the training USEFUL IN YOUR WORK? If yes, please provide an example: (CIRCLE YES OR NO AND WRITE AN EXAMPLE FOR EACH RESPONDENT) کیا حاصل کردہ ٹریننگ آپ کے کام میں معاون ثابت ہوئی؟ RESP No.1 o Yes Example: __________________________________________________________ o No RESP No.2 o Yes Example: __________________________________________________________ o No RESP No.3 o Yes Example: __________________________________________________________ o No RESP No.4 o Yes Example: __________________________________________________________ o No RESP No.5 o Yes Example: __________________________________________________________ o No RESP No.6 o Yes Example: __________________________________________________________ o No RESP No.7 o Yes DELIVER LMIS: Final Evaluation Report 152 Example: __________________________________________________________ o No RESP No.8 o Yes Example: __________________________________________________________ o No RESP No.9 o Yes Example: __________________________________________________________ o No RESP No.10 o Yes Example: __________________________________________________________ o No 14. Did the DELIVER project’s training on the use of the cLMIS improve your skills in supply chain management (i.e. reporting, requisition, and availability of modern contraceptives)? Why do you think so? Please use examples to support your response. ٰ اپنے تجربہ کی بن یں کے یاد پر بتائ Project Deliverکے LMISکی ٹریننگ کی وجہ سے فیملی پالننگ کے اشیاءکی ترسیل کے نظام ئی ؟ آپ کے ایسا سوچنے کی کیا وجہ ہے ؟ براۓ مہربانی اپنے )رپورٹننگ، درخواست اور موجودگی( سے متعلق آپکی مہارت میں بہتری آ دیئے گئے جواب کی روشنی میں وضاحت کیجئے۔ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 153 15. Which information, if any, do you use to determine the quantity of modern family planning contraceptive commodities you need? فیملی پالننگ کی اشیاء کی درکار تعدادکا تعین کرنےکے لیئے آپ کس اعدادوشمار کا استعمال کرتے ہیں؟ RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ 16. Why in your experience is cLMIS useful or not useful? (WRITE AN ANSWER) ِل استعمال ہے / نہیں ہے؟ آپ کے خیال میں cLMISکس وجہ سے قاب RESP No.1 ______________________________________________________________ RESP No.2 ______________________________________________________________ RESP No.3 ______________________________________________________________ RESP No.4 ______________________________________________________________ RESP No.5 ______________________________________________________________ RESP No.6 ______________________________________________________________ RESP No.7 ______________________________________________________________ RESP No.8 ______________________________________________________________ DELIVER LMIS: Final Evaluation Report 154 RESP No.9 ______________________________________________________________ RESP No.10 _____________________________________________________________ DELIVER LMIS: Final Evaluation Report 155 Annex 4: List of Interviews The list has been removed from the report to protect the confidentiality of interview subjects. It is available on request from PERFORM. DELIVER LMIS: Final Evaluation Report 156 Annex 5: List of Documents Reviewed 1. DELIVER project documents: a. Annual Report 2013–2015 b. Quarterly Progress Report (2013–2016) c. Annual Work Plan 2013–2015 d. PakInfo progress indicators e. Midterm evaluation report f. DELIVER lot quality assurance sample (LQAS) survey g. Success stories h. Procurement manuals-contraceptives i. Procurement manuals-essential medicines j. Procurement manuals-cLMIS k. Procurement manuals-vLMIS 2. Demographic Health Survey 2013 3. TDY Report DELIVER LMIS: Final Evaluation Report 157 Annex 6: Desk Research on LMIS Introduction to the Review of SCM and LMIS Background Documents For many years, USAID has provided technical assistance to support Health System Strengthening (HSS) efforts in developing countries. Most recently, the Health Finance and Governance (HFG) project14 and the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) project15 are providing technical assistance aimed at expanding access to both essential health services and related essential health commodities in order to improve health.16 The work of HFG has illustrated the importance of regular integrated measurement (1) of access to health commodities, (2) of the proper delivery of preventive and treatment services, and (3) of the health outcomes expected.17 Systems for monitoring, evaluation, and health services research provide funders with the evidence they need to link health system strengthening investments to the occurrence and distribution of health outcomes, and provide practitioners with universally accepted indicators to measure, monitor, evaluate, and continuously improve progress. Both funders and practitioners need to use measurement methods to understand the key characteristics of the performance, quality, and results of health system interventions. THEORY OF CHANGE Source: USAID/Pakistan 14 See https://www.hfgproject.org/what-we-do/ (accessed October 12, 2016). 15 See http://siapsprogram.org/approach/supply-chain-management/ (accessed October 12, 2016). 16 See https://www.hfgproject.org/what-we-do/. 17 See Generating Evidence to Strengthen Health Systems, available at https://www.hfgproject.org/what-we-do/evidence-and-measurement/ (accessed October 12, 2016). DELIVER LMIS: Final Evaluation Report 158 As illustrated in the above figure, in its theory of change, the USAID/Pakistan Mission recognized the need for strengthening measurement systems when it embarked on the design of its program of support to the Government of Pakistan (GoP) for improving maternal and child health (MCH) outcomes by strengthening access to health services and access to related health commodities. Thus, the USAID/Pakistan DELIVER project is one component (component 4) of a five-component program for strengthening MCH programs in Pakistan: 1. Family planning and reproductive health (FP/RH); 2. Maternal, newborn, and child health (MNCH); 3. Behavior change communications (BCC); 4. Health commodities and supply chain management (SCM); and 5. Health systems strengthening (HSS). In August 2009, prior to devolution of the Pakistan health system which occurred in 2011, USAID/Pakistan selected the John Snow, Inc. (JSI) DELIVER project as the Mission’s implementing partner to provide the GoP with technical assistance in strengthening human and institutional capacity and establishing a national system for managing the supply chain for modern contraceptive commodities. Based on its SCM experience in Bangladesh, JSI developed and implemented a contraceptive logistics management information system (cLMIS) in 26 districts in July 2011, beginning with a limited section of the contraceptive supply chain with the Population Welfare Department (PWD), and later working with all public sector family planning stakeholders—i.e., the Department of Health (DoH), People’s Primary Healthcare Initiative (PPHI), and Lady Health Worker (LHW) program. 18 This was one of the first SCM interventions to be implemented in Pakistan, and USAID selected JSI/DELIVER because of JSI’s 30-year history of international SCM technical assistance and its well￾established logistics motto: “No product, no program.”19 After the launch of the cLMIS in July 2011, JSI expanded the LMIS to report contraceptive and tuberculosis (TB) logistics data from all 143 districts of Pakistan. In May 2013, USAID/Pakistan tasked the DELIVER project with expanding the web-based LMIS to cover and improve the vaccine and cold chain logistics management system in Pakistan. Based on strategic level meetings with all stakeholders—including the Ministry of National Health Services Regulations and Coordination (MoNHSR&C), United Nations Children’s Fund (UNICEF), World Health Organization (WHO), World Bank, Global Alliance for Vaccines and Immunization (GAVI), Japan International Cooperation Agency (JICA), and Pakistan’s provincial and regional governments—DELIVER developed a common vision of the design of a comprehensive, sustainable, and automated vaccine logistics management information system (vLMIS). In the first phase of implementation, DELIVER implemented the vLMIS in 54 polio high risk and priority districts of Pakistan, including 9 districts and 3 towns (Karachi) of Sindh. Based on the success of the system, in February 2015, responding to the request of 18 See USAID/Pakistan: New Logistics Management Information System Incorporates Sustainability and Cost Savings, November 2012: http://deliver.jsi.com/dlvr_content/resources/allpubs/logisticsbriefs/PKNewLMIS.pdf (accessed October 12, 2016). 19 The history of JSI’s experience and success in providing SCM technical assistance is available at http://www.jsi.com/JSIInternet/IntlHealth/techexpertise/display.cfm?tid=1000&id=79 (accessed October 12, 2016). Another USAID partner with many years of successful international experience in building capacity and systems for SCM is Management Sciences for Health (MSH); see “Improving Drug Management in Decentralized Health Systems” (available at http://erc.msh.org/mainpage.cfm?file=2.7.2.htm&module=Drugs&language=English, accessed October 12, 2016) and “Systems for Improved Access to Pharmaceuticals & Services” (available at http://siapsprogram.org/approach/supply-chain-management/, accessed October 12, 2016). DELIVER LMIS: Final Evaluation Report 159 the DoH, Government of Sindh, USAID/Pakistan decided to support vLMIS scale-up in all districts and towns of Sindh. The DELIVER project designed, developed, and deployed two LMIS applications for the public sector that capture multiple levels of storage, consumption, and wastage data from the union council, district, provincial, and national levels for vaccines (vLMIS), contraceptives (cLMIS), and TB commodities, ensuring visibility and accountability of these public sector commodities. The activity included both hard and soft components. Hard components included software, continuous architecture maintenance, and IT equipment procurement, including servers. Soft components included training, supportive supervision, and monitoring. The project provided technical support to the GoP in the areas of contraceptive forecasting, procurement planning, warehouse management, supply chain strengthening, and automating the warehouse and LMIS databases. Unresolved LMIS Issues The web-based applications (cLMIS, vLMIS, and TB-LMIS) are “owned and implemented” by the federal and provincial/regional governments, but under Pakistan’s devolved health system there are a number of uncertainties and a lack of clarity regarding how management decisions will be made on financing, maintenance, and possible modifications for improvement of these software applications.20 JSI has 30 years of experience in developing countries, and there is little or no question about the comprehensiveness and quality of the content of JSI’s many DELIVER documents related to capacity￾building for SCM and LMIS (e.g., training documents, advocacy documents, and guidelines on standard operating procedures); however, as indicated in the documents referenced in footnote 20, there are substantial problems affecting the quality of LMIS data and the effective use of the cLMIS and vLMIS. These problems appear to be at least in part associated with the didactic LMIS teaching method (versus a more practical, applied, on-the-job approach), the limited extent of coverage of required training among stakeholders, the limited use of supportive supervision and mentoring under the DELIVER project and uncertainty about how to improve these practices going forward, the limited retention of trained personnel, the uncertainty about whether the LMIS software applications can be modified, and the uncertainty about how to maintain the LMIS standards in Pakistan’s devolved health system. Although the DELIVER/SCM component of the MCH program aimed to contribute to the overall objective of improving maternal and child health outcomes in focus areas, with a specific emphasis on strengthening the public supply chain to ensure commodity security, many of the essential MCH health commodities are not included in the three “vertical” LMIS applications for contraceptives (cLMIS), Expanded Program on Immunization (EPI) vaccines (vLMIS), and TB (TB-LMIS) commodities. The Team’s Review and Comments on Selected Background Documents 1. DHS 2012-2013: National Institute of Population Studies (NIPS) [Pakistan] and ICF International. 2013. Pakistan Demographic and Health Survey (DHS) 2012-2013. Islamabad, Pakistan, and Calverton, Maryland, USA: NIPS and ICF International. A quick scan through the charts and graphs of Pakistan’s most recent DHS yields important insights into the performance of the government’s health policies and the public health and demographic challenges it 20 See the April–May 2016 TDY Report to USAID by Lauren Hartel on Diversion and Commodity Security; the Family Planning Compliance Monitoring Report, Management Systems International, draft September 2016 (personal communication); and the USAID | DELIVER PROJECT, Task Order 4. April 2016. Rapid Assessment to Determine Current Stock Availability of Contraceptives in Sindh and Punjab, Pakistan. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. DELIVER LMIS: Final Evaluation Report 160 will face in the future.21, 22 These challenges will be exacerbated if there are continued constraints in assuring the availability, accessibility, and affordability of essential health services and related health commodities, such as the full range of modern contraceptives, EPI vaccines, other essential health commodities, and related primary healthcare delivery services under the decentralized health system of Pakistan (see footnotes 21–22). DHS Table 6.8, embedded here, on differences between “total wanted fertility rates” and actual “total fertility rates” provides a good example of what may be an important problem of limited access to family planning services, limited access to the modern contraceptive commodities desired by clients, or both. To what extent are “unmet needs” the result of unavailable services/providers or unavailable commodities at service delivery points for women who wished to delay or reduce their birth rate? DHS TABLE 6.8: WANTED FERTILITY RATES Source: DHS 2012 – 2013 The cLMIS does not provide users with integrated data on the distribution and occurrence of access to logistical and clinical monitoring and evaluation (M&E) data on the availability of both a full range of modern contraceptive commodities and trained providers available to deliver all types of contraceptive services at sub-district-level family planning service delivery sites.23 21 See comments by Richard Cincotta (Wilson Center Global Fellow at the Stimson Center in Washington, D.C.), available at https://www.newsecuritybeat.org/2014/12/pakistans-demographic-health-survey-reveals-slow-progress/ (accessed October 12, 2016). 22 See also Sania Nishtar, Ties Boerma, Sohail Amjad, Ali Yawar Alam, Faraz Khalid, Ihsan ul Haq, and Yasir A. Mirza, “Pakistan’s Health System: Performance and Prospects after the 18th Constitutional Amendment,” The Lancet 381, no. 9884, 2013, 2193–2206. 23 See the Family Planning Compliance Monitoring Report, Management Systems International, draft September 2016 (personal communication); the USAID | DELIVER PROJECT, Task Order 4. April 2016. Rapid Assessment to Determine Current Stock Availability of Contraceptives in Sindh and Punjab, Pakistan. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4; and the USAID | DELIVER PROJECT, Task Order 4. October 2012. Pakistan: Provincial and District Supply Chain Management Situation Assessment. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. DELIVER LMIS: Final Evaluation Report 161 The 2012–2013 DHS covered five administrative units: Punjab, Sindh, Khyber Pakhtunkhwa (KP), Gilgit Baltistan, and the Islamabad Capital Territory (ICT). It did not cover the Federally Administered Tribal Areas or Azad Jammu and Kashmir (AJK). At the provincial level, Pakistan’s MCH indicators had improved only marginally since the last DHS in 2006–2007. Notably, the health sector had failed to keep pace with progress in either Bangladesh or Nepal, both of which lagged behind Pakistan in most public health indicators in the early 1990s. Similarly, as the decline in fertility slowed in Pakistan (at 3.8 children per woman in 2013), fertility rates in Bangladesh (2.3) and Nepal (2.6) had continued their declines. As illustrated in DHS Table 8.2, results from the 2012–2013 DHS indicated only slight improvements, and even some worsening in key indicators which are surely dependent in part on whether essential health commodities are available for the delivery of primary healthcare services in both urban and rural areas. Under-five mortality (the proportion of deaths of children aged less than 5 years) declined to 89 deaths per 1,000 births, down from 94 in the 2006–2007 survey. Compare this to Bangladesh’s 53 per 1,000 and Nepal’s 54, reported in comparable 2011 surveys. Childhood vaccination rates (ages 12 to 23 months) in the surveyed regions rose from just 47 percent, as measured in 2006–2007, to 54 percent. Unsurprisingly, Pakistan’s public health infrastructure appears to operate most effectively in and around Islamabad and least successfully in the rugged, sparsely populated province of Balochistan. Otherwise, each province’s rank order varies from one maternal and child health indicator to another, as DHS Table 8.2 illustrates. DHS TABLE 8.2: TRENDS IN EARLY CHILDHOOD MORTALITY RATES Source: DHS 2012 – 2013 *PDHS stands for Pakistan Demographic and Health Survey. Disparities in the availability of an integrated “package” of both essential health commodities and essential health services can be appreciated from the DHS 2012–2013 MNCH outcome data, as illustrated in the following figure. 24 24 Nishtar et al. DELIVER LMIS: Final Evaluation Report 162 PAKISTAN’S MATERNAL AND INFANT MORTALITY RATES Source: IMR stands for Infant Mortality Rate; MMR stands for Maternal Mortality Rate. For some analysts, the DHS 2012–2013 fertility results provide the most disappointing reflection of household conditions. The report found only a slight decline in total fertility rate, from 4.1 children per woman in 2006–2007 to 3.8 in 2012–2013. Whereas Islamabad’s total fertility rate had declined to 3.0, the other administrative units appear to range closer to four children per woman mark. DELIVER LMIS: Final Evaluation Report 163 VACCINATION COVERAGE AND FERTILITY RATES Source: PDHS stands for Pakistan Demographic and Health Survey; BDHS stands for Bangladesh Demographic and Health Survey. Data from the 2012–2013 DHS indicated that, at the provincial level, about 26 percent of married Pakistani women used modern contraception, a significant jump up from the early 1990s when modern contraceptive use languished below 10 percent, but a small increase from the 2008–2009 assessment of 22 percent. In 2012–2013, about one in five married Pakistani women has an “unmet need for family planning,” a stated desire to delay or limit births over the next two years, without safe and suitable contraception. Recent nationally representative survey data25 and a special survey on contraceptive commodity diversion and security26 obtained at the level of districts and service delivery sites provide some insights into the reasons for the past and current levels of unmet needs for family planning services. They indicate that the reasons surely include the lack of integration at service delivery sites of both the availability and accessibility of the full range of modern contraceptive commodities and the availability of family planning service providers who are trained for the delivery of all of these types of commodities. Richard Cincotta’s review of lessons learned from the 2012–2013 DHS includes the following important remarks27 which have implications for the success or failure of institutionalizing, expanding, and sustaining efficient and effective supply chain management systems and related LMIS for monitoring, evaluating, and improving “vertical” health program “silos” under Pakistan’s devolved health system. “After having been virtually de-funded during the Zia Regime, family planning and related reproductive health programs were reorganized during the mid-1990s and assigned to the Ministry of Population Welfare. The new ministry – assisted by bilateral and international development agencies – managed to assemble a professional cadre of administrators and field workers and an extensive network of community-based 25 See the Family Planning Compliance Monitoring Report, Management Systems International, draft September 2016 (personal communication); and the USAID | DELIVER PROJECT, Task Order 4. April 2016. Rapid Assessment to Determine Current Stock Availability of Contraceptives in Sindh and Punjab, Pakistan. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. 26 See Hartel TDY Report. 27 See Cincotta comments. DELIVER LMIS: Final Evaluation Report 164 ‘Lady Health Workers,’ despite relatively modest funding.28 But the ministry was dissolved in 2010 when the 18th Amendment to Pakistan’s constitution devolved the administration of health services to individual provinces. “Pakistan’s public health service delivery system is now in flux. Critics of health-service devolution argue that provincial governments have neither the expertise nor the funds to support additional services – like those offered by the Lady Health Worker Program. Proponents of service devolution have virtually given up on the central government as a service provider, and point to the successes of some Indian states, particularly in the south, which re-prioritized and modernized their services with little assistance from India’s central government. “Undoubtedly, the 2012-13 DHS will serve as a baseline for future evaluations of Pakistan’s devolution experiment. For the time being, however, it provides a data-rich, but ultimately disheartening update on the country’s public health and demographic progress.” Data from the 2012–2013 DHS and the more recent evaluations mentioned in footnotes 25 and 26 clearly indicate that there is a need for a systems-oriented approach to integrating the availability of health commodities (products) with health services (programs) such as the system illustrated in the following figure from the USAID-funded Health System Strengthening (HSS) approach of building Systems for Improved Access to Pharmaceuticals and Services (SIAPS) which aims to improve health outcomes through the integration of improved access and improved services.29 SIAPS PHARMACEUTICAL SYSTEM STRENGTHENING APPROACH Source: SIAPS 28 Ibid. 29 See the SIAPS approach to supply chain management, implemented by Management Sciences for Health, available at http://siapsprogram.org/approach/supply-chain-management/ (accessed October 13, 2016). DELIVER LMIS: Final Evaluation Report 165 2. Mid-Term Evaluation of the USAID DELIVER Project: prepared independently by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP), February 2013. The midterm evaluation was conducted in 2012 covering the project period from August 2009 to September 2012. The evaluation used both quantitative and qualitative research methodologies, employing primary data collection as well as review and analysis of the secondary data sources. The evaluation was conducted across eight districts in four administrative units: Punjab, Sindh, KP, and AJK. The evaluation aimed to answer five evaluation questions, of which two questions (provided below with findings) were directly related to the end-line evaluation of the LMIS component of the DELIVER project. 1. Have procurement activities been automated, and is the government using the web-based Logistics Management Information System (LMIS) and linking it to procurement planning and forecasting? At the time of the midterm evaluation, the planning commission was using the integrated Contraceptives Logistics Report (CLR-6) consumption data to prepare contraceptive procurement plans in conjunction with JSI/DELIVER. CLR-6 data were entered into the LMIS at the provincial levels for pilot districts and at the central level for all non-pilot districts. In 2012, provincial government agency officials, with the support of JSI/DELIVER, conducted a contraceptive quantification assessment30 and prepared contraceptive procurement tables (CPTs) using data from the integrated CLR-6. The Planning Commission, in collaboration with JSI/DELIVER, utilized the procurement manual in the development of CPTs. The situational assessment used both quantitative and qualitative assessment tools to survey 24 selected districts and their 72 facilities. The assessment findings indicated gaps in the supply chain caused by limited cooperation as well as overlapping responsibilities shared by the DoH and PWD, a lack of institutional commitment to prioritize family planning, and issues with human capacity related to supply chain management. The findings also noted that the distribution system was weak and inconsistent, resulting in stock-outs at the district and facility levels. Moreover, the findings indicated a communication gap among public sector stakeholders, resulting in various vertical supply chains and inefficiencies in the distribution system. The assessment recommended improvements for the DELIVER project to address in supply chain-related managerial and technical skills at the provincial and district levels, in advocating for harmonization and collaboration among stakeholders, in developing an integrated supply chain, and in implementing the LMIS to improve requisitioning and storage for health commodities. At the time of the midterm evaluation, the electronic LMIS was not designed to allow for automated procurement planning. Automation of procurement activities requires that all districts use the electronic LMIS and function as the direct source of utilization data required for generating CPTs. They were not doing this at the time of the midterm evaluation as the electronic LMIS was still being rolled out. The findings of the midterm evaluation indicated that there was a manual rather than an automated link between the web-based LMIS system and production of CPTs and/or procurement planning and forecasting at the federal and provincial levels (pages 28–29). The midterm evaluation in February 2013, 30 See: USAID | DELIVER PROJECT, Task Order 4. 2012. Pakistan: Provincial and District Supply Chain Management Situation Assessment. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. DELIVER LMIS: Final Evaluation Report 166 a subsequent TDY by Lauren Hartel in April–May 2016, and nationally representative surveys in 2016 indicated that these problems continue to threaten the quality and utility of logistics management data.31 2. To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the 8 sampled pilot districts and ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? The 2012 midterm evaluation concluded that the management of the contraceptive supply chain using web-based LMIS technology at the district level was weak, while it was relatively stronger at the provincial level. The PWD performed better at installing and utilizing the software applications than the LHW and the DOH. Effective functioning of the LMIS for use in SCM was found to be limited by certain problems such as high staff turnover, frequent power outages, need for refresher training, and inadequate follow-on support and assistance from JSI/DELIVER (pages 29–30). These problems all persist in 2016 according to the Hartel TDY and the nationally representative surveys mentioned above. With regard to the extent to which JSI/DELIVER achieved its objectives to improve procurement capacity, the midterm evaluation findings indicated that provincial staff had inadequate capacity to quantify, forecast, and prepare procurement plans. As the contraceptive procurement was centralized at the time of the midterm evaluation, provincial staff lacked the opportunity to put the education and training they received into practice (pages 6, 29). 3. Rapid Assessment to Determine Current Stock Availability of Contraceptives in Sindh and Punjab, Pakistan. USAID | DELIVER Project, Task Order 4. Arlington, Va. Prepared by APEX Consulting Pakistan, April 2016. Nature and Purpose of the Study The study was conducted by Apex Consulting Pakistan at the request of USAID to assess the stock availability of eight types of contraceptive commodities—COC (the combined oral contraceptive hormones estrogen and progestogen), DMPA (an injectable contraceptive containing depot medroxyprogesterone acetate), EC (emergency contraception, progestin only pill), IUDs (intrauterine devices; five different brands are FDA approved for use in the United States: ParaGard, Liletta, Mirena, Skyla, and Kyleena), Implanon (etonogestrel implant), Jadelle (two thin, flexible silicone rod implants, each containing 75 mg levonorgestrel), male condoms, and POP (progestogen-only pill)—in Sindh and Punjab at district stores and service delivery points (SDPs) and to identify the gaps within the contraceptive commodity supply and distribution system in these provinces. A secondary purpose was to validate the accuracy of the cLMIS data reported by the GOP. Methodology32 The study used a mixed methodology of quantitative and qualitative research. The survey was carried out using a modified version of the large country-lot quality assurance sampling (LC-LQAS) survey design Investigators selected SDPs and districts where the “lot” was defined as the stakeholder within 31 See the Lauren Hartel TDY Report; the Family Planning Compliance Monitoring Report, Management Systems International, draft September 2016 (personal communication); and the USAID | DELIVER PROJECT, Task Order 4. April 2016. Rapid Assessment to Determine Current Stock Availability of Contraceptives in Sindh and Punjab, Pakistan. Arlington, Va.: USAID | DELIVER PROJECT, Task Order 4. 32 The evaluation team asked their colleague, Prof. Valadez, to provide his assessment as to whether the modified method used by Apex Consultants is valid and reliable as applied in Pakistan. DELIVER LMIS: Final Evaluation Report 167 each province, while supervision areas were defined as the district that acted as an administrative unit to store and distribute contraceptives. Study Timeframe The study was conducted between December 2015 and January 2016. Sample – Quantitative Component The rapid assessment team collected data on a total of 1,991 facilities (71 stores and 1,920 SDPs) including 952 facilities within 10 Punjab districts (30 stores and 922 SDPs) and 1,039 within 11 Sindh districts (41 stores and 998 SDPs). Additionally, 1,122 LHWs (614 in Punjab and 508 in Sindh) were interviewed. Sample – Qualitative Component The qualitative component included 14 in-depth interviews (IDIs) with district store managers and 7 focus group discussions (FGDs) at the SDP level. Respondents among the IDIs and FGDs included staff belonging to each stakeholder working at the district stores, at the SDPs, and in the community. Three types of stakeholders participated in both Punjab and Sindh: the PWD, the DoH, and the LHW program; in Sindh, the PPHI also participated. Key Findings  Dedicated cLMIS operators and their cLMIS training status Only around 50 percent of the visited stores had appointed/dedicated cLMIS operators (Punjab 43 percent, Sindh 66 percent). When disaggregated by stakeholder, approximately half of the PPHI, LHW program, and DoH stores, and two-thirds of the PWD stores had an appointed cLMIS operator. This is not an encouraging finding with regard to institutionalization and sustainability of the cLMIS. (Evaluation Question 2) All the appointed cLMIS operators in Punjab and 93 percent in Sindh reported that they had received the cLMIS trainings. (Evaluation Question 1)  Reporting mechanism District managers in district stores in both provinces reported a high use of CLR-6/cLMIS forms for reporting to higher levels. In Sindh, 100 percent of stores were using CLR-6/cLMIS; 87 percent of stores in Punjab were using this form for reporting, while the remaining stores were reporting either through the manual monthly report or by both mechanisms. This finding is encouraging for institutionalization and sustainability of the cLMIS system. (Evaluation Questions 1 and 2)  Data quality Approximately 65 percent of all stores and 80 percent of SDPs had accurate LMIS reports for COC, DMPA, and male condoms. The data accuracy/consistency is lower at the stores (45 percent of stores had inconsistent LMIS reports). Inconsistencies in the data quality limit the value of data-driven decision-making. (Evaluation Questions 1 and 2)  Stock levels at the time of survey DELIVER LMIS: Final Evaluation Report 168 The stock levels for all the family planning commodities analyzed in the study were found to be inadequate at both the stores and the SDPs. Between 33 and 60 percent of the district stores were understocked for all the assessed family planning commodities. The SDPs, however, were overstocked (page 49). This indicates a lack of attention or an inability to properly use the LMIS, resulting in inappropriate requisition practices and questions about the translation of knowledge acquired through trainings in to practice. (Evaluation Questions 1, 2, and 4) 4. Lauren Hartel: TDY Report to USAID/Pakistan: Diversion and Commodity Security, April 19 to May 7, 2016. This short-term consultancy which used observations and key informant interviews yielded numerous important findings about the use, usefulness, institutionalization, and sustainability of the cLMIS. Selected findings are embedded here because of their relevance to the team’s evaluation questions. Commodity Security Issues According to the Hartel TDY report: “Due to the upcoming phase out of USAID-donated commodities and the beginning of provincial procurement, central stakeholders conceptualized commodity security primarily as the ability of each province to obtain commodities, rather than the ability of provinces to ensure that those commodities reach the end user (last mile delivery). When last mile delivery is discussed, the importance of it is understood, but data surrounding it is not being used to its potential. No stakeholders interviewed at the district or provincial level were able to give a rough estimate of how prevalent stock outs were for their area, for example, though this information is available on the cLMIS. “This seems to have two roots: first, that the cLMIS is not extremely intuitive, causing users to have to pull the data out instead of having a dashboard or other mechanism that actively pushes essential data to them; and second, that there is no clear agreement within and among the stakeholders of who is responsible for preventing stock outs. GOP members at the federal level pointed to the devolution to explain why it fell to the provinces, while provinces felt that stock outs were best dealt with at the district level, and districts kicked it both ways either back up to provinces or down to the facilities.” (Evaluation Questions 1, 2, 3, and 4) cLMIS Reporting related to Commodity Security According to the Hartel TDY report: “The cLMIS does not measure instances of product diversion in any form. This is primarily due to three issues that can adversely affect data quality and data utility: the cLMIS collects data in an inconsistent manner, it does not track identifying product information, and it does not collect information exhaustively at the service delivery point. As a result, the way data is collected and categorized must change in order to begin identifying and reporting instances of diversion [as well as other aspects of commodity security].” (Evaluation Questions 1 and 2) DELIVER LMIS: Final Evaluation Report 169 Improvements in Commodity Security Through Improvements in the LMIS Software and Dashboard33 Ms. Hartel prepared examples of graphic outputs that could improve the use and utility of the LMIS software if the architecture and code of the LMIS can be modified – here are the examples she provided: EXAMPLES OF GRAPHIC OUTPUTS POSSIBLE FROM MODIFIED LMIS Source: Hartel TDY Report, 2016 The report went on to say: “Repackaging existing LMIS information would be an inexpensive and effective way to help drive the importance of commodity security throughout the supply chain instead of mainly at the central level. Additional metrics that could be created using already available data include:  Status of stock: by showing districts whether or not the stock they have requested has been issued from the central warehouse  Filtering by transportation type: check for correlations between stock out frequencies and transportation type (currently documented in cLMIS at Central Warehouse &Storage)  Stocked according to demand: compare CLR-6 automatically calculated in cLMIS with batches issued by CW&S  Inappropriately stocked: number of facilities that have either greater than three months or fewer than one month of stock” 33 In a key informant interview that the evaluation team held with Dr. Inaam Ul Haq at the World Bank, the team was informed that the Bank had offered to fund modifications in the LMIS software but had been unable to determine whether the LMIS software could be modified. DELIVER LMIS: Final Evaluation Report 170 (Evaluation Questions 2 and 4) cLMIS Data Collection Methods According to the Hartel TDY report: “The largest barrier to reporting diversion is that the cLMIS collects data on a transactional basis at the central level, and on a time-bound (monthly) basis at the district level and service delivery points. This difference hinders data comparisons that are necessary to identify discrepancies in product volume across the supply chain that would exist if product were diverted between the central level and service delivery point.” (Evaluation Question 2) Lack of Product Identifiers According to the Hartel TDY report: “Product identifiers are unique, standardized markings that allow individuals and/or organizations to track a specific commodity at every step in the supply chain and ensure it reaches its intended destination. Examples of identifiers include batch number, lot number and product serial number. Though batch numbers are recorded at the central warehouse to ensure proper receipt of product delivery, they are not recorded at the district level or at service delivery points, making it impossible to trace specific commodities once they leave the central warehouse. In fact, none of these identifiers are tracked once products leave the central warehouse. “This is particularly problematic given that frequently several stakeholders operate at the same facility, and share transportation from the central warehouse. Without product identifiers, for example, commodities ordered and intended for a Lady Health Worker could easily be delivered to the storage room of a PPHI program at the same facility.” (Evaluation Questions 1, 2, and 4) Limitations on “Last Mile” Consumption and Use of Commodities Due to Data Aggregation According to the Hartel TDY report: “Product diversion is nearly impossible to identify at the last mile because for most provinces and stakeholders, data from all service delivery points in a given district is aggregated before being inputted into the cLMIS. It is therefore impossible to tell where in a district products are being consumed. Fortunately, SDP-level collection began in several provinces in March of 2015 and has been successful so far. … “In addition to these more technically oriented challenges, broader factors must be taken into account. Most important are the limited user education and capacity to effectively operate the cLMIS, as well as the lack of role clarity for all actors in the supply chain in preventing and responding to instances of diversion.” (Evaluation Questions 1, 2, and 4) Limited Education, On-The-Job Training, Mentoring, and Supportive Supervision of cLMIS Users According to the Hartel TDY report: “The USAID | DELIVER Project has invested an immense effort in educating cLMIS users. When the cLMIS was introduced to a district, they provided a thorough training DELIVER LMIS: Final Evaluation Report 171 to new users and distributed an impressive array of printed reference material. The material – which includes user guides and operation manuals for cLMIS, as well as guides for procurement and logistics management more generally – is tailored towards specific rules and regulations of each province. There is also a support e-mail address (support@lmis.gov.pk) to answer questions and issues as they come up on an ongoing basis. “In spite of this, there are limitations: the manuals are lengthy and often geared towards managers and decision-makers instead of data entry operators and others who work with the cLMIS at the last mile. Moreover, the cLMIS is a live tool that is updated regularly, and employees who use the cLMIS routinely change. There are no regular training sessions in place, and as a result many users use the cLMIS in a very limited manner. When asked in an interview what additional capabilities they would like to have, several cLMIS users requested functionalities that are already available (e.g., automated CLR-6 calculations, and summary data at the district level). Even if diversion reporting were possible, it is likely that not all users would understand how to access – and more importantly, interpret – the data.” (Evaluation Questions 1, 2, and 4) Limited Number of Trained Users and Lack of Clear System for Retention of Trained Users According to the Hartel TDY report: “In addition to a lack of comfort using the cLMIS at the last mile, the small number of trained designated cLMIS users presents its own limit. For each stakeholder there is typically only one cLMIS operator per district, and this user has several responsibilities completely unrelated to the cLMIS. If the cLMIS operator is out of town or otherwise unreachable while completing these other duties, the reporting process is put on hold.” (Evaluation Questions 1, 2, and 4) Lack of Clarity of Roles of Personnel with Various Supply Chain Management Responsibilities According to the Hartel TDY report: “A crucial issue related to reporting, responding and ultimately preventing instances of product diversion [as well as other aspects of commodity security] is the lack of understanding surrounding who is responsible for the product at each link in the supply chain. When several stakeholders were asked what the protocol is when they encounter instances of diversion, almost all simply stated that diversion does not occur. “When pressed on what they would do if it occurred, most then said they would call their supervisor. Additionally, no Standard Operating Procedure for mitigating and/or responding to instances of diversion could be found (though one should keep the brief nature of this TDY in mind - in other words, this does not mean that one does not exist).” (Evaluation Questions 1, 2, and 4) 5. JSI Procurement Manuals for Contraceptive Commodities (National, KP, Punjab, and Sindh) The JSI contraceptive procurement manual was developed for the Population Program Wing, Planning and Development Division, MoNHSR&C, PWD, and DoH personnel who are responsible for procuring contraceptives of good quality on the international market to support the GoP’s FP/RH programs. DELIVER LMIS: Final Evaluation Report 172 The contraceptive procurement manual is based upon best international procurement practices that promote transparency, accountability, and efficiency in the procurement process. It provides detailed information on the basics of procurement, procurement planning and preparation, standard bidding documents, invitation and receipt of bids, evaluation and selection process, award of contract, and delivery procedures. The procurement manuals for the national level, KP, Punjab, and Sindh are similar in structure and content except for Punjab which has an additional section on the procurement process under public and private partnerships. The manuals contain comprehensive information encompassing all stages of procurement and set out the standard procedures with relevant documents at each stage. Training on building human and institutional capacity for efficient and effective contraceptive procurement practices was carried out through a three-day training course which was essentially totally didactic and lacked substantial opportunities for scenario-based interaction and practice. None of the training materials for the contraceptive procurement manuals included a trainer’s guide like the guides that were developed and used for cLMIS and vLMIS training. It is likely that three days is not a sufficient amount of time to provide competency-based training, but there are no JSI documents that provide evidence of sustained post-training competencies. It is a standard practice that trainings have two types of manuals: one for the participants with all the content and material, and one for the trainer with a session-wise guide. The two types of manuals play an important role in ensuring a consistent standard even if the trainers change or there is a time lag between two sets of trainings. (Evaluation Questions 1, 2, and 4) 6. JSI Procurement Manuals for Essential Medicines (KP, Punjab, and Sindh) The JSI procurement manual for essential medicines was developed in English for the DoH personnel in KP, Punjab, and Sindh who are responsible for procurement of essential medicines and supplies. The manual provides information on the key phases of the procurement cycle, from procurement planning and issuing invitations to bid, bid evaluation, supplier selection, contract award, and management. The manual provides step by step instructions for completing standard bidding documents, opening bids from suppliers, evaluating supplier bids, and monitoring the performance of suppliers. The manuals also provide list of essential medicines that should be available at the primary and secondary level of health care based on WHO standards. The content of the manuals for the three provinces is similar in structure and nature. The manuals for KP and Sindh were endorsed by the respective public procurement authorities in the provinces, while the manual for Punjab was endorsed by DoH Punjab. The manual is comprehensive in terms of content and layout, following a logical sequence in a step-wise manner. The only concern is the training on this manual, which was combined with training on the contraceptive procurement manual, and completed in three days. Considering the scope of the subject in these two manuals, three days appears to be insufficient for developing competencies in the combined subjects. Also, as mentioned above, there was no trainer’s manual to provide session-wise guidance on the training. (Evaluation Questions 1, 2, and 4) 7. JSI Logistics Manual for Contraceptives The logistics manual for contraceptives was developed primarily for the public sector departments involved in procurement, storage, and distribution of contraceptive commodities, such as the MoNHSR&C, the Directorate of the Central Warehouse in Karachi, the provincial DoHs, the provincial PWDs, the LHW program, and the MNCH program. The contraceptives logistics manuals were developed in both English and Urdu for Punjab, KP, Sindh, and Balochistan. For Sindh, the manual was DELIVER LMIS: Final Evaluation Report 173 translated to the local language of Sindhi as well. The procurement manuals for the national level, KP, Punjab, and Sindh are similar in structure and content. The manual provides information on:  Basics of logistics, including components of logistics management system;  Purpose and process of product selection;  Forecasting of contraceptive needs, including the process and different methods involved;  Logistics management staff roles and responsibilities;  LMIS, including information on essential data for decision-making, information and recording system, stock keeping, and transaction and consumption records;  Structure of the web-based LMIS, its process and use; data entry and generation of reports in LMIS; and  Warehousing, inventory management, requisition, quality assurance, and safe disposal of expired or damaged commodities. The contraceptives logistics manual comprehensively covers all aspects of logistics management, including the web-based cLMIS system. It provides the reader with information on the web-based cLMIS in an effective manner, displaying snapshot examples at each step of using the cLMIS online. Training on the contraceptives logistics manuals was carried out through a three-day training course; however, considering the detailed content of the manuals, three days seems to be an inadequate period to become competent in the principles and practices of SCM for contraceptives. Unfortunately, the evaluation team cannot find any evidence of the conduct of competency-based post-training evaluations. There was no trainer’s manual as well for the training to provide session wise guidance to the trainers. 8. JSI Training Manuals on Use of the cLMIS Two types of guides were developed for training on the cLMIS: (a) a facilitator’s manual and (b) a participant’s guide. For facilitation of an efficient system of “trickle-down” training, JSI also developed a training-of-trainers (ToT) manual to develop a cadre of master cLMIS trainers, and a guide for the ToT participants. In addition to the guides for training participants, facilitators, and ToTs, JSI developed two manuals for users of cLMIS data: a specific user manual for PWD users and a more general manual for other users. The training facilitator’s manual is well-structured, guiding the trainer on each session with regard to required material, methods of presentation, resource documents, and information about trainer preparation and the activities that are involved in each session. The facilitator’s manual has the required synergy with the participant’s guide that is critical for effective communication and smooth flow during the training. The contents of the manuals for training facilitators and for participants have the following components:  Introduction and objectives  Basic computing skills  Contraceptive pipeline and ordering  Contraceptive LMIS forms and basic logistic concepts  cLMIS introduction, data entry, and requisitions  Online dashboard, reports, graphs, and maps The participant’s manual includes snapshots of pages and charts from the cLMIS website database with instructions at each step on how to make use of the cLMIS data. The training included group work activity on each of the components of the online cLMIS system. The trainings on cLMIS were conducted in three days, though there is no day-wise break-down of the contents of the sessions. There is no DELIVER LMIS: Final Evaluation Report 174 mention of pre and post-test evaluations, nor post-training on-the-job mentoring and supportive supervision for the participants. The evaluation team was not able to find documents that describe evidence of sustained competencies in use of the cLMIS. (Evaluation Questions 1, 2, and 4) 9. JSI Training Manuals on Use of the vLMIS Two types of guides were developed for the JSI trainings on vLMIS: (a) a facilitator’s manual and (b) a manual for district and sub-district users. For facilitation of an efficient system of “trickle-down” training, JSI also developed a ToT manual to develop a cadre of master vLMIS trainers, and a guide for the ToT participants. The facilitator’s manual is well-organized, with a session-wise guide for the trainers with regard to required material, presentations and resource documents, the preparation required by the trainer, and the training activities for each session. The facilitator’s manual has the required synergy with the participant’s guide that is critical for effective communication and smooth flow during the training. The content of the manuals for facilitators and participants have following components:  Introduction and objectives  Basic computing skills  Vaccine supply chain  Basic logistic concepts  Getting started with vLMIS  Inventory management (IM)  Monthly reporting forms  Monthly consumption reporting  Cold chain equipment management (CCEM)  Online dashboards  Vaccine reports and CCEM reports  Inventory management graphs and CCEM graphs and maps The vLMIS user manual includes snapshots of pages and charts from the vLMIS website database with instructions at each step of the vLMIS use. The training also includes exercises on each of the components of the online vLMIS system. The trainings on vLMIS were conducted in four days, though the facilitator’s manual does not provide a day-wise breakdown of the topics. There is no mention of pre- and post-test evaluations, nor post-training on-the-job mentoring and supportive supervision for the participants. The evaluation team was not able to find documents that describe evidence of sustained competencies in use of the vLMIS. (Evaluation Questions 1, 2, and 4) 10. JSI Training Databases The National Training Database The national training database is an aggregate of trainings carried out on use of the vLMIS, cLMIS, LMIS, TB-LMIS, CLM, procurement, warehousing, and SCM data sheets in the same file. The national training database has a limitation: in the “training type” column, users can add the geographical location, tier/level of training, nature of participants, and whether or not refresher training took place, but there is a lack of standardized labeling. This limits the application of filters on the “training type” column and thus makes it difficult or impossible to do a meaningful analysis of trainings. If the training database had been designed and maintained adequately (i.e., with the use of a uniform labeling of types of trainings), the analysis/cross tabulation of types of trainings by geographical region, by the type/level of the participant, and by other variables of interest could have been carried out on the DELIVER LMIS: Final Evaluation Report 175 national database. Moreover, data on the nature, conduct, and participation in workshops were maintained in the same database with similar limitations for meaningful analysis. The nature and purpose of some trainings/courses are not clear from titles such as “First three credit course in HAS.” One other important observation is that, according to the national database, training on the contraceptive procurement manual and the essential medicine procurement manual was a combined training and conducted in three days. Three days by any means are not adequate considering the content and length of the manuals that have been developed. The following gross analysis is the best that could be done at this stage:  A total of 6,746 participants were trained at the national level in different types of trainings under the JSI/DELIVER project. Out of these: o 5,434 were from the DoH; o 470 were from the PWD; o 235 were from the LHW program; o 106 were from Integrated Reproductive, Maternal, and Child Health – Department of Health; o 58 were from GAVI; o 53 were from the MNCH program; o 46 were from the Central Warehouse and Storage Department; o 31 were from the Capital Development Authority; o 11 were from the Marie Stopes Society; o 10 were from the Family Planning Association of Pakistan; o 7 were from GreenStar Social Marketing; o 5 were from the AIDS control program; and o 280 were from other departments.  The disaggregation of 6,746 participants by types of trainings as obtained from individual training sheets is as follows: o 5,024 on vLMIS, o 1,047 on cLMIS, o 51 on LMIS, o 25 on TB-LMIS, o 238 on CLM, o 108 on procurement, o 73 on warehousing, and o 180 on SCM. The vLMIS Training Database The vLMIS database has the same problem that the national database has with non-standardized labeling of the type of training and other column variables, which limits the application of filters as well as the ability to do meaningful analysis. A total of 5,024 participants were trained on vLMIS. The majority of the trainees (4,829) were from DoHs, followed by 58 from GAVI, 26 from the PPHI (the entity responsible for managing Basic Health DELIVER LMIS: Final Evaluation Report 176 Units in Sindh), 13 from UNICEF, 3 from the federal EPI, 2 from the LHW program, 1 from the MoNHSR&C, and 92 from other departments. Among the 5,024 participants, 50 were trained as master trainers on vLMIS; the majority of these (36) were from the DoH. There is no linkage between the coverage levels for EPI vaccines and the number of users trained to use the vLMIS who could monitor, evaluate, and improve the supply chain for EPI vaccines as a component of efforts to improve vaccine coverage levels. The cLMIS Training Database The cLMIS database has the same problem that the national database has with non-standardized labeling of the type of training and other column variables, which limits the application of filters as well as the ability to do meaningful analysis. A total of 1,047 participants were trained on cLMIS. The largest portion of the trainees (397) were from DoH, followed by 278 from the PWD, 161 from the LHW program, 106 from the Integrated Reproductive Maternal Newborn Child Health (IRMNCH) program, 25 from the PPHI (the entity responsible for managing Basic Health Units in Sindh), 29 from the Capital Development Authority (CDA), 10 from the Family Planning Association of Pakistan (FPAP), and 2 from GreenStar Social Marketing. There is no linkage between the frequency of stock-outs and number of users trained to use the cLMIS who could monitor, evaluate, and improve the supply chain for contraceptive commodities as a component of efforts to reduce unmet needs for family planning. Training on Warehousing Seventy-three participants were trained on warehousing tools and standard operating procedures. Forty-one of the participants were from the Central Warehouse and Supplies Department in Karachi, while 32 participants were from the Medical Stores Department in Punjab. Training on Procurement A total of 108 participants were trained on different aspects of procurement. Of these, 18 had 3 days of training in a skill development workshop on conducting international contraceptive procurement in a public sector environment; 20 had training at a workshop on pre- and post-contractual activities; and 70 had combined training on both the contraceptive procurement manual and the essential medicine procurement manual (as mentioned above, this training was 3 days long, a very short period considering the content and length of these two training manuals). 11. Institutionalization, Scale-Up, and Sustainability of vLMIS The memorandum of understanding (MoU) for the scale-up of the vLMIS from 13 to all 36 districts of Punjab province elaborately indicates a commitment from the Government of Punjab for enhanced engagement in planning for the scale up for quality assurance. The MoU also has a commitment from the Government of Punjab to pay to the JSI/DELIVER project the expenses incurred on interventions for phase 1 of the scale-up. Finally, the Government of Punjab also commits to arrange for funds in phase 2 of the scale-up (pages 3–4). These are encouraging steps in ensuring a scale-up of vLMIS in Punjab, but these commitments were between the Government of Punjab and the JSI/DELIVER project, and it is not clear whether and how these commitments will be implemented after the conclusion of the DELIVER project. (Evaluation Question 3) DELIVER LMIS: Final Evaluation Report 177 Annex 7: Qualitative and Quantitative Interviews cLMIS Department of Health Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents DoH Provincial managers Peshawar, Lahore, Karachi, Hyderabad, Quetta, and Muzaffarabad Key informant interviews (KIIs) Purposive 4 4 DoH Provincial data manager or focal person Peshawar, Lahore, Karachi, Hyderabad, Quetta, and Muzaffarabad KIIs Purposive 2 2 DoH District managers Peshawar, Abbottabad, Lahore, Muzaffargarh, Karachi, Hyderabad, and Muzaffarabad KIIs Purposive 7 7 DoH District data entry operators Peshawar, Lahore, Karachi, Abbottabad, Hyderabad, Muzaffargarh, Quetta, Pishin, Muzaffarabad, and Islamabad Capital Territory (ICT) KIIs Purposive 10 10 DoH LHSs Muzaffargarh, Quetta, and Karachi Focus group discussions (FGDs) Purposive 3 21 Total DoH cLMIS KIIs 23 44 FGDs 3 DELIVER LMIS: Final Evaluation Report 178 Population Welfare Department Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents PWD Provincial managers Peshawar, Lahore, Karachi, and Quetta KIIs Purposive 4 4 PWD Provincial data manager or focal person Peshawar, Lahore, Karachi, and Quetta KIIs Purposive 4 4 PWD District managers Peshawar, Abbottabad, Lahore, Muzaffargarh, Karachi, Hyderabad, Quetta, Pishin, Muzaffarabad, and ICT KIIs Purposive 10 10 PWD Data entry operators Peshawar, Abbottabad, Lahore, Muzaffargarh, Karachi, Hyderabad, Quetta, Pishin, Muzaffarabad, and ICT KIIs Purposive 10 10 PWD FWWs Peshawar, Muzaffargarh, Quetta, and Karachi FGDs Purposive 4 32 Total PWD cLMIS KIIs 28 60 FGDs 4 DELIVER LMIS: Final Evaluation Report 179 People’s Primary Healthcare Initiative (PPHI) Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents PPHI Provincial managers or focal person Peshawar, Karachi, and Quetta KIIs Purposive 3 3 PPHI District managers or focal person Peshawar, Abbottabad, Hyderabad and Pishin KIIs Purposive 4 4 Total PPHI cLMIS 7 7 National Government Stakeholders Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Population Welfare Wing National manager Islamabad KIIs Purposive 1 1 Total National cLMIS 1 1 Grand Total cLMIS Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents GRAND TOTAL cLMIS KIIs 59 112 FGDs 7 DELIVER LMIS: Final Evaluation Report 180 vLMIS Department of Health Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents DoH Provincial managers Peshawar, Lahore, Karachi, and Quetta KIIs Purposive 4 4 DoH Provincial data manager or focal person Karachi and Quetta KIIs Purposive 2 2 DoH District managers Lahore, Karachi, and Hyderabad KIIs Purposive 3 3 DoH District data entry operators Peshawar, Lahore, Karachi, and Hyderabad KIIs Purposive 6 6 ASVs Peshawar, Muzaffargarh, Quetta, and Karachi FGDs Purposive 4 29 Total DoH vLMIS KIIs 15 44 FGDs 4 National Government Stakeholders Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents MoNHSR&C (EPI) National manager Islamabad KIIs Purposive 1 1 MoNHSR&C (director) National manager Islamabad KIIs Purposive 1 1 DELIVER LMIS: Final Evaluation Report 181 Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Total national vLMIS 2 2 Grand Total vLMIS Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Grand total vLMIS KIIs 17 46 FGDs 4 Donors/IPs/INGOs Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Donors DFID, KFW, Packard Foundation, UNFPA, UNICEF, World Bank, and WHO Islamabad KIIs Purposive 8 10 NGOs Greenstar, Marie Stopes Society, Family Planning Association of Pakistan (FPAP), and Jhpiego Islamabad and Karachi KIIs Purposive 5 7 Others Agha Khan University, Health Expert, and McKinsey & Company Islamabad, Karachi, and the United States KIIs Purposive 3 4 USAID Health office Islamabad KIIs Purposive 1 2 DELIVER LMIS: Final Evaluation Report 182 Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Project staff JSI COP Islamabad KIIs Purposive 1 1 Total donors/NGOs 16 24 Total (cLMIS and vLMIS) Respondent Type Respondent Location of Interviews Data Collection Method Sampling Number of Interviews Number of Respondents Grand total (cLMIS and vLMIS) KIIs 94 181 FGDs 11 DELIVER LMIS: Final Evaluation Report 183 Annex 8: Trend Analysis Each page of this annex presents a visualization of trends in a supply chain performance indicator and the results of regression analysis of the trends. Each analysis estimates the overall trend for Punjab and Sindh—the two districts on which the project focused (the coefficient of “Time”); the difference, if any, between the trend in these districts and all other districts (the coefficient of “Time x NPFA”); the change, if any, in the trend for project-focused areas after project implementation scaled back on or about September 2015 (the coefficient of “Time x AFE”); and the difference, if any, in the trend for areas on which the project did not focus (the coefficient of “Time x NPFA x AFE”). DELIVER LMIS: Final Evaluation Report 184 Trends in CYP Associated with Three-Month Injection Dependent Variable: Total CYPs Associated with Three-Month Injections Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 14925.239 2480.245 6.018 .000 Time 986.031 113.829 .749 8.662 .000 Not project focused area (NPFA) -8597.425 2862.961 -.218 -3.003 .003 After funding end (AFE) 19154.462 23757.152 .428 .806 .421 Time*NPFA -1001.920 131.405 -.880 -7.625 .000 Time*AFE -826.218 579.268 -.768 -1.426 .155 Time*NPFA*AFE 331.105 107.360 .274 3.084 .002 DELIVER LMIS: Final Evaluation Report 185 Trends in Consumption of Three-Month Injections Dependent Variable: Total Consumption of Three-Month Injections Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 59700.033 9920.981 6.018 .000 Time 3944.143 455.318 .749 8.662 .000 Not project focused area (NPFA) -34389.346 11451.844 -.218 -3.003 .003 After funding end (AFE) 76619.690 95028.603 .428 .806 .421 Time*NPFA -4007.693 525.619 -.880 -7.625 .000 Time*AFE -3304.926 2317.074 -.768 -1.426 .155 Time*NPFA*AFE 1324.430 429.438 .274 3.084 .002 DELIVER LMIS: Final Evaluation Report 186 Trends in CYP Associated with Copper T-380 Dependent Variable: Total CYPs Associated with Copper T-380 Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 105607.576 14437.804 7.315 .000 Time 2801.830 662.615 .418 4.228 .000 Not project focused area (NPFA) -74629.189 16665.637 -.372 -4.478 .000 After funding end (AFE) 112716.270 138293.212 .495 .815 .416 Time*NPFA -2961.544 764.923 -.512 -3.872 .000 Time*AFE -3225.093 3371.990 -.589 -.956 .340 Time*NPFA*AFE 402.105 624.953 .065 .643 .520 DELIVER LMIS: Final Evaluation Report 187 Trends in Consumption of Copper T-380 Dependent Variable: Total Consumption of Copper T-380 Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 22958.186 3138.476 7.315 .000 Time 609.093 144.039 .418 4.229 .000 Not project focused area (NPFA) -16215.252 3622.760 -.371 -4.476 .000 After funding end (AFE) 24497.175 30062.043 .495 .815 .416 Time*NPFA -644.129 166.278 -.512 -3.874 .000 Time*AFE -700.954 733.000 -.589 -.956 .340 Time*NPFA*AFE 87.525 135.852 .065 .644 .520 DELIVER LMIS: Final Evaluation Report 188 Trends in Reporting Rate for BCG-20 Dependent Variable: BCG Reporting Rate Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) -.002 .001 -2.193 .020 Time .000 .000 .868 6.785 .000 Not project focused area (NPFA) .001 .001 .149 .948 .289 After funding end (AFE) .018 .007 1.889 2.575 .010 Time*NPFA 4.472E-005 .000 .164 .935 .388 Time*AFE .000 .000 -2.035 -2.728 .007 Time*NPFA*AFE .000 .000 -.479 -4.181 .000 DELIVER LMIS: Final Evaluation Report 189 Trends in Consumption of BCG-20 Dependent Variable: Consumption of BCG-20 Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 14990.854 14471.320 1.036 .301 Time 3417.215 585.103 .663 5.840 .000 Not project focused area (NPFA) -13763.534 16471.420 -.106 -.836 .404 After funding end (AFE) 143387.075 98529.608 .968 1.455 .147 Time*NPFA -3048.046 677.503 -.775 -4.499 .000 Time*AFE -4000.404 2404.573 -1.123 -1.664 .098 Time*NPFA*AFE 351.487 439.254 .084 .800 .424 DELIVER LMIS: Final Evaluation Report 190 Trends in Reporting Rate for Pentavalent-1 Dependent Variable: Pentavalent Reporting Rate Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) -.002 .001 -1.859 .064 Time .000 .000 .849 6.449 .000 Not project focused area (NPFA) .002 .001 .217 1.470 .143 After funding end (AFE) .017 .007 1.865 2.415 .017 Time*NPFA 2.282E-05 .000 .093 .467 .641 Time*AFE .000 .000 -2.004 -2.558 .011 Time*NPFA*AFE .000 .000 -.469 -3.852 .000 DELIVER LMIS: Final Evaluation Report 191 Trends in Consumption of Pentavalent-1 Dependent Variable: Consumption of Pentavalent-1 Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) 8130.530 28473.223 .286 .775 Time 9260.197 1151.227 .773 8.044 .000 Not project focused area (NPFA) -7408.860 32408.543 -.025 -.229 .819 After funding end (AFE) 189075.320 193863.134 .549 .975 .330 Time*NPFA -8362.262 1333.029 -.915 -6.273 .000 Time*AFE -5480.819 4731.147 -.662 -1.158 .248 Time*NPFA*AFE 488.085 864.259 .050 .565 .573 DELIVER LMIS: Final Evaluation Report 192 Trends in Reporting Rate of Measles Vaccine Dependent Variable: Measles Vaccine Reporting Rate Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) -.151 .103 -1.461 .145 Time .031 .004 .958 7.482 .000 Not project focused area (NPFA) .086 .117 .106 .735 .463 After funding end (AFE) 1.682 .703 1.797 2.394 .017 Time*NPFA -.002 .005 -.077 -.394 .694 Time*AFE -.045 .017 -1.979 -2.598 .010 Time*NPFA*AFE -.011 .003 -.401 -3.386 .001 DELIVER LMIS: Final Evaluation Report 193 Trends in Wastage Rate of Measles Vaccine Dependent Variable: Measles Vaccine Wastage Rate Model Unstandardized Coefficients Standardized Coefficients t Sig. B Beta Beta (Constant) -.048 .031 -1.533 .127 Time .008 .001 .761 6.145 .000 Not project focused area (NPFA) .006 .036 .023 .167 .868 After funding end (AFE) .506 .213 1.725 2.376 .018 Time*NPFA .003 .001 .358 1.903 .058 Time*AFE -.013 .005 -1.827 -2.480 .014 Time*NPFA*AFE -.004 .001 -.508 -4.438 .000 DELIVER LMIS: Final Evaluation Report 194 Annex 9: Conflict of Interest Statements The conflict of interest disclosures have been removed to protect the confidentiality of team members. They are available from PERFORM on request. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523