February 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP) by Brian Agbiriogu, Ghazala Khan, Sajjad Akhtar, and Sarah Khokhar. Mid-term Evaluation of the USAID DELIVER PROJECT ACKNOWLEDGEMENTS The MSI evaluation team expresses thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:  Mr. Javaid Akhtar, Additional Secretary, and Officials of the Population Planning Wing, the Planning Commission, the Government of Pakistan, and Islamabad;  Syed Ilyas Haider, Director of the Central Warehouse in Karachi, and Mr. Asher Durrani, In-charge, LMIS, and staff of Central Warehouse in Karachi;  The Director General of the Department of Population Welfare, the Director General of the Department of Health and the coordinator of the National Lady Health Workers Program in the provinces of Punjab, Sindh, Khyber Pakhtunkhwa and the Azad Jammu and Kashmir (AJK) region;  Officials of the Department of Population Welfare, the Department of Health, and the National Lady Health Workers Program in the districts of Karachi (East), Hyderabad, Lahore, Muzaffargarh, Peshawar, Abbottabad, Rawalakot and Muzaffarabad;  The three Provincial Logistics Managers and the Regional Logistics Manager (AJK) of USAID/John Snow, Inc./DELIVER Project;  The country team of JSI/DELIVER in Islamabad, including Country Director Dr. Muhammad Tariq and the Manager of Commodity Information Systems Mr. Saif Ur Rab; and  USAID officials responsible for the JSI/DELIVER Project. MID-TERM EVALUATION OF THE USAID DELIVER PROJECT February 2013 Contracted under No. GS-23F-8012H and Order No. AID-391-M-11-00001 Monitoring and Evaluation Program (MEP) DISCLAIMER This study/report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of Management Systems International and do not necessarily reflect the views of USAID or the United States Government. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT i CONTENTS Executive Summary................................................................................................................................................................................. 4 Introduction ............................................................................................................................................................................................10 The Development Problem and USAID’s Response....................................................................................................................16 Project Design ........................................................................................................................................................................................18 Evaluation Design and Methodology ................................................................................................................................................22 Findings and Conclusions ....................................................................................................................................................................25 Main Conclusions and Recommendations ......................................................................................................................................35 Annexes ...................................................................................................................................................................................................37 Annex 1: Evaluation Statement of Work .......................................................................................................................37 Annex 2: Desk Review Report for Identifying More and Less Compliant Districts ...........................................48 Annex 3: Provincial Overview ..........................................................................................................................................56 Annex 4: Provincial Statistics .............................................................................................................................................60 Annex 5: Data Collection Instruments ...........................................................................................................................73 Annex 6: Sources of Information ...................................................................................................................................107 Tables and Figures Table 1: Project Summary .................................................................................................................................................... 3 Table 2: The Percentage of Pilot Districts (including SDPs) with Less than One Month of Stock Between January–November 2012 Per Commodity............................................................................ 8 Table 3: Persons Trained in Forecasting, Procurement, Supply Chain Management (SCM) and LMIS ................................................................................................................................................................28 Table 4: LMIS Implementation by District Government Agencies ..........................................................................32 Table 5: Percentage of District Government Stakeholders with Less than One Month of Stock Between January–November 2012 per Commodity ..........................................................................34 Table 6: Level of Effort of Evaluation Team Members ................................................................................................44 Table 7: Punjab-District Level Profile of Compliance Indicators .............................................................................50 Table 8: Punjab-More and Less Compliant Districts ...................................................................................................51 Table 9: Sindh and Khyber Pakhtunkhwa – District Level Profile of Compliance Indicators ...........................52 Table 10: Sindh and Khyber Pakhtunkhwa - More and Less Compliant Districts ...............................................53 Table 11: AJK, FATA and Gilgit Baltistan - District Level Profile of Compliance Indicators ............................54 Table 12: AJK, FATA and Gilgit Baltistan – More and Less Compliant Districts ................................................55 Table 13: Punjab: Non-reporting Districts ....................................................................................................................60 Table 14: Punjab: Months of Stock...................................................................................................................................63 Table 15: Sindh and Khyber Pakhtunkhwa: Non-reporting Districts ......................................................................65 Table 16: Sindh and Khyber Pakhtunkhwa: Months of Stock ....................................................................................67 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT ii Table 17: AJK, FATA and Gilgit Baltistan: Non-reporting Districts ........................................................................69 Table 18: AJK, FATA and Gilgit Baltistan: Months of Stock ......................................................................................71 Table 19: Distribution of KIIs/GDs by Government Levels and Entities ............................................................ 108 Figure 1: Trend in Total Fertility Rate in Pakistan 1970-2009 ..................................................................................11 Figure 2: Trend in Contraceptive Prevalence Rate in Pakistan, 1990-2007 ..........................................................12 Figure 3: Trend in Modern Method-Mix of Contraceptives Since 1990 ................................................................12 Figure 4: Contraceptive Prevalence is Stagnating During 2001-2010, After Sharp Increase During 1990-2000 .......................................................................................................................................13 Figure 5: Procurement and Distribution of Contraceptives and Essential Medicines: Government of Pakistan ............................................................................................................................15 Figure 6: Theory of Change ...............................................................................................................................................17 Figure 7: Cause Analysis Diagram of Factors Affecting the Relative Performance of LMIS Across Pilot Districts ................................................................................................................................................31 Figure 8: Percentage of Pilot Districts (Including Service Delivery Points) with Less than One Month of Stock per Commodity from 2010–2012 (no. of Districts= 7). .....................................33 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 1 ACRONYMS CLR Contraceptives Logistics Report COC Combined Oral Contraceptives CPR Contraceptive Prevalence Rate CPT Contraceptive Procurement Tables CWS Central Warehouse and Supplies DG Director General DOH Department of Health DOPW Department of Population Welfare EAD Economic Affairs Division EDO Executive District Officer FEFO First Expire First Out FPAP Family Planning Association of Pakistan FY Fiscal Year GOP Government of Pakistan HAS Health Services Academy ISO International Standards Organization IUCD Intrauterine Contraceptive Device JSI John Snow, Inc. KII Key Informant Interview KP Khyber Pakhtunkhwa LHW Lady Health Worker LMIS Logistics Management Information System LPC Logistic and Procurement Cells LSAI Logistics System Assessment Instrument M&E Monitoring and Evaluation MDG Millennium Development Goal MEP Monitoring and Evaluation Program MOH Ministry of Health MOPW Ministry of Population Welfare MOS Months of Stock MSI Management Systems International, Inc. NGO Non-Governmental Organization NWFP North West Frontier Province P&D Planning and Development Division PDHS Pakistan Demographic and Health Survey PHC Primary Health Care PLM Provincial Logistics Manager POP Progesterone Only Pills PP Population Planning PWD Population Welfare Department SCM Supply Chain Management SDP Service Delivery Point SO Strategic Objective SOW Statement of Work SOP Standard Operating Procedure TA Technical Assistance MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 2 TACMIL Technical Assistance for Capacity Building in Midwifery, Information and Logistics TOR Terms of Reference TFR Total Fertility Rate UNFPA United Nations Population Fund USAID United States Agency for International Development WMS Warehouse Management Information System MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 3 PROJECT SUMMARY The purpose of the John Snow, Inc. (JSI)/DELIVER Project is to: (a) improve the Central Warehouse and Supplies (CWS) operations; (b) improve procurement capacity and increase the availability of contraceptive supplies for clients; (c) establish a Logistics Management Information System (LMIS); and (d) build capacity and enhance means of quality assurance. Table 1 provides a summary of the project. TABLE 1: PROJECT SUMMARY Project Name/Title JSI/DELIVER Project Contract Information Contract No. GPO-I-00-06-00007-00 Activity Manager Khalid Mahmood (Mr.), Project Management Specialist Project Start Date August 2009 Project Completion Date 2014 Project Location Nationwide USAID Objective Addressed IR 5.1.1: Improved Availability of Contraceptive Commodities to Health Centers and Health Workers Name of Implementing Organization John Snow, Inc. Budget US$10-15 million as technical assistance (TA) for capacity building of federal and provincial governments and US$89 million for procurement of contraceptives and development and operationalization of LMIS MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 4 EXECUTIVE SUMMARY Pakistan has a high unmet need for family planning due in part to the unavailability of contraceptive commodities and stock-outs.1 The Government of Pakistan (GOP) does not allocate sufficient resources toward the procurement of contraceptives and lacks the capacity to forecast, procure and distribute these commodities nationwide.2 THE DEVELOPMENT PROBLEM For the GOP to realize its short-term objectives of “making family planning services available in the remotest corners of the country, reducing the total fertility rate (TFR) from 3.8 percent to 2.15 percent and unmet need from 25 percent to 5 percent, and increasing the contraceptive prevalence rate (CPR) from 30 percent to 60 percent by 2030,” commodity security for all public and private sector outlets needs to be ensured. This entails rational procurement planning, institutionalizing population projections and forecasting, reviewing commodity requirements annually based on consumption patterns and current stock positions, and ensuring timely procurement and delivery of contraceptives.3 As early as 1991, a team from the United Nations Population Fund (UNFPA) evaluated Pakistan's contraceptive requirements and logistics management needs through the year 2000. One finding was that the “current logistics management of moving contraceptive stocks from the Ministry of Population Welfare (MOPW) to Ministry of Health (MOH) outlets was problematic.” The UNFPA team suggested that UNFPA help the government coordinate the procurement of donated contraceptives.4 In 2008, the USAID Technical Assistance for Capacity Building in Midwifery, Information and Logistics (TACMIL) Health Project conducted a situational analysis of the logistics system supporting population welfare in Pakistan and noted that while the MOPW procurement and logistics manual (developed with UNFPA support) focused on procurement, storage and distribution, it did not cover an important component - forecasting requirements. USAID’S RESPONSE TO THE PROBLEM USAID bought into the John Snow, Inc. (JSI)/DELIVER mechanism for use in Pakistan in August 2009. This intervention was designed to provide various supply chain strengthening inputs including developing a Logistics Management Information System (LMIS), undertaking warehouse rehabilitation, ensuring commodity security and supporting procurement capacity. The JSI/DELIVER Project focuses on improving and enhancing the Government of Pakistan’s in-country distribution and health commodity supply chains. JSI/DELIVER works closely with the Pakistan Planning Commission, the provincial and regional Departments of Health and Population Welfare, UNFPA and non-governmental organizations (NGOs) to determine the correct amount of health commodities to order each year5 , establish a modern storage and tracking system, and improve the distribution system throughout the country. It aims to ensure contraceptive commodity security at each level of the health supply chain and promote multilevel coordination on the delivery of health commodities. 1 Zero stocks of any contraceptive commodity at all three distribution levels, i.e., central warehouse, district and service delivery points (SDPs) during the last six months 2 Provincial and District Supply Chain Management Situation Assessment in Pakistan, 2011 3 Pakistan Demographic and Health Survey 2006-07, National Institute of Population Studies, Islamabad, Pakistan 4 United Nations Population Fund (UNFPA): Contraceptive Requirements and Logistics Management Needs in Pakistan; (New York, UNFPA, 1992). 5 Contraceptive commodities are purchased with U.S. Government funds to support the needs of Pakistanis. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 5 PURPOSE AND ASSESSMENT METHODOLOGY The purpose of this assessment is two-fold:  To determine how far JSI/DELIVER has come in achieving the goals outlined in its work plan and scope of work6 ; and  To identify problems that require correction to increase the program’s effectiveness. The evaluation team, working under MSI’s, USAID funded, Monitoring and Evaluation Project (MEP), adopted a mixed methods approach, integrating quantitative and qualitative evidence to shortlist districts from which to collect primary information and answer the evaluation questions. It adopted a two-step process for data collection. In Step 1, the evaluation team analyzed secondary data (LMIS reports) prepared by JSI/DELIVER (and available on the website, www.lmis.Pakistan) from 19 pilot districts to prepare a desk review report on the selection of eight districts for field visits. Within these eight sampled districts the team visited 20 district agencies. An observational checklist was developed to collect information on the physical structure and operation of the central warehouse and district stores. Key Informant Interview (KII) guides were developed to collect information on the various aspects of the Warehouse Management System (WMS), procurement, supply chain management and use of LMIS. In Step 2, during field visits, KIIs were held with the officials of public sector stakeholder agencies i.e., Department of Health (DOH), Department of Population Welfare (DOPW) and Lady Health Workers (LHWs) Program as applicable at all levels of the supply chain, from the districts to the officials at the central warehouse in Karachi. FINDINGS AND CONCLUSIONS – EVALUATION QUESTIONS Evaluation Question 1- To what extent is JSI/DELIVER achieving its objective to institute best practices and automation in the central warehouse? The findings on this question focus on three aspects of central warehouse rehabilitation and institution of best practices - infrastructure, operationalization and automation of inventory management. Infrastructure The central warehouse recently attained an International Standards Organization (ISO) 9001 certification, which indicates that it has been successfully rehabilitated and meets international standards and specifications for pallet and racking systems, commodity storage, physical environment and the application of the First-Expire-First-Out (FEFO) system. Operationalization Central warehouse operations are guided by a logistics/warehouse management document, which has been in place since 1987. Its automation of inventory control and supply chain processes is dependent on technical support from a full-time JSI/DELIVER project staff member. In addition, its operations are managed by 48 staff members, four of them trained in inventory management and eleven trained on equipment such as stackers, computers, barcode printers and scanners. Automation of Inventory Management Inventory management is only partially automated as the barcoding technology is currently in use for only two commodities - Progesterone Only Pills (POP) and an intrauterine contraceptive device (IUCD). The barcoding technology is linked to the WMS and is used for commodity transaction/product management for these two 6 The project goals are to: (a) improve central warehouse and supply (CWS) operations; (b) improve procurement capacity and increase contraceptive supplies;(c) establish a Logistics Management Information System (LMIS); and (d) build capacity and enhance means of quality assurance. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 6 commodities. Inventory management of other commodities is done using a paper-based system. WMS is not integrated with the central warehouse module of LMIS, and paper-based Contraceptives Logistics Report (CLR) forms (1-10) are not digitized. Conclusions JSI/DELIVER has largely met its objective of instituting best practices in physical infrastructure, storage operations and enhancing staff capacity while rehabilitating the central warehouse. The monitoring of two out of five commodities (POPs and IUCDs) is partially automated at all levels of warehousing and inventory control, the exception being the automation of barcoding. This partial barcoding automation along with the non-integration of the WMS with the central warehouse module of LMIS and lack of digitization of paper-based CLR forms within the electronic data entry interface of the WMS reduces the overall efficiency of the central warehouse and the quality of generated data. Evaluation Question 2 - To what extent is JSI/DELIVER achieving its objectives to improve procurement capacity? Findings Over the last three and half years, JSI/DELIVER’s input includes directly procuring commodities worth US$20 million for the GOP, developing a procurement manual for the Planning Commission, and providing technical and financial support to provincial governments for establishing logistic and procurement cells (LPC). Four LPCs are now in use, but none were functional at the time of field visits in December 2012. KIIs with federal and provincial agencies reveal that two Planning Commission officials received training in procurement, while information shared by JSI/DELIVER indicates that seven provincial agency staff members were trained in October 2011.7 Conclusions Improving capacity to procure contraceptives is multi-dimensional. Institutional capacity in terms of infrastructure and procurement manuals is still being developed and is in transition at the provincial level. The evidence on building human capacity in terms of training at the provincial level is mixed. The KIIs with provincial officials indicate that this capacity is weak, while documentary evidence from JSI/DELIVER suggests otherwise. The application of training knowledge in terms of actual procurement actions by the provincial officials cannot yet be verified, as contraceptive procurement will remain centralized until 2014. Evaluation Question 3 - Have procurement activities been automated, and is the government using the web-based Logistics Management Information System (LMIS) procurement manual and linking it to procurement planning and forecasting? Findings The Planning Commission is preparing contraceptive procurement plans in conjunction with JSI/DELIVER using integrated Contraceptives Logistics Report (CLR 6) consumption data, which is entered at the provincial levels for pilot districts and the central level for all non-piloted districts. In December 2012, provincial government agency officials, with the support of JSI/DELIVER, conducted a contraceptive quantification and prepared the Contraceptive Procurement Tables (CPT) themselves using data from the integrated CLR 6. The Planning Commission, in collaboration with JSI/DELIVER, utilizes the procurement manual in the development of CPTs. The electronic LMIS installed in 19 pilot districts is now a direct source of consumption data, which can be used by the GOP and JSI/DELIVER for the preparation of CPTs. However, in its current state, the electronic LMIS is not designed to allow for automated procurement planning. 7 List of total No, of participants in the workshop titled, “Skill Development Workshop for Conducting International Contraceptive Procurement in Public Sector, Oct 24th , 2011, supplied by the country office of /JSI/DELIVER was 22. However only 7 belonged to the provincial agencies and affiliations of 4 officials could not be ascertained from the list. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 7 Conclusions Automation of procurement activities would, to a great extent, require that all districts use the electronic LMIS and function as the direct source of utilization data required for generating CPTs. This is not yet happening as the electronic LMIS is still being rolled out. The above findings indicate a manual link between the LMIS-based data for procurement forecasting and planning rather than an automated link between the web-based LMIS system and production of CPTs and/or procurement planning and forecasting at the federal and provincial levels. Evaluation Question 4 - To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the central, provincial, and eight sample pilot districts to ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? Findings These findings arose from KIIs with provincial and district government officials in eight out of the 19 pilot districts. A structured checklist was also used to assess the electronic LMIS. The web-enabled LMIS is operational in three out of four provincial/regional procurement and logistic cells visited by the evaluation team.8 As of December 2012, all district government officials had been trained in the LMIS and JSI/DELIVER had made computer hardware available to all the pilot district government agencies visited. Forty-five percent of the 20 visited pilot district government agencies are using the LMIS for logistics planning and reporting (see Table 3 in the body of the report for a breakdown of the electronic LMIS). Eight out of 20 pilot districts agencies visited are not connected online. Three pilot district agencies use a local module disk of LMIS, which is hand carried to the agency provincial office to upload the data to the LMIS. With the support of JSI/DELIVER, the GOP has revised the Combined Report and Requisition form (named CLR 6 by the GOP) to include logistics data elements for public and private entities. KIIs with all district government officials revealed the following as key themes influencing the roll-out and use of the LMIS for supply chain management in the districts: no internet connection (8 district agencies), frequent power outages (20 district agencies), turnover of trained staff (9 district agencies), need for refresher training (6 district agencies) and computer illiteracy (5 district agencies). Conclusions JSI/DELIVER has provided hardware and training for the LMIS to all district officials. While management of the contraceptive supply chain using web-based LMIS technology at the district level is weak, it is much stronger at the provincial level. The DOPW is performing better at installing and utilizing the software application than the LHW and the DOH. Ongoing effective functioning of the LMIS for use in supply chain management (SCM) is still limited by certain problems such as high staff turnover, frequent power outages, need for refresher training and inadequate follow-on support and assistance from JSI/DELIVER. 8 For LMIS to be operational in a provincial agency, all that is required is the presence of a desktop computer or laptop (not necessarily dedicated) and an on-line connection. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 8 Evaluation Question 5 - How widespread and frequent are stock-outs at the central warehouse and provincial/district-level warehouses, what is their cause, and when do they occur? Findings Secondary data from seven out of eight pilot districts (Rawalakot was excluded due to non-availability of data) visited are presented as findings in this report. The percentage of districts with less than one month of stock is used as a proxy for a stock-out in the analysis below. The months of stock figure is calculated by dividing the total quantity of stock on hand by the average quantities of monthly consumption. Any districts with less than one month of stock are included in the percentages below.9 At the end of November 2012, there was no stock-out in any district, including at the service delivery point (SDP)- level. The percentage of pilot districts (including SDPs) with less than one month of stock between January–November 2012 per commodity is given below in Table 2. Data analysis for the pilot districts illustrates that stock-outs (as defined above) for each of the five above mentioned commodities have occurred less frequently overtime. In 2010, all of the pilot districts experienced less than one month of stock for one or more of the above commodities, while in 2012 only 14 percent of pilot districts experienced less than one month of stock. In 2012, none of the pilot districts experienced less than one month of stock of POPs and three-month Injectable Contraceptives, as compared to 2010 when 57 percent of districts had less than one month of stock of POPs and 14 percent of districts had less than three month’s stock for three-month Injectable Contraceptives. There remains, however, a considerable level of stock-outs for IUCDs in the central warehouse and all pilot districts. TABLE 2: THE PERCENTAGE OF PILOT DISTRICTS (INCLUDING SDPS) WITH LESS THAN ONE MONTH OF STOCK BETWEEN JANUARY–NOVEMBER 2012 PER COMMODITY Items Percentage Condoms 42% Intrauterine Contraceptive Device (IUCD) 57% Combined Oral Contraceptives (COC) 29% Progesterone Only Pills (POP) 0% Injectable Contraceptives (3 months) 0% Factors affecting stock-outs at the central warehouse and district stores include delays in international shipment to the central warehouse, and inland transportation from the central warehouse to districts. Other issues include use of inconsistent formulas for calculating quantities of contraceptive requisitions, weak coordination among agencies in consolidating CLR 6 forms for aggregate requisitions, and inadequate storage space in district stores. Conclusions The stock situation has progressively improved since the onset of the JSI/DELIVER project; however the maintenance of minimum and maximum levels of stock is not occurring in the 8 pilot districts visited. 9 As an example if two out of seven sampled pilot districts in 2010 (over the 12 month period) had less than one month of stock (down to the level of SDP) the calculated percentage is (2/7)x100=29. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 9 RECOMMENDATIONS The recommendations of the evaluation study address only the following two areas, i.e., automation and capacity building: Automation  The integration of the WMS with the central warehouse module of the LMIS will reduce duplicative efforts in data entry and improve the overall efficiency of the central warehouse and its staff.  The barcoding technology should be extended to include the remaining contraceptives supplied to the central warehouse. The warehouse management system should be redesigned to be interoperable with the online LMIS. Data entry interfaces should be made identical to the Planning Commission’s paper-based system for ease of data transcription and to preclude data entry errors.  It may be useful in the future to develop a module within the LMIS for procurement planning that could serve as an interface to automatically compute contraceptive procurement tables.  While JSI/DELIVER has significantly revamped the logistics data collection and reporting system by integrating the CLR 6 and rolling out the electronic LMIS, it is necessary to incorporate plans to provide feedback to government agencies/staff generating data at the source. The current logistics data flow system is unidirectional with a lot of focus on reporting and little focus on feedback on supply chain performance. Capacity Building  Following the devolution of the procurement responsibility from the MOPW to the provincial DOPW, it is necessary for JSI/DELIVER to intensify its procurement capacity building efforts at the provincial level. KIIs with provincial government officials in each of the three government agencies in the pilot districts revealed limited knowledge of the procurement planning process and involvement in procurement planning. It is therefore paramount to involve provincial officials in the preparation of contraceptive procurement tables and other components of the procurement process.  It is important to intensify capacity building efforts for DOH and LHW provincial and district government agencies so that they may work more closely with the DOPW to better coordinate the supply chain system and streamline logistics reports. This would increase the availability of contraceptives and make data more visible in all three district stakeholder agencies.  Despite the establishment of minimum and maximum levels of contraceptive commodities at the three levels of the supply chain system, it is apparent from field visits that these limits are not adhered to at the district level. JSI/DELIVER needs to intensify onsite technical assistance to ensure that district government agencies calculate the correct amounts for requisition by factoring in the appropriate indicators: minimum and maximum levels of stock, lead time and stock on hand.  Refresher trainings, regular onsite mentoring, and institutionalizing supportive supervision within the GOP are required to ensure utilization of the LMIS at the source of data collection and supply chain management (most critically in AJK). Data will be more centrally visible, but in order to facilitate ownership of the supply chain system by the GOP at all levels, promotion of data management, use and feedback at the district and service delivery points is paramount. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 10 INTRODUCTION Pakistan, with a population of 164.8 million, was the sixth most populous country in the world in 2007. The country’s annual population growth rate is 1.8 percent, with a Total Fertility Rate (TFR) of 4.1 percent. With a low contraceptive prevalence rate of only 30 percent, the total unmet need for family planning services is considered high at 25 percent. Contraceptive use among married women is the highest in Punjab province (33 percent), followed by Sindh (27 percent) and the Khyber Pakhtunkhwa (KP) (25 percent) and is lowest in Balochistan (14 percent). The contraceptive mix across provinces is as follows: in Punjab and Sindh, female sterilization is the most commonly used method, followed by condoms, while in the KP, condoms and withdrawal are the most popular methods. Contraceptive use in Balochistan consists almost entirely of pills and female sterilization.10 Pakistan started public sector family planning activities during 1960-1965 using health department facilities in Pakistan’s four provinces. Between1965 and1970, Pakistan developed a comprehensive and detailed family planning policy and an independent family planning unit was created. From 1970 to 1975, the ‘continuous motivation approach’ was used at the grassroots level. In the 1980s, the role of NGOs was institutionalized, the concept of contraceptive social marketing was introduced, and the National Institute of Population Studies was established. The 1990s saw the elevation of the status of the Population Welfare Division and the establishment of a Ministry of Population Welfare. The Population Policy of 2002 envisioned achieving universal access to family planning and services by 2010 and replacement level fertility by 2020.11 Similarly, the Population Perspective Plan (2002-2012) sought to achieve a contraceptive prevalence rate of 57 percent by 2012.12 Despite the early start of family planning activities, fertility rates only started declining from 6.8 children per woman after1988. The decrease was rapid between 1991 and 2000, but the next decade saw this rate slow from 4.8 to 4.1 children per woman. During this period, the contraceptive prevalence rate rose to 32 percent until 2003 and then dropped to 30 percent. The population growth rate has also stagnated at 2.05 percent, against a target of 1.72 percent annually. The draft of the 2010 National Population Policy stated that the fertility rate could be reduced to 2.1 percent by 2030. In order to attain widespread contraceptive security, an effective and efficient supply chain system is required to ensure the availability of contraceptives for Pakistani people to make voluntary choices regarding fertility.13 Various programmatic constraints have slowed the pace of MOPW’s progress including weak political commitment, inconsistencies in population policies and programs, lack of provincial ownership, weak capacity and commitment level of program personnel, and inadequate allocation of resources. For example, from 2003-2008, the Government of Pakistan committed Rs. 21 billion to the program, but provided only Rs. 14.5 billion. Similarly, for raising CPR from 34 to 45 percent as envisioned in the five-year Population Welfare Plan (2003-2008), contraceptives worth Rs. 2.9 billion were procured and provided against an estimated national contraceptive requirement of Rs. 4 billion. Moreover, during 2008-2010, the program remained on extension with inadequate financial support of Rs. 6.6 billion when Rs. 9.5 billion was originally allocated.14 10 Pakistan Demographic and Health Survey, 2006-2007, National Institute of Population Studies. 11 Manual of National Health Standards for Family Planning Services: Technical Assistance Management Agency to the National Health and Population Facility, Pakistan, Ministry of Health, Ministry of Population Welfare, 2007. 12 Population Sector Perspective Plan (2002 - 2012), Ministry of Population Welfare, Government of Pakistan. 13 Draft National Population Policy 2010, Government of Pakistan. 14 Khan, A. G.: Pakistan Report, 2010, Gillespie Foundation, Population Communication, New York. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 11 FIGURE 1: TREND IN TOTAL FERTILITY RATE IN PAKISTAN 1970-200915 Source: National Institute of Population Studies based on PDS Survey 2007 (FBS) Bilateral and multilateral organizations, such as the United Nations and the World Bank, have historically provided the bulk of donor support for population activities in Pakistan. The U.S. Government’s assistance to Pakistan in the population planning sector has been sporadic for most of the past decade. Until 2005, the U.S. Agency for International Development’s (USAID) highest level of annual funding for population planning sector had been just over US$600,000. However, in 2005, USAID provided US$10.6 million to Pakistan, with nearly 80 percent of the funds directed toward family planning activities.16 ISSUES 1. Administrative Pakistan’s family planning program is administered by two government ministries, the Ministry of Health (MOH) and the Ministry of Population Welfare (MOPW), each with inefficiencies in implementation. In both ministries, family planning services are delivered through weak logistical systems - they lack contraceptives at service delivery points (SDPs), and employ ill-trained and ill-equipped staff to provide services to clients.17 MOPW efforts to promote small family norms and behavior changes through awareness and motivational campaigns focused mainly on urban areas, and failed to target peri-urban and rural populations. The performance of the Lady Health Workers (LHW) Program, an initiative of the MOH in the 1990s, is mixed. Oxford Policy Management (2002) considered it an effective program in delivering family planning services in 2001, while a third party evaluation’s findings in 2009 contradicted that belief.18 2. CPR, Method Mix and Unmet Need The trend of contraceptive method mixing indicates female sterilization and the use of traditional methods as the common approaches in family planning practice. While over the years this has led to a lowering of fertility rates, it also resulted in the decline in use of conventional methods (oral pills, IUCD and injectable contraceptives). The decline in conventional method use exacerbated the persistent unmet 15 Percentages for 2009 are projected. 16 Hardee K., Leahy E., Research Commentary: Population, Fertility and Family Planning in Pakistan: A Program in Stagnation, Population Action International, 2008 Vol.3 Issue 3. 17 Hardee K., Leahy E., Research Commentary: Population, Fertility and Family Planning in Pakistan: A Program in Stagnation, Population Action International, 2008 Vol.3 Issue 3 18 Draft National Population Policy 2010, Government of Pakistan 0 1 2 3 4 5 6 7 1970-75 1984 1986-91 1992-96 1997 -2000 2006 2007 2009 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 12 need for contraception and widened the gap between current- and ever-users of contraceptives, indicating high levels of drop-outs and a high incidence of induced abortions. FIGURE 2: TREND IN CONTRACEPTIVE PREVALENCE RATE IN PAKISTAN, 1990- 2007 Source: Status of Women Reproductive Health and Family Planning Survey FIGURE 3: TREND IN MODERN METHOD-MIX OF CONTRACEPTIVES SINCE 1990 11.9 17.8 23.9 27.6 32.1 29.6 0 5 10 15 20 25 30 35 PDHS 1990 -91 PCPS 1994 -95 PFFPS 1996 - 97PRHFPSS 2000 -01SWRHFPS 2003 PDHS 2006-07 0 1 2 3 4 5 6 7 8 9 Female sterilisation Condom IUD Injectionable Pills PDHS 1990-91 PRHFP 2001-02 SWRHFPS 2003 PDHS 2006-07 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 13 FIGURE 4: CONTRACEPTIVE PREVALENCE IS STAGNATING DURING 2001-2010, AFTER SHARP INCREASE DURING 1990-2000 The use of different contraceptive methods relates to the supply of and support for those methods. Institutional support was provided for female sterilization through the public sector, and for condoms through social marketing in urban areas and the LHW program in rural areas. Despite these outreach efforts, supplies of injectable contraceptives, oral pills and IUDs remained inadequate, consequently causing a drop in their use.19 In view of the projected increase in CPR to 38 percent by 2015 and to 60 percent by 2030 (draft National Population Policy 2010), additional contraceptive requirements of Rs. 42.9 billion are anticipated in the coming years. The policy further states “both commodity security and supply chain management will be improved to avoid overstocking and stock-outs at any level at any time.” 3. Pre-Devolution Before the devolution of the Ministries of Health and Population Welfare in 2010-11, the Government of Pakistan had three major components in the health sector that procured, stored, and distributed drugs/medicines and contraceptives:  The Ministry of Health (MOH), a supervisory body, established policies related to health matters, whereas the Departments of Health (DOH) in the provinces and districts executed these policies. The DOHs also delivered health care services.  Ministry of Population Welfare (MOPW) provided oversight of reproductive health matters and procured most of the contraceptives (through UNFPA) that were distributed throughout Pakistan.  National Program for Family Planning and Primary Health Care (also referred to as the Lady Health Workers (LHW) Program functioned under the MOH umbrella to improve the availability of contraceptive supplies, and thus increase the country’s CPR in rural areas. In the private sector, Green Star works on a program for the social marketing of contraceptives, while Rahnuma￾Family Planning Association of Pakistan (FPAP), an International Planned Parenthood Federation affiliate, has a 19 Khan, A. G.: Pakistan Report, 2010, Gillespie Foundation, Population Communication, New York. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 14 nation-wide service delivery program supported by a network of hospitals and clinics. The latter receives contraceptives through the government procurement system and reports quarterly on its performance to the government. 4. Post-Devolution With the 18th Constitutional Amendment (2010-11) and the 7th National Finance Commission Award of 2010, the health, population, and other social sector ministries were devolved to the provinces. As a result of overall structural adjustments, repositioning of roles and responsibilities, and realignment of administrative setup, the residual set-up of MOPW was placed under the Population Planning Wing (PP Wing), the Planning and Development Section (P&D), and the Planning Commission of Pakistan (Federal body). The PP Wing and P&D are responsible for procurement planning for contraceptives and oversight of the Central Warehouse in Karachi. The Provincial Population Welfare Departments’ role is to compile monthly performance reports from the districts and forward them to the PP Wing. The monitoring role has been curtailed, however, due to the lack of funds. The District Population Welfare Offices’ main responsibilities are to ensure family planning services and contraceptive supplies to SDPs in the district, receive performance reports and prepare the integrated requisition forms (from all three government stakeholders and FPAP) for contraceptives for onward submission to the central warehouse. The Provincial Health Departments are mandated with provision of health and limited family planning services. The National Program for Primary Health Care (PHC) and Family Planning, popularly known as the LHW program, is a vertical program within the health department that is both well-funded and well-staffed. The program provides grassroots-level family planning services: condoms, pills, injectable contraceptives, and referrals for long-term contraceptive services. It is often criticized, however, for more involvement in PHC than family planning service delivery. At the district level, the Executive District Officer (EDO) for health is responsible for the MOH and LHW programs and Department of Population Welfare (DOPW). MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 15 FIGURE 5: PROCUREMENT AND DISTRIBUTION OF CONTRACEPTIVES AND ESSENTIAL MEDICINES: GOVERNMENT OF PAKISTAN Source: USAID Technical Assistance for Capacity Building in Midwifery, Information and Logistics (TACMIL) Project 2008. Government of Pakistan NP-FP and PHC Contraceptives Supply Chain Management Essential Drug Supply Chain Management Essential Drug Supply Chain Management Contraceptives Supply Chain Management Essential Drug Supply Chain Management Three distinct departments deal with similar drugs, but utilize five separate logistics systems targeted for the same community MOH/DOH MOPW MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 16 THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE THE PROBLEM STATEMENT If Pakistan is to meet its population policy goal of 2010, which calls for accelerating fertility transition to achieve population stabilization, the population factor would need to be placed at the center of all its strategic development, poverty reduction and millennium development goal (MDG) planning. In order to realize its short￾term objectives of “making family planning services available in the remotest corners of the country, reducing the TFR from 3.8 percent to 2.1 percent, unmet need from 25 percent to 5 percent, and increasing CPR from 30 percent to 60 percent by 2030,” commodity security for all public and private sector outlets would need to be ensured. This entails planning rational procurements, institutionalizing population projections and forecasting, reviewing commodity requirements annually based on consumption patterns and current stock positions, and ensuring the timely procurement and delivery of contraceptives. As early as 1991, a UNFPA team visited Pakistan to evaluate the country’s contraceptive requirements and logistics management needs through the year 2000. One finding stated that the “current logistics management of moving contraceptive stocks from the MOPW to the MOH outlets was problematic.” The team suggested that the UNFPA help the government coordinate procurements of donated contraceptives.20 Similarly, a Population Council Pakistan Report in 1998 stated, “The supply system to districts is troubled by non￾traceable stocks; serious supply inadequacies occurring at the SDPs, storekeepers and providers oblivious to logistic requirements.” The report’s recommendations were in line with the following: determination of required stock levels, training and provision of manuals, annual inventory, defining stock levels, and non-traceable stocks.21 In 2008, the USAID TACMIL Health Project conducted a situational analysis of the logistics system supporting population welfare and health in Pakistan and noted that both the MOPW and LHW Programs had developed logistics instructional manuals with UNFPA support and trained staff in logistic management. While the MOPW manual focused on procurement, storage and distribution aspects, it did not cover an important component - forecasting requirements. The LHW program suffered from high staff turnover and its pioneering LMIS software application was complex and difficult to use. Also, there was inconsistency in the functioning of provincial procurement committees and in the logistics operations across the various ministries, provinces, and district levels. Non-formal forecasting methods were applied, warehouses lacked shelving and stock or bin cards, forklifts were rarely available, inventory was not properly maintained, and the efficiency and effectiveness of supply chain management was compromised due to poor maintenance of the infrastructure supporting the logistical system. THE THEORY OF THE INTERVENTION USAID/Pakistan’s strategic objective (SO) for the health program is “IR 7.1: Improved quality and use of maternal, newborn, and child health and reproductive services.” This reflects the fact that all Mission activities in reproductive and maternal child health programs are integrated in their design. In line with this objective, the project’s technical assistance is geared towards improving contraceptive product availability and system strengthening, supporting the Mission’s strategic objective and intermediate results. USAID, through IR 7.1, intends to have more services promoting family planning and maternal child health available in districts targeted by the SO (currently underserved areas) and also to have those services delivered at an improved level of quality. It will bring improved reproductive health services closer to vulnerable populations. In November of 2008, upon request from the MOPW, a team 20 United Nations Population Fund: Contraceptive requirements and logistics management needs in Pakistan. (New York, UNFPA, [1992]. 21 Population Council, Pakistan: A Study of Operations of Contraceptive Logistics System of Population Welfare Program, Pakistan, 1998 Apr. vii, 36 [65] p. (Research Report No. 8). MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 17 comprised of ministry officials and representatives from the USAID Mission and JSI/DELIVER Project visited the central warehouse in Karachi. After reviewing and discussing potential improvements needed in the central warehouse, the MOPW asked for USAID’s support in providing technical assistance to carry out improvements. While the team’s focus was on immediate improvements, the activities discussed also covered contraceptive security - ensuring the availability of, and access to, quality contraceptives for all Pakistanis. If improvements are implemented in managing the supply chain through central warehouse rehabilitation, establishing the LMIS, and building governmental (provincial and local) technical capacities, it will lead to more effective delivery of contraceptives and overall sustainability of the delivery system. FIGURE 6: THEORY OF CHANGE Procurement and SCM Rehabilitation of CWH Stabilizing Population Growth Improve Contraceptive Use (CPR) and Reduce Unmet Need Less Leakage and Waste Reduce Out-of-Stock Commodities Timely Availability of Commodities Effective Delivery of Contraceptives Sustainability of Delivery System Capacity Building for Procurement SCM and LMIS LMIS MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 18 PROJECT DESIGN The JSI/DELIVER Project started its operation in Pakistan in August 2009 with USAID support to provide inputs on the Logistics Management Information System (LMIS), warehouse rehabilitation, commodity security, procurement capacity and other support activities related to contraceptive commodities. The JSI/DELIVER Project works nationally, in all districts of Pakistan, and targets those involved in the contraceptive commodity supply chain. Initially, in the JSI/DELIVER Project’s first year in Pakistan, all technical assistance (TA) activities were provided to the Ministry of Population Welfare (MOPW). However, with growing needs at the Ministry of Health (MOH), coupled with the 18th Amendment leading to the devolution of the MOH role to the provinces, all distribution of contraceptives was to be carried out through the MOH. In the second year as well, most activities, where appropriate, involved the MOH’s provision of technical assistance and dissemination. By the end of the third year, the country was fully devolved and most governing power had been transferred to the provinces. Both the MOH and the MOPW no longer exist at the federal level. The JSI/DELIVER field office transitioned to working directly with the provincial health and population departments, liaising with program contacts within the Planning and Development (P&D) Division of the Planning Ministry and Economic Affairs Division (EAD) at the federal level in Islamabad. Sustainability is the cornerstone of any technical intervention from its conceptualization to implementation and completion of the review stages. In order to keep the ownership with the GOP, plans would be regularly discussed with the GOP and stakeholders, their inputs included, progress reviewed jointly, and follow-up actions initiated. PROGRAM IMPLEMENTATION The purpose of JSI/DELIVER is to: 1. Improve central warehouse and supply chain operations; 2. Improve the procurement capacity of the Government of Pakistan; 3. Improve and increase the supply of contraceptives (through procurement22); and 4. Establish the LMIS. The project’s interventions focus on two key dimensions: 1. Improving the performance of the health system by redressing weak supply management systems; and 2. Improving the availability of contraceptives in the public sector by working in partnership with the Ministry of Population Welfare (MOPW) and Ministry of Health (MOH) to achieve contraceptive security through a series of interventions in the above mentioned key focus areas. To meet the objectives of the project the following activities are envisioned: Improvement of Central Warehouse and Supply Chain Operations Technical assistance would be provided for rehabilitating the extension of the central warehouse with a pallet and racking system. This includes procurement and installation of equipment including barcoding, instituting best warehouse management practices, and introducing automated (paper-free) warehouse management. Technical assistance (TA) would also be provided to institute evidence-based warehousing and effective distribution of contraceptive supplies. The collaborating partners: USAID, MOPW, and MOH. 22 Contraceptives are purchased with U.S. Government funds. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 19 Development of Contraceptive Procurement Tables Technical assistance would be available for developing and updating CPTs to better understand the contraceptive requirements for Pakistan involving all major stakeholders. A National Procurement Manual would be developed, endorsed by the GOP, and relevant staff would be trained. Logistics and procurement cells would be set up in key provinces (Punjab, KP and AJK). Later procedures for procuring non-contraceptive items would be outlined and a separate manual developed. This would lead to better national forecasting and procurement planning and improved quantification and resource mobilization. The collaborating partners: the GOP, the private sector, bilateral and multilateral donors, and USAID. Establishment of LMIS Systems including Equipment Provision Technical assistance would be provided for establishing a national level web-based logistics management information system for tracking contraceptive commodities and their utilization and management. This would include carrying out individual district assessments, developing and installing the necessary software and upgrading existing hardware, and procuring hand-held devices for recording contraceptive uptake at the service delivery points. These measures would lead to evidence-based procurement planning and monitoring, and establishing a web-based national LMIS. The collaborating partners: USAID, MOH, MOPW, provincial departments of health, provincial population welfare departments, social marketing organizations and national NGOs. Contraceptive Security Support As the GOP works to improve contraceptive security, other system strengthening activities would be undertaken such as contraceptive security assessments, the revitalization of the Rural Health Centers Committee, coordination amongst stakeholders, the development of a National Contraceptive Security Strategy, and the facilitation of regular forecasting meetings. Advocacy activities would highlight the need for budgeting funds for both contraceptive support as well as transportation costs for commodities. It is critical to continue to press federal and provincial officials to ensure the provision of adequate funds for transportation at the district level for which the directive needs to be reflective of the PC-1 document that the GOP prepares for planning of resource allocations. Training in supply chain management and LMIS operations Technical assistance would be provided in the capacity building process of the GOP and private sector personnel in supply chain management and in operating the LMIS system, including the training of LHWs on the use of hand￾held devices for recording the delivery of contraceptives. This would institutionalize SCM for contraceptives, by introducing a 2-3 credit course for academia and a stand-alone course for public health logisticians, and an internship program to improve the critical mass of logisticians and capacity for SCM for contraceptives. The project would spend US$10-15 million for TA for capacity building of federal and provincial governments and US$89 million on procurement of contraceptives and development and operationalization of the LMIS. PROJECT OPERATION In order to support the technical assistance work planned with the JSI/DELIVER Project, an office was set up in July 2009 with local staff in Islamabad. The country office team currently consists of 23 members including six Provincial Logistics Managers (PLMs) located throughout Pakistan, as well as a staff person within the central warehouse in Karachi to provide specific TA support. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 20 PURPOSE OF THE ASSESSMENT USAID/Pakistan commissioned a mid-term evaluation of the USAID JSI/DELIVER Project to determine the extent to which the program has been successful in building the GOP’s capacity (at both the federal and provincial levels) to manage the supply chain of contraceptives effectively and efficiently. At the end of the project, this function is expected to be taken over by both tiers of the GOP. As indicated in the JSI/DELIVER Evaluation SOW (Annex 1), the purpose behind this assessment is two-fold:  To determine how far JSI/DELIVER has come in achieving the goals set out in its work plan and scope of work23; and  To identify problems that requires correction in order to increase the effectiveness of the program. The intent of this assessment is to: (a) share the results with USAID and the implementing partner- JSI/DELIVER; and (b) initiate course correction; and (c) apply lessons learned to present and future work. ASSESSMENT QUESTIONS As indicated in the JSI/DELIVER assessment SOW, the evaluation focused on five specific questions outlined below: 1. To what extent is JSI/DELIVER achieving its objective to institute best practices (human resource management, infrastructure, operationalization and automation) in the central warehouse? Explanation: a) Has the warehouse been rehabilitated; b) Is the pallet and racking system in place and operational; c) Is the warehouse automated; d) Has the staff been trained (is there a warehousing checklist); e) Have staff job descriptions been revised, are there performance measurement standards for staff, etc.); f) Have periodic evaluations of the warehouse been conducted, and results been used for continual quality improvement; and g) To what extent is the government able to run the warehouse independently or is it dependent on JSI/DELIVER? 2. To what extent is JSI/DELIVER achieving its objectives to improve procurement capacity? Explanation: Did JSI/DELIVER provide adequate technical assistance to build the capacity of the public sector (central and provincial levels) to quantify/forecast contraceptive needs, ensure allocation and disbursement of adequate resources for procurement, and procure contraceptive commodities by the government departments? What measures did JSI/DELIVER use to build the government’s capacity to conduct these tasks? 23 The project goals are to: (a) improve Central Warehouse and supply (CW&S) operations; (b) improve procurement capacity and contraceptive supplies;(c) establish a Logistics Management Information System (LMIS); and (d) build capacity and enhance means of quality assurance. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 21 3. Have procurement activities been automated, and is the government using the web-based logistics management information system (LMIS) procurement manual and linking it to procurement planning and forecasting? 4. To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the central, provincial and eight sample pilot districts to ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? Explanation: This question will explore the effectiveness of JSI/DELIVER’s capacity building efforts to establish a LMIS (including LMIS training, LMIS equipment procurement and operationalization, LMIS infrastructure, and follow-up support for implementation); and the extent to which private (e.g., Green Star, Marie Stopes, People’s Primary Healthcare Initiative) and public stakeholders are compliant with LMIS reporting requirements. Are new technologies, including barcoding, an automated inventory control system at the central warehouse and a web-based LMIS operational? To what extent has the government assumed the responsibility of managing the supply chain, versus being dependent upon JSI/DELIVER for this? 5. How widespread and frequent are stock-outs at the central warehouse and district-level stores and what is their cause and when they occur? Explanation: Stock-outs are an indicator of contraceptive availability and of supply chain capacity. The evaluation will use LMIS data when available to investigate trends in stock-outs and present an overview (with visuals) of these trends. If LMIS data are not adequate or available, the evaluation will rely on site visits or other secondary data to address this question. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 22 EVALUATION DESIGN AND METHODOLOGY EVALUATION DESIGN This section provides an overview of the research design and data collection methods along with a description of the data collection instruments and sampling strategy. The key sources of data for this evaluation included secondary information from JSI/DELIVER management and primary evidence from visits to the Planning Commission, the Provincial and District Departments of Population Welfare Department, the central warehouse, and district stores of the Department of Health (DOH), the (DOPW), and the Lady Health Worker’s (LHWs) offices. The data collection methods helped to determine how successful JSI/DELIVER has been in improving supply chain management through central warehouse rehabilitation, establishing the LMIS, and building governmental (central, provincial and district) technical capacities. The Monitoring and Evaluation Program (MEP) evaluation team adopted a mixed method approach to the evaluation. It adopted a two-stage process for data collection. In Stage 1, the evaluation analyzed the secondary data (LMIS reports) on districts, prepared by JSI/DELIVER (and available on the website www.lmis.Pakistan), to prepare a desk review report on the selection of districts for field visits. In Stage 2 it developed instruments to collect quantitative and qualitative evidence from the selected districts to answer the evaluation questions. The information obtained is analyzed relative to i) two levels of the supply chain system (the central warehouse and district stores), and ii) public sector stakeholder entities (i.e., DOH, DOPW and LHWs) at the central, provincial and district levels as applicable. DATA COLLECTION METHODS The evaluation used the following types of secondary information documented by JSI/DELIVER management in Pakistan/worldwide and USAID (Country Office Pakistan): i) LMIS-based monthly reports generated by the central and district stores on all aspects of contraceptive procurement and supply chain management in the 19 pilot districts since January 2011–November 2012. ii) Project documents including cooperative agreements, Project Monitoring Plans (PMPs), and work plans for the years 2009-2010, 2010-2011, and 2011-2012. iii) The contraceptive procurement manual published by the Government of Pakistan and staff training manuals. iv) The Logistics Handbook that documents best practices in contraceptive procurement and supply chain management.24 The evaluation also used the following methods to collect primary data and information25: i) A Logistics System Assessment Instrument (LSAI) was developed to assess a) the presence and use of physical infrastructure and equipment, b) computerization and operationalization of the LMIS, c) automation of procurement and supply chain processes including inventory control, d) transportation, and e) training of staff. This instrument was used to collect information during site visits to the central warehouse and also sampled district stores. 24 The Logistics Handbook: A Practical Guide for the Supply Chain Management of Health Commodities, Task Order 1, USAID/JSI/DELIVER PROJECT, 2011 25 The following documents were consulted in preparing the instruments: i) Logistics System Assessment Tool (LSAT), USAID/JSI/DELIVER, January 2009, ii) Logistics Indicators Assessment Tool (LIAT), USAID/JSI/DELIVER, 2008, iii) Measuring Supply Chain Performance: Guide to Key Performance Indicators for Public Health Managers, USAID/JSI/DELIVER, May 2010 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 23 ii) An Observational Assessment Instrument was developed to assess the extent to which the central warehouse and district stores were following the best practices and standards in infrastructure of warehouse/store and inventory management. This instrument was used to collect information during site visits to the central warehouse and sampled district stores. iii) Key Informant Interviews (KIIs) with senior officials of the Planning Commission, director of the central warehouse and management of district stores. iv) KIIs with provincial and district DOH, DOPW and LHWs chain officials. v) Interview guide for group discussions with staff of the central warehouse and district stores. DATA ANALYSIS METHODS AND SAMPLING In Stage 1 of the data analysis, pilot district LMIS reports are used to rank the districts as ‘more’ or ‘less’ compliant. Among the 19 pilot districts, a convenience sample of eight is selected (two in each province, one ‘more’ and one ‘less’ compliant district) for site visits and primary data collection. The criteria for selecting ‘more’ and ‘less’ compliant districts is based on the following three indicators: i) percentage of non-reporting months, ii) variations in the months of stock (MOS) of condoms,26 and iii) number of stock in months that is less than one month. The above three indicators are calculated for the 13 months of data from the period July 2011–July 2012.27 Along with the central warehouse in Karachi, the following eight districts were selected based on the above indicators for field visits and KIIs:  Punjab Province, Districts: Lahore and Muzaffargarh  Sindh Province, Districts: Karachi (East) and Hyderabad  Khyber Pakhtunkhwa (KP) Province, Districts: Peshawar and Abbottabad  Azad Jammu and Kashmir (AJK) Region, Districts: Muzaffarabad and Rawalakot Indicator data from the LMIS reports of seven (Rawalakot is excluded due to non-availability of data) sampled districts is used to analyze the stock-outs between January and November 2012 (Qs. 5). Indicator data from the LSAI, including observations, is used for simple descriptive/frequency analysis highlighting and quantifying various aspects of supply chain processes, including automation, barcoding, and automated inventory control. The analysis assesses the effectiveness of supply chain management in the central warehouse and district stores (Qs. 1, 2, and 4). Indicator data from the Observational Assessment Tool is used to qualitatively analyze the extent to which the central warehouse and selected district stores follow best practices in warehouse/store infrastructure and inventory management (Qs. 3 and 4). To supplement findings from LSAI and observational assessment instruments, qualitative assessments on users’ and administrators’ perceptions on the strengths and weaknesses of the procurement and supply chain system and the LMIS is conducted using semi-structured interviews with key informants. KIIs were conducted with 50 officials at three tiers of the government (federal, provincial and district-level) and across the three official entities (DOPW, DOH, and LHW) involved in contraceptive supply chain management.28 The interviews allow for open-ended responses to questions assessing the progress and challenges to the roll-out of the LMIS, including capacity building efforts. Responses are coded and grouped into themes for incorporation into evaluation findings. 26 District-wise reports for stock status are also available commodity-wise. As condoms are by far the largest commodity handled by various district stores, only logistics data on condoms is included in the analysis. 27 The detailed analysis for identification of eight pilot districts from 19 pilot districts is given in annex as Desk Review Report 28 Distribution of KIIs by the three tiers of the government and three official entities is given in Annex 6. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 24 The team held three group discussions with the central warehouse staff (1), LMIS staff of a district store in Lahore under the DOH and LHW (1) and DOPW (1).29 METHODOLOGICAL STRENGTH AND LIMITATIONS The strict definition of stock-out (i.e., 0 stocks of condoms in an entity at three levels, i.e., central warehouse, district and SDPs) during the last six months) is not used in the criteria for selecting ‘more’ and ‘less’ compliant districts. It is replaced by a less stringent definition of district stores with less than one month of stock, as few district stores in the 19 pilot districts were stocked out of condoms. Second, due to time constraints, the instruments were not pre-tested in the field. KIIs developed on the basis of USAID/JSI/DELIVER guidelines for logistical and supply chain assessment were comprehensive. However, the fact that it was a mid-term evaluation and many of the interventions were at the implementation stage, the depth and quality of information gathered did not do justice to the comprehensive format of the KIIs administered in the field and fell short of prior expectations. Specifically, the questions related to improvements between pre- and post￾JSI/DELIVER intervention in the automation process and the forecasting accuracy of commodities. 29 In view of the observation that only one single storekeeper/computer operator is involved in running LMIS system in almost all districts (except in CWH and district stores in Lahore); the GDs were substituted by KIIs. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 25 FINDINGS AND CONCLUSIONS The findings of this mid-term evaluation were obtained amidst ongoing policy, technical, and monitoring interventions by JSI/DELIVER to improve contraceptive availability and data visibility. As the project does not have a baseline or a set of benchmarks against which to measure its achievements, the findings for the five evaluation questions below measure current project achievements against its ultimate objectives,. Thus, precisely measuring annual achievements is difficult and leaves the possibility of underestimating them in a mid-term time frame. These findings should be viewed in the context of a bureaucratic GOP environment and a deteriorating security situation in the country. Implementation of such an important project is further hampered by the complexity of a stagnated and unpredictable transition to a post-devolution scenario. Evaluation Question 1 - To what extent is JSI/DELIVER achieving its objective to institute best practices and automation in the central warehouse? Findings These findings were specifically derived from KIIs with central warehouse management and staff and observation of the central warehouse physical appearance and operations. For this, the findings focus on three aspects of central warehouse rehabilitation and institution of best practices - Infrastructure, Operationalization and Automation of Inventory Management. The central warehouse is expected to have a pallet and racking system that meet international pre-defined specifications, adequate floor space size computed using the potential number of cartons/units of cartons to be stored, proper storage conditions according to international standards, proper and adequate space for inspection of commodities received, stowing commodities, retrieving and shipping commodities; conduction of visual inspection of cartons to detect obvious problems in quality, proper inventory management, adequate implementation of waste management guidelines, well-designed supply (transportation) network, streamlined operations, adequate manpower, and quality monitoring of storage, performance and distribution using innovative technologies, supervision and feedback.30 Infrastructure  The pallet and racking system is in place and meets international standard specifications (10 cm from the floor, 30 cm from walls and 2.5 meters (8 ft.) high).  The storage floor space was increased from 18,000 to 50,000 sq. ft.  Contraceptive commodities are protected from direct sunlight, water and humidity during all seasons.  There is adequate ventilation and temperature regulation.  Cartons are in good condition, well-labeled with international barcodes and arranged upright. The FEFO system is in use.  Computer hardware and barcoding technology are installed with internet functionality and Warehouse Management information System (WMS) is in use.  The fourth layer of racks is not in use as the forklift required to elevate commodities to that level is not available.  Allocation of commodities to racks is not systematic as cartons are placed on any set of racks irrespective of its contents. 30 These expectations for central warehouse infrastructure and operations are obtained from best practices documented in Chapter 8 of the document, “Task Order 1: The Logistics Handbook: A Practical Guide for Supply Chain Management of Health Commodities, USAID/JSI/DELIVER 2011. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 26  The warehouse interface in the central module of the Logistics Management Information System (LMIS) can also be accessed from the central warehouse for supply chain management. Operationalization At the time of the field visit in early December 2012, the central warehouse was in the process of attaining ISO management system 9001 certification. It received its certification on January 15, 2013. A logistics/warehouse management guideline document was developed in 1987 and revised in 2007 prior to the JSI/DELIVER project. There is a full time JSI/DELIVER project staff member in place at the central warehouse who is responsible for providing onsite technical support to central warehouse staff and will eventually be absorbed by the GOP. In conjunction with the GOP, JSI/DELIVER Project is in the process of finalizing a warehouse operations manual. The central warehouse provides 51 job openings, of which 48 are currently filled and three are vacant. Four central warehouse staff have been trained in inventory management at the central warehouse and eleven central warehouse staff are trained on the equipment provided (stacker, computers, barcode printers and scanners). Central warehouse staff is assigned specific responsibilities based on job descriptions; however job descriptions have not been revised since the central warehouse rehabilitation. There is no institutionalized monitoring and evaluation (M&E) system or performance measurement standards for the central warehouse as evidenced by the absence of a Standard Operating Protocol for assessing performance of the central warehouse, specific indicators, and reports of routine performance assessment. JSI/DELIVER, however, conducts quarterly assessments of the central warehouse. There is a contraceptive distribution plan for the current fiscal year (2012-13) developed by JSI/DELIVER with the endorsement of the Planning Commission. The central warehouse management is responsible for the implementation of the distribution plan. JSI/DELIVER facilitated the establishment of contraceptive stock minimum and maximum levels for the central warehouse, the current minimum and maximum contraceptive stock levels in use are 6-9 months. The central warehouse currently contains a 6-9 months’ stock of condoms, Progesterone only pills (POP), combined oral contraceptives (COC), and injectable contraceptives. Copper T-380A (IUCD) has been out of stock in the central warehouse for the last eight months. Central warehouse trucks have been out of order for a year and a half. JSI/DELIVER has alleviated this bottleneck by supporting transportation of commodities from the central warehouse to the district stores in the interim. Automation of Inventory Management and Establishment of Functioning Barcoding System Barcoding technology exists in the central warehouse and is currently in use for only two commodities￾Progesterone Only Pills (POP) and the Intrauterine Contraceptive Devices (IUCDs). Inventory management of other commodities is done using a paper-based system. The barcoding technology is linked to the Warehouse Management System and is used for commodity transaction/product management for the aforementioned two commodities. As a result of barcoding, stock detail reports received and issued from the central warehouse (lot number, manufacturing date, expiration date, quantity of cartons and commodities per carton) for the above two commodities can be viewed in the WMS. The specifications for the warehouse barcoding technology is not compatible with the international barcode system, therefore the barcodes are generated onsite in the warehouse. Three central warehouse staff received training on the WMS; this staff is currently mentored by the onsite JSI/DELIVER staff with a plan to fully transition the management of the system in the near future. The essential central warehouse paper-based tools, Contraceptive Logistics Report (CLR) forms 3, 6, 7 and 8, are not electronically digitized within the WMS and therefore auto-generated reporting of warehouse commodity receipt, inventory management, commodity distribution, issue and receipt voucher and approval is not occurring. The WMS architectural design includes important logistical data elements (beginning balance, quantity issued, quantity received, quantity utilized, losses and adjustments). There is a graphical display of rack functionality in the WMS; however it is not used to allocate cartons to rack positions. As a result of current automation interventions in the central warehouse, the average warehouse processing time has been reduced to two days from lengthy processing times before the JSI/DELIVER project. Conclusions JSI/DELIVER has largely attained its objective of instituting best practices in physical infrastructure, storage operations and enhancing capacity of staff while rehabilitating the central warehouse. The inventory automation and monitoring of two (i.e., POP and IUCD) out of five commodities, is partially automated (except the MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 27 automation of international bar codes) at all levels of warehousing and inventory control. In related areas of automation, barcoding of only two out of five commodities, the non-integration of the WMS with the central warehouse module of LMIS and lack of digitization of paper based CLR forms within the electronic data entry interface of the WMS reduces the overall efficiency of the central warehouse and quality of generated data. Evaluation Question 2 - To what extent is JSI/DELIVER achieving its objectives to improve procurement capacity? Findings Information in project documents provided by JSI/DELIVER as well KIIs with the Planning Commission and DOPW provincial officials revealed that JSI/DELIVER continues to provide technical and financial assistance in strengthening the institutional capacity of the GOP to forecast and procure contraceptive commodities. JSI/DELIVER inputs during the last three and half years include:  Direct procurement of commodities for the GOP – 80 percent of Pakistan’s contraceptive requirements estimated for 2012 were procured using US$20 million contributed by USAID.  Development of a procurement manual for the Planning Commission at the central level; this manual is currently being adapted to include the devolution of forecasting and procurement responsibilities to the provincial governments. The manual includes priority contraceptive commodities and a standardized essential medicines list.  At the central level and in the four provinces/regions included in this study, there are a total of 16 GOP officials that require capacity building assistance to quantify, forecast and prepare procurement plans. KIIs with the federal and provincial officials indicated that only two of the Planning Commission officials out of the 16 officials have been trained in forecasting and procurement. (See Table 3). However, information shared by JSI/DELIVER indicates that seven provincial agency personnel were trained in procurement in October 2011.  KIIs with provincial government officials revealed that JSI/DELIVER is committed to providing technical and financial support in setting up logistic and procurement cells (LPC) at the provincial level. These cells will be responsible for preparing contraceptive procurement tables (CPT) and coordinating and providing oversight to the overall supply chain management system. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 28 TABLE 3: PERSONS TRAINED IN FORECASTING, PROCUREMENT, SUPPLY CHAIN MANAGEMENT (SCM) AND LMIS  Policy interventions at the central level have led to the allocation of Rs. 2.8 billion by the Planning Commission for procurement of contraceptive commodities.  Although Rs. 2.8 billion has been earmarked for procurement for the period 2010–15, the funds have yet to be disbursed for the procurement of contraceptive commodities Conclusions Improving capacity to procure contraceptives is multi-dimensional. Institutional capacity in terms of infrastructure and procurement manuals is still being developed and is in transition at the provincial level. The evidence on human capacity in terms of training at the provincial level is mixed. The evidence from KIIs with provincial officials indicates that the capacity is weak, while documentary evidence from JSI/DELIVER suggest otherwise. Application of training in terms of actual procurement by provincial officials cannot be verified as contraceptive procurement will remain centralized until 2014. Another aspect of training application relates to the timely construction of CPT tables. The findings on this aspect are covered in evaluation question 3 below. Evaluation Question 3 - Have procurement activities been automated, and is the government using the web-based logistics management information system (LMIS), procurement manual and linking it to procurement planning and forecasting? Findings The evaluation teams’ assessment on automation of procurement activities using LMIS is based on KIIs of all provincial officials of three provinces and AJK region. It is also supported by the finding that the LMIS was Forecasting Procurement SCM LMIS Federal Planning Commission 2 2 2 Central Warehouse 1 4 Provincial/Region Punjab 2 6 Sindh 6 4 KPK 3 6 AJK 3 3 Districts in Punjab Lahore 0 5 Muzaffargarh 0 3 Districts in Sindh Karachi (East) 0 1 Hyderabad 0 3 Districts in KPK Peshawar 0 4 Abbottabad 0 6 Districts in AJK Muzaffarabad 0 6 Rawalakot 0 5 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 29 operational in nine out of 20 district level government agencies (Table 4) and none of the provincial Logistics and Procurement cells were operational. The Planning Commission is preparing procurement plans in conjunction with JSI/DELIVER using integrated CLR 6 utilization data. It is expected that in the near future, data from the electronic LMIS at the provincial and central levels will be the source of commodity utilization rates for preparing contraceptive procurement tables (CPTs). In December 2012, the provincial government agency officials, with the support of JSI/DELIVER, conducted contraceptive quantification and prepared contraceptive procurement tables themselves using data from the integrated CLR 6. The electronic LMIS is still in its incipient stage of implementation, and therefore utilization data required for procurement planning is not completely entered into nor retrieved from the LMIS in non-pilot districts and provinces. The electronic LMIS in its current state is not designed to allow for automated procurement planning. JSI/DELIVER provided the following information on the above question: The electronic LMIS is not the only direct source of consumption data used by the GOP and JSI/DELIVER for the preparation of CPTs; however, utilization data for provincial DOPWs entered at the district level and stock data for central level are factored into the procurement planning process. The Planning Commission, in collaboration with JSI/DELIVER, utilizes the LMIS in the development of CPTs. The Planning Commission is preparing procurement plans in conjunction with JSI/DELIVER using integrated Contraceptives Logistics Report (CLR 6) utilization data. Electronic LMIS at the provincial and central levels is providing reliable and up-to-date data on commodity utilization rates for preparing contraceptive procurement tables. The recent CPT 2013, provincial and regional forecasts were prepared based on LMIS consumption trends and stock status. In December 2012, the provincial government officials of DOH, DOPW and LHW, with the support of JSI/DELIVER, conducted contraceptive quantification and prepared contraceptive procurement tables themselves using data from web-based LMIS. The electronic LMIS is still in its incipient stage of implementation for Department of Health and Lady Health Workers Program, and therefore utilization data required for procurement planning for these two departments is not completely entered into nor retrieved from the LMIS. However, observing the trends in increased reporting rates for DOH/LHW, it is likely that LMIS consumption data will be used for procurement planning in the near future. Conclusions Given the mid-term nature of this evaluation and the fact that the electronic LMIS is still being rolled out, automation of procurement activities would, to a great extent, require that all the districts are using the electronic LMIS and are the direct sources of utilization data required for generating the CPTs. Linking i) LMIS data manually with procurement planning and ii) the web-based LMIS system with automated procurement planning and forecasting activities are two different processes. While JSI/DELIVER’s version of findings relate to the manual use of LMIS for procurement planning that may be factually correct; the evaluation team assessment that the LMIS as a tool for automated procurement planning (i.e., the electronic LMIS in its current state) is not designed to allow for automated procurement planning is also correct. Evaluation Question 4 - To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the central, provincial and eight sample pilot districts to ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? Findings These findings arose from KIIs with provincial and district government officials in eight out of the 19 pilot districts. A structured checklist was also used to assess the electronic LMIS. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 30 The web-enabled LMIS is operational in three out of four provincial/regional procurement and logistic cells visited by the evaluation team.31 As of December 2012, all district government officials had been trained in the LMIS and JSI/DELIVER had made computer hardware available to all the pilot district government agencies visited. JSI/DELIVER has made a commitment to provide hardware to the established provincial logistic and procurement cells when the dedicated spaces have been set up. An LMIS manual has been developed by JSI/DELIVER for the GOP to aid in effective use of the LMIS; however it is only being used for data management and reporting in 11 out of 22 government agencies visited. A total of 58 persons have been trained in LMIS in the eight pilot districts and in the provincial and federal agencies (see Table 2). Seventy percent of the 20 pilot district government agencies visited; (with the Department of Population Welfare (DOPW), the Department of Health (DOH) and the Lady Health Worker (LHW) program included), have had the LMIS installed. Forty-five percent of 20 pilot district government agencies are using the LMIS for logistics planning and reporting (see Table 3 for a breakdown of the roll-out of the electronic LMIS). Eight out of 20 pilot districts agencies visited are not connected on-line. Three pilot district agencies use a local LMIS module disk, which is hand carried to the agency’s provincial office to upload the data to LMIS. With the support of JSI/DELIVER, the GOP has revised the Combined Report and Requisition form (named by the government as CLR 6) to include logistics data elements for public and private entities. All the District Population Welfare Officers (DPWOs) interviewed were not aware of any logistics information system in use by private sector entities (Green Star, Marie Stopes and the PPHI). KIIs with all district government officials revealed the following as key themes influencing the roll-out and use of the LMIS for supply chain management in the districts: no internet connection(8 district agencies), frequent power outages (20 district agencies), turnover of trained staff (9 district agencies), need for refresher training (6 district agencies) and computer illiteracy (5 district agencies). Figure 7 further illustrates other important factors influencing the implementation of the LMIS. As part of the implementation of the LMIS in Pakistan, the web-enabled software is hosted on a server within the country and can be accessed at http://lmis.pc.gov.pk/. Conclusions  Provision of training on LMIS to all district officials and hardware by JSI/DELIVER has been implemented. While contraceptive supply chain management using web-based LMIS technology is weak at the district level, it is much stronger at the provincial level. The DOPW are performing better at installing and utilizing the software application than the LHW and DOH.  The continuous and smooth functioning of the LMIS is still limited by a number of problems such as high staff turnover, frequent power outages, need for refresher training and inadequate follow on support and assistance from JSI/DELIVER. 31 For LMIS to be operational in a provincial agency, all that is required is the presence of a desktop computer or laptop (not necessarily dedicated) and an on-line connection. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 31 FIGURE 7: CAUSE ANALYSIS DIAGRAM OF FACTORS AFFECTING THE RELATIVE PERFORMANCE OF LMIS ACROSS PILOT DISTRICTS Human Resources Operationalization Inadequate onsite technical follow on support Poor motivation of staff as salaries are not paid on time Absence and lack use of LMIS guideline/SOPs and job aids at the district level High turnover of trained staff Computer Illiteracy INADEQUATE PERFORMANCE OF LMIS ACROSS No internet DISTRICTS Lack of coordination connection between district Government agencies in integrating reports Incessant power outages Computer bug prone local module of LMIS (unstable local module of LMIS) Policy/Coordination Software/Infrastructure MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 32 TABLE 4: LMIS IMPLEMENTATION BY DISTRICT GOVERNMENT AGENCIES Hardware Software Installed Operational DOPW DOH LHW DOPW DOH LHW DOPW DOH LHW Districts in Punjab Lahore Yes Yes Yes Yes No No ON. N.O. N.O. Muzaffargarh Yes Yes Yes Yes Yes Yes L ON. ON. Districts in Sindh Karachi (East) Yes N/A N/A No N/A N/A N.O. N/A N/A Hyderabad Yes N/A N/A No N/A N/A N.O. N/A N/A Districts in KPK Peshawar Yes Yes Yes Yes Yes Yes ON. ON. ON. Abbottabad Yes Yes Yes Yes Yes Yes ON. ON. ON. Districts in AJK Muzaffarabad Yes Yes Yes Yes Yes Yes L N.O. N.O. Rawalakot Yes Yes Yes Yes No No L N.O. N.O. KEY: N/A Not Applicable N.O. Not Online ON. Online L Local Module Evaluation Question 5 - How widespread and frequent are stock-outs at Central Warehouse and provincial/district-level warehouses and what is their cause and when they occur? Findings The data in this section were extracted from the web-based Logistics Management Information System (http://lmis.pc.gov.pk/). Only the secondary data of seven out of eight pilot districts visited (including service delivery points (SDPs) are presented as findings in this report. Due to the unavailability of data for Rawalakot district in the LMIS, it was excluded from the calculations and the aggregated data of seven districts were used in calculating the percentages below. The percentage of districts with less than one month of stock is used as a proxy for a stock-out in the analysis below. Months of stock is calculated by dividing the total quantity of stock-on-hand by the average quantities of monthly consumption. Any districts with less than one month of stock are included in the percentages below.32 At the end of November 2012, there was no stock-out in any district, down to the level of the SDPs. The percentage of pilot districts (including service delivery points) with less than one month of stock during January 2010–November 2012 per commodity is given in Figure 8. For 2012, the percentage of pilot districts with 32 As an example if two districts out of seven sampled pilot districts in 2010 (over the 12 month period) had less than one month of stock (down to the level of SDP), the calculated percentage is 2/7x100=29%. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 33 less than one month of stock for five commodities is: condoms: 42 percent; IUCD: 57 percent; COC: 29 percent; POP: 0 percent; three-month injectable contraceptive: 0 percent. Data analysis for the pilot districts illustrates that stock-outs (as defined above) of each of the five above mentioned commodities have occurred less frequently overtime. All of the pilot districts experienced less than one month of stock in 2010, while in 2012, only 14 percent of pilot districts experienced less than one month of stock. None of the pilot districts experienced less than one month of stock of POP and three-month injectable contraceptives in 2012, as compared to 2010, when 57 percent of districts had less than one month of stock of POP and 14 percent of districts had less than three month’s stock of injectable contraceptives. There is a considerable level of stock-out of IUCDs in the central warehouse and all pilot districts. FIGURE 8: PERCENTAGE OF PILOT DISTRICTS (INCLUDING SERVICE DELIVERY POINTS) WITH LESS THAN ONE MONTH OF STOCK PER COMMODITY FROM 2010– 2012 (NO. OF DISTRICTS= 7).33 33 Percentages for 2012 include only January–November data. 100 29 14 57 14 28 42 14 85 42 43 57 29 0 10 20 30 40 50 60 70 80 90 100 Condoms IUCD COC POP 3month Injectionables 2010 2011 2012 3 month Injectable Contraceptives MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 34 TABLE 5: PERCENTAGE OF DISTRICT GOVERNMENT STAKEHOLDERS WITH LESS THAN ONE MONTH OF STOCK BETWEEN JANUARY–NOVEMBER 2012 PER COMMODITY Commodities DPWO DOH LHW Condoms 28.6% 100% 80% IUCDs 57.1% 100% COC 14.28% 100% 80% POP 14.28% 80% Injectable Contraceptives (3 months) 14.28% 100% 80% Factors affecting stock-outs at central warehouse and district stores:  KIIs with the Central Warehouse Director and five staff revealed that delays in shipment to central warehouse resulted in a stock-out of IUCDs in some districts and the central warehouse.  KIIS with all the LHW and DOH officials in the pilot districts highlighted issues with the transportation of contraceptives from the central warehouse to their respective district stores.  Use of inconsistent formula for calculating quantities for contraceptive requisitions. Average monthly consumption, stock at hand, lead time, and minimum and maximum levels were not consistently factored into computations.  Delay in consolidation of the integrated CLR 6 due to late submission by the DOH and LHW government stakeholders was identified by key informants as a factor influencing stock-outs in all the pilot districts.  KIIs with all the district government officials revealed that inadequate storage conditions and a lack of space in district stores resulted in the shipment of commodities to the service delivery points whether requests were made/received or not. This was the main reason why many districts stores visited had low quantities of stock. Conclusions The stock situation has progressively improved since the onset of the JSI/DELIVER project; however the maintenance of minimum and maximum levels of stock is not occurring at the eight pilot districts visited. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 35 MAIN CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS JSI/DELIVER has largely attained its objective of instituting best practices while rehabilitating the central warehouse. The inventory monitoring of two out of five commodities is partially automated (except the automation of international bar codes) at all levels of warehousing and inventory control. However non-integration of the WMS with the central warehouse module of LMIS and lack of digitization of paper based CLR forms within the electronic data entry interface of the WMS reduces the overall efficiency of the central warehouse and quality of generated data. The evidence on human capacity in terms of training at the provincial level is mixed. The evidence from KIIs of provincial officials indicates that the capacity is weak, while documentary evidence from JSI/DELIVER suggest otherwise. Application of training in terms of actual procurement by the provincial officials cannot be verified as contraceptive procurement will remain centralized until 2014. The electronic LMIS in its current state is not designed to allow for automated procurement planning Provision of hardware and training on LMIS to all district officials by JSI/DELIVER has been implemented. However while the management of contraceptive supply chain using web-based LMIS technology is weak at the district level, it is much stronger at the provincial level. The continuous and smooth functioning of the LMIS at the district level is still limited by a number of problems such as high staff turnover, power outages, need for refresher training and inadequate follow on support and assistance from JSI/DELIVER. The stock situation has progressively improved since the onset of the JSI/DELIVER project. However, since adherence to minimum and maximum levels of stock at the district level is weak, it has created a situation of uneven availability of commodities at different levels within the supply chain system. RECOMMENDATIONS The recommendations of the evaluation study address only the following two areas, i.e., automation and capacity building: Automation  The integration of the WMS with the central warehouse module of the LMIS will reduce duplicative efforts in data entry and improve the overall efficiency of the central warehouse and its staff.  The barcoding technology should be extended to include the remaining contraceptives supplied to the central warehouse. The warehouse management system should be redesigned to be interoperable with the online LMIS. Data entry interfaces should be made identical to the Planning Commission’s paper-based system for ease of data transcription and to preclude data entry errors.  Though the flexibility of commodity allocation to racks introduces more effective utilization of space at a given point in time, the inconsistent allotment of commodities to racks poses a challenge of ineffective and inefficient inventory control and product handling. Therefore if possible commodities should be assigned to dedicated spaces in the racks for efficient inventory control and product handling. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 36  It may be useful in the future to develop a module within the LMIS for procurement planning that could serve as an interface to automatically compute contraceptive procurement tables.  While JSI/DELIVER has significantly revamped the logistics data collection and reporting system by integrating the CLR 6 and rolling out the electronic LMIS, it is necessary to incorporate plans to provide feedback to government agencies/staff generating data at the source. The current logistics data flow system is unidirectional with a lot of focus on reporting and little focus on feedback on supply chain performance. Capacity Building  Following the devolution of the procurement responsibility from the MOPW to the provincial DOPW, it is necessary for JSI/DELIVER to intensify its procurement capacity building efforts at the provincial level. KIIs with provincial government officials in each of the three government agencies in the pilot districts revealed limited knowledge of the procurement planning process and involvement in procurement planning. It is therefore paramount to involve provincial officials in the preparation of contraceptive procurement tables and other components of the procurement process.  It is important to intensify capacity building efforts for DOH and LHW provincial and district government agencies so that they may work more closely with the DOPW to better coordinate the supply chain system and streamline logistics reports. This would increase the availability of contraceptives and make data more visible in all three district stakeholder agencies.  The launching of formal logistics training at the Health Services Academy is a milestone achievement and pertinent long-term strategy. It is vital that this be followed by more advocacy efforts with the GOP to create substantive provincial government procurement and logistics positions within the three government agencies so that employees of these agencies can avail themselves of this formal training opportunity.  Despite the establishment of minimum and maximum levels of contraceptive commodities at the three levels of the supply chain system, it is apparent from field visits that these limits are not adhered to at the district level. JSI/DELIVER needs to intensify onsite technical assistance to ensure that district government agencies calculate the correct amounts for requisition by factoring in the appropriate indicators: minimum and maximum levels of stock, lead time and stock on hand.  Refresher trainings, regular onsite mentoring, and institutionalizing supportive supervision within the GOP are required to ensure utilization of the LMIS at the source of data collection and supply chain management (most critically in the AJK region). Data will be more centrally visible, but in order to facilitate ownership of the supply chain system by the GOP at all levels, promotion of data management, use and feedback at the district and service delivery points is paramount. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 37 ANNEXES ANNEX 1: EVALUATION STATEMENT OF WORK Background Information - Identifying Information about the Project The JSI/DELIVER Project started its operation in Pakistan in August 2009 with USAID support to provide inputs on Logistics Management Information System (LMIS), Warehouse Rehabilitation, Commodity Security, Procurement Capacity and other support activities related to contraceptive commodities. The project will spend US$10-15 million as technical assistance (TA) for capacity building of federal and provincial governments and US$89 million on procurement of contraceptives and development and operationalization of LMIS. Development Context - Problem or Opportunity Addressed Pakistan has high unmet need for family planning. One reason for this is the unavailability of contraceptive commodities due to stock-outs. The Government of Pakistan does not allocate sufficient resources to the procurement of contraceptives, and lacks the capacity to forecast, procure and distribute the commodities nationwide. Target Areas and Groups The JSI/DELIVER Project works nationally, in all districts of Pakistan, and targets those involved in the contraceptive commodity supply chain. Intended Results If improvement in the supply chain management through central warehouse rehabilitation, establishing LMIS and building governmental (provincial, local) technical capacities is undertaken then it will lead to more effective delivery of contraceptives and sustainability of the delivery system. Approach and Implementation The purpose of JSI/DELIVER is to: (a) improve central warehouse and supply (CH&S) chain operations; (b) improve procurement capacity of the Government of Pakistan; (c) improve supply of contraceptives (through procurement); and (d) establish the LMIS. The project’s interventions focus on two key dimensions:  “Improving the performance of the health system” by redressing weak supply management systems; and  Improve the availability of contraceptives in the public sector by working in partnership with the Ministry of Population Welfare (MOPW) and Ministry of Health (MOH) to achieve contraceptive security through a series of interventions in the above mentioned key focus areas. Existing Data  USAID/JSI/DELIVER Project, Statement of Work  Provincial and District Supply Chain Management Situation Assessment in Pakistan (Final Report)  Quality Assurance Surveillance Plan (QASP) & Performance Monitoring Plan (PMP)  USAID/JSI/DELIVER Project, Task Order 4  Country Quarterly Reports - Routine Contractual Information Forms 1 and 2 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 38 RATIONALE FOR EVALUATION Purpose of Evaluation The purpose of this evaluation is to:  Determine how far JSI/DELIVER has come in achieving the work set out in its work plan and scope of work; and  Identify problems that require correction in order to increase the effectiveness of the program. Audience and Intended Use The results of this evaluation will be shared with USAID and implementing partners. USAID will develop a dissemination plan in accordance with the Evaluation Policy as specified in ADS Chapter 203. Evaluation Questions The evaluation will focus on the following questions: 1. To what extent is JSI/DELIVER achieving its objective to institute best practices (human resource management, infrastructure, operationalization and automation) in the central warehouse? Explanation: a) has the warehouse been rehabilitated; b) is the pallet and racking system in place and operational; c) is the warehouse automated d) has the staff been trained (is there a warehousing checklist); e) have staff job descriptions been revised, are there performance measurement standards for staff, etc.); f) have periodic evaluations of the warehouse been conducted, and results been used for continual quality improvement; g) to what extent is the government able to run the warehouse independently or is it dependent on JSI/DELIVER? 2. To what extent is JSI/DELIVER achieving its objectives to improve procurement capacity? Explanation: Did JSI/DELIVER provide adequate technical assistance to build the capacity of the public sector (central and provincial levels) to quantify/forecast contraceptive needs, ensure allocation and disbursement of adequate resources for procurement, and procure contraceptive commodities by the government departments? What measures did JSI/DELIVER use to build the government’s capacity to conduct these tasks? 3. Have procurement activities been automated, and is the government using the web-based logistics management information system (LMIS), procurement manual and linking it to procurement planning and forecasting? 4. To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the central, provincial and eight sample pilot districts to ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? Explanation: This question will explore the effectiveness of JSI/DELIVER’s capacity building efforts to establish a LMIS (including LMIS training, LMIS equipment procurement and operationalization, LMIS infrastructure, and follow up to support its implementation); and the extent to which private (e.g., Green Star, Marie Stopes, People’s Primary Healthcare Initiative (PPHI) and public stakeholders are compliant with LMIS reporting requirements. Are new technologies, including bar-coding, an automated inventory control system at the central warehouse and a web-based LMIS operational? To what extent has the government assumed the responsibility of managing the supply chain versus being dependent upon JSI/DELIVER for this? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 39 5. How widespread and frequent are stock-outs at central warehouse and district-level stores and what is their cause and when do they occur? Explanation: Stock-outs are an indicator of contraceptive availability and of supply chain capacity. The evaluation will use LMIS data when available to investigate trends in stock-outs and present an overview (with visuals) of these trends. If LMIS data are not adequate or available, the evaluation will rely on site visits or other secondary data to address this question. EVALUATION DESIGN AND METHODOLOGY Evaluation Design This mid-term evaluation of USAID’s JSI/DELIVER Project will provide information on how far JSI/DELIVER has come in achieving the work defined in the program’s work plan and scope of work. The key sources of data for this evaluation will include secondary information from JSI/DELIVER management and primary evidence from visits to the Planning Commission, Provincial and District Population Welfare Department (PWD), Central warehouse () and district stores of Department of Health (DOH), Population Welfare Department (PWD) and Lady Health Worker’s offices. This will help to determine how successful JSI/DELIVER has been in improving supply chain management through central warehouse rehabilitation, establishing the LMIS and building governmental (central, provincial and district) technical capacities. The Monitoring and Evaluation Program proposes a mixed method approach to the evaluation that complies with the USAID evaluation policy, specifically that for performance evaluations. The evaluation will rely heavily on quantitative and qualitative evidence to answer the evaluation questions and shortlist districts from which to collect primary information. The information will be analyzed at i) two levels of the supply chain system (the central warehouse and district stores), ii) public sector stakeholder entities (i.e., DOH, DOPW and LHWs) at the central, provincial and district levels as applicable, and iii) private sector stakeholder entities (i.e. Green Star, Marie Stopes, People’s Primary Healthcare Initiative (PPHI)). Using three indicators: 1) percentage of non-reporting months, 2) average quarterly supply of condoms, pills, Injectable Contraceptives and IUDs, and 3) stock out rate for the period July 2011-July 2012; LMIS analysis of the 19 pilot districts will be used to rank ‘more’ and ‘less’ compliant districts. Among the 19 pilot districts, a convenient sample of eight will be selected (two in each province, one ‘more’ and one ‘less’ compliant district) for site visits and primary data collection. Structured logistics system assessment checklists and KII semi-structured questionnaires will be administered at the central level, provincial administrative levels, central warehouse and eight selected district stores. To facilitate the evaluation within the time-frame required, the eight sample districts will be drawn from the four provinces of the country. Data Collection Methods The evaluation will rely on the following types of secondary information collected and documented by JSI/DELIVER management and the provincial-level DOH and DOPW: 1. LMIS-based quarterly reports generated by the central and district stores on all aspects of contraceptive procurement and supply chain management in the 19 pilot districts since January 2011-September 2012; 2. Project documents including cooperative agreements, PMPs, and work plans for the years 2009-10, 2010- 11 and 2011-12; 3. Provincial DOH and DOPW budget documents; 4. Contraceptive procurement manuals and staff training manuals; 5. Copies of MOUs with MOH and MOPW; and 6. Documented best practices in contraceptive procurement and supply chain management. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 40 The evaluation will also use the following methods to collect primary data and information: 1. A Supply Chain Management Assessment Instrument (SCMAI) will be developed to assess: i) the presence and use of physical infrastructure and equipment, ii) computerization and operationalization of LMIS, iii) automation of procurement and supply chain processes including inventory control, iv) transportation and v) training of staff. This instrument will be used to collect information during site visits to the central warehouse and sampled district stores. 2. A LMIS assessment instrument will be developed to identify the differences between the way a program’s LMIS should work and how it actually works. This instrument will be used to collect information during site visits to the central warehouse and sampled district stores. 3. Key informant interviews (KIIs) with senior officials and staff of the central warehouse and district warehouses. 4. KIIs with provincial DOH, DOPW and LHWs chain officials. Data Analysis Methods Indicator data from LMIS reports of eight sampled districts and central warehouse will be used to analyze monthly and/or quarterly trends in stock outs (Qs.5). Indicator data from the SCMAI, including observations will be subject to simple descriptive/frequency analysis for highlighting and quantifying various aspects of supply chain processes, including automation, bar-coding and automated inventory control. The analysis will assess the effectiveness of supply chain management in central warehouse and district stores (Qs. 1, 2 and 4). Indicator data from the LMIS assessment instrument will be used to assess the working of the LMIS system in the eight sampled districts. The qualitative analysis will assess how well the LMIS system is working in terms of design, operation and use (Qs. 3 and 4). To supplement findings from SCMAI and LMIS assessment instruments, qualitative assessments on users’ and administrators’ perceptions on the strengths and weaknesses of the procurement and supply chain system and LMIS will be conducted using semi-structured interviews with key informants; these will be digitally recorded where possible. The interviews will allow for open-ended responses to questions assessing the progress and challenges to rollout of LMIS, including capacity-building efforts. Responses will be coded and grouped into themes for incorporation into evaluation findings. Methodological Strength and Limitations Identification of ‘more’ or ‘less’ compliant districts as outlined above in the evaluation design section will provide a comparative analysis of the working of the contraceptive supply chain process and identify the weakness and strengths useful to informing the recommendations for the way forward. Identification of ‘more’ and ‘less’ compliant districts depends on the richness of reporting data in the LMIS quarterly reports submitted by the central warehouse and district stores. Gaps in the data as well lack of relevant indicators could lead to a more ‘subjective’ identification and ranking of ‘more’ and ‘less’ compliant districts. EVALUATION PRODUCTS Deliverables The Monitoring and Evaluation Program will submit three main evaluation products: 1. Desk Review Report: The report will be based on quarterly/annual LMIS reports of all 19 pilot districts and will be the basis for selecting performance indicators to distinguish between ‘more’ and ‘less’ compliant districts. Based on the desk review report a total of eight ‘more’ and or ‘less’ compliant districts (four of each) will be selected for site visits, observation and data collection. Due date: December 7, 2012. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 41 2. Debriefing Presentation: At the conclusion of the fieldwork, the evaluation team will prepare and deliver a debriefing presentation on the evaluation findings, conclusions and recommendations to USAID and implementing partners and/or USAID approved other parties. Due date: December 17, 2012. 3. Draft and Final Report: Within one week of the presentation USAID will provide both written and verbal comments. The evaluation team will incorporate these comments into a draft report. After a thorough technical review, the Monitoring and Evaluation Program will deliver the draft report to USAID and implementing partners, if appropriate, for review and comment. Once the Monitoring and Evaluation Program receives comments on the draft report, the evaluation team will incorporate them into the final report, which will then be sent for editing and branding. The Monitoring and Evaluation Program will then deliver the final report to USAID. The final report will be presented within seven weeks of the project start date. Draft report due date: January 8, 2013. Reporting Guidelines The final report will be delivered by the Monitoring and Evaluation Program to USAID in printed and electronic forms. The report will follow standard guidelines laid out in USAID’s Evaluation Policy.34 The report will follow standard guidelines as laid out in Appendix 1 of USAID’s Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced here in Annex 1.35 The Evaluation Design and Baseline Study Report will follow the structure given below:  Title page;  Table of contents;  Table of tables and figures;  Acknowledgements or preface (optional);  Executive summary which will be 3-5 pages in length that summarizes key points (project purpose and background, key evaluation questions, methods, findings, etc.);  Introductory chapter;  The Development Problem and USAID’s Response: This section will describe the development problem USAID wants to address. This will include USAID’s response to the problem, the development hypothesis and project implementation;  Purpose of the mid-term evaluation: This section will include the purpose of the study and state all questions;  Evaluation Design and Methodology: A written design which includes key questions, methods, main features of data collection instruments, and data analysis plan;  Findings and Conclusions: This section will include findings and conclusions for each question;  Main Conclusions and Recommendations;  References; and  Annexes:  The Scope of Work;  All tools such as KII questionnaires for federal and provincial level government official, central warehouse management and staff, district stores staff, SCMAI and LMIS checklists; and  All sources of information properly identified and listed. 34 www.usaid.gov/policy/ads/200/220mab.pdf Accessed; 9 April 2012. 35 www.usaid.gov/policy/ads/200/220mab.pdf. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 42 TEAM COMPOSITION The evaluation team will consist of:  An (short-term) expatriate team leader with experience in health delivery systems, specifically in developing countries; he/she will guide all tasks listed under Section VI and will also be responsible for guiding the evaluation team members listed below. The team leader will be responsible for all deliverables and most importantly, will author and present the draft and final reports;  A (full-time M&E Project staff) senior evaluator with experience in evaluation and social research. He/ She will be a part of team for field visits, analyze data and with the consent of the team leader contribute to writing parts of the report;  A (full-time MEP staff) relationship manager (health) with experience on the Family Advancement for Life and Health (FALAH) project; and  A (short-term) national sector specialist with experience in the delivery of medical goods/contraceptives. EVALUATION MANAGEMENT Logistics This section elaborates on the key tasks required for implementation of a Monitoring and Evaluation Program mid￾term evaluation: 1. Develop a detailed SOW. 2. Recruit an expatriate evaluation team leader and a national subject specialist. Once the evaluation team requirements are determined, the Evaluation Unit will begin to recruit evaluation team members. 3. Collection of project documents including district LMIS reports for writing the Desk Review Report (DRR). 4. Team Planning Meeting (TPM) – Once the team leader and national subject specialist are on board, the Monitoring and Evaluation Program will conduct a TPM to orient the team on the findings of the DRR, evaluation requirements and plan the evaluation fieldwork. During the TPM, the evaluation team will review the evaluation questions, data sources, data collection methods and develop an assessment plan based on the Getting to Answers approach. 5. Field Work – The four member evaluation team will be split into 2 teams with 2 members each and visit the four provincial capital and eight districts to conduct the assessment of the central warehouse and district stores, and KIIs with management and staff of the central warehouse, district stores and government officials. Security issues may prevent the team from visiting the sampled districts during the scheduled time. In such a case, alternative districts may have to be selected or revisions in the study time frame may be required. 6. Debriefing – following the fieldwork and data analysis, MEP staff and the evaluation team will debrief USAID on the evaluation findings. 7. After integrating USAID’s initial feedback from the debriefing, the evaluation team will prepare a draft evaluation report. 8. After incorporating USAID’s and IP comments on the draft report the M&E project will submit the final evaluation report. Scheduling The evaluation work will start after the hiring of expatriate team leader. The hiring process is expected to take 6-8 weeks from August 1, 2012. The complete process, including team planning meeting, desk review report, developing interview tools, field trips to provincial capitals, central warehouse and eight districts and draft report writing will take about six weeks. Thus, the draft report would be available during the week of January 18, 2013. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 43 Activity Nov Dec Jan w 1 w 2 w 3 w 4 w 1 w 2 w 3 w 4 w 1 w 2 w 3 1 Review of Project Background Documents 2 Desk Review Report 3 a. Team Planning Meeting, incl. USAID and IP b. Develop Interview Tools c. Meeting with Relevant Ministries 4 Field Visits to Provincial capitals and 8 Districts 5 a. Preliminary Data Analysis and Debriefing to USAID b. Complete Draft (zero draft) 6 Finalize the report, incorporate feedback from USAID and MSI Team MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 44 BUDGETING The following table highlights Level of Effort (LOE) of each evaluation team member:36 TABLE 6: LEVEL OF EFFORT OF EVALUATION TEAM MEMBERS Tasks Level of effort (days) Team Leader(USN) Team members (CCNs) Senior Evaluation Specialist Sector Specialist RM Health Stage 1:  Review relevant background documents and prepare Desk Review Report 37 6 (Home Base) 20.5 1 4 Stage I1: (Islamabad)  Team Planning Meeting, incl. USAID and IP  Develop interview tools  Meeting with Government of Pakistan (GOP) officials and key donor agencies. 4 4 2 4 4 2 4 4 2 4 4 2 Stage III:  Field visits to provincial capitals and 8 districts. 10 10 10 10 Stage IV:  Complete draft (zero draft)  Presentation to USAID 9 3 9 6 1 Stage V:  Finalize the report, incorporate feedback from USAID and the MSI/MEP team 5 (Home Base) 3 1 1 Total Level of Effort 40 46.5 31 32 36 LOE increase justification TL: 1. Increased number of days (4) required for the preparation for field work, team planning meetings and meeting with GOP officials. 2. One extra day required for field work. 3. LOE for finalizing report was a couple of days less than anticipated. LOE increase justification STTA (sector specialist):1. LOE for desk review not accounted for initially- 1 day, 2. Increased number of days (4) required for the preparation for field work, team planning meetings and meeting with GOP officials. 3. One extra day required for field work. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 45 SOW Annex I-A: Getting to Answers Mid-Term Evaluation of JSI/DELIVER Data Collection Evaluation Question Type of Answer/Evidence Methods Source Sampling/ Selection Data Analysis Methods 1. To what extent is JSI/DELIVER achieving its objective to institute best practices and automation in the central warehouse? Descriptive, comparison of ‘best’ with ‘actual’ practices; Observations; document review; SCMAI questionnaire Project and ‘best’ practice documents; recorded observations in the SCMAI Visit to central warehouse Content analysis of SCMAI questionnaire; 2. To what extent is JSI/DELIVER achieving its objectives to improve procurement capacity? Descriptive, comparison of the ‘on-ground’ situation with the objectives to improve procurement capacity; KIIS; SCMAI Questionnaire; Contraceptive Procurement Manual; and project documents review Central warehouse, district warehouses; Government Visits to central warehouse and purposive sampling of district stores Frequency Tables of data from SCMAI 3. Have procurement activities been automated and is the government using the web￾based Logistics Management Information System (LMIS) and linking it to procurement planning and forecasting? Descriptive; LMIS reports generated by central warehouse/district stores LMIS assessment: LMIS reports generated by district stores and central warehouse Central warehouse and eight district stores Visits to central warehouse and district stores Frequency Tables of indicators from LMIS assessment MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 46 Data Collection Evaluation Question Type of Answer/Evidence Methods Source Sampling/ Selection Data Analysis Methods 4. To what extent has JSI/DELIVER been effective in building the capacity of federal and provincial governments to manage the contraceptive supply chain using modern technology (LMIS) in the 8 sampled pilot districts and ensure a continuous supply of contraceptive commodities? What factors affect the relative performance of the LMIS across districts? Descriptive; Desk Review Report will assess the relative performance of public stakeholders in meeting the LMIS reporting requirements SCMIA and LMIS assessment questionnaire; training manuals; LMIS quarterly reports; KIIs Central warehouse and district stores; DOH, DOPW and LHW Office Purposive sampling of eight districts Frequency Tables of indicators from LMIS Assessment 5. How widespread and frequent are stock-outs at central warehouse and district ware houses and what is their cause and when they occur? Descriptive and Statistical LMIS based central warehouse and district stores data; KIIs with central warehouse/ district store officials Central warehouse and district stores; Central warehouse and purposive sample of eight districts Time Trends in stock outs by districts and central warehouse; District￾wise listing of reasons and causes for stock outs. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 47 SOW Annex I-B: Reporting Guidelines 1. The evaluation report must represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not work, and why. 2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer. 3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report. 4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action. Note: These guidelines are taken from ADS 203.3.2.8 (Documenting Evaluations) - http://transition.usaid.gov/policy/ads/200/203.pdf - which is based on Appendix 1 of USAID Evaluation Policy: Criteria to Ensure the Quality of the Evaluation Report. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 48 ANNEX 2: DESK REVIEW REPORT FOR IDENTIFYING MORE AND LESS COMPLIANT DISTRICTS Introduction The methodology in the Statement of Work (SOW) for the Mid-term Evaluation of JSI/DELIVER requires that the evaluation team conduct a field survey of eight out of the 19 pilot districts in order to evaluate the project. The sample, which is to be selected from the four provinces, is comprised of two districts in each province, one ‘more’ and one ‘less’ compliant. The indicators in the reports on the 19 districts generated by the Logistics Management Information System (LMIS) are used to identify the four ‘more’ and four ‘less’ compliant districts across Pakistan. The initial meeting with the JSI/DELIVER team revealed that the stores in the districts of Balochistan province are yet to be included in the automation project, while some of the districts in Azad Jammu and Kashmir (AJK), the Federally Administered Tribal Areas (FATA) and Gilgit Baltistan (GB) have been included in the first phase of automation. Consequently, the two districts from the province of Balochistan will be replaced by two districts from AJK, FATA and GB. District Selection Methodology An efficient supply chain management and procurement process is dependent on timely reports from various nodes in the supply chain. These include Service delivery Points (SDPs) and district stores in Pakistan. Monthly reports on stock status of contraceptives are manually prepared by SDPs and sent to districts stores. The data are then entered into the automated LMIS system in the pilot districts and transmitted electronically to the central warehouse. In most of the districts, stores are maintained by three different government entities, which are: the Department of Health (DOH), the Population Welfare Department (PWD) and the Lady Health Worker Program (LHW). The first phase of automation was launched in July 2011 in 19 districts. The initial plan was to automate all of the three above entities in each of the 19 pilot districts. However, the operationalization of LMIS has been inconsistent since then, and has yet to be operationalized in some districts. The timelines for operationalization of the LMIS system per district and government entity is included in Annex 2. The indicators used for selecting districts for the mid-term evaluation are outlined below: 1. Percentage of non-reporting months: This indicator highlights the efficiency of the supply chain system. It is the number of stores in a district that do not regularly send reports to central warehouse. The LMIS in Pakistan is used to prepare monthly district-level reports on non-reporting districts. These reports are used to estimate the number of non-reporting months for each of the aforementioned three entities by summing the non-reporting months for each entity for the period July 2011-July 2012. The district aggregate percentage of non-reporting months is computed by the dividing the total number of non-reporting months for all entities by the total number of expected reporting months (39 months) for the period July 2011-July 2012. This percentage is also disaggregated by individual entities using the 13 month period per entity as the denominator. For the purpose of triangulation and sensitivity analysis, two other indicators are calculated from district LMIS reports, namely: 1. Variation in months of stock (MOS) of condoms38: Variation in MOS is a performance indicator highlighting the efficiency of inventory control within the district store. Low levels of stock quantity (in terms of MOS) increase the likelihood of stock outs, and unmet demand over the long term. Conversely, high levels of stock quantity (in terms of MOS) increase the likelihood of product expiration and waste. A small variation in MOS is indicative of a stable supply of condoms and a reflection of the efficiency of the supply chain management system, particularly the quality of inventory control within the district store. 2. Number of stock in months that is less than one month: This is a subset of the aforementioned indicator; it is expected that stores maintain a minimum quantity of one month of stock. This indicator illustrates the likelihood of stores running out of condoms by having stock in month levels of less than 38 District-wise reports for stock status are also available commodity-wise. As condoms are by far the largest commodity handled by various district stores, only logistics data on condoms is included in the analysis. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 49 one month. This indicator is calculated by dividing the stock at hand by the average stock level and determining if the quantity of stock is less than the required minimum level (one month of stock). Findings Punjab In Table 7, at the aggregate level (district), the percentage of non-reporting months across districts vary from 7.7 to 36 percent. At the entity level, these percentages vary from 0 to 77 percent. The following non-reporting behavior is detailed as follows:  Stores of the Population Welfare Department (PWD) are the most regular or timely in submitting the monthly stock positions as nearly all of the stores have zero non-reporting months for the period July 2011-July 2012.  The aggregate percentage of non-reporting months is influenced by the non-reporting of stores managed by DOH and LHW.  In five out of eight districts, the percentage of non-reporting months for LHW stores is higher than the percentage of non-reporting months for DOH stores. The relationship is reversed in the three remaining districts. The last three columns of Table 7 present the results of the two other indicators from the districts of Punjab. Average months of stock in stores range from 0.3 to 53 months across districts and across all entities. Five out of eight PWD stores have a small variation (as indicated by the coefficient of variation) in the MOS, implying that supplies do not exhibit a large magnitude of month-to-month fluctuations. Two out of eight LHW stores have a variation in stock of less than 50 percent. In seven out of eight DOH stores the variation in months of stock is more than 50 percent. The DOH stores have a lower variation in months of stock than the LHW stores. In the last column, the number of months of stock that is less than one month is calculated at the district level by the factoring in the stock in months per entity that is less than one month. The range of the number of months of stock (<1) is from 0-11 months out of a total of 39 months. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 50 TABLE 7: PUNJAB-DISTRICT LEVEL PROFILE OF COMPLIANCE INDICATORS Districts and Entities Percentage of Non-Reporting Months Average Months of Stock Coefficient of Variation39 Months of Stock (<1 month) Bhakkar 8 P&Ds/PWDs 0 5.3 13.5 LHWs 7.7 3.4 52.6 DOH 30.7 0.3 53.8 Overall 12.8 D.G. Khan 4 P&Ds/PWDs 0 6.3 59.6 LHWs 23.1 23.8 164.3 DOH 61.5 1.9 58.2 Overall 28.2 Lahore 0 P&Ds/PWDs 0 4.0 8.8 LHWs 76.9 - DOH 30.7 53.0 148.6 Overall 35.9 Layyah 2 P&Ds/PWDs 0 5.6 27.8 LHWs 23.1 3.5 53.8 DOH 15.4 29.8 51.0 Overall 7.7 Mianwal 3 P&Ds/PWDs 0 5.7 2.5 LHWs 15.4 8.1 60.5 DOH 69.2 0.0 - Overall 28.2 Muzaffargarh 1 P&Ds/PWDs 0 6.0 65.5 LHWs 23.1 14.8 16.7 39 The coefficient of variation is a measure of dispersion. It is equal to the standard deviation divided by the mean. Higher values signify greater dispersion around the mean. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 51 Districts and Entities Percentage of Non-Reporting Months Average Months of Stock Coefficient of Variation39 Months of Stock (<1 month) DOH 0 25.4 27.8 Overall 7.7 Rahim Yar Khan 9 P&Ds/PWDs 0 4.3 3.3 LHWs 69.2 2.4 141.4 DOH 38.5 0.6 44.6 Overall 35.9 Rajanpur 11 P&Ds/PWDs 0 4.1 71.6 LHWs 38.5 2.3 9.2 DOH 23.1 0.5 133.4 Overall 20.5 Table 8 summarizes the identification of more and less compliant districts for the province of Punjab based on the information illustrated in the above Table 1. TABLE 8: PUNJAB-MORE AND LESS COMPLIANT DISTRICTS Indicator More Compliant Districts Less Compliant Districts Percentage of non-reporting months Muzaffargarh Lahore or Rahim Y. Khan Variation in months of stock (MOS) Muzaffargarh40 D.G. Khan or Lahore or Rajanpur Number of months of stock (<1month) Lahore or Muzaffargarh Rajanpur or Rahim Y. Khan Sindh During Phase I, four districts were selected for implementation of the automated LMIS. Of the four districts (three in Karachi and one in Hyderabad), the automated LMIS is currently functional in Hyderabad and Karachi (East) districts. The store in Karachi (East) is managed and operated by PWD. Table 9 provides the district level profiles of compliance indicators in the pilot districts of Sindh and Khyber Pakhtunkhwa. For Sindh, the percentages of non-reporting months for four districts range from 0 to 67 percent. Within Sindh, only PWD stores in Karachi (East) and Hyderabad report months of stock (MOS). The variability in months of stock is higher in Hyderabad as compared to Karachi (East). However, the number of months of stock that is less than one month is higher in Karachi (East) than in Hyderabad for the period July 2011–July 2012. 4 Mianwali District also shows low Coefficient of Variation (C.V) but is not considered for ‘most’ compliant district as 3 out of 39 months have 0 stocks. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 52 Khyber Pakhtunkhwa (KPK) In KPK, three districts were selected during Phase I of the automation process. During the period July 2011-July 2012, the aggregate percentages of non-reporting months for the three districts in KPK range from 15.4 to 35.9 percent. Although Peshawar is compliant to an extent in sending reports, the variability in MOS is much higher than in the other two pilot districts, i.e., Abbottabad and Haripur. TABLE 9: SINDH AND KHYBER PAKHTUNKHWA – DISTRICT LEVEL PROFILE OF COMPLIANCE INDICATORS Districts and Entities Percentage of Non-Reporting Months Average Months Of Stock Coefficient of Variation41 Months Of Stock (<1 month) SINDH Hyderabad 1 P&Ds/PWDs 0 3.06 53.13 LHWs 100 DOH 100 Overall 66.7 Karachi (EAST) 3 P&Ds/PWDs 0 1.73 46.66 OVERALL 0 Karachi-Korangi NA LHWs 0 DOH 100 OVERALL 50 Karachi-Malir NA LHWs 0 DOH 100 OVERALL 50 KHYBER PAKHTUNKHWA Abbottabad 2 P&Ds/PWDs 0 2.40 49.71 LHWs 0 4.96 63.79 DOH 46.1 36.21 92.73 41 The coefficient of variation is a measure of variability of months of stock. It is equal to the standard deviation divided by the mean. Higher values signify greater dispersion around the mean whilst lower values indicate stability in stock. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 53 Districts and Entities Percentage of Non-Reporting Months Average Months Of Stock Coefficient of Variation41 Months Of Stock (<1 month) Overall 15.4 Haripur 5 P&Ds/PWDs 0 2.98 43.40 LHWs 7.7 1.33 52.22 DOH 92.3 Overall 33.3 Peshawar 0 P&Ds/PWDs 0 5.91 149.86 LHWs 23.1 3.26 2.28 DOH 84.6 Overall 35.9 Table 10 summarizes the identification of ‘more’ and ‘less’ compliant districts in the province of Sindh and Khyber Pakhtunkhwa based on the information illustrated in Table 9. TABLE 10: SINDH AND KHYBER PAKHTUNKHWA - MORE AND LESS COMPLIANT DISTRICTS Indicator More Compliant Districts Less Compliant Districts Sindh Percentage of non-reporting months Karachi (East) Hyderabad Variability of MOS Karachi (East) Hyderabad Number of months of stock (<1month) Hyderabad Karachi Khyber Pakhtunkhwa Percentage of non-reporting months Abbottabad Peshawar Variability of MOS Haripur Peshawar Number of months of stock (<1month) Peshawar Haripur AJK, FATA and Gilgit Baltistan In these three administrative regions, five pilot districts were selected during Phase I of the automation process. Of these three districts, the automation process has not been implemented in Khyber Agency due to security reasons. Table 11 gives the percentages of non-reporting months for five districts. The aggregate percentages range from 0- 67 percent. Note that in spite of full reporting by entities in Rawalakot, AJK, the data on MOS is missing. The variability in MOS is the lowest in Khyber Agency and highest in Muzaffarabad. The number of months of stock (<1 month) among the sampled months is the highest in Gilgit followed by Skardu. The results in Table 11 are summarized in terms of most and less compliant districts in Table 12. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 54 TABLE 11: AJK, FATA AND GILGIT BALTISTAN - DISTRICT LEVEL PROFILE OF COMPLIANCE INDICATORS 42 The coefficient of variation is a measure of dispersion. It is equal to the standard deviation divided by the mean. Higher values signify greater dispersion around the mean. Districts & Entities Percentage of Non-Reporting Months Average Months of Stock Coefficient of Variation42 Months of Stock (<1 month) Muzaffarabad 2 P&Ds/PWDs 0 3.72 81.00 LHWs 100 11.17 131.37 DOH 100 Overall 66.7 Rawalakot NA P&Ds/PWDs 0 LHWs 0 DOH 0 Overall 0 Khyber Agency 1 P&Ds/PWDs 0 3.18 45.19 LHWs 100 DOH 100 Overall 66.7 Gilgit 4 P&Ds/PWDs 0 1.72 91.61 LHWs 100 DOH 100 Overall 66.7 Skardu 3 P&Ds/PWDs 0 5.24 81.08 LHWs 100 DOH 100 Overall 66.7 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 55 Table 12 summarizes the identification of more and less compliant districts in the province of Sindh and Khyber Pakhtunkhwa based on ranking of the information given in Table 4. TABLE 12: AJK, FATA AND GILGIT BALTISTAN – MORE AND LESS COMPLIANT DISTRICTS Indicator More Compliant Districts Less Compliant Districts Percentage of non-reporting months Rawalakot Muzaffarabad or Gilgit or Skardu Variability of MOS Khyber Agency Muzaffarabad or Gilgit or Skardu Number of months of stock (<1month) Khyber Agency Gilgit or Skardu or Muzaffarabad MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 56 ANNEX 3: PROVINCIAL OVERVIEW Punjab Districts and Entities Presence of District Warehouse - Yes/No Automated LMIS Operationalized in the District (month and year) Bhakkar P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Sep-11 DOH Yes (Makeshift arrangements made within the existing set up of EDO(H) Feb-12 D.G. Khan P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Sep-11 DOH Yes Oct-11 Lahore P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Oct-11 DOH Yes Oct-11 Layyah P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Sep-11 DOH Yes Nov-11 Mianwali P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Sep-11 DOH Yes (Makeshift arrangements made within the existing set up of EDO(H) Sept-11 (Staff transferred) Muzaffargarh P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the Sep-11 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 57 Districts and Entities Presence of District Warehouse - Yes/No Automated LMIS Operationalized in the District (month and year) existing set up of EDO(H) DOH Yes Sep-11 Rahim Yar Khan P&Ds/PWDs Yes (1 X room Rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Dec-11 DOH Yes (Makeshift arrangements made within the existing set up of EDO(H) Dec-11 (staff transferred) Rajanpur P&Ds/PWDs Yes (1 X room rented building) May-11 LHWs Yes (Makeshift arrangements made within the existing set up of EDO(H) Sep-11 DOH Yes (Makeshift arrangements made within the existing set up of EDO(H) Jan-12 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 58 Sindh and Khyber Pakhtunkhwa Districts and Entities Presence of District Warehouse Yes/No Automated LMIS Operationalized in the District (month and year) Comments Hyderabad P&Ds/PWDs Yes June, 2011 LHWs Yes June, 2011 DOH Yes June, 2011 Karachi East P&Ds/PWDs Yes July, 2012 Karachi-Korangi LHWs Yes DOH Yes District and municipal system changed, Karachi was divided into five separate districts, however, due to conflict in ruling parties, system was collapsed, hardware support delayed until there is clarity about the district system DPIU was being shifted, hardware support delayed Karachi-Malir LHWs Yes DPIU was being shifted, hardware support delayed DOH Yes District and Municipal system changed, Karachi was divided into five separate districts, however, due to conflict in ruling parties system was collapsed, hardware support delayed until clarity about the district system. Abbottabad P&Ds/PWDs Yes June, 2011 LHWs Yes June, 2011 DOH Yes Jan, 2012 Haripur P&Ds/PWDs Yes June, 2011 LHWs Yes June, 2011 DOH Yes June, 2011 Peshawar P&Ds/PWDs Yes June, 2011 LHWs Yes June, 2011 DOH Yes June, 2011 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 59 AJK, FATA and Gilgit Baltistan Districts and Entities Presence of District Warehouse Yes/No Automated LMIS Operationalized in the District (month and year) Muzaffarabad P&Ds/PWDs Yes Oct-11 LHWs Yes Oct-11 DOH Yes Oct-11 Rawalakot P&Ds/PWDs Yes Oct-11 LHWs Yes Oct-11 DOH Yes Oct-11 Khyber Agency P&Ds/PWDs Yes Not Implemented yet due to security reasons LHWs Yes Not Implemented yet due to security reasons DOH Yes Not Implemented yet due to security reasons Gilgit P&Ds/PWDs Yes Oct-11 LHWs Yes Oct-11 DOH Yes Oct-11 Skardu P&Ds/PWDs Yes Oct-11 LHWs Yes Oct-11 DOH Yes Oct-11 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 60 ANNEX 4: PROVINCIAL STATISTICS TABLE 13: PUNJAB: NON-REPORTING DISTRICTS Non-reporting Districts Marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months Bhakkar P&Ds/PW Ds 0 LHWs X 7.7 DOH X X X X 30.7 Overall 12.8 D.G. Khan P&Ds/PW Ds 0 LHWs X X X 23.1 DOH X X X X X X X X 61.5 Overall 28.2 Lahore P&Ds/PW Ds 0 LHWs X X X X X X X X X X 76.9 DOH X X X X 30.7 Overall 35.9 Layyah P&Ds/PW 0 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 61 Non-reporting Districts Marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months Ds LHWs X X X 23.1 DOH X X 15.4 Overall 7.7 Mianwali P&Ds/PW Ds 0 LHWs X X 15.4 DOH X X X X X X X X X 69.2 Overall 28.2 Muzaffargarh P&Ds/PW Ds 0 LHWs X X X 23.1 DOH 0 Overall 7.7 Rahim Yar Khan P&Ds/PW Ds 0 LHWs X X X X X X X X X 69.2 DOH X X X X X 38.5 Overall 35.9 Rajanpur MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 62 Non-reporting Districts Marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months P&Ds/PW Ds 0 LHWs X X X X X 38.5 DOH X X X 23.1 Overall 20.5 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 63 TABLE 14: PUNJAB: MONTHS OF STOCK Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Avg. MOS S.D C.V Bhakkar P&Ds/PW Ds 4.1 3.2 4.0 5.1 6.0 5.6 4.5 3.6 5.9 8.0 7.1 6.1 5.1 5.3 0.71 13.46 LHWs 3.7 3.0 1.8 1.3 1.1 7.8 5.7 4.2 4.1 3.2 3.2 1.2 3.4 1.77 52.64 DOH 0.3 0.5 0.1 0.1 0.3 0.6 0.1 0.1 0.3 0.14 53.87 D.G. Khan P&Ds/PW Ds 3.1 4.7 3.5 4.2 6.3 5.1 5.4 5.1 7.4 9.7 9.7 9.1 8.4 6.3 3.75 59.63 LHWs 0.7 0.7 0.1 10.3 22.7 33.6 26.4 50.1 37.1 55.9 23.8 39.0 3 164.2 8 DOH 0.1 0.2 1.5 1.8 6.0 3.7 1.2 1.3 1.7 1.9 1.13 58.18 Lahore P&Ds/PW Ds 3.0 3.9 4.6 4.9 3.8 3.8 3.9 4.4 3.4 5.1 5.3 3.8 2.5 4.0 0.35 8.77 LHWs DOH 1.7 74.2 82.4 33.1 31.0 23.4 2.5 65.3 102.9 113. 0 53.0 78.7 0 148.6 3 Layyah P&Ds/PW Ds 4.8 4.2 3.0 2.8 2.5 4.1 6.1 4.4 9.0 7.9 9.1 7.9 7.0 5.6 1.56 27.78 LHWs 1.7 1.7 2.6 6.4 6.0 5.3 0.4 3.4 4.4 3.5 1.91 53.86 DOH 0.0 2.1 24.0 30.3 104.6 31.0 36.9 26.2 26.9 24.1 21.5 29.8 15.2 51.05 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 64 Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Avg. MOS S.D C.V 0 Mianwali P&Ds/PW Ds 5.3 4.5 4.7 4.4 4.6 3.7 6.6 7.7 6.6 5.8 7.7 6.5 5.5 5.7 0.14 2.50 LHWs 3.6 3.2 6.1 14.0 10.6 10.4 13.3 1.6 7.3 8.5 10.1 8.1 4.88 60.51 DOH 0.0 0.0 0.0 0.0 Muzaffargarh P&Ds/PW Ds 0.9 2.9 1.8 4.2 3.0 6.5 8.0 8.9 8.1 10.0 9.3 8.5 6.5 6.0 3.96 65.49 LHWs 3.6 2.9 14.4 42.1 18.2 12.0 18.5 15.0 6.4 14.8 2.47 16.73 DOH 2.8 2.8 69.7 71.6 51.1 34.4 17.4 11.6 8.1 10.9 11.8 12.8 25.4 7.07 27.82 Rahim Yar Khan P&Ds/PW Ds 3.8 4.1 0.0 3.6 3.6 3.5 5.0 5.4 5.3 6.9 5.8 4.9 4.0 4.3 0.14 3.29 LHWs 4.7 4.7 0.0 0.0 2.4 3.32 141.4 2 DOH 0.0 0.8 0.7 0.5 0.5 0.0 0.1 0.2 1.5 1.6 1.4 0.3 0.6 0.28 44.66 Rajanpur P&Ds/PW Ds 0.0 3.7 2.7 5.1 5.0 4.0 5.1 5.9 4.9 4.1 32.8 4.1 2.90 71.58 LHWs 0.0 0.0 0.0 3.7 2.6 6.2 3.4 0.3 2.0 0.21 10.47 DOH 3.5 2.5 3.1 0.6 0.4 0.4 0.8 0.4 0.2 1.1 0.4 1.0 1.2 1.77 147.3 1 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 65 TABLE 15: SINDH AND KHYBER PAKHTUNKHWA: NON-REPORTING DISTRICTS SINDH: Non-reporting Districts marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non￾reporting months Hyderabad P&Ds/PWDs LHWs X X X X X X X X X X X X X 100 DOH X X X X X X X X X X X X X 100 Overall 66.7 Karachi (East) P&Ds/PWDs Overall 0 Karachi-Korangi LHWs DOH X X X X X X X X X X X X X 100 Overall 50 Karachi-Malir LHWs DOH X X X X X X X X X X X X X 100 Overall 50 KHYBER PAKHTUNKHWA: Non-reporting Districts marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct -11 Nov￾11 Dec￾11 Jan￾12 Feb -12 Mar -12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months Abbottabad MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 66 P&Ds/PWDs LHWs DOH X X X X X X 46.1 Overall 15.4 Haripur P&Ds/PWDs LHWs X 7.7 DOH X X X X X X X X X X X X 92.3 Overall 33.3 Peshawar P&Ds/PWDs LHWs X X X 23.1 DOH X X X X X X X X X X X 84.6 Overall 35.9 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 67 TABLE 16: SINDH AND KHYBER PAKHTUNKHWA: MONTHS OF STOCK SINDH: Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec-11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Avg. MOS S. D C. V Hyderabad P&Ds/P WDs 1.6 0.5 1.5 2.1 1.5 2.6 4 2.9 4.7 6.1 4.6 3.3 4.4 3.06 1.6 2 53. 13 LHWs DOH Overall Karachi (EAST) P&Ds/P WDs 2.3 2.4 2.0 0.9 2.0 1.6 2.0 3.2 0.7 0.6 2.3 1.9 0.6 1.73 0.8 0 46. 66 Overall Karachi-Korangi LHWs DOH Overall Karachi-Malir LHWs DOH Overall KHYBER PAKHTUNKHWA: Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr-12 May￾12 Jun￾12 Jul￾12 Avg. MOS S.D C.V MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 68 Abbottabad P&Ds/P WDs 0.5 2.3 3.9 3.1 2.7 2.6 1.4 1.1 2.0 4.7 3.4 2.3 1.2 2.4 1.19 49.70 LHWs 5.6 4.4 3.4 1.7 1.4 0.5 8.2 11.3 7.4 4.0 3.6 4.3 8.7 5.0 3.16 63.78 DOH 2.4 2.4 12.5 25.2 26.4 87.0 80.4 53.4 36.2 33.58 92.73 Overall Haripur P&Ds/P WDs 2.4 1.5 2.0 4.4 4.9 3.7 3.1 1.8 2.0 5.0 4.1 2.5 1.4 3.0 1.29 43.40 LHWs 0.7 0.7 0.7 0.7 0.7 1.8 1.8 1.8 1.9 2.5 1.3 0.69 52.21 DOH Overall Peshawar P&Ds/P WDs 3.8 35.2 3.5 2.4 2.7 3.0 3.4 3.9 3.9 5.7 4.4 3.0 1.9 5.9 8.85 149.8 LHWs 3.3 3.3 3.3 3.3 3.3 3.3 3.1 3.2 3.3 0.07 2.28 DOH Overall MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 69 TABLE 17: AJK, FATA AND GILGIT BALTISTAN: NON-REPORTING DISTRICTS AJK, FATA and Gilgit Baltistan- Non-reporting Districts marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months Muzaffarabad P&Ds/PW Ds LHWs X X X X X X X X X X X X X 100 DOH X X X X X X X X X X X X X 100 Overall 66.7 Rawalakot P&Ds/PW Ds 0 LHWs 0 DOH 0 Overall 0 Khyber Agency P&Ds/PW Ds LHWs X X X X X X X X X X X X X 100 DOH X X X X X X X X X X X X X 100 Overall 66.7 Gilgit P&Ds/PW Ds 0 LHWs X X X X X X X X X X X X X 100 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 70 AJK, FATA and Gilgit Baltistan- Non-reporting Districts marked as “X” Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Percentage of non-reporting months DOH X X X X X X X X X X X X X 100 Overall 66.7 Skardu P&Ds/PW Ds 0 LHWs X X X X X X X X X X X X X 100 DOH X X X X X X X X X X X X X 100 Overall 66.7 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 71 TABLE 18: AJK, FATA AND GILGIT BALTISTAN: MONTHS OF STOCK AJK, FATA and Gilgit Baltistan: Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Avg. MoS S.D C.V Muzaffarabad P&Ds/PW Ds 4.9 2.9 2.7 4.5 3.7 2.9 1.5 2.5 0.1 11.3 8.0 2.5 0.9 3.7 3.02 81.00 LHWs 28.1 3.1 2.3 11.2 14.6 131.3 7 DOH Overall Rawalakot P&Ds/PW Ds LHWs DOH Overall Khyber Agency P&Ds/PW Ds 4.6 3.7 2.9 1.2 1.4 0.6 4.2 5.1 3.8 3.9 4.0 2.7 3.2 1.43 45.19 LHWs 2.0 DOH Overall Gilgit P&Ds/PW Ds 1.6 0.8 0.2 1.4 0.3 0.1 2.6 2.7 0.9 5.3 3.0 1.7 1.57 91.61 MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 72 AJK, FATA and Gilgit Baltistan: Months of Stock (Condoms) Jul￾11 Aug￾11 Sep￾11 Oct￾11 Nov￾11 Dec￾11 Jan￾12 Feb￾12 Mar￾12 Apr￾12 May￾12 Jun￾12 Jul￾12 Avg. MoS S.D C.V LHWs 7.8 DOH Overall Skardu P&Ds/PW Ds 5.5 4.3 2.1 1.4 0.7 1.7 0.3 8.6 8.5 6.3 4.5 14.9 9.3 5.2 4.25 81.08 LHWs DOH 1.7 0.4 Overall MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 73 ANNEX 5: DATA COLLECTION INSTRUMENTS Group Discussion Guide – Warehouse Staff Introduction (read verbatim): Assalam-o-Alaikum, My name is ……….. .We are from MSI Pakistan; we are working with USAID in Pakistan and helping them to understand the effectiveness of USAID funded projects. The purpose of this discussion is to talk to warehouse management/staff/ government officials (depending on who it is) and get your views on your experience at the central warehouse. Your answers will be kept confidential. There is no right or wrong answer; we just want to learn about your opinion. May we proceed with the interview? If yes, please proceed with the following questions. Capacity Building 1. How did you learn to complete the forms/records for contraceptives used at the central warehouse? 2. Have you ever received training in the proper storage of contraceptives? Yes 3. Have you ever received training in Logistic Management Information System (LMIS)? Last six months: 4. If so, when was the last training? 5. Did you receive training from JSI/DELIVER on the LMIS? 6. If yes, what were the components of the training? (Probes: LMIS, data entry, data management, data transmission, data analysis: report generation, graphs and LMIS forms) 7. What was your overall experience in the training (in terms of new information, usefulness, and knowledge gained)? 8. To what extent did the training meet the session objectives? 9. Do you know what the training objectives were? 10. To what degree were the training contents and information relevant to your current assignment/job/work situation? 11. Did you do any assessment or test after the training on the content? o 12. Has there been any follow up to training by the JSI/DELIVER project? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 74 13. How did the outcome (information and knowledge gained/topics discussed/decisions made) of this training impact your work? 14. Are there any improvements that could be made to this training (Probe: content, delivery and follow-up) 15. How did you think the trainer/facilitator(s) was in terms of time management of sessions – allocation of adequate time slots for instructions, questions and answers session, and group discussion? 16. How well prepared was the trainer/facilitator(s)? 17. Was the content easily understandable for you? And why? (new concepts, language) 18. Are you satisfied with the session hand-outs that were shared during the session in relation to the presentations made/ topics discussed? 19. On a scale of one to five with one being poor and five being excellent, how would you rate this training/ its contents? 20. What is your impression of how well the commodities are moving down the system? 21. Given the pace of development/ revamping of the central warehouse do you think the government can run the central warehouse independently/ without support? 22. What steps do you believe need to be taken for the government to run the central warehouse and carry out the forecasting/quantification/distribution without support? 23. Is there anything you would like to add? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 75 Key Informant Interview Guide – Central Warehouse Management Date: __________ Name of interviewer: ______________ Position/Title: Time in current position: Introduction (read verbatim): Assalam-o-Alaikum, My name is ……….. .We are from MSI Pakistan; we are working with USAID in Pakistan and helping them to understand the effectiveness of USAID funded projects. The purpose of this interview is to talk to warehouse management/staff/ government officials (depending on who it is) and get your views on how the warehouse is functioning. Your answers will be kept confidential. There is no right or wrong answer; we just want to learn about your opinion. May we proceed with the interview? If yes, please proceed with the following questions. Human Resources 1. What is your job responsibility at the CENTRAL WAREHOUSE and what does this role entail? 2. How many staff do you have working in this warehouse? (Get the position-wise details of the staff/organogram)-Is there a warehouse checklist? (Ask to see the checklist). 3. Are there specific procedures and guidelines outlined to warehouse staff for carrying out logistics responsibilities? 4. Is it possible to have documentation of the procedures and guidelines? 5. Does staff have a written job description that includes logistic responsibilities? (Ask to see the JD) 6. How many central warehouse staff have been trained? How frequently are they retrained? 7. Is there onsite mentoring by JSI/DELIVER? How frequent is the on-site mentoring? 8. What is the number of in-house central warehouse evaluations (assessments) conducted since inception of the JSI/DELIVER project? (Ask for report) 9. Are you in contact with the JSI/DELIVER project for any support? 10. What is the number of requests sent to JSI/DELIVER for assistance in the last 6 months? 11. What is the nature of the most frequent requests sent to JSI/DELIVER for assistance? 12. What is the amount of time spent by JSI/DELIVER staff at central warehouse per working month? 13. Is there any additional support/function you would like the JSI/DELIVER project to undertake? 14. Do you believe there is sufficient staff performing warehouse management functions? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 76 15. How often do you have regular interaction/ meetings with subordinate staff? a) Twice every week b) Once every week c) Once in two weeks d) Once in a month e) As and when required Contraceptive Management and LMIS 1. On average, approximately how long does it take between ordering and receiving contraceptives? 2. Describe the data flow of the logistics reporting system (in terms of what reports are sent in from the districts and timelines)? 3. When are the reports due at the central warehouse? 4. Is the central warehouse receiving more or less reports nowas compared to the pre-automation phase? 5. Is feedback usually provided to districts on reports submitted? 6. If yes, on what kinds of areas is feedback given? 7. Do you collect/keep the LMIS forms? How do you use the information collected on the forms? 8. Do you think the LMIS is working well? 9. What can be done to make the LMIS work better? 10. Is there any other information you think should be entered in the LMIS in addition to what is already being collected? 11. How does the central warehouse use information from the LMIS to assure quality of the Logistics system? 12. Are there key performance/critical indicators used to assess the efficiency and effectiveness of the warehouse? Who carries out the forecasting and do you know the process? 13. Have you ever received training in calculating the contraceptive needs for your warehouse? 14. Who determines the district resupply quantities? Do you know the process by which this is done? 15. Is there an established procedure for placing emergency orders? If yes, what is it ? 16. What (in months) are the minimum and maximum stock levels for contraceptives? (ask per commodity)? 17. Do you know at any given time how many of your commodities are in the warehouse? (How do you know?) 18. Is the pipeline status regularly monitored so procurement decisions can be made to avoid stock outs of commodities? 19. Are there any contraceptives you run out of before resupply? Do you routinely have stock outs of any of your commodities? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 77 20. If yes, list the three most frequent? 21. What do you do when you run out of contraceptives? 22. Is there a surplus of any contraceptives? 23. If yes, list the three most frequent? 24. How long does it take to get supplies if you have stock outs? 25. How are commodities transported to the district? 26. What can be done to ensure the regular supply of contraceptives? 27. Is there thing you would like to add? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 78 Key Informant Interview Guide – Central Warehouse Staff Position/Title: Time in current position: Introduction (read verbatim): Assalam-o-Alaikum, My name is ……….. . We are from MSI Pakistan; we are working with USAID in Pakistan and helping them to understand the effectiveness of USAID funded projects. The purpose of this interview is to talk to warehouse staff and get your views on how the warehouse is functioning. Your answers will be kept confidential. There is no right or wrong answers; we just want to learn about your opinion. May we proceed with the interview? If yes, please proceed with the following questions. 1. Number of years and months you have worked at the warehouse? a) Since joining, b) in the current position 2. What is your job function/role? 3. Do you have a written job description that includes logistics responsibilities? 4. Has your job descriptions been revised after rehabilitation? 5. If yes then, how recently was your job description revised? 6. Are there performance measurement standards for staff? 7. Do you have a hard copy/list of the performance measurement standards? 8. When did you receive your most recent supervision visit? 9. When did you receive your last supervision visit that included contraceptive management (e.g. stock cards checked, reports checked, expired stock removed, supplies checked)? 10. Who is the principal person responsible for managing contraceptives at this warehouse? 11. How often are these reports or orders for contraceptives sent to the higher level? 12. How did you learn to complete the forms/records for contraceptives used at this facility? 13. Does this warehouse use a maximum stock level for contraceptives? 14. Does this warehouse use a minimum stock level for contraceptives? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 79 15. What (in months) is the minimum stock level for contraceptives? 16. What (in months) is the maximum stock level for contraceptives? 17. Do you routinely have stock outs of any of your commodities? 18. Are there any contraceptives you run out of before resupply? 19. If yes, list the three most frequent. 20. What do you do when you run out of contraceptives? 21. Is there a surplus of any contraceptives? 22. If yes, list the three most frequent? 23. What do you do with surplus stock? 24. How long does it take to get supplies if you have stock outs? 25. What can be done to ensure the regular supply of contraceptives? 26. Do you have any idea or can you quantify the type products that may be expired or damaged or lost in the last six months? 27. Have any products in the warehouse expired, been damaged or been lost in the last 6 months? 28. Do you keep a record of that? 29. Aside from more staff and/or salary issues what support would help you to do your job better/ more efficiently? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 80 Key Informant Interview Guide – Provincial Level Officials Date: __________ Name of interviewer: ______________ District: Position/Title: Time in current position: Introduction (read verbatim): Assalam-o-Alaikum. My name is ……….. .We are from MSI Pakistan; we are working with USAID in Pakistan and helping them to understand the effectiveness of USAID funded projects. The purpose of this interview is to talk to warehouse management/staff/ government officials (depending on who it is) and get your views on how the warehouse is functioning. Your answers will be kept confidential. There is no right or wrong answers; we just want to learn about your opinion. May we proceed with the interview? If yes, please proceed with the following questions. 1. How far has JSI/DELIVER technical assistance built government capacity to: a) Quantify/forecast contraceptive needs on past consumption trends i. Independently ii. accurately and iii. timely fashion iv. all three b) Conduct quantification with an acceptable margin of error c) Allocate adequate resources for contraceptives procurement: last three year’s figures? What resources has Govt. of Punjab allocated for contraceptive procurement for next year? How much of the resources are actually disbursed for procurement? d) Improve turnaround time in the last 3 years? What is the turnaround time between finalization of provincial requirements and receiving contraceptive commodities at a. Central level b. District level? e) Improve supply chain management: Have stock out months reduced in PWD/DOH/LHW stores? If answer is YES specify which factors mentioned have contributed to it: training of personnel and implementation of LMIS or both? f) Have there been stock outs of any commodities in the district stores? g)If answer is YES specify which factors have contributed to it? 1. Is the Government able to carry out above functions completely without JSI/DELIVER? Fully Partially MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 81 2. If answer is “partially”, which aspects require further support by JSI/DELIVER? Please specify. 3. Is there a procurement unit at the Provincial level? 4. How many staff members were trained by JSI/DELIVER at provincial and district levels and how many continue to be involved in forecasting and procurement of contraceptive commodities? 5. Is there a system of retraining and mentoring put in place by JSI/DELIVER? 6. Have Procurement Plans for next year been completed? a. Was the Procurement Manual consulted in the preparation and was automated LMIS data used? b. What other indices were used in forecasting and quantification? 7. What was the previous years’ experience with regard to completeness and accuracy of forecasts? 8. Have the forecast estimates by provinces/districts improved after automation? 9. Has a Logistics unit been established at Provincial/district level? a. How many staff been trained in eLMIS and use of Logistics Manual? b. How reliant is staff in SCM: Inventory Control, Storage and Distribution? c. Is it a push or a pull system? 10. How are requisition, supply and distribution done at provincial and district level? 11. What steps has JSI/DELIVER taken to build capacity of staff in SCM and distribution? 12. Is there a system/coordination in place for monitoring provincial and district logistics performance? 13. Do you conduct supervisory visits to the district stores and government entities? 14. If yes, how frequently is this done? What tools are used to conduct the supervisory visits? (Ask to see last report). 15. What is the involvement of the private sector (Marie Stopes, PPHI and Green star) in the logistics system? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 82 16. In what ways do you think the private sector is complying with the logistics requirements of the government? 17. What is your impression of how well the commodities are moving down the system? 18. How do you see the future in terms of commodity procurement and supply chains? 19. As a result of decentralization, what is the relationship between the central warehouse and other levels (provincial and district)? 20. How do you see the three district-level entities- DOH, PWD and LHW program using the LMIS effectively/efficiently in the future? 21. Do you think the district warehouses are ready to scale up supply chain operations? 22. Given the pace of development/ revamping of districts do you think the government can run the district stores independently/ without support? 23. What steps do you believe need to be taken for the government to run the district stores and carry out the procurement/distribution without support? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 83 Key Informant Interview Guide – District Level Officials Date: __________ Name of interviewer: ______________ District: Position/Title: Time in current position: Introduction (read verbatim): Assalam-o-Alaikum, My name is ……….. .We are from MSI Pakistan; we are working with USAID in Pakistan and helping them to understand the effectiveness of USAID funded projects. The purpose of this interview is to talk to warehouse management/staff/ government officials (depending on who it is) and get your views on how the warehouse is functioning. Your answers will be kept confidential. There is no right or wrong answer; we just want to learn about your opinion. May we proceed with the interview? If yes, please proceed with the following questions. 1. How far has JSI/DELIVER technical assistance built government capacity to: a. Quantify/forecast contraceptive needs on past consumption trends i. Independently ii. accurately and iii. timely fashion iv. all three b. Conduct quantification with an acceptable margin of error c. Improve turnaround time in the last 3 years? What is the turnaround time between finalization of district requirements and receiving contraceptive commodities at i. Provincial level ii. District level? d. Improve supply chain management: Have stock out months reduced in PWD/DOH/LHW stores? If answer is YES specify which factors mentioned have contributed to it: training of personnel and implementation of LMIS or both? e. Have there been stock outs of any commodities in the district stores? f. If answer is YES specify which factors have contributed to it? 2. Is the Government able to carry out above functions completely without JSI/DELIVER? Fully Partially 3. If answer is “partially”, which aspects require further support by JSI/DELIVER? Please specify. MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 84 4. How many staff are trained by JSI/DELIVER at the district level and how many continue to be involved in forecasting and procurement of contraceptive commodities? 5. Is there a system of retraining and mentoring put in place by JSI/DELIVER? 6. What was the previous years of experience with regard to completeness and accuracy of forecasts? 7. Have the forecast estimates by districts improved after automation? a. How reliant is staff in SCM: Inventory Control, Storage and Distribution? b. Is it a push or a pull system? 8. How are requisition, supply and distribution done at the district level? 9. What steps has JSI/DELIVER taken to build capacity of staff in SCM and distribution? 10. Is there a system/coordination in place for monitoring district logistics performance? 11. Do you conduct supervisory visits to the SDPs? 12. If yes, how frequently is this done? What tools are used to conduct the supervisory visits? (Ask to see last report). 13. What is the involvement of the private sector (Marie Stopes, PPHI and Green star) in the logistics system? 14. In what ways do you think the private sector is complying with the logistics requirements of the government? 15. What is your impression of how well the commodities are moving down the system? 16. As a result of decentralization, what is the relationship between the central warehouse and other levels (provincial and district)? 17. How do you see the 3 district level entities- DOH, PWD and LHW program using the LMIS effectively/efficiently in the future? 18. Do you think the district warehouses are ready to scale up supply chain operations? 19. Given the pace of development/ revamping of districts do you think the government can run the district stores independently/ without support? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 85 20. What steps do you believe need to be taken for the government to run the district stores and carry out the procurement/distribution without support? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 86 Assessment – LMIS Central Warehouse Staff 1. Is there an electronic logistics management information system in use? Comments: 2. Is ELMIS fully functional? Comments: i. computer exists, ii. availability of power and internet connection) iii. software installed and stable iv. warehouse personnel trained and using software 3. Since when is the ELMIS operational? Records and Forms Check 4. Which of the following forms are computerized? i. Inventory control cards Yes No ii. Bin cards Yes No iii. Stock registers Yes No iv. Bills of lading Yes No v. Shipping records Yes No vi. Requisition/issue vouchers Yes No Records and Forms Review 5. What is the updating Cycle for the following: Last Updated on: i. Inventory control module ii. Bin electronic forms iii. Stock electronic records iv. Bill of lading v. Shipping records vi. Requisition/issue voucher records Crosscheck for Data Consistency within Central Warehouse 6. Check whether shipping/receiving of FP commodities match stock cards? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 87 7. Do information system reports at central warehouse show: i. Beginning balance (stock on hand)? Comments: ii. Quantity issued during a specified reporting period Comments: iii. Losses and adjustments? Comments: iv. Quantities received? Comments: v. Quantity utilized per district? Comments: vi. Minimum level? Comments: vii. Maximum level? Comments: 8. Are information system records reconciled against physical inventories in central warehouse? a. If yes, how is this done? b. How often? 9. Are logistics data used at central warehouse appropriate for: a. continuous monitoring of stock balances? Comments: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 88 b. calculating quantities for re-supply? Comments: 10. Do LMIS guidelines/SOP protocols exist? 11. Do staff understand and follow LMIS guidelines? Crosscheck for Data Consistency between Central Warehouse and District Warehouse 12. Do LMIS reports received at the central Ware house provide information on stock status at the District warehouse (i.e., do central warehouse staff have accurate routine information on which districts are stocked out, under-stocked, adequately stocked, or overstocked)? Please explain. Comments: 13. What is the approximate percentage of information system reports received in time to be used for logistics decisions (ordering, distribution, etc.) at the central warehouse: 14. If below 100 percent, explain why districts don’t report or don’t report on time? 15. What feedback mechanisms are in place to channel logistics information back to district stores? telephone reports meetings supervisory visit other none Comments: Use of LMIS 15. Is the information system used to monitor and evaluate the program’s performance? Comments: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 89 16. How is logistics data recorded, managed, analyzed, and used at central warehouse? 17. What indicators related to logistics and/or product availability does the information system track (e.g., stock￾out rate, percentage of reporting, rational prescribing practices, etc.)? a. Who tracks these indicators? How often? 18. What decisions are based on information system reports? /deliver ____________________________ 19. Are issues data or dispensed-to-user data cross-checked against other data sources (e.g., service statistics, demographic surveys, etc.)? Comments: 20. a. Is logistics information provided to the appropriate decision makers for logistics planning (e.g., Ministry of Health, USAID, NGOs)? Comments: b. What information is provided? c. Who provides the information? d. Who receives the information? e. How often? - f. How is the information used? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 90 21. Other comments on the LMIS: 22. How does the central warehouse ascertain quality of data for these essential data elements? 23. Do the central warehouse staff compare the ELMIS outputs to the paper based system? 24. Do the central warehouse use barcoding technology to track: a. receipts b. issues of commodities c. balance of commodities 29. Is there a built-in trigger mechanism or reminder functionality within the ELMIS to alert staff about stock levels and requisitions? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 91 Observational Assessment Tool – Central Warehouse Karachi A. Physical Facility Description Pre- Post- rehabilitation Comments A.1 What is the total area of the of the facility A.2 What was the built in area of the facility pre/post rehabilitation A.3 How many separate store rooms pre/post rehabilitation A.4 Are the store rooms allocated section-wise? If not, what is the scheme to distribute products across sections. A.5 Details of Storage Equipment i) Rack ii) Shelve iii) Almirah iv) Dunnage v) Trolley vi) Others Yes No A.6 Are there Ceiling fans in the storeroom? If “no”, skip to A.8 A.7 If “yes”, how many & are they in running condition? A.8 Are there Exhaust fans in the storeroom? If “no”, skip to A.10 A.9 If “yes”, how many & are they in running condition? Are there functioning material handling MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 92 equipment? A.10 Is there provision of periodic pest control in the Storeroom? If “no”, skip to A.12 A.11 If “yes”, when was the last pest control done? COMPUTERIZATION B.1 Computer Equipment i) Desk Top ii) Laptop iii) UPS iv) Printers v) Scanners vi) Bar Code Scanners B.2 Is computer network wired or wireless? B.3 Describe the type of internet (either in office or Service Provider internet packages) Internet Dial-up ISDN DSL Wireless DSL B.4 Is there a generator for the entire facility or just for running computers or for non￾computer uses. Do you have a back-up system/server in times of repair or maintenance of the original system or when the original system develops fault? B.5 Would you regard the warehouse to be fully automated? B.6 If not automated what functions in the warehouse are still not automated? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 93 STORAGE/WAREHOUSE CONDITIONS Cond. All OC All IUD Inj 1- mnth Inj 2- mnth Inj 3- mnth Imp￾lants C.1 Products that are ready for distribution are arranged so that identification labels and expiry dates, and/or manufacturing dates are visible C.2 Products are stored and organized in a manner accessible for first-expiry/first-out (FEFO) counting and general management C.3 Cartons and products are in good condition Cartons are stacked correctly C.4 Damaged and/or expired products are separated from good products and are removed from inventory C.5 Products are protected from direct sunlight at all times of the day and all seasons C.6 Cartons and products are protected from water and humidity during all seasons C.7 Storage area is visually free from harmful insects and rodents Extra rooms for i) loading and unloading docks ii) Quality inspection and Quarantine iii) Preparing shipments iv) Offices for Administrative Staff MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 94 C.8 Storage area is secured with a lock and key, accessible during normal working hours and limited to authorized personnel. C.9 Products are stored at the appropriate temperatures throughout the day and during all seasons, according to protocols C.10 Hazardous waste is properly disposed of, such as needles, toxic materials, etc . C.11 The roof is maintained in good condition to avoid sunlight and water penetration at all times of the day and during all seasons C.12 Storeroom is maintained in good condition. (e.g., cleaned, all trash removed, shelves are sturdy, boxes are organized.) C.13 Products are stacked at least 10 cm (4 inches) off the floor C.14 Products are stacked at least 30 cm (1 foot) away from the walls and other stacks C.15 Products are stacked no more than 2.5 meters (8 feet) high Are the Pallets made of metal or wood? C.16 Fire safety equipment is available and accessible Training on Fire Safety C.17 Products are stored separately from insecticides and chemicals Yes No C.18 Do you have a functioning refrigerator(s) to store Lab. Chemicals, vaccines, test kits C.19 If ‘yes’, write down the actual temperature by looking at the internal thermometer inside the MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 95 refrigerator (ideally temperature should be between +4 and +8 degrees centigrade). (note if thermometer is broken or missing). C.20 If you have a freezer, what do you keep in it? C.21 Are refrigerators located away from any surrounding objects? C.22 Is temperature chart up-to-date? (To be up￾to-date, there has to be an entry for the day of the visit.) MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 96 Assessment-LMIS District Store Staff 1. Is there an electronic logistics management information system in use? Comments: 2. Is ELMIS fully functional? Comments: i. computer exists, ii. availability of power and internet connection) iii. software installed and stable iv. warehouse personnel trained and using software 3. Since when is the eLMIS operational? Records and Forms Check 4. Which of the following forms are computerized? Records and Forms Review 5. What is the updating Cycle for the following: Last Updated on: i. Inventory control module ii. Bin electronic forms iii. Stock electronic records iv. Bill of lading v. Shipping records vi. Requisition/issue voucher records Crosscheck for Data Consistency within District Store 6. Check whether shipping/receiving of FP commodities match stock cards? i. Inventory control cards Yes No ii. Bin cards Yes No iii. Stock registers Yes No iv. Bills of handling Yes No v. Shipping records Yes No vi. Requisition/issue vouchers Yes No MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 97 7. Do information system reports at Store show: i. Beginning balance (stock on hand)? Comments ii. Quantity issued during a specified reporting period Comments iii. Losses and adjustments? Comments iv. Quantities received? es Comments v. Quantity utilized per district? Comments vi. Minimum level? Comments vii. Maximum level? Comments 8. Are information system records reconciled against physical inventories in Store? a. If yes, how is this done? b. How often? 9. Are logistics data used at Store appropriate for: a. continuous monitoring of stock balances? Comments: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 98 b. calculating quantities for re-supply? Comments: 10. Do LMIS guidelines/SOP protocols exist? 11. Do staff understand and follow LMIS guidelines? Crosscheck for Data Consistency between the Central Warehouse and District Store 12. Do LMIS reports generated at the district ware house provide information on stock status at the District warehouse (i.e., do district ware house staff have accurate routine information on which SDPs/sub-district stores are stocked out, under-stocked, adequately stocked, or overstocked)? Please explain. Comments: 13. What is the approximate percentage of information system reports received in time to be used for logistics decisions (ordering, distribution, etc.) at the district store: 14. If below 100 percent, explain why SDPs/sub-districts do not report or do not report on time? 15. What feedback mechanisms are in place to channel logistics information back to SDPs/ sub-district stores? Telephone reports meetings supervisory visit other none Comments: Use of LMIS 15. Is the information system used to monitor and evaluate the program’s performance? Comments: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 99 16. How is logistics data recorded, managed, analyzed, and used at district store? 17. What indicators related to logistics and/or product availability does the information system track (e.g., stock￾out rate, percentage of reporting, rational prescribing practices, etc.)? a. Who tracks these indicators? How often? 18. What decisions are based on information system reports? forecasting procurement transport/delivery scheduling supervisory visits inventory management how much to resupply other ____________________________ 19. Are issues data or dispensed-to-user data cross-checked against other data sources (e.g., service statistics, etc.)? Comments: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 100 20. a. Is logistics information provided to the appropriate decision makers for logistics planning (e.g., central warehouse, SDPs/NGOs )? Comments: b. What information is provided? c. Who provides the information? d. Who receives the information? e. How often? monthly quarterly semi-annually annually other______________ f. How is the information used? 21. Other comments on the LMIS: 22. How do the District store ascertain quality of data for these essential data elements? 23. Do the district store staff compare the eLMIS outputs to the paper based system? 24. Do the district store use barcoding technology to track: MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 101 a. receipts b. issues of commodities c. balance of commodities 29. Is there a built-in trigger mechanism or reminder functionality within the eLMIS to alert staff about stock levels and requisitions? MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 102 Observational Assessment Tool JSI/DELIVER-Checklist of Observations From District Warehouse, District_____________ DOH/DPWO/LHW Facility_________________ B. Physical Facility Description Yes No Comments A.1 Is the district warehouse owned by the government? A.2 Is the district warehouse rented by the government A.3 Does the building have a dedicated storeroom? A.4 Is there a separate storeroom or a separate place within the store for FP commodities/Medicines A.5 What is the storage area dedicated to product handling? A.6 Details of Storage Equipment vii) Rack viii) Shelve ix) Almirah x) Dunnage xi) Trolley xii) Others MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 103 A.7 Are there Ceiling fans in the storeroom? (Observe). If “no”, skip to A.9 A.8 If “yes”, how many & are they in running condition? A.9 Are there Exhaust fans in the storeroom? (Observe). If “no”, skip to A.11 A.10 If “yes”, how many & are they in running condition? A.11 Is there provision of periodic pest control in the Storeroom? If “no”, skip to B.1 A.12 If “yes”, when was the last pest control done? COMPUTERIZATION B.1 Computer Equipment vii) Desk Top viii) Laptop ix) UPS x) Printers xi) Scanners xii) Bar Code Scanners B.2 Is computer network wired or wireless? B.3 Is internet connectivity available in your office If “yes” skip to B.5 B.4 Is internet connectivity available in your region B.5 Describe the type of internet (either in office or Service Provider internet packages) Internet Dial-up ISDN MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 104 DSL Wireless DSL B.6 Is there a generator for the entire facility or just for running computers or for lighting and ACs. Do you have a back-up system/server in times of repair or maintenance of the original system or when the original system develops fault? STORAGE/WAREHOUSE CONDITIONS Cond. All OC All IUD Inj 1- mnth Inj 2- mnth Inj 3- mnth Imp￾lants Comments C.1 Products that are ready for distribution are arranged so that identification labels and expiry dates, and/or manufacturing dates are visible C.2 Products are stored and organized in a manner accessible for first-expiry/first-out (FEFO) counting and general management C.3 Cartons and products are in good condition Cartons are stacked correctly C.4 Damaged and/or expired products are separated from good products and are removed from inventory C.5 Products are protected from direct sunlight at all times of the day and all seasons MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 105 C.6 Cartons and products are protected from water and humidity during all seasons C.7 Storage area is visually free from harmful insects and rodents Extra rooms for v) loading and unloading docks vi) Quality inspection and Quarantine vii) Preparing shipments viii) Offices for Administrative Staff C.8 Storage area is secured with a lock and key, accessible during normal working hours and limited to authorized personnel. C.9 Products are stored at the appropriate temperatures throughout the day and during all seasons, according to protocols C.10 Hazardous waste is properly disposed of, such as needles, toxic materials, etc . C.11 The roof is maintained in good condition to avoid sunlight and water penetration at all times of the day and during all seasons C.12 Storeroom is maintained in good condition. (e.g., cleaned, all trash removed, shelves are sturdy and boxes are organized.) C.13 Products are stacked at least 10 cm (4 inches) off the floor MID-TERM EVALUATION OF THE JSI/DELIVER PROJECT 106 C.14 Products are stacked at least 30 cm (1 foot) away from the walls and other stacks C.15 Products are stacked no more than 2.5 meters (8 feet) high C.16 Fire safety equipment is available and accessible Training on Fire Safety C.17 Products are stored separately from insecticides and chemicals Yes No Comments C.18 Do you have a functioning refrigerator(s) to store Lab. Chemicals, vaccines, test kits C.19 If ‘yes’, write down the actual temperature by looking at the internal thermometer inside the refrigerator (ideally temperature should be between +4 and +8 degrees centigrade). (note if thermometer is broken or missing). C.20 If you have a freezer, what do you keep in it? C.21 Are refrigerators located away from any surrounding objects? C.22 Is temperature chart up-to-date? (To be up￾to-date, there has to be an entry for the day of the visit.) MEP Evaluation Report Template 107 ANNEX 6: SOURCES OF INFORMATION Government of Pakistan, Contraceptive Procurement Manual, 2011, July Government of Pakistan, National Population Policy, 2010 (Draft). Hardee K., Leahy E., Research Commentary: Population, Fertility and Family Planning in Pakistan: A Program in Stagnation, Population Action International, 2008 Vol.3 Issue 3. Khan, A. G., Pakistan Report, 2010, Gillespie Foundation, Population Communication, New York Ministry of Health and Ministry of Population, Report on Needs Assessment Study of Drugs and Logistics Management Systems of Pakistan, 2009, USAID TACMIL Project Pakistan. National Institute of Population Studies, Pakistan Demographic and Health Survey, 2006-07. Population Council, Pakistan: A Study of Operations of Contraceptive Logistics System of Population Welfare Program, Pakistan, 1998 April, VII, 36 [65] p. (Research Report No.8). United Nations Population Fund: Contraceptive requirements and logistics management needs in Pakistan. (New York, UNFPA, [1992]. USAID/JSI/DELIVER PROJECT, Task Order 4: Strategy and Work Plan 2012-2013. UASID/JSI/DELIVER PROJECT: Scope of Work, 2009. USAID/JSI/DELIVER PROJECT, Task Order 1: The Logistics Handbook: A Practical Guide for the Supply Chain Management of Health Commodities, 2011. USAID/JSI/DELIVER, Logistics System Assessment Tool (LSAT), 2009, January. USAID/JSI/DELIVER, Logistics Indicators Assessment Tool (LIAT), 2008. USAID/JSI/DELIVER, Measuring Supply Chain Performance: Guide to Key Performance Indicators for Public Health Managers, 2010, May. MEP Evaluation Report Template 108 TABLE 19: DISTRIBUTION OF KIIS/GDS BY GOVERNMENT LEVELS AND ENTITIES Federal Planning Commission 5 Central Ware House 8 43 Population Welfare Department of Health Lady Health Workers Total Provinces Punjab 1 1 1 3 Sindh 1 1 1 3 KPK 1 1 1 3 AJK Region 1 1 1 3 Districts Punjab: Lahore and Muzaffargarh 6 2 3 1144 Sindh: Karachi (East) and Hyderabad 3 - - 3 KPK: Peshawar and Abbottabad 5 2 - 7 AJK: Rawalakot and Muzaffarabad 2 2 2 6 43 Includes six staff members involved in group discussion 44 Includes five staff members involved in group discussion MEP Evaluation Report Template 109