USAID|DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 DECEMBER 2013 This publication was produced at the request of the United States Agency for International Development and prepared independently by Beatriz Ayala-Öström and Igor Novykov, through the Global Health Technical Assistance Project Bridge IV. I. Novykov. Cover Photo by I. Novykov, Kamuzu Central Hospital Pharmacy, 28 November 2013 USAID|DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 DECEMBER 2013 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 13-B4-015) is available in printed or online versions. Online documents can be located on the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Project Bridge IV 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Engility-IRG to the United States Agency for International Development under USAID Contract No. AID-OAA-C-13-00113. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 i ACKNOWLEDGMENTS The USAID/DELIVER Performance Evaluation team is grateful to many individuals within USAID/Malawi and GH Tech and to project counterparts who shared their valuable time and knowledge to help the team understand how the project performed and the challenges. Special thanks go to Chimwemwe Chitsulo and Ugbede-Ojo Abu, both from USAID/Malawi, and also to the Ministry of Health, especially Albert Khuwi, Deputy Director of the Ministry Health Technical Support Services – Pharmaceutical (HTSS-P), for his generous support in facilitating meetings, organizing field visits, and sharing his experiences; and the zonal and district health management teams, together with the staffs of central hospitals and health facilities in Lilongwe, Thyolo, Machinga, Dowa, and Mzimba North. Our thanks also go to USAID/DELIVER staff for their assistance in providing information and sharing with us their implementing challenges and successes, to the Central Medical Stores Trust and its three regional branches, and to other stakeholders, donors, and implementing partners, who were interviewed during this evaluation. Finally, a word of thanks to the Responsible Safari Company for logistics support throughout the in-country phase. ii USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 iii CONTENTS ACRONYMS ........................................................................................................................................ v EXECUTIVE SUMMARY ................................................................................................................. vii Introduction................................................................................................................................. vii Methodology ............................................................................................................................... vii Findings ........................................................................................................................................ viii Recommendations ....................................................................................................................... x I. INTRODUCTION AND BACKGROUND .......................................................................... 1 Evaluation Purpose ....................................................................................................................... 2 Evaluation Questions ................................................................................................................... 2 II. METHODS AND LIMITATIONS ............................................................................................. 3 Methodology ................................................................................................................................. 3 Limitations ..................................................................................................................................... 4 III. FINDINGS ..................................................................................................................................... 5 Strengths and Weaknesses of the USAID/DELIVER Program ........................................... 5 IV. CONCLUSIONS ........................................................................................................................19 V. RECOMMENDATIONS ...........................................................................................................21 Options for Addressing Identified Gaps in Malawian Capacity to Assure Commodity Security .................................................................................................................21 ANNEXES Annex 1. Scope of Work ................................................................................................................25 Annex II. Selected Performance Indicators Matrix ...................................................................43 Annex III. Representatives of Donor Agencies, Governmental Officials, Health Officers and Health Workers Interviewed on the Central and District Levels .................45 Annex IV. Health Facilities Visited by the Evaluation Team ....................................................49 Annex V. Data Collection Tool, Questionnaire for Key Informant Interviews .................51 Annex VI. Evaluation Methodology Summary ............................................................................55 Annex VII. Facility Supply Status Based on LMIS Report, October 2013 ...........................57 Annex VIII. HIV/Lab Tests OP Indicator Status and Evaluation Team Comments ............59 Annex IX. USAID/DELIVER Project Organizational Chart .....................................................61 Annex X. Secondary Data Presented by USAID/DELIVER to the Evaluation Team .........63 iv USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Annex XI. Secondary Data Drawn from E-LMIS (Supply Chain Manager) Available During Field Visits ...........................................................................................................65 Annex XII. Matrix of Field Findings ...............................................................................................69 Annex XIII. Monthly LMIS Form ....................................................................................................71 Annex XIV. Proposed Modification of LMIS Form for Service Delivery Points .................73 Annex XV. Disclosure of Conflict of Interest ............................................................................75 Annex XVI. References ..................................................................................................................79 TABLES Table 1. Evaluation Matrix Plan ......................................................................................................34 Table XI-1. Stock-outs in Thyolo District, Six Months of 2010 (Marked With “X”) .......65 Table XI-2. Stock-outs in Thyolo District, 2011–12 (Days of Stock-out) ...........................66 Table XI-3. Stock-outs in Thyolo District, 2013 (Days of Stock-out) ..................................67 Table XI-4. Average Health Facility Reporting Rates, Thyolo District, 2010–13 (Percent) ............................................................................................................................68 Table XI-5. Average Health Facility Reporting Rates, Machinga District, 2007–13 (Percent) ............................................................................................................................68 FIGURES Figure 1. Average Reporting Rates for Health Facilities, Malawi, 2007–13 (Percent)* .....63 Figure 2. Average Stock-out Rate (%) on all Four Presentations of Coartem, Malawi, 2008–13* ..............................................................................................................................63 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 v ACRONYMS ACT Artemisinin combination therapy AMC Average monthly consumption ARV Antiretroviral drugs CCP Contraceptive commodity procurement CDC U.S. Centers for Disease Control and Prevention CHAM Christian Health Association of Malawi CML Cargo Management Logistics CMS Central Medical Stores CMST Central Medical Stores Trust COP Chief of Party CS Commodity security CSL Commodities Security and Logistics Division, USAID CYP Couple years of protection DFID Department for International Development (UK) DHMT District Health Management Team DHO District Health Officer DPT District Pharmacy Technicians ED Essential drugs EDK Essential drugs kit EDP Essential Drug Program EHP Essential Health Package EUV End Use Verification FP Family planning FPLM Family Planning Logistics Management project GF Global Fund GOM Government of Malawi HMIS Health management information system HPN Health, Population, and Nutrition HSA Health Surveillance Assistant HSSP Health Sector Strategic Plan, Malawi HTSS-P Health Technical Support Services – Pharmaceuticals IHS Imperial Health Sciences (formerly RTT) vi USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 ISSPM Integrated Supportive Supervision and Peer Mentoring JSI John Snow, Inc. KfW KfW Entwicklungsbank (German Development Bank) LA Lumefantrine Artemether LMIS Logistics management information system M&E Monitoring and evaluation MIS Management information system MOH Ministry of Health MOS Months of stock MSH Management Sciences for Health NAC National AIDS Commission NGO Non-governmental organization NMCP National Malaria Control Program NSSD National Stock Status Database PEPFAR U.S. President’s Emergency Plan for AIDS Research PMI President’s Malaria Initiative PMP Performance Monitoring Plan PMPB Pharmaceutical Medicines and Poisons Board PPMR Procurement Planning and Monitoring Report PSC Parallel supply chain RDT Rapid diagnostic test RH Reproductive health RHU Reproductive Health Unit SCM Supply chain management SCMgr Supply Chain Manager (software) SDP Service delivery point SOW Scope of work SPS Strengthening pharmaceutical systems SWAp Sector-wide approach TA Technical assistance TB Tuberculosis TL Team Leader UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 vii EXECUTIVE SUMMARY INTRODUCTION This report assesses the effectiveness of the 2007–13 USAID/DELIVER project in Malawi and identifies strengths, weakness, and lessons learned. It also draws conclusions about the most significant achievements of the project in terms of the objectives of USAID/Malawi and the Government of Malawi for supply chain reinforcement and commodity security (CS). A secondary objective of the evaluation was to assess gaps in supply chain management (SCM) in terms of the project’s current program and approach and to identify potential future programming approaches. Though USAID presence dates back to 1987, when it began focusing on the reproductive health (RH) supply chain, the USAID/DELIVER project has operated in Malawi since 2000, having begun as USAID/DELIVER I, which was implemented by John Snow, Inc. (JSI) and partners. From 2007 to 2010, USAID provided support in Malawi through both the USAID/DELIVER Project and the Strengthening Pharmaceutical Systems (SPS) program. Finally, in the third phase of CS support, since the SPS was closed out in 2010 USAID/DELIVER has been USAID’s sole implementer for SCM activities, playing a central role in supporting the MOH as it improved CS and operated parallel supply chains. The third phase began after thefts were detected from warehouses and even more from health facilities. The project was tapped to rapidly establish and operate its parallel supply chain (PSC) for US government malaria and family planning (FP) products and for the warehousing and distribution of Global Fund–procured malaria commodities. In January 2012, the PSC was expanded to provide warehousing and distribution services for essential drug kits (EDK) under the Essential Health Package (EHP], expanding PSC volume by about 50%. The project undertook the PSC for essential drug kits in 2010. METHODOLOGY The evaluation methodology relied on proven approaches for performance evaluations: document and literature review, drafting of open-ended key informant questionnaires, interviews with implementing and development partners, and site visits. In total, the team visited 25 health facilities, including: visits to one district in each health zone (Lilongwe, Dowa, Machinga, Thyolo, and Mzimba North), during which the team visited at least two government health centers, one Christian Health Association of Malawi (CHAM) health center, and community and district hospitals. The team also visited Zomba and Kamuzu hospitals, the Central Medical Stores Trust (CMST), and the CMST regional branches in Blantyre, Mzuzu, and Lilongwe. The team produced an initial methodology document for USAID/Malawi and presented the findings in a half-day workshop with stakeholders. The team consisted of two independent international SCM evaluators who were in-country November 13–December 12, 2013. The evaluation was performed through the GH Tech Project Bridge IV. The performance evaluation sought to answer the following questions: 1. What were the strengths and weaknesses of the USAID/DELIVER program in Malawi in achieving project objectives? 2. To what extent has USAID/DELIVER contributed to improved national capacity for commodity and SCM? viii USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 3. What are options for addressing identified gaps in strengthening Malawian capacity to assure commodity security? FINDINGS This report presents findings, challenges, gaps, and recommendations for each of these questions. Though the performance of the project has been mixed, USAID/DELIVER has successfully operated PSCs and made procurements on behalf of the United States Government (USG) and has institutionalized a logistics management information system (LMIS) in Malawi. However, the annual work plans mandated by USAID since 2012 do not yield the same results as the five-year country strategies and work plans. The influence of USAID/DELIVER lies in its ability to pursue long-term goals and encourage Malawi government ownership of some or most of the activities. From 2006 to 2011, when the project had a long-term vision and five-year objectives and goals, there was country ownership and alignment of goals and objectives. USAID/DELIVER was seen as working closely with the Health Technical Support Services – Pharmaceutical of the Ministry of Health (MOH/HTSS-P). During that time, the project was located close to the MOH, enabling more meaningful exchange of ideas and sharing of knowledge—USAID/DELIVER was seen to be part of the MOH team rather than an outsider. Since USAID mandated one-year plans for 2012 and 2013, the approach has changed to a short￾term vision and strategy, which has not benefited the project. In addition, performance Monitoring Plans (PMP) were only drafted starting in 2012 and have a number of shortcomings. The 2012–13 PMP, for instance, has 52 process indicators that are linked loosely to the work plan but not to the “established standard indicators” to which the work plan text refers, such as number of people trained, number of facilities visited, and number of meetings held. While some indicators are directed at improving the functionality of facilities (% of facilities with delivery notes, physical inventories, stock cards, etc.), they are input-driven. Even potential output indicators, such as couple years of protection (CYP), are framed as process-based indicators, such as “number of CYPs represented by products dispensed at the health facility level.” Some activities have no indicators, others may have several. Of the 52 indicators, 21—40% of the indicators in the PMP—were reported quarterly for 2012–13 (7 on FP and 14 indicators for Task Order 7). In other words, 60% were not reported during the year. Yet the year-end PMP tracks progress on indicators. This raises the issue of the purpose of the quarterly reports if activities are not fully reported and are poorly defined, lacking substance and validation. Some activities in the work plan were not begun; others were carried out beyond their target, such as “number of unannounced spot checks conducted at third party warehousing and transport operations”. The target of 360 visits was doubled to 716 visits in 12 months. This seems excessive and a waste of human and financial resources because it requires constantly visiting third party operators. No explanation is provided to justify or validate indicator measurements, particularly those that are over- or underachieved. USAID/DELIVER does consistently report stock-outs and facility stocks’ reporting rates, aided by the high reporting rates (83–85%) recently reported through the LMIS. This high level of reporting is recent and can be attributed to the MOH’s recent launch of the successful Integrated Supportive Supervision and Peer Mentoring (ISSPM), with support from USAID/DELIVER and other stakeholders. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 ix Data produced by the outdated LMIS system are not secure or validated; to be used at the central level, they must be thoroughly cleaned. The quality of USAID/DELIVER data for stock￾outs and reporting rates is also inconsistent across documents. For instance, according to the Monthly Inventory Management and Reporting Performance report, the October 2013 reporting rate was 85%; however, in the internal USAID/DELIVER “stock status report 2007–13” the reporting rate is 83%. There are many such examples of data quality issues and lack of verification across documents. Given the paucity of data available, quantification of essential drugs continues to be a system gap. USAID/DELIVER’s skills and expertise would be an asset to the MOH in gathering information for a reasonably robust estimate. USAID/DELIVER has devoted significant resources to capacity building, such as support for training health surveillance assistants (HSA), district health officers (DHO), district pharmacy technicians (DPT), and others managing medicine at the facility level. In 2013, two technical advisors (for monitoring and evaluation [M&E] and data management) were successfully recruited and embedded within MOH/HTSS-P. Between 2007 and 2013, USAID/DELIVER continued to train and support the MOH in use of the LMIS system. Capacity building is evident in the operation of the logistics management system and the ability of programs to do their own forecasting. USAID/DELIVER successfully managed PSCs for malaria (funded by the President’s Malaria Initiative [PMI] and the Global Fund); FP (USAID); and distribution of the essential drug kits (DFID, Norway, and KfW) by outsourcing to private partners. Given full resources for malaria, the PSC has reduced stock-outs considerably (3–10% for Lumefantrine Artemether (LA), based on October 2013 LMIS data). Regular monthly deliveries directly to health facilities have ensured predictability, product use, and reporting and system reliance. However, this approach to ensuring continued supplies of malaria commodities in the pipeline has created significant overstocks at both central and health facility levels. The oversupply is apparent in facilities, where there is limited space to store all commodities and there is exposure to leakage, damage, and expiration due to poor internal controls. According to October 2013 LMIS data, over 30% of facilities have more than six months’ supply (maximum should be three) for any LA, and for the 1x6 presentation there is a 48% overstock (although according to MOH, overstocks have been reduced in recent months). Redistribution activities have no associated indicator and, from primary data collected, are limited to only a few facilities. USAID/DELIVER has been the primary USG procurement agent. The whole USAID/DELIVER management and reporting structure merits thorough USAID review to ensure that for the rest of the contract period the project complies with standard reporting procedures and that meaningful indicators of progress (output, outcome, and impact) are linked to objectives and set reporting formats to track quarterly progress for all indicators. The structural review should also set benchmarks for operational costs compared to other PSC operators in Malawi, since USAID/DELIVER operating costs are double those of others working with the same transparency and accountability parameters. x USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 RECOMMENDATIONS USAID should consider: 1. Reinstating multiyear strategies, work plans, and performance frameworks. 2. Including cost-effectiveness parameters and value for money in future SCM contracts. 3. Performance-based financing that directly links a portion of the funding to results. 4. Promoting discussion within the health donor community about how to strengthen plans for the future “integrated” supply chain and potential support to CMST during the transition period 5. Shifting verification of self-reported data to external organizations that can conduct rigorous representative evaluations for annual performance reviews, LMIS, and assessment of national storage systems. For its part, the USAID/DELIVER Project should consider: 1. Identifying meaningful indicators to track SCM performance and draft PMPs with indicators linked to objectives and to outcomes, outputs, and impact over the strategy period. 2. Drawing up, with USAID approval, multiyear country strategies and work plans, with HTSS￾P inputs from the outset. 3. Improving quarterly report quality by reporting activities and indicators consistently. 4. Working across other commodity groups, such as essential drugs, and collecting data for quantification. 5. Resolving data inaccuracies between reports. 6. Facilitating selection of the e-LMIS by MOH and CMST, the eventual system users. 7. Reviewing the costing structure of PSC and other SCM services and ensuring accountability and transparency. 8. Whether the project will have a role during transition to the integrated national supply chain. 9. Heightening support for capacity-building support in such cross-cutting areas as governance and leadership. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 1 I. INTRODUCTION AND BACKGROUND For the last decade, the Malawi Ministry of Health (MOH) has had chronic problems in the national health supply chain resulting in recurrent stock-outs of essential drugs and other medical commodities. Due to the range of problems (e.g., poor quality of data for procurement planning, shortage of qualified professionals, inadequate infrastructure, and problems in financing), Central Medical Stores (CMS, the MOH pharmacy division), could not ensure efficient coordination of the supply chain. Inefficiencies were compounded by leakage and fraud detected in 2010, which led to loss of the major donor’s confidence in the capacity of CMS to keep the national health system supplied with medical commodities. The acute crisis in 2010 became a decisive reason for the donors to introduce parallel supply chains (PSC) to ensure that essential medical supplies were available at service delivery points (SDPs) throughout the country. Since late 2010, moreover, CMS has been undergoing a process of reform centered on establishing it as a parastatal trust and upgrading the internal procurement, warehousing, and distribution systems. To ensure that commodities are available throughout the health system, over the longer term the GOM strategy is to strengthen the Central Medical Supplies Trust (CMST) and ultimately consolidate parallel supply chains into nationally operated systems (Joint Strategy 2012). Presently, several donor-financed PSCs are in existence, e.g., for HIV, malaria, and family planning (FP) products; these supply facilities directly. In 2012, a multi-stakeholder team drew up a strategy (Joint Strategy 2012) for phased integration of the PSCs into the CMST once specified criteria have been met:  Recapitalization, reform, and successful management of current donated products  Successful expansion of the essential drugs supply chain to all SDPs  Integration of additional PSCs  Integration of procurement functions The objectives of the United States Government (USG), the largest donor to Malawi’s health sector, are to increase availability and utilization of quality Essential Health Package (EHP) services in line with Malawi’s National Health Sector Strategic Plan (HSSP) for 2012–16. With more than 26 years of support, USAID plays a leadership role in working with the Government of Malawi (GOM) to strengthen CS by supporting key vertical programs (malaria, FP), and improvements in cross-cutting systems, such as the national logistics management information system (LMIS). USAID/DELIVER has operated in Malawi since 2000, having started as USAID/DELIVER I, which was implemented by John Snow Inc. (JSI) and partners. From 2007 to 2010, USAID provided support through both the USAID/DELIVER Project and the Strengthening Pharmaceutical Systems (SPS) program.1 In the third phase of CS support, since SPS was closed out in 2010 USAID/DELIVER has served as USAID’s primary implementing partner for SCM activities, not 1 The 2007–10 Strengthening Pharmaceutical Systems (SPS) Program, funded by USAID and implemented by Management Sciences for Health (MSH), provided technical support for the President’s Malaria Initiative (PMI) and PEPFAR activities in Malawi, complementing USAID/DELIVER activities. 2 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 only supporting the MOH in improving CS but also operating PSCs. In 2010, the MOH asked JSI to start warehousing malaria commodities and then to distribute essential drug kits (EDKs). EVALUATION PURPOSE The overriding purpose of this performance evaluation was to assess the effectiveness of the USAID/DELIVER Project and identify lessons learned as a basis for advising USAID/Malawi and the Government of Malawi (GOM) about future investments in supply chain management (SCM). With Malawi facing significant continuing SCM challenges and with USAID commitment to country ownership and strengthening national systems, USAID/Malawi must determine the following: 1. What has worked thus far in terms of support provided through the USAID/DELIVER project? What gaps exist for SCM technical assistance moving forward, so that both current and future USAID supply chain resources can be most effectively utilized to improve commodity security (CS) and build up national SCM systems? The primary objective was to analyze the performance of USAID/DELIVER in Malawi from 2007 to 2013 to identify strengths, weaknesses, and lessons learned and to draw conclusions about its most significant achievements in terms of USAID/Malawi and GOM objectives for supply chain strengthening and CS. A secondary objective was to assess SCM gaps against the project’s current program and approach, and identify potential future programming approaches. Evaluation findings will be used to inform decision-making about program priorities, objectives, interventions, and monitoring and evaluation approaches for current and future USAID/DELIVER annual work plans. The evaluation will also be used to inform USAID/Malawi decision-making for future programming. EVALUATION QUESTIONS 1. What are the strengths and weaknesses of the USAID/DELIVER program in Malawi in terms of achieving project objectives?2 To what extent has USAID/DELIVER contributed to improving national capacity for commodity and supply chain management? What options are there for addressing identified gaps in Malawian capacity to assure commodity security? 2 Sub-questions for each evaluation question are provided later in the text USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 3 II. METHODS AND LIMITATIONS METHODOLOGY A team of two independent consultants conducted this evaluation in compliance with the Scope of Work (SOW; Annex I). The team visited five districts in Malawi between 19-29 November 2013 and interviewed key informants at health facilities, MOH, the CMST, and donor and partner organizations. The team drew on qualitative data collected through guided semi-structured interviews and observation checklists to explain and provide insights into contextual elements that have facilitated or hindered the project in achieving its objectives. USAID/DELIVER documents, such as Performance Monitoring Plans (PMPs) and Work and Strategic Plans, were triangulated with analysis of secondary performance monitoring data. Quantitative data helped to determine USAID/DELIVER performance and the extent to which drug supply and CS have been improved. The goal of Task Order 4 is to bolster country supply chain performance to ensure that commodities are available at SDPs in order to identify the extent to which USAID/DELIVER has helped to improve national capacity for commodity and supply chain management. The evaluation team selected indicators measuring supply chain performance from a guide to key performance indicators developed by the USAID/DELIVER project. It was anticipated that as generator and promoter of these indicators, USAID/DELIVER would be using them to track the success of its interventions. The six indicators selected were stock-out rates, inventory accuracy, forecasting accuracy, facility reporting rate, emergency orders, and the value of unusable stock (see Annex II). The evaluation team planned where possible to track performance over the evaluation period. In addition, representatives of donor agencies and central and district government officials, health officers, and health workers were interviewed (see Annex III). To respond to the first question, “What are the strengths and weaknesses of the USAID/DELIVER program in Malawi in achieving project objectives?” the team conducted face￾to-face interviews, where possible the organization/program head and staff with direct responsibility for SCM activities or monitoring and evaluation (M&E). Where available, documents from 2007 to 2013 were examined. In particular, PMPs, work plans, and quarterly reports from 2007 onward and associated indicators were used to draw conclusions about what indicators, activities, and results were tracked over time; the results were compared to the five￾year strategic plan and annual plans post-2011 to measure performance. MOH and USAID priorities for supply chain strengthening and CS were compared with those of USAID/DELIVER. Senior MOH and USAID staff were interviewed to ascertain their views of USAID/DELIVER achievements in support to their priorities. Representative facilities throughout Malawi were selected for field visits in order to enable rigorous and significant analysis of field data to respond to the second question, “To what extent has USAID/DELIVER contributed to improving national capacity for commodity and supply chain management?” The districts visited and criteria for choice were: Northern Zone: Mzimba North District has a district health office with a maternity department for inpatients only. 4 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Central Eastern Zone: Dowa District has a rural district hospital. Central Western Zone: Lilongwe District is an urban district hospital with both urban and rural health facilities. South Eastern Zone: Machinga District, is recognized as performing well. South Western Zone: The Thyolo District hospital is supported by Médecins Sans Frontières (MSF) and has hired a full-time pharmacy/logistics officer provided with a motorbike who is responsible for supply chain supervision. (See Annex IV for the list of the facilities visited.) Information gathered through the comprehensive data collection tool for field visits (Annex V) was used to respond to the evaluation questions, supplemented by in-depth analysis of LMIS secondary data. Recommendations based on findings from all data sources (see Annex VI) respond to the third question, “What options exist for addressing identified gaps for strengthening Malawian capacity to assure commodity security?” They take into account new supply chain trends and developments in Malawi. LIMITATIONS The evaluation scope of work (SOW) was based on the work of a three-person consultant team; however, the local consultant resigned at the beginning of the evaluation and was not replaced. Although these findings and recommendations are the work of two consultants, given the extent of the SOW, a three- or even four-member team should have been required. However, the team has made every effort to discuss with informants the experience of the evaluation period, identify relevant reports and data, and analyze primary and secondary data to the extent possible. It was difficult to collect sufficient primary data due to a turnover of field staff. Secondary data were also scarce, even though generated by USAID/DELIVER’s own Supply Chain Manager (SCMgr) software. As the USAID/DELIVER project did not consistently collect indicators other than stock-out and reporting rates, this was the only comparison available for performance in the early part of the project. Some informants interviewed seemed to agree about what happened at that time; their views are offered under question 1. The indicators USAID/DELIVER used in reporting are not sufficient to judge the extent to which the project has helped to improve national capacity for commodity and supply chain management. PMPs were only available for 2012 and 2013 and few process indicators were tracked. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 5 III.FINDINGS This section responds to the three questions of the performance evaluation in the SOW. Each question has sub-questions that are discussed individually. The response to the third question consists of recommendations for more effective and comprehensive capacity building and CS based on all the findings. STRENGTHS AND WEAKNESSES OF THE USAID/DELIVER PROGRAM 1. To what extent has the current USAID/DELIVER program addressed the most pressing SCM gaps and priorities of the GOM and USAID? What are the strengths and weakness of specific interventions? The current USAID/DELIVER approach has been based on annual planning since 2012 as mandated by USAID; previously, five-year strategies and work plans were drawn up. Annual plans limit USAID/DELIVER ability to address the most pressing SCM gaps because they are by nature short-term. However, because some gaps have persisted over the years, the project has been able to continue to address them, among them training for lower-level health cadres in supply chain and inventory control and storage practices; assisting the MOH with supportive supervision and peer mentoring and strengthening the MOH HTSS-P department by providing availing two technical advisors; operation of the FP and malaria personal service contracts (PSC); and providing procurement services to USAID. At its request, the MOH receives both financial and human resource support. A major gap is the building of national systems that can eventually take over the PSC functions. While this activity is in the work plan, as yet no interventions have been carried out. Given the paucity of data, quantification of essential drug supply continues to be a system gap. PSC skills and expertise would be an asset to the MOH in gathering information for a reasonably robust estimate that could be used when donors provide additional resources to procure essential drugs. Forecasting Strengths: This has been an activity for which USAID/DELIVER has trained central staff over the years, and it appears that the HIV and malaria programs can now undertake their own forecasting without USAID/DELIVER support. The reproductive health (RH) program also reported producing an annual forecast for 2013 without USAID/DELIVER support. While this should be a welcome development and forecasting capacity building across programs celebrated, USAID/DELIVER seems reluctant to hand this activity back to MOH. Weaknesses: There has been a lack of accuracy in forecasting.3 Due to overestimates for 2011 and underestimates for 2012, the MOH decided to undertake a district forecasting exercise led by UNICEF and collect primary data at the source rather than cleaning and validating SCMgr software data. A complex set of issues, both internal (low morale, limited stocks, limited 3 The CMS Trust reported that the 2011 USAID/DELIVER-led annual forecast was triple the national budget for medicines, but in 2012, USAID/DELIVER was overcautious and the forecast below requirements. The USAID/DELIVER chief of party (COP) stated that errors are addressed by asking GOM and donor partners to either make additional procurements or cancel orders, depending on the forecast. The malaria pipeline has been overstocked for two years as a result of USAID/DELIVER forecasting inaccuracy and plan for no stock-outs. 6 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 capacity, and few facility staff trained in supply chain functions) and external (unpredictable financing, limited infrastructure), have affected both the security and the integrity of SCMgr data. Although the outcome of the district forecast is not yet available, USAID/DELIVER disagrees with the criteria for data collection and therefore the likely outcome. It has reported that in 2014, as in previous years, data will be available through SCMgr. Logistics Management Information Systems Strengths: USAID/DELIVER provided LMIS TA to the MOH to ensure that data are reliable and timely. This has entailed training a number of cadres, particularly district health officers (DHOs), in USAID/DELIVER’s SCMgr software. Reporting rates are high (at least 83% in October 2013), which means greater availability of consumption data. USAID/DELIVER supports hardware and internet access procurements to ensure that DHOs have the tools to report monthly to the central level. Weaknesses: Input data are insecure and SCMgr is outdated. Data can be modified because passwords are per facility not per user. Data keyed into SCMgr are not validated so need to be cleaned before the central level can use them. Further, items listed on LMIS forms do not coincide with the list in the electronic database, which calls for manual inputs and thus opportunities for further inaccuracies. USAID/DELIVER-generated data for stock-out and reporting rates are inconsistent across documents. For instance, according to the Monthly Inventory Management and Reporting Performance report, the October 2013 reporting rate is 85%; however, in the USAID/DELIVER internal “Stock Status Report 2007–13” the reporting rate is 83%. There were many examples of such data quality issues and lack of verification in the documents provided. Since USAID/DELIVER installed SCMgr in 2003 and has devoted significant efforts to training and supporting LMIS, the recent LMIS assessment should have been conducted by an independent external organization. Capacity Building Strengths: In collaboration with MOH and the Clinton Health Access Initiative, USAID/DELIVER contributed to the integrated supportive supervision and peer mentoring (ISSPM) system, which has directly increased reporting levels (in March 2013 the reporting rate was 53%, compared to 83% in October). Similarly, USAID/DELIVER has supported training for health surveillance assistants (HSAs), DHOs, district pharmacy technicians (DPT), and other cadres involved with facility medicine management. Two advisors (for M&E and data management) were recruited in 2013 and successfully embedded within MOH/HTSS-P. Since 2007, USAID/DELIVER has continued to train and lend support to the MOH in use of the LMIS system. Weaknesses: There has been no post-training follow-up to ensure knowledge transfer. Parallel Supply Chains Strengths: USAID/DELIVER manages the PSCs for malaria (supported by the President’s Malaria Initiative [PMI] and the Global Fund to Fight AIDs, Tuberculosis, and Malaria [GF]) and FP (USAID) and used to manage distribution of EDKs (DFID, Norway, and KfW) through private partners. Given full resources for malaria requirements, the PSC has reduced stock-outs considerably (3–10% for LA, according to October 2013 LMIS data). Regular monthly deliveries directly to health facilities have ensured product use, reporting, and system reliance. Weaknesses: The malaria PSC overstocks have major cost implications: USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 7  The central warehouse is operating with more months of stock than necessary (6 months would be enough, but the store usually keeps an estimated 10). Since the operator charges for each box stored, to be cost-effective, the supply chain should be managed efficiently and based on set parameters.  Facility oversupply is apparent; there is limited space to store commodities, and due to poor internal controls they may be exposed to leakage, damage, and expiration. According to October 2013 LMIS data, more than 30% of facilities have more than 6 months’ supply (maximum necessary: 3 months) for any LA, but for the 1x6 presentation the overstock is 48%, even after overstocks were reduced in recent months. There is no indicator associated with redistribution activities; and from primary data it appears that redistribution is limited to a few facilities (see Annex VII for facility supply status as of October 2013). To what extent have the project’s approach and interventions reinforced international best practices? The international best practices on which USAID/DELIVER interventions are based are few. In USAID/DELIVER quarterly reports, no information on best practices was found even where the format allowed for them to be reported. Although USAID/DELIVER did not provide the evaluation team with any best practices applied over the years, the evaluation team identified two recent best practices: 1. Direct monthly deliveries to health facilities within 15 days; and 2. Production of Monthly Inventory Management and Reporting Performance Reports, which started after the two technical advisors were placed at HTSS-P in 2013. If other interventions over the past seven years were supportive of best practices, these were not evident from the data collected. USAID/DELIVER may have intended to reinforce best supply chain practices, such as advocating for storage best practices throughout Malawi or through management of the PSCs, but no actual examples of best practice activities were found. What enabling and inhibiting factors explain the achievement of project results or lack thereof? Enabling Factors The key enabling factor over the years is the extended presence of the project in Malawi, from 2001 as USAID/DELIVER and previously from 1987 as the Family Planning Logistics Management (FPLM) Project. It was GOM confidence in the FPLM-generated RH results that promoted expansion to the EHP. This 26-year presence has meant that USAID/DELIVER could have seen its interventions sustained over time. It also means that the project is thoroughly familiar with the country and its challenges. Procurement activities and operating PSCs are new USAID/DELIVER initiatives not contemplated when the project began. As the de facto USAID implementer for procurement and SCM issues in Malawi after the MSH SPS contract ended in 2010, USAID/DELIVER’s long-standing presence has meant the creation and maintenance of organizational relationships with both the MOH and donor partners, despite high staff turnover. USAID/DELIVER is both well-known and knowledgeable about USAID operations and procedures and has supported USAID requirements over time, particularly as an agent for procuring USAID-donated products. 8 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 More recently, management of the supply chain for malaria commodities has placed USAID/DELIVER in a privileged position to ensure that, given outsourcing to contractors, the PSC conforms to expected high management standards. Inhibiting Factors Given the short-term vision and strategy of the annual plans mandated by USAID, USAID/DELIVER is limited in its strengthening of supply chain and commodity security to short￾term measures. Annual work plans are shared with MOH for comments in their final draft form rather than as the process begins. This problem is exacerbated by limited follow-up of interventions and activities. Work plans only have process indicators (there was no monitoring plan up to Y12), and these are not reported consistently for each annual period. The report of the 2013 assessment of the integrated LMIS logistics management shows a lack of leadership dedicated to implementing activities due to significant turn-over and limited MOH capacity; this may be a negative factor outside USAID/DELIVER’s control if it is not addressed by the MOH. Was there any change in USAID/DELIVER performance between the 2007–10 and 2011– 13 periods? If so, what changed and why? The influence of USAID/DELIVER lies in its ability to pursue long-term goals and GOM ownership of at least some, if not most, of the activities. Between 2006 and 2011, when USAID/DELIVER had a long-term vision and five-year objectives and goals, there was country ownership and alignment with country goals and objectives—USAID/DELIVER was seen as working closely with the MOH/HTSS-P. At that time USAID/DELIVER was located physically close to the MOH, which allowed for more meaningful exchange of ideas and sharing of knowledge, and USAID/DELIVER was seen to be part of the MOH team rather than an outsider. Since USAID mandated one-year plans for 2012 and 2013, the resultant short-term vision and strategy has not benefited USAID/DELIVER. Three informants were able to provide insights into USAID/DELIVER performance in both periods, 2007–10 and 2011–13. Two agreed that the first period was better, but it should be noted that the HTSS-P thought that, physical distance notwithstanding, there was a more collaborative process in 2011–13. HTSS-P attributed this to the fact that USAID/DELIVER showed more flexibility and responsiveness to financial requests for conducting supervision, facilitating information-sharing, and upgrading access to hardware and the Internet. The combination of distance from the MOH and a short term vision related principally to what USAID/DELIVER perceived as the most pressing gaps but seemed to other informants to be less positive. Improving performance perhaps relies less on being physically close to the MOH than in providing meaningful TA that translates into knowledge transfer and capacity building so that MOH can take back functions currently associated with USAID/DELIVER. Successful examples would be the two USAID-funded HTSS-P positions. The MOH wishes to identify more staff to be trained through TA to assure knowledge transfer. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 9 To what extent is the performance M&E plan measuring indicators that are appropriate and sufficient to demonstrate the value of USAID/DELIVER to Malawi? The development of performance monitoring plans (PMPs) follows from the development of annual strategies and work plans. Because PMPs were unavailable before Y12, it is difficult to track progress and to ensure that goals and objectives are on course. If no formal indicators were available through a log frame or performance framework arrangement, it is not clear what sort of benchmarks USAID/Malawi used to ensure that its financing yielded the results expected and that investments were safeguarded by health system and CS improvements. The annual Country Strategy and Work Plan contains objectives and activities but no indicators against which to track performance over time. However, under Performance Monitoring and Evaluation it states that The USAID/DELIVER Project has well-established standard indicators for supply chain management. The project will develop a performance monitoring program with USAID/Malawi that includes a set of indicators to complement the indicators of USAID/Washington and which articulates additional monitoring and evaluation activities required by the project to document project activities.... The project will report on a quarterly basis to USAID/Malawi and to the USAID/DELIVER PROJECT home office (USAID/DELIVER 2012–13). The 2012–13 PMP has 52 indicators; these are linked to the work plan but not to the “established standard indicators” referred to in the work plan narrative. The indicators mostly refer of process: number of people trained, number of facilities visited, number of meetings held, etc. While some indicators do look at improvements in the functioning of facilities (% of facilities that have delivery notes, physical inventories, stock cards, etc.), they are input-driven. Even potential output indicators, such as couple years of protection (CYP) are framed as if process￾based, e.g., number of CYPs represented by products dispensed at the health facility level. Some activities have no indicators, others may have several. Of the 52 indicators, 21 were reported quarterly in 2012–13 (7 on FP and 14 for Task Order 7) — that is, 40% of the PMP indicators. That means that 60% of the PMP indicators were not reported. The year-end PMP however, reports the progress of all indicators. This raises the issue of the purpose of quarterly reports if they do not report activities fully. Some activities in the work plan are not undertaken; Cargo Management Logistics (CML) is listed as receiving six short-term TA visits in Y2012–13, but discussions with CML staff made it evident that these did not take place. Also, “prepare for handover of part or all of PSC operations to CMST as appropriate” has no activities implemented. In other cases, more activities were undertaken than targeted, such as “number of unannounced spot checks conducted at 3rd party warehousing and/or transport operations.” The target was 360 visits in one year but 716 were reported. This seems to be a waste of human and financial resources as this means making visits at least twice a day. No explanation is provided to justify indicator measurements, particularly those that are over- or underachieved. In 2010 Q2, the six indicators followed (use of requisition and issue voucher, physical inventory tally with stock card record, receipt data entries on stock card tally with date on delivery note, stock cards filled in correctly, availability of stock cards, and adherence to storage guidelines) deteriorated from the previous year with no analysis of why and therefore no strategy to improve upon the indicators. This example is common for USAID/DELIVER operations. In 10 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 addition to the PMP process indicators, USAID/DELIVER measures its own performance solely through stock-out and reporting rates, rather than adding indicators that would more comprehensively appraise the Malawi supply chain performance (such as those contained in its own reports to the USAID/DELIVER Head Office). This narrow concept of performance is misleading. While stock availability is paramount in any system, in Malawi it is met at the cost of inefficient and costly SCM arrangements. The last 5-year plan set out the basis for TA and CS support for Malawi. While it did not commit to specific indicators of performance, USAID/DELIVER specified a few that it may or may not use for tracking its own performance. These were:  Product availability on day of visit  Forecast accuracy  Logistic system performance indicators, e.g., data accuracy of stock reports, storage conditions, and inventory management, and timeliness of deliveries to SDPs  Presence of an approved and active CS strategy  Presence of an active and inclusive coordinating committee  Use of local organizations for system strengthening  Active private sector participation in CS strategy implementation  Increases in local financing available for contraceptive procurement  Positive changes in policies, procedures, and resources related to logistics system processes. While some of these indicators are appropriate to measure supply chain performance, none were actually selected and tracked over the 5-year period, as evidenced by their absence from annual work plans and the absence of a PMP. Indicators do not need to be numerous, but some are vital. Weber et al. (2010), presented in the final evaluation of the health sector program of work (2006–10) indicators for quantification accuracy, procurement lead times, and facility reporting rates as a meaningful set of performance indicators. Analysis of indicators provided in quarterly reports since the second quarter of 2007 yielded inconclusive results, given limited availability or no quantifiable data related to stock-out rates, forecasting and inventory accuracy, facilities reporting rate, emergency orders, and damage/losses rate. Further, quarterly reports for 2011 (January/March) included indicators that USAID/DELIVER is unable to measure, such as “number of improvements to laws, policies, regulations, or guidelines related to health services and use of health services drafted with USG support” or “number of regional medical stores RMS stocked out on either screening or confirmatory tests” (see Annex VIII). What are the strengths and weaknesses of USAID/DELIVER’s current management, staffing, and institutional configuration in Malawi in terms of achievement of project objectives? The USAID/DELIVER project has 19 professional positions plus 3 office assistants and 3 drivers (see Annex IX for the organization chart). Except for the two advisers seconded to the MOH, all are based centrally in Lilongwe. The program advisors for contraceptive logistics, malaria, and systems strengthening are key to success because they have the most interaction with MOH programs and provide capacity building. Currently, the malaria advisor post is vacant and the USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 11 chief of party (COP) is fulfilling the role. However, according to the COP there are no vacancies and no staffing challenges because the project is appropriately resourced to execute its current mandate. A contracts manager oversees contracts, work orders, and the entire PSC operation, reporting to the deputy director. The M&E advisor oversees performance and three assistant logistics associates work on cross-cutting LMIS, supported by administrative staff. The project also has five assistant logistics associates, who visit third party warehousing and distribution operators and oversee stock counts and commodity movement operations. Their tasks are to:  Oversee CML during monthly distribution through field spot checks.  Oversee monthly physical counts at Imperial Health Services (IHS).  Oversee loading operations.  Oversee warehouse receipt of products.  Reconcile proofs of delivery with MOH-approved distribution lists.  Reconcile stock status reports.  Validate distribution lists.  Work on CML daily distribution updates and producing distribution updates that are shared with the MOH National Malaria Control Program (NMCP) every two days.  Ensure that all trucks are fit for purpose before they are loaded.  Oversee any trans-shipment in the field during distribution. While these tasks are common in in-house operations (i.e., if USAID/DELIVER were running both storage and distribution systems), it is a heavy-handed, costly, and inefficient duplication of efforts to have five assistants undertaking tasks for which service providers are responsible (for CML ensuring that all trucks are fit for purpose before being loaded, and for IHS overseeing warehouse product receipts). In addition, in Y2012-13, it was reported that 716 spot checks of third party operators were conducted. CML and IHS are paid to do the jobs detailed in their contracts; while USAID/DELIVER should continue to carry out occasional spot checks and validate reconciliations and distribution lists, the present arrangement seems distrustful of how outsourced functions are being handled. By duplicating tasks, USAID/DELIVER is paying for them twice. This may have been justified when the PSCs were set up. Malawian-owned CML was given TA from the outset, which successfully improved operations. Given the fragility of the Malawian economy, to safeguard USAID/DELIVER’s transport investments contracts longer than 12 months may be justified. It will also benefit CML if TA to support activities as in the work plan actually take place. The M&E and data management advisors embedded within MOH/HTSS-P have been successful in supporting MOH counterparts. In conjunction with the chief pharmacist and other HTSS-P programs, a comprehensive and easy-to-use Monthly Inventory Management and Reporting Performance report was developed, with an LMIS summary for analysis of consumption and stock status for a range of commodities. Other programs, such as tuberculosis, have requested inclusion of their program data. 12 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 How effectively has USAID/DELIVER employed project resources (human and financial) to achieve project objectives? How well are program resources and approaches linked (malaria, FP, etc.) to address cross-cutting issues? USAID/DELIVER reported that all of its project resources, both human and financial, are spread equally over the activity portfolio, with no activity being more resource-intensive than any other, though at a granular level one might cost slightly more than another, and LMIS is a major priority because it feeds into all the other USAID/DELIVER activities: quantification, procurement, supply planning, storage, distribution, post-marketing surveillance, training, ISSPM, and management of PSCs. The evaluators consider most USAID/DELIVER inputs to be operational and undertaken for or on behalf of the MOH or USAID, such as procurement or PSC management. Some activities, such as forecasting and supply planning, are led by USAID/DELIVER with MOH participation and other activities, such as LMIS inputs, are undertaken by districts and facilities. USAID/DELIVER manages the LMIS data with some input from MOH programs for their commodity groups. Over the years, LMIS investments have been seen to build capacity. The system is health-sector￾wide and is used by a number of MOH programs; it is also cross-cutting. The 2012–13 work plan mentions that “USG will also concentrate efforts on the cross-cutting areas of human resources for health, infrastructure and leadership, governance, management and accountability,” which are considered particularly critical and difficult problems in Malawi. Throughout, in conjunction with MOH and other partners such as CHAI and Village Reach, USAID/DELIVER investments have supported human resources, with a focus on health facility staff (e.g., DHOs, DPTs, and HSAs) who manage stocks. The placing of the two USAID/DELIVER advisors within the MOH is meant to strengthen the system from within by supporting decision making to enhance HTSS-P capabilities. However, the MOH needs to allocate resources for capacity building and USAID/DELIVER needs to train MOH staff on M&E and data management—two SCM gaps yet to be addressed. The design of a new supervision tool supports district decision-making so that the DHO can make decisions about resupply and redistribution of stocks. More meaningful cross-cutting interventions, such as building leadership and governance capacity, have been hampered by MOH staff turnover and limited capacity. While USAID/DELIVER PSCs are safe and accountable, their cost is higher than similar operations, such as the one UNICEF operates. The MOH Resource Mapping Analysis 20124 shows that CMST charges PSC costs at 12.5% of the value of commodities, UNICEF private operators 17%, and USAID/DELIVER 36%—well over twice as much as UNICEF. The cost of one round of additional kit distributions to all facilities requested from the Norwegian Embassy (after June 2013) was triple what UNICEF charged for the same service conforming to the same conditions. Project resources have been used with limited effect. To what extent has the project been able to effectively address and overcome major challenges, both past and current? The USAID/DELIVER project has been able to address some major bottlenecks effectively, particularly with management of the PSCs, which are likely to continue as a stop-gap measure 4 Joint Strategy for Supply Chain Integration in Malawi reference number 1 page 47 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 13 for the foreseeable future. As the project bypasses national structures to provide secure and safe access to medicines, the gap between the PSCs and the national structures continue to widen; the USAID/DELIVER Y12–13 work plan specified activities to hand over to CMST part or all of the PSC operations without specifying any activity to prepare CMST. Because USAID/DELIVER has concentrated on supporting lower structures, it is not clear that it should be allowed a capacity building role to support CMST or whether the project should include CMST staff in some of its SCM training. Longer-term strategies and vision would enable USAID/DELIVER to have a sustained and meaningful impact on Malawi’s health supply chain structures. However, its performance is limited to what it is mandated to do, as informed by annual plans that may or may not cover such major challenges as the sustainability of supply chain systems. NATIONAL CAPACITY DEVELOPMENT 2. To what extent has USAID/DELIVER contributed to improving national capacity for commodity and supply chain management?  What are the most significant achievements that can be attributed to USAID/DELIVER support with regard to improving commodity security and national systems capacity (separately for malaria, FP, HIV, and cross-cutting issues), and how and why were they achieved? Training and Human Resources Of the 25 health facilities visited, 22 had at least one trained staff member: pharmacy technician or assistant or medical staff (medical assistant, nurse, midwife) or an HSA who was assigned responsibility for managing drug storage. Two HSAs were available at almost all governmental health facilities, as the team observed at Mtwolo and Bwangu Health Centers, Mzimba North District; USAID/DELIVER had trained them in drugs management, and they spent up to 50% of their time assisting or managing facility drug stores. In Dowa district, HSAs are available to count stock only once a week because community duties prevent them from giving drug stores daily attention. The evaluation met pharmacy attendants recently trained by USAID/DELIVER in Zomba Central Hospital and Namitondo Community Hospital (CHAM), Lilongwe District. According to USAID/DELIVER, about 1,200 pharmacy attendants have been trained, and two have been or will be assigned to every Malawi health facility. This significant achievement was possible because of the persistent support of USAID/DELIVER and its partners for professional training on drug management. However, there are still not enough trained cadres, and all interviewees validated high demand for additional training on drug stores management. Three of the health facilities visited did not have trained staff (Thekerani Community Hospital, Thyolo District; Mpiri Health Center [CHAM], Machinga District; and Ekwandene Community Hospital [CHAM], Mzimba North District). Their knowledge and skills were acquired from their predecessors, from experience, and from members of staff leaving the facility. Some staff devote time to teaching other auxiliary staff to assist them. Commodity Deliveries and Stock Status Recognizing the institutional weakness of CMST, which resulted in failure of the MOH supply chain system, the USG and the GF in January 2011 contracted with the USAID/DELIVER Project 14 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 to provide logistics services for introducing PSCs for antimalarials and USG-procured FP commodities. USAID/DELIVER has since provided monthly supplies of both products and assisted with distribution of UNICEF EDKs through September 2013. Regular monthly deliveries (with a 15-day margin), confirmed at all health facilities, resulted in another significant achievement: practically no stock-outs at any health facility of antimalarials for the past six to eight months, although two health facility teams did complain of stock-out of one presentation of Coartem (Nainunje HC, Machinga District and Mpherembe HC, Mzimba North District). The situation with FP and HIV/AIDS commodities was similar: only 4 of 25 health facilities visited complained of stock-outs of Depo-Provera (Mpherembe Health Center and Ekwandene Community Hospital [CHAM], Mzimba North District, and Natenje Health Center and Mitundu Community Hospital, Lilongwe District), and one complained of HIV test kit stock￾outs (Mpherembe). To keep stock-outs low, the malaria supply chain has been overstocked, as was confirmed at practically all health facilities (stock exceeding maximum months of stock [MOS] levels). Five facilities had significant surpluses of antimalarials: Thyolo District Hospital had 22 M)S of Coartem, Thekerani Health Center 20 MOS of Fansidar; and Namitondo Community Hospital and Zomba and Kamuzu Central Hospitals more than 12 MOS of Coartem. There seemed to be no surpluses of FP and HIV/AIDS commodities. Staff at 10 of the 25 health facilities visited mentioned that USAID/DELIVER provide supplies mostly according to their consumption rates but at 4 (Namitondo Community Hospital, Zomba and Kamuzu Central Hospitals, and Mpherembe Health Center) staff stated that the project delivered supplies without considering their consumption. Antimalarial medicines are delivered along with other commodities, such as latex gloves. In those cases, in spite of having surplus stocks SDPs do not refuse the new delivery because they fear that refusal would result in cancellation of future orders. Health commodities, including antimalarials (Coartem), are also received by other donors through parallel vertical programs without adequate coordination with district health management teams (DHMTs) and zonal health offices, which exacerbates the over-supply problem, particularly of 1x6 LA. It was noted that although it is a featured activity of the USAID/DELIVER Y2012–13 work plan, redistribution does not work well for leveling out stocks at different health facilities. For example, while Mtwalo Health Center reported significant surpluses of some essential and antimalarial drugs (doxycycline, albendazol, magnesium sulfate, Coartem), Mpherembe Health Center, which is about 15 km from Mtwalo, chronically suffers from their deficit. Redistribution works better within the CHAM structure, which can resolve issues by telephone. Regular physical inventories are conducted at the end of each month at all health facilities visited. If any discrepancy is found, quantities are corrected on the stock cards, and confirmed by a staff signature near the correction. No statements to justify corrections were found. Storage conditions at the majority of health facilities are poor, with limited area (as little as 3–4 m2 at rural health centers), leaking ceilings, broken shelves, and not enough protected doors and windows. CHAM storage conditions were more adequate. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 15 Reporting and LMIS All government health facilities are required to prepare stock and consumption reports to be sent to the assigned informational hubs by the 5th of every month. During the field visit it was confirmed that in all health facilities visited LMIS forms are filled in and sent on time to the districts, with occasional insignificant delays due to unavailability of transport. This was confirmed by secondary data analysis (Annex X, Figure 1). The achievement is attributed to the USAID/DELIVER focus on LMIS reporting and follow-up through ISSPM and monitoring visits. District hospitals or health centers in district capitals serve as informational hubs. These hubs are computerized and use SCMgr software—an electronic LMIS (e-LMIS) introduced by USAID/DELIVER. The Monthly LMIS Report (Annex XIII) is filled in after the physical inventory and daily dispense registers are reconciled with the stock cards. The form contains a list of medicines and other commodities that includes only a limited number of the items carried through the system. Data from this form are entered into the SCMgr database at the hubs. Often, handwritten additions to the preprinted list are not in SCMgr reference tables, and the manual inputs are vulnerable to errors. New products can be entered into the system only at the central level with MOH authorization. The LMIS form is not sufficiently comprehensive: there is no column to show the closing balance at the end of the month so it cannot be cross-referenced to stock on hand at the beginning of the following month. In addition, three columns refer to stock adjustments when one would suffice. The recommendations section suggests LMIS form content. Stocked-out drugs are registered showing the duration of the stock-out for the month. Apparently, data in this column require complicated processing to be clear; otherwise, it simply alerts to stock-outs without showing their duration. Looking at the form (Annex XIII), it is difficult to draw conclusions about stock status and the needs of the health facility that submitted it. For instance, a report on magnesium trisilicate compound (A0228) is typical: Balance on Hand: 20,000 tablets; Positive Adjustment: + 1,000 tablets; Quantity Used: 4,000 tablets. The data presented in this form: 1. Do not confirm that at the end of that month the facility had 17,000 tablets, as it appears from the figures. Any error in the calculation would be difficult to find and follow up. 2. Do not guarantee multiple positive adjustments: most likely the adjustment demonstrated above is done to correct a calculation error without justifying the nature of the adjustment. Data submitted are stored in the SCMgr database that generates reports. However, in practice these reports were available only at Thyolo District Hospital (see Annex XI). The team was advised that the reports are also available at Dowa District Hospital but could not obtain them due to both a power cut and limited time. Kamuzu Central Hospital complained about insufficient refresher training on use of SCMgr software, which takes a day and a half every quarter. The staff did not use SCMgr to report monthly until a skilled pharmacy technician hired in October began to do so. 16 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Because SCMgr is not installed at zonal health offices, the flow of information is to HTSS-P, an MOH unit that consolidates SCMgr data into a single National Stock Status Database (NSSD) that USAID/DELIVER established. This often results in zonal health authorities receiving insufficient and late information on drug supplies in their zones. SCMgr does not store commodity cost data. Annex XI also shows reporting rates for Thyolo district; at Machinga District Hospital, the only data was on health facilities reporting rates, and no data were available at Mzuzu Health Center, the information hub for Mzimba North District. On the day of the evaluation team visit, the e￾LMIS had not been operational since October due to hardware problems. Senior CMST management lacked confidence that the AccPacc software (a locally produced accounting package currently used by CMST) was the most suitable software to manage the national supply chain in the future, but USAID/DELIVER itself has criticized SCMgr as being inadequate for current operations: “Current technology solutions (SCMgr and the NSSD) are outmoded, not user-friendly, are incompatible with the latest Microsoft Windows and Office products, are difficult to the update with new products, and lack the appropriate data security controls” (ISSPM Report, July, 2013). Since USAID/DELIVER developed, implemented, and supported MOH, DHO, and health facilities in the use of SCMgr, the question is why these issues were not resolved before the software was installed and staff trained. Whatever the future of e-LMIS in Malawi, it would need to be linked to the current MOH HMIS system, DHIS2, so that one comprehensive informational tool can handle epidemiological surveillance and monitoring. CHAM facilities use their own LMIS system based on MS Excel tables, and Baobab Health, an NGO, is piloting its own informational Electronic Prescriptions System (EPS) at Kamuzu Central and Mitundu Community Hospitals. The EPS, though not yet operational, is designed for keeping medical records, including drugs dispensed. Monitoring Supply chain performance is routinely monitored quarterly by zonal health teams and monthly by DHMTs with USAID/DELIVER support (fuel, transport, per diem). The quarterly ISSPM Program was initiated in January 2013, also with USAID/DELIVER support. Its purpose is to supervise medical store performance and on-the-job trainings. In collaboration with partners, HTSS designed and adopted a supervision tool to be used during supportive supervision visits to identify and address logistics management issues that health facilities encounter. To complement the supportive supervision efforts from the central level, a mentorship program was initiated, where pharmacy technicians from selected high performing districts were trained to be peer mentors to their colleagues in the districts. (ISSPM Report, July, 2013). Forecasting and Procurement The HTSS-P does central forecasting and quantification annually; the CMST was not involved in previous years. Annual exercises supported by USAID/DELIVER in 2011 and 2012 failed because needs were not estimated adequately. In 2013, evaluation of bids for an 11 billion kwacha tender on procurement for 12 months of essential medical commodities was done with CMST participation even though financial resources are not sufficient to fund the procurement. District forecasting and quantification are done automatically using SCMgr (where it is functional) based on data reported by SDPs. Before a requisition order is issued, the District USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 17 Drugs Committee should agree on forecasts of commodity requirements. Morbidity patterns were considered in the forecasting. Primary data and the LMIS secondary data were analyzed to understand USAID/DELIVER contributions to national capacity for CS and SCM and formulate the response to evaluation question 2. COMMODITY SECURITY What are the remaining priorities for strengthening Malawian capacity to improve commodity security, in general and for product categories supported by USG programming (malaria, FP, HIV)? Using the USAID/DELIVER definition of CS as the “ability of every person to choose, obtain, and use quality health products whenever they need them,” the evaluation team concluded that USAID/DELIVER made significant contributions to improving pipeline security for antimalarials and FP and HIV/AIDS commodities in Malawi during the period evaluated (2007–13). In the context of the entire country, stock-outs of these commodities could be considered insignificant. However, this was achieved through significant investments that have led to a considerable increase in commodity costs and possibly related waste due to additional storage costs, expiration of unusable stock, damage, and leakage. USAID/DELIVER cost efficiency is lowest among all similar interventions in Malawi by agencies and projects supported by other donors. The cost for commodities provided through the USAID/DELIVER supply chain is higher by 36%, much higher than for any other chain. Stock-outs were significantly decreased by oversupplying PSCs with excessive quantities of commodities, not necessarily by building up national systems. The capacity of Malawi’s health supply chain is still weak in spite of the significant contributions USAID/DELIVER has made for its improvement, such as training, monitoring, and mentoring. In the last nine months, though the reporting rate has gone up significantly from 48% to 83%, due to data limitations and low usability there has not been a notable effect on supply chain performance. The evaluation team field trip validated that a number of health facilities had significant stocks of expired drugs. Stocks from health centers and community hospitals awaiting disposal (which may take years) accumulated at district hospitals over long periods. Bureaucratic procedures and poor MOH and DHMT waste management are to blame. The disposal process is not sufficiently documented. Lists of items to be disposed of are prepared but not signed by witnesses to disposal, not kept on file, and not financially validated. Information on expired drugs is not Stock of expired drugs at Thyolo District Hospital. 18 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 reported to USAID/DELIVER, and the value of expired drugs is not available. USAID/DELIVER does not track waste or damage in health facilities. Weaknesses identified in the national SCM as confirmed by findings but not addressed by USAID/DELIVER are as follows: 1. The low relevance of performance indicators, which do not clarify supply chain performance, makes it impossible to take corrective and timely action, for a number of reasons: – Too little focus on output indicators and attention solely to input indicators (numbers of trainings conducted, meetings, monitoring trips, etc.) rather than indicators for assessing the quality of supply chain performance (inventory accuracy rate, forecast accuracy rate, value of unusable stock, etc.). – The low quality of indicators used by USAID/DELIVER, which often do not fully match SMART principles (they are not specific, measurable, attainable, relevant, and time￾bound). – Inconsistency in the use of indicators, which makes it impossible to follow dynamic changes over a given period (discussed on pages 7 and 8), with the exceptions only of stock-out and facility reporting rates. – Underperformance of e-LMIS (SCMgr) in collecting, processing, and disseminating, which makes for low usability of the data available. Lack of coordination between USAID/DELIVER, MOH, donors, and other stakeholders, especially in distribution planning, procurement, commodity deliveries and donations, so that unusable products accumulate and significant amounts expire. Insufficient expertise of cadres at all levels of the national health supply chain and high demand for professional training throughout the country. Inadequate support for supply chain operations by electronic tools (e-LMIS), of which there are a number used by different players, among them AccPac, CHANNEL, SCM, NSSD, MS Excel, EPS, cStock, and Baobab Health). None of them supports all business processes of procurement and supply management cycle; they tend to be incompatible with each other (different architecture and designed for different purposes, with some exception for SCM and NSSD); and they therefore cannot ensure the accounting accuracy and transparency and precision of analysis that are necessary if the supply chain is to function effectively. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 19 IV.CONCLUSIONS The strengths of USAID/DELIVER have been in the operation of PSCs, in collaboration with MOH, and as a USAID procurement agent. Where necessary, USAID/DELIVER has supported and assisted the MOH in undertaking data collection and supervision, It has also spent significant resources over the years in building information technology capacity. Currently most DHOs report monthly consumption data that programs use to make re-supply decisions. USAID/DELIVER has also helped train health facility staff dedicated to inventory and storage functions and ensured a continuous supply of malaria medicines in a fully resourced supply chain system. More work is necessary to ensure that MOH capacity is being built so that it can take over functions undertaken or managed by USAID/DELIVER. The end-to-end supply chain system for malaria needs to be revised to operate within designated parameters to alleviate storage space constraints, avoid expirations, and decrease stock vulnerability to pilferage and leakage due to poor facility control systems. To ensure that supply chains are sustainable, USAID/DELIVER must draft multiyear strategies with performance benchmarks to assure that MOH and other supply chain functions are strengthened. This will depend on selection of meaningful indicators that allow for measuring performance over time to demonstrate the value and effectiveness of USAID investment and GOM support in promoting priorities. USAID/DELIVER uses third parties to operate PSCs, which are closely overseen; over 700 spot checks in just 12 months were reported. Although the USAID/DELIVER work plan specifies activities to prepare for handing over part of or all of the PSC operations to CMST, none were undertaken in Y2012–13. Traditionally, it was the British Government that over time supported building up of the Central Medical Stores (now with trust status). Due to the recent erosion in confidence in the GOM caused by news of corruption and theft, DFID is reviewing its support to the CMST. Whether this might translate into sustainable development of in-house capacity for CMST to take back functions currently done by PSCs, which would permit a degree of integration, is yet to be seen. In collaboration with MOH and CMST, USAID needs to decide the best way forward. 20 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 21 V. RECOMMENDATIONS OPTIONS FOR ADDRESSING IDENTIFIED GAPS IN MALAWIAN CAPACITY TO ASSURE COMMODITY SECURITY During the integration of PSCs into the national system, given its historical presence in Malawi USAID/DELIVER can provide useful support by, e.g., providing TA to other MOH programs beyond malaria and FP commodities. This would also be an opportunity to strengthen work on such cross-cutting issues as governance and leadership structure, though USAID should think about whether that should be part of its current scope and whether USAID/DELIVER would be the right partner to provide this assistance. What gaps are USAID/DELIVER well-placed to address? USAID/DELIVER: USAID/DELIVER should address gaps in M&E of the performance of all supply chain functions related to assuring CS. The following recommendations to address some of the challenges and shortcomings are: 1. After getting TA on measuring methodologies, devise meaningful indicators, using the UNAIDS 2010 publication as a simple guide to introduction to indicators (Hales et al. 2010). 2. Draft a multiyear plan for the rest of the USAID/DELIVER contract (with approval by USAID) that incorporates relevant indicators for progress, to be tracked both in terms of work plan activities and for capacity building and commodity security: a. Identify performance indicators. Demonstrating initiative, jointly with the MOH and DHOs craft an M&E strategy to follow up supply chain performance using agreed indicators from the USAID/DELIVER Project (TOs 4 and 7). Rather than merely inputs, the indicators should to the extent possible reflect outputs, outcomes, and impact, such as forecasting accuracy rate; inventory accuracy rate; order fulfillment rate; stock-out rate (of tracer medicines); facility reporting rate; stock wastage due to expiration or damage; and reporting accuracy rate. (Stock-out and reporting rates are already tracked.) All indicators must correspond to SMART principles and should cover strategies to ensure that progress is measured. b. Adopt longer-term strategies and vision that will promote sustained and meaningful impact on Malawi’s health supply chain structures. All MOH programs will benefit from monitoring the value chain, which is not done at the moment. It is not clear whether it would be preferable for USAID/DELIVER to do this or to support the Pharmaceutical Medicines and Poisons Board (PMPB) in pharmacovigilance activities and stringent testing of products coming into Malawi, including more efficacious and timely testing of products. Technical assistance in this area may be a task better suited not to USAID/DELIVER but to other contractors with different core competences. c. Improve quarterly reports by reporting on all PMP indicators on which there is progress for that quarter. That would enable both USAID/DELIVER and USAID/Malawi to closely track progress. There is need for narrative to explain under- or over-performing indicators or challenges in attaining targets. 3. Resolve data inaccuracies and inconsistencies between reports. 22 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 4. Design a distribution and risk management strategy that identifies all possible risks of loss or wastage of commodities, measures for their prevention, and the staff responsible. The strategy should clearly determine who does what, why, when, where, and how. It should comprehensively describe the entire process of distribution. Stakeholders should agree upon this strategy and decide who is a. Responsible—who will do the work to achieve each task.; b. Accountable—who is ultimately answerable for correct and thorough completion of each deliverable or task. c. Consulting—whose opinions are sought (subject matter experts). d. Informed—who is to be kept up-to-date on progress. e. Support—resources allocated to those responsible. f. Verifier—who checks whether the procedure follows the requirements set forth in the procedure description. 5. Improve coordination with the MOH through close cooperation and information exchange with DHOs and DMOs: a. Give DHOs and DMOs access to data on supply chain performance. DHOs should have independent authorized access to data stored in the e-LMIS in read-only mode. b. Establish regular coordination meetings with district drug committees, involving other donors and implementing agencies to coordinate deliveries, rationalize distribution, and prevent overlaps. c. Ensure that redistribution activities are implemented effectively. d. Record and analyze reasons for supply interruptions; adjust operations based on lessons learned. e. Coordinate with DHOs on supply chain performance and rationalize distribution to prevent overstocks and duplication. f. Communicate to facilities that they can refuse stock (e.g., malaria medicines) if they have a significant surplus. 6. Build capacity through cascade training for SCM-related staff with an initial training of trainers session for central and district staff. Trained trainers would teach health workers at SDPs how to manage drug stores, conduct physical inventories, use stock cards, document adjustments, register daily dispense, fill LMIS forms, etc. This will be sustainable and cost￾effective. All training should be followed by quality testing to enable measurement of training quality (output). Keep an accurate database of all staff trained and periodically follow up with them to ensure that they are using the knowledge they have acquired. 7. At central level, using the two advisors at HTSS-P, draw up a training curriculum for data management and M&E and train staff selected by MOH for sustainable skills transfer. 8. Introduce a fit-for-purpose e-LMIS centered in stakeholders of the integrated supply chain strategy, eventual users of the system chosen by the MOH HTSS-P and CMST. Consideration should be given to sustainable and cost-effective solutions. USAID/DELIVER is well-placed to build up internal facility controls to prevent commodity insecurity. Data collection and analysis of essential medicines will provide a more comprehensive picture of supply and demand. To ease follow up of collected data and improve data accuracy, the LMIS form should be modified as proposed in Annex XIV. 9. During the transition phase, look into collaborating with CMST for distribution of CMST commodities. Supply chain infrastructure is another potential intervention; USAID/DELIVER USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 23 can assist with minor repairs and equipping of drug stores at all governmental health facilities. 10. Review the malaria pipeline in terms of managing it within set parameters to reduce overstocks, storage costs, and facility overcrowding. To attack overstocks with as much zeal as stock-out rates would considerably reduce PSC management and storage running costs. 11. Limit oversight of third party operations to necessary spot checks and as required. The current human resource intensity for oversight is heavy-handed and not justifiable. Review which activities are USAID/DELIVER-specific and which are required by the operators. Contractor feedback should be sought and taken into account. USAID: 1. Shift verification of self- reported data to external organizations that have the capacity to conduct rigorous and representative evaluations for annual performance reviews, LMIS, and assessment of national storage systems. 2. Review how USAID/DELIVER builds capacity to ensure sustainable development for MOH and its stakeholders. 3. Consider including cost-effectiveness parameters and value for money in future SCM contracts and benchmark PSC operational costs against other PSC operators. 4. Review the annual country strategy and work plan to reinstate multiyear strategic plans that have meaningful (not just process) indicators linked to each objectives and set reporting formats to track quarterly progress for all indicators. 5. Consider performance-based financing that directly links a portion of funding to results. 6. Define the role of USAID/DELIVER in the future integrated supply chain system. 7. Promote discussion within the health donor community (GF and DFID primarily) on plans to support CMST during the transition period. CMST should identify requirements to take on supply chain functions, like the four criteria set out in the report on integrating PSCs into the CMST in the short to medium term. 8. Thoroughly review the whole USAID/DELIVER management and reporting structure to ensure that for the rest of the contract period, the project complies with standard reporting procedures and meaningful indicators of progress (output, outcome, and impact) on which there is consistent reporting. This review should include benchmarking operating costs against other PSC operators in Malawi, since USAID/DELIVER’s costs are more than twice as high as others working with the same transparency and accountability parameters. Opportunities for engaging local partners and better strengthening local systems in line with the objectives of the USAID Forward Implementation and Procurement Reform Objective 2: Strengthen local civil society and private sector capacity to improve aid effectiveness and sustainability Significant achievements have already been made in terms of strengthening local civil society and private sector capacity in Malawi. The network of health facilities run by CHAM, for example, is a public-private partnership. USAID/DELIVER initially provided TA to CML in setting its distribution strategies on a commercial footing. The CMST / GF warehouse (Manobec) mirrors operations in private warehouses. Until CMST assumes management of all functions in the 24 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 supply chain, there is an obvious need to outsource to private operators. USAID can provide capacity strengthening to private SCM firms as follows: 1. With MOH, set out policies, strategies, and guidelines and engage civil and private partners, outlining their roles in the national distribution and risk management strategy. 2. Accurately plan activities and follow up on work plan realization. 3. Organize workshops and prepare work plans with DHMT representatives, SDP staff, third party logistics service providers (3PLs) and community leaders; 4. Keep civil society informed about USAID/DELIVER activities and achievements by, e.g., issuing public information bulletins and disseminating success and incident stories via mass media. 5. Procure goods and services locally where possible. 6. Ensure accountability to society on resources committed and the results obtained. 7. Build the technical capacity of private partners as needed. 8. Orient and motivate all partners to achieve results. Objective 3: Increase competition and broaden USAID’s partner base. USAID/DELIVER The project can alleviate risks of underperformance by working with more than one partner: 1. Hire two or three more transportation companies through an open tender. The total fleet capacity might take into account additional consignments from CMST. 2. Divide the country into geographical zones of responsibility for each transporter (these may coincide with the territories of districts). Each transporter would work in its zone, becoming very familiar with it and thus working more effectively. 3. Ensure that transporters move on routes that are safe and rational. 4. Introduce payment-per-kilometer once trucks are moving on designated routes. 5. Ensure that transporters are timely paid on verified invoices. 6. Independently assess storage infrastructure and identify resources for minor repairs and local contractors to do repairs in pharmacy stores. 7. Evaluate the performance of all contractors on a competitive basis. USAID: The agency should take the opportunity to consider procuring supply chain services through other American contractors who are also familiar with USAID rules and have worked on USAID contracts overseas. SCM assessments, such as national storage systems, and specialized TA, as in pharmacovigilance or governance, would provide a new perspective and an independent approach that would allow for benchmarking innovations, approaches, other modus operandi, and costs. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 25 ANNEX 1. SCOPE OF WORK Global Health Technical Assistance Bridge Project (GH Tech) Contract No. AID-OAA-C-13-00113 SCOPE OF WORK FINAL (11/4/13) I. TITLE: USAID/MALAWI: USAID/DELIVER PROJECT PERFORMANCE EVALUATION Contract: Global Health Technical Assistance Bridge IV Project (GH Tech) II. PERFORMANCE PERIOD Work is set to begin on or about November 1, 2013, over a period of approximately 3 months, with the team’s completion of a final draft report and presentation concluded by approximately mid-January 2014. The final report is to be submitted by mid- to end of February 2014. III. FUNDING SOURCE USAID/Malawi IV. PURPOSE OF ASSIGNMENT With the country facing significant ongoing challenges in supply chain management (SCM), and with a strong USAID focus on country ownership and strengthening national systems, USAID/Malawi must determine what has worked thus far and what gaps exist for targeted technical assistance in SCM moving forward, so that both current and future USAID supply chain resources can be most effectively utilized to improve commodity security and strengthen national SCM systems. The USG is a critical partner of the GOM in making improvements in supply chain systems, and following Malawi’s failure in Global Fund Round Ten, it is imperative that Malawi be able to demonstrate improvements in core SCM-related functions to ensure continued eligibility for Global Fund grant funds. Information on what worked and what has not worked under USAID/DELIVER—focusing primarily on technical assistance for strengthening national systems, as opposed to operation of the PSC—will help the USAID/DELIVER Project, USAID/Malawi, and the MOH to make programmatic decisions regarding the best use of current resources to address national gaps. In addition, information about the strengths and weaknesses of the USAID/DELIVER Project to achieve USAID and GOM objectives around the supply chain is needed to inform mission decision-making about future programming strategies and mechanisms, particularly with regard to implementation of USAID/Forward objectives of building country systems and working through local implementing partners to the extent possible. V. BACKGROUND V.1. Public Health Supply Chains in Malawi Malawi, like most other countries in sub-Saharan Africa, relies mainly on the public sector for the delivery of health care services to its citizens. Most health financing comes from the GOM 26 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 and donors, and approximately 60% of health services are provided through publicly-funded facilities, with another 37% provided through affiliates of the Christian Health Association of Malawi (CHAM). MOH policy is to provide services under the Essential Health Package (EHP) for free at the point of service, though in practice all services at the primary health care level are free (district hospitals have introduced user fees to varying degrees through fee-paying private wings). To ensure commodity availability throughout the health system, the MOH’s long-term objective is the existence of a reliable, integrated national supply chain system capable of delivering health commodities to all public sector facilities as well as CHAM. The current national system is designed to deliver health commodities from the Central Medical Stores Trust (CMST) to regional medical stores (RMS), district hospitals, and health centers. Health centers and community health workers handle “last-mile” distribution to the community for specific commodities. Due to weaknesses and shortages in the CMST system, however, there are presently several parallel supply chains (PSCs) in existence—including for HIV, malaria, and family planning products financed by donors—which provide supplies directly to the facility level. The PSC for ARVs and other HIV products financed by the Global Fund has been in operation since 2004, managed by UNICEF and the MOH with financing from the Global Fund. Another PSC for malaria and family planning products was established in 2010 by USAID and the MOH (with financing from USAID and the Global Fund) following thefts and mismanagement of donated commodities in the Central Medical Store (CMS).5 This PSC is implemented by the John Snow International USAID/DELIVER project and managed by USAID and the MOH. In addition, both public and private sectors have relied increasingly on private sector suppliers in recent years to meet their commodity needs. Since late 2010, CMS has been in a process of transition and reform centered on establishing the CMS as a parastatal trust and reforming internal procurement, warehousing, and distribution systems. The transition has been far from smooth, and in 2011 preexisting institutional weaknesses were reinforced by national shortages of fuel for distribution, limited access to foreign exchange for offshore procurements, and de-capitalization of the CMST revolving fund for drug procurement due to nonpayment of invoices by districts and facilities. This resulted in near-crisis levels of stock-outs both within CMST and at the facility level, and an emergency request to health development partners in July 2011 to support short-term procurements of essential drugs. In response to this request and to prevent further widespread stock-outs of essential drugs and health commodities, a multidonor Essential Drugs Program (EDP) was launched in January 2012 to further augment CMST deliveries and reduce stock-out rates at facilities by delivering essential drug kits directly to the facility level. The EDP is an 18-month project running from January 2012 to June 2013 that delivers kits nationwide on a monthly basis through the JSI￾USAID/DELIVER parallel supply chain system. A key secondary objective of the project was to buy time and free up resources for the GOM and CMST to institute reforms, with technical assistance from the Global Fund-supported Supply Chain Management Agent (SCMA) for operations and from a DFID-funded consultancy for procurement management. The $36 million 5 CMS refers to the Central Medical Stores organization prior to its incorporation as a parastatal trust in 2011, when it became CMST. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 27 EDP is financed by KFW, DFID, and Norway and implemented by UNICEF and USAID, which are responsible for procurement and distribution respectively. The existence of multiple PSC systems has helped to mitigate the impact of weaknesses in the national system on key national health programs, such as malaria, family planning, HIV, and immunization, and it has provided a back-up mechanism for temporary distribution of essential drugs during a period of acute crisis for the CMST. However, given weaknesses in commodity management systems at all levels, parallel delivery has led to greater visibility and accountability for products within the delivery chain, but it has been unable to eliminate stock-outs, which have continued to be chronic at the district and facility levels due to a weakening Logistics Management Information System (LMIS); an irregular supply of some commodities, such as ACTs; challenges in supply and distribution planning; and poor commodity management systems at the district and facility level. Stock-outs and stock imbalances for family planning and malaria commodities have remained chronic. Due to their critical importance for achieving public health objectives and targets, supply chain challenges have remained a significant concern for development partners, including the US Government as well as the Global Fund, which finances nearly all of Malawi’s ARV procurements and a large share of malaria commodities. V.2. USG Support for SCM in Malawi As the largest bilateral donor to Malawi’s health sector, USG objectives focus on increased availability and utilization of quality EHP services in line with the National Health Sector Strategic Plan (HSSP) for 2012–16 and its predecessor Program of Work under the health Sector Wide Approach (SWAp). In the supply chain area, USAID has provided support to the GOM to improve commodity logistics for more than 26 years and continues to play a strong leadership role among development partners in working with the Government of Malawi to strengthen commodity security through the provision of support to key vertical programs (malaria, family planning), as well as support to improve cross-cutting systems, such as the national LMIS. As the CMST reform process has gathered momentum since 2010 with support from the Global Fund, DFID, and UNFPA, USAID support through USAID/DELIVER has focused investments in two main areas: operation of the PSC for malaria and family planning products, and technical assistance to the MOH to strengthen national systems and improve commodity security. Technical assistance has focused on supporting the MOH—including the pharmaceutical services department (Health Technical Support Services/Pharmaceuticals section, HTSS/P), the Pharmacy, the National Malaria Control Program, and the Reproductive Health Program—to strengthen planning and coordination centrally while taking steps to improve commodity management and reporting at the community, district, and facility levels. Key areas of support include national quantification and forecasting for all essential medicines, strengthening of the national LMIS, including installation and operation of Supply Chain Manager at all district hospitals and the center as well as a National Stock Status Database, and support to district, facility, and community levels to improve supervision, training, and reporting to improve logistics management. The majority of this assistance has focused on improving availability and management of malaria and family planning commodities, with additional support designed to address cross-cutting issues that affect management of all health commodities. USAID also supports the MOH centrally and is in the process of recruiting and placing two technical 28 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 advisors inside HTSS/P to support the government’s supply chain planning and information management systems. V.3. USAID/DELIVER PROJECT in Malawi USAID/Malawi health investments are primarily focused in 15 of Malawi's 29 districts: Chitipa, Karonga, Nkhotakota, Kasungu, Dowa, Salima, Lilongwe, Balaka, Machinga, Mangochi, Zomba, Mulanje, Phalombe, Chikhwawa, and Nsanje. However, the USAID/DELIVER Project implementation area covered all 29 districts. The USAID/DELIVER Project has undergone three distinct phases of its Malawi work. USAID’s initial support through the USAID/DELIVER project started under USAID/DELIVER I, implemented by JSI and partners, from 2000 to 2006.6 Secondly, from 2007 to 2010, USAID provided support through both the USAID/DELIVER Project and the Strengthening Pharmaceutical Systems (SPS) program.7 Finally, in the third phase since the closeout of SPS in 2010, the USAID/DELIVER project has served as USAID’s primary implementing partner for SCM activities, playing a central role in supporting the MOH to improve commodity security as well as operating a parallel supply chain system. The third phase of USAID/DELIVER assistance began following the detection of thefts from CMS and the subsequent cancellation of planned USAID technical assistance to CMS in late 2010. In response, USAID/DELIVER was tapped to rapidly establish and operate the PSC for USG malaria and family planning products. The MOH, financed by the Global Fund, established a separate agreement with JSI at the same time for the warehousing and distribution of GF-procured malaria commodities. In January 2012 the PSC was further expanded to provide warehousing and distribution services for essential drug kits under the EDP—expanding the volume of the PSC by approximately 50%. Since 2010 USAID/DELIVER has also served as the de facto sole USAID implementer for supply chain technical assistance and capacity-building in Malawi. The overall goal of the USAID/DELIVER Project is to improve the availability of and access to essential medicines through designing, strengthening, and in the case of Malawi operating safe, reliable, and sustainable supply chains, in collaboration with local partners. The specific objectives of USAID/DELIVER technical assistance vary from year to year and across funding streams, which currently include malaria, HIV, family planning, and maternal and child health and are further defined in annual scopes of work and project work plans. USAID investments in the Malawian supply chain through the USAID/DELIVER Project over the past two years have been substantial, making USAID/DELIVER one of the Mission’s largest projects. In FY 2011, USAID provided over $13 million through USAID/DELIVER (Task Orders 4 and 7, of which approximately $9 million was for procurement of malaria commodities and the remainder was for technical assistance and country operations, including operation of the PSC with monthly deliveries to over 620 health facilities nationwide. In FY 2012, USAID provided approximately $17.7 million in funding to JSI-USAID/DELIVER, of which $13.1 million was for procurement of malaria commodities and $4.6 million for technical assistance (TA), country 6 An end-of-project evaluation was conducted by Chemonics International in October 2006. 7 The Strengthening Pharmaceutical Systems (SPS) Program, funded by the U.S. Agency for International Development (USAID) and implemented by Management Sciences for Health (MSH), provided technical support to the implementation of the President’s Malaria Initiative (PMI) and PEPFAR activities in Malawi from 2007 to 2010, complementing activities implemented by USAID/DELIVER. USAID provided $2.6 million through the SPS project during this period to provide technical assistance to the National Malaria Control Program (NMCP), Pharmacy Medicines and Poisons Board (PMPB), Ministry of Health (MOH) Health Technical Support Services Pharmaceutical Division (HTSS-P), and District Health Offices (DHOs), as well as for HIV activities. Final reports from the SPS project are available. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 29 operations, and PSC deliveries.8 In addition, through a gift to the US Government from DFID, USAID/DELIVER will receive a total of $6,771,625 for the warehousing and monthly distribution of UNICEF-procured essential drugs from January 2012 through June 2013 under the multi￾donor Essential Drugs Project. V.4 Available Resources Available resources to inform the evaluation include, but are not limited to the following: Documents  Malawi Country Strategy & Workplan Task Orders 4 & 7, October 2011–September 2012, and SOW  USAID/DELIVER workplan activity tables and related documents  USAID/DELIVER quarterly and annual reports  2006 Deliver I End-of-Project Evaluation  2010 SPS Program End-of-Project Report  USAID/DELIVER-produced program documents and technical reports (2007–13)  Malawi Country Operational Plan 2012 (COP)  Malawi Partnership Framework and Implementation Plan (PF, PFIP)  Malawi Global Health Initiative (GHI) Strategy  MOH Pharmaceutical Strategy  UNICEF–Essential Drugs Project Spot Check Assessment Report, O&M Associates  Essential Drugs Project (EDP) Memorandum and M&E plan–UNICEF, USAID, KFW, Norway, DFID, CHAM, MOH, and CMS 2011  USAID/DELIVER Q1-2 2012 Supervision Report  MOH 2006 Drug Leakage Study  USAID/DELIVER Q1 2012 EUV Report Stakeholders / Informants  Development partners (Health Donor Group members, including UNICEF, CHAI, UNFPA, WHO, DFID, UNAIDS)  USG agencies (USAID, PEPFAR, CDC)  Government and parastatal entities: – MOH: Health Technical Support Services – Pharmaceuticals Division (HTSS-P), Reproductive Health Unit (RHU), National Malaria Control Program (NMCP), HIV Unit – National AIDS Commission – PMPB – CMST  GOM Technical Working Groups (TWG) that focus on commodity issues, such as – Sexual & Reproductive Health TWG, subcommittee on Family Planning – Drugs & Medical Supplies TWG 8 TA funding sources include USAID HIV (PEPFAR), Family Planning, and Maternal & Child Health program funds. 30 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 – HIV TWG – National Malaria Control Program Commodity Management Task Force – Essential Drugs Project coordination group  Health managers and providers (zonal, district, facility level, and community-level Health Surveillance Assistants) VI. SCOPE OF WORK (SOW) Evaluation Fundamentals VI.1 Scope of the Evaluation The overall purpose of this performance evaluation is to assess the effectiveness of the USAID/DELIVER Project and identify lessons-learned to inform USAID/Malawi and the Government of Malawi (GOM) regarding future investments in SCM programming. With Malawi facing significant ongoing challenges in supply chain management (SCM), and with a strong USAID focus on country ownership and strengthening national systems, USAID/Malawi must determine the following: 1. What has worked thus far in terms of support provided through the USAID/DELIVER Project? What gaps exist for SCM technical assistance moving forward, so that both current and future USAID supply chain resources can be most effectively utilized to improve commodity security and strengthen national SCM systems? The primary objective is to analyze the performance of USAID/DELIVER in Malawi from 2007 to 2013 to identify strengths, weaknesses, and lessons learned, as well as draw conclusions about the most significant achievements of the project vis-a-vis USAID/Malawi and GOM objectives around supply chain strengthening and commodity security. A secondary objective is to assess remaining gaps for SCM strengthening against the project’s current program and approach and identify potential future programming approaches and mechanisms. The findings of the evaluation will be used to inform programmatic decision-making regarding priorities, objectives, interventions, and monitoring and evaluation approaches for USAID/DELIVER Project’s current and future annual workplans. In addition, the evaluation will be used to inform USAID/Malawi decision-making regarding future programming in support of improved commodity security and strengthened national SCM systems. VI.2 Evaluation Questions The evaluation questions are focused on addressing the core objectives described in section V. It should be noted that the majority of the evaluation should focus on USAID/DELIVER’s technical assistance and country operations rather than the operation of the PSC for delivery of essential medicines, malaria, and family planning products. In other words, the main focus is on how the project is supporting improved national supply chain systems and local capacity for the longer term, rather than the effectiveness of USAID/DELIVER commodity warehousing and delivery operations. In addition, the primary focus is expected to be on the 2010–12 period. 1. What are the strengths and weaknesses of the USAID/DELIVER program in Malawi for achieving project objectives? USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 31 a. To what extent has USAID/DELIVER’s current program and approach addressed the most pressing SCM gaps and priorities of the GOM and USAID, and what are the strengths and weaknesses of specific interventions? b. To what extent have the project’s approach and interventions reinforced international best practices? c. What were the enabling factors and inhibiting factors that explain the achievement of project results, or the lack thereof? d. Was there any change in USAID/DELIVER performance between the 2007–10 period and the 2011–13 period, and if so, what changed and why? e. To what extent is the performance monitoring and evaluation plan measuring indicators that are appropriate and sufficient to demonstrate the value of the USAID/DELIVER program in Malawi? f. What are the strengths and weaknesses of USAID/DELIVER’s current management, staffing, and institutional configuration in Malawi toward the achievement of project objectives? g. How effectively has USAID/DELIVER employed project resources, human and financial, to achieve project objectives? How well were programmatic resources and approaches linked (malaria, FP, etc.) to address cross-cutting issues? h. To what extent has the project been able to effectively address and overcome major challenges and bottlenecks to successful project implementation, both past and current? To what extent has USAID/DELIVER contributed to improved national capacity for commodity and supply chain management? a. What are the most significant achievements that can be attributed to USAID/DELIVER support with regard to improving commodity security and national systems capacity (separately for malaria, FP, HIV, and cross-cutting issues), and how and why were they achieved? b. What are the priority remaining gaps for strengthening Malawian capacity to improve commodity security, in general and for key product categories supported by USG programming (malaria, FP, HIV)? What options exist for addressing identified gaps for strengthening Malawian capacity to assure commodity security? a. What gaps is USAID/DELIVER well-placed to address? b. What opportunities exist for engaging local partners and better strengthening local systems in line with USAID Forward Implementation and Procurement Reform (IPR) objectives, through current or future programming? VI.3 Evaluation Design and Methods The evaluation team should design the evaluation methods based on scientifically rigorous methodology, and combine a mix of the most appropriate methods, including desk review, interviews, focus groups, data abstraction and analysis from monitoring data, and other methods 32 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 that may be appropriate to answer the agreed-upon evaluation questions. It is expected that the methods will include a sampling approach that allows conclusions to be drawn on differences, similarities, and contextual factors affecting performance at each level of the system (central, zonal, facility, including both CHAM and public hospitals and health centers, and community), and covering each of the country’s five health zones. The evaluation should utilize a mixed methods design in data collection and analysis, using primary and secondary data sources to answer each evaluation question outlined above. Quantitative data will be utilized to quantify USAID/DELIVER’s performance and the extent to which drug supply and commodity security were improved. Qualitative data will be utilized to explain or provide insights on the contextual elements that have facilitated or hindered the project achieving its objectives. Suggested methods will, at a minimum, include (1) primary data collection through questionnaires or interviews of key informants and focus groups; (2) review of relevant USAID/DELIVER program documents; and (3) secondary analysis of USAID/DELIVER’s performance monitoring data. This section outlines some guidance for the study design for each of the three key result areas for this evaluation. The evaluator is expected to expand and improve upon this guidance as necessary. For each result area, the evaluator is requested to propose the evaluation framework and assessment tools for each evaluation question, highlighting the conceptual model(s) adopted. The evaluation framework should incorporate an analysis of the intervention logic of the program; discussion of any risks and limitations that may undermine the reliability and validity of the evaluation results; and specifications of an indicator or indicators that will be used as a guide in answering each question. A detailed design and evaluation plan is required as the first deliverable. The final design and evaluation plan will be reviewed and approved by USAID before the team begins field work. 1. Quantify the performance of the USAID/DELIVER project in supporting improved commodity management and security. The central role of the USAID/DELIVER project has been to support MOH strengthening of the SCM systems at both national and district levels to ensure consistent availability of essential medicines at health facilities throughout Malawi. USAID/DELIVER’s approach has included a combination of technical assistance (TA) and direct implementation. The project has accumulated quantitative data on both its TA and implementation activities (e.g., commodity procurements, distribution and security—including tracking deliveries, leakage [loss/wastage] and stock-outs, training, and supervision). The methodology in this result area may entail triangulation of routine program data to quantify the extent to which project objectives were met and the associated effects on availability and access to essential medicines by health facilities. Data to facilitate this analysis may be sourced from project procurement and distribution records, as well as the LMIS, which is the national database for SCM service data. Key stakeholder interviews and/or focus group discussions may be utilized to collect qualitative information from relevant stakeholders that would provide context for project performance. Identify the key enabling and inhibiting attributes of successful achievement of USAID/DELIVER’s objectives. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 33 Routine project monitoring data indicates varying levels of project achievement in strengthening supply chain systems at the district level and supporting increased leadership/ownership of SCM improvement at national level. Secondary analysis of the USAID/DELIVER Project data may provide the basis for identifying specific areas in which the project has fulfilled/underachieved its targets under this objective. Informant interviews and/or focus group discussions with relevant stakeholders, including project staff, MOH staff—at national and district levels—may provide insights on the enabling or inhibiting factors to project performance in this result area. It is expected that the analysis would distinguish between internal (USAID/DELIVER-specific) and external factors that have facilitated or inhibited achievement of project objectives. Provide evidence to inform decisions about how USAID/DELIVER and USAID/Malawi’s support for strengthening SCM may be improved. For this result area, USAID/Malawi requests that the evaluator identify key findings from the individual evaluation questions to formulate conclusions and recommendations that will inform decisions on how USAID/DELIVER’s approach may be modified to produce improved outcomes; and considerations for USAID/Malawi to design more effective interventions to strengthen SCM in Malawi. In particular, USAID is interested in identifying actionable recommendations required of relevant stakeholders—development partners, MOH–national level, District Health Offices, and health facilities—to strengthen systems and improve leadership and coordination of SCM in Malawi. Data Collection and Analysis USAID requests that the evaluator summarize the evaluation methodology as part of its detailed design and evaluation plan in a matrix, as per Table 1 below.9 The suggested methods and data sources have been outlined in the discussions of each result area above. However, USAID will rely on the expertise of the evaluator to adjust and improve these suggestions and propose a study design that will ensure strong validity, reliability, and interpretive potential of the results. 9 Another format may be used if the table is not preferred, but any format chosen should contain all the information specified for each question. 34 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Table 1. Evaluation Matrix Plan Evaluation Question Indicator(s) Data Collection method(s) Data Source Sampling Comments The evaluation is expected to use existing data to the greatest extent possible augmented by primary data collected through key informant interviews, focus groups and other tools as determined by the evaluator. Anticipated Constraints to Data Collection and Analysis A number of factors could constrain the ability to collect or analyze data.  Language: Though English is the official language for professional communication, some stakeholders, in particular frontline health workers, may be more comfortable communicating their ideas in Chichewa. The evaluator is encouraged to include individuals fluent in Chichewa on the evaluation team, particularly for the key informant interviews and focus group discussions.  Geography and infrastructure: USAID/DELIVER project stakeholders include staff at over 600 health facilities in all districts of Malawi. Even with sampling, the evaluation will require considerable travel throughout the country to reach relevant health facilities (including Lilongwe, Mzimba, Dowa, Blantyre, Thyolo, Zomba, and Machinga) and their personnel. Sample determination may be affected by the fact that some of the health facilities may not be accessible due to the combination of poor roads and heavy rains.  Data quality: there are some known deficiencies in the quality of the LMIS and the USAID/DELIVER project records. These deficiencies have been documented and will be shared with the evaluation team. Sampling of Sites Because USAID/DELIVER project stakeholders include staff at over 600 health facilities in all districts of Malawi, it is likely that multiple sampling frames will be required to produce representative and generalizable results for each level of interest, namely MOH at national level, District Health Offices, health centers, and their respective personnel who will serve as key informants. USAID/Malawi requests that the evaluator submit a process to determine a sampling plan as part of the first deliverable, the evaluation design and evaluation plan. The team is expected to cover selected sites in each of the five health zones. We expect the team to cover at least one of the four central hospitals, a number of district hospitals, health centers, and if possible some community-based programs that are benefiting from USAID/DELIVER Project activities. VI.4. Technical Requirements 1. A full description of the methodology (or methodologies) to answer each evaluation question should be provided by the evaluation team. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides, will be required in an annex to the final report. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 35 The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in regard to technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the USAID technical officer and cleared by the Program Office. Limitations to the evaluation shall be disclosed in the report, with particular attention to limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparison groups, etc.). Sufficient information should be provided so that a reader can make an informed judgment as to the reliability, validity, and generalizability of the findings. Disclosure of conflict of interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest or describing an existing conflict of interest relative to the USAID/DELIVER project being evaluated. All statements will appear in an annex to the final report. Statement of differences: If a difference arises in the interpretation of the results from the various stakeholders, the evaluation report will include a statement identifying any significant unresolved differences of opinion on the part of funders, implementers, or members of the evaluation team. VI.5 Findings: Empirical Facts Collected During the Evaluation 1. Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should have sufficient evidence and documentation that a reader of the findings can be confident that the findings are based on actual data. Evaluation findings should highlight any regional variations or discrepancies (if applicable). Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. VI.6 Conclusions: Interpretations And Judgments Based on the Findings 1. Evaluation conclusions should be presented for each finding based on the evidence collected by the evaluation team. Conclusions should logically follow from the gathered data and findings. Because conclusions involve interpretation of collected data, they should be explicitly justified. If and when necessary, the evaluator should state his/her assumptions, judgments, and value premises so that readers can better understand and assess them. VI.7 Recommendations: Proposed Actions for Management 1. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. Evaluators should take into consideration the economic and political context of the USAID/DELIVER project, the strengths and weaknesses of MOH institutional capacity, and the feasibility of change and innovation while framing recommendations. 36 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 VII. TEAM COMPOSITION, SKILLS, AND LEVEL OF EFFORT (LOE) The evaluation team will be composed of two international experts, one local technical expert, and one local logistics coordinator who jointly can provide expertise in each of the following key skill areas: 1. Evaluation SCM and logistics Capacity-building/national systems strengthening Public health, with expertise in family planning, malaria, and HIV programming in developing countries Administrative assistance Roles of the Team Members Senior Evaluation Specialist/Team Leader. The team leader (TL) will be responsible for overseeing the team and ultimately for submission of the final draft report to the Mission. S/he will provide team leadership, finalize the evaluation design, coordinate activities, plan and coordinate meetings and site visits, and be responsible for payments of local logistical needs and local staff working with the team. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Malawi team and other major stakeholders and will consolidate reports from other evaluation team members and ensure that a draft report has been left with the Mission on departure. The team leader will take specific responsibility for assessing and analyzing the project’s progress toward targets, factors for such performance, and benefits/impact of the strategies and will compare these with other possible options. The TL should have experience in leading teams of international health experts and working with host country personnel, and should have some experience with public health supply chain issues. S/he should have extensive experience in conducting qualitative evaluations/assessments. Excellent oral and written skills are required, as well as experience in preparing high-quality documents. The TL should have an advanced degree in public health or a related field with a minimum of five years of experience in management and evaluation of public health programs, experience in leading teams of experts in health activities, and experience in developing international health programming. International and Local Technical Experts. It expected that two additional technical experts (one local and one international) will jointly cover all the key skill areas mentioned above; expertise covering both malaria and family planning logistics issues is a requirement. Technical experts should have a minimum of five years of relevant experience in their fields and a master’s degree or equivalent experience. One of the consultants should be local (Malawian) to provide context and linkages to the national program. Experience with PEPFAR and Global Fund programs is highly desirable. Local Logistics Company. The team will be assisted in the field by a local logistics company that will assist the team in managing domestic travel around Malawi, meeting space arrangements, other administrative tasks, and translation as necessary. The evaluator is USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 37 encouraged to include individuals fluent in Chichewa on the evaluation team, particularly for the key informant interviews and focus group discussions. USAID/Malawi M&E specialists will work with the evaluation team as part of the Agency’s efforts to strengthen the Agency's learning from its own experience and work with the successful evaluator under the direction of the TL in  Instrument development and piloting,  Data collection,  Data analysis, and  Synthesis of results.  CVs for the USAID/Malawi M&E specialists will be available upon request. Personnel from the USAID/DELIVER Project will work with the evaluator by providing key documents and insights. The evaluator should not contact the USAID/DELIVER Project directly but coordinate all requests for information through USAID/Malawi. In addition, USAID/Malawi encourages the involvement of representatives from the MOH in the evaluation process. Consultant Conflict of Interest (COI) To avoid conflicts of interest or the appearance of a COI, offers should provide a list of previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors or NGOs evaluated/assessed and information regarding their affiliates. Level of Effort An illustrative table of the LOE is found below. Dates may be modified based on availability of consultants and key stakeholders and the amount of time needed for field work. Activity Team Leader Int’l Specialist Local Specialist POP (illustrative depending on start date) Background Reading/Preparation 5 5 5 11/1 – 11/7 Total Travel Days 7 7 5 N/A Team Planning Meeting/Methodology Planning 3 2 2 11/8-11/13 Fieldwork 12 12 12 11/14-11/25 Data Analysis/Report Writing 14 11 11 11/28-1/13/14 Workshop/Briefings 3 3 3 12/10 Total LOE 41 38 36 10/15 – 1/13/14 *A six-day work week is approved only while in the field to accommodate travel/work days. VIII. LOGISTICS GH Tech will be responsible for all international travel and consultant logistics. 38 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 A six-day work week while working in country is authorized. Local holidays are not authorized. The team (primarily the local logistics coordinator) will be responsible for arranging and scheduling meetings, in-country travel (including vehicle rentals), hotel bookings, meeting space for the findings meeting, facilitating printing, and photocopying when appropriate. A local logistics coordinator may be hired to arrange field visits, local travel, hotels, and appointments with stakeholders. The USAID point of contact for the project will arrange for an initial introductory meeting with appropriate staff at the MOH and JSI. A list of relevant stakeholders and key partners will be provided to the consultant team by the point of contact prior to arrival but the evaluation team will be responsible for expanding this list as appropriate and arranging the meetings. The point of contact and other Mission personnel will be available to the team for consultations regarding technical issues, both before and during the evaluation. IX. DELIVERABLES AND PRODUCTS The team will prepare the following deliverables; all deliverables will require final approval by USAID/Washington. IX.1 Detailed Evaluation Design and Plan10 Through the Team Planning Meeting prior to the commencement of fieldwork, the consultant team will generate in writing a detailed evaluation plan that states the objectives of the evaluation, the questions that will be answered, sampling frameworks, specific methods and instruments to answer each question, sample data analysis tables, and timelines with data collection start and end dates. The following will be important considerations in the detailed evaluation plan: 1. Types of information needed. The types of data needed above and beyond data to completely answer the evaluation questions and identify any limitations of the existing data in terms of effectively answering the evaluation questions. Sources of information. The detailed evaluation plan should identify how existing data will be incorporated and what additional information will be required to accurately and sufficiently answer the evaluation questions. The sources of information that will be used in the evaluation should be described in enough detail that a reader can be confident that the information will be sufficient to meet the evaluation's purposes, given the scope, context, and resources available for the evaluation. The evaluation plan should also discuss how confidentiality of information will be maintained. A sample consent form for all primary data collection should be attached as an appendix and referenced in this section. The consent form should include a description of the evaluation objectives and how the information will be used. Criteria for sampling and selecting participants. Coverage of the USAID/DELIVER project is national—28 districts and their constituent primary health facilities. There are significant logistic and financial barriers to including all health facilities in the evaluation. The consultant team should identify the sampling methods that will be used to answer the evaluation 10 U.S. Department of Housing and Urban Development, Office of Policy Development and Research. A Guide to Evaluating Crime Control of Programs in Public Housing. Washington, DC: Prepared for the U.S. Department of Housing and Urban Development by KRA Corporation; 1997. pp.5.1-5.15. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 39 questions and to articulate any limitations that the method will have on ability to generalize from the findings, conclusions, and recommendations. Methods for collecting information. For each evaluation question, the evaluation plan should specify the methods by which information will be collected (for example: questionnaires, surveys, observations) and the procedures to collect the data. The plan should describe the proposed methodological approaches and how, within the constraints of time and cost, they will yield data that help answer the evaluation questions. The proposed approaches should be grounded in respected methodological frameworks and best-practice literature. Significant or important constraints on the study design should also be identified. Instruments. The evaluation plan should include all instruments that will be used to collect data to answer the evaluation questions, with descriptions of how the instruments were or will be piloted and used. Timeframe for collecting information. The evaluation plan should include a detailed timeframe for the evaluation, including instrument development, fieldwork, and data analysis. Methods for analyzing information. The evaluation plan should detail the practices and procedures that the consultant team will use to analyze the data to answer the evaluation questions. For each evaluation question, at least one blank analysis table graphically displaying the data analysis outputs for each question should be presented. IX.2 Periodic Briefings and Reports The evaluator will provide a progress briefing halfway through the field data collection. IX.3 Oral Briefing of Findings Conclusions and recommendations for each evaluation question shall be provided by the consultant team to USAID/Malawi prior to drafting the evaluation report. The oral briefing should be presented as a PowerPoint presentation. IX.4 Draft Report The evaluator shall provide a draft report to USAID/Malawi (5 hard copies and a soft copy) prior to departure from Malawi. IX.5 Findings Workshop After incorporating comments from USAID into the draft report, the evaluator shall hold a half￾day workshop to present key findings, conclusions, and recommendations. The workshop shall attract at least 30 strategic stakeholders and shall be held in Lilongwe. The evaluators shall be responsible for costs, meeting venue logistics, reimbursement of participants from outside of Lilongwe, and managing invitations to this workshop (with the assistance of an attendee list and initial introductions by USAID/Malawi). The evaluator shall produce a briefer (max. 3 pages) of key findings, conclusions, and recommendations to be distributed to stakeholders. IX.6 Final Report GH Tech shall submit 50 hard copies and 1 electronic copy of the USAID-approved final report. The report shall be in English; should not exceed 30 pages, excluding relevant annexes, (e.g. SOW, interview transcripts/notes, photos and success stories); and shall include matrices and other visuals to consolidate and summarize data. Upon completion the evaluator will submit a 40 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 flash drive comprising all electronic products of the evaluation, including instruments and data in formats suitable for replication of the analysis, final report, and the briefer. X. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment:  Recruit and hire the evaluation team.  Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before Field Work  SOW. Respond to queries about the SOW and/or the assignment at large.  Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants and provide additional information regarding potential COIs with project contractors evaluated/ assessed and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment.  Local Consultants. Assist with identification of potential local consultants, including contact information.  Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line item costs.  Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation). During Field Work  Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.  Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space if available, or other known office/hotel meeting space).  Facilitate Contact with Implementing Partners. Introduce the evaluation team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings.  Identify invitees for findings workshop. USAID will work with the team to identify the 30 participants to be invited to the findings workshop. After Field Work  Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 41 XI. MISSION CONTACT PERSON Chimwemwe Chitsulo M&E Specialist USAID/Malawi cchitsulo@usaid.gov XII. COST ESTIMATE GH Tech will provide a cost estimate for this activity. 42 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 43 ANNEX II. SELECTED PERFORMANCE INDICATORS MATRIX Business Processes Types of Indicators Quality Cost/Financial Productivity Warehousing/ storage Inventory Accuracy Rate Value of Product Damaged in the Warehouse Storage Space Utilization Inventory management/LMIS/ customer response Stock-out Rate Value of Damage/Losses Ratio Facility Reporting Rates Forecasting and quantification Forecast Accuracy % of Average International Reference Price Paid % of Purchase Orders/Contracts Issued As Emergency Orders 44 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 45 ANNEX III. REPRESENTATIVES OF DONOR AGENCIES, GOVERNMENTAL OFFICIALS, HEALTH OFFICERS AND HEALTH WORKERS INTERVIEWED ON THE CENTRAL AND DISTRICT LEVELS Category Represented A Health Donor Group 1 Dr. Patrick Ruldoph KfW Director 2 Hildegunn Tobiessan Norway First Secretary 3 Samuel Chirwa UNICEF PHC Specialist 4 Newton Chagoma CHAI Program Officer- Access to Medicines 5 Sean Donato CHAI Program Manager-Access to Medicines 6 Jean Mwandira UNFPA RH Officer 7 Lamulo Nsaja Kfw 8 Miriam Lutz USAID Office Chief 9 Lily Banda USAID Deputy Office Chief 10 Amy Diallo USAID Health Systems TL 11 Ritu Singh USAID HIV/Aid TL 12 Ruth Madison USAID Family Health TL 13 Monica Olewe USAID Snr Malaria Advisor (by telephone) 14 Veronica Chirwa USAID FP/RH Specialist 15 Chimwemwe Chitsulo USAID M&E 16 Ugbede-Ojo Abu USAID Supply Chain Advisor B Government and Parastatal Entities 17 Dr. Kelita Kamoto MOH/HTSS Director 18 Albert Khuwi MOH/HTSS-P Deputy Director 19 Charles Chimenya MOH/HTSS-P Logistics Pharmacist 20 Flora Khalimba MOH/NMCP/HTSS Commodity Logistics Focal Person 21 Fannie Kachale MOH/RHU Director 22 Jean Malambu MOH/RHU National Coordinator 46 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Category Represented 23 Dorica Chirwa MOH/HIV Dept/HTSS Commodity Logistics Focal Person 24 Chimango Jere National AIDs Commission Head of Procurement 25 Fredrick Mzoma CMST Ag CEO 26 Steward Letapa CMST 27 Joe Kharani CMST Pharmacist in Charge 28 Rex Kuyeli CMST Logistics Officer 29 Peter Mellon CMST Pharmacovigilance and Procurement 30 Austine Omiunu MoH/HTSS-P Supply Chain Advisor 31 David Bagonza MoH/HTSS-P Supply Chain Advisor 32 Moses Chisale CMST Director of Pharmaceutical Operations 33 Mari Piri MoH/RHU Principal RH Officer in FP and Gender C GOM Technical Working Group 34 Caroline K.Ntale MOH/HAD/DMS Logistics Technical Assistant D Other Central Level Stakeholders 35 Phillip Kamutenga USAID/DELIVER COP 36 Dirk Van Wick IHS Director, Malawi 37 Flemings Kapunda CML Director 38 Maureen CML Executive 39 Michael Edwards CML Logistics Management E Persons Met, Field Visit 40 Mr. Kulemera CMST-South Pharmacist in Charge 41 Dr. Malangizo Mbewe South-West Health Zone Office Zonal Officer 42 Nelson Nanchinga Thyolo District Hospital Pharmacist in Charge 43 Enock Tsabola Khonjeni Health Center, Thyolo District Pharmacist in Charge 44 Christopher Nyadani Thekerani Community Hospital, Thyolo District Medical Assistant in Charge USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 47 Category Represented 45 Cecilia Manduza Chipo Health Center (CHAM), Thyolo District Nurse-Midwife 46 Norbert Sumza Chipo Health Center (CHAM), Thyolo District Nurse 47 Kingsley Kapida Machinga District Hospital Pharmacy Technician 48 Willard Rubeni Ntaja Health Center, Machinga District Medical Assistant in Charge 50 Alfred Mdoka Nainunje Health Center, Machinga District HSA-Drug Store Clerk 51 Thokozani Umari Mpiri Health Center (CHAM), Machinga District Nurse in Charge 52 Dr. McDonald Msadala South-Eastern Health Zone Office Zonal Officer 53 Bernadette Chibwana Central Hospital, Zomba Pharmacist 54 Rachel Mwinjiro Mpherembe Health Center, MzimbaNorth District Nurse in Charge 55 Collins Kabogodo Mpherembe Health Center, MzimbaNorth District HSA 56 Kondiwani Kandiero Mpherembe Health Center, MzimbaNorth District Medical Assistant 57 Obrein Mhone Mtwalo Health Center, Mzimba North District Senior HSA 58 Memoria Simfukwe Bwangu Health Center, Mzimba North District HSA-Drugs Clerk 59 Chawezi Mtonga Bwangu Health Center, Mzimba North District Medical Assistant in Charge 60 Agness Hara Ekwandene Community Hospital (CHAM) Community Health Nurse 61 Dr. Collins Mitambo Mzimba North District Health Office, North Zone District Medical Officer 48 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Category Represented 62 Innocent Issa CMST-North Branch Manager 63 Tadala Hamisi Kamuzu Central Hospital, Lilongwe Pharmacist in Charge 64 Mary Ngalanda Namitondo St. Gabriel Community Hospital (CHAM), Lilongwe District Nurse-Midwife in Charge of the Pharmacy 65 Lyton Chimosola Namitondo St. Gabriel Community Hospital (CHAM), Lilongwe District Stock Supervisor 66 Hanifa Likaka Mitundu Community Hospital, Lilongwe District Nurse in Charge of the Pharmacy 67 Labina Kudala Chilima Mthenthela Health Center, Lilongwe District Medical Assistant, Acting in Charge 68 Christopher Fosco Mthenthela Health Center, Lilongwe District HSA-Drugs Clerk 69 Catherine Milanzi Natenje Health Center, Lilongwe District HSA-Drugs Store Clerk 70 Dr. Peter Chaziya Dowa District Health Office District Medical Officer 71 Agness Mpanane'ombe Francisco Palau Community Hospital (CHAM), Dowa District Matron 72 Maria Cruz Francisco Palau Community Hospital (CHAM), Dowa District Responsible for Medical Supply 73 Mr. Seda Chakhaza Health Center, Dowa District Acting in Charge 74 Mr. Banda Chakhaza Health Center, Dowa District HSA 75 Susan Kamada Lulo Communitty Hospital, Mponela, Dowa District Nurse in Charge of the Pharmacy 76 Zeris Kassiya Kasasa Health Center, Dowa District Pharmacy Attendant USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 49 ANNEX IV. HEALTH FACILITIES VISITED BY THE EVALUATION TEAM No. Name 1 Bwangu Health Center, Mzimba North District 2 Central Hospital, Zomba 3 Chakhaza Health Center, Dowa District 4 Chipo Health Center (CHAM), Thyolo District 5 Dowa District Hospital 6 Ekwandene Community Hospital (CHAM), Mzimba North District 7 Francisco Palau Community Hospital (CHAM), Dowa District 8 Kamuzu Central Hospital, Lilongwe 9 Kasasa Health Center, Dowa District 10 Khonjeni Health Center, Thyolo District 11 Lulo Communitty Hospital, Mponela, Dowa District 12 Machinga District Hospital 13 Majani Health Center, Dowa District 14 Mitundu Community Hospital, Lilongwe District 15 Mpherembe Health Center, MzimbaNorth District 16 Mpiri Health Center (CHAM), Machinga District 17 Mthenthela Health Center, Lilongwe District 18 Mtwalo Health Center, Mzimba North District 19 Mzuzu Health Center, Mzimba North District 20 Nainunje Health Center, Machinga District 21 Namitondo St. Gabriel Community Hospital (CHAM), Lilongwe District 22 Natenje Health Center, Lilongwe District 23 Ntaja Health Center, Machinga District 24 Thekerani Community Hospital, Thyolo District 25 Thyolo District Hospital Summary, health facilities visited: Central hospitals 2 District hospitals 3 Community hospitals 6 Health centers 14 Including CHAM health facilities 5 50 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 51 ANNEX V. DATA COLLECTION TOOL, QUESTIONNAIRE FOR KEY INFORMANT INTERVIEWS 1. TRAININGS/MENTORING 1.1. How many professional staff work at your facility? 1.2. How many of them were ever trained by USAID/DELIVER for SCM? 1.3. Could you identify the major topics of the trainings attended? 1.4. Were post-training tests ever conducted to assess the quality of acquired knowledge/skills? 1.5. What mode of assessment was applied, if any? 1.6. What assessment mark or similar have your staffs obtained according to the mode applied? 1.7. Do you have any certificates confirming passing the trainings successfully and the assessment mark attained? 1.8. Are you satisfied with the technical trainings you attended? 1.9. What is the most useful knowledge/skill you have acquired from the training? 1.10. How do you apply the acquired knowledge/skills in your routine job? 2. COOPERATION/PERFORMANCE 2.1. Forecasting/quantification 2.1.1. How many forecasting exercises did you conduct for the period ____________? 2.1.2. How many of them were conducted jointly with JSI? 2.1.3. Who normally does the quantification/forecasting exercise before a procurement process is initiated? 2.1.4. How do you assess the forecasting/quantification process at your supply chain: very poor, requiring significant improvements, good, very good? 2.1.5. Can you justify your opinion with a practical example? 2.1.6. Do you know the principles of needs forecasting for the health supply? 2.1.7. Where did you acquire your knowledge about forecasting and quantification? 2.1.8. How do you see the way forward in terms of improvement of forecasting and quantification? 2.2. Storage/Inventory 2.2.1. How many inventories were conducted for the period (2007–13) at your medical store? 52 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 2.2.2. How many inventories with any physical discrepancy discovered took place during the same period at your medical store? 2.2.3. How do you deal with discrepancies found? 2.2.4. Do you have damaged, expired, or items unaccounted for in your medical store? 2.2.5. How do you report and further deal with these items? 2.2.6. Do you know the value of these items? Is it documented? 2.2.7. How often do you report on existing stock-outs, if any? 2.2.8. How fast are reported stock-outs eliminated after you report them? 2.2.9. Do you have any commodities the stock of which significantly exceeds the maximum level? 2.2.10. Is there any mechanism of redistribution in place for surplus stock? 2.3. Distribution/Transportation 2.3.1. How often do you receive supplies from the CMS? 2.3.2. How are you notified on the date/time of the arrival of trucks? 2.3.3. How many delayed arrivals have you documented? 2.3.4. What were the reasons for the delays? 2.3.5. Who accompanies loads delivered for proper handover purposes? 2.3.6. How would you assess the technical condition of the trucks in terms of load safety (protection from water, dust, security, etc.)? 2.3.7. Where do trucks stay overnight if they arrive too late to be off-loaded upon arrival? 2.3.8. Who does off-load the arriving trucks? 2.3.9. Do you face any delays with off-loading and what are the reasons for that? 2.3.10. How often do you receive supplies that are evidently damaged from being transported? 2.3.11. How do you report on items damaged in transit, and how do you deal with them further? 2.3.12. How many reports on discrepancies discovered when taking over a new load did you submit to your supplier for the period…? 2.3.13. How do you plan distribution from your medical store to the SDPs? 2.3.14. How do you organize transportation of supplies from your medical store to consignees? USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 53 2.3.15. How many SDPs do you have in your region? 2.3.16. In what way do you conduct distribution to them: as a planned campaign, on a regular basis, or upon request received? 2.3.17. What happens if an SDP is closed when the truck has delivered the supplies to it? 2.3.18. Do you maintain communication with the truck’s driver or forwarder clerk while the load is in transit? 2.3.19. How do you assess the distribution/transportation process: very poor, requiring significant improvements, good, very good? 2.3.20. Could you identify gaps and/or major challenges in the distribution/transportation process? 2.3.21. How do you see the way forward in terms of improvement of distribution/ transportation? 2.4. LMIS/Accountability 2.4.1. What e-LMIS do you have in use at your medical store? 2.4.2. How many staff are trained on its use? 2.4.3. Who provided training in the use of LMIS? 2.4.4. Is your LMIS fully used in terms of its available features and options? 2.4.5. What are the reasons preventing usage of your LMIS at full volume? 2.4.6. What major constraints does your LMIS have? 2.4.7. What is the facility reporting rate in your region for the period…? 2.4.8. What are the main reasons of not submitting reports on time? 2.4.9. Are all paper reporting forms you receive transferred into your e-LMIS? 2.4.10. Is your LMIS able to consolidate consumption data on all SDPs in your region? Please demonstrate. 2.4.11. How do you transfer consolidated data to the upper level of the supply chain? 2.4.12. Does your system follow up the value of the stored items, including what is unusable? 2.4.13. What is the total value of all damaged items stored in your medical store today? 2.4.14. What is the total value of all expired items stored in your medical store today? 2.4.15. Please demonstrate all available reports generated by your LMIS that are actual today. 54 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 2.4.16. How do you estimate your e-LMIS: very poor, requiring significant improvements, good, very good? 2.4.17. Could you identify the gaps and/or major challenges of your LMIS? 2.4.18. How do you see the way forward in terms of improvement of accountability? 3. MANAGEMENT/ADMINISTRATION 3.1. Please, give us a breakdown of your working time: % of time spent for performance of certain responsibilities. 3.2. Do you consider you have an exceedingly high workload? 3.3. Are you able to fulfill all your job responsibilities on time? 3.4. Are you satisfied with your job? 3.5. Are there any disciplinary measures in place for SDP staff responsible for reporting if they fail to report on time? USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 55 ANNEX VI. EVALUATION METHODOLOGY SUMMARY Evaluation Question Indicator Data Collection Methods Data Source Comments 1 Technical staff trained and able to undertake forecasting and use LMIS Key informant interviews, secondary data review Health personnel engaged in SCM, PLP and annual work plans Data sources and interviews to ascertain USAID/DELIVER performance up to 2011 may not be quantifiable as the team is reliant on sources of data only available for and people working during that period. 2 Stock-out rates, forecasting and inventory accuracy, facility reporting rate, emergency orders rate, damage or losses rate. Secondary data from e-LMIS reports e-LMIS Supply Chain Manager Facility data triangulated with central records 3 Stock-out, MOS, SOP* Key informants interviews Health personnel engaged in SCM; DHMT * Interviewees were questioned on how they follow standard operation procedures for supply chain management (see Annex II) 56 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 57 ANNEX VII. FACILITY SUPPLY STATUS BASED ON LMIS REPORT, OCTOBER 2013 Malaria Product Out of Stock Stock on Target Over 6 Months of Supply LA 1x6 3% 22% 48% LA 2x6 9% 27% 31% LA 3x6 10% 23% 37% LA 4x6 9% 25% 32% Malaria RDT 7% 21% 37% Quinine dihydrochloride 300 mg/ml, 2ml 2% 14% 64% Quinine hydrochloride 300 mg 22% 23% 41% Sulphadoxine 500 mg/ pyrimethamine 25 mg (SP) 4% 11% 74% 58 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 59 ANNEX VIII. HIV/LAB TESTS OP INDICATOR STATUS11 AND EVALUATION TEAM COMMENTS Indicator Target Achieved (Jan-Mar 2011) Comments Number of people trained in laboratory, HIV test and Malaria RDT Logistics System. 136 129 The final group of health workers were trained this quarter Number of improvements to laws, policies, regulations, or guidelines related to improved and use of health services drafted with USG support – 1 Malawi Laboratory Standardization Guidelines Number of SDPs reporting stock-outs of any HIV test offered by the SDP at any time during the reporting period. – 246 HIV tests only (total number of SDPs=?, representing a x% stock-out rate) Number of RMS stocked-out on either screening or confirmatory HIV test – 1 (out of 3) Southern Region RMS % LMIS reporting rate – 47.4% For HIV tests only, February 2011 1. “Number of people trained…“is an input indicator; reportedly, 129 persons attended training, but it is not clear if they all successfully passed post-training tests or whether USAID/DELIVER considers a trained person once he/she sat the course with no follow up. If that were stated in the Comments column, this achievement would be an obvious output. 2. “Number of improvements to laws…” is not defined correctly since measurement of ‘improvements to laws’ will not yield any meaningful results that could be identified as outputs benefiting supply chain management. USAID/DELIVER also has zero control over changes to laws, which are done by governments. It is not clear from the comments, what specific improvements were made, whether the MOH adopted them or they were just suggestions; or how significant expected impact of these improvements is. Not only is this information missing in the report, it is not clear why this is required when important indicators like inventory accuracy rate or emergency orders are not being measured. 3. For “Number of SDPs reporting stock-outs…” the number given, 246, does not provide any information about the scale of stock-out rates. To appropriately follow best M&E practice, it needs a numerator (the top number of a common fraction which indicates the number of parts from the whole that are included in the calculation, the denominator being the bottom number, which indicates the number of parts in the whole.12 Stock-outs should be expressed as a percentage and the value of the same indicator for the previous period provided for 11 Table from original report “USAID/DELIVER PROJECT, Malawi Field Office Quarterly Report Task Order 4, January–March, 2011”. [Original spelling in the table retained] 12 Hales D et. Al. UNAIDS: An introduction to indicators. Geneva 2010. 60 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 comparison, with the difference between them estimated and the reason for any change (or lack of expected change) explained in the comments. 4. “Number of RMS stocked out…” is irrelevant here: USAID/DELIVER does not supply RMS and therefore cannot be responsible for stock-outs, cannot influence them, and so should not be following this indicator. 5. “% LMIS reporting rate” apply only to HIV tests and only to February 2011, though the report is quarterly. 6. However, in the same report, USAID/DELIVER presents a detailed dynamic chart on stock￾outs of family planning commodities, demonstrating monthly fluctuations for the period from October 2010 through February 2011. The source of the data is Supply Chain Manager. Here it is not clear what this indicator actually expresses. USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 61 ANNEX IX. USAID/DELIVER PROJECT ORGANIZATIONAL CHART 62 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 63 ANNEX X. SECONDARY DATA PRESENTED BY USAID/DELIVER TO THE EVALUATION TEAM Figure 1. Average Reporting Rates for Health Facilities, Malawi, 2007–13 (Percent)* * Data are missing for September and October 2008. There is a progressive trend in improved reporting rate demonstrated in the chart since January 2013 (reporting rate 48%), with the only exceptional drop in June. The best reporting rate was in September – 86%, and slightly decreased in October – 83%. Figure 2. Average Stock-out Rate (%) on all Four Presentations of Coartem, Malawi, 2008–13* *The two peaks in the middle of the figure are explained when CMS failed to make deliveries (August– November 2010), when stock-out rates increased from 14% to 39%, and there was a fuel crisis afflicting Malawi (April–June, 2011). The situation improved. though with fluctuations. Stock-outs are now kept at a low level—1% in October 2013. 64 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 65 ANNEX XI. SECONDARY DATA DRAWN FROM E-LMIS (SUPPLY CHAIN MANAGER) AVAILABLE DURING FIELD VISITS Table XI-1. Stock-outs in Thyolo District, Six Months of 2010 (marked with “X”) Product JUL AUG SEP OCT NOV DEC Male condom No data No data No data Female condom IUCD X X X Jadelle Implanon X X X CycleBead X X X Depoprovera Microgynon Microlute Emergency pills LA 1x6 No data No data No data LA 2x6 LA 3x6 X X X LA 4x6 HIV Test Kit No data USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 66 Table XI-2. Stock-outs in Thyolo District, 2011–12 (days of stock-out) 2011 2012 Product 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 Male condom ND ND ND ND 20 ND ND ND ND ND ND Female condom ND ND ND IUCD ND ND ND ND ND ND ND ND Jadelle Implanon ND ND ND ND ND Cyclebead ND ND Depoprovera 22 Microgynon ND ND ND ND ND ND 31 Microlute ND ND ND ND ND ND ND ND ND ND Emergency pills ND ND LA 1x6 ND 20 ND 3 13 20 LA 2x6 3 3 LA 3x6 LA 4x6 d4T+3TC+NVP No data ND No data ND ND Abacavir 1 ND Nevirapine ND 13 Lopinavir 22 31 Efavirenz, 200 ND Stavudine d4T 10 Tenofovir/Lamiv. ND ND ND Stavudine/Lamiv. 31 Efavirenz, 600 31 Zovidine ND Atazanavir USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 67 Table XI-3. Stock-outs in Thyolo District, 2013 (days of stock-out) 2013 Product Jan Feb Mar Apr May Jun Jul Aug Sep Oct Male condom ND Female condom 28 IUCD 28 30 Jadelle Implanon ND Cyclebead Depoprovera Microgynon ND 30 ND ND Microlute 30 Emergency pills 17 ND LA 1x6 LA 2x6 LA 3x6 LA 4x6 d4T+3TC+N VP ND ND ND 30 ND ND ND Abacavir 30 4 ND ND Nevirapine ND 24 Lopinavir 30 30 31 Efavirenz, 200 ND ND 31 Stavudine d4T ND Tenofovir/ Lamiv. 3 Stavudine/ Lamiv. ? 30 31 1 Efavirenz, 600 ? 30 31 30 Zovidine ? ND ND ND Atazanavir ? 30 31 ND ND 68 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 Table XI-4. Average Health Facility Reporting Rates, Thyolo District, 2010–13 (Percent) 2010 2011 2012 2013 JAN ND 14 95 100 FEB ND 18 100 100 MAR ND 14 100 95 APR ND 32 100 100 MAY ND 91 100 95 JUN ND 100 100 95 JUL 91 100 100 100 AUG 91 100 100 100 SEP 95 100 100 100 OCT 18 95 95 100 NOV 14 100 95 DEC 14 91 100 Table XI-5. Average Health Facility Reporting Rates, Machinga District, 2007–13 (Percent) 2007 2008 2009 2010 2011 2012 2013 JAN 0 62 48 86 67 67 86 FEB 0 62 76 86 67 86 81 MAR 0 24 86 81 81 71 81 APR 0 67 71 81 81 52 86 MAY 0 57 62 71 76 71 86 JUN 0 67 86 71 81 71 86 JUL 0 90 71 76 76 81 81 AUG 0 86 76 71 67 62 76 SEP 0 95 71 71 76 76 95 OCT 0 95 76 81 76 71 86 NOV 0 90 71 81 76 76 DEC 0 90 81 76 76 62 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 69 ANNEX XII. MATRIX OF FIELD FINDINGS 70 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 71 ANNEX XIII. MONTHLY LMIS FORM 72 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 73 ANNEX XIV. PROPOSED MODIFICATION OF LMIS FORM FOR SERVICE DELIVERY POINTS Item ID1 Description2 Unit3 Opening Balance4 Received5 Dispensed6 Losses/ Adjustments7 Closing Balance8 Requested9 00015 Artemether Lumefantrine (Coartem), Tabl., 20 mg/120 mg piece 48 2,400 1,920 - 8 520 5,240 1 Item ID— unique code or number of the commodity item for identification. 2 Description— detailed name of the medicine (commodity) with indication of its characteristics: medicinal form, dosage, strength, etc.; or purpose, material, sizes, etc., for other commodities. 3 Unit — the smallest undivided unit of accounting, which is used for distribution (dispensing) to the final user (patient). 4 Opening Balance — stock on the beginning of the first day of the reporting period; must correspond to the stock at the end of the last day of the previous period. 5 Received — quantities of units received from the suppliers, confirmed with accompanying documents (receipts, delivery notes, waybills, etc.). 6 Dispensed— quantities of units given out at the SDP to the final users (patients); should be confirmed with records, e.g., Daily Dispensing Register. 7 Losses/Adjustments— findings of regular inventories (stocktakes) requiring corrections of stock available. Losses are the quantities of irretrievably lost units due to expiration, damage, or theft; entered as negative values. Adjustments are positive corrections of the stock, recording surplus quantities discovered, which means that miscounting was admitted in previous stocktaking. All losses and adjustments must be explained and documented. 8 Closing Balance— stock on the end of the last day of the reporting period; determined using the formula: Opening Balance plus Received, minus Dispensed, plus Losses/Adjustments. 9 Requested— quantities of units requested from supplier to refill stock for future periods; determined using the formula: average Dispensed multiplied by established maximum MOS, minus Closing Balance, plus or minus justified correction depending on seasonl, epidemiological situation, if necessary. 74 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 75 ANNEX XV. DISCLOSURE OF CONFLICT OF INTEREST Name Beatriz Ayala-Öström Title Independent Procurement and Supply Chain Management Consultant Organization Self-employed Evaluation Position? x Team Leader Team member Evaluation Award Number (contract or other instrument) B4-015 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4.Current or previous work X Xx 76 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 16 December 2013 Name Igor Novykov Title Independent Procurement and Supply Chain Management Consultant Organization Self-employed Evaluation Position? x Team Leader Team member Evaluation Award Number (contract or other instrument) B4-015 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of interest to disclose. Yes No X X USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 77 If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7.Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8.Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9.Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10.Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11.Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12.Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 16 December 2013 78 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 79 ANNEX XVI. REFERENCES Aronovich, Dana, Marie Tien, Ethan Collins, Adriano Sommerlatte, and Linda Allain. 2010. Measuring Supply Chain Performance: Guide to Key Performance Indicators for Public Health Managers. Arlington, Va.: USAID | USAID/DELIVER PROJECT, Task Order 1. Berger, Rene, Noreen Teoh, and Thuguri Mungai. 2006. Malawi USAID/DELIVER. End of Project Evaluation. Washington, DC: Chemonics International. CHIP/Maternal and Child Health Integrated Program, USAID. 2012. National Sexual and Reproductive Health and Rights Strategy (2011–16). Lilongwe, Malawi: Ministry of Health. Clinton Health Access Initiative. 2013. Central Medical Stores Trust (CMST) Warehouse and Distribution Rapid Assessment. Lilongwe. Eichler, Rena, Alex Ergo, Brian Serumaga, James Rosen, Greg Miles, and Mavere Tukai. 2012. Options Guide: Performance-Based Incentives to Strengthen Public Health Supply Chains – Version 1. Bethesda, MD: Health Systems 20/20 project, Abt Associates Inc. Hales D. et al. 2010. An Introduction to Indicators, Geneva: UNAIDS http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/8_2-Intro-to￾IndicatorsFMEF.pdf. Accessed December 16, 2013. Joint Strategy. 2012. Multi Stakeholder Planning Mission, August, Joint Strategy for Supply Chain Integration in Malawi. Lilongwe: Ministry of Health CMS Trust, USAID, The Global Fund, UNICEF, CHAI, UNFPA. JSI/USAID/DELIVER. 2008. Annual Indicator Report, October. (Also available: quarterly reports for task orders 4 and 7 for 2010, 2011, 2012 and 2013.) Lewis, David. 2011. Malawi Health and HIV Procurement and Supply Management Assessment. London, UK : DFID Human Development Resource Centre. Ministry of Health of Malawi. 2011 National Quantification and Supply Planning Report. Lilongwe:: MOH. Nassler, D. 2011. Procurement Oversight for Health Sector. Support to Central Medical Stores Trust – Study on Strategic Development Support. Germany. EPOS Health Management. Program for Appropriate Technology in Health. 2010. Collaborative Requirements Development Methodology and Common Requirements for Logistics Management Information Systems. Seattle, WA: Author. Reproductive Health Supply Coalition. 2009. Optimizing the Reproductive Health Supply Chain. A User’s Guide to Software. Maryland USA. USAID | USAID/DELIVER PROJECT, Task Order 4. 2012. Technical Assistance for Supply Chain Strengthening and Commodity Security in Public Health: Annual Report. October 2011–September 2012. Washington, DC: Author. 80 USAID | DELIVER PROJECT PERFORMANCE EVALUATION MALAWI 2007–2013 USAID/DELIVER PROJECT, Task Orders 4 and 7. 2013. Malawi Country Strategy and Work Plan. October 2012 to September 2013. Washington, DC: Author. USAID/DELIVER. 2013. “FY 2013 PMP Matrix, Final.” Washington, DC: Author. Weber, W., A. O’Connell, B. Ayala-Öström, P. Rockhold, J. Gruber, and U. Enemark. 2010. Final Evaluation of the Health Sector Programme of Work (2006–10). Malawi, Copenhagen: Euro Health Group for Ministry of Health SWAP Secretariat. Wright, Christopher, David Papworth, and Mattias Wiklund. 2013. Malawi: Assessment of the Integrated Logistics Management Information System: Review of the Processes and Software Tools. Lilongwe: USAID/DELIVER. For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge IV Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com