THE GLOBAL HEALTH SUPPLY CHAIN - PROCUREMENT AND SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW February 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Constance Carrino, Tim A. Clary, Lisa Harrington, and George Ruhago. THE GLOBAL HEALTH SUPPLY CHAIN - PROCUREMENT AND SUPPLY MANAGEMENT PROJECT MID-TERM REVIEW FOR LESSONS LEARNED AND THE WAY FORWARD February 2020 USAID Contract No. AID-OAA-C-14-00067; Evaluation Assignment Number: 679 USAID Contract No. 7200AA19M00019; Assignment Number: 050 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information is available from: Global Health Technical Assistance and Mission Support Project 1319 F Street NW, Suite 204 Washington, DC 20004 Phone: (1-202) 991-8600 This document was submitted by GH-TAMS to the United States Agency for International Development under USAID Contract No. 7200AA19M00019. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / iii ABSTRACT The Global Health Supply Chain – Procurement and Supply Management (GHSC-PSM) Project is the primary vehicle through which the U.S. Agency for International Development (USAID) procures and provides health commodities for countries receiving U.S. Government foreign assistance. Concurrently, it provides technical assistance to improve the efficiency, reach, and sustainability of in-country supply chains. This mid-term review looks globally at GHSC-PSM’s work in HIV, malaria, family planning and reproductive health (FP/RH), and maternal, newborn, and child health (MNCH) as it progresses towards meetings project objectives to improve the availability of health commodities, strengthen supply chain systems, and ensure effective global partnerships for the future. The review team found that, after a weak start, GHSC-PSM met difficult targets, including on-time delivery and on-time in-full delivery, and demonstrated declines in total landed cost over the first years of the project that were greater than the savings developed under previous projects. The project is strong in procurement, distribution, and the technical assistance needed to improve in-country systems, as well as key global engagements, especially in FP/RH and MNCH, market research, and innovations, which are all demonstrating encouraging results. Over the project’s second half, continued development of the project’s global supply chain strategy, such as rationalizing like elements of different health areas, would contribute to further success, as would a further, real-time build out of GHSC-PSM’s automated systems focused on reducing bottlenecks and risk. Taking advantage of evaluation and learning opportunities, a clearer theory of change and distinct nomenclatures for technical assistance, and better organization and tracking of global collaboration elements will enhance in-country progress. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / iv ACKNOWLEDGMENTS Thank you to USAID coordinators Christine Hershey, Wezi Munthali, and Meaghan Douglas, and GH Pro and GH-TAMS managers Melinda Pavin and Crystal Thompson, who provided excellent guidance and support throughout this review. We also thank René Berger, Alan Pringle, Lisa Rider, Samantha Salcedo￾Mason, Michael Cohen, Kiersten Rooke at GHSC-PSM, and Sherif Mowafy, Xavier Tomsej, Linda Gutierrez, John Vivalo, and Reena Shukla at USAID. Special thanks to the GHSC-PSM Country Directors and USAID Activity Managers who gave up their time during packed meetings to provide us with the field perspective. Finally, thank you to Terry Redding, our patient technical editor throughout this review, and to Laurie Chamberlain for final formatting on earlier drafts, as well as Kevin Daley at GH Pro and Shane Smith at GHSC-PSM, who managed our interviews. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / v CONTENTS Abstract............................................................................................................................................................................... iii Acknowledgments............................................................................................................................................................ iv Acronyms.......................................................................................................................................................................... viii Executive Summary .......................................................................................................................................................... xi I. Introduction..................................................................................................................................................................... 1 Review Purpose............................................................................................................................................................. 1 Review Questions......................................................................................................................................................... 1 Evaluation Timing .......................................................................................................................................................... 1 II. Project Background ...................................................................................................................................................... 2 GHSC-PSM ..................................................................................................................................................................... 2 USAID Strategy and Management............................................................................................................................. 3 III. Review Methods and Limitations............................................................................................................................. 5 Methodology .................................................................................................................................................................. 5 Limitations....................................................................................................................................................................... 6 IV. Findings........................................................................................................................................................................... 8 Introduction .................................................................................................................................................................... 8 Question 1: How has the Global Health Supply Chain for Procurement and Supply Management (GHSC-PSM) Project progressed across its stated objective and results for global procurement and logistics?........................................................................................................................................................................... 8 Question 2: How has the GHSC-PSM Project addressed risks, bottlenecks, and/or inefficiencies in the global supply chain system?.......................................................................................................................................18 Question 3: How has the GHSC-PSM Project realized cost savings and efficiency since its inception in January 2016, with the consolidation of procurement services under a single award IDIQ contract?..32 Question 4: How have in-country supply chains performed in GHSC-PSM-supported countries during the life of the project? What trends are observed?............................................................................................36 Question 5: How has GHSC-PSM coordinated/collaborated with global development partners to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future?....................................................................................................53 V. Conclusions and Recommendations......................................................................................................................60 Conclusions ..................................................................................................................................................................60 Recommendations......................................................................................................................................................63 Annex I. Statements of Work.......................................................................................................................................68 Annex II. Countries Served by GHSC-PSM............................................................................................................ 103 Annex III. Data Collection Instruments.................................................................................................................. 104 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / vi Annex IV. Sources of Information ............................................................................................................................ 108 Annex V. Evolution Of GHSC-PSM’S Indicators................................................................................................... 114 Annex VI. Performance Dataset Reference Guide............................................................................................... 119 Annex VII. Comments and Potential Gaps on “Core” Indicators.................................................................... 124 Annex VIII. Stocked According to Plan Rates, by Country ................................................................................ 127 Annex IX. Graphs of OTIF and OTD Results....................................................................................................... 129 Annex X. Calculating OTIF and OTD instructions.............................................................................................. 135 Annex XI. Conflicts of Interest................................................................................................................................. 137 Annex XII. Summary Bios of Evaluation Team...................................................................................................... 142 FIGURES Figure 1. Line Items Delivered On Time and In Full, Reported Versus Actual (FY 2016 Q4 to FY 2019, Q3) ........................................................................................................................................................................................ 9 Figure 2. Line Items Delivered on Time, within the Minimum Delivery Window, Reported Versus Actual (FY 2016 Q4 to FY 2019, Q3)........................................................................................................................................ 9 Figure 3. GHSC-PSM M&E Team Reporting Structure ..........................................................................................16 Figure 4. GHSC-PSM End-to-End Global Supply Chain..........................................................................................19 Figure 5. Global Supply Chain Commodity Management.......................................................................................20 Figure 6. Evolution of GHSC-PSM’s Global Supply Chain .....................................................................................22 Figure 7. Gartner Supply Chain Maturity Model......................................................................................................24 Figure 8. AssurX Dashboard Incident Tracker.........................................................................................................31 Figure 9. Total Landed Costs for GHSC-PSM (FYs 2017-18) and Previous Projects (FYs 2014-15) in HIV, Malaria, and FP/RH..........................................................................................................................................................33 Figure 10. GHSC-PSM Total Landed Cost by Key Cost Drivers, TO1 (HIV), TO2 (Malaria), and TO3 (FP/RH)...............................................................................................................................................................................34 Figure 11. SCMS/DELIVER Total Landed Cost by Key Cost Drivers................................................................. 35 Figure 12. GHSC-PSM Commodity Savings by Year for all TOs and TO1 (FYs 2017-2019) .......................36 Figure 13. Spending on GHSC-PSM TA and In-Country Distribution, Versus Commodities, Global Freight, and Logistics.......................................................................................................................................................41 Figure 14. Percentage of Supply Plans Submitted During FY 2019 Q2, by Commodity Group...................44 Figure 15. Timely Reporting Rates of Service Delivery Points via an LMIS for TOs 1-3 (Indicator B3, FY 2017 Q1 to FY 2019 Q2)..............................................................................................................................................46 Figure 16. Stock-out Rates at Service Delivery Points for TOs 1-3 (Indicator B1, FY 2017 Q1 to FY 2019 Q2) ......................................................................................................................................................................................50 Figure 17: Stock Status Observations in Storage Sites Where Commodities are Stocked According to Plan, by Level in Supply System, for TOs 1-3 (Indicator B2, FY 2017 Q1 to FY 2019 Q2)..........................51 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / vii TABLES Table 1: USAID’s GHSC-PSM Results Framework ................................................................................................... 3 Table 2. Countries Participating in Key Informant Interviews................................................................................ 6 Table 3. List of GHSC-PSM “Core” Indicators.........................................................................................................11 Table 4. “Core” Indicators by Objectives 1-3 ..........................................................................................................12 Table 5. Best Practices and the USAID/GHSC-PSM System.................................................................................26 Table 6. Evolution of In-Country TA Elements........................................................................................................38 Table 7. GHSC-PSM Training by Technical Focus, FY 2018..................................................................................48 Table 8. Examples of USAID and Global Fund Field Coordination to Mitigate Stock-Outs .........................59 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / viii ACRONYMS 3PL Third-party logistics ARTMIS Automated Requisition Tracking Management Information System ARVs Antiretrovirals CI Continuous improvement CLEAR Communications, Learning, Evidence, and Analytics for Results COPs Country Operational Plans COR Contracting Officer’s Representative CSCO Chief supply chain officer DATIM Data for Accountability Transparency Impact Monitoring DFID Department for International Development (United Kingdom) DOOR Drugs out of Range System DQA Data quality assurance eLMIS Electronic logistics management information system EUV End-user verification FASP Forecasting and supply planning FBO Faith-based organization FP/RH Family planning/reproductive health FPLM Family Planning Logistics Management Project FY Fiscal year GF Global Fund GH Bureau for Global Health (USAID) GHSC Global Health Supply Chain Program GHSC-BI&A Global Health Supply Chain Business Intelligence and Analytics GHSC-PSM Global Health Supply Chain Procurement and Supply Management Project GHSC-QA Global Health Supply Chain Quality Assurance GHSC-RTK Global Health Supply Chain Rapid Test Kits GHSC-TA Global Health Supply Chain Technical Assistance GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Corporation for International Cooperation) Global FP VAN Global Family Planning Visibility and Analytics Network GPS Global positioning system GSC Global supply chain HP Hewlett-Packard Company GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / ix HQ Headquarters HSS Health systems strengthening ID GH Office of Infectious Disease IDIQ Indefinite delivery/indefinite quantity IP Implementing partner IQC Indefinite quantity contract IR Intermediate Results JSI John Snow, Inc. KII Key informant interviews KMC Knowledge, management, and communications KPI Key performance indicator LLIHN Long-lasting insecticidal hammock net LLIN long-lasting insecticide-treated net LMIS Logistics management information system LOE Level of effort LTTA Long-term technical assistance M&E Monitoring and evaluation ME&L Monitoring, evaluation, and learning MCHN GH Office of Maternal and Child Health and Nutrition MIS Management information system MNCH Maternal, newborn, and child health MOH Ministry of health MOPs Malaria Operational Plans NFO Non-field office NGO Nongovernmental organization OHA GH Office of HIV/AIDS OHS GH Office of Health Systems OTD On-time delivery OTIF On-time, in-full delivery OU Operating units (USAID) PEPFAR President's Emergency Plan for AIDS Relief PfSCM Partnership for Supply Chain Management PIRS Performance Indicator Reference Sheet PMI President’s Malaria Initiative PMU Project Management Units GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / x PO Purchase order POC Point of contact POD Proof of delivery PPMR Procurement, planning, and monitoring report PPR Performance plan and report PR Principal Recipient (Global Fund) PRH GH Office of Population and Reproductive Health QA Quality assurance RDT Rapid diagnostic test (malaria) RFP Request for proposals RHSC Reproductive Health Supplies Coalition RTK Rapid test kit (HIV) S/GAC U.S. Department of State, Office of the Global AIDS Coordinator (PEPFAR) SCMS Supply Chain Management System Project (PEPFAR) SCOR Supply chain operations reference model SCRM Supply chain risk management SDP Service delivery point SOPs Standard operating procedures SOW Statement of Work SPACES Strategic Program for Analyzing Complexity and Evaluating Systems STTA Short-term technical assistance TA Technical assistance TLC Total landed cost TLD Tenofovir disoproxil fumarate/lamivudine/dolutegravir TLP Targeted local procurement TO Task order UNAIDS Joint United Nations Programme on HIV/AIDS UNFPA United Nations Population Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xi EXECUTIVE SUMMARY Purpose and Questions This independent, mid-point review assesses the progress and performance of the new Global Health Supply Chain – Procurement and Supply Management (GHSC-PSM) Project. Findings and recommendations are intended to advise the U.S. Agency for International Development (USAID) and GHSC-PSM on opportunities for continuous improvement, as well as to inform any future initiatives. The review questions include: 1) How has GHSC-PSM progressed across its stated objective and results for global procurement and logistics? 2) How has GHSC-PSM addressed risks, bottlenecks, and/or inefficiencies in the global supply chain system, and throughout the project? 3) Were cost savings and efficiencies realized since the start of the GHSC-PSM Project in January 2016, with the consolidation of procurement services under a single award IDIQ contract? 4) How have in-country supply chains performed in GHSC-PSM–supported countries during the life of the project? What trends are observed? 5) How has GHSC-PSM coordinated/collaborated with global development partners (such as the Global Fund [GF], United Nations Population Fund [UNFPA]) to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future? Background In April 2015, Chemonics International was awarded the GHSC-PSM IDIQ as a single-award, indefinite delivery/indefinite quantity (IDIQ) contract for a possible eight-year period, with a US$9.5 billion ceiling. Task orders (TOs) under the IDIQ were awarded for HIV (TO1), malaria (TO2), family planning/reproductive health (FP/RH) (TO3) in April 2015, and for maternal, newborn, and child health (MNCH) (TO4) in September 2016.1 A protest and subsequent lawsuit kept a stop work order for the IDIQ and TOs in place until the end of December 2015, and the contract began in January 2016. This is the first time USAID has consolidated the procurement for HIV, malaria, FP/RH, and maternal and child health commodities. As a new IDIQ with a new contractor, the first year of the project included a complex transition stage involving the transfer and start up commodity stocks, supplier agreements and country offices, and also the consolidation of regional warehouses. USAID’s goal for GHSC-PSM is to “ensure uninterrupted supply of health commodities to prevent suffering, save lives, and create a brighter future for families across the globe.” Its three objectives are: 1) improved availability of health commodities; 2) strengthened in-country supply chain systems; and 3) effective global collaboration to improve long-term availability to health commodities. 1 There is also a TO for technical assistance in Kenya that is not covered under this review. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xii Findings Interviews and other data collection for this review began May 9 and concluded September 19, 2019, at the time of the first formal briefing for USAID. Overall, the review team found positive working relationships between GHSC-PSM and USAID, especially at senior levels. As might be expected in such a large project, with over 50 USAID Bureau for Global Health (GH) managers and technical advisors involved, working relationships among technical staff in USAID/GH and GHSC-PSM are not uniform, and several respondents noted the desire to see more brainstorming between USAID and GHSC-PSM across all project objectives. Question 1: How has GHSC-PSM progressed across its stated objective and results for global procurement and logistics? On-time, in-full delivery (OTIF) and on-time delivery (OTD) show steady improvement, such that GHSC￾PSM is currently either meeting or exceeding its targets. To achieve this improvement, GHSC-PSM and Chemonics leadership worked closely with USAID to make critical management and operational changes. An action plan with USAID was developed, reported on, and completed. OTIF increased from 7 percent in FY 2017 (Quarter 2) to 85 percent in FY 2019 (Quarter 3), and OTD increased from 31 percent in FY 2017 (Quarter 2) to 92 percent in FY 2019 (Quarter 3). Nevertheless, there remain challenges in measuring the overall performance of the project, as there are significant gaps in measuring the performance of, per the GHSC-PSM Results Framework, the co-equal Objectives 2 and 3 noted just above under “Background.” The data and reporting demands on GHSC￾PSM vary by USAID health element, and this can cause challenges with the monitoring and evaluation (M&E) system. USAID uses both formal and informal meetings among USAID and GHSC-PSM TO M&E representatives, and both sides find this communication has mitigated most day-to-day operational issues; however, structural issues remain. The project follows 29 “core” indicators applicable to all TOs, with an additional 8 indicators applicable to only some of the TOs, and an additional 11 indicators specific to the President’s Malaria Initiative (PMI). Many of these indicators are very complicated and are often, especially for in-country indicators, difficult for general audiences to interpret. Processes for calculating key indicator results (OTIF and OTD) appear to be sufficient, although not sufficiently institutionalized within the broader GHSC-PSM M&E Team. That is to say, many data processes at the Washington headquarters are known by GHSC-PSM staff, but USAID does not always know how key indicators are being managed. GHSC-PSM has sufficient internal field data management systems, with checklists describing sources, timelines, and processes, as well as validation processes. These processes do not negate the need for external data quality assessments, such as data quality assurance (DQA) by USAID. Within the IDIQ, GHSC-PSM is instructed to build the local capacity for “monitoring and evaluating policy and strategy implementation for commodity security in both the public and private sectors and explore application of technologies or other innovations to further such policies,” but it is only obligated to develop an IDIQ Performance Monitoring Plan. Nevertheless, there are 32 country-specific M&E plans, and GHSC-PSM reports that 20 of these plans include country-specific indicators. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xiii GHSC-PSM evaluation activities (both internal and external) are limited, and learning opportunities are recent, primarily internal to the project, and, to date, fairly ad hoc. Currently, under 1 percent (.82 percent) of GHSC-PSM’s programmatic budget—including home office and field-based costs—is spent on monitoring and evaluation (or 2.96 percent when commodities are excluded). There was one M&E director and four specialists employed by GHSC-PSM at the time of this review. Question 2: How has GHSC-PSM addressed risks, bottlenecks, and/or inefficiencies in the global supply chain system, and throughout the project? GHSC-PSM has demonstrated consistent improvement over time in managing the supply chain operations reference (SCOR) model attributes—plan, source, make, deliver, and return/destroy, and enable, measure, and improve—across the board. However, the project faces some challenges as it moves along the Gartner Supply Chain Maturity Model2 that describes how supply chains typically evolve, including multiple missions, multiple decision-makers, multiple processes that may change depending on the situation, the stakeholder, budgetary delays or funding changes, and operating guidance. GHSC-PSM’s global supply chain (GSC)3 management is based on day-to-day developments, and lacks an overarching strategy or “guiding light” around which all supply chain activities are centered and operate, including a strategy for management of the supply chain with consistent protocols, objectives, and conventions. To better mitigate risks and bottlenecks, a build-out of the Automated Requisition Tracking Management Information System (ARTMIS) capabilities is necessary for the system to be more predictive. At present, ARTMIS shows the current status of every order in the system. It has some proactive capabilities, such as forecasting a supplier stock-out based on production runs and capacity. Still, pressure on the system is significant. For example, in FY 2018, ARTMIS processed $888.7 million in procurements, a 33.5 percent increase over FY 2017. Yet, operationally, USAID/GH said they expected that by this juncture ARTMIS would have had more proactive supply chain management capabilities, as well as more analytics capabilities, although at the same time they report that further funding is unlikely. Also important for risk mitigation, GHSC-PSM’s development of its continuous improvement (CI) program shows merit. Built on a CI system (AssurX), the program will eventually incorporate components such as standard operating procedures and workstream management to track reports of an issue, actions taken, and responsibilities undertaken on a weekly basis. The initial focus of the CI MIS effort is supplier performance, theft, and other repeated actions. The CI team also now has a platform for staff to submit issues, such as damaged shipments or theft in a country, that some field respondents said they were using. The review team found that communication about the supply chain(s) between USAID/GH and GHSC￾PSM are very granular, slow, and labor intensive. GHSC-PSM operates by the guiding principles of the 2 Note that the GHSC-PSM contract does not specify nor did USAID ever specify that the project would be assessed against the Gartner Model. 3 GSC is used in this report to refer to both the global supply chain that GHSC-PSM manages and the directorate within GHSC￾PSM where the global supply chain is managed. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xiv SCOR model; however, for the most part, USAID/GH does not. The two organizations think and speak differently when it comes to many supply chain decisions and activities, which creates risk. Question 3: Were cost savings and efficiencies realized since the start of the GHSC-PSM Project in January 2016, with the consolidation of procurement services under a single award IDIQ contract? Using project documentation4,5,6 and raw financial data, comparisons of total landed cost (TLC) were made between GHSC-PSM and the two predecessor projects for TO1 (HIV), TO2 (malaria), and TO3 (FP/RH).7,8 Adjustments were made to ensure comparable comparison over two periods: FYs 2014-2015 near the end of the predecessor projects, and FYs 2017-2018 near the beginning of GHSC-PSM. The percentage of reduction in TLC for GHSC-PSM between 2017-2018 was larger than that found for predecessor projects during the period of available data (2014-2015). Unpacking the commodity-related and headquarters operations costs revealed different cost drivers for GHSC-PSM between FY 2017 and FY 2018. With the exception of TO2 (malaria), in 2017 the cost drivers for the new GHSC-PSM were the global supply chain operations costs. In 2018, the cost drivers shifted to freight costs across the task orders.9 GHSC-PSM reported significant cost savings on commodities across health elements: $89.9 million as of the second quarter of 2019. Cost saving information, although an important accomplishment for GHSC￾PSM, is reported by TO, and as yet the methodology for cost savings calculation is not sufficiently elaborated for verification. Question 4: How have in-country supply chains performed in GHSC-PSM-supported countries during the life of the project? What trends are observed? GHSC-PSM lacks a theory of change or paradigm related to strengthening in-country supply chains; however, as the project got into the field, it adapted its organizational structure at headquarters to meet the needs of field missions and the priorities of the individual TOs. From an original management structure, wherein one of three managing directors was responsible for overseeing all country programs, GHSC-PSM strengthened the staffing in Project Management Units (PMUs) to oversee the work of headquarters-based country and operational staff. Changes included the consolidation of oversight of non-field offices (NFOs) to better serve the needs of countries receiving commodities from GHSC-PSM in which the project had no presence to work with the field mission, government, or other stakeholders. Additionally, the Health System Strengthening (HSS) team, initially a separate unit at GHSC-PSM headquarters, was integrated into country programs. By the end of FY 2018, 41 countries were receiving a mix of short-term technical assistance (STTA) or long-term technical assistance (LTTA) in forecasting and supply planning (FASP) and warehousing and distribution, as well as workforce development, leadership, governance, logistics management 4 GHSC-PSM Quarterly Report, FY 2017, Q4, pp. 113. 5 GHSC-PSM Quarterly Report, FY 2018, Q4, pp. 10. 6 SPACES. “Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER,” 2017. 7 Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES), “Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER,” 2017. 8 TLC for MNCH was not reported for predecessor projects, and thus was not included in this comparison. 9 GHSC-PSM Global Monthly Financial Report, IDIQ Level, 2017 and 2018 monthly reports. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xv information systems (LMIS), laboratory networks, Global Standards (GS), and HIV prevention. Field mission activity managers interviewed made distinctions, not formally made by USAID/GH or GHSC-PSM, between two types of supply chain technical assistance (TA). The first was TA to strengthen the efficiency and effectiveness of procurement and distribution of health commodities. Said by informants to be a GHSC-PSM TA forte, this TA included assistance and the transfer of knowledge logistics, warehousing, information systems, lab networks, training, and tapping the private sector to improve the efficiency and effectiveness of supply chains. The project publishes performance and context indicators by the IDIQ, TOs, countries (by national and sub-region), and commodities in its reporting, including those that can help direct project attention to gaps in in-country supply chains. The second type of TA, referred to by respondents as health systems strengthening, is focused on long￾term capacity building (beyond training), and improvements in decision-making and governance relevant to a country’s journey toward self-reliance in health commodities, and includes work on governance, restructuring elements of the supply chain, workforce development (as opposed to skills training), and pharmaceutical policy and process. GHSC-PSM provides health systems strengthening assistance in various countries, such as Guinea, Malawi, Pakistan, Lesotho, and Nepal, as part of field mission and/or partner government strategies to make the policy and administrative changes necessary to move towards self-reliance. Question 5: How has GHSC-PSM coordinated/collaborated with global development partners (such as the Global Fund, United Nations Population Fund [UNFPA], etc.) to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future. The Objective 3 construct for global collaborations mixes actual global collaborations—global partner engagement, private sector market analyses, and innovations—with cross-Objective functions, such as project communication, social media, research, and collaboration with other USAID/GH projects. GHSC￾PSM reports on the number of engagements, research, and innovations, and provides narratives in reports;10 however, there are no criteria on whether a collaboration fell short, met expectations, or exceeded intentions over time. Two cross-cutting accomplishments that were highly lauded by respondents, one in global engagement and another in private sector market analysis, were initiated early in the project. Additionally, USAID/GH TO3 (FP/RH) and TO4 (MNCH) leaders have effectively brought GHSC-PSM into their engagement strategies as extensions of themselves, and the project has designated points of contact for the work. FP/RH and MNCH global partners know and appreciate the GHSC-PSM team. They discussed how the project contributes to aligning commodity availability with UNFPA at the global level, strengthening the Global Family Planning Visibility and Analytics Network (Global FP VAN), avoiding oxytocin shortages, and mitigating in-country commodity stock-outs. For TO1 (HIV) and TO2 (malaria), the project is tapped by USAID for input on global engagements; however, it is not customarily brought into global partner engagements, nor is the project well understood by the various global partners interviewed. In the field, GHSC-PSM country programs coordinate with the Global Fund and other global partners to mitigate stock-outs. Roll out is seen as important for global collaborations, and respondents explained this takes 10 See Annex VI, Performance Dataset Reference Guide. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xvi one or more of three paths at a time: absorption into the GSC, roll out to partner countries via TA, or roll out within a global partnership. CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS After an unprecedented, overly rapid transition from long-standing contractors, and a difficult start-up in the public eye, GHSC-PSM is credited with a strong comeback. The project is meeting and sustaining key targets of critical interest to USAID over this first half of the project, including OTD and OTIF, and demonstrates declines in TLC. The project manages a complex, worldwide procurement and distribution supply chain, supported by in-country monitoring and TA, and contributes to global engagements, market research, and innovations. Other key conclusions, directed both to GHSC-PSM and USAID, are as follows: Improved availability of health commodities: The project is successful in observing the SCOR model— plan, source, make (not relevant for GHSC-PSM), deliver, return/destroy, and enable, measure and improve—and is working to move up the ladder on the Gartner Maturity Model from a simple, siloed, and reactive supply chain. GHSC-PSM is correctly looking for the ability to predict, anticipate, and influence what is happening within the supply chain, but lacks an overarching strategy for pursuing predictive and proactive supply chain management that could optimize multiple supply chains. The review team believes that USAID and GHSC-PSM collaboration on this strategy development, beginning with systematizing functions across the TO supply chains, is essential to mesh the strategic needs with the supply chain operational realities. Each party has a perspective that is essential in “informing” a collective overarching strategy and enabling execution. The GHSC-PSM continuous improvement program, using its AssurX tool, shows significant progress. Needed is a build-out of ARTMIS capability to be more predictive, and better harmonization of the “language” USAID and GHSC-PSM use to understand USAID’s implementation of the SCOR and Gartner models. Strengthening in-country supply chain systems: The review team agrees with documentation and respondents, including USAID field representatives, that TA to improve the efficiency and reach of supply chain programs in-country is GHSC-PSM’s forte among TA offerings. Field missions appreciate and depend on the project’s expertise in these areas, although there is some question as to whether more TA could be provided by local experts. Of special note is GHSC-PSM’s work to develop, improve, or consolidate LMIS, warehousing and distribution, skills for supply chain workers and managers, and lab efficiency, as well as its use of the private sector. Field respondents also described the progress made by GHSC-PSM; they made a distinction between this direct TA and what they term health systems strengthening (HSS) assistance to governments wishing to undertake systematic changes in governance, management, or human resource development, areas the project is active in and, according to respondents, is doing well in individual countries where they work. Paradigms and approaches were developed for supply chain elements such as human resource development, lab networks, and activity￾based costing, but not for an entire supply chain system, leaving the project in a more reactive position when approached by field missions and USAID/GH to provide TA. USAID and GHSC-PSM need to provide relevant granularity to what the IDIQ calls an in-country HSS TA program. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xvii Effective global collaboration to improve long-term availability of health commodities: USAID was correct to put global collaboration on an equal footing with other IDIQ objectives. GHSC-PSM demonstrates progress in global partner engagement, private sector market analyses, and innovation by joining in on USAID strategies and activities as requested. Highlights across all TOs include GHSC-PSM’s contributions to Global Standards and their private sector market analysis; both are global collaborations, their results are already absorbed by the project’s global supply chains, and Global Standards continue to be rolled out via the TA program. USAID/GH TO3 (FP/RH) and TO4 (MNCH) leads have effectively brought GHSC-PSM into their engagement strategies as extensions of themselves, and the project has designated points of contact for the work. FP/RH and MNCH global partners know and appreciate USAID and GHSC-PSM contributions. For TO1 (HIV) and TO2 (malaria), the project is tapped by USAID for input on global engagements, however they are not as directly brought into global partner engagements. GHSC-PSM does report recent discussions with GF on data sharing, aligning quality assurance processes, and market health initiatives, which could be an opportunity for more robust GHSC-PSM involvement as USAID, PEPFAR, and PMI work to better align with the Global Fund. Cross-cutting issues: The amount of GHSC-PSM’s budget dedicated to M&E is fairly low by international standards, and is extremely low if commodity purchases are included in the denominator. For a high￾profile project such as GHSC-PSM, this is particularly of concern, given both the demands for data and reporting, and the criticality of health commodities for USAID partner countries. Furthermore, evaluative and learning opportunities have not been fully utilized, and to neglect this particular area would be a major failing for both USAID and GHSC-PSM, and USAID needs to ensure that DQAs are conducted. While indicators for Objective 1 follow international standards, results from several of the indicators for Objectives 2 and 3 are difficult to attribute to GHSC-PSM, and do not adequately measure whether the technical assistance provided for strengthening the supply chain (as a sub-unit of the overall health system) is resulting in sustainable or transformative accomplishments. RECOMMENDATIONS 1. USAID should develop an overarching supply chain strategy based on a combination of the SCOR and Gartner Maturity models. (timeframe: ~2 years) 2. As a first step in the development of an overarching supply strategy, USAID should further systematize the supply chain across funding streams. (timeframe: ~1 year) 3. USAID and GHSC-PSM should develop a “Rosetta Stone”-type terminology resource for their staffs working with the GSC program, kicked off by a facilitated workshop. 4. To reduce risks and bottlenecks, USAID should continue to invest in building out GHSC-PSM’s continuous improvement software application, AssurX, to ensure it is integrated with ARTMIS. 5. USAID should decide how it wants to define TLC and other key GHSC indicators now and into the future, if these indicators are expected to be comparable across time. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / xviii 6. GHSC-PSM should develop and disseminate a brief theory of change related to strengthening country supply chains, which would include a simple diagram. 7. USAID and GHSC-PSM should provide more specificity and a clearer nomenclature for the work it undertakes in-country, e.g., direct assistance in executing a supply chain, TA to improve the efficiency and reach of a supply chain, and HSS in concert with government partners to make structural, administrative, and policy changes necessary to move toward self-reliance. 8. USAID and GHSC-PSM should clarify Objective 3, which regards global collaboration, to focus on global engagement, private sector market research, and innovations, and provide long-term oversight of key global collaborations. 9. GHSC-PSM’s budget dedicated to M&E should increase, and include the hiring of additional staff. 10. GHSC-PSM needs to progress from a monitoring function, and look for openings for additional evaluative and learning opportunities. 11. GHSC-PSM should develop an official data dictionary for the Performance Dataset and standard operating procedures (SOPs), for calculating all key indicators beyond what is already detailed in the M&E plan. 12. While possibly not achievable in the remaining life of the project, USAID and GHSC-PSM should consider revisiting and possibly revising indicators for Objectives 2 and 3 in GHSC-PSM’s M&E Plan. 13. GHSC-PSM should hold biannual meetings with USAID M&E personnel to review the Performance Dataset and its contents, and how key indicators are being calculated and managed. 14. External DQAs by USAID of GHSC-PSM need to become routine. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 1 I. INTRODUCTION REVIEW PURPOSE This report represents an independent, mid-term review of the United States Agency for International Development (USAID) Bureau for Global Health (GH) Global Health Supply Chain – Procurement and Supply Management (GHSC-PSM) Project. Its purpose is to assess and document the progress and performance of GHSC-PSM in meeting its core objective, which is to ensure an uninterrupted supply of health commodities for HIV, malaria, family planning and reproductive health (FP/RH), and maternal, newborn, and child health (MNCH)11 activities around the world. Findings and recommendations will be used to identify opportunities for continuous improvements to enhance procurement services and supply chain technical assistance going forward. The primary audience for this review is USAID, especially staff members directly involved with GHSC-PSM. REVIEW QUESTIONS Five questions were posed to assess and document the progress and performance of GHSC-PSM near its midpoint in meeting its core objectives: 1) How has GHSC-PSM progressed across its stated objective and results for global procurement and logistics? 2) How has GHSC-PSM addressed risks, bottlenecks, and/or inefficiencies in the global supply chain system, and throughout the project? 3) Were cost savings and efficiencies realized since the start of GHSC-PSM in January 2016, with the consolidation of procurement services under a single award indefinite delivery/indefinite quantity (IDIQ) contract? 4) How have in-country supply chains performed in GHSC-PSM–supported countries during the life of the project? What trends are observed? 5) How has GHSC-PSM coordinated/collaborated with global development partners (such as the Global Fund, United Nations Population Fund [UNFPA], etc.) to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future? EVALUATION TIMING In-brief contacts between USAID and the evaluation team began with an introductory meeting with USAID points of contact (POCs) on April 30, 2019. Data collection began May 9 and concluded on September 19, 2019, at the time of the first formal briefing with USAID staff. The evaluation report first draft was submitted to USAID by the Global Health Program Cycle Improvement Project (GH Pro) on September 13, 2019. 11 Readers will find two similar acronyms used in this report in similar contexts. MNCH refers to the technical field of maternal, newborn, and child health, as well as Task Order 4. A similar acronym for the USAID Global Health Office of Maternal and Child Health and Nutrition (MCHN) is used to refer to that particular USAID/GH office. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 2 II. PROJECT BACKGROUND GHSC-PSM The GHSC-PSM Project is the primary vehicle through which USAID procures and provides health commodities for U.S. Government (USG) health programs, including HIV, malaria, FP/RH, and MNCH. The project also provides health systems strengthening technical assistance to improve supply chain management and commodity security as part of USAID’s policy framework to support partner countries’ “Journey Towards Self-Reliance.”12 The GHSC-PSM IDIQ contract was awarded in April 2015. However, a protest on the award led USAID to institute a stop work order on contract activities. The protest was denied in August 2015, and then a lawsuit was filed in September 2015. On December 22, 2015, the stop work order for the IDIQ and related task orders (TOs) was lifted, and the contract started in January 2016. GHSC-PSM is the centerpiece of an array of eight commodity procurement and supply chain technical assistance mechanisms, collectively known as the implementing architecture for the GHSC. It is implemented through a single-award IDIQ contract for a possible eight-year period, with a $9.5 billion ceiling. Separate task orders are in place for HIV (TO1), malaria (TO2), FP/RH (TO3), and MNCH (TO4), and for country-specific technical assistance in Kenya (TO5).13 TOs 1-3 were awarded in April 2015, and TO4 in September 2016. GHSC-PSM is led by Chemonics International, with sub-partners including IBM, Kuehne + Nagel Inc., IDA Foundation, Population Services International, SGS Nederland B.V., McKinsey & Company, IntraHealth International Inc., Arbola Inc., Axios International Inc., Panagora Group, and University Research Co., LLC (URC). Antecedents USAID has long procured commodities for global health programs, including the direct procurement of needed family planning products. Successive projects in support of contraceptive or commodity procurement and supply chains dating back to 1986 were all awarded to John Snow, Incorporated (JSI) including: Family Planning Logistics Management (FPLM) I (1986–90); FPLM II (1990–95); FPLM III (1995– 2000); the DELIVER Project (2000–07); and the USAID/DELIVER PROJECT (2006–11).14 From 2005 to 2017 (immediately prior to GHSC-PSM), USAID managed procurement and supply chain operations under two large, single-award Indefinite Quantity Contracts (IQCs), with the DELIVER Project serving malaria, FP/RH, and MNCH programs, and the Supply Chain Management Systems (SCMS) Project implemented by the Partnership for Supply Chain Management (PfSCM)15 covering HIV/AIDS commodities for the President’s Emergency Plan for AIDS Relief (PEPFAR). 12 USAID, “The Journey to Self-Reliance,” https://www.usaid.gov/selfreliance. 13 This review does not cover the work of USAID/Kenya’s TO5. 14 Don Lauro and Dian Woodle, March 2011. USAID/DELIVER Project Mid-term Evaluation. Global Health Technical Assistance Project. 15 PfSCM is a non-profit organization established by Management Sciences for Health (MSH) and the JSI Research and Training Institute. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 3 USAID STRATEGY AND MANAGEMENT USAID’s goal for GHSC-PSM is an “ensured uninterrupted supply of health commodities to prevent suffering, save lives, and create a brighter future for families across the globe.” To reach this goal, USAID identifies and measures progress across three key objectives: 1) improved availability of health commodities; 2) strengthened in-country supply chain systems; and, 3) effective global collaboration to improve long-term availability to health commodities. Table 1 below presents the Intermediate Results (IRs) under the Results Framework. Table 1: USAID’s GHSC-PSM Results Framework Objective 1: Improved availability of health commodities (global procurement & logistics) Objective 2: Strengthened in￾country supply chain systems Objective 3: Effective global collaboration to improve long￾term availability of health commodities IR1.1 Enhanced global health commodity procurement. IR2.1 Improved strategic planning and implementation related to supply chain management and commodity security. IR3.1 Improved strategic engagement with global partners to ensure appropriate strategic coordination. IR1.2 Strengthened global logistic processes associated with the storage and delivery of any health commodity to any point in donor￾supported countries. IR2.2 Improved in-country logistics, including effective and efficient delivery of health commodities to service sites. IR3.2 Global market dynamic research and innovations conducted, shared, and implemented. IR1.3 Ensured adherence to quality assurance requirements. IR2.3 Increased capacity building efforts by implementing strategies to transfer skills, knowledge, and technology for improved and sustained performance. IR3.3 Improved awareness and advocacy to improve availability of essential health commodities. IR1.4 Improved data visibility. IR2.4 Strengthened enabling environments to improve supply chain performance. IR3.4 Improved coordination and collaboration between TOs within the IDIQ and with other USAID supply chain funded activities. Contracting Officer Representatives (CORs) for the IDIQ and Washington, D.C.-based TOs come from four USAID/GH offices. At the time of the review they were: ● Sherif Mowafy, IDIQ COR, USAID/GH Office of HIV/AIDS (OHA) ● Xavier Tomsej, TO1 COR, USAID/GH OHA (supporting PEPFAR) ● Linda Gutierrez, TO2 COR, USAID/GH Office of Infectious Disease, President’s Malaria Initiative (PMI) ● John Vivalo, TO3 COR, USAID/GH Office of Population and Reproductive Health (PRH) ● Reena Shukla, TO4 COR, USAID/GH Office of Maternal and Child Health and Nutrition (MCHN) CORs are advised by varied numbers of country and technical area specialists within their offices. Senior staff within USAID/GH technical offices periodically rotate as the USAID/GH lead/point of contact for supply chain issues in interacting with the USAID/GH front office. An estimated 50 persons within USAID/GH participate in the technical direction and management of this project. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 4 Annual country work plans, approved by all relevant CORs, define in-country activities and, in the cases of TO1 (HIV) and TO2 (malaria), are informed by the requirements of PEPFAR Country Operational Plans (COPs) and PMI Malaria Operational Plans (MOPs), respectively. GHSC-PSM Staffing and Global Reach GHSC-PSM has 384 staff at headquarters, and 1,345 staff in 34 field offices that range in size from two to 222 personnel, of whom just 1.6 percent are expatriates. As of April 2019, the project had provided commodities and/or technical assistance to 67 countries world-wide (listed in Annex II); their catalog lists 3,987 items from 326 suppliers. GHSC-PSM uses three distribution centers for health commodities in Belgium, South Africa, and the United Arab Emirates (UAE), and 4,110 shipment lanes.16 16 Shipment lanes refer to the routes between the points of shipment to the points of delivery. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 5 III. REVIEW METHODS AND LIMITATIONS METHODOLOGY The methodology for the mid-term review of the GHSC-PSM Project was designed to answer the primary questions identified in the Statement of Work (Annex I: see Review Questions) and is structured around the three major objectives of GHSC-PSM’s Results Framework. Per USAID’s request, additional questions and corresponding interview guides were developed, which focused on USAID’s management of the GHSC-PSM Project. The methodology used both quantitative and qualitative approaches, and consisted of the following steps: 1) Initial desk review of background documents, including, but not limited to: ● The original Requests for Proposals for GHSC-PSM; ● The original scopes of work for the IDIQ contract and four global TOs for GHSC-PSM; ● Annual work plans for GHSC-PSM; ● Quarterly and annual reports for GHSC-PSM; ● Monitoring, evaluation, and learning (ME&L) plans for GHSC-PSM; ● Technical reports and presentations prepared by GHSC-PSM; and, ● Additional miscellaneous and relevant documentation (see Annex IV for a full listing). 2) Developing a standardized set of questions for the Key Informant Interviews (KIIs) based on type of stakeholder, as shown in Annex III; 3) Review and revision of the KII questionnaires based on GH Pro and USAID feedback; 4) Conducting KIIs with headquarters (USAID and GHSC-PSM) key informants (May 10–August 9, 2019); 5) Conducting KIIs with GHSC-PSM Country Directors (beginning June 5, 2019, with most occurring July 9-11, 2019); 6) Conducting KIIs with USAID field staff, primarily Supply Chain Activity Managers (July 22-26, 2019); 7) Conducting KIIs with supply chain global partners (July 3-August 9, 2019); 8) Analyzing and verifying monitoring data based on ARTMIS (Automated Requisition Tracking Management Information System) inputs; 9) Mapping of the monitoring indicators against both the results framework and industry standards; 10) Analyzing comparatively GHSC-PSM’s performance and risk using the supply chain operations reference (SCOR) and supply chain risk management (SCRM) models;17 11) Analyzing key selected costs and expenditure trends across the life of the project based on existing project data; 17 The supply chain operations reference (SCOR) model is a process reference model developed and endorsed by the Supply Chain Council as the cross-industry, standard diagnostic tool for supply chain management. The SCOR model describes the business activities associated with satisfying a customer's demand, which includes: plan, source, make, deliver, return/destroy, and enable. Use of the model includes analyzing the current state of a company's processes and goals, quantifying operational performance, and comparing company performance to benchmark data. SCOR has developed a set of metrics for supply chain performance, and Supply Chain Council members have formed industry groups to collect best practices information that companies can use to elevate their supply chain models. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 6 12) A comparison of the total landed cost between GHSC-PSM and previous projects calculated by definitions used in the Global Knowledge Initiative Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES) Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER project report; 18 and, 13) Collecting, synthesizing, and analyzing the results based on the various data collection methods. Key informant interview characteristics (by telephone and in-person) A total of 137 KIIs were conducted in person or by telephone from May 10 to August 9, 2019. Participants in these KIIs included staff from USAID/Washington, USAID field missions, GHSC-PSM headquarters staff, GHSC-PSM field staff, and global partners (e.g., the Global Fund [GF], UNFPA, the Reproductive Health Supplies Coalition). Between July 9 and 11, 2019, 12 GHSC-PSM country directors were interviewed, with two directors being interviewed earlier, in May 2019. Similarly, during the week of July 22-26, 2019, 31 USAID field mission staff were interviewed (see Table 2 for a list of countries). Table 2. Countries Participating in Key Informant Interviews GHSC-PSM Country Directors USAID Field Mission Staff Côte d’Ivoire, eSwatini, Ethiopia, Ghana, Guinea, Kenya, Malawi, Mozambique, Nigeria (present and former), Pakistan, Zambia Angola, Benin, Botswana, Burma, Cameroon, Côte D’Ivoire, Democratic Republic of the Congo, Ethiopia, Ghana, Guinea, Haiti, Jamaica, Kenya, Lesotho, Malawi (2), Mali, Mozambique, Nepal, Nigeria (2), Pakistan, Rwanda, Senegal, South Africa, Tanzania (2), Uganda (2), Zambia (2) LIMITATIONS The primary approach for this review was qualitative data collection via KIIs, and additional quantitative methods as applied for the indicator, SCOR, and cost analyses. As such, the review team encountered several biases and other data limitations that were mitigated through methodological or analytical means. ● Qualitative approach: The opinions of stakeholders are, by their nature, subjective. Team members also may not have accurately recorded or correctly transcribed important data for a variety of reasons, and those data may not be part of the findings, conclusions, or recommendations. ● Selection bias: Key stakeholders may have been inadvertently excluded and those persons who did participate in a KII or group interview may have introduced self-selection bias (either beneficial or detrimental) into the results, limiting the ability to draw definitive findings, conclusions, and recommendations. ● Limited sample of KIIs: The team anticipated that data collection with some key stakeholders might be difficult to schedule because of existing demands on their time. To mitigate this concern, 18 Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES) Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER, January 2017. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 7 the team remained flexible to the best extent possible to accommodate as many key informants and group interview participants as feasible. ● Response bias: Response bias is a common problem for assessments, evaluations, and reviews. For instance, respondents may have given the interviewer positive remarks about the project because they have a vested interest in seeing the project succeed and be continued. ● Recall and recency biases: Some of GHSC-PSM’s interventions have either been completed or were conducted at the beginning of the project. Likewise, the performance of GHSC-PSM may have varied during its implementation. This may, of course, potentially introduce a type of recall and/or recency bias in the respondents, as responses may have varied significantly both in terms of accuracy and opinion if the review is conducted earlier or later during specific implementation. ● Assumptions made within quantitative modelling: For example, the SCOR model does not explicitly address every business process or activity in its analysis, but rather assumes these basic activities (e.g., administration, supporting information systems) to be fundamental supporting processes. Therefore, they are not directly examined. Similarly, for the cost analysis, several assumptions had to be made for the current GHSC-PSM Project. Due to incomparability of cost variables between the current GHSC-PSM and the two predecessor projects, only the cost variables that appeared in both projects were considered, hence potentially underestimating the costs. To prevent biases, the review team used multiple sources of data to triangulate answers for the review questions. By combining data from documents, KIIs from multiple sources, and quantitative analysis, it was believed that any one piece of potentially biased data would not skew the analysis. Another approach that pertains specifically to KIIs was the inclusion of key informants from organizations that do not directly benefit from the reviewed project (i.e., global partners), and the use of their feedback in determining results. The review team thus believes that applying these data collection insights to mitigate the biases and limitations results in robust findings, conclusions, and recommendations. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 8 IV. FINDINGS INTRODUCTION This section examines in turn the five review questions noted in the introduction. Along with the initial questions, USAID program managers made several specific requests for information and analyses within the questions for this mid-term review. In some cases, primarily for Questions 1 and 3, these requests took the review team’s attention into unexpected, yet useful, areas of investigation. As such, readers looking for how GHSC-PSM progressed in meeting project objectives—which is typical for a mid-term review—may benefit from added background details. For example, Questions 2, 4, and 5 include findings on operations, progress, and gaps as they align respectively with the three project objectives: 1) improved availability of health commodities, 2) strengthening in-country supply chain systems, and 3) effective global collaboration to improve long-term availability of health commodities. Within Question 1 below is the response to USAID’s request to verify on-time, in-full delivery (OTIF) and on-time delivery (OTD), which led the review team to take a more in-depth look at the project’s indicators and monitoring and evaluation (M&E) system. Similarly, in Question 3, USAID’s request to compare total landed cost (TLC) in GHSC-PSM and predecessor projects led to findings on how TLC is measured, and a drill-down into what the indicator said about cost drivers. QUESTION 1: HOW HAS THE GHSC-PSM PROJECT PROGRESSED ACROSS ITS STATED OBJECTIVE AND RESULTS FOR GLOBAL PROCUREMENT AND LOGISTICS? 1.1 For two key Objective 1 indicators19 (OTIF and OTD), early performance was sub-optimal; however, during the life of the project to date, there has been steady improvement such that GHSC￾PSM is currently meeting or exceeding its targets. For Objective 1, OTIF (percentage of line items delivered on time and in full, within the minimum delivery window) and OTD (percentage of line items delivered on time, within the minimum delivery window) are two of the key performance measures. As can be seen in Figures 1 and 2 below, early performance (for approximately the first six quarters of implementation) was far below expectations for both OTIF and OTD. The reasons for this sub-optimal performance are well-documented in internal project monitoring documents, such as the quarterly and annual reports, and in previous external reviews.20 What is immediately notable, however, is that there has been a steady improvement in performance against these two indicators for the last several quarters, such that targets are now being either met or exceeded. This not only applies to the overall IDIQ performance, but also to the individual TOs (see Annex VIII for further information). 19 See Annex V for a list of indicators by project objective; this annex also shows the evolution of project indicators over time. A list of core indicators and their connection to the objectives is also found in tables 3 and 4 below. 20 Oversight Investigation: USAID Global Health Supply Chain Contract. Committee on Foreign Affairs, U.S. House of Representatives, October 2018; Early estimates for OTD and OTIF predated the project’s MIS system and in retrospect “actual” numbers were found to be higher. However, GHSC-PSM follows a rule that reported data are not changed once published. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 9 Figure 1. Line Items Delivered On Time and In Full, Reported Versus Actual (FY 2016 Q4 to FY 2019, Q3) Note: “Reported” data are based on the best available data at the time of reporting in GHSC-PSM’s quarterly reports. “Actual” figures from previous reporting may differ as new data about those periods become available, as is the case of late-arriving proof of delivery. “Actual” figures are based on the GHSC-PSM Performance Dataset, downloaded 19 August 2019. Figure 2. Line Items Delivered on Time, within the Minimum Delivery Window, Reported Versus Actual (FY 2016 Q4 to FY 2019, Q3) 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 Percentage Delivered On Time Reporting Period IDIQ Reported IDIQ Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 Percentage Delivered On Time Reporting Period IDIQ Reported IDIQ Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 10 To achieve this improvement, GHSC-PSM and Chemonics leadership worked closely with USAID to make critical management and operational changes. An action plan with USAID was developed, reported on, and completed. Key staffing changes were made and Chemonics committed a “surge team” of senior executives and tapped key project staff to institute improvements in metrics; the project’s management information system (MIS) was also developed. Both USAID and GHSC-PSM noted that the “all-hands-on￾deck” approach was important for moving OTIF from 7 percent in FY 2017, Quarter 2, to 85 percent in FY 2019, Quarter 3, as well as increasing OTD from 31 percent in FY 2017, Quarter 2, to 92 percent in FY 2019, Quarter 3. Nevertheless, as discussed below, there remain challenges in measuring the overall performance of the project as there are significant gaps in measuring performance of, per the GHSC-PSM Results Framework, the co-equal Objectives 2 and 3. 1.2 The data and reporting demands on GHSC-PSM vary by USAID health element, and this can create challenges with the M&E system. Key informant interview data suggest that each health task order associated with GHSC-PSM has its own priorities in terms of data and reporting needs. For example, TO3 (FP/RH), given its substantial history of procurement and technical assistance work with both the current and predecessor projects, has a strong interest in Objective 2 and thus tends to prioritize issues of systems strengthening and sustainability, and highlights issues such as governance and leadership. However, it too requires reporting on several indicators related to Objective 1. This is in contrast to TO2 (malaria), which has a primary focus on malaria elimination and therefore focuses on ensuring commodity delivery and reducing and/or mitigating stock-outs. Still, more recent discussions have centered on examining the reasons behind the stock-outs. Finally, TO1 (HIV) has significantly greater data needs, in terms of quantity, quality (including granularity), and frequency. As such, it relies on multiple sources of data, both within and outside of GHSC-PSM. TO4 (MNCH) was not discussed in interviews. USAID has instituted both formal and informal meetings among TO M&E staff as a means to unify their ideas and requests to GHSC-PSM. Based on feedback from GHSC-PSM stakeholders, these internal discussions at USAID seem to have mitigated most M&E day-to-day operational issues; however, as discussed below, there still appear to be greater strategic issues, which need resolution and alignment. 1.3 Within GHSC-PSM’s M&E plan, Objective 1 relies mainly on Global Standards and appears to be sufficient for reporting purposes. Indicators for Objective 2 and 3 are primarily at the output and outcome levels, and there may be issues with the verifiability of some of the indicators for the same two objectives. GHSC-PSM’s M&E plan, as explained by USAID, is built on the SCOR model, which is noted above and is presented in more detail below in the Question 2 discussion of bottlenecks and risks linked to the performance metrics. GHSC-PSM’s M&E plan and corresponding indicators have evolved during the life to date of the project. This evolution is shown in Annex V. In total, there have been 52 different indicators utilized (excluding those specific to PMI) since GHSC-PSM began implementation. While it is not unusual for indicators to be added, dropped, or have their wording modified, these changes do make the establishment of baselines, the setting of targets, and the monitoring of results more challenging. There is an annual process in which the M&E technical working group reviews indicators, GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 11 results, and their targets, and decides whether any changes should be made to the M&E plan and reporting going forward. The results of these reviews and discussions are then sent to COR team for approval. GHSC-PSM’s IDIQ Project M&E Plan (Learning is not included in the title), which was submitted on February 11, 2019, contains 37 indicators. As shown in Table 3 below, there are 29 “core” indicators21 (i.e., applicable to all TO indicators to measure results22) with an additional 8 indicators applicable to only some of the TOs. An additional 11 indicators specific to PMI are also not included in the table. Table 3. List of GHSC-PSM “Core” Indicators Indicator Indicator 1 On-time, in-full delivery 16 Percentage of required supply plans submitted to GHSC-PSM 2 On-time delivery 17 Percentage of total spent or budgeted on procurement of commodities for public sector services, by funding source 3 Cycle time (average) 18 Percentage of targeted supply chain activities in which the host country entity has achieved technical independence with GHSC-PSM technical assistance. 4 Inventory turns 19 Supply chain technical staff turnover rate 5 Total landed cost 20 Percentage of GHSC-PSM-supported countries that have a functional logistics coordination mechanism in place 6 Temporary waiver percentage 21 Percentage of leadership positions in supply chain management that are held by women 7 Average percentage of shelf life remaining 22 Number of innovations that were developed, implemented, or introduced, and are related to the health commodity market or supply chain best practices 8 Framework contract percentage 23 Number of people trained 9 Average vendor rating score 24 Percentage of required files submitted to the Business Intelligence and Analytics Project (GHSC￾BI&A) 10 Percentage of backlogged line items 25 Percentage of required files timely submitted to GHSC-BI&A 11 Stock-out rate at service delivery points 26 Product loss due to expiry 12 Stocked according to plan at storage sites 27 Product loss due to theft, damage, and other causes 13 Service delivery point reporting rate to the logistics management information system (LMIS) 28 Number of global advocacy engagements in support of improved availability of essential health commodities 21 GHSC-PSM does not use the term “core” indicators. It is used here to clarify what was reviewed. 22 An indicator for “Annual absolute percent consumption forecast error” (B12) is also applicable to all TOs, but is not included in this list. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 12 14 Average rating of in-country data confidence 29 Supply chain policies, regulations, strategies, or standard operating procedures (SOPs) developed or updated with GHSC-PSM assistance 15 Percentage of required annual forecasts conducted Of note, and further illustrated in Table 4 below, is that of the 29 core indicators, 15 are currently being applied against Objective 1, 13 are being applied against Objective 2, and 2 are being applied against Objective 3. Most of the Objectives 2 and 3 indicators are outcome indicators, while a few are output indicators.23 As noted in Table 1 in the Background section above, each of the three Objectives has four associated Intermediate Results. Table 4. “Core” Indicators by Objectives 1-3 Objective Indicators (as numbered in Table 3 directly above) 1. Improved availability of health commodities 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 24, 25, 26, 27 2. Strengthened in-country supply chain systems 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 23, 29 3. Effective global collaboration to improve long-term availability of health commodities 22, 28 In addition to the unequal distribution of indicators among the three Objectives, an examination by the review team of the individual indicators identified potential confusion with 26 of the 29 “core” indicators. These issues, along with clarifications, appear as Annex VII. 1.4 Processes for calculating key indicator results (OTIF and OTD) appear to be sufficient. Two PowerPoint presentations were provided to the review team that explained how GHSC-PSM currently calculates OTIF, OTD, and TLC.24 These two presentations (“How to Calculate M&E Indicators from the Performance Dataset with Pivot Tables: Last Updated: 1 August 2019,” and “Calculating Total Landed Cost”) were examined by the review team and appeared to cover all of the necessary information for calculating these key indicators in conjunction with the previously mentioned list of variables for the Performance Dataset (see Annex VI). The review team verified the results that GHSC￾PSM had both previously and currently presented, in terms of what was officially reported and the actual results (that is, additional proofs of delivery [PODs] may arrive that would allow line items to be included in the given quarter, but were not available at the time GHSC-PSM provided its quarterly report) for OTIF and OTD. There was no material difference between the calculations that GHSC-PSM provided and those done by the review team. 23 Outcome indicators include: 11, 12, 13, 14, 17, 18, 19, 20, 21, and 26; output indicators include 15, 16, 22, 23, 28, and 29. 24 The review team did not examine TLC as part of this sub-question because, per GHSC-PSM correspondence, there was an issue at the time of the review that was causing errors in the price fields of the Performance Dataset (GHSC-PSM was working to correct the problem). Additionally, TLC costs form the primary basis of findings for Question 3, and are examined in that section. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 13 The only item of note, as shown in Annex IX, “Graphs of OTIF and OTD Results,” is that for TO3 (FP/RH) there were time periods when the reported percentage exceeded the actual percentage. This is unusual because typically it would be expected that actual percentages would increase as purchase orders (POs) are fulfilled within the three-week time frame of the OTD and OTIF indicators, as POD documentation is received from destinations too late for inclusion in the reporting. USAID respondents noted that there had been prepositioning of TO3 at the beginning of the project to avoid gaps in commodities during the transition. Although the processes for deriving these key indicators, as noted, appear to be sufficiently robust, there is concern that the knowledge for how they are calculated may not be sufficiently institutionalized within the GHSC-PSM M&E Team. Currently, it appears that the primary responsibility rests with only a few staff within GHSC-PSM and, given the sensitive nature of these indicators, if there is specific GHSC￾PSM staff turnover, there may be delays in reporting. The Performance Dataset is a pre-defined Cognos25 report that serves as the definitive data source for select M&E indicators. The report includes: 1) order and delivery data at the line-item level; 2) all uncancelled line items for the life of the project; 3) all purchase orders, distribution orders, and replenishment orders; 4) line items on hold; and 5) a number of other relevant variables (see Annex VI for a complete listing of all fields). The dataset is updated twice daily, at 6 a.m. and 7 p.m., Washington, D.C., time, and access is restricted to users who receive training from the M&E team and approval from the M&E manager. There were, at the time of the review, 42 users with access to the Performance Dataset, of which two were USAID users. Approximately three-fourths of the other 40 users were trained in the dataset, and the remainder were MIS users with access to administrative functions. 1.5 GHSC-PSM has in place sufficient internal field data validation processes; however, these internal processes do not negate the need for external data quality assessments. GHSC-PSM provided the review team with an Excel file that describes, for each indicator, how GHSC￾PSM reviews and validates data submitted by the field and that are primarily reported against Objectives 2 and 3 (e.g., stock-out rates at service delivery points [SDPs], logistics management information system reporting rates, and supply chain workforce turnover rates). This checklist further describes the sources, timelines, and processes for the validation, as well as any items of note for the various indicators. Per correspondence with GHSC-PSM, field office M&E staff submit completed Excel data tables to the Washington headquarters M&E team by the seventh day after the quarter’s end. Headquarters M&E staff then review the data and send feedback to the field offices that they backstop. Once headquarters approves the data (via email), the field office M&E staff upload them to DevResults, and the data are considered final. It takes about 2-3 weeks to complete the quarterly process for all countries. Additional, basic information is given in the GHSC-PSM M&E plan. The review team found no major issues with this process, although the current M&E team staff backstops between three to nine countries per person for up to all four TOs. This human resource situation may be a risk to GHSC-PSM’s monitoring efforts.26 25 Cognos is a cloud-based business intelligence and analytics solution that provides data analysis and visualizations. 26 Additionally, while not a method for verifying field data, in countries where there is no LMIS, or it is very weak, GHSC-PSM and USAID rely on the end-use verification (EUV) to provide information on stock availability at SDPs. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 14 USAID has not as yet conducted data quality assurance (DQA) for GHSC-PSM, and it should be noted that, per ADS 201: “Missions and Washington OUs (operating units) must conduct a DQA for each performance indicator reported to external entities. This includes all indicators reported in the PPR27 or other external reporting. The DQA must occur after data have been collected on a new indicator and within 12 months prior to the new indicator data being reported. A DQA must be conducted every three years thereafter…Missions and Washington OUs may choose to conduct DQAs more frequently if needed.” 1.6 Per the contract, GHSC-PSM is obligated to develop an IDIQ-level M&E plan. However, USAID and GHSC-PSM reported flexibility in field-specific M&E plans. Within the IDIQ, GHSC-PSM is instructed to build the local capacity for “monitoring and evaluating policy and strategy implementation for commodity security in both the public and private sectors and explore application of technologies or other innovations to further such policies,” but it is only obligated to develop an IDIQ Performance Monitoring Plan. Furthermore, it is only required to include key performance indicators related to: 1) procurement, freight, and logistics; 2) inventory management; and 3) systems strengthening activities. Noticeably absent is any requirement related to Objective 3. There are no additional references to the development of M&E plans within the individual task orders: TO1 (HIV) mentions “monitor” once in reference to ordering commodities; TO2 (malaria) uses “monitor” to discuss risk management; TO3 (FP/RH) uses “monitoring” in reference to order management, the procurement planning and monitoring report (PPMR), and commodity security; and TO4 (MNCH) makes no special reference to monitoring. The GHSC-PSM IDIQ and the TOs are heavily reliant on buy-in funding from USAID field missions, and while there is no reference in these contracts to developing field-specific M&E plans, 32 countries have country-specific M&E plans, of which 20 include country-specific indicators.28 Nevertheless, during field mission interviews, a few activity managers said they felt that GHSC-PSM was not providing sufficient assistance with developing country M&E plans. 1.7 Currently, under 1 percent (.82 percent) of GHSC-PSM’s programmatic budget is spent on monitoring and evaluation (or 2.96 percent when commodities are excluded). The review team requested that GHSC-PSM provide the percentage of its budget spent on M&E activities in order to determine whether there was sufficient funding for adequate USAID reporting requirements and other corresponding activities.29 This request was originally made at a meeting with 27 Performance Plan and Report. 28 Countries with specific M&E plans include: Angola, Burkina Faso, Burma, Cameroon, eSwatini, Ethiopia, Ghana, Guinea, Haiti, Indonesia, Kenya, Lesotho, Malawi, Mozambique, Nepal, Nigeria, Pakistan, Rwanda, Uganda, and Zimbabwe. 29 USAID’s ADS 201 only guidance in terms of what the target for ME&L spending should be is: 1) There is always a trade-off between the cost and the quality of data. Field missions and Washington operating units (OUs) should balance these two factors to ensure that the data used are of sufficiently high quality to support management needs. 2) OUs should devote approximately 3 percent of total program funding to external evaluation, on average. This may include a mix of both required and non-required external evaluations. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 15 the GHSC-PSM M&E Team on May 10, 2019. A complete response was provided on July 29, 2019, via email correspondence. The reported percentages utilized the amount actually spent (not budgeted) to date on M&E and, as such, excluded FY 2019 figures. The denominator included either total programmatic spending including commodities (resulting in the .82 percent calculation noted above), or total programmatic spending excluding commodities (resulting in 2.96 percent). The numerator primarily included the following: 1) Home office (Arlington, VA) M&E costs: a. All M&E team costs of the Communications, Learning, Evidence and Analytics for Results (CLEAR) team; b. Data analytics headquarters team costs; c. Global Supply Chain (GSC) ARTMIS data quality team; and d. Focused TO1 (HIV), TO2 (malaria), TO3 (FP/RH), and TO4 (MNCH) M&E-related activities (e.g., the Contraceptives Securities Indicators Survey, Contraceptive and Condoms report, TO1’s data initiatives, and ad hoc requests for data and reporting). 2) Field-based M&E costs: a. Routine GHSC-PSM M&E; b. Data quality assurance activities; c. End use verification surveys; d. Electronic logistics management information system (eLMIS) support; and e. Support to ministries of health (MOHs) to build capacity and collect data. At the time of the review there were five M&E staff (one M&E manager and four M&E specialists), with two additional positions in the process of recruiting and/or hiring. While the M&E staff have specific roles and responsibilities attached to their positions (country backstopping, specific M&E activities, etc.) the overall responsibility for data governance, according to stakeholder feedback, is more diffused among the CLEAR unit, its organizational sub-units, and some sub-units outside of CLEAR. The general M&E organizational structure is illustrated in Figure 3 below. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 16 Figure 3. GHSC-PSM M&E Team Reporting Structure Source: Taken from GHSC-PSM organigram, 2019 1.8 GHSC-PSM is mainly focused on monitoring activities. Evaluation activities (either internal or external) are limited, and learning opportunities are recent, primarily internal to the project, and, to date, fairly ad hoc. The review team was presented with a list of M&E activities that GHSC-PSM undertakes. This list, in addition to day-to-day headquarters and field office operations, included: 1) Quarterly reporting (e.g., country data validations, GSC reporting, OTD reporting, calculating impact numbers, cost savings, and dwell time); 2) the creation and use of dashboards and monitoring tools; 3) the creation and use of Vender Scorecards; 4) the monitoring of ARTMIS data quality; 5) work done by the Data Analytics and Knowledge, Management, and Communications (KMC) teams; 6) Data for Accountability Transparency Impact Monitoring (DATIM) reporting; 7) DevResults database management; 8) development and implementation of end-use verification surveys; 9) development and implementation of National Supply Chain Assessments; 10) piloting the Drugs out of Range (DOOR) System; 11) undertaking Zika surveys; 12) monitoring contraceptive security indicators; 13) preparing for audits; 14) undertaking data quality assurance; and 15) preparing and implementing other in-country surveys. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 17 What is noticeably missing from the above list appears to be any evaluation activities.30 While there is no specific USAID guidance for how much funding should be budgeted for ME&L in a specific project,31 some organizations such as the United Kingdom’s Department for International Development (DFID), the World Bank Group and the Global Fund recommend between 5–10 percent for M&E32,33. Further, per stakeholder feedback, learning from the enormous amount of data and information that GHSC-PSM is generating has only recently been emphasized, with plans to include some learning activities in the fiscal year (FY) 2020 work plan. At the start of the GHSC-PSM Project, there was an even smaller M&E team, and data/information requests from USAID were more ad hoc and informal (i.e., one-to-one communication outside of the CLEAR/M&E team’s knowledge). Now, however, data and information reporting procedures and guidance have become more formal and standardized with the CLEAR/M&E team, serving as the nexus for vetting all incoming requests and external communications regarding data and results. Likewise, for USAID, most data and information requests, and any involving the use of project resources, come via the TO Directors/CORs. Similarly, there are have been bi-monthly meetings between the GHSC-PSM M&E Team and USAID M&E counterparts starting in Year 1 of project implementation and, per both sets of stakeholders, these meetings have been effective for discussing both short-term challenges and more long-term strategic issues (e.g., moving M&E from simply reporting to more of a management tool function). 1.9 Per stakeholder feedback, the reporting lines between GHSC-PSM and USAID have improved considerably during the life of the project. The review team found positive working relationships between GHSC-PSM and USAID, especially at senior levels on both sides. Respondents said that lines of communication between CORs and USAID/GH leadership and the GHSC-PSM senior executive team demonstrate mutual respect and improves the efficiency of implementation. As might be expected, however, in such a large project with so many USAID managers and technical advisors involved (approximately 50), working relationships among technical staff in USAID/GH and GHSC-PSM are not uniform. Several USAID/GH respondents said GHSC-PSM does not react quickly enough to what they want done, nor does GHSC-PSM pro-actively anticipate a respondent’s needs or propose alternative scenarios. Nonetheless, other USAID/GH respondents said that the project is a very useful partner and USAID has accomplished a lot within the relationship; all respondents gave examples of positive outcomes of their work together. 30 Per USAID’s ADS 201, “Evaluation is the systematic collection and analysis of information about the characteristics and outcomes of strategies, projects, and activities conducted as a basis for judgments to improve effectiveness, and timed to inform decisions about current and future programming. Evaluation is distinct from assessment or an informal review of projects. The purpose of evaluations is twofold: to ensure accountability to stakeholders and to learn to improve development outcomes.” 31 Per ADS 201, “OUs should devote approximately 3 percent of total program funding to external evaluation on average. This may include a mix of both required and non-required external evaluations.” 32 The Monitoring and Evaluation Handbook, IFC Advisory Services in association with the Deutsche Gesellschaft für Internationale Zusammenarbeit (GIZ; German Corporation for International Cooperation) and DFID, June 2018. 33 Guidance for submission of an M&E plan for Global Fund grants, the Global Fund. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 18 A recurring comment from both USAID and GHSC-PSM respondents, although working on varied aspects of the project, is that they each would like to do more brainstorming and exchanges of information with the other side. Some respondents said this brainstorming can occur in technical working groups; several groups, such as the M&E working group, hold regular meetings. More generally, GHSC-PSM respondents noted the improvements in communication and prioritization of tasking from USAID. QUESTION 2: HOW HAS THE GHSC-PSM PROJECT ADDRESSED RISKS, BOTTLENECKS, AND/OR INEFFICIENCIES IN THE GLOBAL SUPPLY CHAIN SYSTEM? 2.1 GHSC-PSM supply chain operations have evolved, performance has improved, and risks and bottlenecks have been reduced. By all accounts, GHSC-PSM had a challenging start due to numerous issues involving the contract and availability of data. GHSC-PSM has improved significantly since project inception in the key areas of supply chain performance to mission, operational efficacy, supply chain cost savings, bottlenecks, and risk reduction, and in day-to-day visibility into supply chain flows, transport, potential bottlenecks, inventory positions, and supplier capacity. Today, the GHSC-PSM supply chain is fully functional as an operational supply chain. The GHSC-PSM supply chain operation is mapped out in Figure 4 (below). As the diagram illustrates, current GHSC-PSM supply chain operations focus on the activities outlined on the right side of the diagram, such as demand and supply planning, commodities procurement and sourcing, demand fulfillment, and transportation management. GHSC-PSM, for the most part, manages the flow of commodities to a country port of entry. Typically, the in-country ministry of health then takes over. In most cases, GHSC-PSM does not manage the in-country supply chain; operationally, this remains a ministry of health responsibility. GHSC-PSM does serve as a technical advisor in many field mission countries, as discussed elsewhere in this report. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 19 Figure 4. GHSC-PSM End-to-End Global Supply Chain Source: GHSC-PSM in-brief, 2019. Broken down more granularly, GHSC-PSM’s supply chain management approach, processes, and tools involve the capabilities and functions diagrammed in Figure 5 (below). The GHSC-PSM “hand off” of goods to respective ministries of health at the ports of entry differentiates the current GHSC-PSM supply chain model significantly from commercial (e.g., pharmaceutical manufacturers) supply chain models, in which the commercial entity controls the supply chain end-to-end (all the way to the patient). The hand off of control is a structural and political issue that varies by country; it lies out of the direct control of the GHSC-PSM supply chain operation. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 20 Figure 5. Global Supply Chain Commodity Management Source: GHSC-PSM in-brief, 2019. At present, GHSC-PSM calls its current supply chain operational model “GSC 3.0.” GHSC-PSM’s organization and operational structure has evolved, and continues to evolve significantly since the project inception. This organizational evolution is critical to GHSC-PSM’s improved performance in the areas of risk management, efficiency, cost savings, and overall performance to supply chain objectives. It should be noted that in the private sector, GHSC-PSM would be considered a highly rapid start-up, requiring learning “on the fly” and a highly adaptive business process culture. The GHSC-PSM current organization structure reflects the myriad demands and activities required of GHSC-PSM in execution of its chartered duties. The project team has matured over more than 3.5 contract years to address the complex needs of the commodities supply chain across multiple task orders and many countries. GHSC-PSM has made a significant investment in human resources, standard operating procedures development and refinement, training and functionality add-ons, and data visibility and analytics capabilities, to meet the needs of the myriad task order supply chains, supplier issues, country and regulatory requirements, and so on. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 21 2.2 GHSC-PSM follows the SCOR model and layers on certain “strategy” capabilities. The SCOR model brings process rigor to the entire supply chain, thereby enhancing performance. The SCOR model delineates the processes that a supply chain typically follows. It was originally designed for commercial supply chains, but has now been widely adapted to suit public sector requirements (e.g., the U.S. Department of Defense). The SCOR model, a process framework for defining, analyzing, and improving supply chain performance, includes defining or mapping the supply chain, measuring its performance, and setting improvement targets. The model focuses on supply chain operational areas and follows a linear progression on the six activities noted below. To a large extent—with the exception of “make” or manufacturing a product—GHSC-PSM follows the SCOR model: 1. Plan 2. Source 3. Make (not relevant for GHSC-PSM) 4. Deliver 5. Return/destroy 6. Enable, measure, and improve The SCOR process model is founded on the tenet of continuous improvement. This requires continuous self-monitoring, evaluation, metrics, and performance analysis; there is no end to the inquiry into and execution of continuous improvement. GHSC-PSM shows every indication of adhering to this core tenet of the SCOR model, as evidenced by its steady performance improvement in key metrics such as OTIF, OTD, and TLC. GHSC-PSM continues working to improve within each of the relevant SCOR attribute areas (#3 excepted). Process improvements in each of these areas are designed to address the key issues raised in Question 2: risks, bottlenecks, and inefficiencies. Everything in global supply chain management is focused on three things for public sector supply chains: Operational performance to mission, risk management, and cost/budget management. Balancing the cost-benefit tradeoffs and making operational and strategic decisions based on those trade-off analyses is essential, and requires an overarching supply chain strategy to do so. The current GHSC-PSM supply chain organization structure reflects the adoption of the SCOR model and the clarity it brings to supply chain operations and organization structure. The organization is aligned around the SCOR functional process areas, with sub-alignments focused on TOs and key processes such as contracting and quality assurance. This structure aligns with the maturation curve of any supply chain, commercial or public. This organizational and operational structure has matured and become more effectively and clearly organized over time (see Figure 6 below). This organizational clarity is seen in the “GSC 3.0” version, the third evolutionary stage, as well as in a clearer organization structure around the TOs and supporting functions, which mirror the SCOR tenets. GHSC-PSM’s rationale for changes centered on inadequate staffing by TO and/or function, poor visibility of orders coming in from multiple teams, extended delays due to multiple hand offs, and two challenges mentioned by several respondents: the lack of SOPs (resulting in bottlenecks and inefficiencies) and, in the first year, the tendency to make commitments that could not be kept. Evolution of GSC from project start to current state Organization Chart • By SCOR function (Plan, Source, Deliver) managed by a Functional Manager • Plan • Source • Deliver Return • Quality • Procurement Teams (By TO and EssentiaI Medicines • Deliver Return • By commodity managed by a Task Order Integrated Supply Chain Manager • Supplier Relations • QA • Plan • Strategic Sourcing • Processing an RO was done manually • Processing an RO to actual delivery involved handoffs between multiple: teams • Procurement Specialist owns RO from receipt to delivery with DR responsibility for logistics Technology • First order was received in April 2016 and managed manually • ARTMIS 1.0 was deployed July 2016 and was used for processing transactions • ARTMIS 3.18 along with a suite of analytical tools are currently used to process transactions Performance • OTIF was 7% in Q2 2017, 368 line items • OTIF was 32% in Q3 2017, 955 line items • OTIF is at 83% in Q2 2019, 1,594 line items USAID Global Health Supply Chain Program GSC 1.0 Project Start – June 2016 GSC 2.0 July 2016 – September 2017 GSC 3.0 October 2017 – Current • Client Relations • Sourcing • Fulfillment • Deliver Return • By Task Order and by Function managed by a Functional Manager • Supplier Relations • QA • Plan • Strategic Sourcing Process GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 22 Figure 6. Evolution of GHSC-PSM’s Global Supply Chain Source: GHSC-PSM in-brief, 2019. The GHSC-PSM evolutionary maturation of organization structure and staffing led to improved performance, as evidenced by the improvement in OTIF and OTD, included the following key elements: ● Improved efficiency of staff structure by function and TO. ● Instituting a single owner, i.e., a procurement specialist, for orders from receipt to delivery, with divided responsibility from various teams. ● Fewer handoffs between teams. ● Increased focus on transaction management and SOPs across various functions. ● Development and use of analytical tools to manage performance. Regarding the “Plan” activity area, GHSC-PSM’s purpose is to “aggregate and analyze demand signals from country programs and supply signals from manufacturers to inform procurements and inventory decision-making.” As an example, PEPFAR countries had to rapidly transition to tenofovir disoproxil fumarate/lamivudine/dolutegravir (TLD) with a limited supply base and changing demand. To accomplish this, USAID and GHSC-PSM coordinated to balance dynamic country demand with GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 23 manufacturing capacity and to ensure timely delivery. By the end of 2018, the project had delivered more than 11.7 million units of TLD to 12 countries. Even with existing products, the introduction of multi-month dispensing (MMD), with its increased packing density of 90 to 180 tablets for 3 to 6 months, shows promise for further efficiency gains while better serving consumers. The “strategy” activities adopted by GHSC-PSM, which are layered on top of its SCOR process model activities and improvements, focus on such activities as understanding market drivers, developing sourcing strategies, establishing long-term framework contracts, and building strategic relationships with suppliers. The goal is to solidify supplier relationships long-term and optimize sourcing and supply chain management where possible. Tools developed under GHSC-PSM to assist in this process include: the order promising tool and freight estimate calculator for planning, the requisition order pipeline dashboard for order management, and the not-OTD tool for continuous improvement. The GHSC-PSM work on market dynamics, strategic sourcing, and supplier relationship management activities is assisted by market intelligence, product life cycle analysis, “should cost” models,34 and supplier score cards. 2.3 GHSC-PSM follows the SCOR model but lacks an overarching strategy for pursuing predictive/proactive supply chain management. SCOR and all best practice supply chains have one thing in common, regardless of being public or private; they have an overarching supply chain strategy that drives the operating strategy. SCOR is an industry best-practice supply chain operations model. The key word is “operations;” SCOR is an operational process model only, not a strategy model. Strategy, and the processes and tools it enables and funds, allow a supply chain to move from strictly reactive to a more proactive and predictive model. That is one of the objectives indicated as important by USAID and GHSC-PSM. The SCOR model alone does not constitute a roadmap toward a more proactive/predictive supply chain. The SCOR model works best when combined with the Gartner Supply Chain Maturity Model.35 Best practice global supply chains use both, to great success. Unilever, for example, committed to adopting both, and within a decade became the number one-rated performing supply chain in the world (Gartner Annual Supply Chain Top 25 Annual Ranking). The Gartner Supply Chain Maturity Model (see Figure 7, below) defines how supply chains typically evolve, from simple, siloed, and reactive-only supply chains (still the most common level of supply chain maturity throughout public and private sectors) to extended, highly orchestrated supply chains that are big-data analytics driven, proactive, and even prescriptive (meaning they understand the market so well that they can not only predict and anticipate what will happen, but can influence what will happen).36 The Gartner Supply Chain Maturity Model is regarded as the accepted maturity model for the evolution of both private and public supply chains worldwide. The model has five stages: 34 That is, models that assist in understanding what a project should cost. 35 Note that the GHSC-PSM contract does not specify nor did USAID ever specify that the project would be assessed against the Gartner Model. 36 For more information on the Gartner supply chain maturity model, see https://www.gartner.com/smarterwithgartner/5- stages-of-logistics-maturity/. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 24 Stage 1- React: Autonomous departments are driving logistics priorities via manual processes and disparate, disconnected systems. The supply chain reacts to events that have already occurred or are occurring that day or week. Stage 2 - Anticipate: The logistics function is beginning to be centralized to improve efficiency and productivity, which starts to enable it to anticipate events, issues, problems, etc. Operational and organizational silos still exist but are beginning to break down. Stage 3 - Integrate: Focus on integrating the various logistics functions and activities into the overall supply chain, and tying that integrated system into the function of the organization. Elimination and amelioration of functional and supply chain product-based silos. Stage 4 - Collaborate: The supply chain is well under control internally; at this stage the organization starts to look outward—to collaborate with suppliers and customers using visibility and analytic tools, shared forecasts, plans, etc. Stage 5: Orchestrate: Logistics and the rest of the supply chain orchestrate processes across an ecosystem of partners, end to end, to capitalize on business or other opportunities. Information flows across the supply chain network in real time, enabling broader visibility and timely, fact-based decision￾making. Figure 7. Gartner Supply Chain Maturity Model Source: Gartner, Inc., 2018. Note: BU = Business Unit. GHSC-PSM organizes and operates its procurement supply chain around the basic tenets of the SCOR model. However, as noted, the SCOR model is a functional supply chain process map, not a strategy. The GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 25 Gartner model provides a basic roadmap for evolving a supply chain, but it does not provide the strategy for doing so. That responsibility falls to the supply chain “owner” organization. 2.4 An overarching strategy can optimize multiple supply chains. There are evidence-based approaches relevant to GHSC-PSM. An overarching supply chain strategy includes a mission statement, a “guiding light” around which all supply chain activities are centered and operate. It also includes a strategy for the management of the supply chain, with consistent protocols, objectives, conventions, etc. An overarching strategy would enable all stakeholders in the supply chain, i.e., USAID/GH and GHSC-PSM, to have a clear vision for operating the supply chain, making informed trade-off decisions, managing service and costs, deciding on problem resolution strategies and tactics, and more. For GHSC-PSM, USAID would determine and maintain the overarching supply chain strategy, with input from and collaboration with GHSC-PSM. USAID would be the “owner” of this strategy. This approach is a staple in the commercial sector, for Apple, Unilever, Pfizer, Caterpillar, Intel, Microsoft, and others. The strategy enables an organization to collaborate with its execution third-party partners to achieve the desired goals. How does this strategy apply to organizations with multiple supply chains? Most global enterprises operate multiple supply chains. These multiple supply chains may be regionally based, product-type based (most common), or “division” based (i.e., business units). Studies show that most large, multinational enterprises have 10 to 25 different supply chains. The “best￾in-class” organizations have half that number. For example, when the Hewlett-Packard Company (HP) and Compaq merged nearly two decades ago, HP analyzed the resulting combined supply chain and found there to be more than 20 discrete and highly duplicative supply chains. That meant special warehousing, special transportation, special inventory management, and special procurement, with no process commonalities, no aggregate procurement of product or services, duplicative and parallel transportation, duplicative and parallel warehousing, sourcing, and the like. There was also no cross￾supply chain visibility in order to optimize activities or investments. The result was high cost, slow or no agility, poor visibility, and loss of competitiveness. By combining commonalities among the 20-plus supply chains, HP was able to winnow its number of discrete supply chains down to 5, saving hundreds of millions of dollars and improving service and agility. USAID and its partner GHSC-PSM essentially operate discrete supply chains by task orders. Each TO has certain, unique requirements, much like discrete product lines in a computer company. However, the discrete supply chains also share core processes that are common, regardless of the TO. For example, best practice supplier procurement and contracting are similar across TOs. Transportation contracting can be aggregated and systematized for lower rates, as GHSC-PSM has done. The U.S. Department of Defense and the National Aeronautical and Space Administration (NASA) are studying this “commonalities” model in an effort to bring down total acquisition and operational supply chain operating costs, while improving service and reliability. The “commonalities” model also reduces risks such as supply chain failure, counterfeiting, intrusion, and theft. In addition to having an overarching supply chain strategy, many commercial entities have a chief supply chain officer (CSCO) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 26 who oversees all the discrete supply chains. This CSCO reports directly to the company president (or equivalent of the organization) and has complete decision-making and budgetary authority to govern all the supply chains/divisions (or whatever terminology is used). These best-in-class supply chains also systematize and standardize supply chain processes across the extended supply chain. Table 5 (below) provides examples of how GHSC-PSM compares. Understandably, federal and commercial supply chains operate differently, and a CSCO may not be feasible in the former. However, a supply chain governing body certainly is possible and worthy of consideration. Regardless of the management overlay structure, federal supply chains have the opportunity to consolidate and systematize discrete and often-times “one off” supply chains under an aggregated umbrella, like a chief supply chain office model or a synchronized model. This group management model does not preempt strategy for, in USAID’s case, individual task orders. It merely sets overarching goals or the mission with task order and partner input on supply chain strategy and objectives overall. Table 5. Best Practices and the USAID/GHSC-PSM System Industry Good Practice USAID/GHSC-PSM Processes Clear mission and operating guidance Multiple missions and operating guidance Elimination of “maverick” decision-making and practices that confuse partners and create inefficiencies and workarounds Multiple decision-makers and multiple processes that may change depending on the situation, the stakeholder, budgetary delays, or funding changes Get out of the “firefighting mode,” i.e., knee-jerk reactions; lack of proactive and even predictive supply chain management Supply chain is managed on past and day-to-day developments as collected in ARTMIS Improvement in all seven performance attributes listed as critical in the SCOR model Consistent improvement over time in managing SCOR attributes across the pertinent process groupings as discussed earlier, e.g., planning, procurement/sourcing, supplier management, deliver, enable, measure The current GHSC structure is designed to protect the objectives and processes of individual offices or initiatives (PEPFAR, PMI) and not to operate in concert, in large part due to how TOs are funded. Because TOs are funded separately, they may opt to operate their supply chains differently and with different resources. Respondents familiar with these differences said that this can result in duplicative activities, specialized reporting and process management, and different transportation and warehousing procedures. 2.5 ARTMIS capabilities need to be more predictive. ARTMIS is the MIS system that GHSC-PSM uses to manage the day-to-day operations of its procurement/supply chain operations, and is an essential tool. Funding any future investment in ARTMIS, such as building out predictive/proactive analytics capabilities, is an issue that requires resolution by USAID and GHSC-PSM. USAID respondents explained that funding for ARTMIS included a GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 27 requirement for predictive/proactive analytics capability, and these capabilities are built into ARTMIS. Nevertheless, according to USAID feedback, they were not being fully utilized at the time of the review. Essentially, to move toward a more proactive supply chain requires better analytics capability in the MIS solution: that is, ARTMIS.37 As defined by GHSC-PSM, ARTMIS is “the procurement system that GHSC￾PSM uses to place, manage, and track orders, as well as manage sourcing and contracts activities.”38 The system tracks average monthly cycle times, average daily order throughput, lead times, order delays, on-time deliveries, and other pertinent, procurement-related information. It provides an at-a-glance order situation dashboard, as well as explicit order information and issues on the detail tabs. ARTMIS’ data reliability has improved steadily over the term of the contract, as has its ability to handle rapidly growing levels of procurement volumes. Evidence of this data reliability shows up in the increased performance levels (OTD and OTIF) of GHSC-PSM. Those performance improvements could not have been achieved without better quality, more comprehensive data. While the review team does not include contract expenditure information in this report, it should be noted that expenditure for ARTMIS reached a high in FY 2017, and has since declined. GHSC-PSM respondents associate this reduction to the completion of the build-out, and in the subcontracting of some of the technical operation to India. ARTMIS transaction volume capabilities have improved. In FY 2018, ARTMIS processed $888.7 million in procurements, a 33.5 percent increase over FY 2017. It also managed delivery of $742.6 million in health commodities, a 79.1 percent increase over FY 2017, according to GHSC-PSM’s 2018 annual report. However, USAID/GH respondents said they expected at this juncture that ARTMIS would have more proactive supply chain management capabilities, as well as more analytics capabilities. Operationally, ARTMIS shows the current status of every order in the system. It does have some proactive capabilities, such as forecasting a supplier stock-out based on production runs and capacity. These are noted in the system, and based on the information, the GHSC-PSM team decides on action plans. For example, a shipment of product can move, only to be held up at a port for 6 months for customs clearance. ARTMIS shows this as a delay, but is unable to anticipate it (assuming there are pattern trends to do so). ARTMIS currently is a fully functional, descriptive, supply chain MIS solution. This means that it tracks supply chain activities and events on a day-to-day, current state basis. Such a descriptive MIS solution is essential to managing a global supply chain operation. It has minimal predictive and analytic capabilities, which are a hallmark of more sophisticated, large, data-based supply chain MISs. Such capabilities can spot and even anticipate trends and supply chain patterns, enable “what if” scenario planning, run simulations, optimize for future requirements, and anticipate supplier issues or market demand variances. Predictive analytics capabilities enable a supply chain to get out ahead of its day-to-day operations, as opposed to constantly operating in a fire-fighting, day-to-day mode. 37 USAID has acknowledged that it does not own the ARTMIS system, considering the IBM software utilized within. 38 In-brief for this review. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 28 ARTMIS is a transactional system that still requires much manual labor and manual data inputting, such as inputting by GHSC-PSM staff from Excel spreadsheets.39 It also requires daily order triage meetings at 4 p.m. to review the ARTMIS dashboard and triage any issues that have occurred or could occur, a highly labor-intensive and reactive exercise.40 It is very important to stress that this kind of descriptive supply chain MIS, which requires significant manual input, is not unusual in either the commercial or public sectors. 2.6 GHSC-PSM development of its continuous improvement program shows significant progress. GHSC-PSM has invested in continuous improvement (CI) since the project’s inception. Early on in the project’s lifecycle, this CI was an operational necessity as part of the rapid operational ramp-up and learning curve. As such, CI efforts were largely manually derived process improvements. GHSC-PSM has had a quality assurance team in place since the project’s early days. This team is charged with coordinating quality assurance activities for various commodities, conducting root cause analyses on issues that arise in implementation, developing and tracking corrective and preventive actions to closure, and related tasks. CI exists on many levels within GHSC-PSM. One key example is the development of a solutions application to automate supplier performance reporting, tracking, and management. This supplier performance reporting tool is essential to holding suppliers accountable for their OTD and OTIF performance, among other metrics. At the time of the report, GHSC-PSM was in the early stages of building a CI software application that eventually will incorporate components such as SOPs, workstream management using Lean Six Sigma,41 corrective action templates, and key performance indicators (KPIs) to track activity and responsibilities on a weekly basis. Under development earlier in the project, and launched in November 2018, the initial focus of the CI MIS effort was supplier performance, theft, and other repeated actions. The CI team also now has an MIS platform that staff can use to submit issues, such as damaged shipments or theft in a country. The CI team takes the following problem-solving approach: 1. Identify the issue and the root cause. 2. Agree on an action plan. 3. Assign an action owner. 4. Decide on an action completion date and milestones. 5. Address the issue and close out the problem. 6. Track frequency and severity; remediate. 39 GHSC-PSM managers note that some “tools are fine-tuned in Excel prior to being programmed into ARTMIS” and that “the freight estimator calculator has now been incorporated into ARTMIS and the OPT is expected to be incorporated by the end of Q2 2020.” 40 While GHSC-PSM does see the benefits of adding AI and Machine Learning, one project expert notes that “will require increased maturity in the upstream supply chain activities, which could be a long-term effort.” 41 Lean Six Sigma in supply chain management is an approach that seeks to improve the quality of process outputs by identifying and removing the causes of defects (errors) and minimizing variability in manufacturing and business processes. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 29 The application “pings” the action owner/manager regarding upcoming milestones or behind-schedule duties; it also has an escalation protocol in case more senior management members need to be notified and/or get involved. If the incident reoccurs, the continuous improvement team reviews the root cause again to assess whether something changed or they missed a driving element. In 2019, the CI team focused primarily on suppliers and supplier quality assurance. The CI application was initially built to deal with day-to-day supplier issues. It is migrating to be a more forward-looking system, like learning to spot potential supplier issues before they become a day-to-day problem. GHSC-PSM or sub-contractor staff can enter an incident into the system, such as from a regional distribution center or a field office. The system uses business intelligence to predict future likelihood of occurrence, for instance, how many more times a laboratory’s suppliers will be late with shipments. This information enables GHSC-PSM to manage the supplier more closely and build in fact-based discussions with the supplier to correct the issue. The system is supporting the migration to performance-based contracting with laboratory suppliers, i.e., providing “up time” versus equipment or supplies, “pieces and parts.” It also supports the development of a more proactive supply chain regarding—at least for now—suppliers. As one GHSC-PSM respondent noted, “The system has allowed us to cut down our response time significantly and help close incidents faster. It has automated a process that used to require thousands of emails and massive spreadsheets.”42 At the time of the report, the CI team was integrating its solution, called AssurX, with ARTMIS. Resolving issues using AssurX goes directly to reducing risks and bottlenecks, particularly in regard to the supplier base. By handling supplier issues immediately, and assigning accountability-to-resolution pathways tied to responsible parties, GHSC-PSM can resolve supplier issues sooner, thereby potentially speeding goods to market, reducing potential stock-outs, better anticipating supply shortages, and other related issues. AssurX, as it builds out and gains acceptance, directly addresses improved and faster resolution of supply chain/supplier risks and bottlenecks. Acceptance was slow at first, but, according to interviews, it is rapidly accelerating as GHSC-PSM in￾country staff see the benefits of resolving issues more quickly and with accountability. Figure 8 (second page following) offers a sample view of how the CI application enables the CI team to record and monitor incidents and their resolution. 2.7 USAID and GHSC-PSM lack a common understanding and lexicon for some areas of overall supply chain management, including risk management. The communication processes between USAID/GH and GHSC-PSM surrounding the global supply chain are very granular, slow, and labor intensive. GHSC-PSM operates by the guiding principles of the SCOR model; GH, for the most part, does not. This means that the two organizations think and speak differently when it comes to many supply chain decisions and activities, creating the potential for misunderstandings, confusion, repeated work, and risks. 42 It should also be noted that the U.S. Department of Defense has been utilizing this so-called “performance-based contracting” for more than a decade, with significant success in certain programs and platforms. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 30 For example, USAID is understandably concerned about commodity theft, but thefts do not cause much of a ripple in the lexicon of the SCOR model unless a stock-out occurs. From GHSC-PSM’s perspective, the warehouse is in stock, so there is not an issue. Thus, if a theft problem arises, it is addressed in an ad hoc manner that may lead to poor publicity for USAID, or other unintended consequences. Alternatively, USAID/GH seldom thinks in terms of the SCOR model and its phases of operation, or a supply chain strategy beyond the SCOR model. It thinks about delivering quality commodities on time to the right place. It thinks in terms of overall reliability and even redundancy, to ensure commodity availability, not about the supply chain in terms of optimizing the SCOR processes. GHSC-PSM, cognizant of this, operates a day-to-day supply chain using the SCOR model with a goal of achieving the SCOR objectives. It modifies the SCOR model, taking into account USAID/GH objectives of supply reliability. While the two approaches are not mutually exclusive, there remains some disconnect in how integrated and rationalized the two approaches are. This is a management issue: for example, the communications needed to resolve and rationalize the issues. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 31 Figure 8. AssurX Dashboard Incident Tracker GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 32 QUESTION 3: HOW HAS THE GHSC-PSM PROJECT REALIZED COST SAVINGS AND EFFICIENCY SINCE ITS INCEPTION IN JANUARY 2016, WITH THE CONSOLIDATION OF PROCUREMENT SERVICES UNDER A SINGLE AWARD IDIQ CONTRACT? 3.1 GHSC-PSM’s TLCs improved across TOs between FYs 2017 and 2018; comparisons with predecessor projects were mixed. In 2017, USAID/GH supported a large study to compare key variables in USAID supply chain projects over time.43 For this review, USAID/GH chose a variable—total landed cost—for making comparisons within GHSC-PSM and between GHSC-PSM and predecessor projects. Using project documentation44,45,46 and raw financial data, comparisons were made between GHSC-PSM and the two predecessor projects for TO1 (HIV), TO2 (malaria), and TO3 (FP/RH). Comparisons for TO4 (MNCH) were not possible, as TLC was not reported in the previous study. TLCs were calculated as a percentage of commodity-related and headquarters operations costs over the value of delivered commodities.47,48 The definition of the denominator between GHSC-PSM and predecessor projects was similar—the total value of all line items that were delivered to customers—with actual dates falling within the stated period. The reporting periods used in this review were FYs 2017 and 2018 for GHSC-PSM (The project started in January 2016; however, complete and reliable data were first available in 2017.), and FYs 2014 and 2015 for the predecessor projects. To ensure reasonable comparability, the review team omitted country-specific logistics costs, demurrage, and security costs from GHSC-PSM costs, as they were not included in the calculation of TLC for the predecessor projects. Nevertheless, it should be noted that comparisons are being made between 2 years under GHSC-PSM and 2 years under the predecessor projects that fall at very different stages in the life of their projects: near the beginning of GHSC-PSM, and near the end of predecessor projects. 43 Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES). “Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER,” 2017. 44 GHSC-PSM Quarterly Report, FY 2017, Q4, pp. 113. 45 GHSC-PSM Quarterly Report, FY 2018, Q4, pp. 10. 46 SPACES. “Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER,” 2017. 47 GHSC-PSM calculates TLC using two cost breakdowns: 1) Freight and Logistics, with expense categories of country-specific logistics, demurrage, drop ship freight, inbound freight, insurance, loss, outbound freight, security, and warehousing; and 2) Headquarters Operations, with expense categories of forecasting and supply planning, MIS, monitoring and evaluation, procurement and warehousing, and distribution. See Annex of GSC Systems Strengthening, FY 2019 Q2 Report, 2019-07-26, p. 13 of 250. 48 GHSC-PSM sets targets for TLC (see Annex C in IDIQ Project Monitoring and Evaluation Plan, February 11, 2019, p. 120 or 125), and since 2017 reports out on TLC (for an example, see Annex of GSC Systems Strengthening, FY 2019 Q2 Report.) GHSC-PSM 2017 2018 Total Landed Cost as a % of all the commodities paid during the period TO1 (HIV) 22% 14% TO2 (Malaria) 28% 24% TO3 (FP/RH) 64% 18% SCMS/DELIVER 2014 2015 Total Landed Cost as a % of all the commodities paid during the period HIV 27% 25% Malaria 17% 20% FP/RH 22% 20% GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 33 Figure 9. Total Landed Costs for GHSC-PSM (FYs 2017-18) and Previous Projects (FYs 2014- 15) in HIV, Malaria, and FP/RH As Figure 9 shows, the TLC within GHSC-PSM improved consistently across the TOs between 2017 and 2018, with significant improvement noted for TO3 (FP/RH). 49 For TO3 under GHSC-PSM, USAID explained that the sharp increase in FY 2017 was primarily due to the prepositioning of commodities during the transition from the predecessor project to GHSC-PSM and the associated costs of ensuring that field needs were met, as well as to serve as a buffer for unanticipated demand. Improvements in TLC were not significant for predecessor projects, and TLC actually increased for malaria between 2014 and 2015. The review team did not investigate why the overall TLC for malaria was higher under GHSC-PSM than the predecessor project. In comparing GHSC-PSM TLC results for FY 2017 and FY 2018 with predecessor projects in FY 2014 and FY 2015, GHSC-PSM showed performance improvements over the predecessor projects in TO1 (HIV) but not TO 2 (malaria). For TO3 (RH/FP), TLC performance in FY 2018 performance improved over the predecessor project. 3.2 Unpacking the commodity-related and headquarters operations costs revealed different cost drivers between GHSC-PSM and SCMS/DELIVER projects. Unpacking the total costs delivering health commodities is key for identifying the main cost drivers50 and hence potential cost saving opportunities. These costs include supply chain operations (forecasting and supply planning, procurement, quality assurance, monitoring and evaluation, and information systems), 49 Much higher declines in TLC are seen for TO1 (HIV) and TO3 (FP/RH) compared to TO2 (malaria), for GHSC-PSM. This may be explained by differences in the handling requirements, which could contribute to higher TO2 commodities handling costs. There appears to be scope for more efficiency gains for TO2. 50 A cost driver is the direct cause of a cost, and its effect is on the total cost incurred. Costs in Millions USD $ Global Supply Chain Operations Freight Warehousing TO1 (HIV/AIDS) 2017 $53.9 66% 28% 7% 2018 $56.7 44% 55% 1% TO2 (Malaria) 2017 $18.9 47.3% 52.6% 0.1% 2018 $44.8 13.6% 86% 0.4% TO (FP/RH) 2017 $8.2 62% 17% 21% 2018 $11 40% 56% 4% GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 34 freight (inbound, outbound, and drop ship freight and insurance), and warehousing. Cost drivers for GHSC-PSM in FY 2017 and FY 2018 are shown in Figure 10. Figure 10. GHSC-PSM Total Landed Cost by Key Cost Drivers, TO1 (HIV), TO2 (Malaria), and TO3 (FP/RH) Note: The cost was calculated from the GHSC-PSM Global Monthly Financial Report, IDIQ Level, 2017 and 2018 monthly reports. In FY 2017, with the exception of TO2 (malaria), the cost drivers for the new GHSC-PSM Project were the global supply chain operations costs. In FY 2018, the cost drivers shifted to freight costs across the task orders. The same can be observed in the analysis completed in the SPACES report for the predecessor projects, with freight costs constituting the largest share (Figure 11, below). Costs in Millions (2016 USD) Freight Global Supply Chain Operations RDC Warehousing Other SCMS HIV 2014 79% 11% 7% 3% 2015 76% 14% 7% 3% DELIVER Malaria 2014 82% 12% 4% 2% 2015 69% 19% 7% 5% DELIVER Family Planning 2014 75% 13% 7% 5% 2015 69% 13% 9% 9% GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 35 Figure 11. SCMS/DELIVER Total Landed Cost by Key Cost Drivers Note: SPACES Global Evaluation of USAID Supply Chain Investments through SCMS and DELIVER, 2017, pp. 35. Original data to modify the chart were not available. For GHSC-PSM, between FYs 2017 and 2018, the value of commodities delivered increased by 21 percent for TO1 (HIV), 40 percent for TO2 (malaria), and 60 percent for TO3 (FP/RH). For the predecessor projects, assuming that the value of commodities delivered increased in the comparison fiscal years, the predecessor projects managed to reduce their freight costs, yet the review team did not have value of commodities information for SCMS/DELIVER. 3.3 GHSC-PSM experienced significant commodity savings since the project began. Cost saving on commodities over the project life stood at $89.9 million by the second quarter of FY 2019 for all TOs (Figure 12, below), with the majority of the savings, about $61.5 million, or 67 percent, being on adult antiretrovirals (ARVs).51 However, these cost savings should be interpreted carefully; the review team pulled the data from GHSC-PSM reports and used the GHSC-PSM method that is used to estimate the cost savings. The team observed that: 1) the method used to estimate the cost saving is not clearly enough elaborated to allow independent verification, and 2) the cost saving is reported on a few selected items, with no clear explanation on the inclusion and exclusion criteria thereof; it is just known that the cost savings occurred. 51 The data were pulled from the GHSC-PSM Quarterly Report, FY 2019, Q2. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 36 Figure 12. GHSC-PSM Commodity Savings by Year for all TOs* and TO1 (FYs 2017-2019)** * “All TOs” refers to the TOs that were reported on cost savings: TO1 (HIV/AIDS), TO2 (malaria), and TO3 (FP/RH). ** The data are from the GHSC-PSM Quarterly Report, FY 2019, Q2. Note that TO2 commodity cost savings are reported on an annual basis and are included in the cumulative total of $89.9 million in savings. These data suggest that there are significant cost savings to USAID; however, field activity managers voiced the perception that the GHSC-PSM contract was more expensive to field missions than the predecessor projects. The GHSC finance team in USAID/GH reported hearing similar complaints from USAID field missions; however, neither they nor the review team have data to verify the statement, nor would it necessarily follow that cost saving in the TO supply chains would translate into lower costs to field missions for technical assistance.52 QUESTION 4: HOW HAVE IN-COUNTRY SUPPLY CHAINS PERFORMED IN GHSC￾PSM-SUPPORTED COUNTRIES DURING THE LIFE OF THE PROJECT? WHAT TRENDS ARE OBSERVED? Central to GHSC-PSM’s ability to influence the strength of supply chains is the advice and technical assistance it provides. USAID’s design for in-country technical assistance (TA) drew on a range of in￾country experiences from predecessor supply chain, pharmaceutical management, and other USAID/GH system strengthening and policy programs. The GHSC-PSM contract calls for “an in-country Systems Strengthening Technical Assistance Program” with the following expectations about its objective and structure: Systems strengthening technical assistance will help strengthen in-country supply chain systems and improve health commodity availability with the ultimate goal of promoting sustainable 52 The question on who benefits from the reported cost savings remains outside the scope of this review. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 37 solutions and country ownership. The systems strengthening technical assistance is typically provided from the contractor’s field offices with periodic short-term technical assistance from the headquarters office. Systems strengthening technical assistance needs are identified by USAID Missions and USAID GH offices.53 The IDIQ delineates, under systems strengthening, areas of work in four main areas: 1) Strategic planning; 2) In-country logistics (with 14 subsections); 3) Capacity building, defined as the transfer of skills and technology; and 4) Enabling environment, (with subsections for finance, human resources, governance and leadership, and policy).54 4.1 Initially, technical assistance was overtaken by procurement and was a bit crowded out by in￾country start-ups. GHSC-PSM and USAID respondents recalled that the roll out of the project’s procurement activities took precedence over TA in the initial years of the project, and providing TA got off to a slow start. A presentation provided to the review team characterizes TA as all GHSC-PSM in-country work, along with international (i.e., headquarters-provided) short-term technical assistance (STTA).55 Table 6 below demonstrates the evolution of TA in project reporting. In 2016, in-country TA was used to implement the transition from preceding projects to GHSC-PSM. The project assessed the countries and extant staff to develop STTA needs for the transition; GHSC-PSM also set up four working groups around the immediate needs of the launch during that period. By the end of 2016, the categories of TA were set, and stayed about the same up to the time of the review. However, efforts to get GHSC-PSM field offices up and running continued to require attention. In FY 2017, reporting began on the in-country operations indicators that provided insights into how public sector supply chains in-country were operating (see section 4.5 below). By FY 2018, reporting indicates that several TA activities were underway (e.g., setting up training programs, integrated global positioning system [GPS] capacity in warehousing and distribution in Ethiopia, and support for private sector provision in Ghana), along with the opening of two new country offices.56 53 AID-OAA-I-15-00004, Section C.3, p. 41. 54 Ibid, pp. 41-51; Other TO contracts referred to the IDIQ, with a few clarifications. 55 GHSC-PSM, 2019, System Strengthening Introduction, GHSC-PSM PowerPoint presentation. 56 GHSC-PSM, Quarterly Report Q2, FY 2018, p. 40. Project offices were open in Niger and Sierra Leone as part of a decision by USAID to provide direct supply chain support to these countries. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 38 Table 6. Evolution of In-Country TA Elements GHSC-PSM Contract57 FY 201658 FY 201759 FY 201860 August 201961 Systems Strengthening TA options: Strategic planning • strategic implementation In-country logistics • commodity qualification and forecasting • supply planning • procurement • warehousing and inventory management • distribution and transportation • waste management • quality assurance • pharmaceutical commodity selection • identification of barriers to importation • loss prevention • product recalls • commodity supply chain design • data collection and information systems • construction, non￾structural or cosmetic Capacity building Enabling environments • finance • human resources • government and leadership • policy Accomplishments • Assessed local staff skills and planned for STTA needs over transition • For nine countries, initiated service agreements or secured warehouse space • Cross-TO task forces set up: • Forecasting and supply planning • Targeted local procurement (TLP) • Warehousing and logistics • Logistics management information systems Accomplishments • Field offices grew to 1,064 staff spanning 38 countries • 11 TA elements updated and validated • New teams & procedures to enhance information sharing between GSC and field offices. Provision of TA more than doubled between first and second half of year Areas of TA: • HIV/TB coinfection • Strategy and design • Engaging national malaria control (NMC) programs in LLIN distribution • Procurement Capacity • Forecasting • Warehousing & distribution • Management Information Systems • Workforce Development • Governance & leadership • Laboratory networks Accomplishments • Forecasting and Supply Planning (FASP) reporting for 33 countries • LMIS for 33 countries • Optimized logistics in 31 countries TA provided to 41 countries Areas of TA • Forecasting and supply planning • Governance and leadership • Global Standards • Health-care waste management • Laboratory networks • Management information systems • Process improvement • Procurement • Strategy and design • Voluntary male medical circumcision (VMMC) • Warehousing and distribution • Workforce development Areas of TA offered by GHSC-PSM • Forecasting and supply planning • Warehousing and distribution • Workforce development and enabling environment • Leadership, procurement, and governance • Management information systems • Laboratory networks • Global Standards • HIV prevention 57 AID-OAA-I-15-00004 Section C.3, pp. 40-51, Elements of Objective 2: Systems Strengthening. 58 GHSC-PSM FY 2016 Q2 Report, pp. 28-29, actions by Systems Strengthening team. 59 GHSC-PSM Semi Annual and Quarterly Report, FY Q4 Final, pp. 45-69. 60 GHSC-PSM FY 2018 Q2 Report, p. 20 accomplishments; p. 43, areas of TA. 61 Communication with GHSC-PSM for most current areas of TA at the time of the review. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 39 4.2 GHSC-PSM’s organizational structure evolved to meet the in-country TA requirements of this large, new project. Project Management Units: GHSC-PSM’s organizational structure at headquarters evolved to meet the needs of the field missions and the priorities of the individual TOs. From an original management structure, in which one of three managing directors was responsible for overseeing country programs, GHSC-PSM strengthened staffing of Project Management Units (PMUs) to oversee the work of headquarters-based country and operational staff. USAID/GH and GHSC-PSM respondents said that they found the more robust management structure useful. For example, the strengthened structure allowed GHSC-PSM headquarters to have senior managers participate in or support headquarters staff in country-specific “four corners meetings” among field missions, USAID/GH, GHSC-PSM headquarters, and GHSC-PSM representatives to address in-country planning and implementation issues. Both USAID and GHSC-PSM respondents said the four corners meetings are both necessary and useful to efficiently review in-country issues and develop understanding—and often consensus—among all parties. Non-Field Office PMU: Another organizational change to facilitate in-country work was to establish a PMU for non-field office (NFO) countries, that is, countries receiving commodities from GHSC-PSM but not TA. NFO countries include a range of country situations, such as those with no other TA, e.g., Yemen, Colombia, and Togo. It also includes countries with other TA programs, such as Tanzania and Francophone countries covered under the GHSC Multi-Award TA project, as well as other bilateral or partner country programs. In these countries, GHSC-PSM works with field missions, other TA partners, and often with relevant government entities, but do not have a country office or representative. The PMU also covers ad hoc support for other U.S. Government entities, such as the Office of Foreign Disaster Assistance and the Peace Corps, or countries such as South Africa that need help with a specific commodity (e.g., female condoms). Having a PMU for this work, said GHSC-PSM respondents, allows for a better exchange of like experiences. For example, GHSC-PSM and field mission respondents gave examples of NFO countries hiring in-country consultants to handle registration issues and clear commodities despite not having a formal GHSC-PSM in-country presence (e.g., Tanzania and Côte d’Ivoire). The structure allows GHSC￾PSM to better understand and manage challenges presented by the NFO model. For example, given that there is no “on-the-ground” representation by the project, GHSC-PSM respondents noted it is more difficult to support or advocate for key changes in the global supply chain being executed, e.g., change of color to white for all long-lasting insecticide-treated nets (LLINs), or uptake of TLD. Field respondents noted difficulties for GHSC-PSM in countries when project representatives participated at the government policy table due to their work on procurement and were then asked by governments for TA. Field mission activity managers from NFO and other countries noted that, when the in-country TA provider was a local entity, building up the local capabilities took priority over adding GHSC-PSM TA. However, field respondents felt that GHSC-PSM assistance should be considered by the field mission in cases in which other USAID TA providers covered supply chain TA and GHSC-PSM had specific capabilities to offer, such as activity-based costing, tracking systems, and solar-powered warehouses. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 40 HSS Team: GHSC-PSM’s Health Systems Strengthening (HSS) team, tasked with leading STTA for the field, was initially a separate unit within the project’s organizational structure. Respondents said that this separation, along with the attention required by the start-up of procurement activities and the lack of an articulated strategy for strengthening supply chains, combined to make it difficult for STTA to get a foothold into countries. In response, GHSC-PSM restructured and integrated the HSS team into the country programs team to allow for more direct communication on both country TA needs and the various types of assistance the project could offer. Following some turnover in the position, in 2019 GHSC-PSM recruited a director for HSS. Each HSS technical element has a technical point of contact, some of whom develop and lead activities in their specializations, for instance human resources, Global Standards, and laboratory networks. HSS team members work with their respective technical USAID or U.S. Government counterparts, and other partner country or international colleagues, and several have articulated plans for implementation in such areas as lab networks and human resource development. While the project still lacks an overarching paradigm or theory of change for strengthening supply chains, GHSC-PSM respondents noted that, overall, there is now a better understanding of STTA work among project staff than in early days. 4.3 TO objectives and USAID/GH priorities influence the types of in-country TA offered. While USAID field missions decide how to program their funds for in-country TA, interviews indicated that USAID/GH, in many cases representing PEPFAR or PMI, has a significant role in which TA activities are developed under the project, and how. USAID/GH staff and cross-cutting groups of technical advisors develop specific activities, often in concert with GHSC-PSM, in a compartmentalized manner. USAID and GHSC-PSM respondents said that PEPFAR and PMI (i.e., TO1 [HIV] and TO2 [malaria]) tend to use GHSC-PSM primarily as a mechanism for the procurement and delivery of their critical health commodities, and use in-country TA to improve in-country implementation and efficiencies of commodities procurement (such as avoiding stock-outs), or assist with instituting changes (such as the introduction of the TLD ARV formulation into PEPFAR field programs). PEPFAR (TO1 [HIV]) also has a series of initiatives to foster evidence-based policies, as evinced by the robust Monitoring Evaluation and Reporting framework and the push for data to validate any decision. PEPFAR mandates that countries track their development using PEPFAR’s Sustainability Indices and Dashboard; PEPFAR also requires the utilization of private sector partners and the transferring of responsibility to country stakeholders through the PEPFAR Local Partner initiative. Recently, PEPFAR announced that 70 percent of PEPFAR resources should go to local partners. PMI (TO2 malaria) is also very interested in commodity security, the institutionalization of evidence￾based policies, and a transference of responsibility from donors to in-country governments and other country stakeholders, such as those in the private sector. TO3 (FP/RH) and TO4 (MNCH) respondents in USAID/GH said they want to see efficiency improvements, but are even more interested in TA that will lead to commodity security, the institutionalization of evidence-based policies, and a transference of responsibility from donors to in-country governments and other country stakeholders, such as the private sector and nongovernmental organizations (NGOs). RH/FP and MNCH are more mature programs and are actively involved in having governments take on the responsibilities and costs of commodities. RH/FP encourages government budget allocations US$ millions, by TO TA and in-country distribution Commodities, global freight, and logistics TO1 (HIV) $348 $1,233 TO2 (Malaria) $166 $47 4 TO3 (FP/RH) $88 $138 TO4 (MNCH) $34 $12 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 41 through initiatives such as FP2020 (discussed in Question 5 below) and MNCH works to integrate commodity needs into countries’ essential medical lists. In total, GHSC-PSM dedicates approximately $636 million to TA and in-country distribution.62 Figure 13, using data provided by GHSC-PSM, shows that while total TA and in-country distribution is much higher for TO1 and TO2, a comparatively greater percentage of TO3 and especially TO4 is dedicated to in￾country technical assistance and distribution versus commodities, global freight, and logistics.63 Figure 13. Spending on GHSC-PSM TA and In-Country Distribution, Versus Commodities, Global Freight, and Logistics Source: Data from GHSC-PSM in-brief, June 2019. When a TA initiative is TO-specific, it is of course supported by the specific TO. For cross-TO technical assistance, costs are routinely shared across USAID/GH TOs in a pro-rated fashion. For example, TO4 (MNCH), which has a much smaller budget than other TOs, is able to fund specific activities that are useful to MNCH, and work plans are constructed to include an appropriate proportion of funding from the TO. Interviews suggest that cross-TO technical assistance is seen as useful by those using it. The working relationships among technical staff in USAID/GH and GHSC-PSM are not uniform. Several USAID/GH respondents said that GHSC-PSM does not react quickly enough to what they want done, nor do they pro-actively anticipate USAID’s needs or propose relevant scenarios. Others said the project is a useful partner, and USAID has accomplished a lot within the relationship. Still others said they see both 62 Data from GHSC-PSM in-brief, June 2019. 63 The review team did not receive a breakout of TA expenditures separate from distribution. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 42 pros and cons with the relationship. A recurring comment from both USAID and GHSC-PSM respondents was that they would like to have more “brainstorming” and planning with the other side. Several GHSC￾PSM respondents said they believed that USAID/GH does not adequately advocate for the project’s experience in assessing supply chain systems or present the project’s successful approaches to field missions, which may benefit from the assistance. Many field mission activity managers said the cost of GHSC-PSM TA is too high, and they equated the high cost with the overuse of international consultants when local consultants were available and encouraged by their field missions. Yet some field mission activity managers said they needed the expertise of international consultants and the approaches they bring. It is not known whether GHSC￾PSM TA costs are impacted by the use of international consultants or what the breakout is of international versus local consultants. USAID does not ask GHSC-PSM to report on consultant breakouts. 4.4 Integration of technical assistance elements occurs at the country level, and distinctions between types of technical assistance are discernable. Field integration: At the end of FY 2018, 41 countries were receiving a mix of STTA and long-term technical assistance (LTTA), 64 and as listed in Table 6 above, the project presently offers eight categories of assistance. Field respondents explained that GHSC-PSM TA is invited into a country based on field mission needs and the perceived strengths of the project. TA is accessed through field mission buy-in, usually as part of an annual work plan that requires both field mission and USAID/GH clearance. The integration of individual TA elements, when it occurs, happens at the country level, within field mission and/or national supply chain or health sector strategies. Issues and accomplishments are reported in quarterly and semiannual reports, and occasionally in field mission success stories. In-country examples were highlighted in an informal poster session at the July 2019 GHSC-PSM Country Directors meeting, providing an understanding of what has happened at the country level of the project to date. Some examples: In Nigeria, the project early on initiated an evidence-based case for private sector participation to increase access to health commodities, GHSC-PSM worked with the competitive bidding process. In May 2019, the Ministry of Health’s finance arm signed the first public-private partnership through the Infrastructure Concession Regulatory Commission (ICRC). In Pakistan, since 2016, GHSC-PSM has provided assistance to an integrated government strategy with the USAID field mission in the form of forecasting and supply planning (FASP) training for financial planning, LMIS assistance (including executive dashboards, SMS/email alerts, and interfacing with the health sectors and Ministry of Health information systems), capacity building to develop guidelines and SOPs, and supply chain management courses in national and provincial institutions. Work in Zimbabwe started with a high potential for supply imbalances and no tool in use by the Central Medical Store. GHSC-PSM developed and launched early warning system software (EWS) in October 2018. GHSC-PSM trained health ministry staff, and the tool was initiated in February 2019, 64 GHSC-PSM Quarterly Report, FY 2018 Q4, pp. 61-66, is used for the country and activity counts in this section. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 43 with results presented to supply chain partners, including risks to the National Aids Council ARV Shipments; this led USAID to use core funds to cover gaps. Aside from meeting the requirements of individual field mission and partner country strategies, GHSC￾PSM is often asked to address a wide range of country-specific challenges and externalities. Again, GHSC-PSM Country Directors provided the following examples: In Cambodia, extreme challenges in reaching forested and protected areas with populations living “off the grid,” led GHSC-PSM to partner with Cambodia’s Ministry of Health and Environment forest rangers, who are now delivering long-lasting insecticidal hammock nets (LLIHNs) in 20 priority provinces. Mozambique’s March 2019 category 4 cyclone damaged or destroyed 14 percent of public health structures and 29 percent of national roads. GHSC-PSM assessed the situation, mobilized additional staff, and re-established information technology and supply chain SOPs. The project managed donated commodities, performed an inventory of existing stock, and restored supply chain operations, while documenting lessons learned. Broad distinctions: In interviews, field mission activity managers made distinctions between two types of supply chain TA. The first is direct TA to strengthen the efficiency and effectiveness of procurement and distribution of health commodities, such as ensuring that in-country counterparts can complete the quantification, forecasting, logistics, and distribution of USAID and other donated or government￾purchased commodities. The second, often referred to by respondents as health systems strengthening, is focused on long-term capacity building (beyond training), and improvements in decision-making and governance relevant to a country’s journey toward self-reliance in health commodities. These distinctions are not formally made by USAID/GH or GHSC-PSM. Mission activity managers and several other respondents interviewed find that procurement and directly related TA (e.g., logistics, MIS, supply chain training) are GHSC-PSM strengths, but do not see the project as an obvious vehicle for work in governance, policy, human resource development, or national and regional strategy development. Some respondents believe that, as a contract,65 GHSC-PSM cannot be sufficiently flexible to engage in ever-changing political and administrative environments, and that the project’s skill set, while good for procurement, logistics, and MIS systems, does not translate into the policy acumen and engagement skill necessary to enable change within fluid public sector structures. Common quotes include “these are different skill sets,” “better at procurement and warehousing; not good in TA,” and “amazing with commodities; not TA.” Nevertheless, in countries where GHSC-PSM is integrated into field mission health system strengthening activities, there is markedly more support for GHSC-PSM’s engagement and policy development skills than in countries where this is not the case. 4.5 Direct supply chain technical assistance to improve efficiency and reach is seen as GHSC-PSM’s comparative advantage. GHSC-PSM is used by many field missions for the immediate TA needs of national and sub-national supply chains, including FASP, LMIS, warehousing and distribution, skills development, and lab 65 The viewpoint of some respondents is that an assistance mechanism would offer more flexibility. FY 19 Target ARVs 94% 85% Lab (HIV diagnostics) 100% 85% RTKs 94% 85% VMMC 80% 60% Malaria commodities 92% 60% FP/RH commodities 88% 85% MCH commodities 67% 60% Condoms 88% 85% Total 90% GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 44 efficiency. Some of this TA focuses on improving the efficiency and effectiveness of supply chains, and leveraging the in-country private sector is commonly supported in these approaches. Respondents said this type of assistance develops and improves the in-country provision of health commodities, while at the same time strengthening the reach of the GHSC-PSM global supply chain in countries where the project is responsible for procurement. Forecasting and supply planning: While complex supply chain environments make objective performance measures of the TA provided difficult, GHSC-PSM does set targets and tracks areas of TA provision across countries. For example, by the end of FY 2018, 39 countries were receiving TA in supply planning, aided in part by GHSC-PSM’s FASP modernization tool. In FY 2019 Quarter 2, targets for supply plans submitted were met and exceeded in all health commodity areas (Figure 14).66 Nevertheless, there is no reporting on the quality of these plans, or local (non-GHSC-PSM) participation in their preparation. Figure 14. Percentage of Supply Plans Submitted During FY 2019 Q2, by Commodity Group Source: GHSC-PSM Quarterly Report, FY 2019 Q2, p. 58. Warehousing and distribution assistance, said field respondents, is requested by national governments when commodities do not reach patients, there are stock-outs, there are concerns about theft, or all of the above. USAID respondents reported hearing similar concerns but also find that field missions and governments ask for assistance to improve efficiency, responsiveness, and sustainability of national systems, regardless of whether stock-out rates are high. Thirty-four countries were receiving this 66 Under the quantification paradigm supported by GHSC-PSM, supply plans take a regularly updated, forward-looking view of demand for 18 months. The approach provides visibility into monthly demand, even if a single quarterly update is not submitted. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 45 assistance by the end of 2018. Country Directors and GHSC-PSM headquarters counterparts shared examples of improved efficiency, and that the ability to safely store and track commodities is central to GHSC-PSM’s response: In Ethiopia, a GHSC-PSM assessment determined that the country needed only 10 warehouses when it had 19, and the transportation system was at a 40 percent utilization rate. The government accepted a new plan, and GHSC-PSM reports that there are now 10 warehouses and fewer partial deliveries. In Malawi, GHSC-PSM sub-contracts to set up prefabricated Storage-in-a-Box units were entered into with private sector providers and faith-based organizations (FBOs). The units met 40 percent of the storage, but the requirement for grid power precluded expansion. In 2018-19, USAID and DFID funded a solar-powered redesign, and the new units are now in 253 health facilities across Malawi. In Zambia, GHSC-PSM engaged a new third￾party logistics (3PL) fleet to distribute commodities and assisted the Medical Stores Limited (MSL), a government entity, in monitoring. MSL’s two-day alert notices demonstrate improvements in loading time and one-day delivery. The Transportation Information Tool (TransIT), developed with support from the Chemonics’ SOLVE Team67 in 2018, is a cloud-based tool that provides constant information on transportation systems, enabling distribution managers to better track shipments and optimize distribution resources. Field mission activity managers and several USAID/GH respondents were positive about it, yet several also said managing misappropriation and theft, including within government facilities, was still very difficult for USAID and the project. One global partner with experience in this area felt strongly that tracking and prosecuting misappropriation and theft was not appropriately developed, noting “you can’t monitor theft through Inspector General audits.” For the complexity of warehousing and distribution, Activity-Based Costing is a business-oriented approach that GHSC-PSM has found to be successful, and is looking to better disseminate 67 SOLVE is a Chemonics corporately funded group set up to support and encourage innovations. TransIT itself is an innovation developed under Objective 3: Global Collaboration. Activity-Based Costing: Warehousing and Distribution (Lesotho) Problem • Drugs not getting out and clients were complaining; Lesotho field mission asked for STTA to help • First look: Distribution center had drugs to deliver but had not done so for two weeks because they were awaiting more commodities • Analysis: Space constraints, including too much of the right and wrong products; outdated center practices Approach • Calculated indirect costs (procurement set up, cold chain, complexity of order) and direct costs (receiving, storage, selection, and transport), and determined true costs of activities • Helped set up competitive requests for proposals (RFPs), establish appropriate mark-up, and develop standard key performance indicators • Long-term consultant on the ground observed to have a hands-on, respectful manner that developed trust Change • Center knows its costs, what to charge as a management fee (far less than based on value of commodities) • Government and Global Fund know the true value of all costs, and through-put has gone up Sources: Interviews; Titus, Ralph L. 2017. “Leadership through Activity-Based Costing.” Presentation at Global Health Supply Chains Conference, Accra. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 46 within USAID. The box at right summarizes the use of the approach in Lesotho. As warehousing becomes more efficient, GHSC-PSM has focused on fine-tuning technologies. For example, GHSC-PSM is interested in innovations to expand temperature and humidity monitoring beyond trucks and warehouses to temporary storerooms, clinic offices, and, for community-based work, health workers themselves. Information systems: The project is called on to improve the functionality, efficiency, and coverage of LMISs, and occasionally to harmonize varied systems. In FY 2018, 32 countries and the Regional Development Mission for Asia (RDMA) region were using the project’s LMIS assistance, based on the capacity in each. For example, field respondents reported that Rwanda’s and Guinea’s MIS systems were moved from paper to automated systems. They have integrated SOPs and tracer commodities to be followed at the district, regional, and central level; Burma’s system was expanded to cover more than essential drugs; and Pakistan’s and Nepal’s earlier systems are being strengthened to improve visibility. GHSC-PSM began to track reporting into LMISs, whether the system was paper-based or automated, in FY 2017. This included percentages of timely submission of reports by service delivery points (SDPs), as well as timely aggregation and/or data entry at any intermediate levels, as required (Figure 15). HIV and malaria SDPs show higher percentages of reporting than FP/RH at the aggregate level shown in quarterly reports. Individual GHSC-PSM country programs set targets for reporting on this indicator. Figure 15. Timely Reporting Rates of Service Delivery Points via an LMIS for TOs 1-3 (Indicator B3, FY 2017 Q1 to FY 2019 Q2)68 68 Sources: Indicator B3: SDP reporting rate to the Logistics Management Information System from FY 2019 Performance Overview annex to GHSC-PSM Quarter 4 Report, pp. 3-4. Indicator B3; GHSC-PSM Annual Report, FY 2018 Annex A. M&E Indicator B3; and GHSC-PSM Semiannual and Quarter 4 Report, FY 2017, Annex A M&E Indicator B3. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 FY 2017 FY 2018 FY 2019 Percentage of SDPs Reporting Reporting Period TO1 (HIV) TO2 (Malaria) TO3 (FP/RH) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 47 USAID activity managers and several USAID country backstops stressed that any assistance, such as strengthening or assisting in the development of LMIS leading to changes in a supply chain, requires inclusive engagement with the relevant national government entities managing and using the systems (some more than others). Respondents reported that for PEPFAR and for some field missions there is strong interest in developing the connectivity of supply chain information systems to link with the facility and user level. For HIV, this would include medical records and viral load information. Laboratory networks: For PEPFAR, the focus on laboratory networks is the third UNAIDS 90-90-90 target.69 GHSC-PSM’s approach is “to link laboratory network optimization to a supply chain system’s forecasting and supply planning, procurement and strategy sourcing, and performance management.”70 The project engages government and other stakeholders and international partners to map and validate the national lab network, referral systems (including transportation routes), and all equipment.71 It also encourages innovation, such as a shift from capital equipment to an all-inclusive rental pricing, which includes reagents, based on expected demand in an optimized network. In Nigeria, for example, GHSC-PSM analysis found patient demand could be met with 16 viral load labs; there were 27 operating under PEPFAR. PEPFAR and government officials came to a consensus and decreased the number to 17, with the MOH deciding to pay for one lab. The National Integrated Sample Referral Network (NISRN) reported improvements; from March through June 2018, figures indicated a 16 percent increase in transport of specimens over a similar period from the year before; there was also a 75 percent increase in health facility coverage, and the sample rejection rates dropped from 36 percent to 8 percent.72 Skills training: In FY 2018, GHSC-PSM trained 9,812 host-country government and other supply chain staff (see Table 7 below). In the first two quarters of FY 2019, GHSC-PSM trained a total of 4,828 individuals.73 69 The 3rd target is that 90 percent of all people living with HIV will have viral suppression, and thus need to know their viral load. 70 GHSC-PSM, Laboratory Network Approach, one pager, no date. 71 GHSC-PSM, The Laboratory Approach for Procurement and Supply Management. Power point presentation; interviews. 72 Ibid., slide 27. 73 Participants may be counted more than once if they attend multiple, discrete training activities. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 48 Table 7. GHSC-PSM Training by Technical Focus, FY 2018 Technical Area Individuals Trained Warehousing and inventory management 3,155 MIS 1,658 Quality assurance 1,643 Human resources capacity development 1,316 Governance and finance 888 Monitoring and evaluation 716 Transportation and distribution 247 Forecasting and supply planning 112 Strategy and planning 67 Procurement 10 TOTAL 9,812 Source: GHSC-PSM Quarterly Report, FY 2018 Q4, p. 84. Overall, GHSC-PSM training includes skills to fill both immediate supply chain operational needs and move forward long-term institutionalization and sustainability, work that is discussed in the next section. Field respondents see this training as necessary for the viability of USAID’s investment in the supply chains, yet voiced concerns about the lack of information on who was trained and whether there was sufficient follow-up supervision or assessment after training. For USAID field activity managers, this concern extended to all donor training activities. GHSC-PSM trains its own staff as well as USAID staff who are involved or responsible for supply chains. These trainings are not included in the previous table. GHSC-PSM respondents gave examples of training accessed by project field staff through the graduate program at the W.P. Carey School of Business, University of Arizona, established with Chemonics support. GHSC-PSM assists with training USAID supply chain staff, and managers responsible for supply chain training. Using the private sector in the supply chain: GHSC-PSM and USAID respondents said they bring the private sector into supply chain implementation as a way to improve efficiency and coverage, and to avoid trying to run supply chains through weak government structures. The project helps identify, support, and manage private sector entities for supply chain operations, such as logistics, storage, distribution, and transportation. Field respondents gave examples: 3PLs were enlisted in Zambia for supplemental warehouse space, in Cameroon for commodity distribution, and in the Democratic Republic of Congo (DRC) for the transport of malaria drugs. In Angola, the private sector is tapped for “everything from port to distribution.” In Pakistan, field respondents report that the transportation supplier sends commodities to the district level, where district governments are responsible for distribution; when districts lack the resources to complete deliveries, GHSC-PSM links them with private companies. Field respondents also note that GHSC-PSM helps countries as they enlist FBO networks in last mile delivery, such as in Malawi, Zambia, and more recently, Rwanda. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 49 Other operations indicators: Along with the service delivery point reporting rates to LMIS (indicator B3) shown in Figure 15 above, USAID asked the review team about two additional in-country operations indicators: stock-out rates at SDPs (indicator B1; see Figure 16, next page), and stock status at storage sites (indicator B2; Figure 17, second page below). Reporting on these indicators by TO for HIV, malaria, and FP/RH began in FY 2017, and continues to appear in quarterly reports by TO, country, and commodity, along with descriptive caveats for each country. In addition, for TO1 (HIV), GHSC-PSM provides current stock-out data for HIV monthly to the U.S. Department of State’s Office of the Global AIDS Coordinator (S/GAC). GHSC-PSM respondents noted that these indicators provide insight into the public health commodity supply chain systems where GHSC-PSM is working, and help identify where project TA may be necessary. In FY 2018, the project reported on indicators B-1 and B-2 disaggregated by country program, store level (e.g., central warehouse, subnational medical store), and health element. GHSC-PSM also provides brief country contextual descriptions related to the indicators in routine reporting,74 however, as is the case with the TA provided, distinctions are not made among the level of direct GHSC-PSM assistance provided. GHSC-PSM correctly cautions those using these indicators75 that: ● GHSC-PSM is one of many entities working to improve country supply chain indicators; it would be hard to attribute change solely to their contributions, ● Success regarding stock levels depends at a minimum on adequate in-country supplies. GHSC￾PSM is among many entities contributing to procurement and inventory management, and ● Especially at the beginning of the project, there were limitations on the quality of data for these indicators, complicating comparison over time. Given these caveats, Figure 16 shows stock-out rates at SDPs. TO3 (FP/RH) demonstrated a significant drop in stock-out rates since the project began (from 29 to 12 percent); TO1 (HIV) has maintained a rate of approximately 8 percent. TO2 (malaria) began FY 2017 Q1 at 13 percent and ended FY 2019 Q2 at 12 percent. 74 Ibid. Performance Annex for indicators B-1 and B-2. GHSC-PSM explained to the review team that the stock risks for all TOs are initially reported either by Procurement Planning and Monitoring Report (PPMR) administrators or GHSC-PSM field offices. In most cases, the Commodity Security Team also sends out requests to responsible parties for clarification and updates. These sources can vary, but generally, one or more of the following sources are used: PPMR administrators, field offices, non-field office teams, and ARTMIS. 75 GHSC-PSM, M&E Plan, February 11, 2019. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 50 Figure 16. Stock-out Rates at Service Delivery Points for TOs 1-3 (Indicator B1, FY 2017 Q1 to FY 2019 Q2)76 The steep drop in stockout rates in FY 2017, reported GHSC-PSM: Can be largely attributed to changes in reporting from Pakistan. Pakistan reports stockout data from as many as 12,000 health facilities… Throughout 2017 they reported stockouts for progestin-only pills and emergency oral contraceptives, both of which are little-used in Pakistan and widely stocked out. In 2017, they ceased reporting on these commodities as they did not fit the definition of “offered” methods require for reporting. Field respondents noted that GHSC-PSM examines different levels of the supply chain system to identify where hold-ups occur to better inform their work, and, as discussed under Question 5 below, USAID coordinates with global partners through GHSC-PSM to address stock-outs within formal global engagements, and less formally at the country level. USAID activity managers for all health areas find that in-country supply chains have always required close vigilance to avoid stock-outs. They also expressed concern that, while new technologies for tracking can represent useful improvements, technologies need to be appropriate for the environments and supervisors using them. The “Stock Status Observations in Storage Sites” indicator is used to identify 76 Sources: FY 2019 Performance Overview annex to GHSC-PSM Quarter 2 Report, pp. 3-4. Indicator B1; GHSC-PSM Annual Report, FY 2018 Annex A, M&E, Indicator B1; and GHSC-PSM Semiannual and Quarter 4 reports, FY 2017, Annex A, M&E, Indicator B1. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 FY 2017 FY 2018 FY 2019 Stock-out Rate Reporting Period TO1 (HIV) TO2 (Malaria) TO3 (FP/RH) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 51 under- or over-stock and expiration issues for individual commodities at distribution sites, to determine whether and/or what type of assistance is needed from GHSC-PSM. This indicator “checks to see if the supply chain system is functioning as it was designed by tracking if both the central level and subnational level medical stores can maintain the designated quantity of stock/months of stock to treat patients or to distribute to treatment facilities or secondary distribution centers.”77 The indicator is broken out by stock status: whether observations find stocks in a warehouse are according to plan, or if there are overstocks, understocks, and/or stock-outs (as shown in the country-by-country chart of FY 2019 Quarter 2 data in Annex VIII, which gives a country-level example of the information provided for a single quarter). These data, along with country-specific caveats reported by GHSC-PSM, combine to highlight in-country areas needing attention. Figure 17 shows the rate of “stocks according to plan” to date for the three TOs. Figure 17: Stock Status Observations in Storage Sites Where Commodities are Stocked According to Plan, by Level in Supply System, for TOs 1-3 (Indicator B2, FY 2017 Q1 to FY 2019 Q2) At the TO level, USAID/GH respondents explained that PEPFAR and PMI are heavily involved with SDP￾level observations as part of their Country Operational Plans (COPs) and Malaria Operational Plans (MOPs), respectively, and are concerned with stock-outs. Procurement, donations, and use anomalies can affect the breakout of this indicator. 77 GHSC-PSM, M&E Plan, February 11, 2019. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 FY 2017 FY 2018 FY 2019 Percentage of Observations According to Plan Reporting Period TO1 (HIV) TO2 (Malaria) TO3 (FP/RH) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 52 Stock-outs are of equal concern for FP/RH; USAID/GH and global partner respondents noted the importance of global contraceptive security, the rational distribution of available commodities, and the increasing demand for various methods in some areas. Over the coming year, USAID and GHSC-PSM are scheduled to introduce an indicator that measures the institutionalization of GHSC-PSM-supported supply chain functions: B-8, “Percentage of targeted supply chain activities in which the host country entity has achieved technical independence with GHSC technical assistance.” This focus on long-term institutionalization beyond the life of GHSC-PSM speaks to the concerns of field mission activity managers, several of whom noted that as projects end, systems and activities that were developed, especially LMIS and the use of private sector approaches, can break down when USAID support ends. 4.6 Bolstering country ownership and moving towards self-reliance is seen to require trust and flexibility. Systems-wide work: Depending on the field mission and partner government objectives, GHSC-PSM TA can and is used to assist in-country supply chains moving towards self-reliance. Field respondents noted that governance and policy development is integrated with other elements into GHSC-PSM’s TA work in various countries: Guinea, Malawi, Pakistan, and Nepal, among others. The highlighted box gives a glimpse of the potential complexity and challenges of this work, using Nepal as an example. Field respondents see embedded staff as a useful way to develop trust as a partner and move changes along. One respondent reflected the thought bluntly: “…as long as they don’t see you like a separate entity, it will work.” Several respondents felt that building trust and knowledge of a national government system to support change within that system was more difficult for GHSC-PSM headquarters staff and international STTA. GHSC-PSM Country Directors gave examples of embedding that they found useful, such as in Lesotho, where a precedent condition for the country’s Global Fund grant is the evolution of the supply chain unit in the Ministry of Health into a directorate. National￾and district-level embeds are assisting in this evolution. Shift to federalism (decentralization of PfSCM responsibilities) in Nepal Administrative Change • After December 2017 elections, Nepal started decentralizing it procurement and supply chain functions to several newly created provinces and 753 Local Level Governments (LLGs) • Its centrally managed Logistics Management Division was downgraded to a Logistics Management Section, reflecting the changing Partnership for Supply Chain Management roles and responsibilities Approach • GHSC-PSM/Nepal began working more at the provincial level, specifically with the piloting of eLMIS in two of Nepal’s provinces • GHSC-PSM/Nepal has also seconded staff (Field Support Officers) to several provinces Change • Government buy-in for a number of the technical interventions (e.g., eLMIS) remains uncertain • It was too early to tell, but the most recent feedback was that there are several challenges which need to be addressed if the supply chain technical assistance is to be sustainable or transformative Source: Tim Clary, July 2019, Evaluation of GHSC-PSM in Nepal, draft document. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 53 USAID activity managers said GHSC-PSM’s embedded staff in government put the project at the policy table, well positioned for encouraging a move towards system strengthening. When working with a partner government along with other GHSC and supply chain-related implementing partners (IPs), most felt that there is clarity on roles. As in-country relationships proceed, GHSC-PSM finds that it is asked to assist with important system improvement changes. For example, in Rwanda, the project is providing assistance to the government’s development of a pharmaceutical pricing policy, and the process for determining prices involves the coordination of advice from public and private organizations. Workforce improvement and sustainability: GHSC-PSM has contributed to the development of a theory of change and is providing TA to improve and better sustain supply chain human resources. Working with the USAID/GH PRH and Office of Health Systems (OHS)78 Human Resources for Health 2030 project, People that Deliver, and UNICEF, the project co-designed a theory of change based on international experience. The resulting roadmap includes activities and indicators for promoting stewardship and leadership at the national level to address workforce needs, and advocate within the public and private health system for a quality supply chain workforce.79 In its workforce improvement assistance, GHSC-PSM is using the theory of change to move beyond competency training and into workforce development in several countries. For example, in response to the common concern of public sector staff turnover, a GHSC-PSM respondent described supporting the development of permanent positions, e.g., pharmacists and information technology experts, as a way to professionalize personnel and retain an institutional history within government organizations. Field respondents and a USAID/GH technical advisor noted that activities are underway. In Angola and Pakistan, supply chain work is being integrated into the offerings of higher education institutions, and an investigation of supply chain-related careers in Rwanda’s entire labor market is underway. QUESTION 5: HOW HAS GHSC-PSM COORDINATED/COLLABORATED WITH GLOBAL DEVELOPMENT PARTNERS TO MITIGATE THE RISK OF STOCK-OUTS OR OTHER SUPPLY IMBALANCES IN COUNTRY SUPPLY CHAINS, FROM THE CENTRAL WAREHOUSE TO FACILITIES NOW AND IN THE FUTURE? 5.1 USAID calls for effective global collaboration: a journey without maps. Global Collaboration: USAID’s GHSC-PSM Results Framework Objective 3 calls for “Effective global collaboration to improve long-term availability of health commodities …through improved engagement, global market research and innovation, improved awareness and advocacy, and improved coordination among GHSC implementing partners and other USAID supply chain activities.” There is no project-wide strategy for how “effective global collaboration” is to occur within the GHSC￾PSM Project. GHSC-PSM regularly joins in on USAID strategies and activities as requested and helps develop activities. The project performance reporting tracks the number of engagements, research, and 78 GH/OHS works with other GH technical offices to help with field health system strengthening. See: USAID, 2015, USAID Vision for Health Systems Strengthening: 2015-2019. Washington, D.C. 79Andrew Brown, Wanda Jaskievicz, Bridget McHenry, Erin Meier, and Dominique Zwinkels, 2018. “Building Human Resources for Supply Chain Management: A Theory of Change,” GHSC-PSM, Washington, D.C. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 54 innovations, and provides narratives in reports.80 There are no criteria on whether a collaboration fell short, met expectations, or exceeded intentions over time. Reporting: USAID’s Results Framework calls for four Intermediate Results under Objective 3, Global Collaboration. These can be summarized as: 1) Improved strategic engagement, 2) Global market dynamic research and dissemination, 3) Improved awareness and advocacy to improve availability of health commodities, and 4) Improved coordination within the IDIQ and other USAID supply chain activities.81 Quarterly reports suggest that the project picked up on relationships held by USAID and previous projects, and planned for their own forms of technical communication, which included a website. In 2016, GHSC-PSM engaged substantively with UNFPA, with convening assistance from GH/PRH, and integrated into the Reproductive Health Supplies Coalition (RHSC).82 They also contracted for and later launched a market dynamics evaluation covering critical aspects of commodity markets for each central task order. During this period, GHSC-PSM drafted a vision and detailed stock-taking of their work in global collaboration, which consolidated the wide range of collaborators the project was or planned to engage with, and lists of activities being undertaken.83,84 Beginning in year two, a range of work by task order was reported in scheduled project documentation to USAID. Most involved the provision of data and expertise, and convening meetings with a variety of partners (e.g., the World Health Organization [WHO] on ARVs; RHSC on the Global Family Planning Visibility and Analytics Network [Global FP VAN]; FP2020 for supply chain data). The latest report reviewed (FY 2019, Q2), discussed GHSC-PSM launching or leading working groups (e.g., TraceNet for LLINs; an ARV Procurement Working Group; the ART Optimization Programme Advisory Committee), and various RHSC working groups in which GHSC-PSM holds key committee positions. There were also updates on technological innovations, such as an unmanned aerial vehicle (UAV) pilot project in Malawi, and TransIT85 being rolled out in Cameroon, Mozambique, and Angola. Respondents indicated that the roll out of global collaborations can take one or more different paths, such as: 1) absorption into the GHSC-PSM GSC (e.g., market analyses, Global Standards); 2) Roll out as an area or element of in-country technical assistance (e.g., Global Standards, TransIT); or 3) Roll out more broadly into countries as part of the collaboration itself (e.g., contraceptive security, Global Standards). Respondents’ views on global engagement: In interviews, USAID, UNFPA, and the Global Fund are seen by colleagues as having the hands-on procurement experience to help mitigate stock-outs or other commodities imbalances in real time. They are also seen as useful partners in planning for long-term 80 See Annex VI. Performance Dataset Reference Guide. 81 AID-OAA-I-15-00004, p. 28. 82 The Reproductive Health Supplies Coalition is a global partnership NGO established in 2004. Membership includes multilateral and bilateral organizations, private foundations, low- and middle-income country governments, civil society, inter￾governmental and non-governmental organizations, and the private sector. USAID and GHSC-PSM are members. 83 Global Collaboration Sub-unit Strategic Plan, October 18, 2016, PowerPoint presentation. 84 As this review ended, GHSC-PSM’s TO3 team was completing an updated Strategic Framework specifically focused on the key global collaboration issue of contraceptive security. 85 As discussed in the previous section, TransIT is a cloud-based, non-subscription system providing information on transportation. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 55 contraceptive security, demand estimations, and other initiatives related to addressing supply imbalances (e.g., FP2020,86 the Global FP VAN, and the Center for Global Development’s work group on the Future of Global Health Procurement). Task orders work with GHSC-PSM in different ways. Respondents said that for TO3 (FP/RH) and TO4 (MNCH), USAID envisions the project as an implementing partner with the resources and personnel to engage with relevant partners. TO3 has a global collaboration lead. As part of this engagement, GHSC￾PSM notes that, in TO3, the project becomes involved in joint decision-making and strategic partnership building. For TO4, the TO Director covers the responsibility. GHSC-PSM’s M&E Team works with FP/RH and MNCH donors (e.g., RHSC and FP2020) on data and analytic issues. Interviews demonstrated that global partner respondents in FP/RH and MNCH know their USAID and GHSC-PSM counterparts, and can outline what the project provides to their partnership. For PEPFAR and PMI (TO1 [HIV] and TO2 [malaria], respectively), respondents said that GHSC-PSM tends to provide data and data analysis, and hosts meetings as directed by USAID colleagues. TO1 and TO2 decided early in the project not to have point persons for global engagement in GHSC-PSM. TO1 respondents noted that neither PEPFAR nor the Global Fund work directly with GHSC-PSM. TO2 respondents noted that PMI is working with GHSC-PSM on Global Standards (discussed below) issues along with the GF and other international procurement agencies (IPAs), but do not regularly conduct donor engagement with GHSC-PSM. Nevertheless, USAID/GH and GHSC-PSM headquarters respondents said that GHSC-PSM TO directors and technical staff from M&E, GSC, and HSS are asked for input for global engagement activities in HIV and malaria; GHSC-PSM noted recent discussions with GF on data sharing, aligning quality assurance processes, and market health initiatives. Global partners working in HIV and malaria relate having good working relationships with USAID counterparts. They mention specific activities, such as alignment engagement between USAID and the Global Fund, reviewing new technologies in the Malaria Access Committee, and harmonizing HIV and malaria indicators. Most in this particular group of respondents said they do not work directly with GHSC-PSM on collaborations with USAID, although none said that they would not. GHSC-PSM does report working on teams with the GF on alignment, and a few respondents noted being briefed on the project or visiting the GHSC-PSM offices; one GF respondent suggested it would be helpful to have a GHSC-PSM organizational chart.87 Communication: When asked about dissemination and internal coordination within the global health community, respondents discussed the GHSC-PSM website, presentations, and reports associated with specific activities, and noted that professional contributions were important. GHSC-PSM reporting discusses the project coordination with other GHSC projects—Global Health Supply Chain Rapid Test Kits (GHSC-RTK), Global Health Supply Chain Quality Assurance (GHSC-QA), Global Health Supply Chain Technical Assistance (GHSC-TA), and earlier with Global Health Supply Chain Business Intelligence and 86 Initiated in 2015, FP2020, is a multi-country, multi-organization initiative to advance access to FP methods to women and girls, using a rights-based approach, focused on government responsibility. 87 As respondent feedback indicates in Section 5.6, working relationships between the GF and GHSC-PSM, especially in mitigating stock outs, do exist in the field. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 56 Analytics (GHSC-BI&A)—without references to any specific coordination. There is no agency-wide public affairs strategy for the project. 5.2 GHSC-PSM is widely credited for work on two cross-cutting initiatives: Global Standards and Market Analysis. Global Standards: Both USAID and global partners credit GHSC-PSM as a key collaborator in the development and dissemination of Global Standards in supply chain management. In March 2019, Version 2.1 of the Global Standards Technical Implementation Guidelines for Global Health Commodities was drafted and co-authored by GHSC-PSM staff, and was endorsed by the Global Fund, United Nations Development Programme, UNFPA, USAID, PEPFAR, PMI, and Stop TB.88 The consensus summarized in the document states that: International procurement agencies (IPAs) that procure and distribute... for developing markets are committed to incorporating lessons learned over the last decade of health supply chain management to enable more secure and efficient supply chains. Implementation and use of global supply chain data standards for product and location identification, data capture (e.g., barcoding), and data exchange as the foundation for business communications across global, regional, and national trading partners is agreed to be central to achieving those objectives.89 Interviewees within USAID and the project view this result as a milestone, both in global partnerships and the project’s own approach to visibility. Interviews and documentation indicate that GHSC-PSM was fully “at the table” in the initiative. As GHSC documentation demonstrates, GHSC-PSM worked with the GS190 and Global Data Synchronization Network (GDSN) community to introduce features to enhance their utility in global health, including obtaining product registration information directly from suppliers, and completing an integration of ARTMIS with the GS1 and GDSN community. GHSC-PSM developed guidance for countries on how to implement the GS1 standards, and worked with Angola, Ghana, and Rwanda to assess their current capabilities and develop plans for improved alignment.91 One global partner cautioned that partner governments will find investment in this approach difficult. Market research: Early in 2016, one of the first global collaboration activities that GHSC-PSM reported on was the initiation of global health commodity markets analyses. Several USAID respondents from different TOs said this was what they were looking forward to from GHSC-PSM. Analyses included in￾depth looks at HIV, malaria, and FP/RH in areas such as: ● Market assessments of HIV rapid test kits (RTKs) and malaria rapid diagnostic tests (RDTs); ● A deep dive into LLIN varieties and sizes; and 88 This work was facilitated by members of the WHO Interagency Supply Chain Group (ISCG) and the GS1 Healthcare Team, comprising 20 representatives of industry. 89 Global Standards Technical Implementation Guideline for Global Health Commodities Version 2.1, March 2019, p. 7.; Note that, separately, WHO is preparing policy guidance on the use of data standards in supply chains for health commodities for member countries. 90 GS1 is a not-for-profit organization that develops and maintains Global Standards for business communication. 91 GHSC-PSM, “Implementation Guidance for Pharmaceutical Traceability: Leveraging GS1 Global Standards,” Version 1.0, February 2019, PowerPoint presentation; GHSC-PSM FY 2018 Annual Report, p. 20; interviews. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 57 ● Analysis of the generics market for contraceptives, and reproductive health data collection and assessment in Latin America, including the purchase of private sector data on a wide variety of reproductive health commodities. USAID respondents reported that these analyses led directly to changes in both HIV and malaria procurement policies; for Latin America, it was used both in internal planning and with global partners, and provided a better view of contraceptive availability in some areas that had graduated from USAID FP/RH assistance. GHSC-PSM also purchased generic oral contraceptives, a first for USAID. The HIV results were reviewed by the U.S. Department of State’s Office of the Global AIDS Coordinator to evaluate priorities, and changes were integrated into PEPFAR Country Operational Plan instructions.92 A USAID respondent (calling the studies “the silver lining of the first year”) and various other USAID/GH respondents noted that the work led to changes in how the global supply chain was approached, e.g., renting equipment, including reagents as a supplier’s responsibility, and changes in the color (white) and heights of LLINs eligible for PMI procurement, to improve efficiency. GHSC-PSM estimated that the changes in HIV policies could save more than $200 million for the period between 2017 and 2021; for malaria, the project estimates a savings of $30 million over the same period.93 USAID respondents who are close to the work corroborated these estimates. 5.3 GHSC-PSM is well-integrated into FP/RH and MNCH global collaborations. UNFPA: USAID’s key counterpart in family planning and reproductive health is UNFPA. The two have a decades-long history of collaborating on contraceptive security, and together, an RHSC respondent estimated that they provide approximately 60 percent of all FP commodities to the public sector, with GHSC-PSM providing USAID’s share. Interviews and documents indicated that USAID and UNFPA collaborate at both the global and country level to mitigate stock-outs. Their field representatives work through intermittent in-country stock-outs, and participate together in other long-term, in-country initiatives in concert with national government and other stakeholders to implement national strategies or to work on specific activities (e.g., harmonizing the distribution of commodities; capturing and managing registration data for FP commodities). A UNFPA respondent noted that, together, UNFPA and GHSC-PSM can develop better solutions based on their complementarity: “In the field, UNFPA covers a broader set of countries than GHSC-PSM, but we often need to wait for their funding to make contraceptive purchases, while GHSC￾PSM can take advantage of the Central Contraceptive Procurement (CCP); thus, its ability to act quickly.” Global FP VAN: The Global FP VAN, established during the 2017 London Family Planning Summit by UNFPA, USAID, DFID, and the Bill and Melinda Gates Foundation,94 uses extant tools such as the procurement planning and monitoring reports,95 and institutional relationships such as the Coordinated 92 PEPFAR’s 2019 Country Operational Plan instructions p. 385 of 419: “To address issues around instrument breakdown/sample backlog due to poor services and maintenance contracts, stock-outs, discrepant/volume commitment pricing and high unit cost per test for reagents, Rather, all countries should stop outright instrument procurement, pursue and secure "all-inclusive" per test pricing via reagent rental agreements using standardized key performance indicators to monitor suppliers, end users, and instruments.” 93 GHSC-PSM, 2017 Semiannual and Q4 Report, p. 62. 94 All four remain members of the steering committee, along with one user representative from John Snow, Inc. 95 PPMRs were initiated by GH/PRH and integrated into the international FP/RH community via RHSC. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 58 Assistance for Reproductive Health Supplies (CARhs), to integrate the many information systems among procurers, manufactures, suppliers, and Central Medical Stores. GHSC-PSM helped draft the consensus document and participated in developing the plan and competition for the technology vendor, as well as field pilots focused on implants and oral contraceptives.96 GHSC-PSM respondents described the close cooperation between USAID and GHSC￾PSM on the initiative, and USAID and global partners said they especially appreciated the difficulty of setting up the procurement for the Global FP VAN technology implementer. Oxytocin storage: At a 2017 RHSC Maternal Health Caucus97 meeting, USAID began a coordinated focus on the scientific evidence of the importance of cold chains and oxytocin (a first line drug of treatment for post-partum hemorrhage) with WHO, UNFPA, manufacturers, wholesalers, governments, and universities. GHSC-PSM helped develop an international consensus, culminating in the WHO-UNICEF￾UNFPA “Joint Statement on Storage and Management of Oxytocin,”98 and co-hosted a workshop with PATH to develop and share a tenable messaging framework.99 MNCH specialists and other respondents involved with supply chains see this as a key success in global coordination. Several USAID and partner respondents said that a key element for GHSC-PSM’s ability to work effectively with RHSC in both GH/PRH and GH/MCHN stems from USAID and GHSC-PSM having individuals with long-term technical expertise in MNCH who have participated on and been elected to leadership positions on RHSC committees. Closing stock-out gaps: TO3 (FP/RH) works with global partners to close the gap on stock-outs for family planning commodities. The GHSC-PSM PPMR team works with the CARhs group to expedite shipments, transfer commodities, track and provide shipment information, and respond to information requests. In FY 2018, for example, the project reported: ● Expediting 13 shipments to Burkina Faso, Burundi, Chad, Liberia, Madagascar, and Tanzania to prevent stock-outs; ● Initiating five transfers of commodities to Benin, Burkina Faso, DRC, Madagascar, and Togo; ● Tracking and providing valuable shipment information on planned and emergency shipments to 36 country data providers; ● Initiating and monitoring the response to seven emergency requests to avert stock-outs; ● Postponing or canceling 10 shipments to avoid or reduce overstocks; and ● Responding to 71 information requests from countries on upcoming shipments and commodity￾related procedures. 96 Reproductive Health Supply Coalition and Global FP VAN, 2019, Forging a “Collective Ask” of the Global FP VAN Technology Vendor. 97 The Maternal Health Caucus of the RHSC is a forum for maternal health and family planning communities to forge a common language for maternal health supply-related challenges. 98 https://apps.who.int/iris/bitstream/handle/10665/311524/WHO-RHR-19.5-eng.pdf. 99 Reproductive Health Supply Coalition. 2018. “Buy Quality Oxytocin, Keep it Cold.” RHSC, Brussels. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 59 5.4. USAID and the Global Fund are pursuing alignment and helping each other to mitigate stock-outs. USAID and GF alignment: Global partners said that the GF, PEPFAR, and PMI provide an estimated 60 percent of HIV and malaria commodities in the countries they serve. USAID and the GF are the largest buyers of HIV and malaria commodities, and both provide commodities within many of the same countries. Respondents on both sides say they work with each other to minimize overlap as situations arise. However, USAID and GHSC-PSM gave examples of gaps in service when the GF Principal Recipients (PRs) were responsible for last mile delivery and did not deliver, indicating that work is needed to assure alignment in the field. Additionally, a few USAID activity managers were not pleased with having to fund quantification costs for GF PRs with funding that should be directed to GHSC-PSM. The GF and USAID report that they are presently making a concerted effort to develop a process for how to move forward in collaboration on medium- and long-term alignment using a shared strategy. On the HIV side, one global partner expected this alignment to lead to better harmonization, resulting in smaller, better-targeted deliveries. Another global partner noted that there is a realization that some countries may not be comfortable with such a high level of collaboration, which may be interpreted as a method of oversight: “Countries may be not happy with the level of collaboration, but most of the strategies are mainly country-driven policies, with a broader mandate to countries.” Mitigating stock-outs: USAID respondents reported positive collaboration between USAID, the GF, and other donors in mitigating country-level stock-outs. For HIV, USAID provides a monthly stock-out report to PEPFAR. These stock-outs are not in GHSC-PSM programs, but the project is called on to expedite an order or redistribute commodities to meet the need. At the country level, USAID activity managers and USAID/GH colleagues gave examples of providing help with HIV and malaria commodities to Global Fund PRs (Table 8). Several respondents indicated they were quite pleased with being able to help. Table 8. Examples of USAID and Global Fund Field Coordination to Mitigate Stock-Outs Country Example of Coordination from Respondent DRC Malaria commodities swapped; Global Fund and USAID implemented an end user verification survey together Ghana A few examples with USAID and Global Fund were discussed, e.g., “earlier in the project, GF helped with bed nets until the USAID ones arrived” Guinea “Global Fund didn’t have the capacity in 2016, and USAID helped” Liberia “USAID and Global Fund help each other with stock-outs in malaria; we don’t need it as much” Mali and Burundi Specific coordination on malaria, but procurement at the Geneva level is improving Rwanda USAID helped provide second-line ARVs Sierra Leone “They are running out of Global Fund-supported ACT (artemisinin-based combination therapy) – we were approached for help” Tanzania “USAID alleviated a shortage with pediatric ARVs” Uganda Inter-warehouse transfers used to improve efficiency or lead to fewer stock-outs; GF and USAID coordinate on this in-country GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 60 V. CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS After an unprecedented, overly rapid transition from long-standing contractors, and a difficult start-up in the public eye, GHSC-PSM is credited with a strong comeback. The project is meeting and sustaining key targets of critical interest to USAID over this first half of the project, including OTD and OTIF, and demonstrates declines in TLC. The project manages a complex, worldwide procurement and distribution supply chain, supported by in-country monitoring and TA, and contributes to global engagements, market research, and innovations. It was important that the review team consider the magnitude of the GHSC-PSM Project. This project is the consolidation of the bulk of procurement and in-country distribution of health commodities from USAID. By April 2019, GHSC-PSM had provided commodities and/or technical assistance in supply chain improvements to 67 countries. Its catalog contains close to 4,000 different items procured from 326 suppliers. The project has three distribution centers and uses over 4,000 shipping lanes. In short, this is a very large, new project for USAID’s Global Health Bureau and, as such, many of these conclusions, in this early stage of the new project, are directed not only to GHSC-PSM but also to USAID. Improved availability of health commodities A delayed start and ramp up of GHSC-PSM created management, operational, administrative, and implementation challenges for the global supply chain. It was clear that GHSC-PSM worked hard and that the project’s supply chain operations have evolved, leading to improved performance and reduced risks and bottlenecks. The project is observing the SCOR model and is working to move up the ladder on the Gartner model. As GHSC-PSM follows the SCOR model it layers atop certain “strategy” capabilities but lacks an overarching strategy for pursuing predictive/proactive supply chain management, a necessity for GHSC-PSM, which could optimize multiple supply chains. The review team believes USAID and GHSC-PSM collaboration on this strategy development is essential to mesh strategic needs with supply chain operational realities. Each party has a perspective that is essential in informing a collective, overarching strategy and enabling execution. GHSC-PSM has experienced an evolutionary maturation of organizational structure and staffing, all leading to improved performance, as evidenced by the improvement in OTIF and OTD. Of note are the investments in human resources, standard operating procedures development and refinement, training and functionality add-ons, and data visibility and analytics capabilities to meet the needs of the myriad TO supply chains, supplier issues, country and regulatory requirements, and other elements. Nevertheless, ARTMIS, and thus the project, should be more predictive. The review team believes that a build-out of ARTMIS capabilities is likely needed for the system to be more predictive. On the positive side, the development of the GHSC-PSM continuous improvement program using AssurX shows significant progress. For continued evolution to occur, better coordination is vital. Interviews suggest that USAID/GH and GHSC-PSM are not speaking the same “language” as they refer to the supply chain, nor do they understand each other’s mission vis-à-vis the implementation of the global supply chain. GHSC-PSM is GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 61 pursuing improvement in relation to the SCOR and Gartner models, while USAID is focused on operational-level implementation of the supply chain. Additionally, there are likely further opportunities for efficiency gains, especially in freight cost reductions. However, a lack of both a clearly defined methodology to enumerate cost savings and a prior prediction of the future cost savings limit the reliability of such reported information. Strengthening in-country supply chain systems Respondents said it took over a year to start seeing the elements of what is referred to in the GHSC-PSM IDIQ as “an in-country Systems Strengthening Technical Assistance Program.” Project attention was initially focused on working through the transition from predecessor procurement projects and setting up country offices. The review team sees this less as a late start issue and more as an eventuality that should have been anticipated, given the criticality of having an uninterrupted supply of health commodities. Early attention should have been given to developing a paradigm, or as one respondent noted, “at least a picture,” that explains the general progression and elements of country supply chain systems, and where the entry points for strengthening can be. Paradigms and approaches were developed for supply chain elements, such as human resource development, lab networks, and activity￾based costing, but not for an entire supply chain system. This left the project in a more “stove-piped” and reactive position when approached by field missions and USAID/GH to provide TA. At this juncture, the project should be able to step back and clearly define what it means to strengthen a supply chain system in-country, highlighting the critical points in the system where GHSC-PSM is able to assist. The review team agrees with relevant documentation and respondents, including USAID field representatives, that TA to improve the efficiency and reach of supply chain programs in-country is GHSC-PSM’s forte among TA offerings. Field missions appreciate and depend on the project’s expertise in these areas, although there is some question as to whether increased TA could be provided by local experts. Of special note is GHSC-PSM’s work to develop, improve, or consolidate LMIS, warehousing and distribution, skills for supply chain workers and managers, and lab efficiency, as well as its use of the private sector. As field respondents described the progress made by GHSC-PSM, they made a distinction between this direct TA and what they term health systems strengthening (HSS) assistance to governments wishing to undertake systematic changes in governance, management, or human resource development, areas in which the project is active and, according to respondents, is doing well in individual countries where they work. At this point in the project, the review team believes USAID and GHSC-PSM should move away from the current practice of describing everything the project does in-country as “strengthening supply chains.” The GHSC-PSM Project provides three types assistance to countries: direct implementation of supply chain management by project staff (direct assistance); assistance to improve the efficiency and reach of a supply chain system (technical assistance); and work in concert with governments and relevant entities towards long-term systemic change (HSS). Understanding and using these distinctions will 1) help better identify how countries are doing along the journey to self-reliance, 2) clarify the relevance of individual indicators used to track progress or sustainability, and 3) facilitate the integration and exchange of supply chain TA and health system strengthening work with partner countries and the wider global health community, including the global partners the project works with now. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 62 In terms of management and monitoring, GHSC-PSM made strategic structural improvements in its headquarters management of country program, including strengthening the Project Management Unit structure to better focus on the development and backstopping of in-country programs in concert with USAID/GH TO leaders. Establishing a non-field office PMU was especially important in that it operates centrally in situations where GHSC-PSM has no in-country presence but is providing commodities. Additionally, integrating STTA teams into country programs was useful for a broader understanding within the project of the TA elements being provided as country planning proceeds. These structures appear to serve the project well and can be expected to do so in the future. Effective global collaboration to improve long-term availability of health commodities GHSC-PSM demonstrates progress in global partner engagement, private sector market analyses, and innovation by joining USAID strategies and activities as requested. Interviews indicated that a global collaboration usually takes more than one of the following three paths: absorption into the GSC, roll out to partner countries via TA, and roll out in concert with global partners. For example, in the Findings section above, GHSC-PSM’s contributions to Global Standards and their private sector market analysis was discussed; both are global collaborations, their results are already absorbed by the GSC, and Global Standards continue to be rolled out via the TA program. Keeping track of these paths and the absorption/acceptance of these global collaborations is important for monitoring. It can help inform future activities and, the review team believes, is more useful than the present practice of monitoring progress by counting the number of activities undertaken. Additionally, it is not strategic to group elements of normal project implementation, such as coordination with other GHSC projects, developing a website, and attending meetings, solely in Objective 3, as they are related to all three project objectives. USAID/GH TO3 (FP/RH) and TO4 (MNCH) leaders have effectively brought GHSC-PSM into their engagement strategies as extensions of themselves, and the project has designated points of contact for the work. FP/RH and MNCH global partners know and appreciate GHSC-PSM contributions on engagements, e.g., mitigating stock-outs with UNFPA, strengthening the Global FP VAN, and avoiding oxytocin shortages. These are commendable contributions for USAID and GHSC-PSM. For TO1 (HIV) and TO2 (malaria), the project is tapped by USAID for input on global engagements. However, for a variety of reasons, it is not directly brought into global partner engagements, nor is the project well understood by global partners. In the field, GHSC-PSM coordinates with the Global Fund and other global partners to mitigate stock-outs. However, as USAID and the Global Fund continue to work on global alignment, the review team believes GHSC-PSM should be brought in as a fuller participant in the process; a review of the approaches used by FP/RH and MNCH colleagues as they approach this task would be useful. USAID was prescient to include global collaboration on an equal footing as an objective for GHSC-PSM. The interconnectivity of global markets, the work of donors and other international procurement agencies, and improvements in science and technology are increasingly more important in the current era, when foreign assistance is expected to do more with less. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 63 Cross-cutting issues The amount of GHSC-PSM’s budget dedicated to M&E is fairly low by international standards and is extremely low if commodity purchases are included in the denominator when deriving percentages. For a high-profile project such as GHSC-PSM, this is particularly concerning, given both the demands for data and reporting, and the criticality of health commodities for USAID partner countries. Furthermore, evaluative and learning opportunities have not been fully utilized, and to neglect this particular area would be a major failing for both USAID and GHSC-PSM. While there is an informal listing of all of the indicators for the Performance Dataset (Annex VI) and notes on the variables, this is not sufficient. GHSC-PSM and USAID M&E counterparts would benefit from more specificity in documentation of the methods used for calculating and managing the key performance indicators. USAID and GHSC-PSM need to ensure they are in compliance with DQA guidance as given in ADS 201. It is recognized by the review team that USAID/GH needs to utilize some global indicators to allow comparative analysis, as well as oversight of the IDIQ. As noted in Annex VII and under Question 4 (Findings section) above, several indicators for Objectives 2 and 3 are difficult to attribute to GHSC-PSM, and do not adequately measure whether the technical assistance provided for strengthening the supply chain (as a sub-unit of the overall health system) will result in any sustainable or transformative accomplishments. For Objective 3, from a strictly indicator-focused standpoint, having only two core indicators that count activities could call into question whether Objective 3 is important to USAID. RECOMMENDATIONS IMPROVED AVAILABILITY OF HEALTH COMMODITIES 1. USAID should develop an overarching supply chain strategy based on a combination of the SCOR and Gartner Maturity models. Working within the constraints of the TO health areas’ objectives and funding structure, develop a strategy in consensus with GHSC-PSM and all stakeholders. Key elements of the strategy would be to: create a single supply chain mission; create common process improvement objectives and SOPs within the SCOR key processes (plan, source, deliver, etc.); and create a strategy for evolving the supply chain toward Stage 2 (e.g., more proactive across the SCOR elements). Systematize the strategy to the extent possible, and expect a 2-year timeframe for completion. 2. As a first step in the development of an overarching supply strategy, USAID should further systematize the supply chain across funding streams. The first step in this process requires USAID and GHSC-PSM to systematize the supply chain across TOs and funding streams. Review each of the task order supply chains with an eye to discovering commonalities, such as in the HP-Compaq example discussed above in the Findings section. Consider applying a “LEGO-like” concept; that is, standardize as much as possible with the building blocks of the supply chain, then customize the remainder, maintaining the focus on standardization.100 100 This is a recommendation from an internal USAID/GH document. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 64 Standardize the processes and management of these common traits to reduce redundancies, reduce costs, etc. Two examples: standardize supplier RFPs and contracting processes where possible; create standardized risk identification and management processes. This type of standardization can be expected to reduce costs, improve agility and decision-making processes, and simplify operational activities for PSM. The time frame is both immediate and ongoing. 3. USAID and GHSC-PSM should develop a “Rosetta Stone”-type terminology resource for their staffs working with the GSC program, kicked off by a facilitated workshop. Develop a “Rosetta Stone” terminology resource to facilitate how staff talk to each other about the supply chain, so that everyone understands the relevant components and the specific intent of all terms and names in the same way (e.g., what is the SCOR model; what is meant by analytics); leadership of both entities should agree on a common lexicon with which to communicate. The process is listen, share, and document; this is not unusual. The opportunity provided is to review the multiple supply chains, determine the commonalities, and develop standardized SOPs for managing common activities and elements. Create a library of SOPs; a library eliminates the issue of learning loss that occurs when personnel leave the organization. 4. To reduce risks and bottlenecks, USAID should continue to invest in building out GHSC-PSM’s continuous improvement software application “AssurX” to ensure it is integrated with ARTMIS. AssurX is particularly important for forward-looking analytics, and it should be integrated with ARTMIS. Results would include better supplier management, reduction in problems related to quality, and better categorization of quality issues, resulting in improved prioritization of actions and remediations. This should take approximately 1 year. 5. USAID should decide how it wants to define TLC and other key GHSC-PSM indicators now and into the future, if these indicators are expected to be comparable across time. USAID wants to compare the progress of their GH supply chain investments over time. USAID should objectively review the definitions of the metrics they expect to compare over time. The definitions of TLC and other key indicators should be clearly referenced with all cost and other data inputs defined, and with a specification of all supply chain costs that are excluded. STRENGTHENING IN-COUNTRY SUPPLY CHAIN SYSTEMS 6. GHSC-PSM should develop and disseminate a brief theory of change for strengthening country supply chains, which would include a simple diagram. USAID and field missions will continue to have specific needs, and the contract will need to remain responsive to those needs. Integrating and articulating how supply chains are strengthened through a theory of change should facilitate communication, and would perhaps provide an educational tool with field missions, governments, global partners, and USAID/GH colleagues as the project undertakes further in-country work. If this is done, it is important that all GHSC-PSM staff are well briefed and able to articulate the theory of change. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 65 7. USAID and GHSC-PSM should provide more specificity and a clearer nomenclature for the work GHSC-PSM undertakes in-country, such as direct assistance in executing a supply chain, TA to improve the efficiency and reach of a supply chain, and HSS in concert with government partners to make the structural, administrative, and policy changes necessary to move toward self-reliance. Transparency is particularly important at a time when the continued need for direct assistance for commodity procurement and distribution by donors is being questioned. Transparency facilitated by clear and specific language will allow the project (and USAID) to better demonstrate and measure USAID’s intents vis-à-vis local ownership and sustainability. As necessary precursors to assessing progress towards self-reliance, the project should ensure that the institutionalization of new approaches and technologies under the TA program are being tracked. For health systems strengthening work, take time to understand the milestones or tipping points as a country takes on responsibility for their own procurement and distribution, and consider international and national expert participation in tracking HSS in countries over time. EFFECTIVE GLOBAL COLLABORATION TO IMPROVE LONG-TERM AVAILABILITY OF HEALTH COMMODITIES 8. USAID and GHSC-PSM should clarify Objective 3, which regards global collaboration, to focus on global engagement, private sector market research and innovations, and long-term oversight of key global collaborations. Begin with a review of existing initiatives, focusing on USAID priorities and GHSC-PSM comparative advantages. Track initiatives over time through their stages of implementation, field roll out, dissemination, and learning. For some initiatives this tracking will be done in concert with global partners (e.g., Global FP VAN, UNFPA); others will be more specific to USAID (e.g., uses of market analysis within and beyond USAID). This does not preclude the addition of new initiatives, nor does it assume that all facets of an initiative will be completed before the end date of the project. For the remainder of what is now Objective 3, USAID and GHSC-PSM will need to develop a clearer placement for project-wide activities, such as: coordination among GHSC and other projects and entities; social media concerning the project; project dissemination, including professional meetings; and one-off meetings, perhaps as a cross-cutting set of activities supporting all three Objectives. CROSS-CUTTING RECOMMENDATIONS 9. GHSC-PSM’s budget dedicated to M&E should increase, and include the hiring of additional staff. While it was noted that each of the task orders has different priorities within M&E, and that the flows of information have become more routinized, this does not lessen the need for robust M&E systems, processes, and staffing; rather, it should increase demand. Additionally, given that TO1 is accessing multiple datasets and systems, TO2 has its own specific indicators, and TO3 is looking at broader issues of systems strengthening, there appears to be a need to better coordinate and holistically address M&E issues. The hiring of additional staff with higher-order tasks around strategic management, coordination, evaluation and learning, and global collaboration (not just M&E back-stopping) would, most likely, help to address these issues. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 66 10. GHSC-PSM needs to progress from a monitoring function, and look for openings for additional evaluative and learning opportunities. As related to Recommendation 9, it is recommended for the remaining life of the project and, specific to any remaining annual M&E work plans, that USAID and GHSC-PSM develop a set of investigative questions and learning activities to begin sharing with the global development community for lessons learned, best practices, success stories, and implementation challenges. This could include, for example, state-of-the-art conferences/workshops, internal evaluations (with U.S. Government participation), brown bags, and communication materials to be shared with partner organizations. 11. GHSC-PSM should develop an official data dictionary for the Performance Dataset and SOPs, for calculating all key indicators beyond what is already detailed in the M&E Plan. GHSC-PSM needs to create an official data dictionary for the Performance Dataset (although there needs to be sufficient M&E staff and capacity to allow this undertaking). Furthermore, while the M&E plans does contain Performance Indicator Reference Sheets (PIRSs), and the M&E team was able to provide the review team with instructions for calculating the three requested KPIs (OTIF, OTD, TLC) using the Performance Dataset, GHSC-PSM should consider incorporating these types of instructions for all relevant indicators into the data dictionary. Also, GHSC-PSM should ensure that USAID is sufficiently informed of (and perhaps involved in) the development of the data dictionary, and subsequently in the computation process, so that USAID M&E counterparts can adequately answer internal and external basic queries about GHSC-PSM’s data. 12. While possibly not achievable in the remaining life of the project, USAID and GHSC-PSM should consider revisiting and possibly revising the indicators for Objectives 2 and 3 in GHSC-PSM’s M&E Plan. There are two main options for remedying this situation at this stage. The first is to revisit and revise most, if not all, of the indicators associated with Objectives 2 and 3, as well as to develop additional indicators or standardized evaluation plans for Objective 3. The main drawback for Objective 2 indicators would be that revisiting the indicators after nearly 4 years of implementation makes the development of baselines and realistic targets very challenging. GHSC-PSM and USAID should consider opening up broader consultations on these indicators with development partners (Gavi [The Vaccine Alliance], the World Bank, the Global Fund), technical agencies (WHO), and other USAID/GH system strengthening and policy development projects that have significant histories in working on health systems strengthening. 13. GHSC-PSM should hold biannual meetings with USAID M&E personnel to review the Performance Dataset, its contents, and how key indicators are being calculated and managed. Twice a year (outside of the normal bi-monthly meetings), GHSC-PSM and USAID should schedule a 1- or 2-hour meeting/workshop to review these issues and discuss any updates to this primarily technical function. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 67 14. External DQAs by USAID of GHSC-PSM need to become routine. USAID should first internally review the previous DQA, its contents, its date of completion, whether any identified issues were resolved, and what changes have occurred within GHSC-PSM’s M&E plan that may require additional DQAs. It is recommended that there be annual, external DQAs conducted to ensure that GHSC-PSM’s data reporting systems and processes are sufficient to assure both internal and external stakeholders that the reported information complies with the five characteristics (validity, integrity, precision, timeliness, and reliability) of high-quality data. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 68 ANNEX I. STATEMENTS OF WORK INITIAL STATEMENT OF WORK Assignment #: 679 [assigned by GH Pro] Global Health Program Cycle Improvement Project (GH Pro) Contract No. AID-OAA-C-14-00067 EVALUATION OR ANALYTIC ACTIVITY STATEMENT OF WORK (SOW) Date of Submission: 10-19-18 Last update: 10-23-19 Amendment #4 TITLE: The Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Project: A Mid-term Review for Lessons Learned and the Way Forward Requester / Client ☒ USAID/Washington Office/Division: GH Bureau (OHA/SCH, ID/MAL, PRH/CSL) Funding Account Source(s): (Click on box(es) to indicate source of payment for this assignment) ☒ 3.1.1 HIV ☒ 3.1.2 TB ☒ 3.1.3 Malaria ☒ 3.1.4 PIOET ☒ 3.1.5 Other public health threats ☒ 3.1.6 MCH ☒ 3.1.7 FP/RH ☒ 3.1.8 WSSH ☒ 3.1.9 Nutrition ☒ 3.2.0 Other (specify): multiple offices/program accounts as indicated here: Original: GH/PRH FP/RH: $35,000; GH/ID MAL: $75,000; GH/OHA HIV/AIDS: $189,170; AA/GH FP/RH: $184; AA/GH MAL: $98; GH/P3 FP/RH: $14,095; GH/P3 MAL: $7,527; GH/P3 MCH: $9,416; Amend2: 50:50 split GH/PRH – FP/RH: $14,456.50 & GH/ID- MAL: $14,456.50; Amend3 – cost realignment only; Amend4: 50:50 split of GH/PRH – FP/RH; GH/HIV – HIV/AIDS. Cost Estimate: Note: GH Pro will provide a cost estimate based on this SOW) Performance Period Expected Start Date (on or about): March 18, 2019 Anticipated End Date (on or about): December 13, 2019 Location(s) of Assignment: (Indicate where work will be performed) The consultants’ home office; GHSC-PSM (PSM) Project Offices in Crystal City, VA; USAID Offices (CP3 and RRB); and other locations in the Washington, DC Area. It is not expected that the evaluation team will visit any of the 33 countries where the Project implements activities; GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 69 however, information will be solicited from select USAID Missions and PSM Field Offices as part of the data collection process. Type of Analytic Activity (Check the box to indicate the type of analytic activity) EVALUATION: ☒ Performance Evaluation (Check timing of data collection) ☒ Midterm ☒ Endline ☒ Other (specify): Performance evaluations encompass a broad range of evaluation methods. They often incorporate before–after comparisons but generally lack a rigorously defined counterfactual. Performance evaluations may address descriptive, normative, and/or cause-and-effect questions. They may focus on what a particular project or program has achieved (at any point during or after implementation); how it was implemented; how it was perceived and valued; and other questions that are pertinent to design, management, and operational decision making ☒ Impact Evaluation (Check timing(s) of data collection) ☒ Baseline ☒ Midterm ☒ Endline ☒ Other (specify): Impact evaluations measure the change in a development outcome that is attributable to a defined intervention. They are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. OTHER ANALYTIC ACTIVITIES ☒ Assessment Assessments are designed to examine country and/or sector context to inform project design, or as an informal review of projects. ☒ Costing and/or Economic Analysis Costing and Economic Analysis can identify, measure, value and cost an intervention or program. It can be an assessment or evaluation, with or without a comparative intervention/program. ☒ Other Analytic Activity (Specify) PEPFAR EVALUATIONS (PEPFAR Evaluation Standards of Practice 2014) Note: If PEPFA-funded, check the box for type of evaluation ☒ Process Evaluation (Check timing of data collection) ☒ Midterm ☒ Endline ☒ Other (specify): Process Evaluation focuses on program or intervention implementation, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. In addition, a process evaluation might provide an understanding of cultural, socio￾political, legal, and economic context that affect implementation of the program or intervention. For example: Are activities delivered as intended, and are the right participants being reached? (PEPFAR Evaluation Standards of Practice 2014) ☒ Outcome Evaluation Outcome Evaluation determines if and by how much, intervention activities or services achieved their intended outcomes. It focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are produced. It is possible to use statistical techniques in some instances when control or comparison groups are not available (e.g., for the evaluation of a national program). Example of question asked: To what extent are desired changes occurring due to the program, and who is benefiting? (PEPFAR Evaluation Standards of Practice 2014) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 70 ☒ Impact Evaluation (Check timing(s) of data collection) ☒ Baseline ☒ Midterm ☒ Endline ☒ Other (specify): Impact evaluations measure the change in an outcome that is attributable to a defined intervention by comparing actual impact to what would have happened in the absence of the intervention (the counterfactual scenario). IEs are based on models of cause and effect and require a rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. There are a range of accepted approaches to applying a counterfactual analysis, though IEs in which comparisons are made between beneficiaries that are randomly assigned to either an intervention or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured to demonstrate impact. ☒ Economic Evaluation (PEPFAR) Economic Evaluations identifies, measures, values and compares the costs and outcomes of alternative interventions. Economic evaluation is a systematic and transparent framework for assessing efficiency focusing on the economic costs and outcomes of alternative programs or interventions. This framework is based on a comparative analysis of both the costs (resources consumed) and outcomes (health, clinical, economic) of programs or interventions. Main types of economic evaluation are cost-minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-benefit analysis (CBA) and cost-utility analysis (CUA). Example of question asked: What is the cost-effectiveness of this intervention in improving patient outcomes as compared to other treatment models? BACKGROUND If an evaluation, Project/Program being evaluated: Project Title: Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Single Award IDIQ Contract Number: AID-OAA-I-15-00 004 Contract Dates: 04/15/2015–11/23/2020 (IDIQ) Project Funding: $9,500,000,000 Implementing Organization(s): Chemonics International Project COR: Sherif Mowafy, IDIQ COR Xavier Tomsej, TO1 COR Linda Gutierrez, TO2 COR John Vivalo, TO3 COR Reena Shukla, TO4 COR Background of project/program/intervention (Provide a brief background on the country and/or sector context; specific problem or opportunity the intervention addresses; and the development hypothesis) The Global Health Bureau (GH) manages an array of central commodity procurement and supply chain technical assistance mechanisms, collectively known as the implementing architecture for the Global Health Supply Chain (GHSC), to support USG field programs and USAID's global technical leadership. These mechanisms are managed across multiple GH technical offices that support HIV/AIDS, malaria, family planning/reproductive health (FP/RH), maternal, newborn and child health (MNCH), and other public health programs. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 71 The Global Health Supply Chain Procurement and Supply Management (GHSC-PSM) contract is the largest of the USAID GHSC mechanisms. It is a single-award indefinite delivery/indefinite quantity (IDIQ) contract with a $9.5 billion ceiling designed with the following purpose: ● To serve as the primary vehicle through which USAID will procure and provide health commodities for all USAID health programs, including but not limited to, HIV/AIDS, Malaria, FP/RH, and Maternal & Child Health. ● To provide systems strengthening technical assistance to improve supply chain management and commodity security in partner countries. rovide effective global collaboration to improve long term availability of health commodities. GHSC-PSM is led by Chemonics International with sub-partners including IBM, Kuenhne + Nagel Inc., IDA Foundation, Population Services International, SGS Nederland B.V., McKinsey & Company, IntraHealth International Inc., Arbola Inc., Axios International Inc., Panagora Group, and University Research Co. LLC. GHSC-PSM supports USAID field programs in meeting their commodity and supply chain management needs through a range of services—including forecasting, quantification, procurement planning, procurement, freight forwarding, warehousing and inventory management, in-country distribution as needed, contract management and administration, systems strengthening, capacity assessment and building, global collaboration, and reporting. There are currently five task orders under the IDIQ. The task orders are Task Order 1 HIV/AIDS, Task Order 2 Malaria, Task Order 3 Family Planning/Reproductive Health, Task Order 4 Maternal Child Health and Zika, and Task Order 5 Kenya Technical Assistance. The first four are centrally managed and provide both commodity procurement and technical assistance, while the Kenya Task Order was awarded and is managed by the Kenya Mission for technical assistance only. The periods of performance for Task Orders 1-3 end in November 2020; for Task Order 4 in September 2021. As of June 2018, GHSC-PSM had served 62 countries through global commodity procurement services and/or technical assistance. Theory of Change of target project/program/intervention The project does not have a theory of change. Strategic or Results Framework for the project/program/intervention (paste framework below) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 72 What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis? GHSC-PSM works globally in the Africa, Asia, Latin America and Caribbean, and Middle-East Regions and has established in-country presence in the following countries: Angola, Botswana, Burkina Faso, Burma, Burundi, Cambodia, Cameroon, Caribbean Region, Central America Region, Ethiopia. Ghana, Guinea, Haiti, Indonesia, Kenya101, Lesotho, Liberia, Madagascar, Malawi, Mali, Mozambique, Namibia, Nepal, Niger, Nigeria, Pakistan, Regional Development Mission for Asia (RDMA), Rwanda, Sierra Leone, Uganda, Vietnam, Zambia and Zimbabwe. GHSC-PSM procures and delivers commodities to these countries and several others. Though the analysis will cover the GHSC-PSM’s implementation at the headquarters for global procurement and logistics and systems strengthening technical assistance in the field, it is not anticipated that the evaluation team will conduct country visits. Evaluation questions related to technical assistance will be addressed through data and document reviews and virtual communication with select countries. Purpose, Audience & Application 101 Technical assistance for Kenya is provided under a separate task order (TO5) and as such is not included in this assessment. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 73 A. Purpose: Why is this evaluation/assessment being conducted (purpose of analytic activity)? Provide the specific reason for this activity, linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. Through this activity GH seeks to: 1) Assess and document the progress and performance of GHSC-PSM, near its midpoint, in meeting its core objectives of (a) Improved availability of health commodities (global procurement and logistics); (b) Strengthened in-country supply chain systems; and (c) Effective global collaboration to improve long-term availability of health commodities across the centrally-managed task orders. The evaluation findings and recommendations will be used to identify opportunities for continuous improvement to enhance procurement services and supply chain technical assistance going forward. a. The evaluation questions will address strategic and operational themes implemented across all health areas (HIV/AIDS; malaria; family planning and reproductive health; and maternal and child health and Zika) within the project's mandate. b. Determine what savings and efficiencies may have been realized through the implementation of the consolidated GHSC-PSM design/approach using available historical data from the predecessor supply chain projects, where feasible, and savings and efficiencies realized over the life of the project. B. Audience: Who is the intended audience for this analysis? Who will use the results? If listing multiple audiences, indicate which are most important. ● The evaluation findings and recommendations will be used primarily by GH staff managing the GHSC-PSM project and GH and USAID leadership to prioritize and inform activities in the final years of the project and inform future procurements in this area. ● Missions who have funded GHSC-PSM activities are additional intended users of the evaluation results as they consider technical priorities and support in the final years of the project and beyond. ● GHSC-PSM (Chemonics and its subpartners) as they consider their strategy for the last years of implementation. C. Applications and use: How will the findings be used? What future decisions will be made based on these findings? As noted in previous sections, the findings of this evaluation will be used to identify areas for improvements and inform programming in the remaining years of the GHSC-PSM Project across the three objective areas. The evaluation will also help to shape the future of the GHSC implementing architecture, as GH begins a comprehensive design of the “next generation” GHSC architecture. Evaluation/Analytic Questions & Matrix: ● Questions should be: a) aligned with the evaluation/assessment purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 74 geographic locale, age, etc.), they must be incorporated into the evaluation/assessment questions. USAID Evaluation Policy recommends 1to 5 evaluation questions. ● State the method and/or data source and describe the data elements needed to answer the evaluation questions Evaluation/Assessment Question Method & Data Source Objective 1: Improved availability of health commodities (global procurement and logistics) 1 How has the Global Health Supply Chain for Procurement and Supply Management (GHSC-PSM) Project progressed across its stated objective and results for global procurement and logistics? a. Document and describe GHSC-PSM Project performance to-date using defined Project key performance indicators (KPIs) (e.g. Total Landed Cost, Cycle Time, On-Time Delivery, etc.) and the Project’s Results Framework. ● Document and data review ● Data analysis of GHSC-PSM’s ARTMIS data and/or data in GHSC-BI&A 2 How has the GHSC-PSM Project addressed risks, bottlenecks, and/or inefficiencies, in the global supply chain system? a. Document the major risks, bottlenecks, and/or inefficiencies in the global supply chain system that may impede procurement and delivery performance. b.Document and describe the process for identifying and managing root causes. c. Document and describe enhancements to the continuous improvement approach the Project might consider going forward. Are there any specific global supply chain functional improvements that should be prioritized going forward? ● Document and data review ● Key informant interviews: − USAID/W staff (CORs and technical advisors) − GHSC-PSM Project staff including —leadership and relevant Global Supply Chain (GSC) and Task Order teams ● Focus group discussions, if possible 3 Were cost savings and efficiencies realized since the start of the GHSC-PSM Project in January 2016, with the consolidation of procurement services under a single award IDIQ contract? a. Comparisons should be made to the initial GHSC￾PSM network design and processes, as well as to the predecessor SCMS and DELIVER projects. In particular, the Global Health Bureau would like to explore (savings and efficiencies across) the following areas: − Commodity sourcing including product costs; − Warehousing and distribution of commodities; − Consolidation of project headquarters staff; ● Document and data review, including review of initial GHSC-PSM network design and available predecessor data to establish baseline ● Key informant interviews: − USAID/W staff (CORs and technical advisors) − GHSC-PSM Project staff including —leadership and relevant Global Supply Chain (GSC) and Task Order teams GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 75 − Automation of information systems and use of technology; − Any others related to the global level. Objective 2: Strengthened in-country supply chain systems 4 How have in-country supply chains performed in GHSC￾PSM supported countries during the life of the project? What trends are observed? a. To what extent has technical assistance from GHSC-PSM contributed to performance improvements? Describe performance within the context of relevant project in-country systems strengthening indicators including but not limited to: i. Stock outs at service delivery points (SDPs); ii. Percentage of stock status observations in storage sites where commodities are stocked according to plan; and iii. Service delivery point reporting rate to the logistics management information system (LMIS). b.Describe the contribution of GHSC-PSM technical assistance to the journey to self-reliance (i.e. supporting the in-country systems to become self￾reliant). c. What efficiencies (or inefficiencies) were identified in GHSC-PSM’s non-presence countries (countries for which GHSC-PSM procured commodities but were not the supply chain technical assistance provider but still had to collaborate with the TA provider)? ● Document and data review ● Data analysis of GHSC-PSM’s DevResults data and/or data in GHSC-BI&A ● Key informant interviews: − USAID Mission staff − USAID/W staff − GHSC-PSM Project staff − Home Office: Health System Strengthening and Country Programs Teams, Project Management Units, Non-field office (NFO) team, etc. − Field Office staff ● Review findings from GHSC USG Customer Satisfaction Assessment (conducted under GH Pro 614) − Survey and interview responses of USAID Mission staff Objective 3: Effective global collaboration to improve long-term availability of health commodities 5 How has GHSC-PSM coordinated/collaborated with global development partners (such as The Global Fund, UNFPA, etc.) to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future? − Document lessons learned and opportunities for improvement. − Describe any new processes or work streams implemented that supported improved donor coordination in the area of supply chain. ● Document and data review ● Key informant interviews: − USAID Mission staff − USAID/W staff − GHSC-PSM Project staff − Global development partner representatives, e.g. The Global Fund, UNFPA, The Gates Foundation etc. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 76 Other Questions [OPTIONAL] (Note: Use this space only if necessary. Too many questions leads to an ineffective evaluation or analysis.) Methods: Check and describe the recommended methods for this analytic activity. Selection of methods should be aligned with the evaluation/assessment questions and fit within the time and resources allotted for this analytic activity. Also, include the sample or sampling frame in the description of each method selected. General Comments related to Methods: Review of key project documents and reports; secondary analysis of existing project data; key informant and group interviews, focus groups (as needed), Mission consultations. ☒ Document and Data Review (list of documents and data recommended for review) This desk review will be used to provide background information on the project, and will also provide data for analysis for this evaluation. Documents and data to be reviewed include, but are not limited to the following: ● IDIQ RFP and Task Order RFTOPs ● Contracts ● M&E Plans (For HQ and by country) ● Performance Data from all quarterly and annual reports ● Project financial data and reports ● SPACES Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER ● Client Satisfaction Survey 2016-2017 ● Assessment of USG Customer Satisfaction with USAID’s Global Health Supply Chain Program (conducted under GH Pro 614) (2018) ◦ Mission staff survey and interviews ◦ Full report ☒ Secondary analysis of existing data (This is a re-analysis of existing data, beyond a review of data reports. List the data source and recommended analyses) Data Source (existing dataset) Description of data Recommended analysis GHSC-PSM M&E data (global and in-country data) Project performance data as defined in the Project M&E Plan (available in GHSC-PSM’s ARTMIS, DevResults, and/or BI&A) Calculation of key global procurement and logistics and in-country/systems strengthening indicators Global procurement and logistics operations and financial data Data generated by the Project’s global procurement and logistics operations Cost comparison analysis (Note: Comparisons should be made to the initial GHSC-PSM network design and processes as well as to the predecessor SCMS and GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 77 DELIVER projects.) Calculation of indicators may include on￾time delivery, on-time and in￾full delivery, total landed cost, cycle time, and other relevant metrics. ☒ Key Informant Interviews (list categories of key informants, and purpose of inquiry) ● USAID Washington (HIV/AIDS, Malaria, Family Planning/PRH, MNCH): Division Chiefs, CORs, technical advisors, country backstops ● USAID Missions ● GHSC-PSM (IDIQ and TO Directors, Global Supply Chain, Country Programs, Health Systems Strengthening Teams) ☒ Focus Group Discussions (list categories of groups, and purpose of inquiry) ☒ Group Interviews (list categories of groups, and purpose of inquiry) Key informants may be interviewed in small groups of similar respondents, as long as all participants feel free to express their own opinions. ☒ Client/Participant Satisfaction or Exit Interviews (list who is to be interviewed, and purpose of inquiry) ☒ Survey (describe content of the survey and target responders, and purpose of inquiry) A USG client satisfaction survey regarding the GHSC architecture is being conducted by GH Pro (#614). The evaluation team should make reference to this as appropriate. Another e-survey for this evaluation may be considered in consultation with USAID. ☒ Facility or Service Assessment/Survey (list type of facility or service of interest, and purpose of inquiry) ☒ Observations (list types of sites or activities to be observed, and purpose of inquiry) ☒ Cost Analysis (list costing factors of interest, and type of costing assessment, if known) ☒ Data Abstraction (list and describe files or documents that contain information of interest, and purpose of inquiry) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 78 ☒ Case Study (describe the case, and issue of interest to be explored) ☒ Verbal Autopsy (list the type of mortality being investigated (i.e., maternal deaths), any cause of death and the target population) ☒ Rapid Appraisal Methods (ethnographic / participatory) (list and describe methods, target participants, and purpose of inquiry) ☒ Other (list and describe other methods recommended for this evaluation/assessment, and purpose of inquiry) If impact evaluation – Is technical assistance needed to develop full protocol and/or IRB submission? ☒ Yes ☒ No List or describe case and counterfactual” Case Counterfactual HUMAN SUBJECT PROTECTION The Analytic Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey, and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community in the public setting. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. An Informed Consent statement included in all data collection interactions must contain: ● Introduction of facilitator/note-taker ● Purpose of the evaluation/assessment ● Purpose of interview/discussion/survey GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 79 ● Statement that all information provided is confidential and information provided will not be connected to the individual ● Right to refuse to answer questions or participate in interview/discussion/survey ● Request consent prior to initiating data collection (i.e., interview/discussion/survey) ANALYTIC PLAN Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data. All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Quantitative data will be analyzed to calculate the indicators selected by the USAID steering committee which are included in GHSC-PSM’s M&E plan. Quantitative data will also be analyzed to assess cost efficiencies over the life of the Project to date and in comparison with the predecessor SCMS and DELIVER projects. Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide context, provide more insights than quantitative data can provide, and answer questions where other data do not exist. Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data) will allow the Team to triangulate findings to produce more robust evaluation results. The Evaluation Report will describe analytic methods and statistical tests employed in this evaluation. ACTIVITIES List the expected activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and Deliverables may overlap. Give as much detail as possible. Background reading – Several documents are available for review for this analytic activity. These include Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Project proposal, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (i.e., DHS and MICS). This desk review will provide background information for the Evaluation Team and will also be used as data input and evidence for the evaluation. Team Planning Meeting (TPM) – A four-day team planning meeting (TPM) will be held at the initiation of this assignment and before the data collection begins. The TPM will: GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 80 ● Review and clarify any questions on the evaluation SOW ● Clarify team members’ roles and responsibilities ● Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion ● Review and finalize evaluation questions ● Review and finalize the assignment timeline ● Develop data collection methods, instruments, tools and guidelines ● Review and clarify any logistical and administrative procedures for the assignment ● Develop a data collection plan ● Draft the evaluation work plan for USAID’s approval ● Develop a preliminary draft outline of the team’s report ● Assign drafting/writing responsibilities for the final report Briefing and Debriefing Meetings – Throughout the evaluation the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include all Evaluation Team experts, but will be determined in consultation with the USAID. These briefings are: ● Evaluation launch, a call/meeting among the USAID, GH Pro and the Team Lead to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH Pro will introduce the Team Lead, and review the initial schedule and review other management issues. ● In-brief with USAID, as part of the TPM. At the beginning of the TPM, the Evaluation Team will meet with USAID to discuss expectations, review evaluation questions, and intended plans. The Team will also raise questions that they may have about the project/program and SOW resulting from their background document review. The time and place for this in-brief will be determined between the Team Lead and USAID prior to the TPM. ● Workplan and methodology review briefing. At the end of the TPM, the Evaluation Team will meet with USAID to present an outline of the methods/protocols, timeline and data collection tools. Also, the format and content of the Evaluation report(s) will be discussed. ● In-brief with GHSC-PSM to review the evaluation plans and timeline, and for the project to give an overview of the project to the Evaluation Team. ● The Team Lead (TL) will brief USAID weekly to discuss progress on the evaluation. As preliminary findings arise, the TL will share these during the routine briefing, and in an email. ● An initial debrief between the Evaluation Team and USAID will be held at the end of the evaluation to present preliminary findings to USAID (evaluation steering committee, CORs, and Division Chiefs). During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the evaluation report. (Note: preliminary findings are not final and as more data sources are developed and analyzed these findings may change.) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 81 ● IP debrief/workshop will be held with the project staff and other stakeholders identified by USAID. This will occur following the final debrief with USAID, and will not include any information that may be deemed procurement sensitive or not suitable by USAID. ● A final debrief between the Evaluation Team and USAID supply chain team will be held at the end of the evaluation to present the final findings. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the final evaluation report. ● A final debrief between the Evaluation Team and USAID senior leadership will be held at the end of the evaluation to present final findings. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the final evaluation report. ● A final debrief between the Evaluation Team and USAID Executive Board will be held at the end of the evaluation to present final findings. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from USAID during the debrief in the final evaluation report. Data Collection – The evaluation team will conduct data collection. Selection of key informants and survey respondents (as needed) will be finalized during TPM in consultation with USAID. Following data collection, the Team will analyze, synthesize and interpret the data in order to answer the evaluation questions. Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations (see Analytic Report below). Report writing and submission will include the following steps: 1. Team Lead will submit draft evaluation report to GH Pro for review and formatting 2. GH Pro will submit the draft report to USAID 3. USAID will review the draft report in a timely manner, and send their comments and edits back to GH Pro 4. USAID will manage implementing partner(s)’s (IP) review of the report and compile and send their comments and edits to GH Pro. (Note: USAID will decide what draft they want the IP to review.) 5. GH Pro will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH Pro 6. GH Pro will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 82 7. Once Evaluation Report is approved, GH Pro will re-format it for 508 compliance and post it to the DEC. The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. Data Submission – All quantitative data will be submitted to GH Pro in a machine-readable format (CSV or XML). The datasets created as part of this evaluation must be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. It is essential that the datasets are stripped of all identifying information, as the data will be public once posted on USAID Development Data Library (DDL). Where feasible, qualitative data that do not contain identifying information should also be submitted to GH Pro. DELIVERABLES AND PRODUCTS Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. Deliverable / Product Timelines & Deadlines (estimated) Launch briefing/introductory meeting April 4, 2019 In-brief with USAID Points of Contact (POCs) April 30, 2019 In-brief with USAID CORs May 2, 2019 In-brief with USAID/PSM May 6, 2019 Workplan and methodology review briefing (questions, roles, sources, methods, instruments, analysis plan, timeline May 9, 2019 Routine check-ins with POC (call or email) Weekly PSM Country Directors meeting July 5-8, 2019 USAID SC Activity Managers meeting July 22-26, 2019 Share PP with POCs on the 5th POCs and CORs morning GH SC50 - afternoon September 19, 2019 PSM – PP by themselves; provide feedback September 20, 2019 Draft report Submit to GH Pro: September 6, 2019 GH Pro submits to USAID: September 13, 2019 Findings debrief for GH Senior Staff September 25, 2019 Debrief for GHSC-50 team October 21, 2019 Feedback from USAID on draft report October 13, 2019 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 83 Draft 2 of report Submit to GH Pro: October 25, 2019 Submit to USAID: October 30, 2019 Feedback from USAID on Draft 2 due November 13, 2019 Final report Submit to GH Pro: November 20, 2019 Submit to USAID: November 25, 2019 Raw data (cleaned datasets in CSV or XML with code sheet) October 17, 2019 Report Posted to DEC December 13, 2019 Estimated USAID review time Average number of business days USAID will need to review the Report? 15 Business days TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Evaluation/Assessment team: When planning this analytic activity, consider: ● Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. ● Team leaders for evaluations/assessments must be an external expert with appropriate skills and experience. ● Additional team members can include research assistants, enumerators, translators, logisticians, etc. ● Teams should include a collective mix of appropriate methodological and subject matter expertise. ● Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity. Similarly, other analytic activities should have a specialist with methodological expertise. ● Note that all team members will be required to provide a signed statement attesting that they have no conflict of interest (COI), or describing the conflict of interest if applicable. Team Qualifications: Please list technical areas of expertise required for this activity: ● List desired qualifications for the team as a whole ● List the key staff needed for this analytic activity and their roles. ● Sample position descriptions are posted on USAID/GH Pro webpage ● Edit as needed GH Pro provided position descriptions Overall Team requirements: The evaluation team should consist of 4 key evaluators external to USAID with: ● In-depth knowledge and experience of supply chain management, with an understanding of supply chain and procurement needs/context in low- and middle-income countries; ● Demonstrated experience leading health sector project/program evaluation/analytics, utilizing both quantitative and qualitative methods; ● Experience in implementation of health activities in developing countries; ● Experience utilizing Microstrategy for data analysis or another, similar system. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 84 Key Staff 1 Title: Team Lead Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation team, and (5) leading briefings and presentations. Qualifications: ● Minimum of 10 years of experience in public health, which includes experience in implementation of health activities in developing countries ● Demonstrated experience leading health sector project/program evaluation/assessments, utilizing both quantitative and qualitative methods ● Excellent skills in planning, facilitation, and consensus building ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent skills in project management ● Excellent organizational skills and ability to keep to a timeline ● Good writing skills, with extensive report writing experience ● Familiarity with health supply chains ● Familiarity with USAID health programs, PMI and PEPFAR ● Familiarity with USAID policies and practices − Evaluation policy − Results frameworks − Performance monitoring plans Key Staff 2 Title: Evaluation Specialist/Data Analyst Roles & Responsibilities: Serve as a member of the evaluation team, providing quality assurance on analytic issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. This includes calculation of key performance indicators for GHSC-PSM (global and country level) utilizing the Project’s data and data available in BI&A. S/he will also calculate indicators, describe trends, and document data gaps or challenges. S/He will participate in all aspects of the evaluation, from planning, data collection, data analysis to report writing. Qualifications: ● At least 10 years of experience in USAID M&E procedures and implementation ● At least 5 years managing M&E, including evaluations and/or assessments ● Experience in design and implementation of evaluations and/or assessments ● Strong knowledge, skills, and experience in qualitative and quantitative analytic tools ● Experience implementing and coordinating others to implement surveys, key informant interviews, focus groups, observations and other evaluation and assessment methods that assure reliability and validity of the data. ● Experience in data management ● Able to analyze quantitative data, which will be primarily descriptive statistics and cross￾tabulations ● Able to analyze qualitative data ● Experience using analytic software GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 85 ● Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data ● Experience conducting secondary analysis of existing quantitative datasets ● Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation ● Strong data interpretation and presentation skills ● Proficient in written and spoken English ● Good writing skills, including experience writing evaluation and/or assessment reports ● Familiarity with USAID health programs/projects ● Familiarity with USAID and PEPFAR M&E policies and practices − Evaluation policies − Results frameworks − Performance monitoring plans Key Staff 3 Title: Supply Chain Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in health commodity procurement and supply chain systems strengthening. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications: ● At least 8 years’ experience with supply chain projects (including health commodity procurement); USAID project implementation experience preferred ● Expertise with supply chain strengthening and integration across health elements is desirable ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Proficient in English ● Good writing skills, including experience writing evaluation and/or assessment reports ● Experience in conducting USAID evaluations of health programs/activities Key Staff 4 Title: Costing Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing technical expertise to evaluate expenditures and assess cost efficiencies based on existing data. S/he will provide technical expertise for the expenditure trends by type of expenditure, and over time and across IPs. S/He will work remotely with the Evaluation team, providing input during planning, data analysis and report writing. Qualifications: ● At least 5 years of experience working with cost analysis for projects working in developing countries; USAID experience desirable ● Experience should include in-depth understanding of evaluating programs from a cost efficiency perspective, including identifying cost saving and efficiencies ● Experience working with projects to extracted expenditure data, including categorizing these expenditures into useful categories of project implementation ● Experience assessing value for money on health and development projects GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 86 ● Experience assessing areas for priority investment, and whether money is being spent where it should be in order to maximize desired results ● Experience working remotely as part of a team ● Proficient in English ● Good writing skills, with experience producing evaluation and/or technical report Other Staff Titles with Roles & Responsibilities (include number of individuals needed): US-based Program Assistant to work part time with the Evaluation Team to arrange interviews, meetings and logistics, and other support duties as needed by the Evaluation Team. S/He will assist the Evaluation Team to arrange interviews, meetings and logistics, and other support duties as needed by the Evaluation Team. S/He will conduct programmatic administrative and support tasks as assigned, and ensure the processes moves forward smoothly. Additionally, s/he will manage the uploading of the e-survey to the website (if part of the final methodology), and will routinely monitor it for response rates, as well as download the data as needed. The Technical Writer/Editor will assist with writing and editing of the report drafts, as well as the final report. This will include providing guidance on structure, language and layout of the report. Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation or assessment activity. ☒ Full member of the Evaluation Team (including planning, data collection, analysis and report development) – If yes, specify who: ☒ Some Involvement anticipated – If yes, specify who: ☒ No Staffing Level of Effort (LOE) Matrix: This LOE Matrix will help you estimate the LOE needed to implement this analytic activity. If you are unsure, GH Pro can assist you to complete this table. a) For each column, replace the label "Position Title" with the actual position title of staff needed for this analytic activity. b) Immediately below each staff title enter the anticipated number of people for each titled position. c) Enter Row labels for each activity, task and deliverable needed to implement this analytic activity. d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title. Level of Effort in days for each Evaluation Team member (one per position) Activity / Deliverable Evaluation/Analytic Team Team Lead Eval Spec/ Data Analyst Supply Chain Specialist Costing Specialist Technical Writer/Editor Program Assistant GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 87 1 Launch Briefing 0.5 2 Desk review 5 5 5 5 2 3 In-brief with USAID 0.5 0.5 0.5 0.5 0.5 4 Team Planning Meeting 9 9 9 9 4 5 Workplan and methodology briefing with USAID 0.5 0.5 0.5 0.5 0.5 6 Eval planning deliverables: 1) workplan with timeline, eval matrix, protocol (methods, sampling & analytic plan); 2) data collection tools 7 Data Collection DQA Workshop (protocol orientation/training for all data collectors; validation of data collection tools) 1 1 1 1 8 In-brief with project 0.5 0.5 0.5 0.5 0.5 9 Preparation for data collection 1 1 2 10 Data collection (including key informant interviews, group discussions) 35 12 18 23 3 11 Data (including secondary) analysis 6 8 6 8 1 12 Initial debrief with USAID steering committee and CORs with prep 1 1 1 1 1 13 IP debrief workshop with prep (with USAID) 1 1 1 1 0.5 14 Debrief with USAID supply chain team with prep 1 1 1 1 0.5 15 Debrief with USAID senior leadership with prep 1 1 1 1 0.5 16 Debrief with USAID Executive Board with prep 0.5 0.5 0.5 0.5 0.5 17 Briefing with GHSC-50 staff with prep 3 3 3 18 Draft report(s) 13 7 9 11 8 2 19 GH Pro Report QC Review & Formatting 20 USAID Report Review 21 Revise report(s) per USAID comments 6.5 3 5 5 5 0.5 22 Finalize and submit report to USAID 23 USAID approves report 24 Final copy editing and formatting 1 25 508 Compliance editing 26 Eval Report(s) to the DEC Revised Total LOE per person 85 52 62 72 14 19 A 6-day workweek permitted, as needed: ☒ Yes ☒ No 6-day or 7-day workweek approved while consultant is on travel status mobilizing or demobilizing from the place of performance: ☒ Yes Travel anticipated: List international and local travel anticipated by what team members. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 88 No international travel will be required for this activity; however, travel to USAID and GHSC￾PSM headquarters (in the Washington, DC area) is expected. LOGISTICS Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s): N/A List recommended/required type of Visa for entry into counties where consultant(s) will work Name of Country Type of Visa ☒ Tourist ☒ Business ☒ No preference ☒ Tourist ☒ Business ☒ No preference ☒ Tourist ☒ Business ☒ No preference ☒ Tourist ☒ Business ☒ No preference Clearances & Other Requirements Note: Most Evaluation/Analytic Teams arrange their own work space, often in conference rooms at their hotels. However, if a Security Clearance or Facility Access is preferred, GH Pro can submit an application for it on the consultant’s behalf. GH Pro can obtain Facility Access (FA) and transfer existing Secret Security Clearance for our consultants, but please note these requests, processed through AMS at USAID/GH (Washington, DC), can take 4-6 months to be granted. If you are in a Mission and the RSO is able to grant a temporary FA locally, this can expedite the process. FAs for non-US citizens or Green Card holders must be obtained through the RSO. If FA or Security Clearance is granted through Washington, DC, the consultant must pick up his/her badge in person at the Office of Security in Washington, DC, regardless of where the consultant resides or will work. If Electronic Country Clearance (eCC) is required prior to the consultant’s travel, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.] If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant may be required complete the one week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (consultants must register approximately 3-4 months in advance). Additionally, there will be the cost for additional lodging and M&IE to take this course. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 89 Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH Pro workspace and travel (other than to and from post). ☒ USAID Facility Access (FA) Specify who will require Facility Access: ☒ Electronic County Clearance (ECC) (International travelers only) ☒ High Threat Security Overseas Seminar (HTSOS) (required in most countries with ECC) ☒ Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days) ☒ GH Pro workspace Specify who will require workspace at GH Pro: GH Pro will provide workspace for team planning and prep work, as needed ☒ Travel -other than posting (specify): Travel to Washington, DC to meet with USAID and GHSC-PSM if not based in the Washington, DC area ☒ Other (specify): Specify any country-specific security concerns and/or requirements GH PRO ROLES AND RESPONSIBILITIES GH Pro will coordinate and manage the evaluation/assessment team and provide quality assurance oversight, including: ● Review SOW and recommend revisions as needed ● Provide technical assistance on methodology, as needed ● Develop budget for analytic activity ● Recruit and hire the evaluation/assessment team, with USAID POC approval ● Arrange international travel and lodging for international consultants ● Request for country clearance and/or facility access (if needed) ● Review methods, workplan, analytic instruments, reports and other deliverables as part of the quality assurance oversight ● Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH Pro website. If the report is internal, then copy editing/formatting for internal distribution. USAID ROLES AND RESPONSIBILITIES Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: Before Field Work ● SOW. ◦ Develop SOW. ◦ Peer Review SOW ◦ Respond to queries about the SOW and/or the assignment at large. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 90 ● Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. ● Documents. Identify and prioritize background materials for the consultants and provide them to GH Pro, preferably in electronic form, at least one week prior to the inception of the assignment. ● Local Consultants. Assist with identification of potential local consultants, including contact information. ● Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. ● Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). During Field Work ● Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. ● Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space). ● Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders. ● Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Field Work ● Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. ANALYTIC REPORT Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports) The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy). ● The report should not exceed 50 pages (excluding executive summary, table of contents, acronym list and annexes). ● The structure of the report should follow the Evaluation Report template, including branding found here or here. ● Draft reports must be provided electronically, in English, to GH Pro who will then submit it to USAID. ● For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing Evaluation Draft Reports found here. USAID Criteria to Ensure the Quality of the Evaluation Report (USAID ADS 201): ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 91 ● Evaluation methodology should be explained in detail and sources of information properly identified. ● Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. ● Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. ● If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. ● If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. Reporting Guidelines: The draft report should be a comprehensive analytical evidence-based evaluation/assessment report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation/assessment will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format: ● Abstract: briefly describing what was evaluated, evaluation questions, methods, and key findings or conclusions (not more than 250 words) ● Executive Summary: summarizes key points, including the purpose, background, evaluation questions, methods, limitations, findings, conclusions, and most salient recommendations (2-5 pages) ● Table of Contents (1 page) ● Acronyms ● Evaluation/Analytic Purpose and Evaluation/Analytic Questions: state purpose of, audience for, and anticipated use(s) of the evaluation/assessment (1-2 pages) ● Project [or Program] Background: describe the project/program and the background , including country and sector context, and how the project/program addresses a problem or opportunity (1-3 pages) ● Evaluation/Analytic Methods and Limitations: data collection, sampling, data analysis and limitations (1-3 pages) ● Findings (organized by Evaluation/Analytic Questions): substantiate findings with evidence/data ● Conclusions ● Recommendations GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 92 ● Annexes ◦ Annex I: Evaluation/Analytic Statement of Work ◦ Annex II: Evaluation/Analytic Methods and Limitations ((if not described in full in the main body of the evaluation report) ◦ Annex III: Data Collection Instruments ◦ Annex IV: Sources of Information • List of Persons Interviewed • Bibliography of Documents Reviewed • Databases • [etc.] ◦ Annex V: Statement of Differences (if applicable) ◦ Annex VI: Disclosure of Any Conflicts of Interest ◦ Annex VII: Summary information about evaluation team members, including qualifications, experience, and role on the team. The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports -------------------------------- The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USIAD separate from the Evaluation Report. -------------------------------- All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation will be submitted electronically to the GH Pro Program Manager. All datasets developed as part of this evaluation will be submitted to GH Pro in an unlocked machine￾readable format (CSV or XML). The datasets must not include any identifying or confidential information. The datasets must also be accompanied by a data dictionary that includes a codebook and any other information needed for others to use these data. Qualitative data included in this submission should not contain identifying or confidential information. Category of respondent is acceptable, but names, addresses and other confidential information that can easily lead to identifying the respondent should not be included in any quantitative or qualitative data submitted. USAID CONTACTS Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Christie Hershey Wezi Munthali Meaghan Douglas Title: Malaria Technical Advisor Supply Chain Performance Technical Advisor Supply Chain M&E Technical Adviser USAID Office GH/ID/MAL GH/PRH/CSL GH/OHA/SCH GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 93 Email: chershey@usaid.gov amunthali@usaid.gov medouglas@usaid.gov Telephone: 571-551-7173 571-551-7517 571-551-7311 Cell Phone: GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 94 FOLLOW-ON STATEMENT OF WORK Assignment #: 050 [assigned by GH-TAMS] Global Health Technical Assistance and Mission Support (GH-TAMS) Contract No. 7200AA19M0019 Task Order under IDIQ GS00Q14OADS121 TECHNICAL ASSISTANCE & MISSION SUPPORT STATEMENT OF WORK (SOW) Date of Submission: 01/06/20 Last update: 01/17/20 Type of Assignment: □ Management Support (e.g. coverage) ■ Technical Assistance I. TITLE: Completion of ‘The Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Project: A Mid-term Review for Lessons Learned and the Way Forward’ II. Requester / Client ■ USAID/Washington Office/Division: GH Bureau (OHA/SCH, ID/MAL, PRH/CSL) III. Funding Account Source(s): (Click on box(es) to indicate source of payment for this assignment) ■ 3.1.1 HIV □ 3.1.2 TB 3.1.3 Malaria □ 3.1.4 PIOET □ 3.1.5 Other public health threats □ 3.1.6 MCH 3.1.7 FP/RH □ 3.1.8 WSSH □ 3.1.9 Nutrition □ 3.2.0 Other (specify): GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 95 IV. Cost Estimate: $21,275 (Note: GH-TAMS will provide a final budget based on this SOW) V. Performance Period Expected Start Date (on or about): January 21, 2020 (tent.) Anticipated End Date (on or about): February 28, 2020 VI. Location(s) of Assignment: (Indicate where work will be performed) All work is expected to be completed in the consultant’s home office. This assignment will not require any travel. VII. Purpose of Consultant Assignment (Purpose and need for consultant) This assignment builds on the recently commissioned ‘The Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Project: A Mid-term Review for Lessons Learned and the Way Forward’ that sought to assess and document the progress and performance of the GHSC-PSM Project, near its midpoint, in meeting its core objectives. Insufficient time under the previous evaluation contract mechanism did not allow the Review Team to respond key questions and potential areas of disagreement posed by USAID and the GHSC-PSM Project following review of the final report. Responses to these questions will help improve the clarity and accuracy of the report. USAID therefore would like to re-engage the Review Team, namely the Review Team Leader (Connie Carrino), to address outstanding issues through the GH-TAMS mechanism. VIII. Objectives of Consultant Assignment Consistent with the ‘Purpose,’ the consultant will: (1) address outstanding questions and areas requiring additional clarification (to the extent feasible) put forth by the USAID supply chain management team and the GHSC-PSM Project; and (2) revise the draft report document accordingly for dissemination. IX. Background for Consultant Assignment (Background of work related to consultant assignment) Reference should be made to the attached Global Health Program Cycle Improvement Project (GH Pro) Assignment #679 Statement of Work and related deliverables. X. SCOPE OF WORK Description of Work & Responsibilities Describe the work to be done by the consultant(s) and what he/she/they will be responsible for. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 96 As noted in previous sections, the consultant will respond to USAID’s and GHSC-PSM’s questions and attempt to reconcile areas of disagreement/difference. The consultant will then update the final report document produced under the GH Pro Assignment #679. Major Tasks & Activities Identify major tasks the consult(s) are expected to implement 1. Document relevant responses according to questions and remarks outlined by USAID and GHSC￾PSM. 2. Incorporate additional information and/or adjust existing report content. XI. DELIVERABLES & MILESTONES List all deliverables in the table below. Provide estimated timelines and deliverable deadline for each. Deliverable / Milestone Timelines & Deadlines (estimated) In-brief with USAID Points of Contact (POCs) to discuss outstanding questions and areas of difference. January 28, 2020 Discussion with GHSC-PSM Project to review questions and feedback. TBD Document detailing responses to combined USAID and GHSC-PSM questions and remarks. TBD Final Report Submit to GH-TAMS: TBD Submit to USAID: TBD Report Posted to DEC February 28, 2020 Estimated USAID review time Average number of business days USAID will need to review deliverables requiring USAID review and/or approval? 3-5 Business days XII. STAFFING Consultant(s) Needed: Individual Consultant ● Team of 2 # of Consultants (estimated) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 97 List the consultant(s) needed for assignment. If only one consultant is requested, only complete information for Consultant 1. Consultant 1: Lead Consultant for the Mid-term Review for Lessons Learned and the Way Forward for GH Supply Chain. NOTE: This position will be consistent with the previous assignment under GH Pro.) Roles & Responsibilities: The consultant will be responsible for: (1) ensuring that all deliverables are met in a timely manner; and (2) liaising with all stakeholders to ensure the final report document addresses outstanding issues to the extent feasible. Qualifications: ● Minimum of 10 years of experience in public health, which includes experience in implementation of health activities in developing countries ● Demonstrated experience leading health sector project/program evaluation/assessments, utilizing both quantitative and qualitative methods ● Excellent skills in planning, facilitation, and consensus building ● Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders ● Excellent skills in project management ● Excellent organizational skills and ability to keep to a timeline ● Good writing skills, with extensive report writing experience ● Familiarity with health supply chains ● Familiarity with USAID health programs, PMI and PEPFAR ● Familiarity with USAID policies and practices − Evaluation policy − Results frameworks − Performance monitoring plans Consultant 2: Editor Roles & Responsibilities: The consultant will be responsible for assisting the Lead Consultant with editing and formatting the draft report and final version of the report. Qualifications: NOTE: This position will be consistent with the previous assignment under GH Pro. XIII. Staffing Level of Effort (LOE) Matrix Instructions: This LOE Matrix can help you estimate the LOE of each consultant attached to this assignment. If you are unsure, GH-TAMS can assist you to complete this table. a) For each column, replace the label "Title" with the actual position title of consultant needed for this activity. b) Enter Row labels for each activity, task and deliverable needed to implement this activity. c) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 98 d) At the bottom of the table total the LOE days for each consultant title. Level of Effort in days for each Consultant attached to this assignment Activity /Task / Deliverable List each activity, task and deliverable expected Consultants LOE (days) Lead Consultant Editor 1 In-brief with USAID 0.25 0.25 2 Discussion with GHSC-PSM Project (tent.) 0.25 0.25 3 Document responses to USAID and GHSC-PSM comments 4 4 Revise report per USAID and GHSC-PSM feedback 4 4 5 USAID Report Review 6 Revise report per final USAID feedback (tent.) 1.5 1.5 7 Finalize and submit report to USAID 2 8 USAID approves report 9 Final copy editing and formatting 10 508 Compliance editing 11 Report to the DEC Total LOE days 10 8 If overseas, is a 6-day workweek permitted X Yes □ No If on travel status, a 7-day workweek is permitted □ Yes □ No XIV. LOGISTICS Visa Requirements List any specific Visa requirements or considerations for entry to countries that will be visited by consultant(s): N/A List recommended/required type of Visa for entry into counties where consultant(s) will work Name of Country Type of Visa □ Tourist □ Business □ No preference □ Tourist □ Business □ No preference GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 99 □ Tourist □ Business □ No preference □ Tourist □ Business □ No preference Clearances & Other Requirements Note: Most Evaluation/Analytic Teams arrange their own work space, often in conference rooms at their hotels. However, if a Security Clearance or Facility Access is preferred, GH-TAMS can submit an application for it on the consultant’s behalf. GH-TAMS can obtain Facility Access (FA) and transfer existing Secret Security Clearance for our consultants, but please note these requests, processed through AMS at USAID/GH (Washington, DC), can take 4-6 months to be granted. If you are in a Mission and the RSO is able to grant a temporary FA locally, this can expedite the process. FAs for non-US citizens or Green Card holders must be obtained through the RSO. If FA or Security Clearance is granted through Washington, DC, the consultant must pick up his/her badge in person at the Office of Security in Washington, DC, regardless of where the consultant resides or will work. If Electronic Country Clearance (eCC) is required prior to the consultant’s travel, the consultant is also required to complete the High Threat Security Overseas Seminar (HTSOS). HTSOS is an interactive e-Learning (online) course designed to provide participants with threat and situational awareness training against criminal and terrorist attacks while working in high threat regions. There is a small fee required to register for this course. [Note: The course is not required for employees who have taken FACT training within the past five years or have taken HTSOS within the same calendar year.] If eCC is required, and the consultant is expected to work in country more than 45 consecutive days, the consultant may be required complete the one week Foreign Affairs Counter Threat (FACT) course offered by FSI in West Virginia. This course provides participants with the knowledge and skills to better prepare themselves for living and working in critical and high threat overseas environments. Registration for this course is complicated by high demand (consultants must register approximately 3-4 months in advance). Additionally, there will be the cost for additional lodging and M&IE to take this course. Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH-TAMS workspace and travel (other than to and from post). □ USAID Facility Access (FA) Specify who will require Facility Access: □ Electronic County Clearance (ECC) (International travelers only) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 100 □ High Threat Security Overseas Seminar (HTSOS) (required in most countries with ECC) □ Foreign Affairs Counter Threat (FACT) (for consultants working on country more than 45 consecutive days) □ GH-TAMS workspace Specify who will require workspace at GH-TAMS: □ Travel -other than posting (specify): □ Other (specify): XV. GH-TAMS Roles and Responsibilities GH-TAMS will coordinate and manage the consultant or consulting team, and provide quality assurance oversight, including: ● Review SOW and recommend revisions as needed ● Provide technical assistance on methodology, as needed ● Develop budget for activity ● Recruit and hire consultants, with USAID POC approval ● Arrange travel and lodging for consultants to primary location of work, as needed ● Request for facility access and/or country clearance (if needed) ● Report production (when a report is a deliverable) – If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH-TAMS website. If the report is internal, then copy editing/formatting for Internal Distribution. XVI. USAID Roles and Responsibilities Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: Before Field Work ● SOW. ◦ Develop SOW. ◦ Peer Review SOW GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 101 ◦ Respond to queries about the SOW and/or the assignment at large. ● Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. ● Documents. Identify and prioritize background materials for the consultants and provide them to GH-TAMS, preferably in electronic form, at least one week prior to the beginning of the assignment. ● Local Consultants. Assist with identification of potential local consultants, including contact information. ● Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. ● Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). After Field Work ● Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XVII. USAID CONTACT PERSON Primary Contact Alternate Contact 1 Alternate Contact 2 Name: Christie Hershey Wezi Munthali Meaghan Douglas Title: Malaria Technical Advisor Supply Chain Performance Technical Advisor Supply Chain M&E Technical Adviser USAID Office/Mission : GH/ID/MAL GH/PRH/CSL GH/OHA/SCH Email: chershey@usaid.gov amunthali@usaid.gov medouglas@usaid.gov Telephone: 202-916-2094 202-916-2163 202-916-2348 Cell Phone: XVIII. REFERENCE MATERIALS Documents and materials needed and/or useful for consultant assignment. ● The Global Health Supply Chain - Procurement and Supply Management (GHSC-PSM) Project: A Mid-term Review for Lessons Learned and the Way Forward Report document (dated December 13, 2019) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 102 ● GH Pro Assignment #679 SOW ● USAID GHSC-PSM Mid-Term Review Statement of Clarifications Memo ● GHSC-PSM Statement of Differences Memo (dated December 9, 2019) GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 103 ANNEX II. COUNTRIES SERVED BY GHSC￾PSM Countries receiving commodities or technical assistance from GHSC-PSM, as of April 2019 (cumulative). Afghanistan Guinea Rwanda Angola Guyana St. Kitts and Nevis Antigua and Barbuda Haiti St. Lucia Bahamas Honduras St. Vincent & the Grenadines Bangladesh Indonesia Senegal Barbados Jamaica Sierra Leone Benin Kenya South Africa Botswana Laos South Sudan Brazil Lesotho Suriname Burkina Faso Liberia Tajikistan Burma Madagascar Tanzania Burundi Malawi Thailand Cambodia Mali Togo Cameroon Mauritania Trinidad and Tobago Chile Mozambique Uganda Colombia Namibia Ukraine Côte d’Ivoire Nepal Vietnam Democratic Republic of the Congo Nicaragua Yemen Dominican Republic Niger Zambia El Salvador Nigeria Zimbabwe eSwatini Pakistan Ethiopia Panama Ghana Papua New Guinea Guatemala Paraguay GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 104 ANNEX III. DATA COLLECTION INSTRUMENTS Guidelines for Key Informant Interviews (KIIs) Informed Consent Statement Thank you for making the time to talk with (us/me) today. If needed: USAID’s Bureau for Global Health has asked GH Pro to conduct an independent review of the Global Health Supply Chain – Procurement and Supply Management (GHSC-PSM) project to assess and document the progress and performance of GHSC-PSM, near its midpoint, in meeting its core objectives. The results of the review will also be used to inform the design of the “next generation” GHSC implementing architecture. As such, we are interviewing relevant PSM stakeholders about the past and current performance of PSM in meeting the health commodity supply needs of USG health programs worldwide, its ability to provide supply chain technical assistance, and its effectiveness in collaborating globally with other procurement and supply chain stakeholders. We will be examining these issues across GHSC-PSM’s four main Task Orders (HIV, malaria, FP/RH, MNCH). You were suggested as a key person to inform this activity and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned about GHSC-PSM. Before we begin, I want to let you know that any information or examples we gather during this interview process will not be attributed to any specific person; though, unattributed quotes may be used in the report. You are also free to not respond to any of the questions or stop the interview at any time. The interview will take approximately 45 minutes to one hour. Do (I/we) have your permission to begin? Before we begin, do you have any questions about this interview? [ ] Consent provided _________ (Interviewer/Recorder initials) Key Informant Interview Guides (by review question) Question 2: How has the GHSC-PSM Project addressed risks, bottlenecks, and/or inefficiencies, in the global supply chain system? KIIs: PSM functional team leads, and USAID TO CORs 1. What has been the biggest challenge or challenges for GHSC-PSM in ensuring on-time delivery (OTD) and on-time, in-full (OTIF) OTIF for health commodities? 2. Do you think PSM has improved the efficiency of the global commodities supply chain? a. If yes, how? b. If no, why not? GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 105 3. What do you believe have been the main driving factors for any changes in supply chain costs under PSM? Please provide examples. 4. What impact(s) have any cost changes had on PSM’s ability to fulfill its three main objectives (improved availability of health commodities, strengthened in-country supply chain systems, effective global collaboration)? 5. Has PSM reduced risk in the GH supply chain? a. If yes, where and how? b. If no, where and why? 6. What do you believe have been the main factors in changes to the cycle time in the GHSC-PSM supply chain? How have these factors varied across task order / commodities? 7. What difference has been made by PSM instituting a root cause analysis process? 8. Does PSM have a sufficient process for continuous improvement? What recommendations, if any, would you make to improve this process? Question 4: How have in-country supply chains performed in GHSC-PSM supported countries during the life of the project? What trends are observed? KIIs: USAID/W and USAID Mission staff; PSM HQ and PSM field staff 1. What do you believe have been the main contributions of GHSC-PSM in the areas of: a. improved availability of health commodities? b. strengthened in-country supply chain systems? c. effective global collaboration to improve the long-term availability of health commodities? 2. Do you believe the project on the right track for achieving the desired results? Why or why not? Please provide specific examples. 3. What have been the main challenges for the GHSC-PSM in ensuring the success of its activities? What corrective actions, if any, need to be taken? Please provide specific examples. 4. At which geographical level (national, provincial, district, etc.) have most of the improvements been demonstrated? Please provide examples. 5. What external contextual factors will have the greatest impact on GHSC-PSM’s implementation and how can any challenges be mitigated? 6. How has GHSC-PSM been able to collaborate with and build upon synergies of other supply chain projects funded by other development partners? Please provide examples. 7. How have supply chains receiving technical assistance via GHSC-PSM interventions advanced to greater self-reliance? Please provide examples. 8. Has the Ministry of Health and other local leadership taken ownership of GHSC-PSM’s activities? Please provide examples. 9. (Optional HQ) In which countries has GHSC-PSM had the most success and most challenges? Please explain why. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 106 Question 5: How has GHSC-PSM coordinated/collaborated with global development partners (such as The Global Fund, UNFPA, etc.) to mitigate the risk of stock-outs or other supply imbalances in country supply chains, from the central warehouse to facilities now and in the future? PSM KIIs: (TO leads, Systems Strengthening/Technical Assistance staff / Global Coordination staff/René Berger and Chris Scott) 1. How has PSM defined global collaboration around the issue of strengthening supply chains? 2. Which have you found most useful, collaborating internationally or at the country level? Please explain. 3. Which TOs benefit most from collaboration with global partners? Why? 4. When should USAID be present in partner coordination for health commodities versus PSM? 5. How has collaboration with global partners been a significant approach to mitigating the risk of stock￾outs and other supply imbalances? Why? 6. How has collaboration helped address supply imbalances from the central warehouse to facilities? Please explain. 7. What do you see as the key issues in health commodities and supply chain that should be addressed via global collaboration during the next few years? USAID KIIs: (CORs and USAID Senior Staff; subset of the GHSC50) 1. How has USAID defined global collaboration around the issue of strengthening supply chains? 2. Is collaboration at the international or country level most useful? Explain 3. Do some TOs benefit more than others from PSM collaboration with global partners? How and why? 4. When should USAID be present in partner coordination for health commodities versus PSM? 5. How has collaboration with global partners been a significant approach to mitigating the risk of stock￾outs and other supply imbalances? Why? 6. How has collaboration helped address supply imbalances from the central warehouse to facilities? Please explain. 7. What do you see as the key issues in health commodities and supply chain that should be addressed via global collaboration during the next few years? 8. What suggestions do you have for PSM’s global coordination activities in the future? USAID partner KIIs: (e.g. GF, Gates Foundation, WHO, RHN, UNFPA, etc.) 1. In what areas of health commodities and supply chain have you collaborated with USAID (e.g. global commodity security, mitigating stock-outs, addressing supply imbalances within a country)? Which have been most effective? 2. As part of that collaboration how did you work with USAID’s implementing partners? Which ones? [PROMPT: prompt with GHSC-PSM if not mentioned] 3. What are your expectations from these collaborative efforts? GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 107 4. Did the coordination involving GHSC-PSM mainly occur within an international forum or among stakeholders at the national level? Please explain [PROMPT: ask for countries intended to benefit]. What are the advantages and disadvantages of focusing on international versus country-level collaboration? 5. When should USAID be present in partner coordination for health commodities versus PSM? 6. What do you see as the key issues in health commodities and supply chain that should be addressed via global collaboration during the next few years? GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 108 ANNEX IV. SOURCES OF INFORMATION Persons Interviewed GHSC-PSM – Headquarters, Arlington, VA Grace Adeya, TO2 (Malaria) Director Natalie Albrow, PMU Country Director for Thailand, Laos, Cambodia, and Vietnam Haley Behre, M&E Specialist René Berger, SVP and IDIQ Director, GHSC-PSM Nagash Borse, Acting TO1 (HIV) Director Andrew Brown, Workforce Development Team Lead and Enabling Environment Michael Cohen, M&E Manager Anita Deshpande, Market Dynamics David DiSilvestro, Business Analyst Adam Dorius, Analyst Scott Dubin, Warehousing and Distribution Team Elias Epstein, TO2 Finance Melinda Fotopoulous, Lab ISC Manager Kate Gulitashvili, PMU Country Director for Ghana, Guinea, Sierra Leone & Central America Suzanne Gold, M&E Specialist Thuy Huong Ha, PMU Director for Asia Xuan-Mai Harpy, TO3/TO4 ISC Manager Andrew Inglis, Advanced Analytics Manager, HHS Irma Karsten, DCP Procurement Manager Colleen Karoliszyn, TO2 ISC Manager Dan Kiesa, Demand Planning Lead Analyst Juan Jaramillo, Plan Team Manager Claude Kadisi, TO3 & 4 Finance Director Andrew Lewis, PMU Director for Nigeria Rebecca Logan, PMU Manager, Non-Field Offices Drew Luten, Strategy Director, GSC Rosa Maldonado, TO1, Finance Rebecca Neagle, TO1 ISC Manager Thidiane Ndoye, TO3 Director Hua Ni, Supply Chain Optimization and Excellence Manager, GSC Shaun O’Neil, PMU Manager for East and South Africa Arthur Ostrega, M&E Specialist Ana De Paiva, QMU & Continuous Improvement Manager Heidi Pilloud, Director, Project Finance GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 109 Gabriel Pincus, Analyst Alan Pringle, GSC Director Ramesh Rajeswaran, GSC Leslie Rider, Managing Director, CLEAR Kiersten Rooke, M&E Associate, CLEAR Samantha Salcedo-Mason, Communications Chris Scott, SVP and Country Programs Director Kim Shelsby, Director, Supply Chain Solutions Division Stefania Slabyj, PMU Country Director for Rwanda, Uganda & Ethiopia Shane Smith, Operations John Stanton, PMU Director, West Africa, Latin America & Caribbean Linda Wennick, MIS Release Manager Danielle Wiedeman, PMU Director, South Sudan/Caribbean/Mozambique Stephanie Wohler, MIS Director Ralph Titus, Director, Health Systems Strengthening Jamil Tokhi, Quality Assurance Manager Ellen Tompsett, TO3, Global Collaboration Matt Wattleworth, Laboratory Team Lead GHSC-PSM – Field Offices Claude Bahati, Country Director, Guinea Michael Egboh, Country Director, Nigeria Phillip Kamutenga, Country Director, Malawi Deogratius Kimera, Country Director, Ghana Dimitri Peffer, Country Director, Mozambique Dan Rhodes, former Country Director, Nigeria Timothy Rosche, Country Director, eSwatini Muhammad Tariq, Country Director, Pakistan Daniel Tedesse, Country Director, Ethiopia Jean-Marc Vander Stichelen, Country Director, Zambia Jayne Waweru, Country Director, Kenya Kate Woods, Country Director, Côte d'Ivoire USAID, Bureau for Global Health Deborah Armbruster, Senior Technical Advisor, MCHN Messai Belayneh, GH/OHA/SCH Julia Bem, GH/OHA/SCH Alan Bornbusch, Chief, Commodity Security and Logistics (CSL), PRH Jematia Chepyator, MCHN GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 110 Meaghan O’Keefe Douglas, TO1; GH/OHA/SCH, POC for review Ramy Guirgus, PRH/CSL Lindizgya Gutierrez, COR TO2, GH/ID/Malaria Lisa Hare, Chief, Supply Chain Branch, Malaria, ID Christie Hershey, TO2 (PMI); GH/ID/Malaria; lead POC for review Clarice Johnson, PRH/CSL, Finance Alexis Leonard, GH/ID/Malaria Clerisse Lemke, GH/ID/Malaria James Maloney, Chief, Supply Chain for Health (SCH), OHA Bridget McHenry, PRH/CSL Glenn Milano, PRH/CSL Sherif Mowafy, COR GHSC-PSM IDIQ; Deputy Chief GH/OHA/SCH Wezi Munthali, TO3 (PRH); GH/PRH/CSL; POC for review Helen Petach, Senior Science Advisor, MCHN Kevin Pilz, PRH/CSL Sharmila Raj, COR GHSC-TA, PRH/CSL Joshua Rosenfeld, GH/OHA/SCH Reena Shukla, Health Officer MCH; COR TO4, MCHN Ellen Starbird, Director, GH/PRH Xavier Tomsej, COR TO1, GH/OHA/SCH John Vivalo, COR TO3, PRH/CSL Julie Wallace, Chief Malaria, ID Jennifer Wray, Pharmacist, GH/ID/Malaria USAID – Bureau for Management, Office of Acquisition and Assistance (OAA) Eduardo Elia, Division Chief, OAA Erin Long, Contract Officer for GHSC-PSM, OAA USAID – Supply Chain Activity Managers Khadidiatou Hadi, Senegal Ugbede Abu, Malawi Adenike Adelanwa, Nigeria Bekele Ashagire, Ethiopia Lamine Bangoura, Guinea Joana Cardao, Angola/Southern Africa Umesh Gupta, Nepal Leah Hasselback, Mozambique Pierre R. Julien, Haiti Noah Kafumbe, Uganda GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 111 Lucien Kalenga, DRC Simplice Kamdem, Côte d’Ivoire William Kanweka, Zambia Nu Nu Khin, Burma Lesego Kitso, Botswana Babatunji Odelola, Nigeria Khalid Mahmood, Pakistan Lumbani Makwakwa, Malawi Alice Micheni, Kenya Ricardo Missihoun, Benin Tiwonge Mkandawire, South Africa Lulu Msangi, Tanzania Hilary Francis Mwale, Lesotho Suzan Nakawunde, Uganda Patrick Nganji, Cameroon Ashley Smith, Tanzania Althea M. Spence, Jamaica Dessalegn Tesfaye, Rwanda Aligui Yattara, Mali Sixte Zigirumugabe, Ghana Rabson Zyambo, Zambia GLOBAL PARTNERS Martin Auton, Global Fund Milka Dinev, Maternal Health Working Group, Reproductive Health Supply Chain (RHSC) Alexis Heaton, RHSC Hitesh Hurkchand, ISG Coordinator, Africa Resource Centre Mouna Jarmouni, Global Fund Lenny Kosicki, U.S. State Department, Office of the Global AIDS Coordinator Desmond Koroma, UNFPA Brian McKenna, RHSC Roberto Mena, UNFPA Susan Nazzaro, Gates Foundation Cristina Paulu, UNFPA Martyn Smith, FP2020 Julia White, RHSC Tom Wood, World Bank/Woods International, LLC GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 112 Bibliography of Documents Reviewed Document review included the original requests for proposal for GHSC-PSM; the scopes of work for the original IDIQ contract, and the four global TO contracts, as well as GHSC-PSM annual workplans, quarterly and annual reports, approval matrixes, and ME&L plans. GHSC-PSM’s Performance Dataset and the Global Monthly Financial Reporting were used for Question 1 and Question 3 analyses. Following are additional sources used by the team. Brown, Andrew. 2018. GHSC-PSM Country Director’s HR systems Maturity Survey Results. GHSC-PSM PowerPoint presentation. Brown, Andrew, Wanda Jaskiewicz, Bridget McHenry, Erin Meier, and Dominique Zwinkels. 2018. Building Human Resources for Supply Chain Management. Arlington VA: USAID, Human Resources for Health, People that Deliver, and UNICEF. Carrino, Constance, Maria Miralles, and Regan Whitworth. 2016. Interim Evaluation for Systems to Improve Access to Pharmaceuticals and Services Project. Washington, D.C.: Global Health Program Cycle Improvement Project. Daymon, Bob. 2019. Using data to improve supply chain operations. CSCMP Hot Topics, January 17, 2019. Gartner, Inc. 2015. Introducing Supply Chain Benchmarking from Gartner. GHSC-PSM. 2017. Work instructions for updating and agreed delivery date and estimated delivery date and applying reason codes. PSM.PROC.WI.001, Rev.001, Dec. 22, 2017. GHSC-PSM. September 2018. Daily Dashboard, a 12-month Journey. PowerPoint presentation. GHSC-PSM. Circa 2018. Unmanned Aerial Vehicle Procurement Guide: specifications, questions and other criteria to consider. GHSC-PSM. Circa 2018. The Laboratory Network Approach for Procurement and Supply Management. GHSC-PSM PowerPoint presentation. GHSC-PSM. Circa 2018. From Local Decisions to Global Markets: A workshop on market dynamics for supply chain professionals. GHSC-PSM. January 2019. Mozambique PSM PSU. Internal memorandum. GHSC-PSM. April 2019. TransIT Overview. GHSC-PSM. April 15, 2019. Using Unmanned Aerial Vehicles for Health Commodity Transport and Delivery in Malawi: Activity Design Plan. GHSC-PSM. 2019. Ambient Climate Monitoring Activity. GHSC-PSM summary. GHSC-PSM. 2019. GSC Introduction and Brief History. PowerPoint presentation (for mid-term review). GHSC-PSM. 2019. System Strengthening Introduction. GHSC-PSM PowerPoint presentation. GHSC-PSM. 2017. Transition Milestones Update (chart). November 13, 2017. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 113 GHSC-PSM. nd. Requisition Order Pipeline Dashboard. GHSC-PSM. nd. Guidance for Making Sense of Reason Codes Handout. HFAC. October 2018. Oversight investigation: USAID global health supply chain contract. Washington D.C.: Members of House Committee on Foreign Affairs and Subcommittee on Africa, Global Health, Global Human Rights, and International Organizations. IFC Advisory Services, in association with GTZ and DFID. June 2018. The Monitoring and Evaluation Handbook. Lauro, Don and Dian Woodle. 2011. USAID DELIVER Project Mid-term Evaluation. Global Health Technical Assistance Project: Washington, D.C. Reproductive Health Supplies Coalition. May 2019. Global Family Planning VAN: Proof of Concept (phase 1) review. Brussels: RHSC Management Unit. Rogers, Roxana and Tim Clary. 2018. Assessment of USG Customer Satisfaction with USAID’s Global Health Supply Chain Program. Washington D.C.: Global Health Program Cycle Improvement Project. Silverman, Rachel, Janeen Madan Keller, Amanda Glassman and Kalipso Chalkidou. 2019. Tackling the Triple Transition in Global Health Procurement: Promoting access to essential health products through aid eligibility changes, epidemiological transformation, and the progressive realization of universal health coverage. Washington, D.C.: Center for Global Development, Final Report of CGD’s Working Group on the Future of Global Health Procurement. Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES). January 2017. Global Evaluation of USAID Supply Chain Investments Through SCMS and DELIVER. Sullivan, S.D., et al. 2014. Budget Impact Analysis—Principles of Good Practice: Report of the ISPOR 2012 Budget Impact Analysis Good Practice II Task Force. Value in Health, 17(1):5-14. The Global Fund. Guidance for Submission of an M&E Plan for Global Fund Grants. USAID. 2015. USAID Vision for Health Systems Strengthening: 2015-2019. Washington, D.C. USAID. 2016. ADS Chapter 201: Program Cycle Operational Plan. Wattleworth, Matt and Holynne Steppe. 2018. Laboratory Network Optimization and Integrated ample Transport Workshop – Zimbabwe. GHSC-PSM PowerPoint presentation. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 114 ANNEX V. EVOLUTION OF GHSC-PSM’S INDICATORS FROM FINDINGS, QUESTION 1. EVOLUTION OF GHSC-PSM’S INDICATORS (SIMPLIFIED) AND CURRENT DATA SOURCES (Gray, filled boxes are for indicators not listed in the corresponding year’s M&E Plan) Indicators (May 31, 2016) Indicators (January 3, 2018) Indicators (February 11, 2019) Data Source (2019) Objective 1: Improved availability of health commodities (global procurement and logistics) 1 On-time, in-full delivery On-time, in-full delivery ( On-time, in-full delivery ( ARTMIS 2 On-time delivery On-time delivery On-time delivery ARTMIS 3 On-time completion rate for QA processes On-time completion rate for QA processes QA Database 4 QA investigation report submission QA investigation report submission QA Database 5 Cycle Time (Average) Cycle time (average) Cycle time (average) ARTMIS 6 Absolute percent supply plan error Absolute percent supply plan error ARTMIS and Supply Plans 7 Inventory turns Inventory turns Inventory turns Regional Distribution Center (RDC) Warehouse Management Systems 8 Total landed cost Total landed cost Total landed cost ARTMIS & Monthly Financial Statements 9 Absolute Percent Error and variant Mean Absolute Percent Error 10 Absolute percent forecast error Absolute percent forecast error ARTMIS & Supply Plans 11 Temporary Waiver Percentage Temporary waiver percentage Temporary waiver percentage ARTMIS & Registration Documents 12 Product at risk percentage 13 Average percentage of shelf life remaining Average percentage of shelf life remaining RDC Warehouse Management Systems 14 Product loss due to expiry Product loss due to expiry RDC Warehouse Management Systems & Country Warehouse GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 115 Management Systems 15 Product loss due to theft, damage, and other causes Product loss due to theft, damage, and other causes Continual Improvement System & Country Warehouse Management Systems 16 Out of specification percentage Out-of-specification percentage Out-of-specification percentage QA Database 17 Supplier concentration 18 Framework contract percentage Framework contract percentage Framework contract percentage ARTMIS 19 Catalog Order Percentage 20 Percentage price variance Percentage price variance 21 Percentage of backlogged line items Percentage of backlogged line items ARTMIS 22 Average supplier rating score 23 Average vendor rating score Average vendor rating score ARTMIS & Vendor Scorecards 24 Average vendor rating score (QA lab vendors) Average vendor rating score (QA lab vendors) ARTMIS & Vendor Scorecards 25 Percentage of ‘complete’ submissions reported to GHSC-BI&A Percentage of complete submissions reported to GHSC-BI&A 26 Percentage of required files submitted to GHSC-BI&A Percentage of required files submitted to GHSC-BI&A GHSC-BI&A 27 Percentage of sampled ‘accurate’ submissions reported to GHSC-BI&A 28 Percentage of ‘timely’ submissions reported to GHSC-BI&A Percentage of required files timely submitted to GHSC￾BI&A Percentage of required files timely submitted to GHSC￾BI&A GHSC-BI&A 29 Percentage complete and on time submissions to global knowledge management platform 31 Supply chain policies, regulations, strategies, or standard operating procedures developed or updated with GHSC-PSM assistance. Supply chain policies, regulations, strategies, or standard operating procedures developed or updated with GHSC-PSM assistance. Narratives GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 116 32 Average percent variance between GHSC-PSM ARTMIS and GHSC-BI&A calculations of key supply chain indicators for Task Order 1 GHSC-BI&A Objective 2: Strengthened in-country supply chain systems 33 Stock-out rate at SDPs Stock-out rate at service delivery points Stock-out rate at service delivery points Country LMIS 34 Percentage of stock status observations in storage sites, where commodities are stocked according to plan, by level in supply system. Stocked according to plan at storage sites Stocked according to plan at storage sites Country Warehouse Management Systems 35 Service delivery point reporting rate to LMIS Service delivery point reporting rate to LMIS Service delivery point reporting rate to LMIS Country LMIS 36 Average rating of in￾country data confidence at the central, subnational, and SDP Average rating of in-country data confidence Average rating of in-country data confidence Data Quality Assessment 37 Percentage of countries conducting annual forecast reviews Percentage of required annual forecasts conducted Percentage of required annual forecasts conducted Quantifications 38 Percentage of countries conducting quarterly supply plan updates 39 Percentage of required supply plans submitted to GHSC-PSM Percentage of required supply plans submitted to GHSC-PSM Supply plans 40 Percentage of total spent or budgeted on procurement of commodities for public sector services by the government, U.S.G., the Global Fund, or other sources disaggregated by program. 41 Percentage of total spent or budgeted on procurement of commodities for public sector services, by funding source Percentage of total spent or budgeted on procurement of commodities for public sector services, by funding source Country finance records 42 Percentage of initially GHSC-PSM-supported supply chain functions carried out by national authorities that are done without external technical assistance Percentage of initially GHSC￾PSM-supported supply chain functions carried out by national authorities without external technical assistance Percentage of initially GHSC￾PSM-supported supply chain functions carried out by national authorities without external technical assistance Country assessment GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 117 43 Supply chain workforce loss ratio 44 Supply chain technical staff turnover rate Supply chain technical staff turnover rate Country workforce data or assessment 45 Percentage of countries that have a functional logistics coordination mechanism in place. Percentage of GHSC-PSM￾supported countries that have a functional logistics coordination mechanism in place Percentage of GHSC-PSM￾supported countries that have a functional logistics coordination mechanism in place Country assessment 46 Percentage of leadership positions in supply chain management that are filled by women (in countries where GHSC￾PSM is providing technical assistance related to workforce development) Percentage of leadership positions in supply chain management that are held by women Percentage of leadership positions in supply chain management that are held by women Country workforce data or assessment 47 Number of people trained Number of people trained Number of people trained Training records 48 Mean absolute percent consumption forecast error Absolute percent consumption forecast error Quantifications & Country LMIS 49 Percentage of GHSC-PSM￾procured or supported molecular instruments that remained functional (previously was a cross￾cutting indicator) Percentage of GHSC-PSM￾procured or supported molecular instruments that remained functional (previously was a cross￾cutting indicator) Country lab reports Objective 3: Effective global collaboration to improve long-term availability of health commodities 50 Number of new innovations that were developed, implemented, or introduced and are related to the health commodity market or supply chain best practices Number of innovations that were developed, implemented, or introduced, and are related to the health commodity market or supply chain best practices Number of innovations that were developed, implemented, or introduced, and are related to the health commodity market or supply chain best practices Narratives 51 Number of global advocacy engagements in support of improved availability of essential health commodities. Number of global advocacy engagements in support of improved availability of essential health commodities Number of global advocacy engagements in support of improved availability of essential health commodities Narratives Cross-cutting (from 2016 M&E Plan) 52 Objectives 1 & 2: Overall customer satisfaction rating for GHSC-PSM services 15 Objective 1 & 2: Percentage of GHSC￾See above GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 118 PSM-procured molecular instruments that remained functional during the reporting period 49 Objective 1 & 2: Percentage of product lost due to theft, damage, or expiry, while under GHSC-PSM control See above GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 119 ANNEX VI. PERFORMANCE DATASET REFERENCE GUIDE FROM FINDINGS, QUESTION 1. PERFORMANCE DATASET REFERENCE GUIDE ● At the line item level ◦ More granular than RO/PO/DO/IO, not as granular as shipments ● Includes all uncancelled line items for the life of project, including all items on hold ● Includes all purchase orders, distribution orders and replenishment orders ● Any data that may have multiple instances will show the most recent data (e.g., a line with two shipments will have two ship dates. The most recent available date will show in this field in the dataset) Notes on Fields: Task Order Task order associated with the RO Condom Adjusted TO All condom and lubricant line items, regardless of funding source, are marked as TO1. All other line items have their usual task order from their RO Country Destination Country RO Number PO DO IO Number Order Number Same as PO DO IO Number. For Distribution orders, “_##” are trimmed off Prime Line Number Prior to PO/DO generation, line number from RO. After PO/DO generation, line numbers are updated to PO/DO line number. Status Name Current ARTMIS status of line item TLP Indicator Global Supply Chain (GSC) or Decentralized Procurement (DCP) Order Type Purchase order, distribution order, or replenishment order. Blank prior to PO/DO release for non-replenishment orders Fulfillment Method Direct Drop (IOs and POs) or RDC (DOs) Transportation mode Land, Sea, or Air “Multiple” refers to line items with multiple shipments that have different modes Item tracer category Product category used for Finance and M&E reporting on GHSC-PSM Product Category Legacy product categories used for other types management and analysis Item ID Prior to vendor identification: 12-digit product SKU After vendor identification: 15-digit item SKU, unique to product and manufacturer Product Name Name of item, including pack size. UOM Unit of Measure (i.e., pack type) Base Unit Type of units contained within UOM/pack Base Unit multiplier Number of base units within the UOM/pack GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 120 Item Volume Item specifications from the catalog Item Volume UOM Item specifications from the catalog Item Weight Item specifications from the catalog Item Weight UOM Item specifications from the catalog Contract Number Emptoris contract number associated with the PO/IO Contract Title Contract type (IDIQ, BPA, BOA, SPA, FFP, FUP, etc.). Blank for distribution orders Framework contract Framework (IDIQ, BPA, BOA) or non-framework (all others) contract type Blank for Distribution Orders Health element Funding code: 31# health area + source (core, field mission, DOD, DFID, etc.) Estimated Lead Time in Days ELT from RO, converted from weeks to days Emergency Order Y/N Supplier Order Pick Up Country This comes from the address provided for the vendor. It might be a headquarters or manufacturing site that is not actually in the country where the order will be picked up. Vendor Incoterm Incoterm governing our agreement with the vendor Destination Incoterm Incoterm governing our agreement with the recipient Unit price List price Price of the item (same as unit price at this time) Order quantity Line item ordered quantity from the RO Line total List price * ordered quantity Shipped Quantity Sum of item quantity for all shipment bookings associated with the line item. (Shipments may be in any LMIS status, i.e., may not have been shipped or delivered yet. Erroneous or cancelled duplicate bookings often continue to flow to reporting, and can create quality problems in this field. Quantities for distribution orders are also often underrepresented, due to known issues with OMS and LMIS that are being worked on.) Delivery Value List price * Shipped Quantity Total Actual Freight Costs Sum of invoiced freight costs across all cost categories. For invoices covering multiple line items, costs are allocated proportionally according to (volume? Item qty? Item value?). In cases where (relevant volume data) is not available, costs are divided equally across all line items in the invoice. Users must be aware of invoicing time lags before trying to run analysis on this field. In-full delivery (IFD) If shipped quantity ≥ ordered quantity, then “Y” Otherwise, “N” If no shipped quantity, then blank On-time delivery (OTD) If there is no ADD, then blank If no shipments have been delivered and the delivery window (-14/+7 days) is not yet closed, then blank If the delivery window has closed and no shipments have been delivered, then “N” If all shipments have been delivered not on time, then “N” If at least one shipment has been delivered on time, then “Y” GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 121 NOTE: This field will populate as soon as the delivery window closes, even if the line item has undelivered shipments. Between -14 and 14 Same rules as OTD within a -14/+14 day delivery window Between -30 and 30 Same rules as OTD within a -30/+30 day delivery window OTIF categories On-time in-full: The full ordered quantity of the line item is delivered within the delivery window (-14/+7) On time not in full: Some but not all of the ordered quantity was delivered within the window. Can occur with split lines, where some shipments fall within the window and some outside, or with short-shipped lines, where the shipped quantity is less than the ordered quantity. Not on time in full: All shipments were delivered outside the delivery window, but the shipped quantity ≥ ordered quantity Not on time not in full: All shipments are delivered outside the delivery window and the shipped quantity < ordered quantity NOTE: If there are any bookings associated with the line item, this field will populate as soon as the delivery window closes, even if the line item has undelivered shipments. OTIF Between -14 and 14 Same rules as OTIF within a -14/+14 day delivery window OTIF Between -30 and 30 Same rules as OTIF within a -30/+30 day delivery window Average Days Late Binned Categorization of delivery timeliness, based on average difference between AcDD and ADD of all shipments associated with the line. Categories are: <-14 (i.e., early), Between -14 and 7 (i.e., on time), Between 8 and 30, >30 Average Days Late Average difference between AcDD and ADD of all shipments associated with the line. Note that a line item with an early shipment and late shipment can average out to an “on time” line item in this calculation, but the line is still not on time. Crosscheck with Number of Shipments if Average Days Late and OTD do not seem to agree. Number of Shipments Number of LMIS bookings associated with a line item. Bookings may be in any status (delivered or undelivered). Delivery Progress 0 – No associated shipments have been delivered 1 – Some but not all associated shipments have been delivered 2 – All associated shipments have been delivered Line Delivery Status Blank – Line item does not have an ADD, so timeliness cannot be determined Delivered – Early: All delivered shipments associated with this line were delivered early. Delivered – On time: At least one shipment associate with the line item was delivered on time. Delivered – Late: All delivered shipments associated with this line were delivered late. Undelivered: All shipments associated with this line item have not yet been delivered, and the on-time delivery window has not yet closed. Status will be updated as shipments are delivered and/or the delivery window closes. Undelivered – Late: All shipments associated with this line item have not yet been delivered, and the on-time delivery window has closed. This means the line item cannot be delivered on-time given the current ADD. On Hold – Line item is in a WCS hold status. Agreed Delivery Date Agreed Delivery Date from RO. If ADD has been changed with a reason code, this will be the current ADD. ADD Year GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 122 ADD Year-Month ADD Fiscal Year ADD Fiscal Quarter Year Order Entry Date OED Year OED Year-Month OED Fiscal Year OED Fiscal Quarter Year Requested Delivery Date RDD Year RDD Year-Month RDD Fiscal Year RDD Fiscal Quarter Year Estimated Delivery Date From OMS EDD Year EDD Year-Month EDD Fiscal Year EDD Fiscal Quarter Year Latest Actual Delivery Date Actual Door Delivery Date of the most recently delivered shipment associated with the line item LAcDD Year LAcDD Year-Month LAcDD Fiscal Year LAcDD Fiscal Quarter Year RO Clarified Date Manually entered WCS milestone date RO Sent to Plan Fulfillment Date Manually entered WCS milestone date RO Sent Sourcing RFX Event Date Manually entered WCS milestone date Initial PSM Source Approval Date System-generated approval date Recipient Approval Date System-generated approval date USAID Approval Date System-generated approval date PO Released for Fulfillment Date System-generated date of PO/DO/IO release to LMIS PO Release Year PO Release Month PO Release Fiscal Year PO Release Fiscal Quarter Committed Goods Available Date From OMS Actual Goods Available Date From OMS GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 123 QA Complete Date Max Pick Update From LMIS. Max Ship Date From LMIS Max Departure Date From LMIS Max Arrival Date From LMIS. Arrival date at port (not delivery to recipient) Order Cycle Time (Latest Actual Delivery Date – Order Entry Date) +1 RO on Hold – Client WCS hold status duration RO on Hold – Recipient WCS hold status duration RO on Hold – Pre-RFx Stage WCS hold status duration RO on Hold – Tech Package Stage WCS hold status duration RO on Hold – Clarifications Stage WCS hold status duration On Hold Pending PO Release OMS hold status duration On Hold Pending RO Release OMS hold status duration Line Item Total Dwell Sum of all hold status durations RO Validation For DOs: (RO Sent to Plan Date – Order Entry Date) For POs: (RO Sent to Sourcing Date – Order Entry Date) Sourcing and Planning For DOs: (Recipient Approval Date – RO Sent to Plan Date) For POs: (Recipient Approval Date – RO Sent to Sourcing Date) USAID Approval USAID Approval Date – Recipient Approval Date Process PO/DO PO Released for Fulfillment Date – USAID Approval Date Manufacture Actual Goods Available Date – PO Released for Fulfillment Date Pick Up Max Pick Up Date – Actual Goods Available Date Deliver Latest Actual Delivery Date – Max Pick Up Date Quality Assurance QA Complete Date – Actual Goods Available Date Reason Code Most recent Reason Code applied to line item Reason Code Text Text associated with current reason code Changed Field Name of the OMS field that was changed in the reason code process Reason Code Duration Duration associated with current reason code Reason Code SharePoint Link SharePoint link to reason code documentation GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 124 ANNEX VII. COMMENTS AND POTENTIAL GAPS ON “CORE” INDICATORS Following are comments, including comments on potential gaps, on the “Core” indicators as numbered in the report text in Table 3, p. 12. While initially prepared by the Review Team, GHSC-PSM and USAID assisted with updates and clarifications during the report review process. Indicator Comments and Potential Gaps 1 On-time, in-full delivery ● Window for what is considered to be “on-time” is not globally standardized; this does not permit easy performance comparability with other similar organizations (e.g., pharmaceutical distributors) ● Will not always be reported accurately in the quarterly reports since some deliveries will occur on-time, but PODs will not come in until past the external reporting period (although updated in annual reports) 2 On-time delivery ● Window for what is considered to be “on-time” is not globally standardized, which does not permit easy comparability ● Will not always be reported accurately in the quarterly reports since some deliveries will occur on-time, but PODs will not come in until past the external reporting period (although updated in annual reports) 3 Cycle time (average) ● Includes dwell times, which may or may not be within GHSC-PSM’s manageable interest ● May not be an accurate measure as better forecasting (and order placement) occurs further in advance102 5 Total landed cost A number of limitations (e.g., exclusion of quality assurance [QA] costs,103 commodity price changes) are discussed in the M&E plan 6 Temporary waiver percentage Still in the process of being developed and measured at the time of the review. Since then, per USAID correspondence, reporting for TO2 started in FY 2019 Q2, and FY 2019 Q3 for TO3 and had continued every quarter since then. There is no requirement for TO1 and TO4. 7 Average percentage of shelf life remaining Limitation as this metric only reports on products held in GHSC-PSM-managed warehouses. 9 Average vendor rating score ● Is actually a measure of “key” vendor rating ● What constitutes a “key” vendor is not clearly defined in the M&E plan, although “critical and strategic” vendors and their criteria for inclusion are referenced in an SOP manual 11 Stock-out rate at service delivery points ● For clarity, the numerator could be re-written to read “Number of service delivery points (SDPs) that were stocked out of a specific tracer product 102 At the time of the evaluation, this was being addressed through an ARTMIS change to deduct dwell time from the order fulfillment cycle. The first dwell-adjusted cycle times should be reported starting in FY 2021. 103 GHSC-PSM does calculate a second version of total landed cost with quality assurance for TO2. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 125 according to the ending balance of the logistics report for the time period of interest (in GHSC-PSM supported regions104)” ● For clarity, the denominator could be rewritten to read “Total number of SDPs that offer the tracer product and which submitted logistics reports in the time period of interest (in GHSC-PSM-supported countries)” 13 Service delivery point reporting rate to LMIS Does not provide information on the data quality (i.e., data may be 100 percent reported but 100 percent inaccurate) 14 Average rating of in￾country data confidence ● Per the project Performance Indicator Reference Sheet (PIRS), it is unclear what this indicator is attempting to measure, as it does not provide sufficient evidence on several of the normal data quality measures (validity, integrity, precision, reliability) and only partially measures timeliness. ● GHSC-PSM limited measurement to availability, accuracy, and timeliness as an effort to develop an assessment/indicator that was not too resource￾intensive to implement on a wide scale. 15 Percentage of required annual forecasts conducted Does not measure if in-country counterparts have taken ownership/leadership of the process or are still primarily reliant on technical assistance 16 Percentage of required supply plans submitted to GHSC-PSM Does not measure if in-country counterparts have taken ownership/leadership of the process or are still primarily reliant on technical assistance 17 Percentage of total spent or budgeted on procurement of commodities for public sector services, by funding source ● May be difficult to accurately measure because of the potential for double￾counting (i.e., direct budget support, World Bank loans, or Global Fund grants implemented by government Project Management Units, etc.) ● Additional limitations are given in the M&E plan 18 Percentage of targeted supply chain activities in which the host country entity has achieved technical independence with GHSC-PSM technical assistance. ● In the process of being redefined during the review. ● Indicator B-8 has since been defined as “Percentage of targeted supply chain activities in which the host country entity has achieved technical independence with GHSC technical assistance,” and awaits final USAID approval. 20 Percentage of GHSC￾PSM-supported ● Functional," although given some definition, could be more precisely defined (“good representation,” “relevant” stakeholders, etc.) 104 The definition of “GHSC-PSM-support” in the PIRS for this indicator focuses on supported “regions” as opposed to countries. For instance, in Nigeria, GHSC-PSM only reports on stockouts from the states in which GHSC-PSM is working. It is not a national￾level picture, because it does not work in all states. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 126 countries that have a functional logistics coordination mechanism in place ● GHSC-PSM asks whether the mechanism has “formal legislative or administrative status,” and whether it has “formal written terms of reference,” but not whether an on-going, supporting entity (e.g. Secretariat) is in place. ● More precise measures (thresholds for quorum met, time to complete follow-up actions, etc.) could be developed to mitigate some of the subjectivity 23 Number of innovations that were developed, implemented, or introduced, and are related to the health commodity market or supply chain best practices ● This is a multiple composite indicator (developed, implemented, introduced), (health commodity market or supply chain best practices) ● It is unclear how this will be made attributable to GHSC-PSM’s efforts ● This is an output indicator 25 Percentage of required files submitted to GHSC￾BI&A Does not measure quality of information submitted 26 Percentage of required files timely submitted to GHSC￾BI&A Only measures one of the five data quality measures 27 Product loss due to expiry ● Only countries in which GHSC-PSM is directly responsible for commodity storage (at any level) and/or distribution are required to report on this indicator. ● However, this indicator in addition to GHSC-PSM’s regional distribution centers, the in-country medical stores. This would imply that GHSC-PSM is either fully managing and/or staffing those corresponding medical stores, which is, most likely, not the situation. 28 Product loss due to theft, damage, and other causes ● This indicator focuses on those products under GHSC-PSM’s “control.” ● It also includes products in transit to the “customer” defined as the recipient designated to receive the delivery. 29 Number of global advocacy engagements in support of improved availability of essential health commodities ● This is partially outside of GHSC-PSM’s manageable interest, as travel must be approved by USAID ● The measure “forum or meetings that happen in a global setting” is limiting as “advocacy engagements” can have a broader definition ● This is an output indicator GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 127 ANNEX VIII. STOCKED ACCORDING TO PLAN RATES, BY COUNTRY Source: GHSC – PSM, “Global Supply Chain M&E Indicator Performance Report, FY 2019 Quarter 2, January - March 2019,” p. 73. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 128 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 129 ANNEX IX. GRAPHS OF OTIF AND OTD RESULTS FROM FINDINGS, QUESTION 1. GRAPHS OF OTIF AND OTD – REPORTED VERSUS ACTUAL RESULTS 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 IDIQ OTIF - Reported versus Actual IDIQ Reported IDIQ Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO1 (HIV) TO1 Reported TO1 Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 130 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO2 (Malaria) T02 Reported T02 Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO3 (FP/RH) TO3 Reported TO3 Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 131 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO4 (MNCH) TO4 Reported TO4 Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 IDIQ OTD - Reported versus Actual IDIQ Reported IDIQ Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 132 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO1 (HIV) TO1 Reported TO1 Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO2 (Malaria) T02 Reported T02 Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 133 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO3 (FP/RH) TO3 Reported TO3 Actual 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% FY2016 Q4 FY2017 Q1 FY2017 Q2 FY2017 Q3 FY2017 Q4 FY2018 Q1 FY2018 Q2 FY2018 Q3 FY2018 Q4 FY2019 Q1 FY2019 Q2 FY2019 Q3 TO4 (MNCH) TO4 Reported TO4 Actual GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 134 Notes from GHSC-PSM (OTIF) "Actual" figures are based on Performance Dataset, downloaded 19 August 2019. Blanks indicate that there were no line items with Latest Actual Delivery Dates in the period (i.e., denominator = 0). As noted in GHSC-PSM's 30 July 2019 submission of the FY 2019 Q3 quarterly report, OTIF, OTD, and other delivery figures submitted in that report were provisional, pending USAID approval of some reason codes. Delivery indicators were to be recalculated following receipt of the USAID first round of feedback and updated in GHSC-PSM's second submission. Final delivery data in the second submission will be based on the 19 August 2019 dataset. The FY 2019 Q3 "Reported" data above is therefore based on the 19 August 2019 dataset and is identical to the "Actual" data. From FY 2016 Q4 to FY 2017 Q3, the data source for this indicator was manual or semi-manual Excel shipment trackers. This dataset was very difficult to keep updated and complete. Our actual number of line items delivered was much higher than was accounted for in those datasets, which is the main driver of variance between reported and current data for those periods. From FY 2017 Q4 to FY 2018 Q2, the M&E and Data Analytics team developed a method to combine K+N's LMIS Shipment Tracker with ARTMIS' Requisition Order History report, which resulted in a significantly more complete and accurate dataset. From FY 2018 Q3 onward, the M&E team moved over to the Performance Dataset, a unified, high￾quality ARTMIS report, to run all procurement and delivery indicators. Notes from GHSC-PSM (OTD) "Actual" figures are based on Performance Dataset downloaded 19 August 2019. Blanks indicate that there were no line items with Agreed Delivery Dates in the period (i.e., denominator = 0). As noted in GHSC-PSM's submission of the FY 2019 Q3 quarterly report, OTIF, OTD and other delivery figures submitted in that report were provisional, pending USAID approval of some reason codes. Delivery indicators were to be recalculated following receipt of the USAID first round of feedback and updated in GHSC-PSM's second submission. Final delivery data in the second submission will be based on the 19 August 2019 dataset. The FY 2019 Q3 "Reported" data above is therefore based on the 19 August 2019 dataset and is identical to the "Actual" data. From FY 2017 Q4 to FY 2018 Q2, the M&E and Data Analytics team developed a method to combine K+N's LMIS Shipment Tracker with ARTMIS' Requisition Order History report, which gave us a more complete and accurate delivery dataset than we had had before. The implementation of this method coincided with the first OTD reporting. From FY 2018 Q3 onward, the M&E team moved over to the Performance Dataset, a unified, high￾quality ARTMIS report, to run all procurement and delivery indicators. GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 135 ANNEX X. CALCULATING OTIF AND OTD INSTRUCTIONS Instructions for calculating OTIF and OTD as received from USAID (July 2019) Instructions for calculating OTIF and OTD as received from GHSC-PSM (August 2019) Using data from ARTMIS (as the primary data source) and GHSC-PSM monthly financial statements calculate the following metrics: OTIF (A1a), OTD (A1b), and Total Landed Cost (TLC) (A5) The relevant report to look at in ARTMIS for OTD and OTIF calculations would be the “Requisition Order History” report, accessible through Team Content/Predefined Reports/Order Tracking. The relevant fields to use are listed below: 1. “Task Order” will provide the task order disaggregation. 2. Note that each unique combination of “RO#”, “PO/DO#” and “Line Number” refers to one order line item. In this report, there can be multiple rows for each order line item, if the order line item has been shipped in multiple shipments. In this case, the “KN shipment number” field will reference the shipment number associated with each shipment. OTD and OTIF calculations are based on the number of unique order line items, not shipments. 3. “Status” provides the current status of the order line item. It will need to be filtered for all non-cancelled orders while calculating the denominator for OTD and OTIF. “Status” equal to Shipment Delivered indicates that an order line item- shipment combination has been delivered; it does not indicate if the order line item has been delivered in full or partially, this will need to be gleaned using the “Ordered quantity” and “Shipped quantity” fields. 4. “Agreed Delivery Date” will provide the agreed delivery date for each order line item. 5. “Actual delivery date” provides the actual date of delivery of the shipment What is the Performance Dataset? • A pre-defined Cognos report developed by the M&E and MIS teams to serve as the definitive data source for select M&E indicators Dataset includes: ● Order and delivery data at the line item level ● All uncancelled line items for the life of the project ● All Purchase Orders, Distribution Orders, and Replenishment Orders ● Line items on hold Data is updated twice daily at 6 AM and 7 PM ET Access is restricted to users who receive training from the M&E team and approval from the M&E Manager Which indicators are run out of the Performance Dataset? • A1a. On-time, In-Full Delivery • A1b. On-time Delivery • A3. Cycle time • A10. Framework contract percentage • A16. Backlog percentage • Delivery and Procurement volumes ($ value and # of line items) • Performance Dataset and/or other ARTMIS data also contributes to indicators A5, A6a, A6b, A7, C7b, and impact totals Calculating OTIF 1. Insert a Pivot Table 2. Add the following fields to Filters: ● Order Type: Select “Distribution Order, Purchase Order and Blank” ● Delivery Progress: Select “2” ● Latest Actual Delivery Date Fiscal Quarter Year: Select the relevant period 3. Add the following fields to Rows: ● Condom Adjusted Task Order 4. Add the following fields to Columns ● OTIF Categories 5. Add the following fields to Values: ● RO Number. ● “Value Field Setting” should be set to Count GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 136 associated with the order line item. In case of multiple shipments per order line item, each row will provide the actual delivery date associated with that particular order line item—KN shipment # combination. 6. “PO/DO#” can be blank, depending on the status of the order line item. 7. Likewise, “Agreed Delivery Date” can be blank if the order line item has not reached the status where it is assigned. This is usually assigned at the “Pending PSM source final processing” stage. The PSM MIS team can provide a list of the various stages in order processing and delivery and associated statuses if required. 8. “Actual Delivery Date” is blank until the order line item- shipment combination has been delivered. 9. An “Agreed Delivery Date” of 2099 indicates that the agreed delivery date has not been finalized yet. 10. “Ordered Quantity” provides the total quantity ordered against the order line item. In the case of multiple shipments for one order line item, the ordered quantity is repeated across the multiple shipment rows for that order line item. 11. “Shipped Quantity” provides the total quantity shipped against the order line item -shipment combination. In the case of multiple shipments for one order line item, each row has the specific quantity shipped against that order line item￾shipment combination. 12. There may be instances where the total shipped quantity against an order line item does not perfectly match the ordered quantity, yet the order line item is considered “shipped in full”. This happens due to manufacturing or shipping batch sizes and in these cases the difference between the two quantities is a very small percentage of the original ordered quantity. 6. Right-click into the Pivot table, select “Show Values As”, and select “% of Row Total” 7. On-Time, In-Full delivery percentages for each Task Order will calculate in the table under the column labeled “On-Time In-Full” Common Variations • Break down by product category: Add Item Tracer Category to Rows below Condom Adjusted Task Order • Calculate for multiple reporting periods: Move Condom Adjusted Task Order into Filters, and moved Latest Actual Delivery Date Fiscal Quarter Year to Rows. Use the “Row Labels” filter to remove future periods and blanks • Calculate by month: Use Latest Actual Delivery Date Year Month instead of Fiscal Quarter Year • Filter by country: Add Country to the Filters and select the relevant country/countries • Filter by DCP v. GSC: Add TLP Indicator to the Filters and select desired procurement method Calculating OTD by Task Order 1. Insert a Pivot Table 2. Add the following fields to Filters: ● Order Type: Select “Distribution Order, Purchase Order and Blank” ● Agreed Delivery Date Fiscal Quarter Year: Select the relevant period 3. Add the following fields to Rows: ● Condom Adjusted Task Order 4. Add the following fields to Columns ● On-time delivery (OTD) 5. Add the following fields to Values: ● RO Number. ● “Value Field Setting” should be set to Count 6. Right-click into the Pivot table, select “Show Values As”, and select “% of Row Total” 7. On-time delivery percentages for each Task Order will calculate in the table under the column labeled “Y” Common Variations • Break down by product category: Add Item Tracer Category to Rows below Condom Adjusted Task Order • Calculate for multiple reporting periods: Move Condom Adjusted Task Order into Filters, and moved Agreed Delivery Date Fiscal Quarter Year to Rows. Use the “Row Labels” filter to remove future periods and blanks • Calculate by month: Use Agreed Delivery Date Year Month instead of Fiscal Quarter Year • Filter by country: Add Country to the Filters and select the relevant country/countries • Filter by DCP v. GSC: Add TLP Indicator to the Filters and select desired procurement method GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 137 ANNEX XI. CONFLICTS OF INTEREST GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 138 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 139 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 140 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 141 GH SUPPLY CHAIN-PROCUREMENT & SUPPLY MANAGEMENT PROJECT: MID-TERM REVIEW / 142 ANNEX XII. SUMMARY BIOS OF EVALUATION TEAM Constance A. Carrino, Team Leader, provides advice to organizations designing, evaluating, and improving foreign assistance policies and programs. This includes leading program evaluations, strategy development, and stakeholder analyses. As a senior executive for USAID, she held key policy, leadership, and international cooperation positions, including: Director, Bureau for Global Health’s (GH) Office of HIV/AIDS and USAID’s deputy principal for President’s Emergency Plan for AIDS Relief (PEPFAR); U.S. Embassy to Japan’s Counselor for International Development; Director, Social Sector Restructuring, USAID Russia and co-chair of U.S.-Russia Maternal and Child Health Subcommittee, and Chief of the Health Policy and Sector Reform Division in GH’s Office of Health and Nutrition. Dr. Carrino is an economist with a national security policy degree and overseas working experience in 28 countries. Tim A. Clary, Evaluation Specialist/Data Analyst, is an international expert in designing, managing, monitoring, and evaluating health and development projects. His work includes developing, evaluating, and providing training in strategic planning, monitoring and evaluation systems, indicator dictionaries and data quality assurance guidelines, and conducting data quality assurance, as well as major elements of health systems strengthening. His clients have included leading multilateral (e.g., the Global Fund, IBRD, IFC, IADB) and bilateral (e.g., USAID, MCC, DFID, GIZ) agencies, United Nations organizations, and NGOs. Dr. Clary is an epidemiologist with specialties in infectious diseases, health systems governance, and environmental determinants of health, with working experience in more than 60 countries. Lisa H. Harrington, Supply Chain Specialist. As president of the lharrington group LLC (LHG), a woman￾owned small business, Ms. Harrington provides strategic consulting services to major private and public organizations in global supply chain risk and volatility management and supply chain security; supply chain best practices, strategy, and execution; supply chain technology, including big data analytics and artificial intelligence; supply chain physical network and management architecture design; robotics and other physical technologies; transportation, distribution, and fulfillment operations; e-commerce; and supply chain talent development. Her expertise covers life sciences and healthcare, defense/aerospace, technology, automotive, consumer goods, fashion/retail, ecommerce, industrial energy and third-party logistics service firms (3PLs), brokers and carriers, software application platforms/developers. Ms. Harrington is a Senior Research Scholar at the Center for Public Policy and Private Enterprise at the University of Maryland’ School of Public Policy, former faculty member at the Robert H. School of Business, University of Maryland, and former Associate Director, Supply Chain Management Research Center, University of Maryland. George M. Ruhago, Costing Specialist, is a physician and health economist with strong statistical and modelling emphases and a broad range of experience studying global public health issues, such as cost analyses of health supply chain, evaluating cost effectiveness of public health programs, estimating the global burden of disease, identifying risks factors for maternal and neonatal health, and cancer. He is widely published in peer reviewed publications (e.g., Lancet, Cost Effectiveness and Resource Allocation, Health Policy Plan). He is a health economics specialist with various certificates and interests, including costs effectiveness, systematic reviews, and health and human rights. Dr. Ruhago is a collaborator on Global Burden of Disease with the Institute of Health Metrics and Evaluation. Global Health Technical Assistance and Mission Support Project 1319 F Street NW, Suite 204 Washington, DC 20004 Phone: (1-202) 991-8600