i June 2020 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International, Inc. The Mid-term Performance Evaluation of GHSC-PSM/Nepal Report - FINAL ii GHSC-PSM/NEPAL MID-TERM PERFORMANCE EVALUATION June 2020 Evaluation Mechanism Number: AID-367-C-15-00001 DISCLAIMER This publication is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. iii TABLE OF CONTENTS ACRONYMS AND ABBREVIATIONS v ABSTRACT ix I. EXECUTIVE SUMMARY 1 II. EVALUATION PURPOSE 1 III. ACTIVITY BACKGROUND 2 IV. EVALUATION QUESTIONS AND METHODOLOGY 6 V. FINDINGS 11 VI. CONCLUSIONS 25 VII. RECOMMENDATIONS (TECHNICAL) 26 VIII. RECOMMENDATIONS (MANAGERIAL) 28 APPENDIX 1: EVALUATION STATEMENT OF WORK 29 APPENDIX II: DOCUMENTS REVIEWED 42 APPENDIX III: KEY INFORMANTS INTERVIEWED 43 APPENDIX IV: DATA COLLECTION INSTRUMENTS (KII GUIDES AND OBSERVATION CHECKLISTS) 46 APPENDIX V: SITE VISIT PICTURES 70 iv LIST OF FIGURES Figure 1: USAID/Nepal’s Objectives for Supply Chain Technical Assistance Under PSM ................................................... 2 Figure 2: Deployment of GHSC-PSM/Nepal Field Support Officers......................................................................................... 3 Figure 3: PSM-Nepal’s Strategic Vision 2018-2021...........................................................Error! Bookmark not defined. Figure 4: GHSC-PSM-Nepal’s Results Framework......................................................................................................................... 4 Figure 5: Training Provided by GHSC-PSM/Nepal .......................................................................................................................13 Figure 6: Stockout rate in SDPs in GHSC-PSM supported regions, TO3 - FP/PRH............................................................17 Figure 7: Composite stock out rate - by FP method (TO3) in GHSC-PSM supported regions ......................................18 Figure 8: LMIS Reporting Rate, Quarter 2, 2019..........................................................................................................................18 LIST OF TABLES Table 1: Evaluation Questions............................................................................................................................................................. 6 Table 2: Site Visits................................................................................................................................................................................... 9 Table 3: Indicators in GHSC-PSM/Nepal’s M&E Plan (Note: performance indicators are in normal text, context indicators are italicized, and the country specific indicator is in bold text) ..........................................................................11 v ACRONYMS AND ABBREVIATIONS AMC Annual Maintenance Costs CDCS Country Development Cooperation Strategy CMS Central Medical Store COP Chief of Party DEC Development Experience Clearinghouse DELIVER Delivering Effective Government for Competitiveness and Inclusive Growth DfID Department for International Development DHO District Health Office DLI(s) Disbursement-linked Indicators DO Development Objective DoHS Department of Health Services EDP(s) External Development Partners eLMIS Electronic Logistics Management Information System FEFO First Expired / First Out FP/RH Family Planning / Reproductive Health FSO Field Support Officer GF The Global Fund to Fight AIDS, Tuberculosis, and Malaria GH Bureau for Global Health GHSC-PSM Global Health Supply Chain – Procurement and Supply Management Activity GoN Government of Nepal GtG Government to Government H4L Health for Life HMIS Health Management Information System HRH Human Resources for Health IDIQ Indefinite Delivery, Indefinite Quantity IMS Inventory Management System IP Implementing Partner IR Intermediate Results KfW Kreditanstalt für Wiederaufbau KII Key Informant Interview LLG Local Level Government LMD Logistics Management Division LMIS Logistics Management Information System LMS Logistics Management Section LWG Logistics Working Group M&E Monitoring and Evaluation MD Management Division vi MNCH Maternal, Neonatal, and Child Health MoHP Ministry of Health and Population NHSSP Nepal Health Sector Support Programme O&M Operations and Management PHD Provincial Health Director PHLMC Provincial Health Logistics Management Center PHO Provincial Health Office PMP Performance Monitoring Plan PMS Provincial Medical Store PR Principal Recipient PSCM Procurement and Supply Chain Management QA Quality Assurance SC Supply Chain SDP Service Delivery Point SOP(s) Standard Operating Procedures SOW Statement of Work SSBH Strengthening Systems or Better Health STTA Short-term Technical Assistance TA Technical Assistance TO Task Order UNICEF United Nations Children’s Fund USG United States Government vii Team Members Tim A. Clary, Evaluation Team Lead Puja Bharati, Supply Chain Management Specialist Devi Prasai, Monitoring and Evaluation Specialist Sumana Devkota Field Researcher Nilam Bhandari, Field Researcher Bina Dhital, Field Researcher Suban Thapa, Field Researcher Jahirul Hussein, Field Researcher USAID/Nepal’s Monitoring, Evaluation, and Learning (MEL) Support Kshitiz Shrestha, Evaluation Specialist (Senior Level) Manorama Adhikari, Deputy Chief of Party Swadesh Gurung, Monitoring and Evaluation Specialist Milima Dangol, Monitoring and Evaluation Specialist viii Acknowledgments This evaluation could not have been possible without the contributions of dedicated USAID and implementing partner staff whose inputs were instrumental to the evaluation team’s efforts. The team especially wishes to express its gratitude to Office Director of the USAID/Nepal Health Office Carrie Rasmussen, Monitoring and Evaluation Advisor Sabita Tuladhar, Senior Health Advisor Nur Pant, Supply Chain Management Advisor Umesh Gupta, and the USAID/Nepal Health Office for their support and guidance throughout the evaluation. The evaluation team also wishes to acknowledge the support it received from the staff of the Global Health Supply Chain – Procurement and Supply Management/Nepal (GHSC-PSM/Nepal). Specifically, the team expresses its appreciation for the assistance provided by Policy and Government Relations Advisor Balkrishna Khakurel, who coordinated meetings and provided introductions to key stakeholders, followed-up on team inquiries, and provided additional requested documentation. The evaluation team is grateful to the staff of all organizations that it interviewed as part of this evaluation and the additional stakeholders who made themselves available to participate in focus group discussions and individual and group interviews. These staff and stakeholders generously provided their time to share valuable information upon which the findings, conclusions, and recommendations are based. ix ABSTRACT The Mid-term Performance Evaluation of the USAID Global Health Supply Chain – Procurement and Supply Management/Nepal (GHSC-PSM/Nepal) activity employed a mixed-methods approach to better understand whether GHSC-PSM/Nepal is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality technical assistance to Nepal’s Ministry of Health and Population in achieving its national supply chain objectives. The evaluation team held interviews with 64 key informants at the central (Kathmandu) level and five local-level sites, as well as completed two observational checklists. Based on the evidence collected, the evaluation team concludes that while some of the technical interventions implemented by GHSC-PSM/Nepal have provided selected benefits to local stakeholders, they have not fundamentally transformed the supply chain or moved the Nepalese supply chain far forward in its journey to self-reliance. Additionally, many of the interventions are unlikely to be sustained after the GHSC-PSM/Nepal activity ends in April 2021. The evaluation team, therefore, recommends during the remaining life of the activity that GHSC￾PSM/Nepal prioritizes three managerial and technical issues, namely 1) adopting a more strategic approach to address the underlying systemic issues within the Nepal supply chain; 2) ensuring that GHSC-PSM/Nepal has sufficient buy-in from its government counterparts for any additional technical interventions; and 3) building the evidence base for the interventions, primarily the Logistics Management Information System (eLMIS), which it has already piloted in two provinces. GHSC￾PSM/Nepal also needs to collaborate and work more closely with other development and implementing partners to ensure that the current and impending challenges that have resulted from federalism are adequately mitigated and to use that restructuring to address new opportunities as well. x 1 I. EXECUTIVE SUMMARY EVALUATION PURPOSE The purpose of the evaluation is to better understand whether the Global Health Supply Chain – Procurement and Supply Management activity in Nepal (GHSC-PSM/Nepal) is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality technical assistance to the Ministry of Health and Population (MoHP) in achieving its national supply chain objectives1. The findings from the evaluation will be used to strengthen current activities through the identification of course￾correcting actions to ensure that the activity will meet its objectives; develop recommendations for future activities; and provide recommendations to the Government of Nepal (GoN) and external development partners (EDPs) in Nepal with evidence of good supply chain management practices. The evaluation is predicated on the assumption that while the context for the federalization of health services in Nepal is changing rapidly and significantly, many lessons from GHSC-PSM/Nepal’s experience will remain relevant going forward, and that identifying these lessons and taking them into account has much potential to strengthen future programming in this area. ACTIVITY BACKGROUND GHSC-PSM/Nepal is a five-year, nearly $16 million activity that began in April 2016 and will conclude in April 2021. It is a buy-in into the centrally managed GHSC-PSM Indefinite Delivery Indefinite Quantity (IDIQ) contract and primarily funded through Task Orders 3 (family planning/reproductive health or FP/RH) and 4 (maternal, neonatal, and child health or MNCH). The prime partner of the GHSC-PSM IDIQ is Chemonics International, Inc. GHSC-PSM/Nepal’s overall goal is to ensure an uninterrupted supply of health commodities to prevent suffering, save lives, and create a brighter future for families in Nepal. This goal is supported by five intermediate results (strategic planning, in-country logistics, capacity-building, an enabling environment, and innovations and research) with twelve areas of technical focus2 (though, some areas are emphasized significantly more than others). GHSC-PSM/Nepal is, in one sense, working nationally; however, several of its activities are being piloted in selected provinces and/or with local-level governments (LLGs) before being rolled out nationally. GHSC-PSM/Nepal’s main in-country counterpart is the MoHP, particularly the Department of Health Services (DoHS) and its supporting organizational units. 1 Details are given in the Nepal Health Sector Strategy (2015-2020) and the Nepal Health Sector Strategy Implementation Plan (2016-2021). 2 Strategy and Planning Technical Assistance, Forecasting and Supply Planning Technical Assistance, Procurement Technical Assistance, Quality Assurance TA, Warehousing and Inventory Management Technical Assistance, Transportation and Distribution Technical Assistance, MIS Technical Assistance, Knowledge Management and Communication, Monitoring and Evaluation, Human Resources Capacity Development Technical Assistance, Global Collaboration, and Other Support. 2 Exhibit 1: Key Evaluation Questions FINDINGS Question 1: To what extent and how has the GHSC-PSM activity contributed to supply chain performance improvements? ● In GHSC-PSM/Nepal’s February 2019 Monitoring and Evaluation (M&E) Plan, 12 of the 13 indicators are either directly taken from PSM’s global M&E plan or slightly modified. Only a few targets have been set for the performance indicators, and many are at the national level. ● Reviewing relevant documents show that the main technical contributions have been related to the electronic Logistics Management Information System (eLMIS), medical store/warehousing improvements, and capacity building for quantification/forecasting. ● Per key informant interviews (KIIs), GHSC-PSM/Nepal is widely regarded as an eLMIS activity, with some key informants also mentioning warehouse improvements and supply chain (SC) capacity building as technical contributions. ● GHSC-PSM/Nepal’s technical contributions primarily have been limited to family planning/reproductive health (FP/RH) and MNCH at the central level mainly due to the fragmented supply chain. At lower administrative levels, contributions have been more cross￾cutting due to field support officer (FSO) presence. Most technical contributions have occurred at the central level. ● The main challenges to GHSC-PSM/Nepal’s implementation include: a shift to federalism; a lack of GHSC-PSM/Nepal leadership and corresponding strategic vision; GoN staff turnover; a need for GHSC-PSM/Nepal to be more inclusive and collaborative in the development of its TA package; and a lack understanding of the Nepalese context in the implementation of some of its activities. ● Per several stakeholders, federalism and SC efforts by other development partners (DPs)/technical assistance (TA) providers could have provided opportunities for GHSC￾PSM/Nepal to both adjust its approach and ensure that its efforts would have had more lasting and effective outcomes. ● While many stakeholders noted and appreciated some of GHSC-PSM/Nepal’s various activities, few could cite a best practice or success story. Question 2: Is the activity on the right track for achieving the desired results by the end of the activity? ● Per reviewed documents, only a limited number of targets were set for reducing stock-outs, increasing service delivery point (SDP) reporting rates to the LMIS, and warehouses instituting first expired/first out (FEFO); thus, the “desired” results are unclear. ● Based on the document review, KIIs, and field observations, there has been uneven and slow progress in increasing SDP reporting rates to the LMIS. Based on interviews with FSOs at the • To what extent and how has the GHSC-PSM project contributed to supply chain performance improvements? • Is the project on the right track for achieving the desired results by the end of the project? • How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? • What has been the experience of participating stakeholders in the transfer of skills in the supported technical areas? 3 Question 1: To what extent and how has the GHSC-PSM activity contributed to supply chain performance improvements? visited field sites, the eLMIS reporting rates have increased in recent months to at least 40 percent. ● Per field observations, the following three issues have been insufficiently addressed for FEFO to be consistent or sustainable: 1) insufficient storage space; 2) inadequate management of the limited space; and 3) uneven implementation of the basic guidelines of warehousing best practices and the monitoring of those practices. Question 3: How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? ● Per most interviewed stakeholders, because of GHSC-PSM/Nepal’s limited focus on only a few technical areas of supply chain strengthening, and with a limited geographical focus, its contribution to developing a self-reliant supply chain in Nepal is minimal. ● Key stakeholders within the MoHP/DoHS technical/operational units reported they have not been sufficiently involved in GHSC-PSM/Nepal’s technical decisions and have instead been bypassed for approvals from senior leadership. ● The level of GoN “ownership” of GHSC-PSM/Nepal’s activities is mixed. At the central level, there is a strong desire by senior management to see eLMIS scaled-up, but there is a lack of ownership by technical staff. At the provincial and LLG levels, there is limited ownership of the activity. ● Almost all stakeholders noted that most of the activities would not be sustained after GHSC￾PSM/Nepal’s closure without continued assistance. Some stakeholders noted that, most likely, they will regress to previous practices. Question 4: What has been the experience of participating stakeholders in the transfer of skills in the supported technical areas? ● Per stakeholder interviews (done both at the central and implementation level), the eLMIS has had limited effect on decision-making in Provinces 5 and 6, mainly due to the slowness of the system/software and report outputs and limited access. ● The main challenges stated by stakeholders in implementing the eLMIS system include the software not being user-friendly, slow internet connectivity (and lack of an offline version), inability to modify incorrect data on-site, insufficient and/or inappropriate data fields, the off-site trouble-shooting system, and insufficient buy-in from multiple GoN levels. ● The challenges for eLMIS scale-up and sustainability include technical (internet connectivity, technical assistance for troubleshooting), financial (unclear future operational and technical assistance costs), and, possibly, legal/contractual (Nepal Public Procurement Act). ● At the provincial level, stakeholders are utilizing the procurement and forecasting guidelines and an annual procurement plan (and in some sites, corresponding training has been provided). ● Based on field observations, some good warehousing practices have been institutionalized; however, based on KIIs, these practices will not be sustained given basic infrastructure issues within the medical stores and warehouses. ● Per KIIs, there is an informal agreement that the GoN will take greater ownership of the supply chain pharmacists via increased domestic funding in future years. ● Based on field observations and interviews, while the FSOs have provided some technical assistance, their current job function is to act as seconded staff primarily. ● Per interviewed stakeholders, the Logistics Working Group (LWG) has not met in more than one year and, therefore, participants were unable to provide specific answers regarding its 4 Question 1: To what extent and how has the GHSC-PSM activity contributed to supply chain performance improvements? technical contributions. Other forums were cited as contributing to improving a steady and continuous flow of commodities. CONCLUSIONS • Most of GHSC-PSM/Nepal’s interventions have provided limited technical fixes to address underlying systemic issues in supply chain management. Furthermore, GHSC-PSM/Nepal has not obtained enough buy-in for these technical fixes, thereby limiting their prospects for sustainability. • Some of the challenges could not be foreseen (federalism) and some were outside of GHSC￾PSM/Nepal’s manageable interest (e.g., GoN staff turnover); however, there were missed opportunities and a lack of a clear strategic vision and leadership for addressing fundamental supply chain issues. • This lack of a greater strategic approach is a shared responsibility among all relevant partners. • GHSC-PSM/Nepal’s efforts are unlikely to have any transformative or sustainable effect on the “supply chain” in Nepal. RECOMMENDATIONS (TECHNICAL AND MANAGERIAL) 1) If possible, review and revise the GHSC-PSM/Nepal M&E Plan to make it more Nepal specific. 2) Use the remaining life of the activity for examining the eLMIS results in Provinces 5 and 6 to build the evidence base for the effectiveness, acceptability, and sustainability of GHSC-PSM/Nepal’s eLMIS approach and specific intervention. 3) Possibly roll out eLMIS to one or two more provinces with the remaining provinces to be part of any follow-on activity, if appropriate. 4) If possible, untie or delay the eLMIS Disbursement Linked Indicators (DLIs). 5) Continue and finish the Central Medical Store (CMS) work, but none should be labeled “model” medical stores and increase the number of monitoring visits by GHSC-PSM/Nepal staff. 6) Work in collaboration with other DPs and TA providers on policies and procedures for developing a pooled procurement mechanism and potentially shared SC systems (logistics, quality assurance or QA, disposal methods, etc.) which can be utilized by LLGs. 7) Work with the GoN at all levels to ensure that supply chain roles and responsibilities are defined and understood at all levels for all stakeholders (this can be done, for example, via policy memos and training sessions). 8) Partner with other DPs and implementing partners (IPs) in building the capacity of lower administrative levels to manage their new (defined) procurement and supply chain management (PSCM) roles and responsibilities. 9) Work with other TA providers and DPs to advocate for an integrated supply chain at a minimum across health elements (can be done via meetings/forums, technical briefs, and/or position papers). 10) For any future TA or other activities, GHSC-PSM/Nepal needs to be more inclusive (e.g., require a sign-off sheet from technical/operational units and staff) in developing materials, planning training sessions, and developing its overall strategic approach. 11) To ensure that it is “fit for purpose” in its final months of implementation in meeting any revised objectives and corresponding targets: 5 a. Review and, if needed, adjust GHSC-PSM/Nepal’s organizational structure (both in-country and its relationship to GHSC-PSM/HQ). b. Ensure that short-term technical assistance (STTA) is from repeat providers who understand the Nepalese context. c. Examine the Kathmandu versus field staff proportions. d. Review GHSC-PSM/Nepal job descriptions to ensure they match any new strategic undertakings. 12) Provide additional training/re-training to FSOs to reorient them from a staff role to a TA provider role to build the capacity of local counterparts. 13) Continue hosting/supporting various meetings (quantification, forecasting, pipeline, LWG), but, re-orient them to include more strategic discussions. 1 II. EVALUATION PURPOSE In January 2019, USAID/Nepal released a statement of work (SOW – Appendix 1) for the mid-term performance evaluation of the GHSC-PSM/Nepal activity. Per the SOW, the purpose of the evaluation is to better understand whether GHSC-PSM/Nepal is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality technical assistance to the MoHP in achieving the national supply chain objectives3. The findings from the evaluation will be used to strengthen current activities through the identification of course-correcting actions to ensure the activity will meet its objectives; develop recommendations for future activities; and provide recommendations to the GoN and EDPs in Nepal with evidence of good supply chain management practices. The evaluation is predicated on the assumption that while the context for the federalization of health services in Nepal is changing rapidly and significantly, many lessons from GHSC-PSM/Nepal’s experience will remain relevant going forward, and that identifying these lessons and taking them into account has much potential to strengthen future programming in this area. The primary audiences for the evaluation report are the USAID/Nepal Mission and IP staff, as well as other relevant stakeholders (GoN counterparts, external development partners (EDPs), and other technical implementers). It is intended that the report will allow USAID/Nepal to ground truth and discuss the findings and conclusions in detail, utilize the recommendations for the remaining life of the activity, and inform future supply chain investments. The final report also will be uploaded to the Development Experience Clearinghouse (DEC) for public use. The evaluation report may be disseminated to a wider audience through other platforms, such as workshops or publication of separate documents if requested by USAID/Nepal. 3 Details are given in the Nepal Health Sector Strategy (2015-2020) and the Nepal Health Sector Strategy Implementation Plan (2016-2021). 2 III. ACTIVITY BACKGROUND USAID’s Bureau for Global Health (GH) manages an array of central commodity procurement and supply chain technical assistance mechanisms to support United States Government (USG)-field programs and USAID’s global technical leadership as part of the Global Health Supply Chain (GHSC) Activity. These mechanisms are managed across multiple GH technical offices that support HIV/AIDS, malaria, FP/RH, MNCH, and other public health programs. The Procurement and Supply Management (GHSC-PSM) Activity is a single award IDIQ contract for commodity procurement and supply chain systems strengthening technical assistance. The prime implementing partner is Chemonics International, Inc. Through a buy-in to the central GHSC-PSM Activity, USAID/Nepal is supporting the implementation of the PSM activity under Task Orders 3 and 4 (TO 3 and 4), which are dedicated to family planning/reproductive health (FP/RH) and maternal, neonatal, and child health (MNCH) respectively. GHSC-PSM/Nepal is a nearly $16 million, five-year activity (April 2016-April 2021) to strengthen the supply chain, including procurement, in Nepal. Its three objectives are presented in Figure 1, its current Strategic Vision (2018-2021) is presented in Figure 3, and its Results Framework is presented in Figure 4. Figure 1: USAID/Nepal’s Objectives for Supply Chain Technical Assistance Under PSM GHSC-PSM/Nepal builds upon previous procurement and supply chain technical assistance provided under the Delivering Effective Government for Competitiveness and Inclusive Growth (DELIVER) Activity and Health for Life (H4L) logistics contract. When DELIVER ended its activities in 2013, it was believed that the MoHP would be able to lead procurement and supply chain management activities in Nepal with minimal technical assistance in the future. However, this assumption was premature as the MoHP could not sustain the supply chain. This coupled with a series of unexpected events (e.g., the 2015 earthquake), the re-organization of the MoHP and the resultant downgrading of the Logistics Management Division (LMD) to the Logistics Management Section (LMS) under the Management Division (MD), and the transition to the federal system of governance and the resulting uncertainty created concerning responsibilities for the supply chain, has resulted in significant challenges for the MoHP. In response, USAID/Nepal and other EDPs, such as the United Kingdom’s Department for International Development (DfID), are providing procurement and supply chain technical assistance in support of the Nepal Health Sector Strategy Implementation Plan: 2016-2021, which has as its first outcome: “Rebuilt and strengthened health systems: Infrastructure, HRH management, procurement and supply chain management.” Although GHSC-PSM/Nepal is, in one sense, working nationally, several of its activities (e.g., eLMIS, FSOs) are being piloted in selected provinces and/or with LLGs before being rolled out nationally, as well as, its placement of FSOs as illustrated in Figure 2. • Global commodity procurement and logistics. • Systems strengthening technical assistance. • Global collaboration to improve the long-term availability of health commodities. 3 Figure 2: Deployment of GHSC-PSM/Nepal Field Support Officers Following the December 2017 elections, Nepal began its decentralization process as specified in the Constitution of 2015. Seven newly formed provinces were given autonomy to govern various functions previously under the responsibility of the central government, and 753 LLGs (municipalities) also were given greater powers of self-governance. Accordingly, GHSC-PSM/Nepal had to flexibly rethink its strategic vision to ensure that each level of government is accountable for improving the health of Nepalis. Therefore, GHSC-PSM/Nepal currently has a focus on strengthening the health supply chain system at provincial and LLG levels in coordination with and providing support to the MD of the DoHS at the central level. Figure 3 below provides a graphic representation of GHSC-PSM/Nepal’s strategic vision, outcomes, and objectives for the next three years and Figure 4 shows GHSC-PSM/Nepal’s current Results Framework as embedded within USAID/Nepal’s Country Development Cooperation Strategy (CDCS), Development Objective 3 (DO 3: Increased Human Capital), and Intermediate Result (IR) 3.2: A healthier and well-nourished population. Figure 3: PSM-Nepal’s Strategic Vision 2018-2021 DATA Availability & Use Forecasting and Supply Planning Procurement Quality Assurance Budget/Finance Warehousing & Inventory Management Transportation & Distribution Good Governance Improved Supply Chain performance End-to-end Visibility Financial Sustainability Capacity Building Federal and local government ensures uninterrupted supply of quality essential health commodities in health facilities 4 Figure 4: GHSC-PSM-Nepal’s Results Framework GHSC-PSM/Nepal Goal: To ensure uninterrupted supplies of health commodities to prevent suffering, save lives, and create a brighter future for families in Nepal Intermediate Result 1: Strategic Planning Improved strategic planning and implementation related to supply chain management and commodity security Intermediate Result 2: In-County Logistics Improved in-country logistics including effective and efficient delivery of health commodities to service sites Intermediate Result 3: Capacity Building Increased capacity building efforts by implementing strategies to transfer of skills, knowledge, and technology for improved and sustained performance Intermediate Result 4: Enabling Environment Strengthened enabling environments to improve supply chain performance Intermediate Result 5: Innovations and Research Innovations and research conducted, shared and implemented Indicator Reference Themes: In-country performance and sustainability Indicator Reference Themes: In-country sustainability; cross-cutting Intermediate Result 1: Strategic Planning Output 1.1 LMD’s Performance capacity strengthened Output 1.2 Demand forecasting methodologies standardized Output 1.3 Process for forecasting, quantification and pipeline strengthened and institutionalized Output 1.4 Sourcing strategy SOPs developed and advocated for implementation uptake Output 1.5 Groundwork laid for annual procurement plan Intermediate Result 2: In-Country Logistics Output 2.1 Warehouse management improved for commodity availability Output 2.2 Improved efficiency of transportation and distribution Intermediate Result 3: Capacity Building Output 3.1 Strengthen local vendors capacity to supply quality products and right products on time Output 3.2 Forecasting and supply planning capacity developed at different supply chain levels Output 3.3 Local level supply chain staffs’ knowledge one￾bidding enhanced Output 3.4 Supply chain staffs’ competency to implement eLMIS enhanced Intermediate Result 4: Enabling Environment Output 4.1 Visibility of supply chain data increased Output 4.2 Improved use of data to inform supply chain management decisions Output 4.3 Improved coordination and support between relevant stakeholders Output 4.4 Supply chain pharmacist’s role strengthened to support warehouse Intermediate Result 5: innovations and Research Output 5.1 eLMIS developed and rolled out Output 5.2 Advocate for quality assurance mechanism for TO4 and TO4 products 5 Under its original SOW, GHSC-PSM/Nepal was to support comprehensive system strengthening to achieve the goal of transitioning from assistance from external support to sustainable supply chain systems. GHSC-PSM/Nepal was expected to provide innovative technical assistance in all areas of the supply chain, including management and governance, to address root issues of underperformance. The key priorities for USAID’s supply chain technical assistance in Nepal are: 1) support foundational supply chain activities; 2) support development of a cadre of health supply chain professionals; 3) ensure the availability of reliable data for decision-making; and 4) support the transition of key supply chain management functions from central to provincial and municipal level governments. To ensure progress in these four key priority areas, GHSC-PSM/Nepal supports interventions in twelve technical areas (although, it emphasizes some technical areas more than others): 1. Strategy and Planning Technical Assistance. 2. Forecasting and Supply Planning Technical Assistance. 3. Procurement Technical Assistance. 4. Quality Assurance TA. 5. Warehousing and Inventory Management Technical Assistance. 6. Transportation and Distribution Technical Assistance. 7. Management Information System (MIS) Technical Assistance. 8. Knowledge Management and Communication. 9. Monitoring and Evaluation. 10. Human Resources Capacity Development Technical Assistance. 11. Global Collaboration. 12. Other Support. 6 IV. EVALUATION QUESTIONS AND METHODOLOGY The performance evaluation questions appear in Table 1 below. In developing and answering these questions, the evaluation team focused on placing them within USAID’s overall framework, “The Journey to Self-Reliance. Table 1: Evaluation Questions Evaluation Area(s) Evaluation Questions Context and Design 1. To what extent and how has the GHSC-PSM activity contributed to supply chain performance improvements? A. Per documented performance figures, what have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? B. Per relevant stakeholders, what have been the main contributions of GHSC￾PSM/Nepal in the areas of supply chain technical assistance? Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? C. At which geographical level (national, provincial, LLG) have most of the improvements been demonstrated? D. What have been the main challenges (technical, administrative/managerial, contextual) for GHSC-PSM/Nepal in ensuring the success of its activities? How have these challenges been mitigated? E. What factors (contextual or otherwise) have led to opportunities for activity implementation? F. What have been the best practices and/or success stories related to activity implementation? Implementation 2. Is the activity on the right track for achieving the desired results by the end of the activity, including: A. Reducing stock-outs at service delivery points? B. Increasing SDP reporting rates to the LMIS? C. Increasing the percentage of warehouses with FP and MNCH commodities managed by a FEFO system? D. Other areas of significant technical assistance focus? Sustainability 3. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance (e.g., mobilizing adequate resources effectively and implementing transparent supply chain functions effectively)? A. To what extent has GHSC-PSM/Nepal’s technical assistance contributed to this trajectory/journey? B. How has the activity’s support improved supply chain management performance and enhanced the capacity of the supply chain cadres within the LMS? C. Has the activity secured ownership of activity activities from the MoHP, provincial, and municipal leadership? D. Is GHSC-PSM/Nepal on track to achieve a greater capacity to sustain performance in the absence of external assistance once its technical assistance is finished? Empowerment 4. What has been the experience of participating stakeholders in the transfer of skills in the supported technical areas? 7 Evaluation Area(s) Evaluation Questions A. Has the eLMIS data been useful in the overall decision-making around supply chain management? What are the challenges in the overall implementation, operation, and management of the eLMIS? To what extent have stakeholders at all levels of the federal structure taken ownership of the eLMIS? B. What will be the greatest challenges to further eLMIS scale-up? C. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? D. Have good warehousing practices been institutionalized and, if not, what additional steps need to be undertaken to ensure their sustainability? E. Has the MoHP taken ownership (i.e., budgeting) for the supply chain pharmacists funded under Red Book and, if not, what additional steps need to be undertaken to ensure this? F. What have been the main technical contributions of GHSC-PSM/Nepal’s FSOs? G. What have been the main contributions of the LWG to ensuring a sustainable supply chain? 3.1 METHODOLOGY The evaluation team primarily used a qualitative methodology done via KIIs, either with individuals or small groups. The team also conducted quantitative data collection via site visits (health facilities, provincial health offices, central medical stores and warehouses, and eLMIS implementation sites) utilizing both pre-existing data and results from facility observational checklists for medical store warehouses and eLMIS sites. See Appendix 4 for the tools utilized. Specifically, the evaluation team collected data through the following methods: ● A review of program documents. ● Semi-structured key informant interviews either with individuals or in small groups. ● Observational checklists (eLMIS sites, medical storage warehouses). As identified in Appendix 2 of this report, the evaluation team reviewed both before and during the evaluation several key government and activity-related documents. Further, the evaluation team used semi-structured questionnaires to collect qualitative data during KIIs and group interviews and checklists to collect both quantitative and qualitative data from implementation sites. The evaluation team used the questionnaires flexibly during interviews to tailor questions to the specific knowledge and experience of the respondents and allow stakeholders to address issues which were of importance to them. Finally, observational checklists were conducted at provincial medical stores, district medical stores, central medical store, and eLMIS implementation sites to determine the usefulness, effectiveness, and sustainability of the interventions. This included, for example, whether good warehousing practices were still in place, whether there has been a full conversion to the eLMIS (versus the paper-based system), time saved through eLMIS, whether there have been increases in SDP reporting rates, and whether accurate forecasts are being developed. 3.1.1 SAMPLING STRATEGY (KEY INFORMANT INTERVIEWS) 8 The team included the following stakeholders in the evaluation (for the list of actual interviewees, please see Appendix 3). 1) HQ Staff (USAID/W and GHSC-PSM): The evaluation team conducted remote interviews with appropriate USAID/W and GHSC-PSM staff at the headquarters level, which included those involved in overall management, technical assistance and backstopping, and reporting. 2) GoN/MoHP Officials: The team also held interviews and meetings with appropriate government officials at the national, provincial, and municipality levels that have interacted with the GHSC￾PSM/Nepal’s program activities (nine interviews were held at the national/central level, while an additional 23 were held at the provincial and lower administrative levels). The interviews at the national level, for example, included the director, MD section chief, procurement & LMS section chief, eLMIS-section chief, and chief of the CMS. At the provincial level, KIIs were conducted, for example, with the following: the health officer in charge, provincial health director, chief of the provincial medical store, pharmacist, and staff responsible for eLMIS and forecasting. At the municipality level, KIIs were conducted, for example, with the CEO and health coordinator of the respective municipalities, chiefs of district medical stores, and nearest health facility in-charge. 3) USAID/Nepal Staff: Six interviews were held with USAID/Nepal staff, including, the Health Office director, MNCH team lead, and supply chain management advisor. 4) IPs: Interviews and meetings were held with appropriate implementing partners at the national and local levels, including, the acting country director/chief of party, team leads for health systems strengthening, management information systems, policy and government relations, and operations along with their relevant staff (a total of five local GHSC-PSM staff). To receive local-level input, the evaluation team also interviewed the six current FSOs and met with relevant staff of other IPs, such as the Nepal CRS Company and individuals implementing the Nepal Health Sector Support Programme (NHSSP), for a total of four additional IP staff. 5) Other EDPs: The evaluation team interviewed the relevant staff of DfID, the United Nations Populations Fund (UNFPA), Kreditanstalt für Wiederaufbau (KfW), the United Nations Children’s Fund (UNICEF), the Global Fund Principal Recipient (GF PR), and the German Corporation for International Cooperation (aka GIZ). 6) LWG Members: The evaluation team met with a sample of LWG members with high levels of attendance and participation at the LWG meetings. 7) Supply Chain Pharmacists: The evaluation team interviewed a subset of the 15 supply chain pharmacists. 3.1.2 SAMPLING STRATEGY (SITE VISITS) While as previously mentioned, GHSC-PSM/Nepal’s activities are considered to be implemented nationwide, to date, many of the interventions (capacity building, eLMIS, etc.) have been in Provinces 5 and 6. To see the full effects of GHSC-PSM/Nepal’s technical assistance, the evaluation team chose areas which have received significant support versus areas which have received less support. The details for the sites, including the criteria utilized in determining them, appear in Table 2 below. 9 Table 2: Site Visits Province Site Rationale for Selection 3 Kathmandu Tools pilot site ● Logistically feasible. ● Presence of key stakeholders at the national level. ● Has received technical assistance for areas of interest in the piloting of the tools/observation checklists. Additional Site Visits Province Site Rationale for Selection 5 Banke ● Significant technical assistance and training provided at this site in multiple areas. ● Presence of FSO in Province 5 placed at Provincial Medical Store (PMS)/Nepalgunj for the commodity supply for Province 6. ● Can include municipality and health facility visits. 5 Bardia ● Significant technical assistance and training provided at this site in multiple areas. ● Can include municipality and health facility visits. 5 Rupendehi/Butwal ● Significant technical assistance and training provided at this site in multiple areas. ● Presence of an FSO in Province 5 placed at Butwal. ● Can include municipality and health facility visits. 4 Pokhara ● Presence of sub-regional warehouse. ● (More limited) technical assistance given in earlier years of activity implementation. ● Can include municipality and health facility visits. 2 Bara (Pathlaiya) ● Significant technical assistance provided to the warehouse/medical store to become a “model” facility. ● Some involvement in eLMIS activities. ● Presence of an FSO in Province 2. 3.2 LIMITATIONS As expected, the evaluation team encountered several biases and other data limitations that were mitigated through methodological or analytical means. ● Qualitative approach: The primary approach for this evaluation was qualitative data collection via KIIs and group interviews and, to a lesser degree, qualitative and quantitative questions within the proposed checklists. The opinions of stakeholders are, by their nature, subjective. Team members may not have accurately recorded or correctly transcribed important data for a variety of reasons (such as fatigue and difference in understanding), and those data may not be part of the findings, conclusions, or recommendations.4 4 This is mainly a generic statement as the vast majority of KIIs were done with 2 or more team members present who then subsequently verified their notes with each other. For the small minority of interviews which were done by individual team members, this may have been an issue. 10 ● Selection bias: Key stakeholders may have been inadvertently excluded, and those persons who did participate in a KII or group interview may have introduced self-selection bias (either beneficial or detrimental) into the results limiting the ability to draw definitive conclusions. ● Limited sample of KIIs: The team anticipated that data collection with direct and indirect beneficiaries might be difficult to schedule because of existing demands on their time. To mitigate this concern, the team remained flexible to the best extent possible to accommodate as many key informants and group interviewees as possible. ● Response bias: Response bias is a common problem for activity evaluations. For instance, respondents may have given the interviewer positive remarks, such as in capacity building, about an activity because they would like to participate in additional training in the future. ● Recall and recency biases: Some of GHSC-PSM/Nepal’s interventions have either been completed or were conducted at the beginning of the activity, which, of course, potentially will introduce a type of recall and/or recency bias in the respondents as responses may have varied significantly both in terms of accuracy and opinion if the evaluation had been done either earlier or later during implementation. To combat biases, the evaluation team used multiple sources of data to triangulate answers on evaluation issues. By combining data from documents, observation checklists, and KIIs from multiple sources, the evaluation team believed that any one piece of biased data would not skew the analysis. Another approach that pertains specifically to KIIs was the inclusion of key informants from organizations that do not directly benefit from the assessed activity, and the use of questions about specific examples of knowledge use. Additionally, team members used direct observation based on knowledge of the evaluation environment to assist in identifying potential bias in responses. 11 V. FINDINGS Evaluation Question 1: What have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Have those contributions been only limited to the areas of RH/FP and MNCH, and if not, what other health areas have benefited from the technical assistance? 4.1.1 Contributions of GHSC-PSM/Nepal to the GoN’s Supply Chain Priorities GHSC-PSM/Nepal contributes to several of the main outcomes of the Nepal Health Sector Strategy 2016-2020 (1c1: Improved procurement and 1c2: Improved supply chain management system)5, Policy 6.4.5 (eHealth), and 6.10 (production of medicine and supply chain management) of the National Health Policy 2019.6 It also contributes toward Sustainable Development Goal 3 (Ensure healthy lives and promote well-being) 7 of Nepal by increasing access to quality medicine and health commodities. However, the indicators chosen to measure those contributions which appear within the February 2019 M&E Plan of GHSC-PSM/Nepal have mainly been directly taken from the GHSC-PSM global M&E Plan or slightly modified. Of the 13 indicators, only one output indicator, "Number of monitoring visits in which GoN participated," is specific to Nepal. Additionally, only two of the indicators are routinely collected and analyzed (stock-out rate of the SDPs and SDP LMIS reporting status), with some of the indicators being monitored only on a yearly basis (e.g., annual forecasting errors), and with a number of indicators (e.g. ,average rating of in-country data confidence, stock status observations, and FEFO) that have yet to be monitored. The overall planned analysis of the generated quantitative data, per the M&E Plan, appears to be limited. Thus, it remains unclear whether the M&E Plan and its corresponding indicators and process have adequately captured GHSC-PSM/Nepal’s supply chain efforts. Table 3: Indicators in GHSC-PSM/Nepal’s M&E Plan (Note: performance indicators are in normal text, context indicators are italicized, and the country-specific indicator is in bold text) Performance Indicators Sustainability Indicators Cross-cutting Indicators 1. Stock-out rate at SDPs. 2. Percentage of stock status observations in storage sites, where commodities are stocked according to plan, by level in the supply system. 3. SDP reporting rate to the LMIS. 4. Average rating of in-country data confidence at the central and provincial medical stores. 8. Percentage of the total budgeted on procurement of commodities for public sector services by government, USG, The Global Fund, and other sources. 9. Percentage of initially GHSC-PSM-supported supply chain functions carried out by national authorities that are done 11. Number of new innovations that were developed, implemented, or introduced and related to the health commodity market of supply chain best practices. 12. Number of people trained in health system strengthening. 13. Number of monitoring visits in which the GoN participates. 5 Ministry of Health and Population (2015). Nepal Health Sector Strategy 2015-2020. Kathmandu. 6 Ministry of Health and Population (2019). National Health Policy 2076 BS. Kathmandu. 7 National Planning Commission (2017). Nepal’s Sustainable Development Goals Status and Roadmap: 2016-2030 12 Performance Indicators Sustainability Indicators Cross-cutting Indicators 5. Absolute percent consumption forecast error, with mean absolute percentage error (MAPE) and bias variants. 6. Conducted annual forecast (yes/no). 7. Conducted quarterly supply plan update (yes/no). without external technical assistance. 10. Functional logistics coordination mechanism in place (yes/no). 4.1.2 Main Technical Contributions of GHSC-PSM/Nepal 1. Based on the document review and feedback from KIIs, it appears that GHSC-PSM/Nepal’s major contribution is the eLMIS; indeed, several KIIs regard GHSC-PSM/Nepal as an eLMIS activity. However, it should be noted that eLMIS is only one component of supply chain strengthening, which itself is a sub-component of health systems strengthening. As a result, many stakeholders commented that the activity had delivered less toward the other IRs within its results framework and, thus, toward the strengthening of supply chain management in Nepal. The eLMIS has been designed, planned, and piloted in the 57 sites, including six CMSs, two provincial medical stores (PMSs), 22 district stores within Provinces 5 and 6, and four LLGs and their corresponding 23 health facilities. Most KIIs agreed that the eLMIS has improved data visibility to some extent and provides a real-time stock level of commodities, as well as generates requisition, usable receipts, and handover forms. However, during the field visits, it was observed that the paper-based system is still being used in some sites due to the limited skills of the operators and a generally low level of confidence in the data generated by the eLMIS. In fact, many sites continue to operate two systems; namely, either the offline IMS/Excel or paper-based system and eLMIS due to the belief that the data of the eLMIS is less reliable and valid. Finally, the evaluation team also observed that wherever GHSC-PSM/Nepal piloted eLMIS, access to and use of the data is limited to an insufficient number of users. For example, the Provincial Health Directorate Office has no access to eLMIS data in Province 6. 2. According to stakeholders, the second major contribution of GHSC-PSM/Nepal is in forecasting and supply planning of FP/RH, essential medicines, vaccines, etc., at the central level and in Provinces 1, 5, and 7. As a result of capacity building, Province 5 has done its forecasting and has developed an annual procurement plan to be used for budgeting in the upcoming fiscal year, which provides it with a solid basis for procurement. GHSC-PSM/Nepal also has developed guidelines on procurement and supply chain management for LLGs and a strategic sourcing and health commodities catalog in collaboration with NHSSP and shared these documents with the provinces and LLGs. Several stakeholders also noted the usefulness of the quarterly national pipeline report. 3. A few respondents mentioned that warehouse improvements and capacity building have been additional technical contributions to strengthening the supply chain in Nepal. For example, the Provincial Health Logistics Management Centers (PHLMC, formally known as Regional Medical Stores) were reorganized with the support of GHSC-PSM/Nepal. Pallets, steel racks, steel cupboards, weighing machines, fire extinguishers, trolleys, and additional supplies were provided to improve warehouse management. However, only a few good warehouse management practices 13 were in place (supplies kept on pallets, stock levels were visible, boxes were stacked at a usable height, and a passage was kept clear for trolleys) at the time of the field visits. For example, both ventilation and lighting remain inadequate, and temperatures at all visited medical stores and warehouses exceeded (usually between 38 to 50 degrees centigrade) the appropriate standard. Further, some warehouse practices, such as keeping bin cards up to date, ensuring that supplies were stacked in an upright position, storing supplies batch-wise, and maintaining a clean environment, were unevenly implemented. 4. GHSC-PSM/Nepal has contributed to strengthening the capacity of MoHP staff; a total of 127 staff (13 from SDPs, 100 from District Health Offices or DHOs, and 14 from Provincial Health Directorate) were trained on basic health logistics in 2017, and more than 230 individuals were trained on procurement (182 from DHOs, 35 from Provincial Health Office and 13 from SDPs). Similarly, a total of 232 LLGs were trained on reporting modules of eLMIS in 2018/2019. See Figure 5 below for additional information. 5. In addition, training materials were developed on eLMIS; however, per stakeholder feedback, the quality of some of the training remains less than satisfactory. Additionally, per stakeholders, there has been no follow-up to the training. Most stakeholders mentioned that the basic health logistics training was adequate but that limiting the eLMIS training to one day as part of the HMIS rollout training without providing passwords was, per KIIs, not a good use of resources. Few eLMIS operators stated that the given training was adequate to operate the software and those without passwords quickly forgot the acquired skills. Figure 5: Training Provided by GHSC-PSM/Nepal 2 14 100 13 0 20 40 60 80 100 120 CENTRAL HEALTH OFFICIALS REGIONAL HEALTH OFFICIALS DISTRICT HEALTH OFFICIALS SERVICE DELIVERY POINT OFFICIALS Basic Health Logistics May-July 2017 35 182 13 REGIONAL HEALTH OFFICIALS DISTRICT HEALTH OFFICIALS SERVICE DELIVERY POINT OFFICIALS Training on Procurement in May-July 2017 14 Source: GHSC-PSM, 2019. 6. Per stakeholder responses, field observations, and the document review, the technical contributions of GHSC-PSM/Nepal have largely been limited to FP/RH and MNCH (except for some training sessions, quantification/forecasting, and procurement planning meetings) at the central level, mainly due to the verticalization of the supply chain in Nepal. The physical facilities and supply planning practices at the central level do not easily lend themselves to the integration of supply chain management as most are both limited in size and managed by separate organizational units. However, based on-site visits to provincial and district facilities, GHSC-PSM/Nepal’s contributions have been more integrated across health elements due to integrated supply chain management practices at these lower administrative levels. Within provinces and districts, supply chain activities were more integrated except for HIV-related commodities. 7. Per the KIIs and the document review, most of GHSC-PSM/Nepal’s technical contributions have occurred at the central level due to the centralized approach of planning and implementation of eLMIS, quantification and forecasting, and the capacity building which has been provided. However, given the implementation of federalism, many stakeholders questioned this approach. For example, according to stakeholders in Province 6 (Karnali), there is a great need for technical assistance on procurement and supply chain management, but, thus far, the province has received limited technical support. For example, during a site visit there, no quantitative forecasting of medical commodities was observed. 4.1.3 Main External and Internal Challenges for GHSC-PSM/Nepal External Challenges 1. Federalism: According to the constitution of Nepal, basic healthcare is the sole responsibility of the LLGs; therefore, the procurement function shifted from the district to the municipality level. As a result, the procurement and supply chain fragmented to 753 units or the corresponding number of LLGs. Moreover, the capacity of the LLGs in procurement and supply chain management is limited, and virtually no quality control mechanism exists at the LLG level. Resources directly flow to the municipalities, but their capacity on procurement and supply chain appears to be very limited based on field observations and stakeholder feedback. These issues are compounded by the allocation of those who have the training and greater experience in supply chain management being moved to the municipalities and SDP levels and those who have less training allocated to PHLMCs. For example, the PHLMC established in Gandaki Province (4) is currently not functional due to staff shortages and limited warehouse and the shifting of the warehouse from Nepalgunj to Surkhet (Province 6). 0 50 100 150 PROVINCE 1 PROVINCE 2 PROVINCE 3 GANDAKI PROVINCE PROVINCE 5 PROVINCE 6 SUDUR PSCHIM 137 136 119 85 109 79 88 0 46 54 0 0 79 53 eLMIS Reporting Module Training 2018/2019 Total LLGs Trained Palika 15 As will be discussed as part of Evaluation Question 2, roles and responsibilities among the three levels of government require further clarification if the LMIS and procurement supply chain management system are to operate effectively. For example, confusion regarding the reporting lines of LMIS, has resulted in decreased LMIS reporting. Additional issues resulting from federalism include a dysfunctional “pull” system whereby LLG health facilities should be requesting medicines and supplies, but due to limited capacity, they are still operating under a “push” system; prices of medicines and supplies are increasing due to fragmented procurement; and the roles and responsibilities between the three levels of government still need further clarification. Thus, there appears to be a diminishing sense of responsibility and accountability for procurement and supply chain management. 2. Staff turnover: During the field visits, many stakeholders cited staff turnover at all levels of the MoHP as a major challenge which impeded the ability of the activity to retain buy-in from key personnel, as well as having counterparts with sufficient skills and experience, both specific to GHSC-PSM/Nepal and, more generally, with supply chain management. For example, the evaluation team observed that many of the trained storekeepers who had significant experience in procurement and supply management had been transferred to other offices, and their replacements had not received the necessary training on basic logistics and the eLMIS. Internal Challenges 1. Lack of PSM leadership: GHSC/PSM-Nepal has lacked a consistent country director/chief of party (COP) during its implementation. There have been three COPs within 3.5 years, and, which, according to stakeholders, has constrained its relationship with the MoHP and subsequently limited its ability to provide technical and strategic leadership on supply chain issues. Additionally, several government counterparts expressed confusion with GHSC-PSM/Nepal’s management style because they were often approached by various staff with conflicting messages and requests and, thus, were unclear of whether GHSC-PSM/Nepal had a cohesive and strategic vision for its work. 2. Lack of an inclusive and collaborative management approach: Additionally, several key MoHP stakeholders reported that GHSC-PSM/Nepal’s process for receiving MoHP approvals is to by-pass mid-level managers and technical officers and directly approach senior management for approval of proposed activities and final deliverables. For example, while preparing for the rollout of the eLMIS, there was limited consultation with the desk officer and program manager. At the provincial level, GHSC-PSM/Nepal was supposed to work closely with the Provincial Health Director (PHD), but stakeholders within Karnali Province noted that GHSC-PSM/Nepal directly approached the director of the Management Division at the central level and issued a letter through the MD to increase the reporting status of eLMIS without sufficient consultation with the provincial level. This management style has, per several stakeholders, damaged the relationship with the government and, indeed, many key counterparts expressed considerable frustration with GHSC￾PSM/Nepal. Moreover, because they were not involved in the technical decision-making process, MoHP staff also noted concerns about GHSC-PSM/Nepal’s technical capabilities. 3. Management of the relationship between HQ and the field office and STTA: Several stakeholders, including some GHSC-PSM/Nepal staff, expressed concerns about PSM headquarters’ relationship with its field office, specifically citing the tendency of HQ staff to overmanage the field office and its activities, with the field office being both constrained to a deferential role and slowed in its implementation of activities. Further, stakeholders reported that GHSC-PSM’s headquarters sends STTA without sufficient country experience and, as a result, they fail to understand the country context. A few stakeholders noted that they spent considerable time providing technical assistance to the in-coming consultants so that the STTA could understand the supply chain 16 challenges in Nepal adequately. While stakeholders did appreciate the global experience that STTA brought to the country, they believed it needs to be additionally customized to reflect Nepal’s specific needs. 4.1.4 Missed Opportunities Federalism not only provided a significant challenge to GHSC-PSM/Nepal, but also an opportunity to work at the provincial and LLG levels to build their capacity to effectively procure commodities and manage the commodity supply chain (e.g., TA for procurement and supply chain capacity building, quality control, and general management development of health coordinators and CEOs). This was most frequently cited as a missed opportunity by those exact counterparts located at the lower administrative levels. Several stakeholders reported that GHSC-PSM/Nepal could have coordinated better its training and technical assistance efforts with other activities. For example, the rolling out of eLMIS along with the HMIS at the municipality level could have been better coordinated with the MoHP and other partners (e.g., GIZ, the United Nations Children's Fund, and the USAID-funded Strengthening Systems for Better Health or SSBH). While many stakeholders noted and appreciated some of GHSC-PSM/Nepal’s various activities, few could cite a best practice or success story. The quantification of commodities, forecasting exercises, and quarterly pipeline reviews were the activities most cited by stakeholders as generally improving their capacity in supply chain management. EVALUATION QUESTION 2: Is the activity on the right track for achieving the desired results by the end of the activity? As per reviewed documents, only a limited number of targets were set for reducing stock-outs, increasing SDP reporting rates to the LMIS, and warehouses instituting FEFO; thus, it is unclear what the “desired” results are. Further, as discussed under Evaluation Question 1, data on warehouse institutionalization of FEFO have not been routinely collected and analyzed. 4.2.1: Stock-outs at SDPs The goal of GHSC-PSM/Nepal is to ensure the uninterrupted supply of quality essential health commodities; therefore, as planned, the evaluation team tried to track for stock-out rates at the SDPs. However, the stock-out rate is a higher-level indicator of supply chain management, and it depends on many factors, such as adequate procurement, forecasting and planning, supply chain management, prescribing and dispensing patterns, etc. Regardless, it should be noted that the stock-out rate of the FP/RH has increased from 7.8 percent in Quarter 1 to 10.7 percent in Quarter 2, 2019 (see Figure 6 below) in GHSC-PSM/Nepal supported regions, especially given that by that time GHSC-PSM/Nepal had been implementing activities for nearly three years8. 8 Per the GHSC-PSM/Nepal February 2019 M&E Plan, this indicator is disaggregated by GHSC-PSM supported sites and non-supported sites. 17 Figure 6: Stockout rate in SDPs in GHSC-PSM supported regions, TO3 - FP/PRH Source: GHSC-PSM Quarter 1 and 2 M&E Annexes (DRAFT) Several stakeholders reported that efforts have been made to reduce the stockout rate of FP/RH commodities by redistributing oral combined contraceptives from overstocked SDPs to less stocked districts (for example, from Rolpa to Banke District, and from Panchthar to Morang). However, its effect was negligible in reducing the stock-out rate due to the limited supply and higher demands for combined oral contraceptives. As mandated, provinces and LLGs have procured some listed essential medicines to avoid the stockout at SDPs, but quality remains questionable because of their lack of capacity to provide quality assurance. Figure 7: Composite Stock-Out Rate - By FP Method (TO3) in GHSC-PSM Supported Regions Source: GHSC-PSM Quarter 1 and 2 M&E Annexes (DRAFT Global Report) As shown in Figure 7 above, the stock-out rates for several specific family planning methods, likewise, increased instead of decreasing in 2019. For example, the stock-out rates of combined oral 7.8% 10.7% 0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 2019 Q1 2019 Q2 6.7% 6.1% 13.6% 8.1% 9.1% 18.3% 17.6% COMBINED ORAL METHODS INJECTABLE CONTRACEPTIVES IMPLANTABLE CONTRACEPTIVES COPPER BEARING IUCD 2019 Q1 2019 Q2 18 contraceptives increased from 6.7 percent in Q1 to 8.1 percent in Q2, 2019. Similarly, the stock-out rate of injectable contraceptives increased from 6.1 percent in Q1 to 9.1 percent in Q2 and implantable contraceptives from 13.6 percent in Q1 to 18.3 percent in Q2. Most stakeholders noted that this was due to the non-procurement of those commodities by relevant government officials. 4.2.2 SDP Reporting Status The eLMIS has been implemented for more than one year and has encountered many challenges. As per the document review, the reporting rate of the eLMIS was only 24 percent in by Q2, 2019 (see Figure 8 below). Per stakeholder interviews, the LLGs had stopped reporting to the PHD and MD due to the confusion created by the introduction of the federal structure. The LLGs had thought that they were a separate level of government and, therefore, they believed there was no need to send the LMIS reports to the provincial and central levels. Likewise, some Provincial Health Offices (PHOs) were reluctant to accept the LMIS reports from the LLGs. This has adversely affected the reporting status of the SDPs. A circular issued by the MD clarified this misunderstanding, and, subsequently, the reporting rates have slightly improved. However, it still currently is only approximately 40 percent as reported by interviewed FSOs. Second, as will be discussed as part of question 4, many SDPs continue to use two types of system. Thus, eLMIS data entry and usage is limited. While the reporting status is slowly improving, per stakeholders, it is still too low to use data in the decision-making process. For example, many LLGs do not utilize the eLMIS requisition forms to report and request commodities regularly. Figure 8: LMIS Reporting Rate, Quarter 2, 2019 Source: GHSC-PSM Quarter 2 M&E Annex (DRAFT) Per interviewed stakeholders, few efforts have been made by GHSC-PSM/Nepal to increase the reporting rates. This could have been done, for example, by telephone conversations with SDPs, in-person TA provided by the FSOs to counterparts, and greater dissemination and discussion of the circular issued by the Management Division. 24% 24% TO3 PRH, 2019 Q2 TO4-MCH, 2019 Q2 19 4.2.3 Instituting FEFO According to both stakeholders and field observations, the FEFO method appears to be applied when sending commodities to districts and SDPs. Most interviewed stakeholders reported that the eLMIS helps maintain FEFO. There is no compliance data on the institutionalization of FEFO. Therefore, it is difficult to assess ongoing adherence to FEFO. According to the field observations, it appears that warehouse space does not permit health commodities to be kept in batch-wise columns due to the limited space. During site visits, the evaluation team noted that the following five issues need to be addressed if application of FEFO is to be consistent and sustainable: 1) the inadequate storage space in warehouses and medical stores, 2) the physical space is not being adequately managed, 3) the basic standards of good warehousing practices are unevenly applied, 4) the adherence to the standards is not routinely monitored by MoHP staff (nor GHSC-PSM staff), and 5) some warehouses are not properly ventilated to prevent the commodities from expiring prematurely. (See pictures in Appendix 5.) Moreover, the “model” medical store in Pathlaiya, which is currently under construction, while increasing the space to accommodate additional commodities does not appear to be adequately ventilated nor equipped with a temperature control system to withstand the high temperatures of the region, nor does it have sufficient lighting. Evaluation Question 3: How has the supply chain in Nepal through the GHSC￾PSM/Nepal interventions advanced to greater self-reliance (e.g., mobilizing adequate resources effectively and implementing transparent supply chain functions effectively)? 4.3.1 The Journey to Self-Reliance Per most interviewed stakeholders, because of PSM’s limited focus on only a few technical areas of supply chain strengthening (and with a limited geographical focus), its contribution to developing a self￾reliant supply chain in Nepal is minimal. As mentioned previously, most stakeholders perceive GHSC￾PSM/Nepal’s primary focus is on eLMIS, which is only one sub-component of a functional procurement and supply chain management system. And, given the concerns stakeholders expressed about the implementation of the eLMIS (see below in question 4), most also do not believe that GHSC-PSM/Nepal will have made a significant impact by its closure in April 2021 in moving the supply chain in Nepal forward in being self-sufficient. 4.3.2 Stakeholder Buy-in and Ownership Key stakeholders within the MoHP/DoHS technical/operational units stated that they have not been sufficiently involved in GHSC-PSM/Nepal’s technical decisions and have instead been bypassed for approvals given by senior leadership. Indeed, some key technical counterparts expressed significant concerns about the overall management style of GHSC-PSM/Nepal with a few stating that they believed that GHSC-PSM/Nepal counterparts were more interested in fulfilling work plan requirements than in taking the time to consult with counterpart technical units to ensure the technical accuracy and appropriateness of its interventions. Additionally, the level of GoN “ownership” of project activities is mixed. At the central level, there is a strong desire by senior management to scale up eLMIS, perhaps, due to the Disbursement-linked Indicators (DLIs) related to the eLMIS, which the World Bank and DfID are utilizing to make disbursement decisions. At the provincial and LLG level, based on informant interviews and observations, there is limited ownership of the activity. Many had little knowledge of the activity other 20 than the specific interventions (e.g., eLMIS, medical warehouse/storage improvements, FSO placements, capacity-building, etc.) that had been implemented in their respective sites. As will be noted below in question 4, having the FSOs function more as seconded staff rather than TA providers has further undermined ownership at the local level. 4.3.3 Sustainability None of the interviewed stakeholders stated that the supply chain would be sustainable without external assistance after GHSC-PSM/Nepal’s closure in April 2021. When questioned about the timeframe for when the supply chain in Nepal would be self-sufficient without external assistance, a range of answers was given from 5 to 20 years, and most respondents noted that sustainability was difficult to predict based on possible external shocks. Further, some stakeholders noted that, most likely, they will regress to previous practices. This was particularly notable in regard to warehousing practices as most respondents noted that the medical stores were both inadequate in size and had been built 40 to 50 years ago when standards were considerably different. Evaluation Question 4: What has been the experience of participating stakeholders in the transfer of skills in the supported technical areas? 4.4.1 The Usefulness of eLMIS Data, Challenges in the Overall Implementation of eLMIS, and eLMIS Ownership As mentioned previously, eLMIS has been implemented in all CMSs, four central sub-stores, two provincial medical stores, twenty-two districts, four LLGs, and twenty-three health facilities of Provinces 5 and 6 with the intended outcome to have one system in use for public health supply chain management. Of the four LLGs where eLMIS is being implemented, only three sites are functional. The online Inventory Management System (IMS) was discontinued in 22 districts of Province 5 and 6, the CMS, and the PMS in Butwal and Nepalgunj. The online IMS is still in use in 55 districts of Provinces 1, 2, 3, 4, and 7. Additionally, the plan was to discontinue the desktop-based reporting application with the implementation of the eLMIS, but, in practice, both systems are in place in some facilities. During the interviews, stakeholders were aware that the data are useful for quantification and budgeting processes, but the eLMIS has had limited effect on decision-making due to partial coverage and low reporting status in Provinces 5 and 6. It also was noted that the use of consumption data in forecasting is limited because of issues with data migration. There also are concerns regarding data quality as some health facilities continue to maintain three systems (eLMIS, online IMS, and paper-based) because of their lack of confidence in eLMIS data. The main challenges stated by stakeholders in the implementation, operation, and management of the eLMIS system include: • Software not being user-friendly: As per KIIs, most stakeholders shared that the system is good, but not user-friendly. Features such as data analysis and reporting are adequate but do not fully meet the needs of the end-users. Stakeholders also mentioned that the software is complex, and it requires several steps to generate. For example, the Handover Form can take between two to eight hours to produce. Since the system’s reporting rate is low, commodities are not dispatched based on eLIMIS reported stock levels, but on a distribution plan, which is provided by the respective divisions. For example, for mental health drugs, corresponding hospitals provide the distribution lists. Likewise, the Family Welfare Division provides the distribution list for all FP commodities to be supplied to each province/district. Thus, by default, the supply chain of these sites is still a “push” system. 21 • Slow internet connectivity: Though the eLMIS system was optimized at 512 kilobits per second (kbps) to address internet connectivity, most stakeholders shared that it is difficult to use the eLMIS because of the internet connectivity. Some reported that there are frequent disruptions of internet connectivity, which is compounded by out-of-date desktop computers. For example, in Banke and Bardiy, the operators could not open the eLMIS software for observation as the computers kept on restarting. Most KIIs reported that they require an online, as well as, an offline version for the smooth functioning of supply chain management activities as an offline version will work during disruptions of internet connectivity. • Inability to modify incorrect data on-site: The eLMIS currently doesn’t allow the user to edit the data to correct inadvertent errors. For example, at times, it becomes an issue when a delivery truck is outside the PHLMC store and is ready for the loading of commodities, and the eLMIS user has typed in the wrong digits. Hence, the user must prepare a paper-based handover form and then email the Level 2 helpdesk at Teku to solve the problem. Resolving this type of issue can take up to three working days. In the meantime, the storekeeper must utilize a paper-based system. • Insufficient and/or inappropriate data fields: The eLMIS dashboard shows accumulated stock-out of commodities but doesn’t give item specific information. For example, the software generates the batch-wise information, but it doesn’t provide a commodity expiration date in the summary report, which is critical for decision-making. A separate expiry report must be generated and manually updated. • The off-site trouble-shooting system: There are different levels of help-desk support: Level 1 (L1) and Level 2 (L2) supports users through helpdesk phone calls, emails, and remote support or through site visits of eLMIS officers if the problem cannot be fixed virtually. Level 3 (L3) support is responsible if there are software bugs identified or complex support issues that L1 and L2 cannot handle. The L3 support is situated in Sri Lanka. Per stakeholder feedback, the L1 & L2 can support the eLMIS users within an hour to a day; however, the L3 support can take up to a week to fix the identified problem. For example, in Banke, the eLMIS operator prepared the handover form of two batches of the ciprofloxacin 500mg (50,000 tabs by two batches: 20,000 from batch 1 and 30,000 from batch 2). The stock automatically got deducted from batch one but not from batch two. After reporting the issue to L3 support, it took approximately one week to resolve this issue. • Insufficient buy-in from multiple GoN levels: o Lack of information: During the KIIs, stakeholders shared that they had not known about the eLMIS’ licensing and maintenance costs until six months before the interviews and most had expressed serious concerns about its affordability when GHSC-PSM/Nepal presented those costs. o National review by eLMIS users: Most of the stakeholders stated that they are unaware of the implication of scale-up of eLMIS. Many of them raised the issue of the validity and reliability of the data. o Reluctance of mid-level managers: The mid-level managers are currently reluctant to scale-up eLMIS as they believe that the problems cited above first need to be sorted out. They are waiting for responses from GHSC-PSM/Nepal on a 13-point improvement plan forwarded by the MD. o Stakeholders shared that GHSC-PSM/Nepal needed to examine more in-depth the competitive advantages of using commercial and/or open-source before procuring the chosen software from Bileeta and had on-going concerns regarding the licensing arrangements and the ability to modify the source code if needed. 22 4.4.2 Further Challenges for eLMIS Scale-up In examining the geographical diversity, population size of the LLG’s, morbidity patterns, and access to the internet and its connectivity, most interviewed stakeholders believe, it will be very challenging to roll out eLMIS in the remaining LLGs (749) based on the finding of current four LLGs. The challenges for scale-up and sustainability include: • Technical challenges: o Lack of interoperability: The health facility codes must be the same to ensure interoperability of the HMIS and eLMIS system. But it is unclear, per stakeholders, whether the two systems are utilizing the same codes. Further, to customize the eLMIS software, the source code is required, and, currently, it is uncertain whether the vendor (Bileeta) will provide it. o Data Migration: At present, data migration is limited to the central level. The provincial levels do not have access and capacity to migrate data from one system to another. o Training: To date, GHSC-PSM/Nepal has trained only four municipalities out of 753. The activity must train more than 1,500 health workers (two persons per each site) for implementation of eLMIS. They also need to conduct a Master Training of Trainers to roll out the eLMIS. Moreover, per stakeholder feedback, the quality of training remains questionable. Many of the visited sites shared that they need to update their knowledge and skills. The projected training cost is $295,800. o Hardware: All the remaining facilities require hardware that will need to be purchased, installed, and prepared for the implementation of eLMIS. • Financial challenges: o The GoN may not be able to sustain the system because of the high costs of purchasing and maintaining a new eLMIS system or paying for the licensing fee and potentially high maintenance costs of the existing system. Per PSM’s eLMIS scale-up plan, the total cost for scale-up is approximately $1.345 million, including annual maintenance costs, an eLMIS support cost of $530,000, and a licensing fee of $301,200. Stakeholders noted that the information on annual maintenance costs and licensing was shared only six months ago. They also shared that the future operational and transactional costs are unclear as different estimates had been shared. Table 4: Phase 2 and Phase 3 Details FY 2019: Phase 2 Scale-up Details FY 2020 & Beyond: Phase 3 Scale-up Details ● 58 new sites: 3 PSMs, 55 PHOs. ● 753 LLGs online (reporting only). ● Old systems will be completely retired and eLMIS will be the only system for logistics: o All provincial medical stores will be operational on eLMIS. o LLGs and health facilities reporting through eLMIS. o Approximate Budget: $1,345,775. ● Nationalized data hosting. ● Full LLG and HFs automation. ● Annual maintenance cost (AMC) and support contracts. ● Transactional licenses and training for LLGs and health facilities. ● Sustained monthly reporting statistics. ● Approximate Budget (FY 2020): $3,906,330. ● Approximate Budget (FY 2021): $7,819,180 / • Legal/policy challenges: Per stakeholders, the Nepal Public Procurement Act prohibits them from contracting a single entity for procurement of goods or services. The GoN must select a vendor through a 23 nationwide, open, and transparent tendering process. In other words, after GHSC-PSM/Nepal ends in 2021, even if the GoN decides to select an eLMIS vendor through open tendering, it cannot ensure that the current vendor will be selected. Several stakeholders also raised the issue about the access of source code and the implications for future usage of the current GHSC￾PSM/Nepal supported eLMIS system. A few stakeholders shared a concern that the foreign location of the server in Singapore may violate the ICT Policy and the Electronic Transactions Act, 2063 (2008), although a review of the act by the evaluation team did not find this to be an issue. 4.4.3 Capacity Building Through TA for Forecasting to Ensure a Secure Supply of Commodities With the federal structure in place, GHSC-PSM/Nepal provided training and/or TA in quantification and forecasting in Provinces 1, 4, 5, and 7. As per the KIIs and document review, in Province 5, they are utilizing the procurement and forecasting guidelines developed by GHSC-PSM/Nepal, and in Province 4, they are utilizing the quantification skills As mentioned earlier, Province 5 has drafted an annual procurement plan that will be budgeted for in fiscal year 2019/2020. Some participants of Province 4 training shared that the two-days of training on the large number of topics covered was insufficient. 4.4.4 Institutionalization of Warehousing Practices Based on field observations, some good warehousing practices have been institutionalized, such as the storage of commodities on racks and pallets, visible and labeled expiration dates, utilization of bin cards, and maintenance of the FEFO system. However, the quality assurance aspect of commodities lacked in Provinces 4, 5, and 6 (FSO presence is limited in Province 4). For example, in the PHO of Bhairawaha, all commodities were stored on racks providing easy access for counting and general management and there was a separate area for the storage of liquid formulations. In general, the rooms were clean, and expiration dates were visible. However, the room temperature was 38ºC, which could hasten the expiration dates of the commodities. In contrast, at the PHLMC in Butwal, a few commodities were stored near the staircase due to lack of storage space and quality assurance was not prioritized. Due to limitations of storage space, a needs assessment of the warehouse was done by GHSC￾PSM/Nepal, but the recommendations are yet to be shared. The PHLMC has a budget for the construction of the building. As communicated, on March 20, 2017, the PHLMC also forwarded the estimated budget for the refurbishment and installation of air conditioning so that an adequate storage temperature can be maintained. To date, however, they have yet to receive a response. Per stakeholders at the visited sites, most likely, these warehousing practices will not be sustained given basic infrastructure issues within the medical stores and warehouses. 4.4.5 Ownership (i.e., Budgeting) for the Supply Chain Pharmacists Funded Under Red Book During the KIIs, stakeholders shared that the pharmacists have certainly improved the situation of supply chain management by improving the storage situation. There are 16 pharmacists funded by USAID through the Red Book government-to-government (GTG) program. It was agreed that the GoN would create the positions of pharmacists as per the recommendations of the organization and management (O&M) survey. USAID will gradually withdraw its support. However, it has not happened yet due to the restructuring of the government under federalism and subsequent budget limitations. 24 4.4.6 Main Technical Contributions of FSOs Based on field observations and interviews, while the FSOs have provided some technical assistance, their current job function is to primarily act as seconded staff to the PHLMC and central medical stores. The high turn-over of PHLMC and PHD staff forces the FSOs to work as regular staff and perform activities assigned by the immediate government supervisors. For example, they perform activities not mentioned in their job descriptions, such as preparing the Handover Form, the Entry Reporting Form, and bidding documents for procurement (which is a major responsibility of the procurement section). These duties limit their field mobility and time to provide TA to PHDs, PHOs, and LLGs. FSOs had limited exposure and training in supply chain management (most received four-days basic logistics management training and on-the-job training) prior to their field postings; thus, their knowledge and skills may have limited their capacity to provide TA for some time. 4.4.7 The Main Contributions of the Logistics Working Group (LWG) Interviewed stakeholders noted that the LWG has not met in more than one year at the central level. Thus, they were unable to provide specific answers regarding the LWG’s technical contributions. At the provincial level, stakeholders shared that they have procurement committees, but no provision for LWGs where they can discuss relevant issues related to supply chain management of commodities. The procurement committee is represented by government officials. For example, in Province 5, the PHD chairs a coordination committee for all involved stakeholders which is evolving, and in the next fiscal year this forum will be utilized to address issues related to the supply chain. The forums, such as the LMIS taskforce, quarterly pipeline meeting, consensus quantification and forecasting, procurement meetings, and eLMIS meetings were cited as contributing to improving a steady and continuous flow of commodities. However, only some of the abovementioned meetings (quarterly pipeline meeting and consensus quantification and forecasting) were conducted regularly. Most relevant stakeholders noted that while these various forums provided an opportunity to share reporting results and to have technical discussions, they lacked an overall strategic direction to discuss issues, such as project sustainability or the systemic problems within the supply chain. 25 VI. CONCLUSIONS As noted in the Background and Findings section, the procurement of commodities and the supply chain in Nepal is fragmented across several different fundamental elements. The two greatest concerns observed as part of this evaluation are that there continues to be partially verticalized systems for different health issues and, with the implementation of federalism, the roles and responsibilities for procurements and management of the supply chain, along with corresponding authorities and budgets, have been devolved to lower administrative levels (provinces and LLGs), further fragmenting the system. While the decision to federalize the system was outside of GHSC-PSM/Nepal’s manageable interest and the fragmented supply chain was in place before the start of GHSC-PSM/Nepal, the activity’s strategic response to both issues has lagged. Rather than using federalism as an opportunity to re-think how to address these underlying systemic challenges and needed changes, GHSC-PSM/Nepal instead limited its focus on short-term technical fixes; thus, most likely, at the conclusion of GHSC-PSM/Nepal’s activities, the “supply chain” will neither be significantly transformed nor sustainable. Further, while again, it is recognized turnover in GoN staffing is outside the control of GHSC￾PSM/Nepal, there is currently insufficient buy-in from government counterparts at all levels. Indeed, some current key DoHS staff and leadership expressed significant frustration with how GHSC￾PSM/Nepal has implemented its interventions. This has the most substantial implications for the eLMIS which, per KIIS and field observations, has limited counterpart support. Given the significant investment by USAID in the eLMIS, as well as, the tying of DLIs to its implementation, GHSC-PSM/Nepal will need to focus its efforts on correcting the current situation without delay. To be clear, it is one of the conclusions of this evaluation that ensuring a cohesive and sustainable procurement and supply chain is a shared responsibility amongst all relevant partners, including not only GHSC-PSM/Nepal and USAID, but other external development partners, technical agencies, implementers, and the GoN. However, GHSC-PSM/Nepal’s lack of consistent leadership and a clear strategy cannot be overlooked in contributing to inadequately addressing fundamental supply chain issues. With just 22 months of implementation left for GHSC-PSM/Nepal, urgent action is needed to review and revise its approach to better meet the identified challenges. 26 VII. RECOMMENDATIONS (TECHNICAL) 1. Review and revised the GHSC-PSM/Nepal M&E Plan As discussed above, almost all performance indicators are taken from PSM’s global M&E Plan, with only one specific to Nepal. Therefore, they may be insufficient for measuring the performance of supply chain management technical assistance, which GHSC-PSM/Nepal is providing. GHSC￾PSM/Nepal needs to seriously consider reviewing and revising its M&E plan to capture better the activities which it is undertaking. However, this must be balanced by GHSC-PSM/Nepal’s closure in 22 months and whether there is enough time to collect and analyze new data against new targets adequately. One additional item to consider would be whether it is worthwhile for GHSC￾PSM/Nepal to develop its own internal performance monitoring plan (PMP) and share some of the results from those measures with USAID/Nepal. 2. Build the evidence base for the eLMIS GHSC-PSM/Nepal should use the remaining life of the activity for examining the eLMIS results in Provinces 5 and 6 to build the evidence base for the effectiveness, acceptability, and sustainability of its eLMIS approach and specific interventions. The evaluation team is not doubting the value of eLMIS in general in strengthening Nepal’s supply chain, but, rather, whether the system and approach GHSC-PSM/Nepal has utilized are worth continuing, given the number of noted issues. Building the evidence base of GHSC-PSM/Nepal’s eLMIS can be done by GHSC-PSM/Nepal’s monitoring data and a separate, independent appraisal, incorporating those results into a national review and discussion. 3. Limit the roll-out of the eLMIS GHSC-PSM/Nepal may consider rolling-out its eLMIS to one or two more provinces with a focus on corresponding LLGs (based on the results of the review mentioned above) with the remaining provinces to be part of any follow-on supply chain activity (post-April 2021) if the results are favorable. 4. Untie or delay the eLMIS DLIs GHSC-PSM/Nepal and USAID/Nepal need to discuss with other EDPs, primarily the World Bank and DfID, the untying or delaying of the eLMIS-related DLIs. Because the eLMIS has encountered several bottlenecks, including the system’s limited reporting capacity to provide robust evidence for decision-making, these challenges should not jeopardize future disbursements of grant and loan monies as having the DLIs tied to GHSC-PSM/Nepal’s implementation of its system appears to have placed undue pressure on its roll-out despite all off the identified challenges. 5. Finish the medical store/warehouse improvements GHSC-PSM should continue and finish the CMS work, but none should be labeled “model” medical stores as the new facilities may still have some fundamental issues to be addressed, such as providing adequate ventilation, lighting, and temperature control. To ensure warehouse practices and physical improvements continue, GHSC-PSM/Nepal needs to increase the number of monitoring visits by its staff (either Kathmandu-based or by sufficiently trained FSOs – see recommendation below). 6. Advocate for a centralized, independent procurement agency. As discussed above, of great concern to the evaluation team is that the 753 LLGs will become individualized procurement and supply chains, greatly increasing transaction costs and pricing for medicines and supplies. Further, it is challenging to provide TA to all LLGs to establish their own procurement and supply chain mechanisms. To avoid this outcome, GHSC-PSM/Nepal should work 27 in collaboration with other development partners and TA providers (e.g., GIZ, NHSSP, UNFPA, UNICEF) on policies and procedures for developing a pooled procurement mechanism and potentially shared supply chain systems (logistics, QA, disposal methods, etc.) which can be utilized by LLGs. While the evaluation team recognizes that, because of federalism, there might be hesitancy by the GoN in undertaking the development of a centralized, independent procurement agency, GHSC-PSM/Nepal can provide additional evidence for its need by providing scenarios (centralized versus decentralized), corresponding cost estimates, and possible regional study tour to show counterparts the advantages of a centralized procurement mechanism. 7. Ensure supply chain roles and responsibilities are defined and understood All level of government (federal, provincial, and LLGs) are involved in procurement and supply chain management of commodities. To effectively operate and sustain procurements and the eLIMS, all levels of the GoN need to fully understand their corresponding roles and responsibilities within the procurement and supply chain system. Therefore, GHSC-PSM/Nepal should work with relevant partners to develop guidance and training for all stakeholders and ensure that this information is sufficiently disseminated and understood. 8. Shift capacity-building to lower administrative levels As mentioned previously, federalism should provide the basis for a basic shifting in GHSC￾PSM/Nepal’s approach, specifically regarding capacity-building at lower administrative levels. Therefore, the evaluation team recommends that GHSC-PSM/Nepal works with its partners in developing the capacity of lower administrative levels to manage their newly defined PSCM roles and responsibilities (as given in Recommendation #7 above). Of importance, depending on whether aspects of the supply chain are centralized or not (see Recommendation #6 above), build the capacity of LLGs such that they can provide their quality assurance on procured items. Given that there are 7 provinces and 753 LLGs and only 22 months of implementation left, this may necessitate a discussion among partners (GHSC-PSM/Nepal, USAID, GoN, other EDPs, and IPs) as to which administrative units to prioritize for capacity building. 9. Advocate for greater integration of the supply chain While it is recognized here that a greater integration of the supply chain across health elements is certainly not the sole responsibility of GHSC-PSM/Nepal, it can work with other TA providers and development partners to advocate the GoN for a better-integrated supply chain. This advocacy work could be done via meetings/forums, technical briefs, and position papers. 28 VIII. RECOMMENDATIONS (MANAGERIAL) 10. Ensure MoHP/DoHS inclusivity in developing technical assistance package For any future technical assistance/activities, GHSC-PSM/Nepal needs to be more inclusive in developing materials, in planning training sessions, and in its overall strategic approach. This may include, for example, requiring a sign-off sheet from MoHP technical/operational units and staff acknowledging that they have reviewed and approved proposed activities. Additionally, given the turnover in GoN staff, this also may provide a safeguard for GHSC-PSM/Nepal in its dealings with new staff. 11. Review GHSC-PSM/Nepal’s organizational structure To ensure that it is “fit for purpose” in its final 22 months of implementation in meeting any revised objectives and corresponding targets, GHSC-PSM/Nepal should: ● Review and, if needed, adjust its organizational structure (both in-country and its relationship to PSM/HQ) to ensure it has the sufficient structure, staff, and autonomy to be responsive to the evolving supply chain needs in Nepal. ● Ensure that STTA is from repeat providers/international consultants who understand the Nepalese context. ● Examine the proportions of staff who are in Kathmandu versus the field/areas of technical focus. ● Review GHSC-PSM/Nepal job descriptions to ensure they match any new strategic undertakings. The evaluation team does not recommend that GHSC-PSM/Nepal should undertake any radical restructuring (firing and/or hiring) of its staff based on the findings, as well as, its closure date, as it will be very difficult for GHSC-PSM/Nepal to recruit, train, and retain staff without significantly disrupting implementation. 12. Re-orient the FSOs and ensure they understand their TA role GHSC-PSM/Nepal must work with its FSO sub-contractor to ensure that both its current FSOs and the FSOs which are scheduled to be hired are provided sufficient (re)training such that they are clear that their role is as a TA provider and not seconded staff. Further, GHSC-PSM/Nepal should ensure that the FSOs are provided sufficient resources (time and budget) so that they can provide satisfactory and consistent TA to their corresponding catchment areas. 13. Ensure GHSC-PSM/Nepal hosted meetings also incorporate strategic discussions GHSC-PSM/Nepal should continue hosting/supporting its various meetings (quantification, forecasting, pipeline, LWG), but re-orient them to include more strategic discussions to address the more systemic supply chain issues identified above. 29 APPENDIX 1: EVALUATION STATEMENT OF WORK MID-TERM PERFORMANCE EVALUATION Statement of Work (SOW) for USAID/NEPAL’S Global Health Supply Chain-Procurement and Supply Management (GHSC-PSM) Activity Draft FINAL January 2019 30 Background The Logistics Management Division (LMD) was established in 1993 under the Department of Health Services (DOHS) with a mandate to effectively and efficiently manage health commodities and delivery of healthcare services that will ensure the rights of citizens for quality healthcare services. The LMD operates through a network of two central stores - Teku and Pathalaiya; five regional medical stores (RMS); 75 District Public Health Office (DPHO) stores; and over 4,000 health facilities. With the restructuring of the Ministry of Health and Population (MOHP) in 2018, the LMD is renamed the Logistics Management Section (LMS) under the Management Division (MD). The major functions of the LMS are to forecast, quantify, procure, store, and distribute health commodities to the service delivery points throughout the country through its network of stores. It is also responsible for the repair and maintenance of bio-medical equipment, instruments, and transportation vehicles. In 2015, the MOHP developed the Nepal Health Sector Strategy (NHSS), which guides all activities in the health sector from 2016-2021. One of the national goals in the NHSS is to improve the efficiency, economy, and transparency of procurement and the supply chain of essential medicines and equipment. The LMS through its network is serving 4,000 health facilities, which include 207 primary health care centers (PHCs) and more than 3800 health posts (HPs) that serve diverse communities across Nepal’s terrain, hilly, and mountainous zones. Reforming the procurement and distribution processes of essential medicines and the health sector have long been an area of discussion within the Government of Nepal (GON) and its external development partners (EDPs). To this effect, in April 2015, The MOHP approved the procurement reform plan dubbed the “Concept Note for the Restructuring of Logistics Management Division (LMD).” The Plan includes 16 action points, or outputs, that are intended to improve the effectiveness, efficiency, and transparency of the procurement process. The Plan includes some outputs related to improving the in￾country supply chain and logistics information flow. Stemming from discussions on procurement reform, a recommendation was made on September 19, 2014 by the Procurement Reform Committee of the DOHS/LMD. One of the three principles for improving health sector procurement included an organization and management (O&M) study to be conducted on the restructuring of LMD. The external development partners (EDPs) strongly support the implementation of the actions in the procurement reform plan and the recommendations outlined in the O&M report. Previous support through USAID projects: From 1994-2013, the USAID|DELIVER project contributed to significant gains across the supply chain, including but not limited to implementation of a web-based LMIS, warehouse management, capacity building through activities including basic health logistics and LMIS training, developing a demand-based supply system, and private sector engagement. The DELIVER project successfully contributed to decreasing stock-out rates and improved in-country capacity to the point in which the MOHP took over the procurement and management of the supply chain. The DELIVER project ended its technical assistance in 2013. However, USAID/Nepal continued to support the CRS, a social marketing company, with contraceptives and condoms that were procured through DELIVER. In September 2013, USAID/Nepal awarded the Health For Life (H4L) Logistics contract to Lifeline Nepal (a local NGO) to provide Technical Assistance to ensure the sustained availability of essential and auxiliary health commodities in health facilities. The Health for Life (H4L) logistics contract ended in 2016 as the scope was limited to only 14 districts. Despite all the support, Nepal’s public health supply chain suffers from frequent staff turnover at all levels and difficult geography. These issues were aggravated by a number of natural and policy-related challenges, including the earthquake in April 2015 leading to reallocation of regularly programmed funds 31 to relief efforts, blockades at the border with India resulting in fuel shortages, and the intervention of the Commission for the Investigation of Abuse of Authority (CIAA) that led to a year-long lapse in central level procurement of medicines as well as staff shortages and staff reassignments. In this context, facility stock out rates for essential drugs at the end of 2015 swelled from an average of around 20-25% to 50%. To address this issue, USAID/Nepal allocated funds to the Global Health Supply Chain Program￾Procurement and Supply Management (GHSC-PSM), a five-year global mechanism, implemented by Chemonics International Inc, to provide TA to the MOHP. Description of the Activity Activity Name Global Health Supply Chain Program-Procurement and Supply Management (GHSC-PSM) Implementer Chemonics International Inc. Cooperative Agreement/Contract # AID-OAA-I-15-00004 Total Estimated Ceiling of the Evaluated Project/Activity (TEC) $15,975,000 Life of Strategy, Project, or Activity April 2016 – April 2021 Active Geographic Regions National Geographic Coverage Development Objective(s) (DOs) DO3. Increased human capital IR3.2 A healthier and well-nourished population Sub -IR 3.2.1 Quality of Health Services improved and Sub-IR 3.2.2 Use of and Access to Health Services Increased USAID Office Health Office, USAID/Nepal The purpose of the GHSC-PSM activity is to ensure the uninterrupted supply of health commodities in support of the U.S. government-funded public health initiatives around the world. The activity provides direct procurement and supply chain management support to the President’s Emergency Plan for AIDS Relief (PEPFAR), the President’s Malaria Initiative (PMI), USAID’s family planning and reproductive health program and the Office of Maternal and Child Health and Nutrition. To support the U.S. government￾funded global health activities, GHSC-PSM manages an array of health commodity procurement services and provides system strengthening technical assistance across all elements of a comprehensive supply chain. At the country level, GHSC-PSM supports strategies and priorities under the following three objectives: ● Global commodity procurement and logistics. ● Systems strengthening technical assistance. ● Global collaboration to improve the long-term availability of health commodities. 32 Map of Intervention Area Purpose of the Mid-Term Evaluation This statement of work describes the scope for the mid-term evaluation of GHSC-PSM/Nepal. The purpose of this evaluation is to better understand whether the activity is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality TA to the MOHP in achieving the national supply chain objectives. The findings from the evaluation will be used to strengthen current activities through the identification of course-correcting actions to ensure the activity will meet its objectives; develop recommendations for future activities; and provide recommendations to the Government of Nepal and External Development partners in Nepal with evidence of good supply chain management practices. Evaluation Questions The following evaluation questions will guide the final evaluation process. In answering these questions, the evaluation team must analyze how the activity has impacted the overall supply chain system in Nepal to-date. 1 To what extent, in what ways, and with what challenges has the GHSC-PSM activity contributed to supply chain performance improvements? Given the activity’s performance challenges over the last 2 and a half years, what are the factors that influenced or hindered activity implementation performance? A. How were TO4 TA funds used by PSM to strengthen MCH commodity supply? How did PSM leverage the co-mingling of TO3 and TO4 TA funds to gain activity efficiency? ● Quarterly, annual reports, etc. ● Document and data review ● Key informant interviews with relevant stakeholders including USIAD and IP 33 B. Has the combined fund led to efficiencies for improving the MCH commodity supply chain in the national supply chain system? C. Was the activity able to track its budget and accruals accurately? 2 Is the activity on the right track for achieving the desired results by the end of the activity? What is the performance of the in-country systems strengthening indicators including but not limited to: A. Stock-outs at service delivery points (SDPs); B. Percentage of stock status observations in storage sites where commodities are stocked as planned; C. Service delivery point reporting rate to the logistics management information system (LMIS); and D. Percentage of warehouses with FP and MNCH commodities managed as per FEFO system. ● Document and data review ● Data analysis of GHSC-PSM’s Dev Results data ● Key informant interviews 3 To the extent possible, describe how the supply chain in Nepal through the GHSC-PSM intervention has advanced to greater self-reliance (mobilize adequate resources effectively, implement a transparent supply chain functions effectively), a greater capacity to sustain the high levels of performance documented above in the absence of external assistance. A. To what extent has the GHSC-PSM’s technical assistance contributed to this trajectory/journey? As appropriate, include a reference to relevant activity indicators, but do not limit the response to the indicators. B. How has the activity support improved supply chain management performance and enhancing the capacity of the supply chain cadres within the Logistics Management Section? C. Has the activity secured ownership of project activities? ● Document and data review ● Key informant interviews ● USAID Mission staff ● USAID/W staff ● GHSC-PSM Activity staff ● Home Office: Health System Strengthening and Country Programs Teams, Project Management Units, etc. ● Field Office staff ● Review relevant findings from USG customer satisfaction assessment of Global Health Supply Chain Program (GHSC)—survey and interviews of Mission staff (conducted under GH Pro 614) 4 What has been the experience of eLMIS users in management supply chain activities? H. How effective is the eLMIS implementation? I. Has the eLMIS data been useful in data visualization, transparency and supply chain decisions in the overall supply chain management? J. What are the challenges in the overall operation and management of the eLMIS? (system’s user friendliness/ease of use, how well are users trained on the use, system’s strengths and weaknesses, etc.) K. The extent to which stakeholders at all levels of the federal structure have taken ownership of the eLMIS? Key informant interviews (KII): ● Users at different tiers (central, regional, district, LLG and SDPs) ● Review eLMIS reports ● Comparison of performance in eLMIS implemented and other sites Based on the review of the Nepal program, what would the evaluation team recommend post-2018 in terms of strategic technical assistance and interventions to enhance the Agency’s GHSC programs? 34 Particularly the following questions to be answered as a whole: What course corrections may be needed? What focus areas/priorities are recommended for the final years? What do we know about how well the TA and innovations led to performance improvements in the supply chain? Mid-Term Evaluation Design and Methodology The midterm evaluation for the GHSC-PSM/Nepal will be a performance evaluation relying on qualitative methods, and secondary quantitative data analysis. The evaluation contractor will further refine a rigorous methodology, including a sampling strategy that will yield meaningful insights into key activity outcomes as specified by the evaluation questions. The evaluation team, in collaboration with USAID, will finalize the evaluation methodology before fieldwork begins. USAID expects that, at a minimum, the evaluation team will: ● Review and analyze existing performance information/data; ● Conduct site visits for field testing survey instruments [when applicable and feasible]; ● Meet and interview USAID project participants, partners, and host country government counterparts at appropriate levels; ● Interview USAID/Nepal staff, USAID /Washington, and a representative number of experts working in the sector. ● Upon award, the team members shall familiarize themselves with documentation about the activity and USAID’s current assistance in the public health supply chain. USAID will ensure that this documentation is available to the team before the start of the work. Data Collection Methods 1) Desk review of documents and secondary data a. Document review: A thorough review of activity documents including data analysis of the GHSC-PSM’s DevResults data and/or data in the GHSC-BI&A M&E plan and Results framework; monitoring data to identify key trends in the availability of FP and MNCH commodities; LMIS reporting and warehousing practices in Nepal. b. Review of findings from interviews and customer satisfaction survey: A thorough review of available data and findings from interaction with key informants and customer satisfaction survey and interviews of Mission staff (conducted under GH Pro 614) 2) Primary data collection a. Focus group and key informant interviews: FGDs and KIIs will be conducted with various stakeholders such as relevant stakeholders at MOHP, central stores, regional medical stores, district medical stores, representatives from EDPs/stakeholders working in supply chain b. Field observations: The evaluation team should visit and make strategic observations of selected central medical stores, regional medical stores, district medical stores, LLG stores, and HFs. The observations can be done simultaneously on warehousing practices, maintaining FEFO, physical count, and record observation for commodities stock with focus group and key informant interviews. c. The evaluation team must develop criteria for selection of selected central medical stores, regional medical stores, district medical stores, LLG stores, and HFs to ensure that analysis captures the range of TA provided by PSM. Evaluation Team 35 The evaluation team should include one Team Leader, one Procurement and Supply Chain Expert, and one M&E Specialist with the following additional responsibilities and competencies: Team Leader The team leader must be a senior-level professional with at least ten years of experience, preferably involved in similar assignments. The team leader must have a Master’s in Public Health, medicine, business administration, health sciences, pharmacy, or equivalent. S/he must have successful experience showing competency to supervise other technical experts and produce a high-quality report. Expertise and experience should include: ● Strong research and analytical skills. ● Strong knowledge of procurement and supply chain system. ● Prior experience as the team leader of similar assignmen.t ● Strong team management and supervisory skills. ● Demonstrated lead writer with strong technical English writing skills. ● Experience working with a range of government officials, donors, local NGOs, academia. Qualifications: ● Minimum of 10 years of experience in public health, with at least 5 years of experience in designing, implementing, managing, monitoring, and evaluating health programs. ● Strong evaluation, organizational, and management skills, with a minimum of 5 years of experience with evaluation tools and methods. ● Prior experience in leading evaluation teams; excellent skills in planning, facilitation, and consensus-building. ● Good familiarity with Nepal’s public health supply chain management. ● Excellent interpersonal skills, including experience successfully interacting with host country government officials, and other stakeholders. ● Strong oral and written communication skills, with extensive report writing experience. ● Experience working in Nepal or the South Asia region preferred. Procurement and Supply Chain Expert The Procurement and Supply Chain expert is a senior-level supply chain professional having demonstrated knowledge and skills in supply chain management including warehousing, forecasting, quantification, and supply planning. This expert must work under the supervision of the Team Leader and will be responsible for supporting the team leader in technical design and methodology, coordination with Management Division (MD)/LMS and partners, LMIS data analysis and report writing. Qualifications: ● Master’s degree or higher level of education in public health, pharmacy, nursing, business administration, or equivalent degree in a relevant technical area. ● At least eight years’ experience with public health supply chain projects with USAID project implementation experience preferred. ● Excellent interpersonal skills, including experience successfully interacting with host country government officials, and other stakeholders. ● Understanding of public health program implementation, monitoring, and evaluation, including design and implementation of evaluations. ● Good presentation and writing skills, including report writing experience. 36 M&E Specialist The M&E Specialist Serve as a member of the evaluation team, providing expertise and quality assurance on evaluation methods and issues. S/he should participate in planning and briefing meetings, data collection, data analysis, and development of presentations and reports. S/he must work under the supervision of the team leader and will be responsible for supporting the Team Leader in design and methodology, training and monitoring field researchers, data analysis, and report writing. Qualifications: ● Master’s degree or higher level of education in a relevant technical area. ● Experience in health program implementation and monitoring and evaluation, including design and implementation of evaluations. ● Strong qualitative evaluation and analytical skills. ● Good presentation and writing skills, including report writing experience. ● Familiarity with USAID health programs/projects. ● Familiarity with USAID M&E policies and practices, including evaluation policies, results frameworks, and performance monitoring plans. The team leader or the associated firm must recruit additional short-term statistician, data processor, data entry person, logistics support person, and field researchers to carry out data collection at relaxant HFs. Key Documents for Review ● GHSC-PSM Annual Work Plans ● GHSC-PSM CMEP ● GHSC-PSM Progress Reports (Annual, quarterly) ● Nepal Health Sector Strategy (NHSS) 2016- 2021 ● Nepal Health Sector Strategy Implementation Plan 2016-2021 ● Nepal Demographic and Health Survey (NDHS) 2016 ● Annual Report, DoHS/ MOHP ● eHealth Strategy ● eLMIS related planning, implementation and progress reports ● Nepal Health Facility Survey 2015 ● e-LMIS assessment report Key Stakeholders to be Consulted The following are suggested stakeholders for the evaluation team to consult. The team will propose their target stakeholders as part of the evaluation design. ● GON officials (MOHP, PPMED, DOHS, MD/LMS, NHTC, FWD)- 10 officials ● Provincial Director, Regional Medical Store, District Medical Stores, LLGs and HFs-3 from Province 5 & 6 only ● EDPs representatives (DFID, UNFPA, UNICEF, KfW, World Bank, GIZ and NHSSP) ● Other non-EDP NGOs ● USAID/W staff (3) ● USAID Mission (3) ● GHSC-PSM staff (3) 37 Timeline for Mid-Term Evaluation The following is a tentative timeline for the evaluation tasks; the detailed timeline will be developed during the team planning meeting and as part of finalizing the evaluation design. The evaluation is expected to start in April 2019 and be completed by early July 2019. Total duration of evaluation: April 2019 to July 2019 Key Evaluation Activities Timeframe Pre-Planning ● Meeting with the partners to plan for the Mid-Term evaluation, finalize Scope of Work and finalize core competencies and level of experience of the evaluation team members ● Contact possible Team Leader / member candidates/ firms ● Finalize the evaluation team (members and the team leader) ● Organize all documents and make them available to the evaluation team ● Hire contracted evaluation Team Leader and members from MEL project January 2019 January 2019 February 2019 March 2019 Planning ● Consult with the Ministry of Health and Population ● Review of existing reports, and project records ● Develop data collection tools ● Develop evaluation design ● In-briefing with USAID/Nepal to discuss the plan and design ● Develop and share evaluation design with stakeholders ● Identify project staff who will participate in the review process ● The evaluation team selects samples ● Arrange all logistics ● Discussion with GHSC-PSM to ensure that appropriate steps are being taken before the implementation of the mid-term evaluation April 2019 Implementation ● Introductory meeting between the evaluation team and the stakeholders (GON, partners, key project staff, EDPs) ● Briefings with USAID/Nepal: mid-term briefing to discuss desk review findings; periodic briefings as agreed upon during initial in-briefing ● Fieldwork (interviews, FGDs, observations, analyses, triangulations) ● Make a presentation to USAID/Nepal and GHSC-PSM project staff separately on the preliminary observations to validate the findings and interpretations May-June 2019 Reporting ● Data transcription, collation, and coding ● Prepare a draft report following the guideline and submit to USAID/Nepal and GHSC-PSM project staff ● Collects comments from relevant reviewers including USAID and IP July-August 2019 ● Incorporates reviewer’s feedback in the draft report ● Make a detailed presentation of the results including recommendations to GHSC-PSM project staff and USAID ● Hold a recommendation workshop that includes the presentation of findings and conclusion ● Finalize the mid-term report for wider circulation September 2019 38 Deliverables The following is a set of evaluation deliverables, with the expected timing of each adhering to the above Evaluation Timeline. 1. Evaluation Design: Following the discussions of plan and design with USAID/Nepal, the evaluation team must submit an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) Background and Project Description including Project Theory of Change; (2) Key Evaluation Questions; (3) Methodology (including data collection methods, sampling, and data analysis); (4) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (5) draft questionnaires and other data collection instruments or their main features; (6) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and/or a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology, where applicable); (7) Evaluation work plan including the anticipated schedule and logistical arrangements and a list of the members of the evaluation team, delineated by roles and responsibilities; (8) known limitations to the evaluation design; and (9) a dissemination plan. 2. Discussion of plan and design: An in-briefing with USAID/Nepal upon the hiring of Team Leader for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and/or to adjust the Statement of Work (SOW), and the evaluation design, if necessary. 3. Desk Review Analysis of data and Briefings: The evaluation team is expected to hold a mid￾term briefing with USAID after completing the desk review component on the findings of the desk review and its implications on the evaluation questions, including potential challenges and emerging opportunities. The team will also provide the evaluation COR/manager and GHSC-PSM Activity Manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in￾briefing. If desired or necessary, weekly briefings can be arranged. 4. Presentation of Initial findings to the Mission and GHSC-PSM project: At the end of the fieldwork, the evaluation team is expected to present their initial findings to USAID/Nepal and the GHSC-PSM separately. This presentation will provide an opportunity for the Mission and project to validate any factual inaccuracies and develop better ownership of the outcomes of the evaluation later. The presentation should also provide an overview of the likely recommendations that seem to be emerging based on the team’s reflection of data collection work. 5. Draft Evaluation Report: The evaluation team will submit a draft evaluation report with a table of contents that includes: Executive summary, Introduction, Evaluation Purpose and use, Project Background, Methods and Limitations, Findings, Conclusions, and Recommendations for USAID that is not more than 20 pages in length, single-spaced in TNR 12 point font, excluding the executive summary and annexes. 6. Recommendations Workshop and Presentation: Approximately two weeks after sharing the draft report (contingent on Mission and IP availability), the team will undertake a one-day joint workshop with USAID and IP with the presentation of key findings and conclusions, and brainstorming session and facilitated discussions to ground-truth conclusions and recommendations for the final report. 7. Final Evaluation Report: A final evaluation report of not more than 25 pages in length, single￾spaced in TNR 12-point font, excluding annexes, with an executive summary of not more than 3 pages 39 in length, within 10 working days of receipt of consolidated comments in electronic format from USAID. A Comments Matrix should also be included with a list of comments from USAID and responses from the evaluation team addressing each comment. All the qualitative data collected as part of this evaluation must be submitted as an annex to the final report, either in summarized format or transcripts, with PII removed. The Final approved evaluation report must be submitted to the Development Experience Clearinghouse (DEC). Mid-Term Evaluation Report Format The Evaluation Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy: https://www.usaid.gov/evaluation/policy). − The report should be 30 pages (excluding executive summary, table of contents, acronym list, and annexes). The structure of the report should follow the Evaluation Report template, including branding (https://www.usaid.gov/branding). Draft reports must be provided electronically, in English, to MEL who will then submit it to the USAID Health Office for review. For additional guidance, please see the Evaluation Reports and How-To Note on preparing Evaluation Draft Reports found at https://usaidlearninglab.org/evaluation-toolkit. USAID Criteria to Ensure the Quality of the Evaluation Report (USAID ADS 201): ● Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. ● The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. ● Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. ● Evaluation methodology should be explained in detail and sources of information properly identified. ● Limitations to the evaluation should be adequately disclosed in the report, with attention to the limitations associated with the evaluation methodology. Annex I. GHSC-PSM NEPAL RESULTS FRAMEWORK The GHSC-PSM’s results framework conveys the development hypothesis implicit in the project goal and intermediate results, as well as the cause-effect relationships between project outputs and intermediate results. Hence, the results framework provides a foundation for work planning and performance monitoring. The GHSC-PSM’s global goal is to ensure the uninterrupted supply of health commodities to prevent suffering, save lives, and create a brighter future for families across the globe. Within this global goal, the GHSC-PSM Nepal’s goal is to ensure uninterrupted supplies of health commodities in support of U.S. Government (USG)-funded public health initiatives. The results of the GHSC-PSM activity will contribute to the achievement of USAID/Nepal’s Development Objective 3: Increased human capital with Intermediate Result 3.2. Healthier and Nourished Population. The GHSC-PSM activities support Task Order 3 (TO3), Population and Reproductive Health (PRH) and Task Order 4 (TO4), Maternal and Child Health (MCH) through an array of services for health commodity procurement and related systems strengthening technical assistance that encompass different elements of a comprehensive supply chain. To attain the GHSC-PSM goal, the activity works towards the following four intermediate results: 40 Exhibit 1: GHSC-PSM Nepal Results Framework ● Intermediate Result 1: Strategic Planning. Improved strategic planning and implementation related to supply chain management and commodity security. ● Intermediate Result 2: In-Country Logistics. Improved in-country logistics, including effective and efficient delivery of health commodities to service sites. ● Intermediate Result 3: Capacity-Building. Increased capacity building efforts by implementing strategies to transfer of skills, knowledge, and technology for improved and sustained performance. ● Intermediate Result 4: Enabling Environment. Strengthened enabling environments to improve supply chain performance. Nepal Development Cooperation Strategy (DO 3): Increased Human Capital Nepal Development Cooperation Strategy/USAID Mission Intermediate Result 3.2: A healthier and well nourished population GHSC-PSM Goal: Ensured uninterrupted supply of health commodities to prevent suffering, save lives, and create a brighter future for families across the globe GHSC-PSM/Nepal Goal: To ensure uninterrupted supplies of health commodities to prevent suffering, save lives, and create a brighter future for families in Nepal Intermediate Result 1: Strategic Planning Improved strategic planning and implementation related to supply chain management and commodity security Intermediate Result 2: In-County Logistics Improved in-country logistics including effective and efficient delivery of health commodities to service sites Intermediate Result 3: Capacity Building Increased capacity building efforts by implementing strategies to transfer of skills, knowledge, and technology for improved and sustained performance Intermediate Result 4: Enabling Environment Strengthened enabling environments to improve supply chain performance Intermediate Result 5: Innovations and Research Innovations and research conducted, shared and implemented Indicator Reference Themes: In-country performance and sustainability Indicator Reference Themes: In-country sustainability; cross-cutting 41 Ensuring availability of commodities to end-users requires well-performing in-country supply chain systems from the central warehouses to the service delivery points. Core systems strengthening elements encompass the activities listed in the first column of Exhibit 2 below. Exhibit 2 also illustrates fidelity between the in-country core activities, intermediate results, and indicator reference themes. The GHSC-PSM is responsible for ensuring that the in-country program activities, including sustainability efforts, map to its results framework, by closely monitoring progress toward objectives using the in￾country performance/sustainability and cross-cutting indicators detailed within the plan. Achievements per indicator are reported in quarterly and annual performance reports. Annex 2. Mapping of GHSC-PSM’s Core Elements to Intermediate Results and Indicator Reference Themes Core Elements of Systems Strengthening Intermediate Results Indicator reference themes a. Strategic Planning: Support strategic planning and implementation related to supply chain management and commodity security. Intermediate Result 1: Improved strategic planning and implementation related to supply chain management and commodity security. In-country performance and sustainability b. In-Country Logistics: Provide technical assistance in health commodity quantification and forecasting, supply planning, procurement, warehousing, inventory management, distribution and transportation, healthcare waste management, quality assurance, product selection, identification of barriers to importation, loss prevention, recalls, supply chain design, data collection, and construction. Intermediate Result 2: Improved in-country logistic, including effective and efficient delivery of health commodities to service sites. In-country performance and sustainability c. Capacity-Building: Implement strategies to transfer health commodity supply chain management skills, knowledge, and technology to the partner country, and identifying and overcoming barriers to effective skills transfer. Intermediate Result 3: Increased capacity building efforts by implementing strategies to transfer of skills, knowledge, and technology for improved and sustained performance. In-country sustainability Cross-cutting d. Enabling Environments: Advocate for change through collaboration with key stakeholders to formulate and implement new and better policies, to allocate resources effectively, to engage and coordinate multi￾sector efforts to improve health supply chains, and to compile and present the data necessary for sound decision-making. Intermediate Result 4: Strengthened enabling environments to improve supply chain performance. In-country sustainability Cross-cutting 42 APPENDIX II: DOCUMENTS REVIEWED 1) Consolidated Annual Procurement Plan (CAPP), Fiscal Year 2017/2019, Ministry of Health, Department of Health Services, August 2017 2) Department of Health Services 2071/72 (2014/2015), Annual Report 3) Department of Health Services 2072/73 (2015/2016), Annual Report 4) Department of Health Services 2073/74 (2016/2017), Annual Report 5) The Electronic Transactions Act, 2063 BS (2006) 6) Electronic Transaction Rules 2064 BS (2007) 7) GHSC-PSM Nepal Electronic Logistics Management Information System (eLMIS) Nepal eLMIS Scale￾up Plan, 2019 8) GHSC-PSM Nepal Monitoring and Evaluation Plan, March 2019 9) GHSC-PSM Nepal Annual Report, October 1, 2016 – September 30, 2017 10) GHSC-PSM In Nepal Annual Report, October 1, 2017 – September 30, 2018 11) GHSC-PSM Nepal Work Plan Fiscal Year 2017 12) GHSC-PSM Nepal Work Plan Fiscal Year 2018 13) GHSC-PSM Nepal Work Plan Fiscal Year 2019 14) Information Communication Technology (ICT) Policy, 2072 BS 15) National Consensus Quantification of Essential Drugs, Vaccines and Program Commodities (Family Planning, MNCH, Nutrition, HIV/AIDS, TB, and Leprosy), Workshop Report, March 19-20, 2018 16) National Pipeline Review Meeting Report (Stock Status Review Report), Health and Family Planning Commodities, February 2019 17) Nepal Demographic and Health Survey, 2016 18) Nepal Fact Sheet Direct Government-to-government Support to the Ministry of Health and Population (2013-2019) 19) Nepal Health Facility Survey 2015 20) Nepal Health Sector Strategy, 2015 –2020 21) Nepal Health Sector Strategy (NHSS) Implementation Plan, 2016-2021 22) National Health Policy 2076 BS (2019) 23) National Planning Commission (2017). Nepal’s Sustainable Development Goals Status and Roadmap: 2016-2030 24) PSM eLMIS Status Reports: May 2017 – January 2019 25) Statement of Work, USAID Nepal, Global Health Supply Chain - Procurement and Supply Management 43 APPENDIX III: KEY INFORMANTS INTERVIEWED MoHP and DoHS 1) Mahendra Prasad Shrestha, Director for International Cooperation Division, MoHP 2) Guna Raj Lohani, Director General, DoHS/MoHP 3) Ramesh Prasad Adhikari, Director, Management Division, DoHS 4) Dr. Surendra Prasad Chourasia, Chief of LMIS, Management Division, DoHS 5) Sharad Sharma, Chief of IHMIS, Management Division, DoHS 6) Gyan Bahadur BC, Chief of LMIS Unit, Management Division, DoHS 7) Deepak Adhikari, Public Health Officer (Procurement Section), Management Division, MoHP 8) Rewati Thapa, Public Health Inspector, Central Medical Store 9) Madhu Pokhrel, Chief Finance Controller, DoHS, MoHP MoHP (site visit interviews) 10) Maya Acharya, Sr, ANM, Shankarnagar HP (Rupendehi) 11) Rabin Adhikari, eLMIS Support Officer, Provincial Health Office (Bardiya) 12) Ganesh Aryal, Health Coordinator, Rajapur Municipality (Bardiya) 13) Rita Bhandari, Director, Karnali Provincial Health Directorate (Surkhet) 14) Kashi Nath Banjhade, Health Coordinator, Tilottama Municipality (Rupendehi) 15) Dinesh Chapagain, Director, Provincial Health Office (Butwal) 16) Dal Bahadur Dangi, eLMIS Officer, Medical Store (Nepaljung) 17) Samjhana Dhungel, Senior ANM, Province Health Office (Pokhara) 18) Tara Prasad Gaira, Sr. AHW, Shankarnagar HP (Rupendehi) 19) Janak Raj Giri, Cold Chain Supervisor, Provincial Medical Store (Nepalgunj) 20) Buddhi Raj Kafle, Chief, Province Health Logistics Management Center (Butwal) 21) Khim Bahadur Khadka, Director, Provincial Health Office (Pokhara) 22) Tuk Maya Kunwar, Senior A.N.W., Naudadha Health Facility (Kaski) 23) Kopil Dev Mahato, Refrigerator Technician, Medical Store (Nepalgunj) 24) Alita Paudel, Pharmacist, Provincial Health Office (Banke) 25) Shiva Paudel, Storekeeper, Province Health Office (Pokhara) 26) Dilli Raj Pokharel, Store In-charge, Provincial Logistic Management Centre (Banke) 27) Devi Lal Poudel, Malaria Inspector/Storekeeper, Province Health Office (Butwal-Rupandehi) 28) Ishwori Poudel, Assistant Health Coordinator, Tilottama Municipality (Rupandehi) 29) Krishna Sapkota, CEO, Tilottama Municipality (Rupandehi) 44 30) Dr. Bhawani Prasad Sharma, Health Administrator, Provincial Health Office (Pokhara) 31) Padma Shrestha, PHN, Health Office (Bardiya) 32) Suraj Shrestha, Computer Operator, Provincial Health Office (Bardiya) USAID/Nepal 33) Umesh Gupta, Supply Chain Management Advisor 34) Belay Mengistu, (former) MNCH Team Lead 35) Nur Pant, Senior Health Technical Adviser 36) Carrie Rasmussen, Director, Health Office 37) Sabita Tuladhar, Strategic Information and Research Adviser 38) Monica Villanueva, Deputy Health Office Director USAID/W 39) Daniella Abrampah, Program Assistant, FP/RH Commodities Security Logistics Division, 40) Deborah Armbruster, Senior Maternal and Newborn Health Advisor 41) Ramy Guirguis, Senior MIS Advisor, FP/RH Commodities Security Logistics Division GHSC-PSM/Nepal 42) Balkrishna Khakurel, Policy and Government Relations Advisor 43) Ananta Lamichanne, LMIS Director 44) Mahesh Pokharel, Director Operations and Field Support 45) Raju Sapkota, Finance Officer 46) Heem Sundar Shakye, Director for HSS GHSC-PSM/Nepal Field Support Officers 47) Bhibhatsa Bhusal, Province 5 48) Gyanendra Chaudhary, Field Support Officer, Province 7 49) Indra Bahadur Gharti, Province 5 50) Shailendra Patel, Field Support Officer, Province 1 51) Madan Kumar Prajapati, Province 2 52) Alina Shrestha, Field Support Officer, Province 3 GHSC-PSM Headquarters 53) Tenzin Khando, Project Manager for Nepal 54) Wayne Lifshitz, Acting COP for PSM/Nepal / Country Programs Asia Region Director 45 Development and Technical Partners 55) Annapurna Bhattarai, Supply Officer, UNICEF 56) Manav Bhattaria, Senior Health Specialist, The World Bank/Nepal 57) Saurav Bhattarai, Deputy Chief Technical Advisor, GIZ/Nepal 58) Nichola Cadge, Health Advisor, DfID/Nepal 59) Shanker Raj Pandey, Head of KfW Office Nepal, KfW/Nepal 60) Jiblal Pokharel, Managing Director, Nepal CRS Company 61) Latika Maskey Pradhan, Assistant Representative, UNFPA/Nepal 62) Rabin Shrestha, Senior Operations Manager, Save the Children 63) Rohit Shrestha, Director for Administration and Logistics, Nepal CRS Company 64) Ramesh Kumar Sharma, Senior Public Procurement Advisor, NHSSP 46 APPENDIX IV: DATA COLLECTION INSTRUMENTS (KII GUIDES AND OBSERVATION CHECKLISTS) Consent Statement for Key Informant Interviews and Group Interviews Thank you for meeting with us today. We are from CAMRIS International, Inc., an independent firm that is conducting a study for USAID on the Procurement and Supply Management Project (PSM), which is being implemented by Chemonics International, Inc. As part of this work, we are evaluating PSM to learn more about its progress. Our team will be speaking with many individuals as part of the evaluation. We asked to meet with you because we are looking to understand your experiences with PSM and how PSM has or has not helped improve Nepal’s supply chain. You have been selected for this interview because, based on our understanding, you have interacted with PSM in a variety of settings. Your participation is voluntary, and you are entitled to decline to participate at any point before or during the discussion. For reports we write on this study, your answers will be combined with those of other people and presented in a summary format. We will share the report with USAID, and the report will be more widely disseminated after it is finalized. Your comments will not be attached to your name, so your responses will be anonymous—although, we will be taking notes (and recording your voice) so that we can remember later what you tell us. Any information you provide that might identify you will be kept confidential to the fullest extent under local law and U.S. Government policy. You should also feel comfortable asking any questions at any time during the conversation. What you tell us will be used to understand the results of PSM in this area better, and to improve the PSM program, so we hope you can be honest with us in your responses. We expect this interview will take one hour. Do you have any questions before we begin? By saying “yes,” and participating in this evaluation, you are indicating that you voluntarily consent to participate and have had an opportunity to ask any questions about your participation. Will you participate in this research study? You may answer yes or no. [Note: Consent will be obtained orally]. Yes, I am willing to participate. No, I am not willing to participate. 47 DRAFT KII/Group Interview Guide for GoN/MoHP (LMS, MD, etc.) 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. At which geographical level (national, provincial, municipality, facility) have most of the improvements been demonstrated? 4. What have been the main challenges, either internal or external, for GHSC-PSM/Nepal in ensuring the success of its activities? 5. Do you believe that the project is on the right track for achieving the desired results by the end of the project? What corrective actions, if any, need to be taken? 6. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 7. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? Please provide examples. 8. Do you believe skills and expertise have been transferred and will be sustainable? Please provide examples. 9. Has the eLMIS data been useful in the overall decision-making around supply chain management? What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of the eLMIS? Please provide examples. 10. How has the project’s support improved supply chain management performance and enhanced the capacity of the supply chain cadres within the LMS? In what technical area do you think it has had the greatest impact? What is the difference between the performance of before and after the training? 11. Has the project secured ownership of project activities from MoHP, MoSD, and municipal leadership? Please provide examples. 48 DRAFT KII/Group Interview Guide for GHSC-PSM/W and GHSC-PSM/Nepal 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. At which geographical level (national, provincial, LLG) have most of the improvements been demonstrated? Please provide examples. 4. What have been the main challenges, either internal or external, for GHSC-PSM/Nepal in ensuring the success of its activities? How has PSM mitigated those challenges? 5. Is the project on the right track for achieving the desired results by the end of the project, including: a. Reducing stock-outs at service delivery points (SDPs)? Why or why not? b. Increasing SDP reporting rates to the logistics management information system (LMIS)? Why or why not? c. Increasing the percentage of warehouses with FP and MNCH commodities managed by a first expired/first out (FEFO) system? Why or why not? d. Other areas of significant technical assistance focus? Why or why not? 6. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 7. How has the project’s support improved supply chain management performance and enhanced the capacity of the supply chain cadres within the LMS? Please provide examples. 8. Has the eLMIS data been useful in the overall decision-making around supply chain management? What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of eLMIS? 9. To what extent have stakeholders at all levels of the federal structure taken ownership of the eLMIS? 10. Has the MoHP taken ownership (i.e., budgeting) for the supply chain pharmacists funded under Red Book and, if not, what additional steps need to be undertaken to ensure this? 11. What have been the main technical contributions of GHSC-PSM/Nepal’s FSOs? Please provide examples. 12. What have been the main contributions of the Logistics Working Group (LWG) in ensuring a sustainable supply chain? Please provide examples. 49 DRAFT KII/Group Interview Guide for USAID/W, USAID/Nepal 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. At which geographical level (national, provincial, LLG) have most of the improvements been demonstrated? Please provide examples. 4. What have been the main challenges, either internal or external, for GHSC-PSM/Nepal in ensuring the success of its activities? How has PSM mitigated those challenges? 5. Do you believe that the project is on the right track for achieving the desired results by the end of the project? What corrective actions, if any, need to be taken? 6. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 7. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? To what extent has GHSC-PSM/Nepal’s technical assistance contributed to this trajectory/journey? Please provide examples. 8. How has the project’s support improved supply chain management performance and enhanced the capacity of the supply chain cadres within the LMS? Please provide examples. 9. Is GHSC-PSM/Nepal on track to achieve a greater capacity to sustain performance in the absence of external assistance once its technical assistance is finished? 10. How has PSM/Nepal worked to mobilize additional supply chain resources? Please provide examples. 11. Has the project secured ownership of project activities from MoHP, MoSD, and municipal leadership? Please provide examples. 50 DRAFT KII/Group Interview Guide for EDPs 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. What have been the main challenges, either internal or external, for GHSC-PSM/Nepal in ensuring the success of its activities? 4. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 5. How has GHSC-PSM/Nepal been able to collaborate with and build upon synergies of other supply chain projects funded by EDPs? 6. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? Please provide examples. 7. To what extent has GHSC-PSM/Nepal’s technical assistance contributed to this trajectory/journey? 8. How has the project secured ownership of project activities from MoHP and local leadership? Please provide examples. 9. Is GHSC-PSM/Nepal on track to achieve a greater capacity to sustain performance in the absence of external assistance once its technical assistance is finished? In general, what is your organization’s commitment to future supply chain assistance? 10. Has the eLMIS data been useful in the overall decision-making around supply chain management? What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of eLMIS? Please provide examples. 11. What have been the main contributions of the Logistics Working Group (LWG) to ensuring a sustainable supply chain? Please provide examples. 51 DRAFT KII/Group Interview Guide for LWG Members 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. What have been the main challenges, either internal or external, for GHSC-PSM/Nepal in ensuring the success of its activities? How has PSM mitigated those challenges? 4. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 5. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? Please provide examples. 6. To what extent has GHSC-PSM/Nepal’s technical assistance contributed to this trajectory/journey? Please provide examples. 7. Is GHSC-PSM/Nepal on track to achieve a greater capacity to sustain performance in the absence of external assistance once its technical assistance is finished? 8. Has the eLMIS data been useful in the overall decision-making around supply chain management? Please provide examples. 9. What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of eLMIS? 10. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? Please provide examples. 11. What have been the main contributions of the Logistics Working Group (LWG) in ensuring a sustainable supply chain? Please provide examples. 52 DRAFT KII/Group Interview Guide for IPs (e.g., CRS and NHSSP staff) 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. At which geographical level (national, provincial, LLG) have most of the improvements been demonstrated? Please provide examples. 4. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? 5. How has GHSC-PSM/Nepal been able to collaborate with and build upon synergies of the activities which are done as part of your work? Please provide examples. 6. How has the project support improved supply chain management performance and enhanced the capacity of the supply chain cadres within the LMS? Please provide examples. 7. Is GHSC-PSM/Nepal on track to achieve a greater capacity to sustain performance in the absence of external assistance once its technical assistance is finished? 8. Has the eLMIS data been useful in the overall decision-making around supply chain management? What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of eLMIS? Please provide examples. 9. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? 10. What have been the main technical contributions of GHSC-PSM/Nepal’s FSOs? Please provide examples. 11. What have been the main contributions of the Logistics Working Group (LWG) in ensuring a sustainable supply chain? Please provide examples. 53 DRAFT KII/Group Interview Guide for Provincial/Local Level Counterparts (Health officers, forecasters, medical store/warehouse and eLMIS staff) 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, and other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. Do you believe by its finish date of April 2021 that the project will have achieved the following: a. Reduced stock-outs at service delivery points (SDPs)? Why or why not? b. Increased SDP reporting rates to the logistics management information system (LMIS)? Why or why not? c. Increased the percentage of warehouses with FP and MNCH commodities managed by a first expired/first out (FEFO) system? Why or why not? d. Other areas of significant technical assistance focus? 4. What corrective actions, if any, need to be taken? 5. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 6. How has the project secured ownership of project activities from local leadership? Please provide examples. 7. Has the eLMIS data been useful in the overall decision-making around supply chain management? Please provide examples. 8. What are the challenges in the overall operation and management of the eLMIS? 9. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? Please provide examples. 10. Have good warehousing practices been institutionalized and, if not, what additional steps need to be undertaken to ensure their sustainability? 11. What have been the main technical contributions of GHSC-PSM/Nepal’s FSOs? Please provide examples. 54 DRAFT KII/Group Interview Guide for FSOs 1. What do you believe have been the main contributions of GHSC-PSM/Nepal in the areas of supply chain technical assistance? Please provide examples. (If needed, prompts include strategic planning, in-country logistics, eLMIS, capacity development, other.) 2. Have those contributions been only limited to the areas of RH/FP and MNCH and, if not, what other health areas have benefited from the technical assistance? Please provide examples. 3. Is the project on the right track for achieving the desired results by the end of the project including: a. Reduced stock-outs at service delivery points (SDPs)? Why or why not? b. Increased SDP reporting rates to the logistics management information system (LMIS)? Why or why not? c. Increased the percentage of warehouses with FP and MNCH commodities managed by a first expired/first out (FEFO) system? Why or why not? d. Other areas of significant technical assistance focus? 4. What corrective actions, if any, need to be taken? 5. Are there any best practices or success stories that have emerged during PSM’s implementation? Please provide examples. 6. How has the project secured ownership of project activities by local leadership? Please provide examples. 7. Has the eLMIS data been useful in the overall decision-making around supply chain management? Please provide examples. 8. What are the challenges in the overall operation and management of the eLMIS? How might these challenges affect the scale-up of eLMIS? 9. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? Please provide examples. 10. Have good warehousing practices been institutionalized and, if not, what additional steps need to be undertaken to ensure their sustainability? 11. What do you think have been your main technical contributions to improving the supply chain in Nepal? 55 DRAFT KII/Group Interview Guide for Supply Chain Pharmacists 1. What have been the main challenges either internal or external for GHSC-PSM/Nepal in ensuring the success of its activities? How has PSM mitigated those challenges? 2. How has the supply chain in Nepal through the GHSC-PSM/Nepal interventions advanced to greater self-reliance? Please provide examples. 3. Has the project secured ownership of project activities from MoHP and local leadership? Please provide examples. 4. How has technical assistance for forecasting improved the ability of local counterparts to plan for and ensure a secure supply for FP and MNCH commodities? Please provide examples. 5. Has the MoHP taken ownership for your position as a supply chain pharmacist and, if not, what additional steps needs to be undertaken to ensure this? 6. What have been your main technical contributions to improving the supply chain in Nepal? 56 Observation Checklist for eLMIS at CMS/Provincial Level Evaluation Team, Global Health Supply Chain – Procurement and Supply Management (GHSC – PSM) USAID/Nepal has asked CAMRIS’s Monitoring, Evaluation, and Learning Activity to conduct a mid-term performance evaluation of USAID’s Procurement and Supply Management (PSM) Project. The purpose of this evaluation is to understand better whether GHSC-PSM/Nepal is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality technical assistance to the Ministry of Health and Population (MoHP) in achieving the national supply chain objectives. As such, relevant eLMIS users will be interviewed about their role in the implementation of eLMIS at CMS/PMS/LLGs. Of particular importance is the perspective from persons working in the field; thus, we would greatly appreciate your feedback. Please be assured that any information or examples gathered as part of this survey will be confidential and not attributed to any specific person. Evaluation Team Member: Site/Facility Observed: Date of Visit: Name of Person(s) Contacted Position/Title Questions Explain in Detail 1. Is eLMIS implemented at your administrative/geographic level? Yes No 1a. If yes, have you received training from PSM on eLMIS? Yes No “Yes,” which interface (mobile, online, offline) was used for the training? 57 Questions Explain in Detail 2. How do you rate your level of satisfaction with the training? 1= Dissatisfied 2 = Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied 2a. If you rated your level of satisfaction as either “Dissatisfied” or “Somewhat Dissatisfied,” how could the training be improved? Comments: 3. Please describe your experience using the eLMIS system. Comments: 4. How do you mainly use the eLMIS data? Comments: 5. Have you used the eLMIS data for decision￾making? Yes No If yes, please provide some examples. 6. Does the eLMIS software address the needs for logistics management? Yes No If no, what modifications are needed with the software? 7. Do you use the data produced by eLMIS for the activities as mentioned below? a. Forecasting and quantification Yes No Comments: 58 Questions Explain in Detail b. Procurement Yes No Comments: c. Supply planning Yes No Comments: d. Inventory management (including warehousing) Yes No Comments: e. Budgeting and financing Yes No Comments: 8. How frequently do you experience problems in using the eLMIS system? Daily Weekly Monthly Never Other (please specify): 9. How responsive is the eLMIS support system at the centre when you encounter a problem with the eLMIS? Immediate Within a day Within a Week Within a month Other - Specify Level 1: Help desk Level 2: LMIS Officer Level 3: Engineer 10. To what extent are you satisfied with the eLMIS system? (Tick in the right column) 1. Software Support Services Level of Satisfaction a. Level 1: Help desk 1= Dissatisfied 2 = Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied If dissatisfied or somewhat dissatisfied, please explain. b. Level 2: LMIS Officer 1= Dissatisfied 2 – Somewhat dissatisfied If dissatisfied or somewhat dissatisfied, please explain. 59 Questions Explain in Detail 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied c. Level 3: Engineer 1= Dissatisfied 2 – Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied If dissatisfied or somewhat dissatisfied, please explain. 2. Data Migration 1= Dissatisfied 2 – Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied If dissatisfied or somewhat dissatisfied, please explain. 3. Output Level of Satisfaction a. Output tables/reports 1= Dissatisfied 2 – Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied If dissatisfied or somewhat dissatisfied, please explain. b. Graphs 1= Dissatisfied 2 – Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied If dissatisfied or somewhat dissatisfied, please explain. 60 Questions Explain in Detail 5= Very satisfied c. Dashboard 1= Dissatisfied 2 – Somewhat dissatisfied 3 = Neutral 4 = Somewhat satisfied 5= Very satisfied If dissatisfied or somewhat dissatisfied, please explain. 11. Do you think the eLMIS has improved the quality of logistics data? If so, how? Yes No If no, please provide recommendations for improvement. 12. Is there a data verification system in place? Yes No If yes, please describe the mechanisms to verify the data. 12.a. If a data verification system is in place, how often is the data verified? Frequency of verification: 13. Can the software be improved to address your needs? If yes, how? Yes No If yes, please provide additional information. 14. What were the main challenges faced during the implementation of eLMIS software? 61 15. What do you believe has been the main contribution (if any) of eLMIS activities in improving the supply chain and its management? 16. If additional sites were to implement the eLMIS, what do you believe are the main lessons learned that could benefit those new sites? Thank you for your participation! 62 Observation Checklist for Medical Stores/Warehouse (Central, Provincial, District) Evaluation Team, PSM-GHSC Global Health Supply Chain USAID/Nepal has asked CAMRIS’s Monitoring, Evaluation, and Learning (MEL) Project to conduct a mid￾term performance evaluation of USAID/Nepal’s Procurement and Supply Management (PSM) Project. The purpose of this evaluation is to understand better whether GHSC-PSM/Nepal is on track to meeting its stated objectives and results, and the activity’s capacity to provide quality technical assistance to the Ministry of Health and Population (MoHP) in achieving the national supply chain objectives. Of importance is the perspective of persons working in the field; thus, we would greatly appreciate your feedback. Please be assured that any information or examples gathered as part of this survey will be confidential and not attributed to any specific person. Evaluation Team Member: Site/Facility Observed (Province/district/LLG and name or identification number of the facility) Date of Site Visit: Name of Person(s) Contacted Position/Title Questions and Themes Comments/Explanation Planning and Forecasting 1. Is there a logistics working group (LWG) at your level? Yes No Don’t know 1.a. If yes, do they meet regularly? Yes No 1.b. If the LWG meets regularly, how often do they meet? Circle one: Weekly If “Other,” please provide additional detail. 63 Questions and Themes Comments/Explanation Monthly Quarterly Bi-annually Annually Other 2. Are you aware of the criteria for forecasting required drugs? Yes No 2a. If yes, what are the criteria for forecasting required drugs? (For example, is it based on guidelines? Please verify by asking to see the documents.) Criteria for Basis of Forecasting: Means of Verification: 3. Are stakeholders involved in forecasting at each geographic/ administrative level? Yes No Don’t know 3.a. If stakeholders are involved, please describe which stakeholders participate in the forecasting. 4. Is there a consolidated procurement plan at the provincial level? Yes No Don’t know Please verify that the plan is available. Procurement 64 Questions and Themes Comments/Explanation 1. How is technical assistance being provided for procurement at your administrative level? Please explain. Comments: 2. Is there a procurement committee for your administrative level? Yes No Don’t know 3. Have you done procurement as per the most recent Consolidated Procurement Plan? (Verify that the document exists.) Yes No Don’t know Verification: 3.a. If no, what are the challenges? (Please explain.) Examples of challenges: o Technical o Financial o Legal o Structural o Other (e.g., transport/venders) Comments (Explain the challenges): Capacity Development 1. Have you/your staff received training from PSM on logistics management and/or eLMIS/? Yes No Don’t know 65 Questions and Themes Comments/Explanation 1.a. If any of your staff have received training on eLMIS, please estimate how many. Number of staff who have received eLMIS training: 2. Has the training contributed to improved performance? Yes No Don’t know If yes, please provide some examples of improved performance. Expansion of eLMIS 1. Does the eLMIS software address the needs of logistics management? Yes No Don’t know Comments: 2. How is technical assistance being provided when you encounter a problem with the eLMIS? Comments: 3. Can the software be improved to address your needs? If yes, how? Yes No Don’t know If yes, please provide additional information: 4. Are you using the data for decision￾making? (Please explain) Don’t know If yes, please provide additional detail: 66 Questions and Themes Comments/Explanation 5. Do you think eLMIS has improved the quality of logistics data? Yes No Don’t know If no, please provide recommendations for improvement. 6. Is a data verification system in place? Yes No Don’t know If yes, please describe the mechanisms to verify the data. 6a. If a data verification system is in place, how often is the data verified? Frequency of verification: Monitoring and Evaluation 1. What is the process for monitoring the stock of the health facility (or facilities) under your jurisdiction? Comments: 2. How often do you monitor the stock of key commodities? Circle one: Never Monthly Quarterly Bi-annually Annually Other If “Other,” please provide additional Detail. 67 Questions and Themes Comments/Explanation 3. Are action plans with recommendations developed after monitoring visits? Yes No Don’t know If yes, please describe how the implementation of recommendations is verified. Warehouse Management Practices 1. Verify that the following warehouse practices are being implemented: a. Are supplies stored in a dry, well-lit, well-ventilated storeroom out of direct sunlight? Yes No Comments: b. Are there any visible signs of water coming into the storage area? Yes No Comments: c. Is the fire safety equipment available and accessible? Yes No Comments: d. Are cartons stacked on the pallets, away from the walls and other stacks, and not more than 2.5 m (8 ft.) high? Yes No Comments: e. Are medical supplies stored away from insecticides, chemicals, old files, office supplies, and other materials? Yes No Comments: 68 Questions and Themes Comments/Explanation f. Are store supplies in a manner accessible for counting and general management? Yes No Comments: g. Is there a process for the disposal of damaged or expired products immediately? (Please verify the process) Yes No Comments: Means of verification: h. What is the current temperature in the storage area? (Verify by looking at storage facility thermostat) Current temperature: Verification of the presence of thermostat within storage area (Circle one): 1 = Yes, there is a thermostat, and it is working 2 = Yes, there is a thermostat, and it is currently not working 3 = There is no thermostat/means for monitoring and/or regulating the temperature Comments: What were the main challenges faced during the implementation of the supply chain management activities under the support of the PSM project? 69 What do you believe has been the main contribution of PSM’s activities in improving the supply chain and its management? Do you have any suggestion to improve the project? Thank you for your participation! 70 APPENDIX V: SITE VISIT PICTURES On-going construction at the “model” warehouse at Pathlaiya. 71 APPENDIX VI: DISCLOSURE OF CONFLICTS OF INTEREST Name Timothy Allen Clark Title Consultant Organization Self-employed Evaluation Position?  Team Leader ☐ Team member Evaluation Award (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number{s), if applicable) GHSC-PSM Nepal I have real or potential conflicts of interest to disclose. ☐Yes  No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is on employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) ore being evaluated. 2. Financial interest that is direct; or is significant though indirect in the implementing organization{s) whose projects ore being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAJD operating unit managing the evaluation or the implementing organization(s) whose project{s} are being evaluated. 5. Current or previous work experience with an organization that may be seen as on industry competitor with the implementing organization(s) whose project(s} are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date