HCSM EOP Evaluation, March 2016 Page 1 Final Version (07 April 2016) April 2016 This report was prepared by the HCSM End of Project Consulting Team for review by the United States Agency for International Development (USAID). END OF PROJECT EVALUATION OF THE MANAGEMENT SCIENCES FOR HEALTH (MSH)/HEALTH COMMODITIES AND SERVICES MANAGEMENT (HCSM) PROJECT HCSM EOP Evaluation, March 2016 Page 2 END OF PROJECT EVALUATION OF THE MANAGEMENT SCIENCES FOR HEALTH (MSH)/HEALTH COMMODITIES AND SERVICES MANAGEMENT (HCSM) PROJECT Cover: St Elizabeth Chiga Health Centre, Kisumu - Pharmacy This publication was produced for the United States Agency for International Development. It was prepared by William Kiarie, Nelson Gitonga, Nelly Rangara and Charles Kandie Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements The evaluation team would like to thank the many respondents at both the national and county level who participated in this evaluation through focus group discussions or as key informants. All the participants notwithstanding their busy schedules found time to participate in this evaluation. They offered candid and informed views about HCSM and offered suggestions for future program design. We are greatly Indebted to USAID Kenya office for their input at all stages of this evaluation. In particular, we acknowledge the support and guidance of Alice Micheni, the project’s Agreement Officer’s Representative. Finally our special gratitude goes to MSH staff led by John Chimumbwa who provided us with all the project information and documents we required and additionally efficiently organized the logistics that supported our travel, office space and obtaining appointments with evaluation respondents. HCSM EOP Evaluation, March 2016 Page 3 Table of Contents Acronyms and Abbreviations ............................................................................................................ 5 EXECUTIVE SUMMARY................................................................................................................... 6 1. INTRODUCTION.................................................................................................................. 11 2. BACKGROUND.................................................................................................................... 11 2.1 Country and Portfolio Context and Landscape ................................................................... 11 2.2 Project Background.......................................................................................................... 11 3. EVALUATION DESIGN AND METHODOLOGY ..................................................................... 12 3.1 Evaluation Objectives and Guiding Questions..................................................................... 12 3.2 Proposed Evaluation Approaches ...................................................................................... 13 3.3 Method of Selecting Counties, Facilities and Study Participants............................................. 13 3.4 Selected Counties and Health Facilities.............................................................................. 14 3.5 Key Informant Interviews and FGDs Conducted ................................................................. 14 3.6 Data Collection Tools...................................................................................................... 14 3.7 Data Analysis .................................................................................................................. 15 3.8 Ethical Considerations...................................................................................................... 15 3.9 Evaluation Limitations ..................................................................................................... 15 3.10 Evaluation Team............................................................................................................. 16 4. FINDINGS............................................................................................................................. 16 4.1 Strengthening Commodity Systems at National Level .......................................................... 16 4.2 Strengthening of Health Commodity Management at County Level....................................... 19 4.3 Supporting Improved Coordination and Harmonization in Commodity Management ............. 21 4.4 Improving Stewardship and Leadership at Sub-County Level ................................................ 22 4.5 Trends in Availability and Reporting Rates for Health Commodities...................................... 23 4.6 Pharmaceutical Sub-Sector Governance and Service Delivery............................................... 23 4.7 Rational Drug Use, Quality Assurance and Patient Safety ..................................................... 24 4.8 Laboratory Commodities Supply Chain.............................................................................. 26 4.9 Ownership and Sustainability ............................................................................................ 27 4.10 Best Practices and Innovations.......................................................................................... 28 4.11 HCSM Implementation Challenges and Solutions ............................................................... 29 4.12 Commodity Management and Pharmaceutical Systems Gaps ................................................ 30 4.13 Other Findings ................................................................................................................ 31 5. Conclusions........................................................................................................................... 34 6. Lessons with Potential for Scale-up and informing Future Direction ............................................. 36 7. Recommendations and Future Direction ................................................................................... 36 HCSM EOP Evaluation, March 2016 Page 4 8. Annexes ................................................................................................................................ 38 Annex 1: Guiding Evaluation Questions........................................................................................ 38 Annex 2: List of documents to be reviewed .................................................................................. 39 Annex 3: Key Informants Questionnaire....................................................................................... 40 Annex 4: County Commodity Security Technical Working Group FGD Tool................................... 44 Annex 5: HCSM Training Recipients FGD Tool............................................................................. 46 Annex 6: USAID Respondents Questionnaire ............................................................................... 48 Annex 7: List of Evaluation Respondents....................................................................................... 49 Annex 8: HCSM Performance against Key Indicators ..................................................................... 56 Annex 9: Availability of Commodities and Commodity Management tools by Facility Level ................ 64 HCSM EOP Evaluation, March 2016 Page 5 Acronyms and Abbreviations ADR Adverse drug reaction ADT Antiretroviral Dispensing Tool AIDS Acquired Immune Deficiency Syndrome AL Artemether and Lumefantrine APHIA Plus AIDS Population and Health Integrated Assistance Plus ART Antiretroviral therapy CHMT County Health Management Team CHRIO County Health Records Information Officer DHIS District Health Information Software EDITT Electronic Dispensing and Inventory Tracking Tool EMMS Essential medicines and medical supplies EOP End of Project FP Family Planning GOK Government of Kenya HCSM Health Commodities and Services Management KDHS Kenya Demographic Health Survey KEML Kenya Essential Medicines List KEMLCL Kenya Essential Medical Laboratory Commodity List KEMSL Kenya Essential Medical Supplies List KEMSA Kenya Medical Supplies Authority KISM Kenya Institute of Supplies Management KMLTTB Kenya Medical Laboratory Technicians and Technologists Board KMTC Kenya Medical Training College LLIN long-lasting insecticidal net LMIS Logistics Management Information System M&E Monitoring and evaluation MOH Ministry of Health MTC Medicines and Therapeutics Committee NASCOP National AIDS and STI Control Program NMCP National Malaria Control Program NMS National Malaria Strategy NPHLS National Public Health Laboratory Services OJT On-job-training PEPFAR Presidents Emergency Plan for AIDS Relief PHP Priority Health Program PMS Post-Marketing Surveillance PMT Pipeline Management Tracker PPB Pharmacy and Poisons Board PQMP Poor-quality medicine products PSU Pharmaceutical Services Unit PV Pharmacovigilance QOC Quality of care RMHSU Reproductive and Maternal Health Services Unit RTK Rapid test kit TB T uberculosis TOT Training of trainers TWG T echnical working group UNFPA United Nations Population Fund UON University of Nairobi USAID United States Agency for International Development USG United States Government EXECUTIVE SUMMARY 1. Introduction and Background HCSM is a five-year project (April 1, 2011 to March 31, 2016) funded by USAID and implemented by MSH. The project’s overall goal is to strengthen commodity security and pharmaceutical services with specific focus on the priorities of its three funding streams - PEPFAR (HIV & AIDS), PMI (Malaria) and POP (FP/RH). The project initially adopted a district-focused approach in reaching peripheral health facilities. But after the 2013 General Elections in Kenya and with the devolution of health service delivery to counties, the project revised its coverage/scope of support to focus activities in 13 counties out of the total 47 counties. The 13 counties selected were identified as high need with regard to the existing USAID funding streams/programs, namely PEPFAR, PMI and POP. In 2014/15 financial year, the project added two more counties - Uasin Gichu and Elgeyo Marakwet mainly focusing on strengthening reproductive health commodity management. The project had three strategic Objectives – Strengthening commodity management, strengthening pharmaceutical services and strengthening laboratory supply chain. 2. Evaluation Design and Methodology The evaluation had four strategic objectives including - assess progress towards achievement of the project goal, objectives and IRs; identify accomplishments and challenges encountered during the project’s implementation; assess the rationale for geographical focus; and make recommendations and develop a concept paper on possible prospective programmatic scope for future technical assistance support based on these new realities. The assessment was guided by 13 evaluation questions. The methodology employed a comprehensive desk review, focus group discussions of training recipients and members of the county commodity security TWGs and national and county level key informants. The field work covered 9 counties, 7 of which were HCSM focus counties while two had received limited support. The focus counties visited were Migori, Kisumu, Homa Bay, Siaya, Elgeyo Marakwet, Kakamega and Kilifi. In the focus counties, the evaluation interviewed members of the County Commodity Security TWG, and health managers of the county hospital and one other health facility. The other two counties visited were Machakos and Kiambu. A total of 177 respondents were interviewed. Data obtained from key informants’ interviews and FGDs was organized into key themes in line with the evaluation questions. The evaluation was carried out by four Kenyan consultants including two public health experts, one national level public pharmaceutical expert and one county level pharmaceutical expert. 3. Findings Below is a summary of Key findings organized by the evaluation questions: 3.1 Strengthening Commodity Management at the National Level Working with the Pharmaceutical Services Unit (PSU) of MoH, HCSM supported the review and dissemination of key policy guidelines to guide commodity management. These documents included supportive supervision guidelines, forecasting and quantification guidelines, Kenya Essential Medicines List (KEML), Kenya Essential Medical Supplies List (KEMSL), and Essential Medicines and Medical supplies (EMMS) guidelines. PSU has not done any follow-up to monitor or evaluate how and if the guidelines are being used at the county and facility level. HCSM supported the University of Nairobi’s School of Pharmacy to develop and mount a 2-year, masters course in Pharmacoepidemiology and Pharmacovigilance. Since its inception the program has graduated a HSCM EOP Evaluation Report, March 2016 Page 7 total of 17 students. HCSM has worked very closely with Priority Health Programs (PHPs) – HIV, Malaria and FP to strengthen commodity management. Key successes include establishment of commodity security TWGs, conducting regular forecasting and quantification (F&Q) exercises, conducting commodity pipeline monitoring and development of training packages. F&Q data has been used by all the three PHPs to mobilize resources and reduce commodity funding gaps. Other successes with PHPs include improvement of commodity reporting rates and development of electronic commodity management tools. 3.2 Commodity Management at the County Level HCSM has built the capacity of counties to address health commodity management issues by establishing functional commodity security TWGs in all the 15 counties they support and by supporting commodity management capacity building. A key success has been conducting regular F&Q exercises to determine commodity requirements and Lobby county assembly members for increased commodity budgets. HCSM has also helped introduce electronic commodity management tools such as ADT and EDITT at the county level. HCSM established 39 model sites with the aim of using these sites as learning centres to help other health facilities improve commodity management practices. However, the evaluation team found that the model site concept is not working well and awareness about these sites is very low. 3.3 Improved Coordination and Harmonization in Commodity Management The commodity security TWG has been the main mechanism of supporting coordination and harmonization of commodity management activities. However, there is minimal involvement of non-state actors especially FBOs and the private sector in the TWGs. Coordination and harmonization has also been strengthened through regular forums that bring together staff involved in commodity management from national and county levels. The evaluation team did not find any formal framework for ensuring and monitoring coordination and harmonisation between various USG and non-USG partners. Linkages between the national MOH and county MoH are extremely weak and relations between the two are characterized by mistrust and hostility. 3.4 Stewardship and Leadership at Sub-National Level This was mainly done through the county commodity security TWGs as reported in Section 3.2 above 3.5 Commodity Availability and Reporting Strengthened commodity management at national and county level has led to improved commodity availability mainly as a result of F&Q, mobilization of additional funding for commodities, pipeline monitoring and development of monthly commodity status 2-pagers. The reporting rates for the key health commodities have been on an upward trajectory. HCSM has rolled out malaria, FP and nutrition commodity reporting on the DHIS2 platform. Over the period of the project, FP commodity reporting rates increased from 24% to 83% nationally (91% HCSM target counties), HIV Nutrition commodity reporting improved from 0% to over 58% nationally (68% HCSM target counties) while Malaria commodity reporting rates improved from about 40% to 70% nationally (93% HCSM target counties). 3.6 Pharmaceutical Sub-Sector Governance and Service Delivery With HCSM support, PSU developed a number of policy documents to support governance and service delivery including KEML and Kenya National Pharmaceutical Policy (KNPP). HCSM also helped strengthen pharmacy professional associations through support for their strategic plan HSCM EOP Evaluation Report, March 2016 Page 8 development and implementation. Other areas of strengthening pharmaceutical service delivery included strengthening commodity management supportive supervision and conducting regular quality of care surveys for the national malaria program. 3.7 Rational Drug Use, Quality Assurance and Patient Safety HCSM supported the Pharmacy and Poisons Board (PPB) to strengthen pharmacovigilance (PV) and post marketing surveillance (PMS) programs. In addition, PPB produces a monthly PV 2-pager and a regular health information newsletter targeting health workers. Through HCSM support, Kenya now has the 4th highest ADR reporting rates in Africa and PPB has been appointed a regional centre of excellence in pharmacovigilance by NEPAD. PPB has taken a number of regulatory decisions such as product recalls based on PV and PMS findings. HCSM has supported the establishment of Medicines and Therapeutic Committees (MTCs) at national, county and facility level. However, the evaluation team found that most of the MTCs are not fully operational. 3.8 Laboratory Commodity Supply Chain Laboratory programs have been fully incorporated into the commodity security TWGs and forecasting and quantification (F&Q) activities at both the PHPs and county level. The project has also rolled out a commodity management capacity building program targeting laboratory staff at all levels and has supported improved lab commodity reporting. 3.9 Ownership and Sustainability The PHPs were of the view that the work they had done with HCSM – TWGs, quantification, pipeline monitoring and PV was largely sustainable but may require short to medium term TA (up to three years) and some budgetary support. However most county respondents felt that the journey towards strengthening capacity of counties to better manage pharmaceutical commodities and services had just begun and there was need for more concerted and well-coordinated support (technical and financial) to fill in existing capacity gaps, consolidate skills and competencies and bring their capacity to a sustainable level. 3.10 Best Practices and Innovations The project has supported several interventions that are innovative and constitute best practice. Examples include the use of F&Q data by Kilifi county TWG to lobby their county government to enhance their health commodity budget leading to the budget increasing from Ksh 204 million to Ksh 369 million. Another best practice is the improvements in defaulter tracing following the adoption of Antiretroviral Dispensing Tool (ADT) by Kodiaga Prison health centre in Kisumu County. 3.11 HCSM Implementation Challenges HCSM faced a number of implementation challenges. These included the very broad geographic scope during the first two years and then the significant mid-stream re-focusing of the project’s scope. Devolution was accompanied by significant changes in staffing at both national and county levels that affected program continuity. Other challenges included competing tasks especially with county MoH staff and weak partner coordination especially at the county level. 3.12 Management and Pharmaceutical Systems Gaps A number of commodity management and pharmaceutical systems gaps remain. These include frequent shortages of commodities especially those that are not funded by partners. Other gaps include high reliance on manual commodity management processes, existence of multiple HSCM EOP Evaluation Report, March 2016 Page 9 commodity management and reporting platforms, inadequate engagement and involvement of the private sector and limited use of integrated approaches in commodity management improvement interventions. Commodities storage especially at the county level is inadequate and poses risks to the integrity/quality of the stored commodities. There is limited or no budgets for many commodity management functions such as printing of reporting tools, capacity building and supportive supervision. 3.13 Other Findings Counties that had not received continuous support from HCSM had weaker commodity management practices. They had not carried out F&Q and data quality reviews and had less dynamic TWGs. Based on desk review findings and respondent views there was a strong geographic targeting rationale for the project’s HIV and Malaria components as the focus counties had disproportionately higher disease burden. However, the rationale was weak for FP commodities as FP needs do not exhibit a similar geographic polarity that is seen with HIV and Malaria. Respondents reported that HCSM technical staff were committed and technically proficient. However, some respondents pointed out that HCSM had lost some key staff and the replacements did not have the same level of experience and expertise. 4. Conclusions The evaluation team is of the view that HCSM has done quite well and achieved most of the set targets at a time of rapid country governance and policy change and in an unsettled health service delivery environment coupled with significant midstream refocusing of its own mandate. Notable successes were the establishment of commodity security TWGs at national and county level, support for F&Q, national level PV and PMS, improved commodity reporting rates and support for electronic commodity management tools. Some of the areas where the program was less successful include establishment of facility MTCs, the model site concept and introduction of a mobile app for accessing treatment guidelines. Involvement of the private sector was also patchy. 5. Recommendations The recommendations formulated are intended to guide the design of a future commodity management support program. These recommendations are summarized below: Finding Recommendations 1. Many commodity management processes are manual. Further, where electronic data management systems exist, multiple platforms are in use.  Enhance the use of integrated electronic inventory and dispensing tools at all levels of health service delivery and adopt common ICT platforms. 2. Multiple commodities reporting systems are currently in use and some are not accessible to most health and program managers  All commodity reporting should be migrated and integrated to DHIS2 3. MTCs and patient safety interventions have not taken root at county and facility level  Set up/reactivate facility MTC's 4. The impact of HCSM has been much less at sub-county and lower levels  Develop a mechanism to ensure that commodity management interventions reach the sub-county and lower levels including primary health facilities 5. The linkages between national programs and county MoH are extremely weak. In  Support the Intergovernmental Committee on Health and its respective TWG's to HSCM EOP Evaluation Report, March 2016 Page 10 Finding Recommendations addition, the PHPs still manage most of their functions centrally improve harmonization and coordination  Support greater devolution of PHPs’ commodity management functions 6. Existing information products on health commodities focus primarily on health workers and not consumers  Strengthen consumer education and awareness and introduce consumer surveys and consumer reporting. 7. Many commodity management interventions and improvements especially at the national level remain largely vertical  Design greater integration and adopt more health system approach in future health commodity strengthening programs 8. GOK financial contribution to procurement of health commodities such as FP, HIV and Malaria remains low. In addition, many commodity management functions such as printing of reporting tools and capacity building are not budgeted for at both national and county level  Advocate for increased GOK financial support at national and county level to support health commodities procurement and management 9. The Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) play a critical role in setting standards and building the capacity of commodity management practices in Kenya  Explore engagement of the Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) in strengthening health commodity procurement and inventory management. HSCM EOP Evaluation Report, March 2016 Page 11 1. INTRODUCTION The Health Commodities and Services Management (HCSM) is a five-year project (April 1, 2011 to March 31, 2016) funded by United States Agency for International Development (USAID) and implemented by Management Sciences for Health (MSH). The project works to improve the health of Kenyans by increasing access to and use of quality and safe essential health products and services. The project’s overall goal is to strengthen commodity security and pharmaceutical services across the health system with specific focus on the priorities of its three funding streams - PEPFAR (HIV & AIDS), PMI (Malaria) and POP (FP/RH). This evaluation process began on January 18, 2016. The evaluation assessed the HCSM project‘s retrospective performance and effectiveness in achieving its goal and objectives from April 2011 to December 2015. The evaluation findings and recommendations are expected to inform USAID’s future programming in this area. 2. BACKGROUND 2.1 Country and Portfolio Context and Landscape In March 2013, Kenya moved to a devolved system of government that transitioned fiscal, planning, and oversight autonomy from central authorities to 47 new counties. National government had the remaining function of setting policy, standards and norms to guide service delivery. Midway into the life of this project, OGAC provided guidance that led to a shift in emphasis on HIV/AIDS treatment vis-a-vis other elements of HIV care, prevention and health systems strengthening. In addition, the guidance provided for shifts in geographical coverage for PEPFAR funded mechanisms to areas with high disease burden. 2.2 Project Background In April 2011, USAID awarded the HCSM project to Management Sciences for Health as an Associate Award under the Strengthening Pharmaceutical Systems Project. HCSM was designed as part of USAID/Kenya national level health systems strengthening support programs but was also required to reach regional areas directly or through collaborations and linkages with regional service delivery programs and other stakeholders. It was to specifically focus on strengthening pharmaceutical policy and services, laboratory systems and commodity management within the Kenya Ministry of Health (MOH) and peripheral facilities. From inception in April 2011 to around mid-2013, the project implemented activities in the former 8 provinces of the Country having adopted a district-focused approach in reaching peripheral health facilities. However, after the 2013 General Elections in Kenya and with the devolution of health service delivery to counties, the project in consultation with USAID revised its coverage / scope of support to focus activities in 13 selected counties out of 47. The 13 counties selected were identifies as having highest need with regard to the existing USAID funding streams/programs, namely PEPFAR, PMI and POP. In 2014/15 financial year, the project added two more counties - Uasin Gishu and Elgeyo Marakwet mainly focusing on strengthening reproductive health commodity management. Towards the end of 2015, the project enrolled 3 more counties - Kiambu, Tharaka Nithi and Machakos but program implementation had not fully taken off in these counties at the time of the EOP evaluation. HSCM EOP Evaluation Report, March 2016 Page 12 2.3 Project goals, Strategic objectives and Intermediate Results The stated goal of the HCSM project is to strengthen commodity security and pharmaceutical services which is to be achieved through three strategic objectives each having related intermediate result areas as shown below: 3. EVALUATION DESIGN AND METHODOLOGY 3.1 Evaluation Objectives and Guiding Questions The overarching objectives of the evaluation were:  Assess progress made towards achievement of the project goal, objectives and intermediate results (IRs), in the context of agreed upon annual work plans and available budgets.  Identify and put into context the accomplishments and challenges encountered during the implementation of the project and draw conclusion on the project’s overall performance and effectiveness  Assess the rationale and success of geographical focus and re-focus scope and focus through the life of the project  Make recommendations on best practices, innovations, scale-up ready results and approaches; as well as identified pitfalls and challenges and how these could be addressed  Using lessons learnt, the best practices experienced, and the new Kenya governance dispensation; develop a concept paper on possible prospective programmatic scope for future technical assistance support based on these new realities. The evaluation sought answers to 13 guiding evaluation questions, which were developed to establish the appropriateness of the technical assistance approach of the project to the aspect of the commodity management system that it addresses. (Refer to Annex 1) Strategic Objective I Strengthened MoH Commodity Management IR 1.1. Strong & effective MoH Stewardship & technical leadership in supply-chain management/commodity security IR 1.2. Effective coordination & harmonization of GoK & development partners activity in the subsector by supply chain ICC IR 1.3. Peripheral health facilities able to account for & manage commodities in them effectively IR 1.4. Effective & efficient commodity management systems in the private sector (faith-based & commercial sector organizations) Strategic Objective II Strengthened Pharmaceutical Services IR 2.1. Strengthened Pharmaceutical subsector governance IR 2.2. Improved delivery of pharmaceutical services IR 2.3. Strengthened medicines quality assurance & pharmacovigilance IR 2.4. Improved pharmaceutical information acquisition & management Strategic Objective III Strengthened Laboratory Services IR 3.2. An efficient & effective laboratory supply chain HSCM EOP Evaluation Report, March 2016 Page 13 3.2 Proposed Evaluation Approaches The proposed methodologies were in line with the USAID Evaluation Policy and the Assessing and learning document ADS 203. The methodology included a mix of qualitative and quantitative data collection and analysis approaches. The evaluation was aimed at assessing the project’s achievements against set goals and establishing its contribution to strengthening the respective health system areas. The assessment was not designed to measure attribution. The specific approaches that were applied are: 3.2.1 Comprehensive Document Review A systematic desk review was carried out based on the objectives of the evaluation and the 13 guiding questions and sub-questions. The documents reviewed included key project documents such as work plans and quarterly and annual reports. (Refer to Annex 2 for the full list of documents reviewed). Findings were categorised based on evaluation questions. 3.2.2 Focus Group Discussions (FGD) The evaluation conducted Focus Group Discussions with members of the County Commodity Security Technical Working Groups and recipients of HSCM training courses in the selected counties. 3.2.3 Key Informant Interviews Key informant interviews were conducted using a semi-structured interview tool. The interviews collected in-depth qualitative information on the performance of HCSM and also obtained participants’ suggestions on the scope and design of any follow-on program. 3.2.4 Completion of the Quantitative Data Checklist A quantitative data checklist was used to track the performance of the project against key PMP indicators. 3.3 Method of Selecting Counties, Facilities and Study Participants 3.3.1 The evaluation team randomly selected 5 out of the 10 counties the project targets in the Western region, and one each in the Coast (out of three) and Rift Valley (out of two). In addition, the team selected two counties not targeted by the project after the re-focusing. For logistical convenience, these two counties were randomly selected from the three counties that border Nairobi – Kiambu, Kajiado and Machakos. A total of 9 counties were selected for inclusion in the fieldwork. 3.3.2 Health Facilities In each county, the team visited the county hospital and a health centre not too far from the county hospital but not in the immediate vicinity. In addition, the evaluation team visited some HCSM model sites within the counties covered. The team included some health facilities from the FBO sub-sector. In the facilities, the team conducted key informant interviews with facility in￾charges and heads of pharmacy and laboratory services HSCM EOP Evaluation Report, March 2016 Page 14 3.3.3 FGD and KII Respondents Commodity Security TWG FGD: These were made up of the members of the TWG that were available during the field visit. HCSM Training Recipients: These were drawn from county staff that had gone through any training conducted by HCSM KII Respondents: These were selected in consultation with HSCM so as to identify respondents with adequate experience working with and knowledge of HCSM 3.4 Selected Counties and Health Facilities The table below shows the counties and health facilities that were selected and visited: Counties Targeted By HSCM Selected Counties Selected Health Facilities Western Region: Kisumu, Siaya, Kisii, Nyamira, Vihiga, Migori, Busia, Kakamega, Bungoma, Homa Bay Kisumu  Kisumu County Referral Hospital  St. Elizabeth Chiga Health Centre***/**** Siaya*  Siaya County Hospital  Sega Mission Health Centre Homa Bay*****  Homa Bay Teaching & Referral Hospital  Marindi Health Centre Migori  Migori County Referral Hospital****  Ogwedhi Health Centre Kakamega  Kakamega County Referral Hospital****  Bukura Health Centre Coast: Kilifi, Mombasa, Kwale Kilifi  Kilifi County Hospital  Matsangoni Health Centre Rift Valley: Elgeyo Marakwet, Uasin Gichu Elgeyo Marakwet  Iten County Hospital  Chepkorio Health Centre Counties Not Targeted by HSCM considered for selection** Kiambu, Kajiado, Machakos Kiambu  Kiambu County Hospital Machakos  Machakos Level 5 Hospital *Evaluation team had initially selected Homa Bay. The HCSM team however requested we include Siaya because of the work done there **conveniently chosen due to proximity to Nairobi *** FBO facilities ****HCSM Model site ***** the evaluation team had selected Nyamira county randomly but the USAID team asked Nyamira be replaced with Homa Bay which has the highest HIV burden nationally 3.5 Key Informant Interviews and FGDs Conducted The evaluation had a total of 177 respondents – 128 key informants and 49 FGD participants. (Refer to annex 7 for the complete list of respondents) 3.6 Data Collection Tools The following Data collection tools were developed to guide data collection: • Commodity Security TWG FGD Guide HSCM EOP Evaluation Report, March 2016 Page 15 • HCSM Training Recipients FGD Guide • Key Informants Interview guide • Checklist to collect quantitative data on the projects performance against PMP indicators. (Refer to Annexes 3, 4, 5 and 6 for data collection tools) 3.7 Data Analysis All the data collected was analysed to generate both qualitative and quantitative information. Each evaluator typed out their field work notes from key informant interviews and FGDs into a categorised matrix. The evaluators organized their findings by the 13 evaluation questions and another category of “other findings”. Similarly desk review findings were organized by the 13 evaluation questions and another category of “other findings”. A summary of the quantitative data was prepared based on the data collected using the quantitative data checklist and the project’s PMP report. Comparative analysis was done for data from counties that had received continuous HCSM report and those where support was discontinued after the project’s refocusing. Another comparative analysis was done for commodity and commodity tools availability by service delivery level in Kisumu County. Working as a team, the evaluators manually collated all the data, analysed and identified emerging themes and salient issues for each evaluation question from the key informant interviews, FGDs, desk review and quantitative data. Findings and recommendations were based on the emerging themes for each evaluation question and were discussed, counterchecked and triangulated with the various data sources before inclusion in the evaluation report. The evaluators used the key findings identified to draft scalable lessons, conclusions and recommendations. Initial findings were also validated through briefing sessions with UASID Kenya. 3.8 Ethical Considerations All respondents signed a consent form and were assured of anonymity and confidentiality. Findings are written in such a way that they do not reveal individual respondents as the source of the information. 3.9 Evaluation Limitations Below are some of the evaluation limitations identified: • Changes in the Project’s Geographic and Programmatic Approach There were significant changes in the programmatic and geographic approach in the life of the program. Some geographic areas were dropped from the project while others were added including some in the last 6 months of the project. This made it difficult to compare activities and results across counties due major changes during the project’s life. Further, the baseline data was collected at different times based on when a county came on board while some of the baseline data had been done for a district and hence did not apply to the entire county • Impact of Devolution on Health Services Devolution of health services constitutes a major discontinuation in the lifespan of the project. Devolution resulted in large scale movements of staff mainly from the national level to counties and also within counties. This affected institutional memory and as a result during the evaluation there were many respondents who only had limited experience of the HCSM project HSCM EOP Evaluation Report, March 2016 Page 16 3.10 Evaluation Team The evaluation was carried out by four Kenyan consultants including two public health experts, one national level public pharmaceutical expert and one county level pharmaceutical expert. 4. FINDINGS The findings have been organized around the evaluation questions. For some questions, the findings are further disaggregated under different thematic areas including the three priority health program areas - HIV, Malaria and Family planning. Some additional findings outside evaluation questions are reported under “Other Findings”. 4.1 Strengthening Commodity Systems at National Level Evaluation Question1: To what extent has the project strengthened commodity security systems at national level? (Strategic Objective 1, IR 1) Key Activities Planned by HCSM: Capacity building; develop and disseminate key policy documents; develop national LMIS; support the national level Interagency Coordinating Committees (ICC); Strengthen commodity information reporting; Support TWGs; Enhance DHIS2 and migrate all PHP commodities; Support roll-out of electronic commodity management tools; 4.1.1 National Level Commodity Management Programs HCSM supported a number of activities to strengthen commodity management at the national level. Key activities included provision of technical and financial support for the development of critical commodity management guidelines, policy documents, training curricula and training packages. Key documents developed or reviewed include supportive supervision guidelines, forecasting and quantification guidelines, Kenya Essential Medicines List (KEML), Kenya Essential Medical Supplies List (KEMSL), Essential Medicines and Medical supplies (EMMS) guidelines. All these documents were reviewed to reflect the transition to a devolved system of delivering health services. These documents were disseminated to all counties through two County Pharmacists’ forums that were supported by HCSM. The Pharmaceutical Services Unit (PSU) of National MoH reported that HCSM was the sole source of support for all policy and training materials that the unit developed. However, the evaluation team was informed that although these policy documents have been developed and disseminated, there has been no follow-up to see how and if they are being used at the county and facility level. The lack of monitoring by national MoH on the use of guidelines at the county level was attributed to the prevailing mistrust and hostility between county and national governments and also due to budgetary constraints. 4.1.2 Expanding Pre and In-Service Training The project has supported development and roll-out of training materials and training packages that have supported national level commodity management capacity building. Some of these training programs have also been cascaded to the county level. Some of the specific national level training programs that HCSM has supported include:  Masters Course in Pharmacoepidemiology and Pharmacovigilance HCSM supported the University of Nairobi - School of Pharmacy develop and mount a 2-year, masters course in Pharmacoepidemiology and Pharmacovigilance. The program has graduated a HSCM EOP Evaluation Report, March 2016 Page 17 total of 17 students in two lots since its inception. Most of the students enrolled have been sponsored by the government. Before the introduction of this course, the country did not have a single pharmacist trained in pharmacoepidemiology and pharmacovigilance. A number of graduates of this program are serving in key health institutions including national referral hospitals, NASCOP, National Malaria program and NTLP.  One –Day Course in Pharmacovigilance for Final Year Pharmacy Students MSH has also supported a one-day pharmacovigilance course for final year UON undergraduate pharmacy students. The university is yet to integrate this training into their undergraduate curriculum.  Commodity Management Pre-Service Short Course HCSM has supported a 5-day pre-service short course in commodity management for final year Diploma Pharmaceutical Technology students at the Kenya Medical Training College (KMTC). The training has not been integrated into the pre-service curriculum. 4.1.3 National Level Management of HIV Commodities NASCOP is charged with the responsibility of national level planning and coordination of HIV commodities including ARVs, OI drugs, nutritional products and rapid test kits. HCSM support to NASCOP has included development of training curricula and materials, development of a wide range of tools and support for the National Commodity Security TWG. The project also helped incorporate HIV nutritional products and laboratory commodities into the KEMSA LMIS which previously only handled ARVs and OI drugs. A key achievement of HCSM was supporting the establishment of a National HIV Commodities Security Technical Working Group that draws its participants from NASCOP, KEMSA and key partners that support HIV commodities. The strengthening of the forecasting and quantification capacity by HCSM was said to have been particularly useful and helped improve stocking levels and also supported resource mobilization for various commodities. NASCOP used the F&Q approach and tools to develop the successful GFATM funding concept. HCSM has helped NASCOP develop a well-established commodity planning and management cycle that addresses key commodity management issues. The commodity planning and management cycle includes monthly commodity review meetings, annual F&Q sessions and biannual reviews of forecasts. HCSM also supported NASCOP develop an electronic tool to allocate rapid test kits to 6000 testing centres nationally. HCSM has supported the development of a number of electronic inventory management tools including ADT (Antiretroviral Dispensing Tool) and EDITT (Electronic Dispensing and Inventory Tracking Tool) that have helped make available accurate commodity data in a timely way. HCSM has also supported NASCOP prepare for migrating their commodity data to DHIS2. This migration will give HIV commodity visibility to more stakeholders including NASCOP and managers at the county level and health facilities and also make it easier to compare commodity data with health services data that also sits on DHIS2. With the support of HCSM, NASCOP develops a monthly information 2-pager on HIV commodity status that is shared with a wide range of stakeholders who include the national health managers and program partners involved in HIV programs. HCSM has also helped NASCOP start tracking forecast accuracy. Early data suggests that forecast accuracy is improving. The country has experienced improved availability of HIV commodities and this could be partly attributed to activities that HCSM has supported including forecasting and quantification, and pipeline monitoring. It was reported that there are significant challenges with forecasting and allocation of HIV rapid test kits leading to frequent shortages. The reasons for this HSCM EOP Evaluation Report, March 2016 Page 18 are poorly understood at national and county level. Possible contributory causes suggested include leaking of commodities to other markets and/or poor data quality. It was also reported that management of HIV nutritional commodities remains a challenge largely due to the fact that these commodities are procured and distributed by multiple partners and coordination is weak. 4.1.4 National Level Management of Malaria Commodities HCSM provided support to the NMCP to strengthen malaria commodity management, pharmaceutical services and the rational use of malaria drugs. In the words of the program, HCSM support facilitated the availability of ‘the right commodities at the right time’. Key activities included forecasting and quantification, pipeline monitoring and tracking, and development of tools and guidelines. The malaria commodity reporting is through DHIS2. With the support of the project, NMCP has been holding annual pharmacists’ meetings to provide commodity management and pharmaceutical services training. With HCSM support, NMCP has conducted annual educational forums for laboratory staff drawn from the national and county levels. NMCP acknowledged the key role played by HCSM in building capacity for commodity management at national and county level. HCSM also supported the Malaria ICC and the National Malaria Commodity security TWG’s and sub-committees. NMCP rated HCSM as the strongest partner of the drug management sub￾committee. HCSM supported the training of private sector health workers in commodity management and provided tools and guidelines. NMCP reported that the most useful HCSM interventions were monthly stock status reports, support for DHIS2, support for the biannual Quality of Care (QOC) surveys and forecasting and quantification. Key results were reported to include increased diagnostic capacity from 50% to 98% due to enhanced availability of RDT’s and also reduction in use of AL from 14 million to 12 million doses mainly due reduction on symptomatic treatment of malaria and increased compliance with Malaria treatment guidelines. 4.1.5 National Level Management of FP Commodities The commodity management unit of the RMHSU is tasked with ensuring FP and maternal health commodity security. This public health program suffered the most immediately post devolution after counties received funds but failed to budget for FP commodities. The total commodity requirement is about USD 13 to 15 Million of which the total GOK contribution in 2015 was a paltry Kes 50 million against a contribution of Kes 600 million before devolution. Some of the FP commodity management areas supported by HCSM include quantification and forecasting, monthly pipeline monitoring, quarterly meetings of the RH ICC, training of CHMT members on FP commodity management and support for improved FP data reporting that rose from 20% to 80% between 2013 and 2015. HCSM supported RMHSU to engage counties in clusters to address reporting and data quality issues. It was reported that there was an issue with the accuracy of one of the quantification rounds leading to calls for emergency procurement. RHMSU reported that as a result of the HCSM support commodity availability is improving. RMHSU reported that leakage of FP commodities (mainly oral contraceptive pills and injectables) into the private retail pharmacies remains a challenge. The unit noted that weak pharmacovigilance for FP commodities as a key risk. Some stakeholders reported that RMHSU has capacity constraints as the organization has only one person responsible for all commodity management issues. HSCM EOP Evaluation Report, March 2016 Page 19 4.2 Strengthening of Health Commodity Management at County Level Evaluation Question 2: To what extent has the project strengthened overall health commodity management systems at sub-national level i.e. county and peripheral health facilities? (Strategic Objective 1, IR 3) Key Activities Planned by HCSM: Establish county commodity security TWGs; Assess and strengthen county commodity management capacity; Establish supportive supervision; Strengthen commodity reporting; Support data quality review; Scale-up ADT; Convene a national health commodity security consultative forum; Develop model sites; Provide job aids 4.2.1 Establishment of County Commodity Security Technical Working Group HCSM has established functional County Commodity Security Technical working groups in all 15 counties that they supported. The members of the TWG typically include the County Pharmacist, County Medical Laboratory Technologist, County Nutrition Coordinator, County HRIO and Program coordinators – HIV, TB, Malaria and RH. The TWG has helped counties address the myriad health commodities management challenges they face. This is particularly so given the historical realities where counties (formerly as provinces and districts) and facilities played a minimal role in planning and managing commodities, a function which was largely the preserve of the national MoH and Priority national health programs. The evaluation team found that TWGs hold regular meetings and lead a number of key commodity management activities including F&Q, supportive supervision, quarterly data quality review, commodity redistribution and improvements of commodity reporting. Many TWGs have lobbied their counties for additional funds to support improvements in commodity management including refurbishing and expanding storage space. The Migori County Commodity Security TWG successfully lobbied the county government for the construction of a larger county store for health commodities. Prior to this the county did not have a county store and used the limited storage space at the County Hospital. In Siaya, the County Commodity Security TWG successfully lobbied Health facility management boards in two sub￾counties to refurbish pharmacy stores. The TWG structure and activities were reported to have been particularly useful in strengthening nutritional and laboratory commodity management which was even weaker than that of pharmaceutical commodities. The Homa Bay nutrition coordinator reported that through the TWG activities they had increased the number of central ART sites that provide nutritional supplements from 3 to 8 (one for each sub-county) and satellite sites from 33 to 92. 4.2.2 Forecasting and Quantification The counties reported that forecasting and quantification was very useful in helping them plan for their health commodities needs. The counties reported that initially they had very little information Mind-set Change: Viewing Commodities as Valuable For many years our facilities received commodities from KEMSA often without ordering or paying for them. This entrenched a laissez faire culture where people viewed commodities as having no cost. An example is the casual attitude to commodity expiries. Following the support of HCSM and the fact that we now pay for our commodities, a new mind-set has evolved that sees commodities as valuable and appreciates the need for an accountable and efficient system. County Director of Health HSCM EOP Evaluation Report, March 2016 Page 20 to use for commodity planning as previously a commodity “Push” system was in place and counties did not even have reliable historical consumption data. A county respondent informed the evaluation team that following F&Q – “For the first time we now know what commodities we require and what budget we need to procure them” 4.2.3 Resource Mobilization for Health Commodities A number of counties reported that they had used F&Q data to lobby the county government for additional funds to support procurement of commodities and to support commodity management related infrastructure. Siaya County has successfully used the FQ data to lobby for more commodity funds leading to a 50% increase in their health commodity budget. 4.2.4 Integrated Approach to Commodity Management With the leadership and coordination afforded by the County Commodity Security TWG, many counties have adopted an integrated commodity management approach that covers all categories of commodities including pharmaceuticals, non-pharms (gloves, IV kits, syringes, dressings etc), nutritional products and laboratory commodities. In some counties such as Migori, they also did F&Q for “free” program commodities such as ARVs. The Migori county team reported that they found having information on these commodities an eye opener as they were not aware that these “free” commodities cost so much. Integration has involved a joint approach to commodity planning, supportive supervision and use of common commodity management tools, storage and reporting approaches. Siaya county reported that commodity management improvement initiatives in respect to storage and record keeping had also been extended to vaccines and TB drugs which are still centrally procured by national programs. The teams were also working together on commodity reporting. An example is in Siaya, where the county nutritionist reported that even in Sub-Counties that did not have nutritionists, they were still achieving very high reporting rates because other functions such as pharmacy provided the needed oversight. 4.2.5 Use of Electronic Tools for Commodity Management HCSM provided counties with support for electronic commodity management tools. The Antiretroviral Dispensing Tool (ADT) was developed by MSH and has been in use for many years. It was reported that it was of great help in ensuring accurate inventory records. HSCM has recently developed Electronic Dispensing and Inventory Tracking Tool (EDITT) which is currently being piloted in a number of facilities. Unlike the ADT, it is able to handle other commodities in addition to ARVs, link to DHIS2 and handle other inventory management processes beyond dispensing. It was found that multiple commodity management IT systems were in use procured by county governments and partners. Examples include IQCARE by CDC, Fansoft, Medboss, and HCMP by CHAI. A number of health facilities including some busy Patient Support Centres (PSC) run completely manual dispensing and inventory management systems. The evaluation team found that Homa Bay County Hospital with over 6500 active HIV patients had no electronic commodity management system Strengthening Nutritional Commodities Initially there were absolutely no records on nutritional commodities - receipts, issues etc. Stores were accessible to multiple people. With HCSM support, we now have accountability documents for every process. This has minimized misuse and loss and we believe it has increased the amount of products reaching our clients. - County Nutritionist HSCM EOP Evaluation Report, March 2016 Page 21 4.2.6 Model Sites HCSM designated 39 sites in the 15 focus counties as model sites. These were evaluated and their capacity built to ensure they attained a high level of commodity management standards so as to be a platform for building the capacity of other sites. The evaluation team found that most staff in the designated model sites were not even aware that their facilities were model sites or what the model site concept entailed. It is only one staff from a sub-county hospital in Nyanza that could articulate the model site concept and its implications. A few of the model sites visited had glaring commodity management issues including poor health commodity storage practices 4.2.7 Scaled-Up Recruitment of Staff Involved Commodity Management It was found that many counties had recruited significant numbers of additional health workers including those with a major role in commodity management such as pharmacists and laboratory technicians. The table below shows the changing staffing picture for Kisumu County. Cadre No in 2013 No in 2015 Pharmacists 35 49 Pharmaceutical Technologists 9 44 Laboratory Technologists 70 112 Nutritionists 8 8 4.3 Supporting Improved Coordination and Harmonization in Commodity Management Evaluation Question 3: How has the implementation of HCSM improved co-ordination and harmonization of the activities of the various players (donors & IPs) in the health commodity supply chain arena? (Strategic Objective 1, IR2) Key Activities Planned by HCSM: Support MoH to convene a national health commodity security consultative forum (involving key stakeholders) to share and disseminate best practices and discuss emerging issues on health commodity supply chains. In December 2015, a County Forum on Commodity Management was held during which all the focus counties and national programs presented papers on best practice. Below are other findings under this question. 4.3.1 GOK, Donors & Implementing Partners At the national level, the HCSM project worked closely with priority health programs and MOH program departments to strengthen commodity management and pharmaceutical services. In the opinion of MOH public health programs at national level – Malaria, HIV (NASCOP), DRH and TB, HCSM adopted a well-coordinated approach in implementing their activities and supported national priorities. The national level TWG’s and committees played a critical role in harmonising and coordinating commodity planning, funding and programming among donors (USG and non￾USG), GOK and implementing partners. However most of the TWG were focused mainly on vertical disease program commodities and rarely included non-state actors. HSCM EOP Evaluation Report, March 2016 Page 22 At the county level HCSM worked together with other USG partners and projects such as FACES, EGPAF, APHIAPlus, PIMA, CHAI and Ampath in implementing its commodity management and pharmaceutical services activities. It’s important to note however there is no formal framework for ensuring and monitoring coordination and harmonisation between various USG and non-USG partners and cooperation currently depends on the goodwill of program staff and managers. 4.3.2 National & County Government Coordination Coordination and harmonisation between the county and national government has been a challenge due to teething problems of understanding and implementing the devolved system of governance. The envisaged inter-governmental coordination forums as defined in the Inter￾governmental Relations Act are only beginning to take shape and define their scope and mechanisms of operation. HCSM project worked at the national and county level largely independently and there was minimal effort or activity to bring together county and national governments to address commodity management issues in a harmonised and coordinated manner. The only notable attempts to bring the two levels of government together were the two county pharmacist forums held in 2013 and 2015 that brought together county pharmacists, the national PSU, KEMSA and the public health programs. 4.3.3 Engagement of FBO/Private Sector While HCSM project included FBO’s (CHAK, KCCB and selected FBO facilities) and some private sector actors (KPA, PSK and selected private for profit facilities) in commodity management and pharmaceutical services activities, these stakeholders were generally left out of TWG’s and other committees at national and county level. Multi-sectoral national commodity stakeholder forums did not take place. At the sub-national level the forums were held in only two counties - Kisumu and Kilifi. 4.4 Improving Stewardship and Leadership at Sub-County Level Evaluation Question 4: Has the project achieved its objective of improving stewardship and leadership for commodity management at sub-national level? (Strategic Objective 1, IR 3 and Strategic Objective 2, IR 1) Most of the stewardship and governance improvements were implemented through Strategic Objective 1, IR 3 and the findings are reported under Evaluation Question 2 above. Under Strategic Objective 2, IR 1, the project set out to create forums bringing together county health managers and other stakeholders to disseminate key national documents such as supportive supervision guideline, quantification guidelines and product disposal guidelines. This was achieved through the County Pharmacists Forums. The creation and capacity building of the County commodity TWG enabled the counties to take leadership and drive the commodity management agenda and activities in the county. It gave health commodities issues greater visibility in the wider The Policy Role of the National MoH – Why It is Not Working It is no longer clear to us what our job is. There are only so many policies and guidelines that we can review and develop. Further, with relations with counties completely broken down, we do not even know which new policies are needed and which old ones need to be reviewed. We also have no idea how and if the ones we have disseminated are being used. - National MoH Respondent HSCM EOP Evaluation Report, March 2016 Page 23 county leadership and elevated commodity availability and access matters to the highest decision making levels. 4.5 Trends in Availability and Reporting Rates for Health Commodities Evaluation Question 5: What trends in availability of and reporting rates for key health commodities have been observed over time, and to what extent has the project contributed to improvements in the trends? (Strategic Objective 1) Key Activities Planned by HCSM: Support to MoH to include HIV, FP and TB commodities data collection and reporting systems to the DHIS-2; Enhance DHIS2 4.5.1 Commodity Availability The HCSM project has supported PHPs to conduct F&Q, mobilize additional commodity funds, undertake pipeline monitoring and develop monthly 2-pager commodity status reports. The pipeline tracking was done in all the programs – HIV, Malaria, TB and FP. Pipeline monitoring helped commodity planners to make prompt decisions to delay or bring orders forward and in some cases advised on change of treatment regimens where there were global supply challenges. All these interventions and similar ones at the county level have supported increased commodity availability. It was reported that in the last year there was zero stock-outs of ARVs at the national level. A common commodity availability concern was the frequent shortages of HIV RTKs. This was attributed to poor data quality and possible leakage into other markets. 4.5.2 Commodity Reporting The reporting rates for the key health commodities have been on an upward trajectory. HCSM has rolled out malaria, FP, nutrition commodity reporting on the DHIS2 platform. National FP commodity reporting rates increased from 24% to 83% nationally (91% for HCSM counties), HIV Nutrition commodity reporting improved from 0% to over 58% nationally (68% for HCSM counties) while Malaria commodity reporting rates improved from about 40% to70% nationally (93% for HCSM counties). It was reported that improved reporting rates for malaria commodities helped improve clinical practice. This was because teams could compare commodity and health service data and identify cases where Malaria was being treated symptomatically contrary to guidelines. HCSM activities included capacity building, support for migration to DHIS2, supporting data quality review meetings and giving staff air time for electronic transmission of reports. Concerns were raised that increased reporting was not always accompanied by improvements in data quality. In all the 9 counties visited there was a common challenge of availability of registers and reporting tools. The evaluation team was informed that in the transition to devolution the responsibility of procuring reporting tools between the national and county MoH was not defined. As a result, both the national and county MoH did not budget for them. 4.6 Pharmaceutical Sub-Sector Governance and Service Delivery Evaluation Question 6: To what extent has the HCSM project improved the pharmaceutical sub-sector governance and the delivery of pharmaceutical services? (Strategic objective 2 IR1 and IR2) Key Activities Planned by HCSM: Finalize and disseminate policy guidelines and training packages; Develop operational plans for pharmacy professional associations; Develop/ review; Support bi-annual Malaria Quality of Care (QoC); Conduct an LLINs assessment HSCM EOP Evaluation Report, March 2016 Page 24 4.6.1 Development and Use of Policy Documents A number of key pharmaceutical policy documents have been reviewed and disseminated. These include, Kenya Essential Medicines List (KEML), Kenya Essential Medical Supplies List (KEMSL) and Essential Medicines and Medical supplies (EMMS) guidelines. The evaluation found minimal documented operationalization of these policy documents. It was also reported that the delay in the adoption of the health policy and the finalization of the Health Bill have led to delays in the finalization and rollout of some policy documents including the Kenya National Pharmaceutical Policy (KNPP) also known as Sessional Paper No. 4 of 2012. 4.6.2 Strengthening Professional Associations HCSM supported the Pharmaceutical Society of Kenya (association for pharmacists) develop an operational plan for their strategic plan. The organization has been able to implement the operational plan and credit the support they received from HCSM with the strengthening of the organization in terms of improved HR capacity and capital base. HCSM also supported the revision of the strategic plan and development of an implementation plan for the Kenya Pharmaceutical Association (the association for the pharmaceutical technologists) 4.6.3 Commodity Supportive Supervision At the county level, it was reported that the roll-out of a commodity dedicated supportive supervision guided by a detailed checklist had been very helpful in both assessing and improving the status of health commodity management at the facility level. This was said to have contributed to improvements in commodity storage, record keeping and commodity redistribution. 4.6.4 Malaria Quality of Care Surveys HCSM worked with NMCP and other partners to conduct biannual Malaria Quality of Care surveys. The surveys addressed a number of areas including the availability of Malaria drugs and RDTs, commodity record keeping and reporting, and availability and adherence to treatment guidelines. HCSM also supported a survey undertaken to monitor use of Insecticide Treated Nets to inform LLIN communication strategy 4.7 Rational Drug Use, Quality Assurance and Patient Safety Evaluation Question 7: To what extent has the implementation of the HCSM project improved systems for promoting the rational/appropriate use of medicines, product quality assurance and patient safety? (Strategic Objective 2, IR 3) Key Activities Planned by HCSM: strengthen the national, county and facility Medicines and Therapeutic Committees (MTC); Review and update PV guidelines and SOPs; Strengthen the 12 existing sentinel PV sites; Support PPB for medicine safety communication; Support the MOH and PPB to develop and rollout a PMS strategy/framework 4.7.1 Pharmacy and Poisons Board interventions HCSM has provided the Pharmacy and Poisons Board (PPB) with support in Pharmacovigilance, medical information and Post Marketing Surveillance (PMS). Using HCSM support, the PPB produces and distributes a regular health information newsletter targeting health workers. Other areas that HCSM has provided support to the PPB include carrying out PMS campaigns, HSCM EOP Evaluation Report, March 2016 Page 25 development of an electronic tool for reporting Adverse Drug Reactions (ADRs) and suspected poor quality medicines, and printing of pharmacovigilance tools and job aids. In collaboration with NASCOP and the Malaria program, PPB conducted cohort event monitoring through newly established sentinel sites to actively identify ADRs and suspected poor quality medicines instead of relying on spontaneous reporting. One PV cohort study for malaria drugs has been completed while another one for ARVs is ongoing. The PPB produces a monthly 2-pager status report on Pharmacovigilance that is distributed to all County Directors of Health and County Pharmacists. Kenya now has the 4th highest ADR reporting rates in Africa and PPB has been appointed a regional centre of excellence in pharmacovigilance by NEPAD. PPB reported that they have been able to take regulatory action on the basis of their pharmacovigilance and PMS findings. For example, PPB pharmacovigilance data was used to make a case for Kenya migrating from Stavudine to Tenofovir based ARV regimens. PPB also discontinued the use of a version of Oxytocin following complaints of low potency. NASCOP also reported that they had worked together with PPB to recall a batch of Zidolam in Nyanza following a WHO alert. NASCOP has also worked with PPB to investigate a number of other cases of suspected poor quality of medicines and unusual rates of ADRs. PPB’s PMS campaigns have covered Malaria, FP, ARVs and TB drugs. PPB reported that PMS has revealed a declining trend in the number of poor quality medicines in the market as shown in the chart below: % of Post Marketing Surveillance Samples Passing Testing Source: HCSM Annual Report 2014/15 4.7.2 National, County and Health Facility Medicines and Therapeutic Committees (MTC) HCSM supported the establishment or reconstitution of Medicines and Therapeutic Committees (NMTs) at the national, county and facility level. However, the evaluation team found that with a few exceptions, the MTCs are not operational. Many meet infrequently and suffer from quorum hitches. Some of the areas that MTCs were found to be addressing include adherence to Malaria treatment guidelines and provision of Continued Medical Education (CME). 4.7.3 Mobile App for Downloading Treatment Guidelines HCSM developed a mobile app to help health workers download standard treatment guidelines. However, the evaluation team found extremely low awareness of the app while rates of reported use of the app were even lower. Out of 9 counties visited it is only in three counties – Migori, Siaya and Kakamega where we found respondents that were aware of the App and it is only in Siaya that health workers reported having used the app. Those who were aware of the app but had not used it reported that they had problems logging into the database. 84.10% 76.00% 90.00% 99.10% 0.00% 20.00% 40.00% 60.00% 80.00% 100.00% 120.00% 2010 2011 2012 2014 HSCM EOP Evaluation Report, March 2016 Page 26 4.8 Laboratory Commodities Supply Chain Evaluation Question 8: Has the HCSM project achieved the objective of strengthening the supply chain for laboratory commodities? (Strategic Objective 3, IR 2) Key Activities Planned by HCSM: Build the capacity of Lab county coordinators and CHMT on supply chain; Support supportive supervision, OJT, mentorship & CMEs on lab commodity management; Support reporting of lab commodities; Finalize and disseminate the lab essential commodity and tracer lists; Support F&Q for lab commodities The HCSM project focused on the intermediate result area on commodity supply chain. The other two intermediate result areas were to be implemented by the CDC funded MSH Strengthening Public Health Laboratory Systems (SPHLS) program. HCSM has made progress in improving the management of laboratory commodity supply chain. In the award document1 the initial plan was to support the establishment of a national laboratory commodity steering team to coordinate the interventions in the laboratory commodity management. This national committee was never established and the laboratory commodity management issues were handled through the PHPs and county commodity security TWGs. The areas in which the laboratory commodity supply chain has been strengthened include: 4.8.1 Stewardship for commodity management In the PHPs, laboratory managers were part of the commodity security committee. They were also members of the county security commodity TWGs. As part of the County TWGs they participated in commodity support supervision teams.2 To help in monitoring the performance of laboratory services a list of laboratory tracer commodities was developed as part of the KEMLCL. 4.8.2 Forecasting & Quantification For the three PHPs – NASCOP/GFATM, NMCP, and NLTP the diagnostic laboratory commodities used in each of these programs were integrated in the quantification exercise. 3,4,5 To assist the laboratory officers to conduct F&Q, HCSM developed laboratory commodity quantification job aid. In the county, F&Q also incorporated laboratory commodities. To ensure rational selection of laboratory commodities, HCSM supported the development of a Kenya Essential Medical Laboratory Commodities List (KEMLCL) in 2013 which was informed by the Kenya National Clinical Guidelines. 4.8.3 Capacity building for Commodity Management In all the PHPs and counties visited, laboratory officers had participated in the integrated commodity management training. However, to address the knowledge gaps in laboratory commodity management, HCSM developed and rolled-out a curriculum for laboratory commodity management and an implementation guide and a facility Standard Operating Procedure. 4.8.4 Laboratory Commodity Logistic Information System The HCSM has supported the MoH to review the laboratory commodity record keeping and reporting tools. They also trained laboratory officers on the use of these tools for reporting. 1 Cooperative Agreement No. AID-623-LA-11-00008 2 Supportive Supervision Manual for Health Commodities Management Second Edition September 2014 3 National Quantification Report for HIV-related commodities for FY 2014/15 & Forecast for FY 2015/16 and FY 2016/17 4 Malaria Commodities Quantification And Supply Planning Review For FY2013/14 Technical Report September 2013 5 Kenya National Quantification for Tuberculosis and Leprosy commodities 2014/2015 HSCM EOP Evaluation Report, March 2016 Page 27 Counties reported that HCSM had supported lab commodity reporting by providing courier services for transporting tools, providing mobile phone airtime for data uploading and supporting printing of reporting tools. Currently the laboratory commodities are reported in the DHIS2. 4.8.5 Impact of Laboratory Commodity Initiatives on PHPs HIV The strengthening of the laboratory commodity supply chain management has enabled the HIV program (NASCOP) improve the availability of its diagnostics. This is through improved reporting rates which in turn provides data for national F&Q exercise and supports resource mobilization. All the counties visited and NASCOP reported that availability of the testing kits has improved and stock outs reduced despite the continuing challenges with RTK accountability Malaria The main laboratory commodity used in the NMCP is the Rapid Diagnostic Test kits (RDTs). In 2010, the Malaria treatment guidelines were revised to change from presumptive treatment to parasitic diagnosis before treatment with AL. The HCSM project supported NMCP in the development and rollout of a curriculum on RDT use. Below is an example from Siaya County showing how tracking Malaria testing and AL consumption helped reduce the treatment of clinical malaria. Source – Siaya Commodity Security TWG reports 4.9 Ownership and Sustainability Evaluation Question 9: In implementing activities, has capacity been built within MoH (National & County) and has there been a transition plan to host country ownership for sustainability? Within the national level MOH, HCSM interacted mainly with the PSU, PPB, NPHLS and the public health programs – Malaria, DRH and HIV. In the view of most stakeholders interviewed at national level there has been significant skills transfer. The PHPs were of the view that the work they had done with HCSM – establishment of TWGs, quantification, pipeline monitoring and PV was largely sustainable but may require limited TA and some budgetary support. However, the malaria program identified laboratory commodity management as lagging behind and needing more support. The laboratory TWG was only started in 2015. The DRH staffing also needs to be strengthened to effectively carry out its mandate. PPB informed the team that with introduction of 375957 370699 283978 271267 46706 171534 162666 375416 457594 496834 0 100000 200000 300000 400000 500000 600000 2011 2012 2013 2014 2015 No of Cases Trends of confirmed and clinical malaria cases 2011 - 2015 Siaya county Clinical Malaria Confirmed Malaria HSCM EOP Evaluation Report, March 2016 Page 28 a new levy on pharmaceutical imports, they will have adequate funds to support PV and PMS activities. The new GFATM grant has funding for a number of commodity management activities including capacity building both at national and county level. KEMSA held itself out as an example of a national institution whose capacity was built through various programs in the past and is now working towards financial sustainability. PSU felt that exit of HCSM would spell doom for the unit given budgetary and TA constraints None of the national and county institutions supported by HCSM had specific post-HCSM transition plans. At the county level, there was a general acknowledgement of significant skills transfer and capacity improvement in the management of commodities and pharmaceutical services. Most county commodity security TWG’s agreed that they had acquired skills in forecasting and quantification and understood how to use the data to better manage the whole supply chain cycle – ordering, receiving and verification, storage and inventory management. They were also better able to carry out on-the- job training and supportive supervision and data review activities. Counties have rapidly recruited additional pharmacy, nutrition and lab staff which will support sustainability. However most county respondents felt that the journey towards strengthening the capacity of counties to better manage commodities and pharmaceutical services had just begun and there was need for more concerted and well-coordinated support (technical and financial) to fill in existing capacity gaps, consolidate skills and competencies and bring their capacity to a sustainable level. Any loss of support at this stage in their view would significantly roll back the gains made. EDITT was given as a good example of an intervention at risk. The software is new and unstable hence requiring ongoing support to fix software bugs. 4.10 Best Practices and Innovations Evaluation Question 10: What are the documented best practices and innovations, both in terms of technical contributions and in systems strengthening approach, by the project during its period of implementation? The project documents do not provide a working definition of best practices and innovations. However, there are reports of practices from the Public Health Programs and the focus counties which are considered to have significantly improved the way that commodities are managed and can be considered as best practice. The HCSM convened a County Forum on Health Commodity in December 2015. During this meeting representatives from the 15 counties that the HCSM had supported came together to share their best practices which have been documented in a compendium.6 From the interviews conducted by the evaluation team the following are examples of best practices that have been documented. 6 Transforming Health Commodity Management in a Devolved System – A Compendium of Best Practices and Innovations from Counties Greater Ownership by National Programs HCSM has particularly been successful in building the commodity management capacity and ownership of NASCOP. For many years donor representatives led most commodity management tasks including forecasting and quantification, hosted meetings, took meeting notes and often wrote program reports. Today NASCOP takes a lead in virtually all these activities – International Health NGO Respondent HSCM EOP Evaluation Report, March 2016 Page 29 4.10.1 Kilifi County – Forecasting and Quantification and Use of F&Q Data for Budget advocacy With the support of HCSM, the Kilifi CHMT constituted a multi-disciplinary team that undertook an F&Q exercise. The F&Q report was submitted to the County Assembly Health committee as a tool for lobbying for increased budgetary allocation for health commodities. Based on their F&Q data, their annual health commodities budget was increased from Ksh 204 million to Ksh 369 million 4.10.2 Kakamega County – Commodity Data Reporting at Khwisero Sub-County The quality of commodity management data is important for decision making. Supported by the HCSM project, the Kakamega CHMT and Khwisero Sub county HMT intervened to address the poor reporting of commodity data at the sub-county including lack of or incomplete reporting, inadequate reporting tools, and low ownership of the reporting process. The County HRIO gave the Sub County commodity managers rights to the DHIS2, photocopied the reporting tools, used a checklist to identify facilities that had not reported, implemented an SMS and WhatsApp platform to send reminders and conducted facility and sub-county reviews before uploading data. The reporting rates for all commodities in the sub-county rose from an average of about 60% (2014) to 95% (2015). 4.10.3 Kisumu County – Kodiaga Prison Health Centre - Improved HIV Defaulter tracing using ADT Kodiaga prison health centre was using a manual system for recording patient data. As a result, identifying defaulters was difficult. This situation however changed in 2014 when the health facility embraced a fully automated HIV patient and commodity information management system through the use of the Antiretroviral Dispensing Tool (ADT). This was done with the support of the HCSM project. The tool made it possible for the facility to detect and trace defaulting clients as the tool flagged out those that did not turn up for their appointments to collect their ARV drugs. Clients were then followed up and those that were traced were reinstated on the treatment program. 4.10.4 Pharmacy and Poisons Board - Assuring Quality of Medicinal Products in Kenya through Post Marketing Surveillance The presence of poor quality medicines has become a public health concern especially in low income countries. Such medicines endanger patient safety, lead to treatment failure and development of drug resistance and represent a waste of financial resources. In 2011, the MOH and the PPB took steps to strengthen the country’s capacity for surveillance systems to track and detect poor quality medicines in the Kenyan market. With support from the HCSM they developed and implemented a national system for monitoring and reporting suspected poor quality medicinal products (PQMP) through PMS and adverse drug reactions tracking. PPB has also been able to make several evidence-based regulatory decisions including withdrawal of medicines; closure of a pharmaceutical company; recall of medicines and rescheduling of some antibiotics for hospital use only. Consequently, there is a trend of reducing incidence of PQMPs on the market. 4.11 HCSM Implementation Challenges and Solutions Evaluation Question 11: What challenges has the project faced in activity implementation and how can these be addressed? The HCSM project encountered a number of implementation challenges. These include a very broad initial geographical scope with the project covering 64 districts against a target of 50 districts HSCM EOP Evaluation Report, March 2016 Page 30 by September 2012 spread out over all the previous 8 provinces. As a result, the project resources were spread very thin thus reducing implementation effectiveness. In addition, there was a mid-stream programmatic and geographic focus change. Other challenges included weak coordination and harmonization between various partners (each with different approaches and systems), and weak linkages between the national and county MoH. Most county respondents said that HCSM did not have adequate regional coordinators leading to inadequate level of support and delayed implementation of several activities. A number of county meetings and commodity management activities had to be rescheduled due to the unavailability of the regional coordinator. The creation of devolved units meant that HCSM had to engage afresh with a new set of health sector leaders and county governance structures. This was further complicated by the initial difficult relationship between the national and county governments and health leadership. It was also reported that some HCSM staff had difficulties relating well with county CHMTs. In one case the replacement project staff had big challenges engaging and influencing the county CHMT. Although the project was required to harmonize and coordinate with other USG projects at the national and county level, there was no clear framework for this coordination and harmonization. For example, while it was a good idea to propose that some commodity management activities should be handed over to existing partners and projects at the county, the implementation modalities were not clear and budgetary implications were not addressed.. Commodity management activities particularly capacity building and the strengthening of reporting and data quality required support up to facility level but the project had limited mandate and resources to effectively reach this level. Working with other service delivery partners was proposed but there was no framework to ensure that this happened. It was left to the goodwill of project managers. County CHMT’s had their hands full with many activities of their own and multiple donor programs. Competing tasks delayed implementation of planned activities. Examples mentioned included TWG meetings not being held as regularly as intended and frequent rescheduling of trainings. A recurrent complaint from county stakeholders was that reimbursements for stipulated allowances and costs (transport, per diem etc.) for meeting participants were below the stipulated rates for MOH and took very long to be reimbursed (2 to 4 months delay) after activities were completed. This served as a participation disincentive to health workers who in many cases had to finance their transport and other costs pending the reimbursements. Addressing the challenges The next design of the program should determine the framework for coordination of activities for all the organizations that are funded by USG. The rest of the specific recommendations are dealt with in the relevant section of this report. 4.12 Commodity Management and Pharmaceutical Systems Gaps Evaluation Question 12: What gaps still exist within the country’s health commodity management and pharmaceutical systems? How could these gaps be addressed in future? National Level Commodity Management Programs HSCM EOP Evaluation Report, March 2016 Page 31 Some of the commodity management gaps that plague the sector include frequent stock-outs of essential health commodities, high staff turnover which occurs without proper handing over and issues with the reporting for commodities and data quality. The most frequently mentioned commodities where periodic stock outs occurred included RTK’s, blotting paper for DB test, AL, Artesunate injection, Ferrous Sulphate, SP for malaria in pregnancy prophylaxis and commodities for gene expert tests (Falcon tubes). In the case of AL, availability of formulations for all weight bands was erratic. There is also leakage of products (i.e. cannot be accounted for) such as FP (oral and Injectable contraceptive), RTKs, nutritional supplements and possibly Artemether Lumefantrine (AL). It was mentioned that the leakage of FP commodities was mainly to private pharmacies - this could point to an unmet need in the market. Analysis of availability of health commodities and commodity management tools in Kisumu County showed there was no significant difference in stock-outs by level of health facilities – levels 2 to 4. Refer to Annex 9 for details of Kisumu County analysis. There is inadequate end-to-end visibility of the entire Procurement and Supply Chain Management (PSCM) systems particularly in the downstream supply chain. All the 6000 plus facilities send their ART stock requests to KEMSA each month. This data is not visible to NASCOP and to county managers. Every month NASCOP obtains reports from KEMSA and conducts verification and then allocates for re-supply. This information is however not visible to the county level managers and thus they cannot oversee, redistribute or control stocks in any way. Other gaps identified include:  Manual reporting and transportation of hard copies for transcription into the DHIS2 and LMIS leads to inaccuracies and lateness.  Poor integration of the different reporting platforms into the DHIS2 (for example the sms reporting developed by TUPANGE and the KEMSA LMIS).  Inadequate engagement and involvement of the private sector at county level despite them being key providers of FP, HIV and malaria services  Inadequate HR capacity for commodity management at RMHSU  Current RH supply chain/commodity management program focuses almost exclusively on FP and does not include other lifesaving maternal health commodities such as oxytocin, Magnesium Sulphate and Calcium Gluconate  There is inadequate advocacy for FP commodity budgets at both levels of government.  There is inadequate use of ICT in health commodity planning and management. This creates challenges of efficiency and data quality  Frequent shortages of basic data reporting tools  Lack of budgets at the national and county level to support basic commodity management functions including supportive supervision, commodity redistribution, tool printing and data review. These activities continue to rely heavily on donor funds.  Storage facilities at county, sub-county and facility level are woefully inadequate in terms of space and in some cases inappropriate in terms of the required environmental control. The assessment team encountered commodities stored at very high temperatures and humidity which can compromise product quality. 4.13 Other Findings The findings below are those that the evaluation team found relevant to the objectives of the evaluation yet are outside the evaluation questions 4.13.1 Comparison of Performance of HCSM Focus Vs Non HCSM Counties and Initial Focus counties and those that were added Later The evaluation team found the following differences between HCSM focus counties and those counties where HCSM had done little work especially in the last two years HSCM EOP Evaluation Report, March 2016 Page 32  HCSM focus counties had a more dynamic Commodity security TWGs  No systematic F&Q was conducted in non-HCSM counties. For example, Kiambu had done one F&Q as a district but no F&Q had been conducted after the formation of Kiambu County in 2013. The same applied to Machakos  There were greater challenges with availability of commodity management tools in non-HCSM supported counties  Commodity dedicated supportive supervision and data quality review was not done in non￾HCSM supported counties  There was limited commodity management capacity building in non-HCSM counties. In Kiambu, a laboratory commodity management TOT was conducted in 2012, but after HCSM ceased activities in the county, the training was never rolled out. The evaluation team also further analysed project activities and results to see if there were any differences between Kilifi which was one of the 13 counties prioritized in 2013 and Elgeyo Marakwet that was added in the financial year 2014/15. Below are some of the differences noted:  The F&Q exercise in Kilifi county was comprehensive capturing all the health commodities while in Elgeyo Marakwet the F&Q exercise only covered priority health programs’ commodities  Kilifi had a better developed health commodity reporting system compared to Elgeyo Marakwet  Kilifi County had in place a well structured countywide redistribution of slow moving and short expiry health commodities while Elgeyo Marakwet was still grappling with enormous challenges in redistribution of health commodities particularly for far flung areas of the county.  There is greater use of ICT in commodity management in Kilifi compared to Elgeyo Marakwet  Kilifi was also ahead of Elgeyo Marakwet in other commodity management areas including infrastructural improvements, PV, MTCs, capacity building and coordination with FBOs and the private sector. 4.13.2 Rationale for Project’s Geographic Focus The evaluation team found that the revised geographic focus of HCSM was appropriate for addressing HIV and Malaria. Although the focus counties account for only 34% of national population, they are home to 55% of the people living with HIV nationally. This indicates that focusing on the 15 counties addressed a large part of the national HIV burden and the associated commodity needs. 11 out of 15 (73%) of the focus counties are classified as Malaria endemic again showing that in terms of Malaria burden, the geographic selection of counties is justifiable. However, the rationale for selecting the 15 counties for FP commodity strengthening is weak. This is because FP needs do not exhibit a similar geographic polarity that is seen with HIV and Malaria. Indeed 6 out of the 15 focus counties have a contraceptive prevalence rate that is above the national average. There is also no strong rationale for including Kisii and Nyamira counties as they are not classified as Malaria endemic, do not have a high HIV burden and their contraceptive prevalence rates are above the national average HSCM EOP Evaluation Report, March 2016 Page 33 4.13.3 HCSM’s Technical Capacity and Approach Almost all national and county MoH respondents and partners interviewed were of the view that HCSM staff were highly committed and had high technical proficiency. It was reported that project staff had interacted well with their national and county counterparts, exhibited a high degree of professionalism and engaged and consulted their counterparts adequately. It was however pointed out that HCSM had not succeeded in replacing some of their technical staff in Malaria and FP that exited the project with staff of equivalent experience and expertise and that this eroded the project’s ability to influence policy Some of the county staff felt that the regional HCSM coordinators that supported multiple counties struggled to respond to the counties’ technical demands in a timely fashion. 4.13.4 Role of Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) The evaluation team reviewed the roles of the Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) in supporting commodity management and explored their possible inclusion in a future commodity management program design. The roles of the Public Procurement Regulatory Authority include monitoring, assessing and reviewing the public procurement and asset disposal system. The authority also has the role of setting standards and providing technical assistance. Kenya Institute of Supplies Management (KISM) is a membership organization for procurement and supply management practitioners. The Institute is established and operates as a corporate body promoting learning, development of best practices, and application of the same to the practice of procurement and supply chain management. Some of the evaluation team’s findings on the two institutions as potential partners in a health commodity management program include:  Both KISM and the Public Procurement Regulatory Authority primarily focus on procurement. For example the18-part Public Procurement and Asset Disposal Act, has only one short part – Part XIII that deals with inventory control, asset and stores management and distribution. All the other parts primarily address themselves to procurement issues. The bulk of complex public sector health commodities procurement is done by KEMSA or by partners. At the county level the bulk of health commodity procurement is at a fairly basic level involving primarily the placing of orders with KEMSA and paying for those orders. The national Priority Health Programs also have minimal involvement in commodity procurement beyond forecasting and quantification.  Current KISM training programs mainly target high level supply management practitioners and again primarily address procurement training areas.  As a result there is a disconnect between what counties’ health departments and health facilities need which is broad-based capacity building to support planning for and managing commodities and what the Public Procurement Regulatory Authority and KISM focus on which is high level procurement systems and capacity building. Country Led Approach “I have not had any bad experience working with HCSM. They have not forced anything on us. This has been a program that has truly been government led and owned” – Priority Health Program Respondent HSCM EOP Evaluation Report, March 2016 Page 34 5. Conclusions 5.1 Overall Project Performance The evaluation team is of the view that HCSM has done quite well at a time of rapid change and in an unsettled health service delivery environment coupled with significant midstream refocusing of its own mandate. The project achieved most of the targets it set. Refer to annex 8 on HCSM’s performance against indicators. Perceptions of its performance were quite high with national and county MoH respondents and relatively lower with USAID informants. The program’s impact appears to have been highest at the national and county level, going down at the sub-county and lower levels. This is most likely because the project largely intervened at the national and county level. At the national level, the project was very successful with the priority health programs especially HIV and Malaria, and also with its interventions at the Pharmacy and Poisons Board. Notable successes were the establishment of commodity security TWGs at national and county level, support for F&Q, national level PV and PMS, improved commodity reporting rates and support for electronic commodity management tools. Some of the areas where the program was less successful include establishment of facility MTCs, the model site concept and introduction of a mobile app for accessing treatment guidelines 5.2 Project’s Impact on National Level Commodity Management Systems The project has built significant commodity management capacity and strengthened systems at the national level. The establishment of commodity security TWGs for the HIV, Malaria and FP national programs has created an effective and sustainable mechanism for national level commodity management including forecasting and quantification, improved commodity reporting, strengthened information management systems and enhanced capacity building capability. The migration of commodity reporting from multiple vertical systems to DHIS2 has been completed for Malaria and FP commodities and is underway for HIV commodities. There have been significant national level system improvements for HIV and Malaria commodities but progress has been slower for FP commodities. At the national level there has also been significant strengthening of pharmacovigilance and post marketing surveillance through the support extended to the Pharmacy and Poisons Board. Work done with the Pharmaceutical Services Unit of the national MoH in developing national policy guidelines for commodity management and pharmaceutical services has been less successful due to challenges of effective dissemination and follow-up. This has in turn reduced the intended trickle down of improved commodity management practices and systems to counties that HCSM does not support. 5.3 Project’s Impact on County Level Commodity Management Systems The commodity management capacity of counties supported by HCSM has grown rapidly almost from scratch. The establishment of the county commodity security TWGs has constituted an excellent platform on which counties have strengthened commodity management systems and practices. Key successes at the county level include annual forecasting and quantification, use of F&Q data to lobby for enhanced health commodity budgets, supportive supervision and improvement in record keeping and storage infrastructure and systems. Another notable success HCSM Program Effectiveness “This is the only partner (HCSM) that has not constructed anything yet their impact is being felt” Sub-County Pharmacist HSCM EOP Evaluation Report, March 2016 Page 35 at the county level compared with the national level is the growth of an integrated commodity management approach. 5.4 Implementation Challenges and Gaps The project faced a number of significant implementations challenges that affected the discharge of its mandate. Some of the challenges include the very broad geographic and programmatic mandate the project had at its inception. Significant midstream changes on the project’s focus made necessary by devolution created further challenges. Implementation of devolution was accompanied by large staff movements at both national and county level that affected institutional memory and program continuity. Lack of a clear coordination framework between HCSM as a national mechanism and USG implementing partners constituted another challenge. A number of gaps continue to plague commodity management at national, county and facility level. Stock-outs of health commodities are still prevalent and especially for commodities that are not procured by partners. Multiple commodity management and reporting systems remain in use due to uncoordinated partner and government decision making. In some cases such as the KEMSA LMIS, commodity data is not accessible to county and national level managers. A notable gap remains the limited use of electronic systems to manage commodities. Most commodity management processes especially at county and facility level are manual leading to challenges with reporting and timely re-stocking. Another challenge is lack of budgets to support commodity management beyond the cost of procuring commodities. This includes lack of budgets to print reporting tools, train staff and undertake supportive supervision. 5.5 Sustainability of HCSM Interventions Many of the national level interventions are fairly sustainable given that most national level institutions have had their capacity built over many years and most of these organizations such as the priority health programs and the Pharmacy and poisons board are relatively well funded. Interventions at the county level are currently not sustainable given the very short history of support and the fact that commodity planning and management responsibility is new to county governments. The new Global Fund grant is supporting a number of commodity management areas at both national and county level. The areas covered include migration of commodity reporting to DHIS2, tool printing and commodity management capacity building. The Global Fund grant should therefore confer some level of sustainability post HCSM. HSCM EOP Evaluation Report, March 2016 Page 36 6. Lessons with Potential for Scale-up and informing Future Direction Evaluation Question: In light of the evaluation findings, what lessons can be identified to inform scale￾up of successful interventions or be applied for future projects? This section addresses the above question which was originally question 13. The table below is a summary of lessons learnt with potential for scale-up and informing future direction. Focus Area Lessons Learnt 1. Commodity computerization and Reporting  Use of electronic commodity management systems such as EDITT enhances operational efficiencies and minimizes data errors  There is need to adopt common commodity management and reporting ICT platforms  Commodity reporting should be done on DHIS2 to increase information access and make it easier to compare commodity and health services data 2. Commodity Management  A number of HCSM initiatives such as support for commodity security TWGs, F&Q and capacity building have significantly improved commodity management at national and county level 3. Pharmacovigilance, post marketing surveillance and patient safety  Improved PV and PMS at national, county and facility level is essential to support rational drug use and ensure patients’ safety 4. Coordination of multiple agencies involved in lab commodities  There is need to improve coordination of organizations with a role in lab commodity management including National Public Health Laboratory Services, KMLTTB, KEMSA, Priority Health programs and county MoH departments 5. Linkages between national and county levels  There is need for strong linkages between the national and county level to support policy development, dissemination and implementation 7. Recommendations and Future Direction This being an End of Project (EOP) evaluation, the proposed recommendations primarily address issues of future program design. These are elaborated in the Concept Note that the evaluation team has developed for USAID Kenya. Below are the recommendations proposed by the evaluation team: Table: List of Recommendations Finding Recommendations 1. Many commodity management processes (for all health commodities including lab commodities) including ordering, receipt, issues and reporting are manual. Further, where electronic data management systems exist, multiple platforms are in use with different capabilities. For example, ADT is only capable of handling ARVs at the dispensing level  Enhance the use of integrated electronic inventory and dispensing tools at all levels of health service delivery. These tools should use a common platform and link with other health information systems HSCM EOP Evaluation Report, March 2016 Page 37 2. A number of commodities including HIV and lab commodities use vertical reporting systems that are not accessible to most health and program managers  All commodity reporting should be migrated and integrated to DHIS2 3. Medicines and Therapeutic Committees (MTCs) and patient safety interventions have not taken root at county and facility level  Set up/reactivate facility MTC's to drive improvement in rational drug use and patient safety including strengthening the availability and use of standard treatment guidelines. 4. The impact of HCSM has mainly been at the national and county level. There has been much less impact at sub-county and lower levels  Develop a mechanism to ensure that commodity management interventions reach the sub-county and lower levels including primary health facilities  Continue strengthening county commodity security TWGs to support county wide commodity management improvements 5. The linkages between national programs and county MoH are extremely weak and this has negatively impacted policy formulation, dissemination and implementation. In addition, the PHPs still manage most of their functions centrally even when many tasks such as commodity planning and allocation is best done at the county and facility level  Support the Intergovernmental Committee on Health and its respective TWG's to improve harmonization and coordination between national and county level and between programs in respect to the planning and management of health commodities  Support dissemination and follow-up of implementation of key pharmaceutical services and commodity management policy guidelines  Support greater devolution of PHPs’ commodity management functions 6. Existing information products on health commodities focus primarily on health workers and not consumers  Strengthen consumer education and awareness and introduce consumer surveys and consumer reports. 7. Many commodity management interventions and improvements especially at the national level remain largely vertical  Design greater integration and adopt more health systems approach into future health commodity strengthening programs 8. GOK financial contribution to procurement of health commodities such as FP, HIV and Malaria that have traditionally been supported by donors remains sub-optimal. In addition, many commodity management functions such as printing of reporting tools and capacity building are not budgeted for at both national and county level  Advocate for increased GOK financial contribution at national and county level to support health commodities procurement and management 9. The Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) play a critical role in setting standards and building the capacity of commodity management practices in Kenya  Explore engagement of the Public Procurement Regulatory Authority and Kenya Institute of Supplies Management (KISM) in strengthening health commodity procurement and inventory management. HSCM EOP Evaluation Report, March 2016 Page 38 8. Annexes Annex 1: Guiding Evaluation Questions 1. To what extent has the project strengthened commodity security systems at national level? 2. To what extent has the project strengthened overall health commodity management systems at sub-national level i.e. county and peripheral health facilities? 3. How has the implementation of HCSM improved co-ordination and harmonization of the activities of the various players (donors & IPs) in the health commodity supply chain arena? 4. Has the project achieved its objective of improving stewardship and leadership for commodity management at sub-national level? 5. What trends in availability of and reporting rates for key health commodities have been observed overtime, and to what extent has the project contributed to improvements in the trends? 6. To what extent has the HCSM project improved the pharmaceutical sub-sector governance and the delivery of pharmaceutical services? 7. To what extent has the implementation of the HCSM project improved systems for promoting the rational/appropriate use of medicines, product quality assurance and patient safety? 8. Has the HCSM project achieved the objective of strengthening the supply chain for laboratory commodities? 9. In implementing activities, has capacity been built within MoH (National & County) and has there been a transition plan to host country ownership for sustainability? 10.What are the documented best practices and innovations, both in terms of technical contributions and in systems strengthening approach, by the project during its period of implementation? 11.What challenges has the project faced in activity implementation and how can these be addressed? 12.What gaps still exist within the country’s health commodity management and pharmaceutical systems? How could these gaps be addressed in future? 13.In light of the evaluation findings, what lessons can be identified to inform scale-up of successful interventions or be applied for future projects? HSCM EOP Evaluation Report, March 2016 Page 39 Annex 2: List of documents to be reviewed Below are the documents that will be reviewed by the evaluation team:  Annual and quarterly project reports  Assessment reports  Award documents  Commodity management documents including tools and guidelines  County documents including tools, guidelines, MoUs, work plans and strategic plans  Laboratory supply chain documents  Project M&E documents including plans, PMP and data quality assessment reports  Pharmaceutical policy and services documents including o MTC guidelines o Pharmacovigilance guidelines and reports o CPD guidelines o Kenya National Pharmaceutical Policy o Standard Treatment Guidelines  Project management documents including work plans and financial reports  HCSM success stories HSCM EOP Evaluation Report, March 2016 Page 40 Annex 3: Key Informants Questionnaire Guidance for Interviewers Consider relevance and applicability of all questions to specific key informants. Skip questions not relevant to respondent being interviewed You may add probing and clarifying questions where appropriate during the actual interview. A) Opening Question How have you worked /interacted with MSH/HCSM? B) Evaluation Questions Evaluation Question 1: To what extent has the project strengthened commodity security systems at national level? - (IR1.1) (Respondents: National respondents including PSU, PHPs, KEMSA, CHAK, KCCB, UON School of Pharmacy, KMTC, Partners) 1.1 In which ways has HCSM strengthened national program commodity management? (Probe about the National Commodity Security TWG, commodity quantification, tool development and dissemination, pipeline monitoring, forecasting, commodity information management - DHIS2/EDITT, commodity resource mobilization) 1.2 What evidence is there that commodity security has been strengthened? 1.3 In which specific ways has the project strengthened commodity management in the FBO/private sub-sector? 1.4 How helpful or otherwise has the HSCM training been? Comment on the TOT programs. 1.5 To what extent has commodity management been integrated to pre-service training (For KMTC and UON, School of Pharmacy)? What are early results? 1.6 What HCSM interventions have been particularly helpful? 1.7 Are there specific successes/issues for the national level commodity management by program area – HIV/TB, Malaria, FP/Reproductive Health? Evaluation Question 2: To what extent has the project strengthened overall health commodity management systems at sub-national level i.e. county and peripheral health facilities? – (IR1.3) (Respondents: County, health facility, county level partners) 2.1 In which ways has HCSM strengthened county commodity management? (Probe about the County Commodity Security TWG, commodity quantification, tool development and dissemination, commodity information management, commodity resource mobilization) 2.2 What evidence is there that commodity security has been strengthened? 2.3 In which specific ways has the project strengthened commodity management in the FBO/private sub-sector? 2.3 What HCSM interventions have been particularly helpful? 2.4 Are there specific successes/issues for the county commodity management by program area – HIV, Malaria, FP/Reproductive Health? 2.5 Are there specific issues/successes for county commodity management in general from a HSS perspective? Evaluation Question 3: How has the implementation of HCSM improved co￾ordination and harmonization of the activities of the various players (donors & IPs) in the health commodity supply chain arena? (IR2) HSCM EOP Evaluation Report, March 2016 Page 41 (All respondents) 3.1 In which way has the HSCM improved coordination and harmonization across various players (GoK, FBOs and partners) - cite some examples? (Probe - stakeholders’ forums, has the commodity Security TWG helped involve non-state stakeholders, collaboration with FBO sub￾sector) 3.2 How has the above contributed to an improved supply chain and commodity management? 3.3 How has HCSM supported collaboration between the county MoH and national MoH on commodity management? 3.4 How has the Project collaborated with USG and non-USG partners in strengthening commodity management? 3.5 How can coordination and harmonization be improved in a future program? Evaluation Question 4: Has the project achieved its objective of improving stewardship and leadership for commodity management at sub-national level? (IR 1.3) (All Respondents) 4.1 How well or otherwise are County Commodity Security TWGs functioning? 4.2 How helpful or otherwise has the HSCM training been in improving county and facility level capacity for commodity management? Comment on the effectiveness of the TOT program. 4.3 How useful was the recent county commodity management forum? Evaluation Question 5: What trends in availability of and reporting rates for key health commodities have been observed overtime, and to what extent has the project contributed to improvements in the trends? (Respondents: PHPs, County and Facility) 5.1 What has been the trend in the availability of key health commodities? 5.2 What has been the trend in reporting rates for key health commodities? 5.3 In which way has HCSM contributed to trends in commodity availability and reporting rates? 5.4 What feedback do you have on the reporting tools and information system for health commodities? Evaluation Question6: To what extent has the HCSM project improved the pharmaceutical sub-sector governance and the delivery of pharmaceutical services? (IR2.1 and IR 2.2) (Respondents: National, county and facility respondents as relevant) 6.1 Comment on the effectiveness of the following HCSM interventions in the improvement of pharmaceutical sector governance and delivery of pharmaceutical services (Probe on awareness about, usefulness and involvement of wider stakeholders in the interventions) 6.1.1 Review of the Kenya Essential Medicines List (KEML) and the Development of the Kenya Essential Medical Supplies List (KEMSL) 6.1.2 Support to National medicines Therapeutic Committee including review of the Kenya Clinical Guidelines 6.1.3 Finalization and dissemination of the Kenya National Pharmaceutical Policy (KNPP)/ sessional paper No 4 6.1.4 Development of strategic and other plans for professional associations (PSK and KAPI) (Ask question to PSK and KPA only) 6.1.5 Development of Public Service pharmaceutical charters 6.1.6 Establishment of Medicines and Therapeutics Committees in hospitals 6.1.7 Access of clinical guidelines through mobile phones 6.1.8 Support supervision for health commodities 6.1.9 County pharmacists’ forums 6.1.10 Quality of care surveys HSCM EOP Evaluation Report, March 2016 Page 42 Evaluation Question 7: To what extent has the implementation of the HCSM project improved systems for promoting the rational/appropriate use of medicines, product quality assurance and patient safety? (IR2.3) (Respondents: PPB, UON School of Pharmacy, NASCOP and Facility level respondents as relevant) 7.1 In which way has HCSM improved systems for promoting the following? A) Rational/appropriate use of medicines B) Product quality and patient safety? 7.2 What challenges have you faced in promoting the rational/appropriate use of medicines, product quality and patient safety and what are the gaps Evaluation Question 8: Has the HCSM project achieved the objective of strengthening the supply chain for laboratory commodities? (IR 3.2) (Respondents: County, PHPs, Chief Med Lab Technologist, Board registrar, Professional association, NPHLS, FBOs, Facilities Partners) 8.1 To what extent has HCSM succeeded in strengthening the supply chain for laboratory commodities? Cite some examples 8.2 In what specific ways have the above successes improved HIV/TB programs? Evaluation Question 9: In implementing activities, has capacity been built within MoH (National & County) and has there been a transition plan to host country ownership for sustainability? (Respondents: HCSM, National, County, PHPs, Partners) 9.1 To what extent has capacity to manage commodities been build at the National MoH? Cite some examples 9.2 To what extent has capacity to manage commodities been build at the County MoH? Highlight major capacity gaps 9.3 Which significant gaps remain in the capacity of National and county MoH to manage commodities? 9.4 Has there been a transition towards a country ownership of commodity management for sustainability? Has this been guided by a plan? 9.5 What sustainability issues persist? Evaluation Question 10: What are the documented best practices and innovations, both in terms of technical contributions and in systems strengthening approach, by the project during its period of implementation? (Respondents: All) 10.1 What best practices and innovations have been implemented with the support of HCSM in the following areas? Commodity Management at national, county and Facility Level Pharmaceutical Services governance, policy and delivery Laboratory Services strengthening at national, county and Facility Level PHP – HIV. Malaria and FP Integration of training and tools Other HSS areas 10.2 Have these best practices been documented and disseminated Evaluation Question 11: What challenges has the project faced in activity implementation and how can these be addressed? (Respondents: All) HSCM EOP Evaluation Report, March 2016 Page 43 11.1 What implementation challenges has the HCSM Project faced? (Such as Technical operation, regulatory/policy, ownership) 11.2 What have been the contributory factors to these challenges? 11.3 Highlight specific challenges by PHP – HIV, Malaria, and FP/RH 11.4 How can these challenges be addressed? Evaluation Question 12: What gaps still exist within the country’s health commodity management and pharmaceutical systems? How could these gaps be addressed in future? (Respondents: All) 12.1 What gaps still exist within the country’s health commodity management (including lab commodities) and pharmaceutical systems at the following levels? For each of these suggest how these gaps could be addressed in future 12.1.1 National MoH 12.1.2 County MoH 12.1.3 Health Facilities 12.1.4 Priority Health programs – HIV, Malaria, FP/RH 12.1.5 FBO/NGO/Private sub-sector 12.1.6 PPB and Pharmaceutical professional associations 12.1.6 Health Training Institutions 12.1.7 KEMSA 12.1.8 Other 12.2 Comment on the adequacy or otherwise of the geographic coverage of the HCSM Project. Specify for each priority health program – HIV, Malaria, FP/RH 12.4 In addressing existing gaps in commodity management, which functions/activities are best carried out at the county level and which are better done at the national level? Give reasons? 12.3 How should a future program targeting the above gaps be structured in respect to geographic and programmatic scope? Evaluation Question 13: In light of the evaluation findings, what lessons can be identified to inform scale-up of successful interventions or be applied for future projects? Output will come from evaluation findings HSCM EOP Evaluation Report, March 2016 Page 44 Annex 4: County Commodity Security Technical Working Group FGD Tool FGD – County Commodity Security TWG Members Introduction by Facilitator Hello, my name is [facilitator name] Thank you for taking the time to participate in this focus group discussion. It is part of a larger end￾term evaluation of the MSH-HCSM Project, a national health initiative implemented in Kenya from April 2011 to March 2016. The MSH-HCSM supports the Kenya Government’s efforts in building systems to ensure health commodities security. The evaluation’s objectives are to find out what has worked well, what hasn’t, the achievements to date, and any impediments in the implementation of the project. It also aims to determine what needs improving, and how, so that future programs can achieve their anticipated results. This focus group discussion is being conducted to learn about the trainings that were offered by the HCSM project and their applicability in your work. The ground rules for participation in the focus group discussion are as follows: First, confidentiality - what we talk about today will not be attributed to anyone but to the discussion as a whole and there will be no mention of individuals or quotes attributed to an individual in the notes taken. Second - Please turn off your cell phone, or if this is not possible set it on silence mode. If you have to take a call, please step outside the room. Third - Please respect the opinions of others. When responding you can mention that you disagree, politely, and state your opinion. Let others finish before you start speaking. This discussion may move from question to question in no particular order and that’s okay. Please feel free to participate as much as possible, there are no right or wrong opinions because your experience is not the same as anyone else’s experience. Fourth - Your attendance is voluntary and you may leave the discussion at any time. Those are the ground rules unless there is any participant who feels that an important rule has been omitted. We will start by introducing ourselves - mention your name; professional cadre; and the facility you work in. Discussion Questions and Issues 1. How have you worked with MSH –HCSM? In which ways has HCSM strengthened county commodity management? How has the County Commodity Security TWG operated? HSCM EOP Evaluation Report, March 2016 Page 45 What difference has the County Commodity Security TWG made? What evidence is there that commodity security has been strengthened? What HCSM interventions have been particularly helpful? 2. Are there specific successes/issues for the county commodity management by program area – HIV, Malaria, FP/Reproductive Health 3. Which significant gaps remain in the capacity of the county MoH to manage commodities What implementation challenges has the HCSM Project faced? (Technical operation, regulatory/policy, ownership) 4. What have been the contributory factors to these challenges 5. What gaps still exist within the county’s health commodity management (including lab commodities) and pharmaceutical systems 6. What changes would you propose for any new USAID program intended to strengthen commodity management Thank you so much for your time! HSCM EOP Evaluation Report, March 2016 Page 46 Annex 5: HCSM Training Recipients FGD Tool FGD – HCSM IN-SERVICE TRAINING RECIPIENTS Introduction by Facilitator Hello, my name is [facilitator name] Thank you for taking the time to participate in this focus group discussion. It is part of a larger end￾term evaluation of the MSH-HCSM Project, a national health initiative implemented in Kenya from April 2011 to March 2016. The MSH-HCSM supports the Kenya Government’s efforts in building systems to ensure health commodities security. The evaluation’s objectives are to find out what has worked well, what hasn’t, the achievements to date, and any impediments in the implementation of the program. It also aims to determine what needs improving, and how, so that future programs can achieve their anticipated results. This focus group discussion is being conducted to learn about the trainings that were offered by the HCSM Project and their applicability in your work. The ground rules for participation in the focus group discussion are as follows: First, confidentiality - what we talk about today will not be attributed to anyone but to the discussion as a whole and there will be no mention of individuals or quotes attributed to an individual in the notes taken. Second - Please turn off your cell phone, or if this is not possible set it on silence mode. If you have to take a call, please step outside the room. Third - Please respect the opinions of others. When responding you can mention that you disagree, politely, and state your opinion. Let others finish before you start speaking. This discussion may move from question to question in no particular order and that’s okay. Please feel free to participate as much as possible, there are no right or wrong opinions because your experience is not the same as anyone else’s experience. Fourth - Your attendance is voluntary and you may leave the discussion at any time. Those are the ground rules unless there is any participant who feels that an important rule has been omitted. We will start by introducing ourselves - mention your name; professional cadre; and the facility you work in. Discussion Questions and Issues 1. Tell us which of the MSH – HCSM supported in-service training you attended. (and whether it was for ToTs) 2. How relevant was the training to your work? 3. What are your views about the quality of the training you received in respect to: HSCM EOP Evaluation Report, March 2016 Page 47 a) content, b) trainers, c) training materials, d) course delivery, and e) course venue? 4. Let us know in which way the training has been useful in respect to the following: a) Enhanced skills for you to perform your tasks? b) Ability to offer on-the-job training to other colleagues? c) Others? Specify_________ 5. What do you think could be done to improve the quality of the training? 6. What other comments do you have about the in-service trainings that would be important for future design of trainings? Thank you so much for your time! HSCM EOP Evaluation Report, March 2016 Page 48 Annex 6: USAID Respondents Questionnaire BROAD QUESTIONS FOR USAID 1. In your opinion to what extent has the HCSM Project met its mandate in the 3 strategic objective areas a) Strengthened Commodity Management within MoH and peripheral facilities for effective service provision: b) Improved Pharmaceutical Policy and Service Delivery for Effective Provision of Health Care to Clients c) Strengthened Laboratory systems and service delivery 2. In your opinion what are the outstanding issues in commodity management at National, County and Facility levels. 3. Share ideas on future direction of USAID support in the area of commodity management and pharmaceutical service delivery systems in terms of: a) Programming b) Geographic scope c) National vs County mechanism / focus HSCM EOP Evaluation Report, March 2016 Page 49 Annex 7: List of Evaluation Respondents # Name Designation Organization Partners 1. Biwott Kevin Director EMC -HEALTH ELGEYO MARAKWET 2. Dickson Mwakangala Technical Director PATH FINDER INT MOMBASA 3. Lucy Matu Dts EGPAF NAIROBI 4. Mboya Dennis Ta FACES KISUMU 5. Chris Forshaw Senior Pharmaceutical Advisor DANIDA NAIROBI 6. Davis Karambi Associate Director – Access Programs CHAI NAIROBI 7. Paul Mwaniki President PSK NAIROBI 8. Sybil Nakitare Clinical Coordinator CHAK 9. Dan Okoro Program Analyst UNFPA 10. Mukabi James COP APHIA PLUS WESTERN KISUMU 11. Nancy Aloo SDO MCSP - JHPIEGO KISUMU 12. Milka Cho Health Advisor DFID 13. Yasmin Chandani Project Director In Supply JSI 14. James Riungu JSI 15. Jane Mwangi Lab CDC(KE) CDC 16. Mildred Shieshia Resident Malaria Advisor USAID / PMI 17. Sheila Macharia Senior Health Advisor USAID K AND EA 18. Alice Micheni PMS Logistics & Facilities USAID K 19. Stanley Bii USAID 20. Alex Kinoti USAID 21. James Odek USAID 22. Daniel Wacira Prog. Mgt. Specialist USAID/PMI 23. Ann Buff CDCRA PMI CDC National Public Sector Institutions 24. Bernard Sande SACMLT NATIONAL PUBLIC HEALTH LABORATORY 25. Jonah Maina Program Manager REPRODUCTIVE & MATERNAL HEALTH SERVICES UNIT 26. Guantai A.N. Dean UON – SCHOOL OF PHARMACY HSCM EOP Evaluation Report, March 2016 Page 50 # Name Designation Organization 27. Faith Okalebo Senior Lecturer UON – SCHOOL OF PHARMACY 28. Margaret Oluka Senior Lecturer UON – SCHOOL OF PHARMACY 29. Eunice Mutemi TA Nutrition NASCOP 30. Susan Njogu Program Manager NASCOP 31. Roseline Warutere Program Officer NASCOP 32. Caroline Olwande PM GF NASCOP 33. Waqo D. Ejarsa Head Nmcp NMCP 34. Samwel Kigeni Program Officer NMCP 35. Rebecca Kiptui Program Officer NMCP 36. Memusi D. Naisae Program Officer NMCP 37. Fred Siyoi Deputy Registrar PPB 38. Edward Abwao ACP PPB 39. Josphat Mbuva SCDP PSU 40. Sarah Chuchu Senior Deputy Chief Pharmacist PSU 41. J. Munyu CEO KEMSA 42. Rose Njuguna Senior Lecturer KMTC - NAIROBI 43. Phoebe Kigundu Lecturer KMTC - NAIROBI 44. D. Kirubu Lecturer KMTC - NAIROBI 45. Richard Muthama ACP NLTD - P County Level Respondents – Kilifi 46. Bilali Yusuf Mazoya CDH MOH - KILIFI 47. Humphrey Mundu Chief Lab Tech County Commodity Security Twg 48. Clare Obonyo Assistant Chief Pharm County Commodity Security Twg 49. Esther Mwema Chief Registered Nurse County Commodity Security Twg 50. Patrick Mugao Makazi Chief Lab Tech County Commodity Security Twg 51. Ronald Mbunya Principal Nutritionist County Commodity Security Twg 52. Cecilia Wamalwa Assistant Chief Pharm County Commodity Security Twg 53. Lucas Mwero Co Matsangoni Health Centre 54. Annete Mbucho Mlt Matsangoni Health Centre 55. Patience Pendo Pharm Tech Matsangoni Health Centre HSCM EOP Evaluation Report, March 2016 Page 51 # Name Designation Organization 56. Khadoa Mbua Nutritionist Training Recipients - Kilifi County Referral Hospital 57. Olive Jefwa Stores Clerk Training Recipients - Kilifi County Referral Hospital 58. Patrick Chibungu Stores Clerk Training Recipients - Kilifi County Referral Hospital 59. Titus Mwamuganga Storeman Training Recipients - Kilifi County Referral Hospital 60. Peter Sifa Mwanyonyo Stores Assistant Training Recipients - Kilifi County Referral Hospital 61. Mlongo M. Atiki Spho Training Recipients - Kilifi County Referral Hospital 62. Sylvestor J. Mwango Procurement Supplies Training Recipients - Kilifi County Referral Hospital 63. Patience M. Ndiso Deputy Chief Pharm Tech Training Recipients - Kilifi County Referral Hospital 64. Pamela C. Kabibu No I/C For Med Sup Kilifi County Referral Hospital 65. Raphael Kalama Clmt Kilifi County Referral Hospital County Level Respondents - Kisumu 66. Onyango D. Cdh Moh - Kisumu 67. Maurice Ouma Mlt St. Elizabeth Chiga Mission Hospital 68. Joel Waudah Pharm Tech St. Elizabeth Chiga Mission Hospital 69. Josephine Makaya Pharm Tech St. Elizabeth Chiga Mission Hospital 70. Elizabeth Donde No I Kodiaga Prisons Health Centre 71. Jayne Odada Mlt Kodiaga Prisons Health Centre 72. Naomi Jelagat Pharm Tech Kodiaga Prisons Health Centre 73. Sarah Ogola Rco Kodiaga Prisons Health Centre 74. Kevin Otieno Pharm Tech Training Recipients – Nyalenda Hc 75. Festus Ogada Pharmacist Training Recipients – Ahero Ch 76. Teresa Okiri Sno Training Recipients – Kisumu Crh 77. Rose Maoga No Training Recipients –Miranga Sch 78. Alfred Omullo Mlt Training Recipients –Kisumu East 79. Rosemary Jagongo Mlt Training Recipients – Kisumu West 80. Alice Adoma No Training Recipients – Jootrh 81. Obala Neto Pharmacist Training Recipients – Kisumu West 82. Kennedy Orure Dcmlt / Mlt County Commodity Security Twg 83. Isaiah Ogwalo C Pharm T County Commodity Security Twg HSCM EOP Evaluation Report, March 2016 Page 52 # Name Designation Organization 84. Perez Akello Sno County Commodity Security Twg 85. Ogollah Hellen Cmlc County Commodity Security Twg 86. Otieno Lawrence Acp County Commodity Security Twg 87. Nyang’wara Leon Dcasco County Commodity Security Twg 88. Lilyana Dayo Cmcc County Commodity Security Twg 89. Emmah Obegi Pharmacist Kisumu County Referral Hospital 90. Benjamin Nyangema Lab Technologist Kisumu County Referral Hospital County Level Respondents - Kakamega 91. Elpharet Agiso Nurse Bukura Health Centre 92. Cynthia Rajula Hrio Bukura Health Centre 93. Felix Amalia Co Bukura Health Centre 94. Shiundu Richard Martin Pharm Tech Bukura Health Centre 95. Stephen Imanya Mlt Bukura Health Centre 96. Zacharia A. Okiya Medical Lab Tech Kakamega County Referral Hospital 97. Babra Murila Pharmacist Kakamega County Referral Hospital 98. Watamba Michael Pharmacist Kakamega County Referral Hospital 99. Emisiko James Smlt Kakamega County Referral Hospital 100. David Oluoch Ag. Cdh County Director Of Health Rgd 101. Faustina Sakavi Cmcc County Director Of Health Rgd 102. Emisiko James Smlt County Director Of Health Rgd 103. Jonathan Majan Cmcl County Director Of Health Rgd 104. Michael Ruto Hrim County Director Of Health Rgd 105. Emmanuel Otiko Pharm Tech Training Recipient - Kakamega County Referral Hospital 106. Joan Kadesa Pharmacist Training Recipient - Kakamega County Referral Hospital 107. Sarah Nandwa Mlt Training Recipient - Kakamega County Referral Hospital 108. Faustina Sakavi Cmcc County Commodity Security Twg 109. Charles Muyekho Sno County Commodity Security Twg 110. Madson Malongo Sno County Commodity Security Twg 111. Josina Sikolia Sc.Nut.O County Commodity Security Twg 112. Violet Osundwa Sno County Commodity Security Twg HSCM EOP Evaluation Report, March 2016 Page 53 # Name Designation Organization 113. Emisiko James Smlt County Commodity Security Twg 114. Ruto Mike Hrim County Commodity Security Twg County Level Respondents – Elgeyo Marakwet 115. Michael Kipkoech Sco Chepkorio Health Centre - Keiyo South 116. Grace Kombech Sctlc Chepkorio Health Centre - Keiyo South 117. Yator Agnes Sc Pharmacist Chepkorio Health Centre - Keiyo South 118. Samson Keter Scmlt Chepkorio Health Centre - Keiyo South 119. Joseph Riwomoi Officer I/C/ Chepkorio Hc Chepkorio Health Centre - Keiyo South 120. Castro Mugalla Med Sup Iten County Referral Hospital 121. Luka Kiptarus Lab Technologist Iten County Referral Hospital 122. Violet Kiprop Pharm Tech Iten County Referral Hospital 123. Jonathan Tanui Crhc - Emc Iten County Referral Hospital 124. Luka Kiptarus Lab Technologist Training Recipients 125. Violet Kiprop Pharmaceutical Technologist Training Recipients 126. David Cheruiyot Clinical Officer Training Recipients 127. Josphat Maiyo Casco County Commodity Security Twg 128. Kiprop Gideon C. Pharmacist County Commodity Security Twg 129. Lydia Chemno Cno County Commodity Security Twg 130. Charles C. Kosgei Fh / Programs Coordinator County Commodity Security Twg 131. Maximilia Barasa No County Commodity Security Twg 132. Jacob Ayienda Acpho County Commodity Security Twg 133. Mike Koima Chrio County Commodity Security Twg County Level Respondents - Machakos 134. Nkatha Mutungi County Pharmacist County Commodity Security Twg 135. David N. Maundu Laboratory Manager County Commodity Security Twg 136. Stella Mutinda County Nutrition Officer County Commodity Security Twg 137. Philomena Muthoka County Commodity Nurse County Commodity Security Twg County Level Respondents – Kiambu 138. Christopher Kimaru CMLT County Commodity Security Twg 139. Lily Kimuhu County Pharmacist County Commodity Security Twg HSCM EOP Evaluation Report, March 2016 Page 54 # Name Designation Organization 140. Edward Ndungu County Nutritionist County Commodity Security Twg 141. Margaret Gathecha County Logistician County Commodity Security Twg 142. Rosalind Murugami Adns County Commodity Security Twg County Level Respondents – Migori 143. Gondi J.O. CDH Moh - Migori 144. Florence Akeyo County Nutritionist Moh – Migori 145. Boniface Olalo Lab Coordinator Migori County Referral Hospital 146. Stephen Oyugi Nutritionist Migori County Referral Hospital 147. Erick Omondi Pharmacist I/C Migori County Referral Hospital 148. Edwin Okemwa Pharmacist Migori County Referral Hospital 149. Wyclffe Onyango Pharm Tech Ogwedhi Health Centre 150. Carolyn A. Onyango CCHA Ogwedhi Health Centre - Faces 151. Faustine Adhiambo Pharmacist Uriri Sub County Hospital 152. John Paul Genga Pharm Tech Awendo Sub County Hospital 153. Obura Chrishan HRIO Awendo Sub County Hospital County Level Respondents – Siaya 154. Samuel Omondi CDH Moh - Siaya 155. Mary Wambura CTLC Moh - Siaya 156. Geoffrey Mwai Pharmacist Siaya County Referral Hospital 157. Douglas Okoto SCMLC County Commodity Security Twg 158. Benter A. Rieko SCPHN Moh - Ugenya 159. Carey Abuya Sc Pharmacist Moh – Alego Usonga 160. Felix Odhiambo Oloo Sc Pharmacist Moh - Ugenya 161. Nancy Olunga Ag. County Pharmacist County Commodity Security Twg 162. Solomon O.Onyango SCPHN Moh – Alego Usonga 163. William Mayi HRIO County Commodity Security Twg 164. Eunice Fwaya Pharmacist County Commodity Security Twg 165. Daniel Ojiambo MLT Sega Mission Hosp 166. Ann Wamaya PHARM TECH Sega Mission Hosp County Level Respondents - Homa Bay HSCM EOP Evaluation Report, March 2016 Page 55 # Name Designation Organization 167. Ruth Olango HRIO Marindi Health Centre 168. Victor Awino MLT Marindi Health Centre 169. Vivian Achieng Pharm Tech Marindi Health Centre 170. Susan Onyango Nutritionist Marindi Health Centre - Egpaf 171. Francis Aila CNC County Commodity Security Twg 172. Judith Niver A. Oyuta CMLC County Commodity Security Twg 173. Waringa Vincent DCHD County Commodity Security Twg 174. Okari Fredrick Pharmacist Homa Bay County Referral Hospital 175. Nancy Atieno Osewe Nutritionist Homa Bay County Referral Hospital - Egpaf 176. Barrack Odinda MLT Homa Bay County Referral Hospital 177. Isaiah Matundura Pharm Tech Homa Bay County Referral Hospital - Egpaf Annex 8: HCSM Performance against Key Indicators Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) SO 1: Strengthened MoH commodity management IR 1.1 Strong and effective MoH stewardship and technical leadership in supply chain management/commodity security Indicator 1.1.1: Functional priority programs (HIV, FP, Malaria, TB and Lab) commodity security committees at national level Non-functional As per the indicator definition, 4 National TWG including Malaria, HIV, FP and TB programs are functional. Have formal TORs, meet regularly (minutes available) and undertake national supply chain activities to ensure commodity security. TB dropped from HCSM support in 2014 Committees functional and mainstreamed into the MOH structure Indicator 1.1.2: Proportion of priority programs and key MoH departments [including NASCOP, DLTLD, DOMC, DRH, NPHLS] able to generate monthly commodity stock status reports Not institutionalized, Sporadic. The 3 TWGs under HCSM Support are functional, as denoted by the progress made in implementation of the planned activities. FP, Malaria, HIV programs and Laboratory have been developing 2 pagers and stock status as required to guide the various supply chain components. The TB program in no longer under HCSM support (since 2013/14) The 3 PHPs have conducted annual quantification and review consistently. Quantification of commodities (indicator 1.1.3) has been largely a success, with increased GOK commitments. A level of dependency still exists, with expectation for continued assistance from HCSM by MOH Commodity stock status reporting mainstreamed into the MOH structure PSU, key MoH programs, other MoH departments managing health commodities Indicator 1.1.3: Proportion of priority programs [including NASCOP, DLTLD, DOMC, DRH, NPHLS] and key MoH departments mentored by HCSM that are able to independently undertake commodity forecasting and quantification None Annual commodity forecasting and quantification independently undertaken by priority programs and key MoH departments Indicator 1.1.4: Percent difference between forecasted consumption and actual consumption for ARVs TDF+3TC+EF V Forecast error was not a practice prior to 2012. HCSM- designed forecast -- 3% 3% 0% <25% TDF+3TC+N -- 23% 15% 14% HSCM EOP Evaluation Report, March 2016 Page 57 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) VP error innovative tool AZT+3TC+N introduced in 2012 VP -- 12% 19% 17% AZT+3TC+EF V (pediatrics) -- 22% 34% 5% Indicator 1.1.5: Progress on a milestone scale in development of a functional harmonized national LMIS None National MoH lead LMIS TWG formed and functioning The envisaged LMIS deeded infeasible due to costs. HCSM led innovation – introduction of a platform within DHIS2 for commodity reporting. Piloted successfully in Malaria program DHIS2 currently institutionalized in Malaria, FP programs and the laboratory department for commodity reporting. HIV and TB in progress. Existence of functional national LMIS LMIS framework developed Existence of functional national LMIS IR 1.2: Effective coordination and harmonization of GoK and development partners’ activity in the sub-sector by the procurement and supply chain ICC (PSC-ICC) Indicator 1.2.1: Progress on a milestone scale in formation of national coordinating mechanism for health products, technologies and related services Revised Terms of Reference Due to devolution and reorganization of governance structures this has not been prioritized. PSC-ICC functional Developed annual calendar / Schedule of activities Meets regularly and disseminates meeting minutes Coordinating health commodity and services management activities IR 1.3 Peripheral healthcare facilities able to account for and manage commodities effectively Indicator 1.3.1: Proportion of health facilities submitting commodity usage reports to the central level for priority program commodities [ART, Malaria, TB, FP] ART: 84% [ordering points] 92% 97% 93% 97% 96% At least 90% FP: 51% [ stores] 52% 0% 13 % (DHIS2) 80% 86% At least 80% of health facilities submitting commodity usage Malaria: 62% [ordering points] 48% 66% 62% 66% 76% reports to the central level HIV Nutrition 2% 44% 63% HSCM EOP Evaluation Report, March 2016 Page 58 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) Indicator 1.3.2: Number of health workers trained in commodity management through USG programs 0 285 4557 1287 1336 1596 Capacitate regional and facility staff in commodity management in 210 districts/15 counties Indicator 1.3.3: Proportion of health facilities in priority regions having tracer commodities [on the day of the survey/support supervision visit, disaggregated by tracer commodity and sector(Public, Private, FBO)]sector(Public, Private, FBO)]) Essential médicines 14.1% DMPA 80% 91% Essential médicine 40% Non-pharms 35% (2013 : County baseline) (2015: National – QOC 9/10) At least 80% Non-pharms 8.8% TB Patient Pack 81% 87% Artemether / Lumefantrine (AL) tablets 20mg/120mg (24s) 80% 69% Specific tracers not assessed AZT/3TC/NVP 300mg/150mg/2 00mg FDC tabs 74% 81% Capsules Amoxicillin 250mg 80% 82% Tablets Paracetamol 500mg 79% 80% Tablets Cotrimoxazol e 480mg 81% 91% ORS 500ml Sachet 88% 80% Indicator 1.3.4: Percentage of USG assisted SDPs in priority regions reporting stock-out of more than 7 days for a set of tracer health commodities in the last 3 months [disaggregated by tracer commodity AZT/3TC/NVP 300/150/200 tab 4.8% 0% 2% 9% Less than 5% facilities reporting stock-out of more than 7 days for a set of tracer health commodities DMPA(FP): 26.4% 18% 15% 9% TB patient pack 22.9% 42% 6% 13% AL all sizes 25% AL 24s 8% 23% 23% Capsules Amoxicillin 250mg (Not 27% 27% HSCM EOP Evaluation Report, March 2016 Page 59 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) and sector(Public, Private, FBO)] Tablets Paracetamol 500mg assessed as It wasn’t part of the indicators then) 29% 25% Tablets Cotrimoxazole 480mg 22% 13% ORS 500ml Sachet 10% 25% Indicator 1.3.6: Proportion of health facilities in priority regions where physical stock and record counts are in agreement for selected tracer commodities (disaggregated by tracer commodities and sector(Public, Private(FBO)) AZT/3TC/NVP 300/150/ 200 tab (ART) 50.8% 34% 85% At least 80% of health facilities with physical stock and record counts in agreement DMPA(FP) 51.2% 40% 90% TB patient pack (TB) 51.9% 32% 85% AL all sizes (Malaria) 60.3% 53 % 80% Capsules Amoxicillin 250mg Not assessed 59% 81% Tablets Paracetamol 500mg 52% 81% Tablets Cotrimoxazole 480mg 55% 81% ORS 500ml Sachet 41% 74% Indicator 1.3.7: Proportion of health in priority regions facilities reporting to have received integrated supportive supervision visits within the past 3 months disaggregated by sector(Public, Private(FBO)) 40% 60% 69% 65% 80% 70% Indicator 1.3.8: Number of functional regional commodity security Non-existent O counties 13 14 Functional commodity security committees set up in all the HSCM EOP Evaluation Report, March 2016 Page 60 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) committees established (disaggregated by county and facility regions/15 counties IR 2.1: Strengthened Pharmaceutical sub-sector governance Indicator 2.1.1: Progress on a milestone scale in development of pharmaceutical strategy or KNPP and its implementation KNPP adapted by the cabinet. Draft revised KNPP available and awaited cabinet approval See detailed narrative Pharmaceutical strategy or KNPP implemented by at least 10% of The KNPP the outlined targets sessional paper #4 approved by parliamentary KNPP Strategy developed Indicator 2.1.2: Progress on a milestone in implementation of updated strategic plans for KPA and PSK KPA: 2009-2012 strategic plan PSK: Strategic plan exists (2009- 2014); No implementation plans PSK implemented over 80% of the priority activities outlined in the Strategic and Operational Plan Revised KPA and PSK strategic plans and implementation plans in place IR 2.2: Improved delivery of pharmaceutical services Indicator 2.2.1: Percentage of health facilities with the most current edition of Kenya National STGs and EML STGs EML Both 47.1% 57% 50% 74% 60% 90% 90% Indicator 2.2.2: Percentage of Malaria cases treated according to recommended treatment guidelines. 22% 27% 45% 46% 60% 60% Indicator 2.2.3: Percentage of medicines prescribed from the essential medicines list 94% Not tracked 95% HSCM EOP Evaluation Report, March 2016 Page 61 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) Indicator 2.2.4: Proportion of counties in the target regions with a functional Medicine and Therapeutic Committee Counties were non-existence 3 C/SC MTCs activated and operationalized (Busia, Unguja and Ukwala) 13 Hospital MTCs functional (See detailed narrative) At least 3 counties with functional medicine and therapeutic committee IR 2.3: Strengthened Medicines Quality Assurance and Pharmacovigilance Indicator 2.3.1: : Number of pharmacovigilance related report received at central level (disaggregated by type of report: ADRs and PQMR) ADRs: 1400 (Sept 2011) PMP: 175 (Sept 2011) 3175 159 1475 198 999 166 1163 (till Dec 15) 185 ADRs: 10000 PMP: 330 Indicator 2.3.2: Regulatory actions taken (during the reporting period) consequent on pharmacovigilance activities No data available 4 4 5 4 5 (target was 1 per annum) IR 2.4: Improved Pharmaceutical Information Acquisition and Management Indicator 2.4.1: Progress on a milestone scale in development and implementation of the PMIS framework  Review of PMIS situational report  PMIS indicators developed  PMIS Framework  Pharmaceutical Information Portal available to support decision making Done Done Done Initial milestones realized, but final implementation put on hold indefinitely due to effects of devolution which necessitated re￾prioritization at MOH level Pharmaceutical Information Portal available to support decision making IR 3.1 : Strengthened Laboratory Subsector Leadership and Governance Indicator 3.1.1: Existence of a functional laboratory commodity security committee that addresses issues related to lab commodities Inactive TORs and member ship Awaits formal approval from the MOH (Director of Medical services). Functional lab commodity sub￾committee HSCM EOP Evaluation Report, March 2016 Page 62 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) criteria defined IR 3.2: An Efficient and Effective Laboratory Supply Chain Indicator 3.2.1: Proportion of health facilities submitting commodity usage reports to the central level for lab priority commodities [RTKs and CD4) HIV Test Kits 50% CD4 50% 1% - 35% - 42% - HIV Test Kits 80% CD4 80% Figures based on DHIS2. Alternative platform exists in NASCOP Indicator 3.2.2: Proportion of facilities in priority regions reporting stock-out of more than 7 days for a set of tracer laboratory commodities (disaggregated by lab tracer commodity (RTKs and CD4 and RDTs for Malaria)) HIV Test Kits 11.7% Malaria RDT 67.0% 38% 7% 24% 10% 14% HIV Test Kits <5% Malaria RDT <5% in targeted regions Indicator 3.2.3: Proportion of health facilities in priority regions having laboratory tracer commodities, on the day of the survey/support supervision visit [disaggregated by tracer commodity] RTKs (screening) MrDTs (Not assessed as It wasn’t part of indicators then) 97% 69% 92% 78% Only applies to facilities offering the services Indicator 3.2.4: Proportion of health facilities in priority regions where physical stock and record counts are in agreement for a set of tracer laboratory commodities (disaggregated by lab tracer Determine 55.2% Unigold 64.9% 27% Not a tracer item 95% Determine 80% Unigold 80% Malaria RDT: 80% in targeted ( HSCM EOP Evaluation Report, March 2016 Page 63 Indicator Base-line 2011 2012 2013 2014 2015 Final Target (March 2016) commodity (RTKs, CD4 and RDTs for Malaria)) Malaria RDT: 46.8% 8% 78% malaria endemic) regions IR 3.3: Improved Accessibility of Quality Essential Lab Services Indicator 3.3.4: Percentage of facilities able to conduct malaria testing (Microscopy and RDTs) 45% 50% 65% 75% 98% 85% HSCM EOP Evaluation Report, March 2016 Page 64 Annex 9: Availability of Commodities and Commodity Management tools by Facility Level 1. INTRODUCTION This table is based on support supervision data from the last visits to SEME and KISUMU EAST sub-counties (of KISUMU COUNTY) in the first quarter of the current financial year 2015/2016. The tool used collects data in three commodity domains i.e. Laboratory, Pharmaceuticals and EMMS (non-program pharmaceuticals and non-pharms). For each of these it collects data on (i) Storage areas; (ii) Inventory management (stocks); (iii) Reference materials; and (iv) availability and use of MIS tools. In addition for laboratory it also does an RTK – EUV (Rapid Test Kit - End user verification). For the commodities there are designated tracer lists to be used for uniformity. 2. REPORT SUMMARY Facility Name KEPH Level Laboratory Inventory RTK EUV Laboratory MIS Tools Pharmacy Inventory Pharmacy MIS Tools EMMS Asat Beach Dispensary 2 0 24% - 28% 77% 14% Dago Jonyo Dispensary 2 0 43% 0 18% 86% 12% Korwenje Dispensary 2 55% 22% - 30% 90% 8% Kuoyo Kaila Dispensary 2 0 14% - 28% 81% 11% Langi Kawino Dispensary 2 0 0 - 67% 82% 11% Nduru Kadero Dispensary 2 0 14% - 33% 65% 10% Opapala Dispensary 2 0 10% - 10% 68% 4% Rodi Dispensary 2 40% 70% - 48% 92% 6% Kibos Prison Dispensary 2 45% 52% 50% 69% 83% - Got Nyabondo Dispensary 2 - - - 38% 65% - Chiga Dispensary 2 45% 57% 38% 33% 63% - Bodi HC 3 40% 0% 71% 50% 86% 15% Miranga SCH 4 20% 33% 100% 43% 55% - Migosi SCH 4 45% 43% 75% 35% 100% - Lumumba SCH 4 47% 57% 60% 25% 94% - Gita SCH 4 40% 14% 60% 23% 88% - 3. FINDINGS The sample consists of four SCH (level 4); one HC (level 3); and 11 Dispensaries (level 2). 3.1 Availability of Pharmacy Inventory (tracer program items) The level 4 facilities scored an average of 32% and level 2 an average of 37% although the dispensaries had higher scores ranging from 10 - 69% while the SC hospitals ranged from 23 – 43%. The only level 3 in the sample had a score of 50%. 3.2 Availability and Use of Pharmacy MIS Tools The availability and use of the MIS tools in pharmacy was on average 84% in the level 4 facilities and 77% in the level 2 with the dispensaries scores ranging from 63 - 92% while the SC hospitals ranged from 55 – 100%. The only level 3 (health centre) in the sample had a score of 86%. 3.3 Availability of Laboratory Inventory (tracer program items) The majority of the level 2 facilities did not have any laboratory inventory and of the four which did the average score was 38% (40 – 55%). The average score for the level 4 facilities was 46% (20 – 47%). The only level 3 (health centre) in the sample had a score of 40%. 3.4 Availability and of Use of Laboratory MIS Tools Only two of the sample level 2 facilities had a score for this indicator (38% and 50%) which averaged to 44%. The average score for the level 4 facilities was 74% (60 – 100%). The only level 3 (health centre) in the sample had a score of 71%. 3.5 RTK End user verification HSCM EOP Evaluation Report, March 2016 Page 65 Two of the level 2 facilities had no score for this indicator but the other nine had an average of 34% (10 -70%). The level 4 facilities had an average of 37% (14 – 57%). The only level 3 facility in the sample also had a score of 0%. 3.6 EMMS (Non-program commodities – pharmaceuticals and non-pharmaceuticals) This indicator is intended to enable the supervisors check on the non-program commodities based on a designated tracer list for essential medicines and medical supplies. The eight level 2 facilities that where it was surveyed had an average score of 7% and the one level 3 facility in the sample had a score of 15%. There was no score for all the four level 4 facilities in the sample. 4. DISCUSSION 4.1 Availability of Pharmacy Inventory (tracer program items) On average the scores for all levels of facilities appear to be the same at 32% and 37% (although not subjected to statistical evaluation). The only issue would appear that the scores are relatively low but this may be explained by a closer look at the list of trace items used for the survey. The list presupposes that all the eight items on the list will be available at all facility levels on the day of the visit.The fact is that not all facilities have all of these items – some are not designated to have them e.g. a dispensary which is not a treatment centre for HIV or TB would not stock items no. (7) and (8). Similarly not all facilities are designated to give item no. (4). The AL pack (24s) which is in this list may not be available always but the other pack sizes (6s, 12s and 18s) may be available but will not count for the survey. The other non-program item nos. (1), (5), and (6) have varying levels of availability at any given time. This would depend on regularity of supply as well as the knowledge of the persons managing the commodities and how close the day of the visit is to the supply date. 4.2 Availability and Use of Pharmacy MIS Tools The average score for level 4 facilities at 84% was higher than that for level 2 facilities at 77% (although not subjected to statistical evaluation). The only level 3 (health centre) in the sample had a score of 86%. This indicator measures two things – availability and use of the tools. It is based on a scoring of a set of 13 data tools consisting of 4 programs daily activity registers; 7 program monthly/quarterly reports; and 2 pharmacovigilance tools. It is observed that at all levels the scores were high (above 75%) but it could not be 100% because not all the tools are designated to be at all the levels / facilities. Program services are sited at facilities where the requisite qualified staffs are available. 4.3 Availability of Laboratory Inventory (tracer program items) The majority of the level 2 facilities did not have any laboratory inventory and of the four which did the average score was 38% (40 – 55%). The average score for the level 4 facilities was 46% (20 – 47%). The only level 3 (health centre) in the sample had a score of 40%. This indicator is based on tracing the availability of HIV Rapid Test Kits (Screening, Confirmatory, and Tie Breaker) and the Malaria Rapid Diagnostic Tests (RDTs). The low scores here can be attributed to the perennial management problems of the RTKs (as has been noted in the report in the areas of laboratory commodity management) 4.4 Availability and of Use of Laboratory MIS Tools Only two of the sample level 2 facilities had a score for this indicator (38% and 50%) which averaged to 44%. The average score for the level 4 facilities was 74% (60 – 100%). The only level 3 (health centre) in the sample had a score of 71%. This indicator measures two things – availability and use of the tools. It is based on a scoring of a set of 10 data tools consisting of 4 transaction documents; 4 program monthly/quarterly reports; and 2 HIV-related HIS reports. It is observed that at all levels the scores were much higher at level 4 facilities and low at level 2 facilities with only 2 scoring for this indicator. The following observations may be made on this indicator: (i) many of the lower level facilities do not have laboratory facilities / or qualified laboratory staff hence the low / non-existent TRACER ITEMS 1. Amoxicillin caps 250mg 2. Artemether/ Lumefantrine tabs 20mg/120mg (24's) 3. Cotrimoxazole tabs 480mg 4. Depot Medroxyprogesterone Acetate inj 5. ORS sachets 6. Paracetamol tabs 500mg 7. TB Patient Pack 8. Zidovudine/ Lamivudine/ Nevirapine 300mg/150mg/200mg tabs HSCM EOP Evaluation Report, March 2016 Page 66 scores; (ii) not all the program lab tests are conducted at all facilities; and (iii) this may reflect that the lab commodity management may still require more work especially at the lower level facilities. 4.5 RTK End user verification Two of the level 2 facilities had no score for this indicator but the other nine had an average of 34% (10 -70%). The level 4 facilities had an average of 37% (14 – 57%). The only level 3 facility in the sample also had a score of 0%. This indicator is intended to compare the commodity inventory data against service delivery data so as to determine if there is any leakage in the supply chain. Once again the low scores here can be attributed to the perennial management problems of the RTKs (as has been noted in the report in the areas of laboratory commodity management) 4.6 EMMS (Non-program commodities – pharmaceuticals and non-pharmaceuticals) This indicator is intended to enable the supervisors check on the non-program commodities based on a designated tracer list for essential medicines and medical supplies. The eight level 2 facilities that where it was surveyed had an average score of 7% and the one level 3-facility in the sample had a score of 15%. There was no score for all the four level 4 facilities in the sample. The low scores do not necessarily point to low availability of the EMMS. It is to be observed that the commodity support supervision emphasizes on the four program areas (HIV, TB, TB and Malaria) and obtaining data for these areas therefore this part of the tool is often filled only when time allows as it is not prioritized for HCSM program reporting.