Afya Ugavi Mid-Term Review, May 2019 Page i USAID AFYA UGAVI ACTIVITY MID-TERM REVIEW REPORT May 26, 2019 Submitted by: Insight Health Advisors. Afya Ugavi Mid-Term Review, May 2019 Page ii USAID AFYA UGAVI MID-TERM REVIEW REPORT May 2019 Prepared for: Chemonics Kenya. Prepared by: Insight Health Advisors – Dr. W. Kiarie, Dr. N. Gitonga & D. Kiongo. Award No: Contracted under USAID Global Health Supply Chain Program – Procurement and Supply Management Project (GHSC-PSM) Contract No. AID-OAA-I-15-00004; Task Order Contract No. AID-615-TO-16-00011. DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Afya Ugavi Mid-Term Review, May 2019 Page 1 TABLE OF CONTENTS ACKNOWLEDGEMENT ...................................................................................................................................................3 LIST OF ACRONYMS AND ABBREVIATIONS...........................................................................................................4 EXECUTIVE SUMMARY.....................................................................................................................................................5 Methodology .....................................................................................................................................................................5 Findings 5 Conclusion.........................................................................................................................................................................8 Recommendations...........................................................................................................................................................8 1. INTRODUCTION TO THE MID-TERM REVIEW......................................................................................... 10 2. CONTEXT AND BACKGROUND................................................................................................................... 11 2.1. History of USG SC/CM support............................................................................................................... 11 2.2. Afya Ugavi Activity Background ................................................................................................................ 11 2.3. Activity Goal, Mandate and Coverage ..................................................................................................... 11 3. EVALUATION APPROACH AND METHODOLOGY................................................................................ 13 3.1. Inception.......................................................................................................................................................... 13 3.2. Data Collection ............................................................................................................................................. 13 3.3. Data Analysis.................................................................................................................................................. 14 3.4. Sampling of Organizations, Counties, Sub-Counties and Facilities................................................... 15 3.5. Report writing ............................................................................................................................................... 16 3.6. Ethical Considerations................................................................................................................................. 16 3.7. MTR Limitations............................................................................................................................................ 16 3.8. Evaluation Team............................................................................................................................................ 17 4. EVALUATION FINDINGS ................................................................................................................................... 18 4.1. AU Mid-Term Evaluation Findings at a Glance...................................................................................... 18 4.2. Detailed MTR Findings................................................................................................................................. 22 5. CONCLUSIONS ON SC/CM FINDINGS........................................................................................................ 43 6. LESSONS LEARNED AND BEST PRACTICES ............................................................................................... 43 7. RECOMMENDATIONS ........................................................................................................................................ 45 8. CRITIQUE OF ACTIVITY DESIGN AND PROPOSED FUTURE DIRECTION .................................... 48 9. ANNEXES ................................................................................................................................................................. 51 Annex 1: Evaluation Matrix ........................................................................................................................................ 51 Annex 2: List of Literature Review Documents................................................................................................... 54 Annex 3: Key Informants Interview Questionnaire - National Respondents................................................ 56 Annex 4: Key Informants Interview Questionnaire - County and Sub-County Respondents .................. 60 Annex 5: County and Facility Checklist.................................................................................................................. 63 Annex 6: AU Activity Performance Checklist....................................................................................................... 64 Annex 7: List of National Level Informants............................................................................................................ 67 Annex 8: List of County Level Informants ............................................................................................................. 62 Afya Ugavi Mid-Term Review, May 2019 Page 2 LIST OF TABLES Table 1: Summary of AU Activities performance ......................................................................6 Table 2: Summary of AU’s rating based on pillars of a functioning health supply chain system.........................................................................................................................................6 Table 3: AU Activity Geographic Scope.................................................................................11 Table 4: List of National Level Organizations that Participated ............................................15 Table 5: Sampled and Non-Sampled AU Counties..................................................................15 Table 6: Number and Gender of County Level Respondents...................................................16 Table 7: Performance Rating by Area of Evaluation ......................................................18 Table 8: Ratings Based on Health Supply Chain Pillars.........................................................19 Table 9: Respondents Perception of AU Performance and Impact.........................................25 Table 10: Baringo County Budget Trends...............................................................................33 Table 11 : AU Interventions Quality Perceptions Score .........................................................34 Table 12: Perception scores for Efficient Use of Resources by AU ........................................35 Table 13: Percentage % Workload Growth in Isiolo County Post UHC Introduction ...........37 Table 14: AU Interventions Targeting Priority Health Programs ..........................................40 Table 15 : Actual Expenditure by Period ................................................................................41 Table 16: Actual expenditure by Health Focus Area...............................................................41 Table 17 : Projected End of Year 3 Expenditure.....................................................................41 Table 18: Identified Best Practices..........................................................................................44 Table 19: List of Recommendations Arising from the MTR ....................................................45 Table 20: AU Activity Design Matrix .....................................................................................49 Table 21: Recommendations on AU Activity Design Changes................................................50 LIST OF FIGURES Figure 1: USAID Framework for AU......................................................................................12 Figure 2: % Stock-Out Rates of all ALs at Service Delivery Points........................................22 Figure 3: National SDP Reporting Rates to DHIS2................................................................23 Figure 4: National Malaria Commodity Reporting Rates - Feb 2018 - Jan 2019 ..................23 Figure 5: Months of Stock for Malaria Commodities (National) - Feb 2019 .........................24 Figure 6 : FP Commodity Reporting Rates – Baringo............................................................31 Figure 7: Percent of Total Turkana County FP Clients (Long-Term Methods)......................48 Figure 8 : 2018 Vs 2017 FP Client Growth - Turkana County ...............................................48 Afya Ugavi Mid-Term Review, May 2019 Page 3 ACKNOWLEDGEMENT The evaluation team is grateful to the many respondents at national and county level who were generous with their time, shared their views candidly and freely shared with us many supporting documents. In several counties, county and sub-county CSTWGs allowed us to sit in their meetings and thus contributed valuable time from their scheduled meetings. Respondents also took time to show us their pharmacies and health commodity stores. We are greatly indebted to the Afya Ugavi team that provided us with both logistical and technical support at all stages of the mid-term evaluation. We acknowledge the support and guidance of Jayne Waweru, Afya Ugavi Chief of Party and her technical team led by James Riungu and John Ngamau. We are also grateful to the staff of Insight Health Advisors that efficiently organized the logistics for our travel and accommodation. Afya Ugavi Mid-Term Review, May 2019 Page 4 LIST OF ACRONYMS AND ABBREVIATIONS AIDS AL Acquired Immune Deficiency Syndrome Artemether Lumefantrine AMREF African Medical and Research Foundation APHIAplus AIDS, Population and Health Integrated Assistance AU Afya Ugavi CCC Comprehensive Care Centres CDCS Country Development Cooperative Strategy CHAI Clinton Health Access Initiative CHMT County Health Management Team CS TWG Commodity Security Technical Working Group DFH Department of Family Health DHIS District Health Information System DHIS2 District Health Information System 2 platform EMMS Essential Medicines and Medical Supplies FGD Focus Group Discussion FP/RMNCAH Family Planning/Reproductive Health, Maternal, Newborn, Child and Adolescent Health GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria. GHSA Global Health Security Agenda GoK Government of Kenya HIV Human Immunodeficiency Virus IHA Insight Health Advisors KEMSA Kenya Medical Supplies Authority KII Key Informants Interview KSCSS Kenya Supply Chain Systems Strengthening MCA Member of the County Assembly MEL monitoring, evaluation and learning MoH Ministry of Health MTR Medium Term Review NASCOP National AIDS/STI Control Program NCAHU Newborn Child and Adolescent Health Unit NCD Non-Communicable Diseases NMCP National Malaria Control Program PMP Performance Management Plan RHSMU SC/CM Reproductive Health Services Management Unit Supply Chain and Commodity Management SCHMT Sub-County Health Management Team SoW Scope of Work UHC Universal Health Coverage UON University of Nairobi USAID U.S. Agency for International Development USG United States government Afya Ugavi Mid-Term Review, May 2019 Page 5 EXECUTIVE SUMMARY Afya Ugavi (AU) is a 5-year activity managed by Chemonics Kenya and funded by USAID running from October 2016 to September 2021. The goal of the Activity is to ensure there is a well-functioning, high performing supply chain system that provides communities with adequate, safe, affordable, and reliable supply of health commodities, when and where needed. The expected outcomes of the Activity are to strengthen national level systems for supply chain and commodity management and to strengthen county oversight and implementation of supply chain services. The Activity is being implemented in collaboration with the Government of Kenya’s (GOK) Ministry of Health (MoH) and other stakeholders to strengthen the health supply chain systems for HIV/AIDS, malaria, Family Planning/ Reproductive Health, Maternal, Newborn, Child and Adolescent Health (FP/RMNCAH). AU supports activities at the national level and in 14 high disease burden, under-resourced, underserved counties with the aim of strengthening health supply chain systems for HIV/AIDS, Malaria, and Family Planning and Reproductive Health. The mid-term review (MTR) was conducted by Insight Health Advisors (IHA) in the second quarter of the third year of the AU Activity to (1) assess progress made since inception of the activity towards specific objectives of AU, (2) generate recommendations to assist AU in meeting intermediate results and outcomes within the remaining project period, (3) understand if AU is working in accordance with its stated objectives and (4) provide recommendations to USAID on how they can use AU and other service delivery partners to implement their newly defined supply chain priorities. Methodology The approach applied by IHA consisted of an inception report drafted at the beginning of the evaluation to ensure the evaluators’ understanding was completely aligned with that of AU’s technical team. Thereafter, collection of both qualitative and quantitative data was carried out. A comprehensive secondary data review was carried out to understand the country’s health, socio-economic, and demographic contexts and the performance and impact of the AU Activity. Primary data collection was carried out at the national, county and facility levels. Structured, interviewer-administered survey questionnaires were administered to a purposive sample of relevant and knowledgeable stakeholders. Organizations interviewed at the national level included KEMSA, NASCOP, NMCP, DFH, MoH, COG, Ministry of Agriculture & Livestock, AMREF and GFATM. Six counties (Baringo, Busia, Homa Bay, Isiolo, Kakamega and Turkana) out of the total 14 counties under each of the three program focus areas (Malaria, HIV/AIDS and RMNCAH) were purposively sampled. The evaluation team also made observations on the state of the supply chain and commodity management especially at the facility level. Data collected from key informant interviews was manually organized along the Activity’s technical intervention areas and the three health thematic areas - HIV/AIDS, Malaria and FP/RMNCAH. Quantitative data was analyzed using Microsoft Excel and graphical outputs and tables developed. This midterm evaluation report has been organized around the evaluation questions, sub-questions and tasks provided by AU Activity with guidance from the USAID Evaluation Policy and coherent flow of concepts and findings. The limitations of the MTR included difficulty in securing appointments with some national level informants due to suspension of USAID Funding at National Level. Many national level respondents also had limited knowledge of the AU Activity especially since its interventions are mainly at the county level. A few key respondents were not available in Turkana county CSTWG level. However, the team was able to interview sub-county CS TWG and staff at facility level. Availability of key staff was not a problem in the other five counties visited. Findings Overall findings: Afya Ugavi Mid-Term Review, May 2019 Page 6 AU Activity interventions were reported to be of high quality with regards to technical content and execution. AU used findings of the baseline and other assessments, and stakeholder engagement in designing its interventions. AU factored in feedback from its beneficiaries in designing or re-designing its tools and interventions. Performance of AU Activities varied across different areas of evaluation based on the analysis of the data collected. A summary of the AU Activity performance is summarized in table 1 below: Table 1: Sum m ary of AU Activities perform ance MET EXPECTATIONS PARTIALLY MET EXPECTATIONS BELOW EXPECTATIONS Overall AU Performance Overall HIV/AIDS Performance Support to Counties - Senior Leadership Engagement Overall Malaria Performance Overall FP/RMNCAH Performance Support to National Public Health Programs (HIV/AIDS and FP/MNCAH) Support to Counties - CSTWG Global Health Security Agenda Activity Visibility and Branding Integration of interventions A summary of AU’s rating based on pillars of a functioning health supply chain system are shown in table 2 below: Table 2: Sum m ary of AU’s rating based on pillars of a functioning health supply chain system MET EXPECTATIONS PARTIALLY MET EXPECTATIONS BELOW EXPECTATIONS Inventory management Strengthening Forecasting and supply planning National SC/CM Strategic Planning Health Commodity Re-Distribution Commodity Management Advocacy for increased County funding for health commodities Supply Chain Audits Pharmacovigilance Supportive supervision Rational Drug Use Functional County and sub-County CSTWGs Data capture tools County Strategic and Annual Operational planning EMMS Tool (Dashboard) Regional and National Forums Planning and budgeting Capacity Building (Training, CMEs, OJT and Mentorship) Improving reporting rates & quality (timeliness and completeness) Data accuracy Health Commodities Dashboards (FP, Malaria and HIV) Minimizing Leakages and expiries Quantification As is elucidated in the main report, the way the AU Activity was designed presented some challenges and constraints in the overall performance of the Activity. Overview of specific findings: Afya Ugavi Mid-Term Review, May 2019 Page 7 AU has made progress in rationalizing and simplifying its Performance Management Plan (PMP). The grouping of the AU indicators into three categories - Cross cutting indicators (5), Sustainability indicators (5) and Program based indicators (7) as articulated in AU Year 3 monitoring, evaluation and learning (MEL) plan represents real progress on how the activity monitors and evaluates its work. AU Year 3 MEL plan has a total of 17 indicators down from 50 in the year 2 MEL plan. The evaluation team found that three sustainability indicators were either difficult to measure or suffered from reliability and validity problems. Governance challenges at the National government level led to suspension of USAID funding to national level MoH institutions. This has had a great impact especially on planned national level interventions. However, the effect has not been uniform across the program areas. HIV/AIDS appeared to be the most affected. Nevertheless, AU provided varying degrees of SC/CM support and collaborationto national level institutions such as KEMSA, DFH, NMCP and NASCOP. Development of a National SC/CM strategic plan that was to be developed in Year 1 of the activity is yet to be done. This activity was dropped following the suspension of national level activities but has been reintroduced in the year 3 workplan. However, AU supported County strategic plans and annual workplans (AWP) development. County Commodity Dashboards, support to CSTWGs and facility level supportive supervision/OJT were identified by respondents as best practices that were transformative to SC/CM at the county level. Some inter-county variations in the quality of AU interventions were noted largely due to different disease program approaches in respective counties. Lack of disposal of expired products is a lingering problem leading to products that expired years ago taking up scarce storage space and posing hazards. Implementation of commodity management in designated focal facilities /COLs faced some challenges including lack of a coherent approach in most of the visited counties and there were a few issues with the quality of renovations carried out. In all the counties visited, automation of clinical services and commodity management was found to be patchy and fragmented. While automation of clinical and commodity management is not an AU mandate, the prevailing situation has adversely affected AU work due to challenges with availability of timely and accurate commodity data especially at the facility, sub-county and county levels. Commodity dashboards have contributed in alleviating this challenge but are not a comprehensive solution by themselves. AU is increasingly working towards promoting sustainability of its SC/CM interventions particularly at the county level. Some of these initiatives include strengthening and working through county leadership and governance structures (county and sub-county CS TWGs), capacity building initiatives for skills transfer and partnering with local institutions such as the University of Nairobi Health IT. In order to better institutionalise SC/CM functions at county level, a concept note has been developed by AU that proposes the establishment of county Health Products & Technologies Units (HPTU) that would drive and oversee all activities within the supply chain system. However significant sustainability challenges remain. Sustainability of AU interventions was constrained by; initial verticalized design and approach of the interventions; limited county financing for SC/CM support activities; a trend of reduction in county commodity budgets in a number of counties visited; public health program commodities are currently funded centrally through partner and national government support and counties have no visibility on the costs of the same; HRH limitations (numbers, retention and skills) in the context of expected rapid increase in demand from UHC initiatives and; minimal engagement of high-level county leadership by AU for advocacy and SC/CM ownership. AU has also not actively engaged with inter-governmental and national level forums and institutions such as Council of Governors, Directorate of Pharmaceutical Services. Directorate of Pharmaceutical Services is a new entity that replaces the former national MoH Pharmacy department. AU support to these institutions was affected by suspension of USAID financial and technical support to national level institutions while Pharmacy and Poisons Board is not part of the current AU Activity mandate. Specifically, the evaluation team further found that there were limited AU interventions to advocate for increased & sustained domestic commodity funding to support transition to a self-reliance model Afya Ugavi Mid-Term Review, May 2019 Page 8 in-line the USAID Policy Framework – “Ending the Need for Foreign Assistance.” It is however recognized that the role of AU to support enhancement of health commodity funding is limited to its support for development of health strategic plans and annual workplans. Introduction of Universal Health Coverage (UHC) has led to a major spike in demand for health services and commodities. This means that roll-out of UHC poses significant SC/CM challenges and opportunities for AU to mitigate and explore. Based on the MTR, the evaluation highlighted the following lessons learned: 1. Integration in implementing SC/CM interventions enhances efficiency and impact. 2. There is need to engage national level institutions in policy dialogue and formulation to secure adequate resources and promote a harmonized approach to SC/CM interventions countrywide. 3. Involving beneficiaries in design and implementation of SC/CM interventions contributes to better designs, greater ownership and skills transfer/increased local capacity. 4. Versatility/adaptability considering contextual changes: It is important for AU to continue demonstrating greater versatility in consideration of contextual changes. A good example of this is the support AU extended to Isiolo county to support their UHC health commodities quantification. 5. Advocacy for enhanced and ring-fenced health commodities financing is crucial in achieving commodity security and moving towards greater sustainability. 6. Strengthening County Leadership and Governance Structures is fundamental to ensure strong health stewardship by the county governments and accountability to citizens on all health maters including commodity security. Conclusion The following conclusions emerged from the mid-term evaluation findings: • AU has effectively and efficiently discharged its mandate and is on track to achieve program goals and targets. • The suspension of USAID funding of national MoH institutions has significantly hampered the implementation of national level activities and weakened goodwill and coordination between national and county levels. • The AU core package of interventions for SC/CM strengthening at county and national level is appropriate and meets key SC/CM needs and gaps. • At the county level, it was reported that health EMMS commodities budgets and actual disbursements were in decline in a significant number of counties visited. Further, there is perceptible lack of knowledge, confidence and openness in discussing county commodity budgets and financial flows. This has huge implications on plans to migrate to an era of greater self-reliance. • For greater effectiveness and in response to the changing SC/CM landscape (especially UHC) some design and implementation changes including scaling up existing promising interventions are required for AU going forward. • Some of the proposed changes, such as expanding geographical scope and scaling up best practices and innovations, may require additional funding. Recommendations Afya Ugavi Mid-Term Review, May 2019 Page 9 The main recommendations arising from the findings of the MTR include: 1. Afya Ugavi should support a comprehensive assessment to establish the value of program commodities supplied to counties and other recipients. This should be compared to county health commodity budgets to develop “stress test” scores for county migration to greater self-reliance in health commodities. KEMSA should also start providing health facilities with receipts and statements showing the value of program commodities to provide visibility for program costs. 2. AU should develop a strategy to scale up promising interventions (tools, methodologies, lessons) within the 14 (fourteen) supported counties and beyond, in partnership with other USG and non-USG implementing partners. Examples include the County Commodity Dashboard and the integrated supportive supervision tool. In this regard the current ongoing scale up of the Malaria commodity dashboard should be expanded to include all the other program commodities and EMMS. 3. AU should reduce the number of focal facilities/COLs to one per county but significantly expand the depth and quality of support. In addition, AU should develop a design and budget pack to support roll-out by counties. Partnerships should be explored with USG partners on ways of enhancing focal facilities and COLs. 4. Re-align AU activities with UHC, expand support for UHC SC/CM components at the national and county level and advocate for the necessary policy and regulatory reforms. 5. AU needs to support counties to address the issue of safe disposal of expired products to protect consumers and the environment and free much needed storage space currently taken up by a legacy of stockpiled expired products. 6. AU Activity Design Changes should include broadening geographical footprint of FP/RMNCAH and HIV/AIDS interventions to cover all sub-counties in the targeted counties. A health focus lens should only be used for selecting target counties. Once selected, all counties should adopt similar implementation approaches and interventions covering all three health focus areas and EMMS. The Activity should fully integrate interventions to include HIV/AIDS, Malaria, FP/RMNCAH and EMMS in all the 14 target counties and strengthen collaboration with service delivery partners particularly execution of facility support activities. This will significantly leverage available resources and magnify impact. 7. Current funding based on the task order is likely to finance the Activity up to the end of Year 4 at the current rates of expenditure. For the Activity to run up to year 5 at the current rate of spending and if it is to adopt design and implementation changes to be more effective, USAID may need to consider additional funding. 8. AU Activity mandate needs to be expanded to support scaled-up advocacy interventions for increased commodity funding by national and county governments in a sustained and ring￾fenced manner. Exploring private sector engagement in various components of SC/CM to support migration to a more self-reliant model will ensure sustainability of the program. In collaboration with other USG partners AU needs to develop a national and county transition roadmap to support migration towards a more self-reliant model especially for program commodities. Afya Ugavi Mid-Term Review, May 2019 Page 10 1. INTRODUCTION TO THE MID-TERM REVIEW Afya Ugavi is a 5-year activity managed by Chemonics Kenya and funded by USAID. The activity started in October 2016 and runs until September 2021. The mid-term review (MTR) was conducted in the second quarter of the third year of Afya Ugavi (AU) Activity. Purpose of the Mid Term Review The Scope of Work (SOW) developed for the Afya Ugavi MTR expected the consultants to undertake the following: • Assess progress made towards specific objectives of AU, its impact on the target stakeholders (policy makers at county and national levels), and Kenya’s disaster preparedness; • Generate recommendations to assist AU in meeting intermediate results and outcomes within the remaining project period; • Understand if AU is working in accordance with its stated objectives and whether all the funded implementing partners are working in accordance with the USAID/Kenya mission’s new supply chain priority objectives; • Provide recommendations to USAID on how they can use Afya Ugavi and other service delivery partners to implement their newly defined supply chain priorities. In carrying out the MTR, the consultants were further guided by the evaluation questions and sub￾questions and the consultants’ tasks detailed in the SOW. Refer to Annex 1 for the evaluation matrix indicating the consultants’ tasks. The evaluation questions and sub-questions that the MTR addressed are: EQ1: How effectively and efficiently has Chemonics implemented the Afya Ugavi activity? • EQ1.1- Is the Afya Ugavi activity on track to achieve its stated objectives? • EQ1.2- What was the quality of the work conducted by Chemonics? • EQ1.3- How efficiently have resources been utilized in the Afya Ugavi implementation? • EQ1.4- How effective has the project’s measurement approach (framework, indicators, data collections, etc.) been to monitoring project implementation? • EQ1.5- What are the most effective sustainability strategies that Afya Ugavi has employed and is the GOK likely to sustain these advances after the program is finished? EQ2: How effective has the design of Afya Ugavi been in addressing the supply chain development goals of the Government of Kenya and of USAID/Kenya? • EQ2.1- How have GOK counterparts benefitted from this activity at the national level? At the county level? What about other implementing partners? • EQ2.2- How has the initial design been affected by evolving contexts and changing beneficiary needs? And was it able to respond and change to these influences, if necessary? • EQ2.3- Is the design of Afya Ugavi activity an appropriate allocation of resources and responsibilities in the USG funding environment for the Kenyan public health supply chain? • EQ2.4- Are there more effective supply chain technical assistance approaches that should be considered in the Kenyan context? The MTR was carried out in March and April 2019. Afya Ugavi Mid-Term Review, May 2019 Page 11 2. CONTEXT AND BACKGROUND 2.1. History of USG SC/CM support Kenya has been a recipient of USG assistance for strengthening health supply chain and commodity management functions for nearly two decades. This has resulted in significant Supply Chain (SC) and Commodity Management (CM) gains at the national level and in the latter years following devolution at the county level as well. This progress has been achieved by providing technical assistance and building the capacity of respective priority health program departments of the national Ministry of Health, particularly theNational AIDS/STI Control Program(NASCOP), National Malaria Control Program (NMCP) and Department of Family Health (DFH). USAID has supported a number of other national level institutions that support health SC/CM. These institutions include the Pharmaceutical Services Unit of MOH, Pharmacy and Poisons Board (PPB) and the Kenya Medical Supplies Authority (KEMSA). Following the promulgation of the 2010 constitution, USAID has extended its SC/CM support to counties given the fact that the health function is now largely devolved. This support was initially provided by USAID to Health Commodities and Services Management (HCSM) project and currently through Afya Ugavi. Some USAID SC/CM support at the county level is also provided through a number of USG service delivery partners. 2.2. Afya Ugavi Activity Background The USAID-funded AfyaUgavi Activity runs for five years between October 2016 and September 2021 and aims at building on the supply chain and commodity management gains the country has made over the last decade and to further catalyze a new level of health commodities supply chain performance. AU supports activities at the national level and in 14 high diseaseburden, under￾resourced, underserved counties with the aim of strengthening health supply chain systems for HIV/AIDS, Malaria, and Family Planning and Reproductive Health. The Activity is currently in its third year of implementation. The Activity uses the six key pillars of a functional health supply chain system as its guiding framework. These pillars are: • Access to health products and technologies • Strong governance structures with defined responsibilities • Quality service delivery through safety of health products • An adequate, well-trained, health workforce. • Information management systems • Commodity Financing 2.3. Activity Goal, Mandate and Coverage The AU Activity has a clearly defined goal, mandate and coverage based on the 2016 USAID contract and subsequent revisions. Table3 is a summary of the goal, mandate and coverage. Table 3: AU Activity Geographic Scope Activity Goal: Ensure there is a well-functioning, high performing supply chain system that provides communities with adequate, safe, affordable, and reliable supply of health commodities, when and where needed. Afya Ugavi Mid-Term Review, May 2019 Page 12 Level # Organization/County Health Focus Area National 1 National AIDS/STI Control Program (NASCOP) 2 National Malaria Control Program (NMCP) 3 Department of Family Health (DFH) County 1. Homa Bay 2. Uasin Gishu 3. Isiolo 4. Mombasa 5. Kisumu 6. Migori 7. Siaya 8. Busia 9. Bungoma 10. Vihiga 11. Kakamega 12. Baringo 13. Samburu 14. Turkana GHSA 1. Multiple Key HIV/AIDS Malaria FP/RMNCAH The goal of the Activity is to ensure there is a well-functioning, high performing supply chain system that provides communities with adequate, safe, affordable, and reliable supply of health commodities, when and where needed. The Activity is being implemented in collaboration with the Government of Kenya’s (GOK) Ministry of Health (MoH) and other stakeholders to strengthen the health supply chain systems for HIV/AIDS, malaria, Family Planning/ Reproductive Health, Maternal, Newborn, Child and Adolescent Health (FP/RMNCAH) at the national level and in14 high burden, under-resourced and underserved counties. At the national level, technical support is extendedto the constituent programs which include National AIDS/STI Control Program (NASCOP), National Malaria Control Program (NMCP) and the Department of Family Health (DFH). There is also additional support in line with the Global Health Security Agenda’s (GHSA) plans to improve Kenya’s capacity to help create a safe and secure environment from infectious disease threats and elevate global health security as a national and global priority. 2.4. USAID Framework and Strategic Objectives The activity has two outcomes that contribute to the activity goal - ensure there is a well-functioning, high performing supply chain system that provides communities with adequate, safe, affordable, and reliable supply of health commodities, when and where needed. The activity goal further contributes to the USAID/Kenya Country Development Cooperative Strategy (CDCS) Development Objective 2 - Strengthen health and human capacity. Figure 1: USAID Fram ework for AU Afya Ugavi Mid-Term Review, May 2019 Page 13 USAID/Kenya CDCS Development Objective 2: Strengthen health and human capacity  Activity Goal: Ensure there is a well-functioning, high performing supply chain system that provides communities with adequate, safe, affordable, and reliable supply of health commodities, when and where needed   Outcome 1: Strengthened national level systems for supply chain and commodity management Outcome 2: Strengthened county oversight and implementation of supply chain services     Output 1.1: Strengthened technical leadership and coordination for commodity management Output 1.2: Improved supply chain logistics and commodity management Output 2.1: Increased human and institutional capacity for commodity management Output 2.2: Improved commodity security environment Source: Based on AU Revised Monitoring, Evaluation and Learning (MEL) Plan, 2018 3. EVALUATION APPROACH AND METHODOLOGY The MTR involved the collection of both qualitative and quantitative data and adhered to USAID’s Evaluation Policy. The approach in achieving the MTR’s objectives was based on the following: • A strong supply chain and sustainable health systems focus. • A consultative approach that entailed deliberate and strategic engagement of relevant actors and stakeholders to foster ownership, collaboration and lasting change. • Development of evidence-based recommendations and designs that give room to new knowledge and future projections. • Contextualization of best practices and models. The key approaches that were applied in the MTR included: 3.1. Inception To ensure the evaluators’ understanding was completely aligned with that of AU’s technical team, the evaluators worked collaboratively with the AU team at all stages of the evaluation. The evaluation team developed a concise inception report that was discussed with the AU team to align the scope and expectations and agree on logistical matters. The MTR was guided by the inception report. 3.2. Data Collection Data collection employed the following broad approaches: Afya Ugavi Mid-Term Review, May 2019 Page 14 3.2.1. Secondary Data Review A comprehensive secondary data review was carried out to understand the country’s health, socio￾economic, and demographic contexts and the performance and impact oftheAU Activity. The literature review was aligned to the evaluation questions, sub-questions and consultants’ task developed by AU. Refer to Annex 1for the evaluation matrix. The documents reviewed included national and county policy and planning documents, relevant evaluation and assessment reports and a wide range of AU Activity documents including workplans, Monitoring and Evaluation (M&E) plans and reports, quarterly and annual progress reports. Refer to Annex 2 for a complete list of documents that were reviewed. 3.2.2. Primary Data Collection The following tools were used to collect data at the national, county and facility levels: Key Informant Interview Questionnaires (KIIs) Structured, interviewer-administered survey questionnaires were administered to a purposive sample of relevant and knowledgeable stakeholders to get their views on the health sector and supply chain systems and more specifically on the performance of AU Activity. The questionnaires also sought views from informants on possible changes to the design of AU Activity in the remaining implementation period. Two distinct Key Informant Interview Questionnaires were used. These are: • KIIs Questionnaire - For National Level Respondents - Refer to Annex 3 • KIIs Questionnaire- For County Level Respondents - Refer to Annex 4 Quantitative Data Collection Checklists • County and Facility Level Checklist - Refer to Annex 5 The checklist used at this level looked at staffing levels for staff involved in SC/CM and stock status for HIV, Malaria, FP and RMNCAH tracer commodities. Data on commodity financing was also collected and availability of relevant strategic documents and minutes of relevant meetings checked. • AU Activity Performance Checklist -Refer to Annex 6 This checklist collated quantitative data from AU based on the Activity’s Performance Management Plan (PMP). 3.2.3. Field Observation The evaluation team made observations on the state of the supply chain and commodity management especially at the facility level. The observations focused on pharmacies and drug stores where the team observed cleanliness, organization, labeling, shelving and racking, presence and use of stock control documents such as bin cards, security measures, firefightingequipment, air-conditioning, temperature and humidity monitoring and display and use of job aids. 3.3. Data Analysis All the data obtained from key informants’ interviews was manually organized into key themes based on the evaluation matrix developed from the MTR SOW – refer to Annex 1. The data was further organized along the three health thematic areas - HIV/AIDS, Malaria and FP/RMNCAH. Quantitative data from the checklists was analyzed using Microsoft Excel and graphical outputs and tables were developed. Afya Ugavi Mid-Term Review, May 2019 Page 15 3.4. Sampling of Organizations, Counties, Sub-Counties and Facilities National Level All the three national level organizations that are supported by the AU Activity – NASCOP, NMCP and DFH were selected to participate in the evaluation. Additionally, five other organizations shown in Table 4 also participated in the evaluation based on purposive sampling. Table 4: List of National Level Organizations that Participated # Organization No. of Respondents Female Male Total 1 KEMSA 1 1 2 2 NASCOP 1 1 3 NMCP 2 2 4 DFH 1 1 5 MoH 3 3 6 COG 4 3 7 7 Ministry of Agriculture & Livestock 1 2 3 8 AMREF 1 1 9 GFATM 1 1 Total 8 13 21 Refer to Annex 7 for the complete list of national level respondents. County Level The selection of counties to be covered under the MTR was based on a ‘stratified/categorized’ purposive sampling to ensure that we included a balanced mix of counties supported under each of the three program focus areas (Malaria, HIV/AIDS and RMNCAH). Six out of the total 14 supported counties were selected. The counties are Homa Bay, Isiolo, Kakamega, Busia, Baringo and Turkana. Table 5: Sampled and Non-Sam pled AU Counties Level # AU Supported County Health Focus Area Sampled for Evaluation County 1. Homa Bay Yes 2. UasinGishu 3. Isiolo Yes 4. Mombasa 5. Kisumu 6. Migori 7. Siaya 8. Busia Yes 9. Bungoma 10 Vihiga 11 Kakamega Yes 12 Baringo Yes 13 Samburu Afya Ugavi Mid-Term Review, May 2019 Page 16 Level # AU Supported County Health Focus Area Sampled for Evaluation 14 Turkana Yes Key HIV/AIDS Malaria FP/RMNCAH The selected counties represent all the three disease focus areas. The evaluation team interviewed county and sub-county CSTWGs and also facility pharmacy and clinical staff who were purposivelysampled based on availability during the MTR period. See Table 6 for county level respondents: Table 6: Num ber and Gender of County Level Respondents # County Health Focus Area No. Of Respondents Female Male Total 1 Isiolo HIV/AIDS 5 11 16 2 Kakamega Malaria 6 9 15 3 Homa Bay HIV/AIDS & Malaria 8 8 16 4 Busia Malaria 9 15 24 5 Baringo FP/RMNCAH 8 13 21 6 Turkana FP/RMNCAH 3 7 10 Total 39 63 102 Refer to Annex 8 for the complete list of county level respondents 3.5. Report writing The report has been organized around the evaluation questions, sub-questions and consultants’ tasks provided by AU Activity. In writing the report, the evaluation team has factored in guidance from the USAID Evaluation Policy and coherent flow of concepts and findings. 3.6. Ethical Considerations Participation in the evaluation was voluntary. All persons identified as respondents were verbally encouraged to participate in the evaluation but also informed that they had a choice of opting out. The evaluation report does not reveal the identity of individual respondents as the source of cited information or quotes. 3.7. MTR Limitations The limitations of the MTR were: Afya Ugavi Mid-Term Review, May 2019 Page 17 • Suspension of USAID Funding at National Level: This made it difficult to secure appointments with some national level informants especially from DFH. Many national level respondents also had limited knowledge of the AU Activity especially its interventions at the county level. • Availability of Key Respondents: In Turkana, the team was only able to interview one member of the County CSTWG. However, the team was able to interview sub-county CS TWG and staff at facility level. Availability of key staff was not a problem in the other five counties visited. 3.8. Evaluation Team The Insight Health Advisors evaluation team members consisted of: • Dr. William Kiarie – Team Leader, SC/CM and HSS expert. • Dr. Nelson Gitonga – HSS Expert – Governance, Policy, HCF, PPP. • Mr. David Kiongo – M&E expert. Afya Ugavi Mid-Term Review, May 2019 Page 18 4. EVALUATION FINDINGS Findings of the mid-term review have been organized around the evaluation questions, sub-questions and consultants’ tasks based on the evaluation Matrix – Refer to Annex 1. The findings in this section relate to Evaluation Question 1 - How effectively and efficiently has Chemonics implemented the Afya Ugavi activity? Findings for Evaluation Question 2 - How effective has the design of Afya Ugavi been in addressing the supply chain development goals of the Government of Kenya and of USAID/Kenya? - are largely addressed in section 8 of this report. The exception is findings for sub-question 2.1 on benefits to GOK counterparts and implementing partners that are addressed in this section. 4.1. AU Mid-Term Evaluation Findings at a Glance A) General Ratings Performance of AU activities varied across different areas of evaluation based on the analysis of the data collected. Each specific area of evaluation was accorded a performance rating according to the colour scheme Green (Meets Expectations), Yellow (Room for Improvement), and Red (Below Expectation). Table 7shows performance rating by area of evaluation. Table 7: Performance Rating by Area of Evaluation # Area of Evaluation Performance Rating Remarks 1. AU Organizational Capacity (Systems, Structures and People) All respondents that had interacted with AU spoke of it as an efficient and responsive organization with competent and committed staff. 2. Overall AU Performance Given some of the challenges the Activity faced at the beginning, including suspension of USAID funding to national institutions and also the protracted health sector industrial action, the activity has established itself well as the key provider of SC/CM TA especially at the county level. AU activities are making a significant difference in the performance of supply chain and the wider health systems. 3. Overall HIV/AIDS Performance Concerns raised and noted are partially to do with the design. This is discussed more detailed MTR findings. Also, national level interventions seem to have been affected more by the suspension of USAID funding to national institutions compared to other programs such as Malaria. HIV/AIDS activities also had a late start. 4. Overall Malaria Performance The evaluation team found that the Malaria program had the most comprehensive design in terms of geographic scope and package of interventions. Its work at the national level had also not suffered as much as the other two areas as a result of the suspension of USAID funding to national MoH institutions Afya Ugavi Mid-Term Review, May 2019 Page 19 # Area of Evaluation Performance Rating Remarks 5. Overall FP/RMNCAH Performance The program had relatively little involvement at the national level compared to the malaria program. The MNCAH supply chain (excluding FP) component was poorly developed and AU is working to establish this in partnership with DFH and the counties. Selection of target counties and sub-counties places this Activity on an uphill task with regards to performance as these are regions that are not traditionally receptive to FP/RMNCAH practices. 6. Support to Counties The execution of the Activity in the target counties has been a success including having county-based staff for close linkages and skills transfer. 7. Support to National Public Health Programs Refer to remarks on HIV/AIDS and FP/RMNCAH 8. GHSA This component had a late start but the foundations have been laid. However, some respondents still felt that progress was slow. Reports however suggest that the mission had anticipated challenges and were pleased with the progress and outcome. Future funding is uncertain at this stage. 9. Integration of interventions Most efforts to adopt an integrated approach in AU interventions started in the second year and are accelerating in the third year 10. Organizational Visibility and Branding Unlike other USAID implementing partners there was no branding of AU in the facilities visited, not even in the pharmacies and stores they have refurbished. The little visibility observed was through job aids in some facilities. Success stories have not been printed and distributed B) Rating Based on Pillars of a Functioning Health Supply Chain System Table 8: Ratings Based on Health Supply Chain Pillars # Pillar of a functioning health supply chain system AU Interventions Performance Rating Remarks 1. Commodity Management system Strengthening Forecasting and supply planning Quantification had been done in three counties evaluated but not in the other three. At the national level AU had supported HIV/AIDS and Malaria commodity quantification Inventory management (Record keeping, stores management) There was excellent progress at all levels visited including in health centres and dispensaries. Commodity Management Focal There was significant variation in the way this activity has been implemented in different counties. In some counties staff were not aware of the Afya Ugavi Mid-Term Review, May 2019 Page 20 # Pillar of a functioning health supply chain system AU Interventions Performance Rating Remarks Facilities/Centres of Learning and Renovation and equipping of Pharmacies and stores implementation roadmap and viewed pharmacy and stores rehabilitation as the main goal. The standard of renovation and refurbishment was not uniform in the facilities visited. In some facilities the quality of the renovations was below expectation based on feedback from users and consultants’ observations. Renovation documentation in respect to designs and budgets was not available for use in lobbying counties to replicate these efforts Health commodity Re-Distribution This was consistently cited as being very helpful and having improved stocking and reduced expiries, despite being a reactionary practice in supply chain. Supply Chain Audits The audits were very comprehensive. In Kakamega an audit had helped unearth commodity losses through mismatched deliveries. Supportive supervision Respondents liked the comprehensive approach supported by the checklist. Also, were happy with the scoring capability and linkage to OJT and mentoring 2. Leadership and Governance Functional County and sub-County CSTWGs The evaluation team found that CSTWGs at both the county and sub￾county were functional and were helping provide a coordinated and harmonized approach in SC/CM across functions and commodity categories. The CSTWGs provide a good foundation for sustainability beyond AU National SC/CM Strategic Planning Development of a National SC/CM strategic plan that was to be developed in Year 1 of the activity is yet to be done. NASCOP and DFH do not have current commodity strategic plans County Strategic and operational planning(AWP) Where AU supported strategic plan and AWP development, their support was highly appreciated. A case in point is the development of a costed SC/CM strategic plan for Isiolo Regional and National Forums Although not many people interviewed had attended these forums those that had said they found them relevant and helpful and allowed them to network with their county and national level counterparts. 3. Safety of health products Pharmacovigilance Involvement of AU in these interventions has largely been limited to training/orientation. AU has also not worked with the Pharmacy and Poisons Board on strengthening these areas. Rational Drug Use Afya Ugavi Mid-Term Review, May 2019 Page 21 # Pillar of a functioning health supply chain system AU Interventions Performance Rating Remarks 4. HRH Capacity Building (Training, CMEs, OJT and Mentorship) Extensive training has been done that has reached a significant proportion of relevant health staff. Most respondents in counties mentioned supportive supervision, OJT and CME’s as being very helpful for their work. 5. Information Systems Data capture tools AU had supported availing of data capture tools e.g. Malaria Commodities Daily Activityregister by working at both the national and county level. However, most data tools were manual and electronic tools in use were fragmented and often unlinked. Specifically, commodity electronic tools were not linked with electronic service delivery tools Improving reporting rates (Quality, timeliness and completeness) Significant gains reported and attributed to several AU interventions including facilitation of facility in/charges and data review meetings. Quality of data still lags completeness and timeliness Dashboards (FP, Malaria and HIV) This was rated as the most helpful AU intervention. It had greatly enhanced visibility of commodity data thus supporting data driven decision making. Respondents were happy with the DHIS2 inter-operability of the dashboards. EMMS Tool (Dashboard) The design and roll-out of this tool is still evolving and there are lingering issues for example with DHIS2 inter￾operability. Respondents are however looking forward to having this tool fully operationalized as it will give them full visibility of not only program commodities but also of EMMS 6. Commodity Financing Planning and budgeting This was done as part of quantification and AWP. The Isiolo SC/CM Strategic plan was also fully costed. However, this had not been done for a number of counties visited. Advocacy for increased County funding for health commodities In the counties visited, health commodities budgets were on a downward trajectory. AU was said to have established excellent relations with mid-level county managers but rarely engaged top level county leadership. Although AU does not have an express mandate to support advocacy for enhanced county level funding for EMMS commodities, it has an indirect role through support for strategic plan and annual workplan development Minimizing Leakages and expiries More rational ordering and re￾distribution made possible by the commodity dashboards has reduced Afya Ugavi Mid-Term Review, May 2019 Page 22 # Pillar of a functioning health supply chain system AU Interventions Performance Rating Remarks expiries. More robust inventory management including better record keeping and conducting of supply chain audit was said to have helped detect and reduce leakages and identify commodity management skills gaps. A lingering challenge is the timely and safe disposal of expired commodities. Key: Meets Expectations Partially Met Expectations Below Expectations 4.2. Detailed MTR Findings 4.2.1. Sub-Question 1.1: Is the AfyaUgavi Activity on track to achieve its stated objectives? Sub-Question 2.1: How have GOK counterparts benefitted from this activity at the national level? At the county level? What about other implementing partners? 4.2.1.1. AU Performance against PMP Indicators, baseline and Targets AU has demonstrated improvements in several key indicators in their PMP. It was not easy tracking these changes as the PMP has undergone significant changes. Figure 2 and 3, highlight examples of how the program has performed against its PMP indicators and targets. Figure 2: % Stock-Out Rates of all ALs at Service Delivery Points Source: AU Year 2 Q4 PMP Report 0 1 2 3 4 5 6 7 8 9 10 2016 BASELINE YEAR 1 YEAR 2 Q1 YEAR 2 Q2 YEAR 2 Q3 YEAR 2 Q4 7 9.9 5 4.6 4.5 5.4 Afya Ugavi Mid-Term Review, May 2019 Page 23 Stock-out rates had decreased significantly after a spike in year 1 and were lower than baseline in year 2. However, the reduction seems to have plateaued. It's important to note that these averages mask some significant variability at facility levels. For example Bushiri Health Centre (Kakamega County) had 42 days out of stock for RDT's and Bumala Health Centre (Busia County) had 78 days out of stock for AL (6 by 2) in the first quarter of 2019. Figure 3: National SDP Reporting Rates to DHIS2 Source: AU Year 3 Q1 PMP Report Similar progress is evident in NCMP where the monthly national reportingrates for commodities have improved from 84% to 89% between February 2018 and January 2019. The reporting rates in the lake endemic counties that have benefitted from AU support have over 90% reporting rates. Increase in Community Case Management (CCM) commodity reporting rates have also been observed. Figure 4 shows the national commodity reporting rate Feb 2018 – Jan 2019. Figure 4: National Malaria Com m odity Reporting Rates - Feb 2018 - Jan 2019 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 2016 BASELINE YEAR 1 YEAR 2QI 2018Q2 YEAR 3 Q3 YEAR 4 Q4 YEAR 3 Q1 FP MNH HIV Malaria Afya Ugavi Mid-Term Review, May 2019 Page 24 Source: NMCP AU support has also resulted in improvement in stocking level at the national level for program commodities. Most of the NMCP commodities were found to be within the recommended 9-18 months of Stock holding. Figure 5 shows months of stock for malaria commodities (Feb 2019) at national level. Figure 5: Months of Stock for Malaria Com m odities (National) - Feb 2019 Source: NMCP 4.2.1.2. AU Performance against Annual Workplan A review of AU workplans and quarterly and annual reports showed that the activity was executing its workplans and reasons for any significant deviations were given in the annual and quarterly reports. However, many national level activities were interrupted by USAID’s suspension of support to national level MoH institutions. 4.2.1.3. AU Organizational Performance and Impact The evaluation team found that the activity has established itself well especially at the county level and its interventions are making a difference in the status of supply chain and the wider health system. AU has made these achievements notwithstanding significant challenges including suspension of USAID Afya Ugavi Mid-Term Review, May 2019 Page 25 funding to national MoH institutions and the protracted industrial action by health workers. Respondents that had interacted with AU rated it highly and spoke of it as an efficient and responsive ‘organization’ with competent and committed staff. Table 9 summarizes perceptions of county level interview respondents on AU: Table 9: Respondents Perception of AU Perform ance and Im pact Perception Area Weighted Perception Score Isiolo Homa Bay Kakamega Busia Baringo Turkana Aggregate for the Six Counties Relevance of interventions 4.9 4.6 4.3 4.3 4.6 4.3 4.5 Effectiveness of interventions 4.6 4.5 4.2 4.3 4.8 4.6 4.5 Knowledgeable staff 4.8 4.4 4.3 4.7 4.8 4.4 4.6 Collaborative staff 4.8 4.6 4.0 4.5 4.8 4.3 4.5 Overall impact 4.4 4.4 3.9 4.1 4.5 4.1 4.2 Source: Baringo County Health Department Note: Perception was based on a scale of 1 to 5 with 1 being below average, 2 average, 3 good, 4 very good, and 5 excellent. County level respondents said the AU Activity has had significant impact on SC/CM function. The most frequently mentioned areas of AU impact were: • Improved commodity reporting (cited in 11 interviews) • Reduced expiries (cited in 10 interviews) • Enhanced commodity management knowledge and skills (cited in 9 interviews) • Improved inventory management (cited in 9 interviews) • Improved commodity stocking (cited in 8 interviews) • Functional CS TWG (cited in 5 interviews) 4.2.1.4. Achievement in AU Cross Interventions Commodity Security Technical Working Groups (CSTWG) Afya Ugavi has established or revived County CSTWGs in all the 14 counties in which it operates. The establishment of CSTWGs involved development of TORs to guide the CSTWGs activities. The membership of the County CSTWGs includes key CHMT members that play key roles in the health commodity supply chain and management. AU has also facilitated the regular meeting of the County CSTWGs. AU has supported the establishment and operationalization of Sub-County CSTWGs in all sub-counties in Malaria focus counties and in the targeted sub-counties in FP focus counties. The evaluation team established that both county and sub-county CSTWGs are fully functional and hold regular meetings with good quorum. Minutes of the meetings are taken and circulated and from review of the minutes there is good follow-up of agreed actions. In the Malaria counties, we observed good linkages between County and Sub-County CSTWGs with select members of the Sub-County CSTWGs regularly attending County CSTWGs meetings. The coordination between the two CSTWGs was weaker in FP counties where attendance of County CSTWG meetings by Sub-County CSTWG members was erratic and the linkages between the two levels tenuous. Overall, we found that the CSTWGs serve as a hub for all health CS/CM activities, including those supported by other partners or by the counties themselves. Respondents reported that the CSTWG’s had significantly improved coordination and harmonisation of SC/CM activities, teamwork, problem-solving, Afya Ugavi Mid-Term Review, May 2019 Page 26 information sharing and learning. They had also given greater visibility and focus to SC/CM issues at county level. Support for Strategic Plans and Annual County Workplans Development • County Level AU has supported the development of County Health Strategic Plans and also the annual workplan development process. It was reported that this has helped ensure that the strategic and annual workplans adequately address issues of planning, procurement and management of health commodities. However, in some of the counties visited such as Homa Bay, Kakamega and Busia, CSTWG members interviewed were uncertain about the extent to which their existing county health strategic plans addressed health commodity issues. However, all the county and sub-county CSTWGs interviewed had actively participated in the process of developing Annual County Health Workplans and felt that these adequately addressed issues of health commodities. In Isiolo County, AU supported the development of Kenya’s first costed County Health Commodity Strategic Plan. The County’s CSTWG members felt that this strategic plan would be extremely useful in mobilizing resources for health commodities from the county, national governments and from partners. Along with the SC/CM Strategic plan, Isiolo also developed a county drug formulary to foster more efficient use of commodity funds and strengthen rational drug use. A review of the Isiolo Health Sector Strategic and Investment Plan, 2018-2023, revealed significant congruence with the AU package of interventions. The strategies included in the county SC/CM strategic plan were: training on quantification, commodity management, quarterly support supervision, capacity building on commodity management, strengthening of commodity re-distribution and provision of pharmacovigilance tools. • National level In Year 2, AU supported the review of the 2009-2018 Kenya Malaria Strategy. AU specifically supported the review of the Procurement and Supplies Management (PSM) section of the strategy. In Quarter 1 of the third year, AU started supporting NMCP to develop and cost the next Kenya Malaria Strategy (KMS) for the period 2019 to 2023. This strategy has since been launched. Monthly and Quarterly Facility In-Charges Meetings AU has regularly supported facility In-charges meetings. Depending on the counties and the health focus areas, these are held at the county and/or sub-county level. During these meetings a wide range of health supply chain and commodity management issues are addressed including review of stock status and commodity reporting. Although these meetings may have a slight bias towards the AU health focus area, the respondents interviewed stated that the meetings addressed broad health SC/CM issues across health areas including EMMS. In the FP focus counties, AU worked very closely with USG partners Afya UZAZI and Afya TIMIZA in facilitating these meetings. Respondents credited holding of regular facility In-Charges meetings with most of the improvements in service and commodity reporting rates. Health managers from large counties and/or with poor infrastructure such as Isiolo, Turkana and Baringo were particularly appreciative of this support given the huge distances and poor roads between facilities. In counties such as Baringo and Turkana where AU does not cover all the sub-counties, it was reported that AU or other partners facilitated staff from facilities in non-AU sub-counties to attend the facility In-charges meetings. Quantification • County Level Afya Ugavi Mid-Term Review, May 2019 Page 27 In Isiolo, AU supported a comprehensive quantification exercise for all health commodities including EMMS and non-pharms. The Isiolo team reported that the quantification exercise was invaluable in the development of their health commodities estimates for the ongoing UHC pilot project. Isiolo is one of the four counties piloting UHC that is fully funded by the national government. As part of this program, the national government is funding the county’s entire health commodity budget for a whole year. This includes EMMS, lab commodities and non-pharms. AU also supported a quantification exercise in Baringo and Turkana and one was planned for Busia in July 2019. Quantification exercises were yet to be carried out in Homa Bay and Kakamega. • National Level At the National level, AU has been most active in supporting quantification exercises and quantification reviews for the Malaria program. In Year 1, AU supported aquantificationexercise for Malaria commodities as part of a successful Global fund grant proposal. In Year 2, AU conducted another quantification exercise for malaria commodities. In year 2 Afya Ugavi participated in the national annual HIV quantification workshop and also carried out forecast analysis for antiretroviral medicines. Commodity Management Focal Facilities and Centres of Learning The evaluation team found that AU is at different stages of establishing Commodity Management Focal Facilities or Centres of Learning (COL) in all the six counties visited. The two main interventions introduced by AU in creating focal facilities and centres of learning was capacity building through CMEs and renovation and refurbishment of pharmacies and stores including fitting of shelves/racking and furniture. In some focal facilities quality improvement teams (QIT) and work improvement teams (WIT) had been set up. All the respondents interviewed were of the view that introduction of focal facilities and centres of learning was an effective intervention in the improvement of health SC/CM in their counties. Most respondents interviewed at the county and sub-county levels stated they were consulted in the selection of these facilities and were aware of the criteria used in the selection. However, the Homa Bay County CSTWG members stated they were not consulted in the selection of the COLs. The evaluation team found that the understanding of the concept including the stages of evolution into a focal facility or centre of learning was poor especially in the two FP focal counties visited – Baringo and Turkana. In these counties, respondents viewed renovation of pharmacies and drug stores as the key goal and the only requirement for transitioning to commodity focal facilities. Understanding of the concept was highest in Isiolo and Kakamega where the local teams had worked with AU to draw roadmaps towards achieving the goal of creating commodity focal facilities and centres of learning that met defined standards. Respondents did not have plans on how the good SC/CM practices in commodity focal facilities and centres of learning would be cascaded to poorer performing facilities. However, one commodity management focal facility reported having received staff from 4 facilities who had come to learn how to improve their own stores. The evaluation team also found that in some of the facilities visited such as Emining in Baringo and Lorugum in Turkana, the quality of renovation was poor. Overall the team found a big disparity in how this intervention has been rolled out in different counties with high clarity of the intervention in some counties and complete lack of clarity in others. End User Verification In Isiolo, AU supported an end user verification exercise for HIV and other commodities to track the SC/CM documentation trail to ascertain that medicines ordered are received, distributed and dispensed to reach the end user. The exercise revealed that program drugs supplied by and large reached intended users. Pharmacovigilance and Poor-Quality Products Afya Ugavi Mid-Term Review, May 2019 Page 28 AU has trained county staff involved in health SC/CM on pharmacovigilance (PV). There were no other interventions of AU directed at strengthening PV. Respondents reported several Adverse Drug Reactions (ADRs) with HIV, TB and EMMS commodities. However, although these ADRs were documented in patient records it appears that most were not reported to the Pharmacy and Poisons Board (PPB). For cases reported to PPB, respondents complained that in most cases they got no further feedback other than the online acknowledgement. Respondents also reported a number of poor-quality products mainly EMMS commodities that were reported to PPB and KEMSA. However, respondent complained that KEMSA had neither collected the poor-quality commodities nor compensated counties for the loss. Supportive Supervision AU has supported quarterly supportive supervision visits in all the counties they support. The supervision is guided by a comprehensive checklist. In the first year, the checklist was largely biased to the focus health area supported by AU in the respective county. However, in the second and third year, AU has introduced(and in some counties is in the process of introducing) a more integrated checklist that addresses a wider range of commodities - program, EMMS and non-pharms. Supportive supervision was one of the AU interventions ranked highest by respondents for impact. Respondents stated that the comprehensive checklist was one of the reasons they found supportive supervision so useful. The checklist was used to document and score health commodities SC/CM performance and to identify skills gaps. AU and county health managers conducted OJT and mentorship in identified areas of weakness during the support supervision. In Baringo we were also informed that AU facilitated county staff to revisit health facilities to mentor staff on identified knowledge and skills gaps. The Isiolo CSTWG stated that AU supported broad supportive supervision where health commodity issues were only a part. They were of the view that AU needed to support commodity focused supportive supervision. In some counties such as Busia it was reported that going through the supportive supervision checklist was very time consuming that they were not able to cover all the facilities in a sub-county within the one week that AU facilitated the supervision. They were of the view that the checklist should remain as is, but more days should be added for the activity. An innovation noted by the team was the ranking of sub-counties based on supportive supervision scores. Good performing sub-counties were recognised and given awards. This was said to have motivated all sub-counties to improve. Health Commodity Re-Distribution AU supported the redistribution of commodities from facilities that had excess stock to those with deficits. Success of the redistribution was based on the use of commodity dashboards (county-wide stock data visibility from facility to county) and facility visits during supportive supervision and OJT which were facilitated by AU. Respondents credited significant improvements in stock availability and reduction of expiries to the redistribution. Renovation and equipping of Pharmacies and stores AU had provided equipment and devices to support pharmacies and stores. These included thermometers, hygrometers, calculators and tablet counters. The evaluation team found that these devices were being correctly used by the facilities visited. For example, temperature charts in the drug stores visited were regularly updated. It was however noted that only a few commodity stores had air-conditioning equipment (donated by AMREF and other partners) and hence temperature control was a challenge despite keeping good charts. AU was also undertaking renovation of pharmacies and Afya Ugavi Mid-Term Review, May 2019 Page 29 stores in designated Commodity Management focal facilities/Centres of Learning. The renovation included erecting shelves and racking, providing furniture, improving ventilation and painting the stores and pharmacies. In Isiolo, AU had also constructed a covered commodity receiving bay to protect deliveries from the elements. Respondents were very appreciative of the renovations undertaken and reported that this had led to improvements in commodity management. In Turkana, it was reported that three health facilities – Kasegol Etom Dispensary, Lobei Health centre and Lomil Dispensary successfully approached the World Bank Supported RBF program to renovate their stores based on the renovations AU had done in focal facilities. Capacity Building AU has supported several capacity building initiatives. These include rolling out a generic and structured health commodity training, including leadership development. This appears to have reached a large proportion of health managers that have a role in health supply chain and commodity management in most of the counties visited. The program also supported several CMEs most of which were biased to the AU health focus area in the specific county. The largest number of CMEs were conducted in Malaria focus counties. In some counties, AU conducted TOT training while in others they used existing trainers in the county. Respondents interviewed stated that the SC/CM training and CMEs had improved motivation and helped enhance knowledge and skills as evidenced by better commodity management practices and improving scores based on the supportive supervision checklist. AU recently conducted a SC/CM leadership program (Capstone Project) that participants rated very highly and were especially excited about the project they must undertake as part of the training. OJT and mentorship were conducted as part of the supportive supervision. However, the evaluators found that OJT and mentorship were not structured, and no distinction was made between the two. Respondents reported that training needs for SC/CM remain, especially in areas of commodity budgeting and quality improvement and also there remains a need to cover newly recruited employees and those working in lower level facilities. Regional and International Forums AU has supported several regional forums and facilitated participation in one international forum. The team only met a few respondents that had attended these forums, but those that had attended stated that the forums were very helpful. The regional forumshadenabledthem establish linkages and supportive relationships with other counties and national level institutions such as NMCP and the DHIS2 support team from national MoH. Improving Commodity Data Quality and Reporting AU has supported several initiatives intended to improve commodity data quality and reporting. These include: • Supporting data quality review meetings • Supporting Facility In-charges meetings • Support for County and Sub-county CSTWG meetings that discuss data quality as an agenda item • Support for quarterly supportive supervision where commodity data is reviewed • Promoting use of commodity dashboards These interventions have led to significant improvements in reporting rates - timeliness, completeness and quality especially of commodities for the AU focal health areas – FP, HIV and Malaria. Data from the AU PMP and county Commodity Dashboards shows a clear improvement trend in reporting rates. In most cases, these stand at or near 100% although quality (particularly data accuracy) lags behind Afya Ugavi Mid-Term Review, May 2019 Page 30 timeliness and completeness. It was however reported that these gains can be very fragile as they are very dependent on partner facilitation and in the past have see-sawed from near perfect to quite low. Afya Ugavi Mid-Term Review, May 2019 Page 31 Figure 6 : FP Com m odity Reporting Rates – Baringo Source: Baringo County Health Department Health Commodities Dashboards AU has developed comprehensive health commodities dashboards for FP, HIV and Malaria. The dashboards hold a wide range of county, sub-county and facility level health commodities information including stock status and reporting rates. AU has incorporated DHIS2 functionality for all the dashboards which ensures that, as long as reliable data is being fed into the DHIS2, then the Dashboard generates accurate statistics and summaries. In counties such as Baringo and Turkana where AU does not cover all the sub-counties, the Dashboards were still comprehensive and incorporated facilities in non-AU sub-Counties. The respondents at county and sub-county level reported that before the dashboards were introduced, they did not have access to accurate and timely health commodity information at facility level (facility to county data visibility) and especially for facilities in far flung areas of large counties such as Isiolo and Turkana. The respondents reported that the Dashboards helped them make informed decisions especially in placing rational orders and timely re-distribution of products from over-stocked to under-stocked facilities. In addition to the dashboard for the specific focus health area(s), AU has gradually introduced dashboards for the other program areas. For example, Turkana which is an FP focus county now also has Malaria and HIV dashboards. The County CSTWG in Baringo (An FP Focus County) stated that when they had a Malaria outbreak, AU introduced a Malaria dashboard that enabled them track malaria commodities stocks in the affected sub-county which was very useful in enabling them control the outbreak. Dashboards were mentioned by most respondents as a best practice. AU Activity has made some progress towards supporting national scale-up of Malaria commodities dashboard. AU has provided technical assistance in collaboration with HealthIT, for the development of a dashboard application within DHIS2 that mirrors the functionality provided in the Excel-based dashboard. This Malaria commodities dashboard is linked to KEMSA LMIS with data being exchanged between DHIS2 and KEMSA LMIS In the recent past AU has introduced or is in the process of introducing an EMMS tracking tool in all the counties in which it operates. In developing this tool, AU has sought the inputs of the different counties and especially in deciding which EMMS commodities should be prioritized for inclusion. Automation of Services and Commodity Management 0 10 20 30 40 50 60 70 80 90 100 AU Supported Sub￾Counties Non-AU Supported Sub-counties County Average 92.9 88.8 90.5 Afya Ugavi Mid-Term Review, May 2019 Page 32 In all the counties visited, automation of clinical services and commodity management was found to be patchy and fragmented. Many CCCs were using either IQCARE or Kenya EMR for services data and Web ADT for commodity data. In some cases, the services data was captured at the point of care while in other cases it was captured on paper and transferred to the electronic tools later. Many CCCs complained of struggling to keep Web ADT working due to lack of IT support. ADT was initially developed by HCSM the AU forerunner. EDITT, also developed by HCSM, had been abandoned by the users visited also due to lack of IT support. Another challenge reported was the fact that IQCARE and Web ADT are not linked to DHIS2. In some of the counties, there were different IT systems in use for outpatient care some of which had an inventory management module. Where such systems were available, they were mainly used in the county referral hospital and in some cases in one or more sub-county hospitals. None of the counties visited had comprehensive Information Technology (IT) systems/Electronic Medical Records (EMRs) or even plans to introduce them. Except for the commodity Dashboard AU was not providing any support towards automation of supply chain and commodity management at the county level. While automation of clinical and commodity management is not an AU mandate, the prevailing situation has adversely affected AU due to challenges with availability of timely and accurate commodity data especially at the facility, sub-county and county levels. Commodity dashboards have ameliorated this challenge, but they are not a comprehensive solution by themselves. Integration of Interventions across Health focus Areas The program has done a very good job of integrating SC/CM strengthening to cover other commodities in addition to those of the focal health area. This integration drive started in the second year but has accelerated in the third year of the activity. In Kakamega and other Malaria counties, integration has been formalized in a pilot program that covers Kakamega and Kisumu. The main way in which AU has supported an integration approach include: • Use of an integrated supportive supervision checklist • Rolling out additional dashboards and tools to cover additional health focus areas including EMMS • Carrying out broad quantification for all health focus areas – HIV, malaria and FP/RMNCAH and EMMS • Collaborating with USG supported implementing partners covering additional health focus areas One county respondent opined that AU needs to make the integration initiatives more intentional and systematic and less ad hoc. Advocacy for Enhanced Health commodity Funding The role of AU in supporting enhanced county budgets to support procurement of health commodities and to fund SC/CM support activities has largely been limited to support for ensuring commodity issues and financing are adequately addressed in health strategic plans and annual workplans. A major observation of the evaluation team was the low level of confidence and knowledge exhibited by County CSTWGs in discussing health commodity financing and budgets. Their involvement appears to be limited to the AWP process and we did not find a robust, multi-year budgeting culture for health commodities. It was also reported that AU has developed excellent relationships with the county “At the beginning of the program , AU staff only spoke about FP com m odities - how we ordered, where we stored them , how we m aintained bin cards, how we reported, how we tracked expiries etc. they were like an FP cult” County Pharm acist” “We initially viewed AU as a Malaria program , but today we see them as a health com m odity m anagem ent strengthening partner” Sub-County CS TWG m em ber Afya Ugavi Mid-Term Review, May 2019 Page 33 technical teams but needed to improve their influence with the county political actors including CEC members, MCAs and the governor. Baringo, Busia and Kakamega all reported that their health commodity budgets have been in decline over the last four years. For example, health commodity budgets for Baringo County had declined by nearly 5 % between 2016/17 and 2018 /19 financial years as depicted in Table 10. Table 10: Baringo County Budget Trends Budget Category Financial Year (Million Kes) 2018/19 Vs 2016/17 % Growth 2016/17 2017/18 2018/19 Total County Budget 6,511 6,959 8,029 23.3 Health Budget 2,355 2,547 2,708 15.0 Medical Supplies Budget 174 151 166 -4.6 Health budget as % of total budget 36.2 36.6 33.7 -2.5 Medical supplies budget as % of health budget 7.4 5.9 6.1 -1.3 Source: Baringo County Health Department In Homa Bay, the county approved 70%, 26% and 24% of requested budgets for pharmaceuticals, lab commodities and non-pharms respectively. Health commodity budgets had also come down for Busia with only 37% of budget requested being approved for the 2018/19 financial year. Kakamega county staff reported that in addition to health commodity budgets being slashed, a significant amount of the budget was consumed by pending bills from the previous financial year. Another finding was that despite regular annual budgeting, all the counties visited (except Kakamega) still do not fund CM/SC support activities such as transport, airtime, printing and provision of tools, cost of meetings (data review meetings, TWG meetings and facility in-charges meetings) and supportive supervision costs. These SC/CM support activities are essential for sustaining and improving the performance of the CM and SC functions. All counties mentioned this challenge as a major barrier to the sustainability of the work done and progress achieved through AU support. There is also little awareness and no visibility about the cost of program commodities that are supplied at no cost to the counties. In the new era of expanding UHC initiatives, public health facilities will generate additional funds from services delivered to NHIF members. Unfortunately, the public financial management Act significantly constrains public facilities from using such funds to procure commodities and supplies (both essential commodities and supplementary commodities for insured member benefits, for example medicines for managing NCD’s). Some counties such as Busia are in the process of amending the PFM Act to better ring-fence the funds for healthcare use and allow greater facility autonomy to apply the funds to improve services, including commodity availability. Two counties visited have serious challenges paying KEMSA bills regularly and this has affected the supply and availability of EMMS commodities. Some of these counties have ended up doing their own procurements from local distributors potentially exposing themselves to the risks of poor-quality commodities and significantly higher commodity prices. Commodity Reporting Tools Counties reported huge challenges accessing current commodity reporting tools. Kakamega was the only county with significant quantities of service and commodity reporting tools. In other counties it was reported that these tools were included in the budget but were never procured. AU was helping counties with photocopying of some of the commodity tools. However, the tools are still largely Afya Ugavi Mid-Term Review, May 2019 Page 34 manual and fragmented. AU has introduced digital dashboards for commodity management and this has alleviated the challenges, though they remain unresolved. The FP, Malaria and HIV dashboards are fully integrated with DHIS2. This is a big advantage over earlier dashboards. Job Aids AU had developed several SC/CM job aids that the evaluation team found displayed in many facilities and that were being used to improve commodity management. The job aid that the team found being used the most was the expiry tracking chart. Global Health Security Agenda (GHSA) AU Activity has an additional objective of strengthening Kenya’s capacity to prevent, detect, and respond to human and animal infectious diseases threats through the Global Health Security Agenda (GHSA) partnership. GHSA is a partnership of international organizations and non-governmental stakeholders whose overall goal is to build countries’ capacity towards a world safe and secure from infectious disease threats. Afya Ugavi conducted a situational assessment in Year 2 to determine Kenya's general disaster preparedness. The findings informed the development of the Emergency Supply Chain (ESC) Framework for Kenya. AU gave technical and logistical support to the GHSA partners meetings. AU supported the development and costing of the National Action Plan for Public Health Systems (NAPHS) medical countermeasures matrix. AU, working with two lead ministries - the Ministry of Health (MoH) and the Ministry of Agriculture, Livestock, Fisheries, and Irrigation (MoALFI) -and the Council of Governors developed adraft Emergency Supply Chain (ESC) Framework. This draft has been tested using an emergency response simulation exercise. Respondents were of the view that implementation has been slow. They also felt that there was need for more involvement of counties in the process of finalizing and implementing the ESC framework. Funding for AU’s GHSA activities beyond the current year is uncertain. 4.2.2.Sub-Question 1.2: What was the quality of the work conducted by Chemonics? Interview respondents stated that AU interventions were of high quality both by design and execution. It was also reported that AU had factored in feedback from its beneficiaries in designing or re-designing its tools and interventions. A good example is the development of the EMMS tool which was collaboratively designed by county teams and AU. This included the selection of EMMS tracer commodities and other non-pharm commodities to be included in the tool. Such collaboration ensured that interventions and tools developed were well aligned with client/beneficiary needs thus satisfying the cardinal definition of quality - “fitness for purpose”. The evaluation team also found that AU had used findings of the baseline and other assessments in designing its interventions. Table 11shows the perception ratings of county respondents on quality of AU interventions. Table 11 : AU Interventions Quality Perceptions Score Perception Area Weighted Perception Score by Respondents Isiolo Homa Bay Kakamega Busia Baringo Turkana Aggregate for the Six Counties Quality of interventions 4.8 4.5 3.8 4.3 4.7 4.0 4.3 Note: Perception was based on a scale of 1 to 5 with 1 being below average, 2 average, 3 good, 4 very good, and 5 excellent. Afya Ugavi Mid-Term Review, May 2019 Page 35 Concerns however were expressed on the quality of store renovations carried out in Turkana (Lorgum Sub-County Hospital) and Baringo (Emining Health Centre) and it was reported that some remedial measures had been taken to improve the same. 4.2.3. Sub-Question1.3: How efficiently have resources been utilized in the Afya Ugavi implementation Based on the views of interview respondents and the analysis of the evaluation team, AU has in general used the resources at its disposal efficiently. Table 12: Perception scores for Efficient Use of Resources by AU Perception Area Weighted Perception Score Isiolo Homa Bay Kakamega Busia Baringo Turkana Aggregate for the Six Counties Efficient use of resources 4.3 4.6 4.3 4.5 4.6 4.5 4.5 Note: Perception was based on a scale of 1 to 5 with 1 being below average, 2 average, 3 good, 4 very good, and 5 excellent. The evaluation team found that the Malaria component has been the most cost efficient based on national and county reach and overall volume activities including sub-counties reached - 63 versus nine for FP/RMNCAH and other indicators such as number of people trained. It is also the view of the evaluation team that scaling-up integration of interventions has had a positive impact on efficiency. 4.2. 4 Sub-Question1.4: How effective has the project’s measurement approach been in monitoring project implementation? AU has done a good job of rationalizing and simplifying its Performance Management Plan (PMP). The year 1 PMP was far too broad and had many indicators that were peripheral to the activity scope or were difficult to measure. The grouping of the AU indicators into three categories - Cross cutting indicators (5), Sustainability indicators (5) and Program based indicators (7) as articulated in AU Year 3 MEL Plan represents real progress on how the activity monitors and evaluates its work. AU Year 3 MEL plan has a total of 17 indicators down from 50 in the year 2 MEL plan. The evaluation team found that three sustainability indicators were either difficult to measure or suffered from a validity problem. These indicators are: • Indicator Name: initially Afya Ugavi focussed on promoting supply chain functions to be carried out by national and county authorities without external technical assistance. Concern: For this to be valid, there must be a proactive AU plan preferably with timelines to build the capacity of the targeted institutions to independently carry identified supply chain functions without external support. For areas such as HIV commodities where it is envisaged some form of external support will continue for years to come, it is difficult to delineate those functions that get external support and those that do not. • Indicator Name: Supply chain maturity level demonstrated through the Centres of Learning Maturity Index. Concern: A high maturity index may reflect extent of AU support and not necessarily greater ownership by counties and sustainability. • Indicator Name: Number of new innovations (including operations research studies) that were developed, implemented, or introduced and are related to the health commodity market Afya Ugavi Mid-Term Review, May 2019 Page 36 or supply chain best practices. Concern: There is need for a more robust operational definition of “innovation” and “best practice” in the context of AU Activity. The proposed definition of innovation -new technologies, new products, new approaches and/or operational research studies developed appears inadequate in the absence of demonstrable impact on SC/CM outcomes. 4.2.5 Sub-question1.5: What are the most effective sustainability strategies that Afya Ugavi has employed and is the GOK likely to sustain these advances after the program is finished? The evaluation team found that there were a number of AU interventions and broader contexts that were either fostering or constraining sustainability. Sustainability was fostered through: • Support to County and Sub-County CS TWG and various national level TWG’s. • Capacity building and skills transfer (particularly the supportive supervision, OJT, mentorship and CME’s). • Working through and strengthening existing county and sub-county structures and systems. • Partnering with national institutions such as Nairobi University Health IT. • Collaboration with other USG partners to synergise resources and broaden impact. Sustainability was constrained by: • Initial verticalized design and approach of the AU interventions. • Field level constraints for sustainability including: – Limited Financing for SC/CM support activities (Most counties’ budgets limited to procurement of commodities). – A trend of reduction in county commodity budgets in a number of counties visited. – Public health program commodities are currently funded centrally through partner and national government support. Transitioning this funding to national and county levels will be a major leap. – HRH limitations (numbers, retention and skills) in the context of expected rapid increase in demand from UHC initiatives. • Minimal engagement of high-level county leadership by AU - (County Executive Committee Members (CEC), Chief officers, County Assembly committees) and Council of Governors (COG) for advocacy and ownership. Given these findings, sustainability of AU interventions will depend on the program enhancing its support for mobilisation of greaterSC/CM budgets at national and county level including budgeting for program commodities and SC/CM activities such as supervision and data review meetings. To achieve this AU needs to target high level county leadership - (CEC, Chief Officers, County Assembly committees) and COG. Other health system strengthening interventions necessary to foster sustainability (such as HRH, leadership and governance, health informatics architecture, private sector engagement) are outside the scope of AU. However, AU may need to play the role of collecting relevant data and information on how these health system components affect SC/CM and advocate for their strengthening or initiate partnerships and collaborations to close the gaps. In order to better institutionalise SC/CM functions, AU has developed a concept note titled - “Establishment of a Health Products and Technologies Unit at County Level of the Health System.” The paper highlights the current unsustainable situation where SC/CM roles and functions are spread over multiple offices and departments within the County Health Department leading to poor coordination and inefficiencies. The concept note proposes the establishment of a county Health Products & Technologies Unit (HPTU) that will organize, monitor, and support all activities within the Afya Ugavi Mid-Term Review, May 2019 Page 37 supply chain system. The proposed functions of a HPTU include Supply chain data management, Quantification; Monitoring & Evaluation; Supervision; Supply Chain Design; Training; and Coordination and Collaboration. AU will advocate for the full staffing of this unit and proposes to provide needed tools and capacity building. 4.2.6 Other Relevant Findings • Changes to the national and County Context Affecting AU Several changes in the National and county context have implications on the performance of theAU Activity. These changes include: o Suspension of USAID funding to national level MoH institutions. This has had a great impact especially on national planned level interventions. However, the effect has not been uniform across the program areas. HIV/AIDS appeared to be the most affected. o Insecurity: Insecurity experienced in Turkana, Isiolo and Baringo has affected ability of health and AU Activity staff to travel freely within the county. One of the major casualties is supportive supervision visits to facilities which affects all other SC/CM indictors in the affected areas. o Protracted HRH Industrial Action – This has especially affected doctors and nurses. For example, during the nurses’ strike there were significant reduction in service and commodity reporting rates. Health workers strikes were also said to have led to expiry of program commodities especially test kits as patient numbers dwindled. Some of the strikes have been national while others have affected specific counties. o Introduction of Universal Health Coverage (UHC) leading to a major spike in demand for health services and commodities. • SC/CM Implications of Universal Health Coverage (UHC) Two of the four pilot UHC counties in Kenya are AU supported. These are Isiolo and Kisumu. The other two pilot counties are Nyeri and Machakos. In Isiolo 45% of the Isiolo UHC funding (Kes 391 million) is going to health commodities. It was reported that the rolling out of the UHC program had significantly affected the AU Activity. The pilot phase will run for one financial year – 2018/19. Major changes and challenges reported include: o Massive jump in outpatient workload including particularly in outpatient clinics, laboratory services and pharmacy clients. Table 13 illustrates the situation in Isiolo County. Table 13: Percentage % Workload Growth in Isiolo County Post UHC Introduction Service % Year on Year Workload Growth (February Week 3) Family Planning 103 Laboratory Services 9 Pharmacy 156 Source: Isiolo County CHMT service delivery reports o Over-stretched commodity storage capacity - With support of the National government the county has received huge quantities of products both pharmaceutical and non-pharms. o Lack of integrated EMR’s to efficiently manage the growing workload -The County is expecting financial support to strengthen IT infrastructure to support both service delivery and commodity management. Afya Ugavi Mid-Term Review, May 2019 Page 38 o Appropriate use of resources and risk of crowding out county funds -For the current financial year, the county has made big savings originally budgeted for purchase of health commodities. The evaluation team however found that there are no plans to utilize these savings to strengthen SC/CM systems and infrastructure. • Commodity Expiry The issue of commodity expiry remains a low to moderate level problem for the counties visited. Most of the expiries affect program commodities - HIV, FP and Malaria. In most of these cases, respondents mentioned that low shelf-life commodities were “pushed” to them beyond their consumption capacity. It was however, reported that all commodities with less than six months shelf￾life at the time of issuing by KEMSA are despatched with the express written concurrence of the county commodity managers (pharmacist and lab coordinator) and the assurance that the health facilities receiving them will be able to use the commodities before expiry. The challenge of accurately forecasting demand for Malaria commodities in regions that experience highland Malaria was also said to have contributed to expiries. County respondents stated that the problems of expiries were smaller for EMMS commodities. This was said to be because these are commodities the county pays for and could therefore reject low shelf-life supplies. Also given the limited county commodity budgets, the risk of over-ordering and hence expiry risk was small. Lengthy industrial action especially by nurses and doctors had also contributed to product expiries. However, respondents reported that with better commodity planning, more rational ordering and AU supported re-distribution, the problem of expiries was becoming smaller. A lingering problem is one of disposal of expired products leading to products that expired years ago taking up scarce storage space. The challenges with disposal of expired products were said to arise from the following: o Constraints imposed by the Public Procurement and asset Disposal Act, 2015 o Failure by priority Health Programs to collect expired products in a timely manner o Lack of recommended incineration facilities for disposal of medical products and devices o Fear of victimisation of facility staff for old expiries for which they consider themselves not responsible. The evaluation team found that except in one instance in Homa Bay, counties are not costing expired products. • Staffing of the SC/CM Function It was reported that most SC/CM function was understaffed particularly with regards to pharmacists, pharm-techs and store staff. Health commodities management functions including dispensing, record keeping and reporting in most health centres and dispensaries was carried out by nurses that also had clinical duties. With pressure on counties to reduce the wage bill it is unlikely there will be big changes in the number of pharmaceutical staff. Another concern raised was dependence on partner employed staff especially in the management of CCCs. Homa Bay Referral hospital lost 5 pharm-techs who were laid off by EGPAF in 2018. They reported that this had adversely affected their operations. In Baringo.14 out of 15 staff working in the Kabarnet County Referral Hospital are contracted by a USAID implementing partner. Staff in Homa Bay also complained about frequent and long delays in the payment of salaries that affects staff morale. Staff turnover (new staff, transfers, separations) was posing a challenge in growing and sustaining SC/CM knowledge and skills. • Commodity Leakages and Pilferage County, Sub-County and facility respondents reported that this was not a significant problem. This was attributed to wide availability of program commodities in GoK and Faith Based Organisations Afya Ugavi Mid-Term Review, May 2019 Page 39 (FBO) facilities and the special marking of KEMSA commodities. Supply Chain and Commodity audits and end user verification exercises had not found significant unaccounted for commodities. Most variances between physical stocks and quantities indicated in bin cards were found to be due to documentation errors. There were two cases of missing malaria commodities at the level of KEMSA – facility delivery interface. The commodity delivery and receiving process underwent some changes to tighten/eliminate the loopholes exploited. • Branding While the AU Activity is clearly doing a commendable job in the counties, their visibility in terms of branding is poor. The only place we found an AU logo is in small print at the bottom of their job aids. Other USAID service delivery partners such as Afya Uzazi and Afya Timiza have been more aggressive in branding and visibility at county level. • Variation in Quality of Implementation by AU While the evaluation team in general found an acceptable standard of implementation by the activity in all the counties visited, there were also discrepancies noted. For example, the understanding and ownership of the commodity management focal facilities and CoL concept was high in some counties such as Isiolo and Kakamega but very low in others including Busia, Baringo and Turkana. We also found that in some counties, AU job aids were displayed in all facilities visited while we could not find single job aid in a few facilities in other counties. For example, we saw very few job aids in health facilities in Kakamega and Homa Bay while the opposite was true in Isiolo and Turkana. • Partner Collaboration The evaluation team found excellent collaboration with USG partners in FP counties where AU worked closely with AfyaTimiza and Afya Uzazi and even shared offices. It was reported that AU and these partners work very closely especially on activities such as data quality review meetings, facility In-charge meetings and supportive supervision. It was also reported that AU had worked very closely with AMREF on the Malaria community program. Partnership collaboration with USG HIV implementing partners in non-HIV counties appeared weak. For example, in Busia, Ampath Plus reported virtually no collaboration with AU and knew very little about AU SC/CM interventions. 4.2.7 Evaluation Findings by Health Focus Area This section gives a brief overview of the AU activity support to the three health focus areas - HIV/AIDS, Malaria and FP/RMNCAH. National Level At the national level, AU has worked with all three national programs responsible for coordination of these three health areas. These institutions are NASCOP for HIV/AIDS, NMCP for Malaria and DFH for FP/RMNCAH. At the national level, AU has provided a set of interventions to strengthen the capacity of these organizations to support the SC/CM function in respective health focus areas across the country. Implementation of AU interventions at the national level for all the three programs was disrupted by suspension of USAID funding for national level MoH institutions. However, the evaluation team found that there was minimal disruption for support to NMCP but significant disruption to support to NASCOP and DFH. The reason for this difference was not clear but it may have been influenced by individual efforts and initiative of staff at USAID and AU. AU also mainly worked with KEMSA primarily on quantification and National Commodity Pipeline Monitoring particularly for Malaria program commodities. The national level interventions in the AU Activity design are largely Afya Ugavi Mid-Term Review, May 2019 Page 40 similar across the three national health programs. These interventions are shown in the Table 14with implementation status by health program: Table 14: AU Interventions Targeting Priority Health Program s AU Intervention NASCOP NMCP DFH Support of strategic plan development No Support. Organization does not have a specific SC/CM plan Supported 2009-2018 Kenya Malaria Strategy the review and development of the new 2019 to 2023strategy Organization does not have a current SC/CM plan. Some support given to review the lapsed Kenya National Reproductive Health Commodity Security Strategy 2013-2017 Quantification Supported in Year 2 of the Activity Comprehensive support provided for a successful Global Fund proposal AU supported a forecast review in Year 1 and 2 National commodity Pipeline Monitoring and development of monthly stock status reports (2-pagers) and quarterly Procurement, Planning and Monitoring Reports (PPMR) AU has worked with KEMSA every month to support preparation of PPMR and National stock status reports. Continuous Support provided Some support provided in Year 1and 2. Not as comprehensive as the support to NMCP Bi-Annual National Pharmacists Meetings Not done Conducted for Malaria Program Not done Quality of Care Surveys Not Applicable Done Not Applicable Import Clearance of Malaria commodities Not Applicable Done Not Applicable For FP/RMNCAH, interventions have focused largely on the FP component. The MNCAH commodities are primarily EMMS, a category that has received little support from USAID Activities. AU’s support for MNCAH component has largely been limited to the development of an integrated commodity tracking tool with few tracer commodities among them Oxytocin, Magnesium Sulphate, Chlorhexidine 7.1% and Zinc/ORS. County Level At the county level, AU has introduced a nearly identical package of interventions in all targeted counties regardless of the health focus area. This makes it difficult and largely unnecessary to disaggregate findings by health focus area. Minor differences include the fact that in Malaria counties, interventions are carried out in all sub-counties, in selects sub-counties for FP/RMNCAH and at county level for HIV/AIDS focus counties. Also, the number of Commodity Management Focus facilities/COLs per county is different based on the health focus – four in Malaria counties, three in FP counties and one in HIV counties. There are also specific interventions for different health focus counties carried out from time to time for example at the request of USAID such as supply chain audit and support for community case management (CCM) for Malaria counties and support for commodity stakeholder forums for FP/RMNCAH counties. With greater integration of interventions, the differences by health focus area are blurring further. Afya Ugavi Mid-Term Review, May 2019 Page 41 4.2.8 AU Activity Financial Status The mid-term evaluation reviewed the broad financial status of the activity to-date. Actual Financial Expenditure as of End of March 2019 At the end of March 2019 - Quarter 2 of Year 3, the activity had an actual expenditure of USD 12,414,451against an obligated amount of USD 14,250,285. The total obligated amounts as of March2019 represented 66% of the total estimated task order ceiling amount of USD 21,640,167. Actual expenditure of USD 12,414,451represents 57% of the total estimated task order ceiling amount. Table 15 : Actual Expenditure by Period Period Expenditure (USD) Year 1 4,473,970 Year 2 4,854,989 Year 3 Q1 and Q2 3,085,492 Total 12,414,451 Table 16: Actual expenditure by Health Focus Area Health Focus Area Actual Expenditure (USD) HIV 3,518,400 Malaria 5,034,447 MCH 602,635 FP/RH 2,494,932 GHSA 764,037 Total 12,414,451 Table 17 : Projected End of Year 3 Expenditure Period Expenditure (USD) Year 1 Actual 4,473,970 Year 2 Actual 4,854,989 Year 3 Budgeted amount 5,797,554 Projected Total at the end of Year 3 15,126,513 Balance from total estimated task order ceiling amount of USD 21,640,167 at the end of Year 3 6,513,654 Source: Tables 15 to 17 - AU Financial Reports. Afya Ugavi Mid-Term Review, May 2019 Page 42 Table 17 shows that at the current level of expenditure and without review of the task order ceiling amount of USD 21,640,167, the activity will have enough money to run only until the end of year 4. This means that at the current level of funding and activity implementation, the Activity is likely to run out of funds by end of year 4 as opposed to the planned year 5. This reality has implications for any planned scale up of interventions or best practices/innovations or expanded geographical coverage. Afya Ugavi Mid-Term Review, May 2019 Page 43 5. CONCLUSIONS ON SC/CM FINDINGS The following conclusions have emerged from our mid-term evaluation findings: • AU has effectively and efficiently discharged its mandate and is on track to achieve program goals and targets. • The suspension of USAID funding of national MoH institutions has significantly hampered the implementation of national level activities and weakened goodwill and coordination between national and county levels. • The AU core package of interventions for SC/CM strengthening at county and national level is appropriate and meets key SC/CM needs and gaps (subject to necessary changes in approach and emphasis based on MTR findings). • At the county level, there has been a decline in health commodities funding in a significant number of counties visited. Further, there is perceptible lack of knowledge, confidence and openness in discussing county commodity budgets and financial flows. This has huge implications on plans to migrate to an era of greater self-reliance. • For greater effectiveness and in response to the changing SC/CM landscape (especially UHC) some design and implementation changes including scaling up existing interventions are required for AU going forward. For details on the design and implementation changes, refer to Sections 7 –“Recommendations” and Section 8 –“Critique of Activity Design and proposed future Direction.” • Some of the proposed changes, such as expanding geographical scope and scaling up best practices and innovations, may require additional funding. For details on the design and implementation changes, refer to Sections 7 –“Recommendations” and Section 8 –“Critique of Activity Design and proposed future Direction.” 6. LESSONS LEARNED AND BEST PRACTICES 6.1. Lessons Learned The AU team as highlighted in the quarterly and annual reports has identified several useful lessons learned. Based on the MTR, the evaluation team wishes to highlight the following lessons. • Integration of Interventions enhances efficiency: An integrated approach in implementing SC/CM interventions is not only more cost effective but also better meets the needs of national and county institutions and clients and promotes sustainability. • Need to Engage National Level Institutions: Even in the devolved context, national institutions remain important for their role in policy formulation and guiding a harmonized approach to SC/CM interventions. National institutions such as national MoH and KEMSA retain a big role in financing, procuring and distributing health commodities. • Importance of Involving beneficiaries in design and implementation of interventions: Engaging beneficiaries in designing and customizing SC/CM interventions contributes to better designs and greater ownership and build their capacity. • Versatility Considering Contextual Changes: It is important for AU to demonstrate versatility in light of contextual changes. A good example of this is the support AU extended to Isiolo county to support their UHC health commodities quantification. Afya Ugavi Mid-Term Review, May 2019 Page 44 • Advocacy for enhanced health commodities financing is crucial: Health commodity budgets especially at the county level have been in decline. This not only affects service delivery but it constitutes a big barrier to greater sustainability. • Strengthening County Leadership and Governance Structures: This is fundamental to ensure strong health stewardship by the county governments and accountability to citizens on all health maters including commodity security. 6.2. Best Practice Given the time available, the evaluation team did not conduct a thorough review to identify the activity’s best practice. However, based on our interviews and discussion with many respondents, we identified two interventions as best practice as shown in Table 18. Table 18: Identified Best Practices Best or Promising Practice Rationale Commodity dashboards Innovative aspects include DHIS2 linkage and disaggregation at facility, sub-county and county level, colour coding to highlight stock status. The evaluation team found that the dashboards are routinely used by county and facility managers for decision making on areas such as order quantities and re-distribution. Supportive Supervision Innovative aspects include the use of a comprehensive integrated checklist, capability to score performance of different inventory management elements, linkage to OJT and mentorship, capability to check if action has been taken on previously identified issues and capability to rank facilities, sub-counties and counties based on performance. Afya Ugavi Mid-Term Review, May 2019 Page 45 7. RECOMMENDATIONS Table19givesthe main recommendations arising from the findings of the MTR. In line with the AU mid-term review scope of work, these recommendations cover the AU Activity scope/mandate and future directions in view of relevant contextual changes: Table 19: List of Recom m endations Arising from the MTR # Finding Recommendation Implementer 1. There are limited AU interventions to advocate for increased & sustained funding to expand the program scope and support migration to a self-reliance model in￾line the USAID Policy Framework – “Ending the Need for Foreign Assistance” 1.1) AU to routinely monitor county commodity budgets and disbursements for EMMS commodities in collaboration with COG. USAID/AU 1.2) AU to scale-up advocacy interventions to increase commodity funding by national and county governments in a sustained and ring-fenced manner. AU 1.3) AU to explore private sector engagement in various components of SC/CM to support migration to a more self-reliant model. USAID and AU 2. Implementation of commodity management focal facilities /COLs has faced challenges of lack of a coherent approach and there are issues with the quality of renovations Propose that AU reduces the number of focal facilities/COLs to one per county but significantly expand the depth and quality of support. The package should be enhanced to include high quality renovations and refurbishment, modular metallic racking, air conditioning and commodity management ICT infrastructure. With the current approach, resources are spread too thin thus reducing impact. In addition, AU should develop a design and budget pack to support roll-out by counties. This should include design approaches and average costs for renovation e.g. cost of racking per square metre, cost of AC equipment, fire-fighting equipment, pallets etc. The pack should also provide information on broad design approaches, principles and standards such as flooring materials, racking options, ventilation, cold-rooms and refrigeration design etc. This information is particularly important in supporting the journey to self-reliance. AU should explore partnerships with USG partners on ways of enhancing focal facilities and COLs. AU 3. There is limited automation of the SC/CM function AU should play a greater role in supporting the design and possible piloting of a comprehensive ICT platform to manage the SC/CM function at the county level, with national linkage. Piloting can start with the focal facilities/COLs. The platform adopted should be linked and interfaced with service delivery EMR’s (or be a module of a comprehensive EMR). USAID and AU Afya Ugavi Mid-Term Review, May 2019 Page 46 # Finding Recommendation Implementer 4. AU has not been active with Inter-governmental and national level organizations such as COG, NASCOP, DFH Directorate of pharmaceutical services and Pharmacy and Poisons board AU to review and expand its engagement with national level institutions and conduct focused policy advocacy and dialogue, leadership and governance strengthening to foster sustainable commodity security. This needs to be done in close collaboration with other USG partners (HP plus, Tupime etc). USAID and AU 5. Some inter-county discrepancies in the quality of AU interventions were noted AU to strengthen internal institutional learning to leverage and scale-up uptake of emerging good practice within the entire organization. Options include “brown bag” meetings. AU 6. AU has a low visibility at national, county and facility level Develop and roll-out an engagement and communication strategy to enhance AU visibility and branding. Print and distribute success stories, operational research findings, case studies etc. AU 7. The roll-out of UHC poses significant SC/CM challenges and opportunities 7.1) Re-align AU activities with UHC. USAID and AU 7.2) Expand support for UHC SC/CM components at the national and county level and advocate for the necessary policy and regulatory reforms. USAID and AU 8. The national SC/CM strategic plan is yet to be developed Support development of the National SC/CM strategic plan to fill clearly identified SC/CM gaps. Clearly identify its owner/driver and build their capacity as needed. AU 9. There is a lingering problem of disposal of expired products leading to products that expired years ago taking up scarce store space AU needs to support counties address the issue of safe disposal of expired products to protect consumers and the environment and free much needed storage space currently taken up by a legacy of stockpiled expired products. USAID/AU 10. Counties and facilities have no visibility of the cost of program commodities supplied to them. Lack of awareness of what program commodities cost creates a moral hazard risk and constitutes a big barrier to preparing for transitioning to an era of greater self￾reliance. 10.1 Afya Ugavi should support a comprehensive assessment to establish the value of program commodities supplied to counties and other recipients. This should be compared to county health commodity budgets to develop “stress test” scores for county migration to greater self-reliance in health commodities. USAID and AU 10.2 KEMSA should start providing health facilities with receipts and statements showing the value of program commodities to provide visibility for program costs. USAID/KEMSA 11 Commodity Dashboards and supportive supervision were identified by respondents as best practices that were transformative to SC/CM at the county level Work started to scale-up Malaria Commodities dashboard to the national level, should be expanded to cover all AU developed commodity dashboards and the supportive Supervision checklist. AU should scale-up national adoption of these tools and interventions in partnership with other USG partners and the National Directorate USAID and AU Afya Ugavi Mid-Term Review, May 2019 Page 47 # Finding Recommendation Implementer of Pharmaceutical services. The dashboards are DHIS2 linked making national roll-out relatively easy. AU should develop a proposal on such scale￾up including financial implications. 12 There are significant sustainability challenges for AU interventions In collaboration with other USG partners AU needs to develop a national and county transition roadmap to support migration towards a more self-reliant model especially for program commodities. USAID and AU 13 AU has developed a concept note that proposes the establishment of county Health Products & Technologies Units (HPTU) that will organize, monitor, and support all activities within the supply chain system. AU should support the establishment of HPTUs initially in the counties they support and later advocate for national roll-out. These units should fully integrate management of all commodities including program and EMMS including lab and nutritional commodities. AU/USAID 14 At the current level of expenditure and with the currenttask order ceiling amount of USD 21,640,167, the activity will have enough money to run only until the end of year 4. For the activity to run for the 5 years as per the task order, USAID needs to increase the task order ceiling amountfrom the current USD 21,640,167. USAID Afya Ugavi Mid-Term Review, May 2019 Page 48 8. CRITIQUE OF ACTIVITY DESIGN AND PROPOSED FUTURE DIRECTION Our recommendations on future direction of AU over the remaining period are based on the design shortcomings identified. A highlight the main issues with the current design of the AU Activity include: i. Selection of Target Counties The eight Malaria counties selected for support by AU collectively represent a significant proportion of the national Malaria disease burden and therefore Malaria commodities consumption and demand. However, the same is not true for HIV and is even less so for FP which is the main component of the FP/RMNCAH health focus area. It can be argued that FP uptake in the selected counties is so low that FP commodities availability is unlikely to be “rate limiting” in efforts to expand uptake of FP services. The problem for FP is further aggravated by selection of sub-counties where FP uptake is lower than county average. In these sub-counties behaviour change communication and other demand creation activities are the main need. Figure7 for Turkana County illustrates this issue. Figure 7: Percent of Total Turkana County FP Clients (Long -Term Methods) Source: Afya Timiza Figure 8 : 2018 Vs 2017 FP Client Growth - Turkana County Source: Afya Timiza 0 20 40 60 80 2017 2018 28 25 72 75 AU Sub-Counties non-AU Sub-Counties -40 -20 - 20 40 60 80 100 Long-Term Methods Short Term Methods 45 -1.1 72 -18 AU Sub-Counties Non-AU Sub -Counties Afya Ugavi Mid-Term Review, May 2019 Page 49 Figure 8shows that selected AU (and Afya Timiza) sub-counties account for a small part of the total FP clients and they are growing more slowly than the non-AU sub-counties. ii. Geographic coverage Within Selected Counties The three health focus areas have different approaches on geographic coverage within selected counties. The HIV focus area largely targets the county HQ for most of its interventions withminimal direct interventions at the sub-county level. FP/RMNCAH on the other hand works in a few select sub-counties. Malaria has the widest geographic coverage as it targets both the county HQ and all the sub-counties within the selected counties. The rationale for the narrower coverage by HIV and FP/RMNCAH health focus areas relative to the Malaria approach is not clear. This is especially so given the relatively small differences in funding streams by health focus area. iii. Interventions Package The SC/CM interventions package is very similar across the three health focus areas. Malaria however has a few additional interventions including quality of care surveys, biannual pharmacists’ meetings and support for community components. Overall, we find that selected SC/CM strengthening interventions across the three health focus areas are consistent and rational. However, one county pharmacist challenged AU to go beyond support for basic technical SC/CM components such inventory management to supporting the full spectrum of SC/CM framework including having a broader system strengthening approach. iv. Integration of Interventions In the first year, delivery of SC/CM interventions was largely vertical. The interventions in FP/RMNCAH counties focused primarily on FP commodities. The same approach applied to Malaria and HIV focus counties. This approach exacerbated some of the concerns highlighted in sections 1 and 2. The narrow health focus and failure to incorporate interventions targeting EMMS commodities reduced the value health facilities and counties derived from the AU Activity. For example, without targeting EMMS one cannot address MNCAH commodities issues. From the second year both the counties and USAID have pushed for greater integration of SC/CM interventions across health focus areas and inclusion of EMMS. v. Learning& Knowledge Management AU needs to strengthen both internal and external learning and knowledge/best practice sharing to quickly and efficiently scale up promising practices. Although operational research is mentioned as an important strategy in the implementation of AU Activity, we only came across one case in Busia County and the local teams were unsure of the next steps in the ongoing study. vi. Technical focus In malaria supported counties we received feedback that the malarial program is generally focused on one testing methodology (RDT) and one treatment commodity (AL). Other important diagnostic methods (such as microscopy for species diagnosis and confirmations) and treatment modalities (2nd line treatment and EMMS needed for supportive treatment) are generally not supported by NMCP and AU. Table 20 captures the broad AU design elements: Table 20: AU Activity Design Matrix Afya Ugavi Mid-Term Review, May 2019 Page 50 Design Attribute Health Focus Area HIV/AIDS Malaria FP/RMNCAH Operational Level County County and Sub-County Sub-County Geographic Reach within the County Mainly County HQ All Sub￾Counties Some Sub￾Counties Breadth of Intervention Package ++ +++ ++ Patient/Client reach in targeted Counties and Sub￾Counties ++ +++ + Based on the analysis of the program design, we are making the following recommendations in table 21 regarding the broad program approach and design: Table 21: Recom m endations on AU Activity Design Changes # Finding Recommendation Implementer 1 Geographic targeting for FP and HIV/AIDS interventions is narrow and limits the impact of these programs. For example, FP targets nine sub-counties while Malaria targets 63 sub-counties. Broaden geographical footprint of FP/RMNCAH and HIV/AIDS interventions to cover all sub￾counties in the targeted counties (Essentially adopt the Malaria model for all the health focus areas). A health focus lens should only be used for selecting target counties. Once selected, all counties should adopt similar implementation approaches and interventions covering all three health focus areas and also cover EMMS. USAID and AU 2 Integration of interventions has only recently started Fully integrate interventions to include HIV/AIDS, Malaria, FP/RMNCAH and EMMS in all the 14 target counties and strengthen collaboration with service delivery partners. This will significantly leverage available resources and magnify impact. USAID and AU 3 Response to and alignment with significant contextual changes AU Activity’s scope/mandate, coverage and implementation approach should be aligned with UHC and USAID’s ‘road to self-reliance’. Afya Ugavi Mid-Term Review, May 2019 Page 51 9. ANNEXES Annex 1: Evaluation Matrix Evaluation Question Evaluation Sub￾Questions Consultants Tasks for the Evaluation EQ1: How effectively and efficiently has Chemonics implemented the Afya Ugavi activity? EQ1.1: Is the Afya Ugavi activity on track to achieve its stated objectives? Compare baseline with mid-term progress and assess whether the objectives and outcomes are on track to being achieved. Assess each program intervention disease area and document progress against the Annual Work Plans (AWPs) and the MEL plans for 2016/17 and 2017/18. Assess achievements in other cross intervention areas such as data review forums, facility in charges meetings, and supportive supervision among others, and how these multi-sectoral approaches have led to a holistic improvement in supply chain and the health service delivery continuum. EQ1.2: What was the quality of the work conducted by Chemonics? Assess tools, manuals, guidelines or approaches that the program developed or used, their suitability towards health supply chain development and their suitability in the current devolved environment * EQ1.3: How efficiently have resources been utilized in the Afya Ugavi implementation? Assess the Activity’s implementation in relation to financial expenditure, existing gaps and measures towards financial security. Suggest realistic modifications necessary for the remaining duration of the project based on perceived shortcomings and gaps in project financial planning, implementation and project management structure** EQ1.4: How effective has the project’s measurement approach (framework, indicators, data collections, etc.) been to monitoring project implementation? Assess how the data or information emerging from, continuous assessments, health commodity system audits operational research and any other special studies carried out by the project has been impactful in implementation. Review the current Monitoring & Evaluation Plan, and specifically the Activity indicators to determine their relevance and alignment to the scope and depth and provide suggestions for improvement. Review monitoring tools and data collection methods to draw conclusions about their effectiveness in measuring the project’s Afya Ugavi Mid-Term Review, May 2019 Page 52 Evaluation Question Evaluation Sub￾Questions Consultants Tasks for the Evaluation performance and make suggestions for improvement as needed. EQ1.5: What are the most effective sustainability strategies that Afya Ugavi has employed and is the GOK likely to sustain these advances after the program is finished? Comment on the sustainability and potential replicability of interventions, especially with in terms of integration into the health delivery system at facility level. Other Assess and comment on unplanned results, unexpected constraints and new interventions not included in the AWPs. EQ2: How effective has the design of Afya Ugavi been in addressing the supply chain development goals of the Government of Kenya and of USAID/Kenya? EQ2.1: How have GOK counterparts benefitted from this activity at the national level? At the county level? What about other implementing partners? EQ2.2: How has the initial design been affected by evolving contexts and changing beneficiary needs? And was it able to respond and change to these influences, if necessary? Assess any modifications of interventions so far and comment on whether these modifications are justified and record their outcomes. EQ2.3: Is the design of Afya Ugavi activity an appropriate allocation of resources and responsibilities in the USG funding environment for the Kenyan public health supply chain? Afya Ugavi Mid-Term Review, May 2019 Page 53 Evaluation Question Evaluation Sub￾Questions Consultants Tasks for the Evaluation EQ2.4: Are there more effective supply chain technical assistance approaches that should be considered in the Kenyan context? Other(These tasks arise more from data analysis while others call for recommendations based on evaluation findings) Document innovative interventions and strategies being used by the Activity and provide, as a highlight, an innovative idea, promising practice, or good practice that is being advanced by this program, providing as much evidence-base as possible at the time of the evaluation. Document good practices and lessons learned in the implementation of the project; and Identify issues, risks and challenges within the project, which can be obstacles in achieving project results, and provide recommendations to overcome these. Suggest realistic modifications necessary for the remaining duration of the project based on perceived shortcomings and gaps in project financial planning, implementation and project management structure. Present specific recommendations for USAID, the Ministry of Health, County Health Management Teams and collaborating partners regarding the remaining period of the project that will ensure that all objectives are met; Provide recommendations for potential extension or future phase of the project beyond its current funding period; and Provide recommendations to USAID on how Afya Ugavi and other implementing partners’ interventions can be realigned alongside the mission’s supply chain priority objectives. Afya Ugavi Mid-Term Review, May 2019 Page 54 Annex 2: List of Literature Review Documents Category/Level Documents Evaluation question predominantly addressed National – Policies and Strategic Plans, Service delivery and SC/CM Reports NHSSIP III. NHA. Pharmaceutical Policy. UHC Roadmap. Malaria Strategy. MNCAH (including FP) Strategies. KNASP. Public Health Programs Surveys (KAIS etc). KEMSA Strategic Plan and Performance Reports. EQ 2 USAID KEA USAID Health Strategic Framework and SC/CM Strategic Framework. USAID SC/CM survey and Audit reports. Other Relevant USG programs SC/CM reports. Previous USAID SC/CM Project Evaluation reports. EQ 2 County – Polices and Strategic Plans, Service delivery and SC/CM reports County Health Strategic Plans. County SC/CM reports. County Public Health program plans/policies & reports (Malaria, HIV, MNCAH/FP as relevant) EQ 1 & 2 Facility SC/CM Tools and routine reports. Service delivery reports (General and public health programs) EQ 1 AU Project Project TOR and deliverables as per AU contract. List and roles of all implementing subcontractors or partners Project PMP and or M/E framework or log frame. Project quarterly and annual performance reports to date. AU Project budget allocation, utilization and financial management reports. Any USAID feedback on project reports. County engagement/workshop reports CHMT - AU meetings minutes. Any beneficiary feedback reports. EQ 1 Others Global SC/CM benchmark reports and frameworks. Other non-USG donor projects related to SC/CM EQ 2 Afya Ugavi Mid-Term Review, May 2019 Page 55 Afya Ugavi Mid-Term Review, May 2019 Page 56 Annex 3: Key Informants Interview Questionnaire - National Respondents Key Informants Questionnaire - National MoH Respondents (NMCP, NASCOP, DFH) Name of Respondent:___________________________________________________ Organization/Location: __________________________________________________ Interview Date: ___________________________ Guidance for Interviewers • Consider relevance and applicability of all questions to specific key informants. Skip questions not relevant to respondent being interviewed • You may add probing and clarifying questions where appropriate during the actual interview. Section A: Evaluation Question 1 1. In which areas have you interacted with Afya Ugavi activity? 2. To what extent has Afya Ugavi activity focused on your priority supply chain and commodity management areas and what are your views on their efficiency and responsiveness? 3. In which areas has Afya Ugavi helped you improve your supply chain and commodity management system s and practices? Probe on the following if not mentioned: CM Area Specific Interventions I. Commodity Management system Forecasting, supply planning, ordering, inventory management, reporting, centres of Learning II. Leadership and Governance County commodity plan, CSTWGs, III. Safety of health products Pharmacovigilance, post marketing surveillance IV. HRH AC/SM staffing, capacity building, gender, SOPs, mentorship, OJT V. Information Systems Data capture tools, transmission process, data quality, data analytics, Data for Decision Making, Dashboards VI. Commodity Financing Planning and budgeting, advocacy for County funding, leakages, expiries 4. What has been the result or impact of above interventions on your supply chain and commodity management systems and practices 5. What has been the overall impact of AU’s assistance on the broader national/ county’s health system s including service delivery? 6. What is the impact of AU support in the CSTWG’s effectiveness? 7. What specific challenges does the CSTWG face in carrying out its mandate and what additional support can AU provide? 8. What national, regional or county coordination forums on SC/CM were carried out in the last two years? What support did AU provide and how helpful have the forums been? Afya Ugavi Mid-Term Review, May 2019 Page 57 9. What supply chain and commodity management tools has Afya Ugavi helped you develop or made available for your program? (This should include IT Systems - Probe on dispensing tools). 10. How have these tools helped you better manage your commodities? What have been the shortcomings? What are the gaps? 11. Do you have adequate, skilled staff to effectively manage the SC/CM function? Provide data on SC/CM staffing levels disaggregated by cadre, gender -National, County and sub-county, facility) Cadre Staffing Norms Numbers available Male Female Total Pharmacists Pharm Tech Nursing officers* Laboratory technologist/technicians* Procurement officers Nutritionists* Others (specify) *Those involved in SC/CM 12. Share with us the SC/CM capacity building activities supported by AU and partners in the last two years? Probe on TOT, OJT, Mentorship and others. Obtain data on number of staff trained/mentored 13. What has been the impact of the training on the attitude and performance of staff? 14. What knowledge and skills gaps remain unaddressed in respect to SC/CM in your organization and at county and facility level? 15. How have you followed-up on the training done to determine its effectiveness? Share your training evaluation reports 16. Share your experience where relevant on the Centres of Learning (Commodity Management focal Facilities) supported by AU - successes and challenges 17. Share with us the systems you are using for SC/CM information management at national, county and facility level 18. What support has AU and other partners provided for strengthening/improving SC/CM information systems 19. What are the successes, gaps and challenges in using these systems? Probe on end to end visibility, electronic data collection and transmission, inter-operability and linkages of IT systems supporting SC/CM, fragmentation of systems, integration with DHIS2. 20. Share your latest national commodity reporting rates and trends 21. Share with us your stock status reports for the last two years focusing on stock-outs, under￾stocking, overstocking and expiries Afya Ugavi Mid-Term Review, May 2019 Page 58 22. Share with us the support received from AU and partners to improve the quality and safety of health commodities – Probe - pharmacovigilance, post market surveillance, Medical and therapeutic committees and impact of interventions. 23. Have you experienced cases of poor quality/unsafe commodities? Provide details and share action taken and any reports. 24. Have you experienced cases of leakage/pilferage of commodities? Provide details and share remedial action taken and any reports 25. Share your multi-year commodity budget showing funding sources, utilization and funding gaps? What is being done to address funding constraints and imbalances? How has AU supported your resource mobilization efforts and with what results? 26. Do you have a commodity management strategic plan? How have you used it to improve SC/CM? Share a copy Section B: Evaluation Question 2 27. What aspects of the AU program design elements have been most helpful in strengthening SC/CM? Give examples and reasons 28. What aspects of the AU program design elements would you want to be changed to better meet your SC/CM needs/priorities? Give reasons 29. What new interventions or implementation approaches should AU adopt in the next 3 to 5 years to better strengthen county and national level SC/CM? 30. Are there any current intervention/ Approaches that AU should discontinue? Why? 31. What opportunities are there for better integration of AU interventions across health focus areas – Malaria, HIV and FP/RMNCAH at the national, county and facility level? 32. What contextual changes at national and county levels have affected the performance of AU projects either positively or negatively? Probe policy and regulatory change, structural changes, staff movements, governance and accountability challenges and funding? 33. To what extent is the county and national government able to take over (manage and finance) the technical assistance provided by AU and other partners in SC/CM? 34. What elements of the AU supported interventions would you rate as best practice/ innovative? Why? 35. Which intervention/s currently supported by AU has the greatest potential for national/county replicability and scale up? Section C: Broader and Recommendations Related Questions 36. On a scale of 1 to 5 (I being below average, 2 average, 3 good, 4 very good, and 5 excellent) rate the performance of AU in the following areas: Parameter Rating 1 2 3 4 5 Relevance of interventions Effectiveness of intervention Quality of interventions Efficient use of resources Knowledgeable staff Collaborative staff Overall impact Sustainability of interventions Afya Ugavi Mid-Term Review, May 2019 Page 59 37. Any other suggestion, comments, concerns or questions that may help strengthen the AU Activity in the coming 3 to 5 years? Thank you for Participating in this Evaluation Afya Ugavi Mid-Term Review, May 2019 Page 60 Annex 4: Key Informants Interview Questionnaire - County and Sub-County Respondents Key Informants Questionnaire - County and sub-County Respondents Name of Respondent:___________________________________________________ Organization/Location: __________________________________________________ Interview Date: ___________________________ Guidance for Interviewers • Consider relevance and applicability of all questions to specific key informants. Skip questions not relevant to respondent being interviewed • You may add probing and clarifying questions where appropriate during the actual interview. Section A: Evaluation Question 1 1. In which areas have you interacted with Afya Ugavi activity? 2. To what extent has Afya Ugavi activity focused on your priority supply chain and commodity management areas and what are your views on their efficiency and responsiveness? 3. In which areas has Afya Ugavi helped you improve your supply chain and commodity management system s and practices? Probe on the following if not mentioned: CM Area Specific Interventions I. Commodity Management system Forecasting, supply planning, ordering, inventory management, reporting, COL II. Leadership and Governance County commodity plan, CSTWGs, III. Safety of health products Pharmacovigilance, post marketing surveillance IV. HRH AC/SM staffing, capacity building, gender, SOPs, mentorship, OJT V. Information Systems Data capture tools, transmission process, data quality, data analytics, Data for Decision Making, Dashboards VI. Commodity Financing Planning and budgeting, advocacy for County funding, leakages, expiries 4. What has been the result or impact of above interventions on your supply chain and commodity management systems and practices? 5. What has been the overall impact of AU’s assistance on the broader county’s health system s including service delivery? (Probe for specific impact in the AU focus health area) 6. Do you have a CSTWG? If yes what support has AU provided? Share the TORs and copy of minutes of last meeting. Share its impact and challenges. 7. What national, regional or county coordination forums on SC/CM were carried out in the last two years? What support did AU provide and how helpful have the forums been? 8. What supply chain and commodity management tools has Afya Ugavi developed or made available for your county? (This should include IT Systems, dispensing tools). Share the benefits accrued from these tools and shortcomings Afya Ugavi Mid-Term Review, May 2019 Page 61 9. Are you applying these tools only in the health area supported by Afya Ugavi or also in other areas? Cite some examples 10. What gaps still exist in respect to supply chain and commodity management tools? 11. Do you have adequate, skilled staff to effectively manage the SC/CM function? Provide data on SC/CM staffing levels disaggregated by cadre, gender -National, County and sub-county, facility) Cadre Staffing Norms Numbers available Male Female Total Pharmacists Pharm Tech Nursing officers* Laboratory technologist/technicians* Procurement officers Nutritionists* Others (specify) *Those involved in SC/CM 12. Share with us the SC/CM capacity building activities supported by AU and partners in the last two years. Probe on TOT, OJT, Mentorship and others, number of staff trained 13. What has been the impact of the training on attitude and performance of staff? 14. What knowledge and skills gaps remain unaddressed in respect to SC/CM? 15. How have you followed on the training done to determine its effectiveness? Share your training evaluation reports 16. Share your experience with the Centres of Learning where applicable - successes and challenges? 17. What suggestions do you have on improving the effectiveness of the Centres of Learning concept (Commodity Management Focal Facilities)? 18. Share with us the systems you are using for SC/CM information 19. What support has AU and other partners provided for strengthening/improving SC/CM information systems 20. What are the successes, gaps and challenges in using these systems? Probe on end to end visibility, electronic data collection and transmission, inter-operability and linkages of IT systems supporting SC/CM, fragmentation of systems, integration with DHIS2. 21. What support have you received from AU and partners to strengthen data quality - completeness, accuracy and timeliness? 22. Share your latest commodity reporting rates and trends for AU focus commodities 23. Share with us your stock status reports for the last two years focusing on stock-outs, under￾stocking, overstocking and expiries? - (Note: Complete the checklist on tracer commodities availability on the day of the visit) Afya Ugavi Mid-Term Review, May 2019 Page 62 24. Share with us the support received from AU and partners to improve the quality and safety of health commodities – Probe - pharmacovigilance, post market surveillance, Medical and therapeutic committees and impact 25. Have you experienced cases of poor quality/unsafe commodities? Provide details and share action taken and any reports. 26. Have you experienced cases of leakage/pilferage of commodities? Provide details and share remedial action taken and any reports 27. Share your multi-year commodity budget and utilization showing funding sources and funding gaps in total and for priority health programs. What challenges has the county health department in mobilizing adequate resources. What is being done to address funding constraints and imbalances? How has AU supported your resource mobilization efforts and with what results? 28. Is commodity management financing and planning integrated in your county health strategic plans and workplans. Please share these documents with us. If yes, how has this helped? Section B: Evaluation Question 2 29. What aspects of the AU program design elements have been most helpful in strengthening SC/CM? Give examples and reasons 30. What aspects of the AU program design elements would you want to be changed to better meet your SC/CM needs/priorities? Give reasons 31. What new interventions or implementation approaches should AU adopt in the next 3 to 5 years to better strengthen county and facility level SC/CM? 32. Are there any current intervention/ Approaches that AU should discontinue and why? 33. What opportunities are there for better integration of AU across health focus areas – Malaria, HIV and FP/RMNCAH at the county and facility level? 34. What changes at national and county levels have affected the performance of the AU project either positively or negatively? Probe policy and regulatory change, structural changes, staff movements, governance and accountability challenges and funding? 35. To what extent is the county able to take over (manage and finance) the technical assistance provided by AU in SC/CM? 36. What elements of the AU supported interventions would you rate as best practice/ innovative? Have this been shared with other stake holders? Section C: Broader and Recommendations Related Questions 37. On a scale of 1 to 5 (I being below average, 2 average, 3 good, 4 very good, and 5 excellent) rate the performance of AU in the following areas: Parameter Rating 1 2 3 4 5 Relevance of interventions Effectiveness of intervention Quality of interventions Efficient use of resources Knowledgeable staff Collaborative staff Overall impact Sustainability of interventions 38. Any other suggestion, comments, concerns or questions that may help strengthen the AU Activity in the coming 3 to 5 years? Afya Ugavi Mid-Term Review, May 2019 Page 63 Annex 5: County and Facility Checklist Note: The data in this table will be collected at either county or facility level. Where county hold data for multiple facilities this data will be collected at county or even sub-county level Level Data Category Specific Data to collect County Health commodity Financing Total health expenditure – budget and actual Health commodity expenditure - budgets and actual Health Strategic and other plans Copy of County Health Investment and Strategic Plan Copy of County Workplan Facility Staffing Numbers of staff involved in commodity management and trends Stock status Stock status for tracer commodities - Refer to table in Question 11 of County KII Availability of tracer commodities for HIV, Malaria and FP/RMNCAH on day of visit - Refer to annex 12 Commodity Reporting rates Reporting rates and trends for HIV, Malaria and FP/RMNCAH commodities Capacity building and supervision Number of staff trained on commodity management No of staff mentored on SC/CM Number of SC/CM supervisory visits Afya Ugavi Mid-Term Review, May 2019 Page 64 Annex 6: AU Activity Performance Checklist Output Project performance indicators Targets Attainment 2016/2017 Attainment 2017/2018 Source of Data Cross cutting indicators 1.2: Improved supply chain logistics and commodity management Service Delivery Point (SDP) reporting rate DHIS2 Stock out rate at service delivery points (SDPs) Percentage of stock status observations in SDPs, where commodities are stocked according to plan 2.1 Increased human and institutional capacity for commodity management Number of people trained by project activities or events. 2.2 Improved commodity security environment % of county health budget requirements for procurement of essential medicines and commodities covered by counties in their budgets Sustainable Indicators 2.2 Improved commodity security environment % of total county budget for health allocated to supply chain system strengthening (excluding procurement budget) 2.1: Increased human and institutional capacity for commodity management # Initially Afya Ugavi supported supply chain functions carried out by national and county authorities without external technical assistance. 1.2 Improved supply chain logistics and commodity management 2.1 Increased human and institutional capacity for Supply chain maturity level demonstrated through the Centres of Learning Maturity model Afya Ugavi Mid -Term Review, May 2019 Page 65 commodity management 2.2 Improved commodity security environment 2.1 Increased human and institutional capacity for commodity management 2.2 Improved commodity security environment Number of supply chain innovations developed and in use Malaria STANDARD INDICATOR Number of artemisinin - based combination therapy (ACT) treatments purchased with USG funds Number of RDTs purchased with USG funds Number of injectable Artesunate treatments purchased with USG funds FP/RMNCAH Program Indicator 1.2 Improved supply chain logistics and commodity management 2.2 Improved commodity security environment Average stock out rate of contraceptive commodities at Family Planning service delivery points • Oral contraceptive • Implant • Male condom • DMPA • IUCD Afya Ugavi Mid-Term Review, May 2019 Page 66 1.2 Improved supply chain logistics and commodity management 2.2 Improved commodity security environment Indicator Name: Average stock-out rate of contraceptive commodities at Family Planning service delivery points Afya Ugavi Mid-Term Review, May 2019 Page 67 Annex 7: List of National Level Informants 62 Annex 8: List of County Level Informants County: Isiolo 63 County: Homa Bay 64 County: Kakamega 65 County: Busia County: Baringo County: Turkana