1 IBTCI: Mid-Term Performance Review of AfyaInfo The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. D JANUARY 5, 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by the International Business & Technical Consultants, Inc. (IBTCI) Evaluation Services and Program Support (ESPS) in Kenya. EVALUATION MID-TERM PERFORMANCE REVIEW OF AFYAINFO FINAL REPORT USAID KENYA ESPS ii IBTCI: Mid-Term Performance Review of AfyaInfo MID-TERM PERFORMANCE REVIEW OF AFYAINFO This publication was produced for the United States Agency for International Development. It was prepared under the Indefinite Delivery/Indefinite Quantity (IDIQ) contract to provide monitoring and evaluation services to USAID Kenya’s Office of Population and Health (OPH) and the Regional Health and HIV/AIDS Office (RHH), Task Order Number 4 by International Business & Technical Consultants, Inc. (IBTCI) Evaluation Services and Program Support (ESPS) in Kenya. DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii IBTCI: Mid-Term Performance Review of AfyaInfo TABLE OF CONTENTS EXECUTIVE SUMMARY............................................................................................................................... 1 1. INTRODUCTION .................................................................................................................................. 6 2. BACKGROUND ..................................................................................................................................... 7 3. METHODOLOGY .................................................................................................................................. 8 4. FINDINGS AND CONCLUSIONS....................................................................................................... 9 5. FUTURE DIRECTIONS AND PRIORITIES.......................................................................................21 ANNEX 1: AFYAINFO CONSORTIUM, STAKEHOLDERS AND PARTNERS.....................................23 ANNEX 2: SCOPE OF WORK....................................................................................................................29 ANNEX 3: CONSULTANT CVS ...............................................................................................................51 ANNEX 4: SIGNED ATTESTATIONS OF NON-CONFLICT OF INTEREST ......................................57 ANNEX 5. DETAILED METHODOLOGY................................................................................................. 59 ANNEX 6: BIBLIOGRPAPHY.....................................................................................................................70 ANNEX 7. DATA COLLECTION TOOLS ................................................................................................72 ANNEX 8: SCHEDULE, ORGANIZATIONS AND NUMBER OF RGD AND FGD PARTICIPANTS . 79 ANNEX 9: SCHEDULE, ORGANIZATIONS AND NUMBER OF SGD PARTICIPANTS....................80 ANNEX 10: SCHEDULE, LIST AND NUMBER OF KEY INFORMANT INTERVIEWS .......................81 ANNEX 11: YEAR 3 AFYAINFO PMP TRACKING..................................................................................82 iv IBTCI: Mid-Term Performance Review of AfyaInfo ACRONYMS AI Abt International APHIAPlus AIDS, Population and Health Integrated Assistance Service Delivery Partners API Application Program Interface APR Annual Performance Reporting AWP Annual Work Plan CDC Centers for Disease Control and Prevention CDOH County Department of Health CEC County Executive Committee CHEO County Health Executive Officer CHIS Community Health Information System CHMT County Health Management Team CHSU Community Health Services Unit COBPAR Community-Based Program Activity Report COR Contracting Officer’s Representative CSO Civil Society Organization CU Community Unit DANIDA Danish International Development Agency DDIU Data Demand and Information Use DHIS2 District Health Information Software Version 2 DivHIME Division of Health Information, Monitoring and Evaluation DOD Department of Defense DP Development Partners DQA Data Quality Assurance DQI Data Quality Improvement EMR Electronic Medical Record ERP Enterprise Resource Planning eSCM Electronic Supply Chain Management ESPS Evaluation Services and Program Support FGD Focus Group Discussion GOK Government of Kenya HCSM Health Commodities and Services Management HIS Health Information System HIV Human Immunodeficiency Virus HMIS Health Management Information System HRIO Health Records Information Officer HSSF Health Sector Service Fund IBTCI International Business & Technical Consultants, Inc. ICT Information and Communication Technology iHRIS Integrated Human Resources Information System IP Implementing Partners IT Information Technology K2D KePMS to DHIS2 v IBTCI: Mid-Term Performance Review of AfyaInfo KEMSA Kenya Medical Supplies Agency KePMS Kenya Program Monitoring System KHWIS Kenya Health Workforce Information System KII Key Informant Interview KL Knowing Limited KMTC Kenya Medical Training College KNBS Kenya National Bureau of Statistics KU Kenyatta University LKM Learning and Knowledge Management LMIS Logistics Management Information System M&E Monitoring and Evaluation MCUL Master Community Unit List MFL Master Facility List MIAS Malaria Information and Acquisition System MOH Ministry of Health MTR Mid-Term Review MTRT Mid-Term Review Team NACC National AIDS Control Council NASCOP National AIDS/STD Control Program NGO Non-Governmental Organization NHIS National Health Information System NIMES National Integrated Monitoring and Evaluation System OGAC U.S. Office of Global AIDS Coordination OPH Office of Population and Health (USAID) PEPFAR The U.S. President’s Emergency Plan for AIDS Relief PMP Performance Monitoring Plan POCC Project Oversight Coordination Committee RGD Roundtable Group Discussion rHRIS Regulatory Human Resources Information System SAPR Semi-Annual Program Reporting SCHRIO Sub-County Health Records Information Officer SDOH State Department of Health SDP Service Delivery Partner SGD Small Group Discussion SOW Scope of Work TNA Training Needs Assessment TRG Training Resources Group TWG Technical Working Group UON University of Nairobi USAID United States Agency for International Development USAID/K United States Agency for International Development/Kenya USG United States Government WHO World Health Organization 1 IBTCI: Mid-Term Performance Review of AfyaInfo EXECUTIVE SUMMARY Kenya’s national health information system (NHIS) has been in existence since the early 1970s. A comprehensive assessment of the NHIS and National Monitoring and Evaluation System supported by the United States Agency for International Development/Kenya (USAID/K) in August 2010 found that current practices were driven more by short-term program reporting requirements than by national coherence or sustainability. Inadequate governance or coordinating structures were also identified as gaps. In response to the assessment findings, USAID/K contracted with Abt Associates International (AI) and its consortium1 in May 2011to implement AfyaInfo, a five-year national health systems strengthening project. AfyaInfo primarily addresses health information systems with the goal of gradually improving the coherence and effectiveness of a national data system for tracking health problems, activities, resources and outcomes by supporting the Government of Kenya (GOK) to design and build a single unified web-based health information system. The activity is aligned with the USAID Implementation Framework 2010–2015’s goal of “sustained improvement of health and well-being for all Kenyans.” During August and September 2014, an independent review team commissioned by USAID/K carried out a mid-term review (MTR) of AfyaInfo. The purpose of the MTR was to assess progress made in implementing the contract and to determine the factors that have facilitated or hindered the implementation of activities planned in the first three years (June 2011–May 2014). The review further explored strategies for sustaining key results at the national, county, sub-county and health facility levels. The evaluation questions to be answered by the MTR are: 1. What progress has the activity made in addressing the key challenges and gaps of NHIS (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009–2014)? 2. What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. 3. What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support the Ministry of Health (MOH) in sustaining the use of NHIS? 4. Based on the key findings and conclusions drawn from questions 1–3, what are the key recommendations on the strategic, programmatic and management directions that the Mission should consider for mid-course changes to the contract? The primary audiences for the review are US Government (USG) agencies (Department of Defense (DOD), the Centers for Disease Control and Prevention (CDC), USAID/K) and the Government of Kenya. Secondarily, the report is intended for AI, USAID/Washington and other interested implementing partners. It is expected that the results of the MTR will inform decisions on: 1. The number of counties that the project should cover with IT infrastructure; 1 Centers for Health Solutions, ICF International, Kenya Medical Training College, Knowing Limited, Training Resources Group and the University of Nairobi. 2 IBTCI: Mid-Term Performance Review of AfyaInfo 2. The number of counties to be covered with technical assistance on information system organizational capacity development; 3. The appropriate approaches to implementing activities under this contract; 4. The appropriate sustainability strategies that would ensure long-term use of the integrated national information system. These programmatic and management decisions will be used to guide the implementation of project activities in the remaining years of the project’s life cycle. Methodology Based on the document review and three key review questions, the mid-term review team (MTRT) developed roundtable group discussion (RGD), focus group discussion (FGD) and key informant interview (KII) guides. The data collection covered 17 days in Nairobi from August 26 to September 12. During this period, the evaluation team facilitated five RGDs, four FGDs and seven small group discussions (SGDs) with a total of 86 participants. The SGDs compensated for important stakeholders who were unable to attend RGDs. In addition, the evaluation team interviewed 22 key informants. RGD, FGD and KII respondents comprised representatives from the following stakeholder categories: the MOH, various national health programs, regulatory bodies and national councils, County Health Executive Officers (CHEOs) from seven counties and Health Records and Information Officers (SCHRIO) from 13 sub-counties, development partners, AfyaInfo consortium member organizations and USG funded service delivery partners (SDPs). Following the interviews and roundtables, the evaluation team consolidated findings into the form of integrated tables where they were compared, cross-referenced and triangulated for consistency and patterns of key findings. Progress in Addressing Key Challenges and Gaps of the NHIS 1. What progress has the activity made in addressing the key challenges and gaps of NHIS (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009–2014)? A trend analysis on the use of the health information system showed that, by FY 2013, more than 80% of health facilities across all counties, except Turkana County, had access to a functional HIS with uninterrupted use for at least 24 months before sign-off.2 A similar analysis on community units, between FY 2012–13, showed that 38 out of 47 counties had improved use of the HIS with uninterrupted use before sign-off for at least 24 months. The activity assisted with the formulation and finalization of the Data Quality Assessment (DQA) protocol, which was launched in October 2013. The protocol outlined DQA roles and responsibilities, and detailed the procedures, processes, assessment and supervision tools that must be used to assure data quality at all levels, from the facility, community, sub-county, and county to the national level. The protocol, however, is yet to be institutionalized. High staff turnover was a major issue, resulting in untrained staff managing the data, which in turn led to data inconsistencies or errors that are not caught or corrected. 2 The sign-off time per the project contract is May 2016, therefore 24months before sign off is May 2014.The project defined the achievement of a functional HIS to be primarily demonstrated through timeliness and completeness of reporting through DHIS2. By May 2014, all counties except Turkana were reporting data through DHIS2 by the 15th of every month as expected per MOH reporting timelines. 3 IBTCI: Mid-Term Performance Review of AfyaInfo The Master Facility List’s (MFL) role within the NHIS is seen as the country’s single, authoritative repository for health facility information. Although both the DHIS2 and the MFL had been in existence and operational prior to this activity, AfyaInfo has furthered their uses and functionalities and developed Application Program Interfaces (APIs) for the MFL and DHIS2, creating a possible channel for data exchange between them and with other systems. AfyaInfo supported the MOH to create a health sector M&E Technical Working Group (TWG). This mechanism is intended to drive the process of strengthening the MOH’s M&E agenda and Learning and Knowledge Management (LKM) products, under which the Kenya health sector strategic plans’ M&E Framework and Guidelines (2010) were developed. Other technical support from the activity involved a data demand and information use (DDIU) assessment (2013), which provided the basis for drafting a DDIU strategy (2013) to guide LKM system development and deployment at facility, county, and national levels. Some respondents felt that though there was involvement in gathering the data, there was little or no involvement in generating and disseminating the documents. AfyaInfo supported the MOH to plan for the organizational and institutional changes mandated by devolution in two ways: by assisting to conduct a comprehensive institutional review of the HIS to inform necessary changes, and by continuing to assist to review and revise key governing documents. The GOK’s HIS Policy 2010–2030 and HIS Strategy 2009–2014 needed to reflect the changed operating environment for the health sector following the implementation of devolution. Conclusions Success in the integration and interoperability of various parallel and sub-systems with DHIS2 has been slow and a lot remains to be done. This falls partly outside of AI’s management because different organizations of departments hold ownership of these systems. However, the activity could be on track towards achieving a functional HIS with some adjustments to priorities led by the MOH and with buy-in by all stakeholders at all levels. 2. What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. AfyaInfo did not anticipate the complexities created by devolution during the design of the activity. The activity had to realign its operations and conform to the new, restructured government management system from one national unit to 47 devolved government units. The new structure came 21 months post-award and slowed down the implementation of activities between March, 2013 and December, 2013.3 The GOK HIS Annual Work Plans (AWPs) guided by the original HIS Strategic Plan 2009–2014 have nominal target budgets and are not fully funded in the national budget. According to respondents, although technical capacity exists within local training institutions, poor information communication technology (ICT) infrastructure, such as unreliable electricity and poor internet connectivity, as well as lack of coordination between ICT-related activities impeding service delivery pose ongoing challenges to implementation. The respondents further noted that tools to collect primary data are not always available.4 3 RGD with mid-level MOH staff. 4 A DQA conducted on USAID’s APHIAPlus programs in February 2014 confirms that data collection tools for primary data are not always available at the facility level; further, some facilities were using outdated tools. 4 IBTCI: Mid-Term Performance Review of AfyaInfo Respondents noted that lack of leadership from both the MOH and the activity hampered the implementation of HIS activities and the MOH structure hindered progress in implementation. RGD participants from the Division of HIS voiced dissatisfaction that AI leadership addressed their HIS priorities first, that they were not transparent and did not share activity documents to help in the planning of joint activities. Conclusions The devolved government structure introduced demands on the project that could not have been anticipated during the design of activities, thus impacting speed of implementation. Inadequate funding from the MOH also contributed to not achieving the expected outputs during the first and second years. Perhaps more problematic, however, is the expressed discord in leadership, management and coordination between the MOH and AI management staff due to competing interests and priorities. 3. What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support the MOH in sustaining the use of NHIS? Responses to this question from participants in the FGDs, RGDs, SGDs and KIIs regarding their thoughts on sustainability strategies are noted below: 1. Senior officials view capacity building in the use of the NHIS and in the establishment of LKM platforms as a critical requirement for country ownership; 2. AI is working towards ensuring national and county government and academia have appropriate departments and relevant capacity; 3. Passage of the proposed National Health Bill will give legal backing to the HIS, which respondents deem necessary. It also lays out the roles and responsibilities of stakeholders to the HIS at various levels, which will help provide clarity in the long-term implementation of HIS; 4. Sustainability will depend on active participation and increased engagement by Kenyan institutions as well as private entities. The University of Nairobi’s (UON) involvement provides a mechanism for sustainability; 5. There is a need for USAID to engage the private sector—e.g. Xavier, Denisoft and Savannah Informatics—in their funding mechanism; other respondents noted that as long as USAID funds the development of HIS, there is zero incentive for the GOK to engage the private sector; 6. To achieve sustainability in technical development, AI needs to engage major technical vendors—e.g. HP, IBM— to work on ICT infrastructure networks, client devices and computers, and mobile service provider like Safari.com to lower the cost of recurrent expenditure on Internet use; 7. Advocacy is needed to increase awareness and sensitization on the importance and value of HIS and health data with governors, county executives and county assemblies for buy-in. Conclusions The original concept of AfyaInfo provides the ingredients for sustainability. AfyaInfo’s inputs on IT and capacity building are cumulative and will likely have long-term benefits. However, the MTRT concludes that for the most part, the MOH has not taken ownership of the process in developing 5 IBTCI: Mid-Term Performance Review of AfyaInfo the NHIS. Further, beyond the appreciation of why a NHIS is valuable, responses related to sustainability indicate a lack of responsibility or vision within the MOH for how to achieve sustainability. Instead, the MOH is looking to AI and/or USAID to create sustainability. Until the GOK/MOH believe they are leading the process and are willing to allocate sufficient funding to operationalize the DHIS fully in the long-term, the important gains made by this activity will not be sustainable. 4. Based on the key findings and conclusions drawn from questions 1–3, what are the key recommendations on the strategic, programmatic and management directions that the Mission should consider for mid-course changes to the contract? Question 1 AI, in close collaboration with the MOH, should establish an integrated work plan setting priorities, responsibilities and funding levels required by each with timelines to integrate and ensure interoperability of the various parallel sub-systems within DHIS2 by the end of the activity. Question 2 1. MOH needs to set funding priorities regarding support to infrastructure, tools and human capacity, discuss with AI and determine a feasible plan for funding inputs, designating responsibility for each. 2. Once the MOH has set funding priorities in support of NHIS, USAID might consider continuing with the activity with the understanding that USAID funding will decrease as MOH funding increases annually. 3. As a matter of priority, USAID/OPH, in collaboration with AI headquarter leadership, should determine how best to resolve the leadership and management issues that are currently seen to be hampering the implementation of HIS activities. 4. Establish an Oversight Committee with the MOH taking the lead, supported by a USAID focal person, as well as AI leadership and training institution representatives, with a clear mandate to harmonize the AWPs and monitor progress and execution of activities. Question 3 1. AI should increase advocacy to ensure the MOH at all levels understands the importance of a robust and accurate HIS and its relevance in day-to-day management and strategic planning. 2. AI should ensure that the MOH is in a leadership role, activities are implemented as per MOH priorities and the MOH owns the process moving forward, with AI in a support role only. 3. USAID and/or AI should work with the MOH to determine budget requirements for each activity and require the MOH to budget increasing funds towards sustaining each activity upon handover to the MOH. 4. To achieve sustainability in technical development, AI should work with the MOH to engage the private sector in addressing ongoing ICT infrastructure needs and a mobile service provider to lower the costs of recurrent connectivity expenditures. 6 IBTCI: Mid-Term Performance Review of AfyaInfo 1. INTRODUCTION USAID/Kenya’s (USAID/K) Office of Population and Health (OPH) signed a five-year, $32.8 million contract on June 22, 2011 with Abt International (AI) and its consortium partners (Annex 1 provides the list of partners and responsibilities for each) to design and build a single unified web￾based health information system (HIS) in Kenya. Named AfyaInfo, the activity aims to improve gradually the coherence and effectiveness of a national data system for tracking health problems, activities, resources and outcomes. The activity further supports the Government of Kenya (GOK) in institutionalizing data quality assurance practices including completeness and accuracy at all levels. The activity is aligned with the USAID Implementation Framework 2010–2015’s goal of “sustained “improvement of health and well-being for all Kenyans”. Specifically, this project responds to the Framework’s Area 2: Health Systems Strengthened for Sustainable Delivery of Quality Services (USAID/Kenya Implementation Framework 2010–2015). The complete and accurate program data will inform program planning and decision-making, ultimately helping to improve the health of Kenyans. During August and September 2014, International Business & Technical Consultants Inc. (IBTCI), carried out a mid-term review (MTR) of AfyaInfo. The purpose of the MTR was to assess progress made in the execution of the contract and to determine the factors that have facilitated or hindered the implementation of planned activities in the first three years (June 2011–May 2014). The review further explored strategies for sustaining key results at the national, county, sub-county and health facility levels. The evaluation questions from the performance review’s scope of work (SOW) (See Annex 2 for complete SOW) are: 1. What progress has the activity made in addressing the key challenges and gaps in the national health information system (NHIS) (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009-2014)? 2. What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. 3. What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support the Ministry of Health (MOH) in sustaining the use of NHIS? 4. Based on the key findings and conclusions drawn from questions 1–3, what are the key recommendations on the strategic, programmatic and management directions that the Mission should consider for mid-course changes to the contract? The primary audience for the review is US Government (USG) agencies (Department of Defense (DOD), the Centers for Disease Control and Prevention (CDC) and USAID/K) and the Government of Kenya. Secondarily, the report is intended for AI, USAID/Washington and interested implementing partners. USAID expects that the results of this MTR will inform decisions on: 1. The number of counties that the activity should cover with Information Technology (IT) infrastructure; 2. The number of counties to be covered with technical assistance on information system 7 IBTCI: Mid-Term Performance Review of AfyaInfo organizational capacity development; 3. Choices among the appropriate approaches to implementing activities under this contract; 4. The appropriate sustainability strategies that would ensure long-term use of the NHIS. 2. BACKGROUND Kenya initiated its national health information system in the early 1970s and has since added partners and a policy framework to it. In August 2010, USAID/K supported a comprehensive assessment of the national monitoring and evaluation (M&E) system and NHIS. 5 The assessment aimed to help in designing a mechanism that would work with the MOH to address identified informational gaps. It observed the short-term, ad hoc nature of many of the coordinating mechanisms in play and identified inadequacies in how data is reported, aggregated and interpreted. The assessment found that current practices were driven more by short-term program reporting requirements than by national coherence or sustainability. Additionally, inadequate governance or coordinating structures were identified as gaps. The assessment concluded that “Kenya’s NHIS includes routine service data, census and vital statistics, surveys, surveillance and other… statistics…, however, integration and interoperability is limited”. The assessment’s key findings, conclusions and recommendations were organized under four thematic areas: (i) Data Collection, Quality and Access; (ii) Technology, Processes, Protocols and the Human Interface; (iii) Policy, Organizational Development and Management; and (iv) Information Products, Data Use and Knowledge Management. Development Hypothesis: The AfyaInfo development hypothesis states that: if the GOK/MOH develops, implements and manages a single, integrated web-based NHIS, then there will be increased use of strategic information for activity management, policymaking and decision-making in the Kenyan health sector.6 Key Tasks and Specific Activities to Address the Problem: The specific activities of AfyaInfo have been organized into three key tasks with specific deliverables, each with related quality and performance standards. Task 1: Establish a strong, unified and integrated web-based, host country-owned and managed NHIS that generates quality data used at all levels to improve health service delivery, with 100% coverage of counties, at least 80% coverage of health facilities and 80% coverage of community units in every county, by September 2013. The specific activities under task 1 are to:  Conduct a comprehensive systems requirements analysis and produce a requirements analysis plan with associated costs;  Establish an IT infrastructure (hardware, software and user technical services including policies and protocols) that supports the development, deployment and maintenance of a unified, integrated web-based NHIS;  Take up management of the Kenya HIV/AIDS Program Monitoring System (KePMS)7 and support the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) partners in using this 5 Luoma, Marc et al. August 2010. Kenya Health System Assessment 2010. Bethesda, MD: USAID Health Systems 20/20 project, Abt Associates Inc. 6 See Annex 2, SOW; section A.2.1, Development Hypothesis. 7 Hosted by PEPFAR, the KePMS has functioned as a parallel reporting system, primarily among PEPFAR implementing partners, and has not been integrated with Kenya’s National AIDS/STD Control Program (NASCOP). 8 IBTCI: Mid-Term Performance Review of AfyaInfo system for reporting semi-annual and annual performance results until such a time that a one unified and integrated web-based host country NHIS is fully functional, and KePMS is transitioned into it;  Integrate community health information system, Community-Based Program Activity Report (COBPAR) system and KePMS into one strong unified and integrated web-based NHIS;  Establish a functional national data warehouse (databank) with the appropriate data storage capacity, data confidentiality and data security for every user type;  Develop appropriate performance indicators for assessing the progress. Task 2: Based on the geographic coverage of task 2 (100% coverage of counties), establish a functional GOK-managed learning and knowledge management (LKM) system that improves the culture of information generation, knowledge capturing and information use by September 2015. The specific activities under task 2 are to:  Develop a GOK-managed LKM system for the health sector;  Conduct training needs assessment for MOH staff on management of LKM systems and produce a training needs assessment report;  Conduct capacity building programs (including training on specific technical areas) to develop institutional and human capacity to launch and manage the LKM agenda in the health sector;  Develop a range of appropriate information products, create demand for these products and establish relevant public awareness and dissemination forums and systems to ensure use of these information products;  Develop appropriate performance indicators and benchmarks for assessing the progress. Task 3: Establish a functional health management information system (HMIS) division that is capable of passing a USAID pre-award responsibility determination assessment on leadership and management, financial and procurement capability. The specific activities under task 3 are to:  Develop and implement appropriate capacity building programs to strengthen management and coordination structures based on already existing policies and governance structures;  Develop and implement appropriate capacity building programs to strengthen financial, technical and human resources management systems;  Develop appropriate performance indicators for assessing yearly progress. 3. METHODOLOGY IBTCI fielded two independent consultants, a Senior Expatriate M&E Specialist and a Senior Kenyan M&E Specialist (See Annex 3 for consultant CVs) to form the MTR team (MTRT). Both team members attested to having no conflict of interest in conducting the MTR (See Annex 4 for signed attestations). The MTR was comprised of three phases: (1) document review that provided the basis for designing data collection instruments (group discussion and key informant interview guides), (2) data collection, and (3) data analysis and report writing. The MTR review of documents and fieldwork started on July 24, 2014 and ended on September 17, 2014. The methodology was based on the SOW (a detailed methodology is presented in Annex 5). The MTRT reviewed relevant documents provided by the USAID/K and AI. The list of reviewed documents is presented in Annex 6. 9 IBTCI: Mid-Term Performance Review of AfyaInfo Based on the document review and the four key review questions enumerated in the Introduction above, the MTRT developed roundtable group discussions/focus group discussions (RGDs/FGDs) and key informant interview (KII) guides (Annex 7). Pilot-testing the guides was not necessary as the questions were meant to be open-ended to allow participants to provide longer answers and to facilitate discussion. The MTRT conducted five RGDs, four FGDs, and seven small group discussions8 (SGDs) totaling 86 participants from August 26 to September 12. Respondents comprised representatives from the following stakeholder categories: the MOH, various national health programs, regulatory bodies and national councils, county health executive officers (CHEOs) from seven counties, sub-county health records and information officers (SCHRIO) from 13 sub￾counties, development partners, AfyaInfo consortium member organizations and USG funded service delivery partners (SDPs). (Schedule, list of represented organizations and number of RGD, FGD and SGD participants are presented in Annexes 8 and 9.) The SGDs compensated for important stakeholders who were unable to attend RGDs. In addition, the evaluation team interviewed 22 key informants (schedule and list of KIIs is presented in Annex 10). Limitations: Due to the recent restructuring of the MOH and devolution, various GOK officials rotated out of their national and county positions and many newer officials lacked the institutional knowledge and history with AfyaInfo to share in-depth information with the MTRT. Eleven respondents from the MOH were unable to attend the initial round of RGDs and as a result, SGDs were held to capture their input. However, responses to Question 3 (sustainability) were less than optimal as the respondents did not seem to have a clear vision of how they (the MOH) could participate in ensuring the sustainability of the HIS. While RGDs/FGDs/SGDs were a valuable component of the review, those within the group could have been influenced by the opinion of other participants; therefore, the results from these groups may not be generalized. 4. FINDINGS AND CONCLUSIONS 4.1 Review Key Question 1 What progress has the activity made in addressing the key challenges and gaps of NHIS (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009-2014)? Findings During its first year, AfyaInfo completed a comprehensive baseline assessment of the systems9 considered to be priority data sources for the NHIS. This assessment identified the NHIS Information and Communication Technology (ICT) needs in terms of hardware, software, and user technical services including policies and protocols, along with the preparation of a roadmap for progress toward an integrated national health information system. Using the findings of this systems assessment, in Year 2 and Year 3, AfyaInfo developed the minimum new HIS interoperability standards, “Kenya Standards and Guidelines for e-Health Systems Interoperability,” which outline a set of rules for interoperability in health data reporting. 8 SGDs are comprised of four or fewer respondents. 9 District Health Information Software Version 2 (DHIS2), Master Facility List (MFL), Kenya Health Workforce Information System (KHWIS), Integrated Human Resources Information System (iHRIS), Kenya Medical Supplies Agency (KEMSA) commodity systems, Kenya Quality Model for Health (KQMH), KePMS, COPBAR, Health Sector Services Fund (HSSF), Malaria Information and Acquisition System (MIAS), Regulatory Human Resources Information System (rHRIS), Electronic Supply Chain Management (eSCM), Civil Society Organizations (CSO) Database, Health Commodities and Services Management (HCSM), Electronic Medical Records (EMRs). 10 IBTCI: Mid-Term Performance Review of AfyaInfo Functional and Integrated Health Information System A trend analysis on the use of the health information system showed that, in FY 2013, more than 80% of health facilities, across all counties except Turkana County, had access to a functional HIS with uninterrupted use for at least 24 months before the AfyaInfo project sign-off date. A similar analysis on community units, between FY 2012-13, showed that 38 out of 47 counties had improved use of the HIS with uninterrupted use for at least 24 months before sign-off. Twenty counties reported more than 50% community units had uninterrupted use for at least 24 months before sign￾off.10 AfyaInfo finalized the NHIS Infrastructure Deployment Framework, the Infrastructure Deployment Implementation Strategy and the related procurement plan. Complete and Accurate Reporting 18 Months after System Deployment The activity assisted with the formulation and finalization of the Data Quality Assessment (DQA) protocol, which was launched in October 2013. The protocol outlined DQA roles and responsibilities, and detailed the procedures, processes, assessment and supervision tools that must be used to assure data quality at the facility, community, sub-county, county and national levels. The protocol, however, is yet to be institutionalized. In FY 2012, independent data quality audits 18 months after system deployment showed that 91.3% of health facilities reported complete and accurate data as required by facility-based programs in the health sector through the NHMIS.11 However, a DQA conducted by IBTCI in February 2014 found that there were inconsistencies in terms of reliability in the data collection tools at the facility level, and the majority of facilities reviewed did not use tally sheets—instead, they used their own methods and calculations for data. Registers and summary tools were not always completed, and in some cases non-existent. High staff turnover was a major issue, resulting in untrained staff managing the data, which in turn leads to data inconsistencies or errors that are not caught or corrected.12 By Years 2 and 3, quarterly print and electronic materials about health information were distributed at all levels. AfyaInfo developed a standardized seven-module comprehensive package of NHIS strengthening materials and methods targeting health managers, other data generators and data consumers for District Health Information Software Version 2 (DHIS2), Master Facility List (MFL) and Master Community Unit List (MCUL), used for capacity building purposes in Year 2 through a collaborative process with other stakeholders. A comprehensive training plan was developed and the trainings kicked off in the second quarter of Year 3. AfyaInfo also trained a pool of 150 national trainers drawn from the MOH, faith-based organizations, SDPs, training institutions and private facilities on the NHIS curriculum. To strengthen the pre-service HIS capacity, following up on the training needs assessment (TNA), the activity supported the revision of the Health Records & Information Management curriculum at Kenya Medical Training College (KMTC) and Kenyatta University (KU) to include the seven-module comprehensive curriculum. Meanwhile, in-service, country-wide NHIS training targeted County Health Management Teams (CHMTs), data managers and community data managers. AfyaInfo and the MOH trained 27 data managers of Kenyatta National Hospital (KNH). The data managers were trained in data entry, reporting and data manipulation for DHIS2, and issued with user rights. The aim of this training was to ensure that KNH, the highest-volume facility in the country, can report through DHIS2, which it 10 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 2nd quarter. 11 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 2nd quarter. 12 IBTCI. May 2014. Data Quality Assessment conducted in February 2014. 11 IBTCI: Mid-Term Performance Review of AfyaInfo hadn’t been due to lack of skills in data entry through DHIS2 among its health records personnel. NHIS training at the national and county levels will continue in the first quarter of Year 4.13 Transitioning of Parallel Systems into One Unified National HIS As of May 2014, the following systems, subsystems and data sets have been integrated and/or made interoperable in varying degrees with DHIS2. The current list of systems, subsystems and data sets and their achieved level of operability includes:  Master Facility List and Master Community Unit List (full interoperability);  MFL and Regulatory Module (full interoperability);  MFL and DHIS2 (semi-automated: 50% interoperability; manual process/invocation required);  DHIS2 functionality has been expanded to include three data sets from these systems: Health Sector Service Fund (HSSF), Kenya Quality Model of Health (KQMH) and e-Supply Chain Management System (eCMS). The MFL’s role within the NHIS is seen as the country’s single, authoritative repository for health facility information.14 Although both the DHIS2 and the MFL had been in existence and operational prior to this activity, AfyaInfo has furthered their uses and functionalities and developed Application Program Interfaces (APIs) for the MFL and DHIS2, creating a possible channel for data exchange between them and with other systems. The Regulatory Human Resources Information System (rHRIS), regulatory boards and councils and various other entities were introduced to MFL codes and are at different stages of engagement regarding its adoption. A number of other health information systems and subsystems including those of the Kenya Medical Supplies Agency (KEMSA) and enterprise resource planning (ERP) tools have adopted or are in the process of adopting the unique MFL codes. In addition, MFL was enhanced with geocoding, although not always updated accurately, and the Regulatory Module was completed and connected with the MFL. However, the MFL is not updated regularly and service information is outdated. According to respondents, this is partly because the “sub-county health records information officers don’t have the required administrative rights to update the systems”.15 AfyaInfo led the development and operationalization of the MCUL, a web-based database designed to capture basic data about more than 2,250 Community Units (CU) across Kenya. Its unique identifier code for each CU allows data on that unit to be linked to data on its associated “link facility” in the MFL. These unique identifiers at the core of the MCUL for community units and MFL for health facilities also make it possible to exchange data with the Kenyan DHIS2 platform, which contains routine health service delivery data. According to respondents, MCUL information is also outdated. COBPAR is a standalone system and functions to collect data mostly through Civil Society Organizations (CSOs) supported by the SDPs, but it is not linked to the Community Health Information System (CHIS), which is currently neither integrated nor interoperable. AfyaInfo is addressing this by supporting the MOH to strengthen data collection at the community level and to integrate the country’s leading community health information systems into the NHIS. Kenya’s community health data is mainly captured in CHIS, owned by the MOH’s Community Health 13 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 4th quarter. 14 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 4th quarter. USAID’s support for the creation of a MFL is a critical initiative in moving towards system integration. The web-based facility database, provides a unique set of health facility codes—the primary key necessary to link all the facility based databases and to integrate information systems. (AfyaInfo Year 1 2nd Quarterly Report) 15 FGD SCHRIOs. 12 IBTCI: Mid-Term Performance Review of AfyaInfo Services Unit (CHSU); COBPAR is managed by the National AIDS Control Council (NACC), and KePMS is owned by the USG’s PEPFAR. With AI facilitation, the head of the CHSU provided the CHIS Integration Technical Working Group (TWG) with an approved model of data integration for the attention of the M&E Director of NACC. NACC leadership agreed in principle to sharing essential data sets between COBPAR and DHIS2. AI hopes to formalize this agreement in the first quarter of Year 4, with the goal that the technical teams of CHSU and NACC will begin sharing data between COBPAR and DHIS2.16 Management and Phasing out of KePMS KePMS is a computerized database for the management and analysis of PEPFAR’s treatment and prevention indicators designed to be responsive to PEPFAR reporting requirements. AfyaInfo coordinates with all USG SDPs and manages KePMS. The activity successfully supported USAID/K and the SDPs in their reporting through the common KePMS platform for semi-annual and annual performance reports in 2011, 2012 and 2013. However, some system glitches have been consistently reported in KePMS. These include the network version not being stable and data elements not adding up to the total.17 AyfaInfo contributed to the broader efforts to harmonize MOH 71118 and MOH 73119 at the facility level between DHIS and KePMS. According to the 2012 Baseline Study as well as information from officials at the National AIDS and Sexually Transmitted Diseases Control Program (NASCOP), AfyaInfo and the MOH, these forms now cover about 70% of the KePMS (PEPFAR) reporting requirements. The KePMS to DHIS2 (K2D) transition is scheduled to take place in the fourth year of the activity, with the software platform tested. According to USAID respondents, “If we are to phase out KePMS, there needs to be more work done on DHIS2 to accommodate KePMS, though their changes are frequent and outside the stakeholders’ control”. Further, although data sources are the same, there are disparities in accuracy.20 A K2D TWG was formed to oversee the transition consisting of two sub￾committees: M&E (oversees the process of transition) and system enhancement (advises on what changes are needed in DHIS2 to ensure it can support PEPFAR reporting needs and other system requirements). In addition, training was conducted for KePMS M&E data users to incorporate the changes in PEPFAR reporting guidance from the U.S. Office of Global AIDS Coordination (OGAC). During the MTR focus group discussions and key informant interviews, the respondents were in agreement that there is a need for discussions at a higher level to ensure that USG and GOK agree on the way forward with regard to having DHIS2 as the main health information system. ICT Service Desk There are discrepant views regarding the effectiveness of the ICT Service Desk. Some MOH and DHIS respondents noted that the Service Desk is either “non-existent” or exists but is “non￾functional,” while others noted they received help. During the last quarter under review (April–June 2014)21, AfyaInfo completed Phase 1of NHIS infrastructure procurement for the Service Desk including the delivery and installation of specified IT equipment to MOH headquarters. The new 16 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 4th Quarter. 17 Abt Associates Inc. AfyaInfo Annual Report Year 2; IBTCI. February 2014. Data Quality Assessment. 18 MOH 711 is the summary tool compiled at the facility level for reporting HIV and MCH data and uploaded into DHIS2 on a monthly basis. 19 MOH 731 is the summary monthly data tool for collecting HIV-related data for uploading to DHIS2. 20 IBTCI. May 2014. Data Quality Assessment conducted in February 2014. 21 The MTRT understands the period of this review ends in May 2014, however, it was difficult to disaggregate achievements in just April and May from the April–June Quarterly Report. 13 IBTCI: Mid-Term Performance Review of AfyaInfo infrastructure included three servers (live, test and backup units), two headsets (microphone and ear pieces for use by service desk staff in responding to requests from the field), one private automated branch exchange system for landlines, and four mobile phones. The School of Computing and Informatics, housed at the University of Nairobi (UON), is one of the MOH selected sites. Upon commissioning, the NHIS Service Desk will be accessible to internal and external users by phone and through an established web portal. Installation of the NHIS Service Desk is scheduled to be completed in the first quarter of Year 4.22 The service Desk will improve user access to services such as remote troubleshooting and advice on problems encountered with ICT infrastructure use and use of HIS priority systems. Functional Learning and Knowledge Management System The MOH was not familiar with the LKM concept and “selling the concept took time”. 23 The fact that the MOH had no M&E unit meant that it was not easy to anchor a LKM system.24 AfyaInfo supported the MOH to create a health sector M&E TWG as a mechanism to drive a process of strengthening the MOH M&E agenda and LKM products, under which the Kenya health sector strategic plans’ M&E Framework and Guidelines (2010) were developed.25 Other technical support from the activity involved a data demand and information use (DDIU) assessment (2013), which provided the basis for drafting a DDIU strategy (2013) to guide LKM system development and deployment at facility, county and national levels.26 The activity supported the development of MOH information products including the HIS annual statistical report, quarterly reports, HIS bulletin and DHIS2 dashboards. These products are available to the MOH, however respondents noted during the RGDs and FGDs that they have not yet been distributed to all levels in the country and therefore are not being systematically used. By the third year, there were reliable and timely web￾based public health information databases, including the regulatory module and eSCM. Respondents indicated that there was heightened interest to access and use DHIS2. They reported that they made more frequent calls to the HIS team, requiring help in manipulating or accessing information on DHIS2. Some respondents felt that though there was involvement in gathering the data, there was little or no involvement in generating and disseminating the documents. HMIS Division Institutional Capacity AfyaInfo supported the MOH to plan for the organizational and institutional changes mandated by devolution in two ways: by assisting to conduct a comprehensive institutional review of the HIS to inform necessary changes, and by continuing to assist to review and revise key governing documents. The GOK’s HIS Policy 2010–2030 and HIS Strategy 2009–2014 needed to reflect the changed operating environment for the health sector following the implementation of devolution. These key documents have been revised and are pending MOH approval for their launch. AfyaInfo provided support in this process of strategy revision; it worked with the Division of Health Information/M&E (DivHIME) select committee to draft HIS inputs into the draft health bill. During this process, three functional units were created under the DivHIME: E-health, HIS and M&E, following the recommendations of a restructuring committee comprised of both HIS and AfyaInfo staff. AfyaInfo also supported numerous studies involving assessments, frameworks, baseline surveys, training curricula and strategic plans that were completed (e.g. Organizational Assessment for 22 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 3rd Quarter. 23 AfyaInfo SGD. 24 AfyaInfo FGD; HIS staff FGD. 25 AfyaInfo FGD. 26 Abt Associates Inc. AfyaInfo Quarterly Report Year 3, 2nd Quarter. 14 IBTCI: Mid-Term Performance Review of AfyaInfo Division of HIS (2011), National HIS Institutional Assessment, National M&E Framework, ICT infrastructure, HIS Readiness Assessment for 36 out of 47 counties and HIS Strategic Plan for two counties, to cite only a few). At the request of the MOH, AfyaInfo seconded an ICT advisor to the MOH to build capacity in systems maintenance and support, and to build functional linkages between the staff of the ICT and HIS Divisions. AfyaInfo also facilitated dissemination forums of the HIS assessment reports in each of the three counties (Uasin Gishu, Homabay Kisumu and Busia) where AfyaInfo will pilot activities aimed at strengthening overall county planning and stakeholder coordination. Conclusions Some of the key barriers to performance to date were extrinsic, related to larger national processes (constitution, politics, new appointees, new system of devolution, sorting out roles and leadership). But other key barriers have been intrinsic and related to a lack of adaption to complex situations by AfyaInfo’s leadership. Functioning HIS Success in the integration and interoperability of various parallel and sub-systems with DHIS2 has been slow and a lot remains to be done, some of which is outside of the management of AI due to ownership of these systems being held by different organizations or departments. However, the activity could be on track towards achieving a functional HIS with some adjustments to priorities led by the MOH and with buy-in by all stakeholders at all levels. Complete and Accurate Reporting 18 Months after System Deployment Complete and accurate reporting remains a challenge and will likely remain so until such time as the MOH is able to address issues of staff retention and standardized data collection tools are available at all times and levels. Despite these challenges, AI has done a remarkable job in providing continuous training at all levels in an effort to improve complete and accurate reporting. Management and Phasing out of KePMS Unless the reporting requirements and the rigor in data collection between DHIS2 and KePMS can be harmonized, the MTRT does not foresee the phasing out of KePMS going according to schedule. The expectation that the system will be able to adequately support PEPFAR reporting requirements is likely unrealistic within the timeframe outlined. ICT Service Desk AI is making strides to equip and operationalize the service desk; however, the perceptions of respondents were varied. It could be because procurement and operationalizing came late in the period under review and not everyone is aware that it is functioning and available to provide needed assistance. Functional Learning and Knowledge Management System The LKM concept and activities appear to be largely unclear and misunderstood by many stakeholders at all levels within the MOH. While the activity has generated a lot of information, its dissemination and use for management, planning and decision-making appears to be limited particularly at the county, sub-county and facility levels. 15 IBTCI: Mid-Term Performance Review of AfyaInfo HMIS Division Institutional Capacity AfyaInfo has provided and continues to provide institutional capacity building to the HMIS division; however, it is not clear what the uptake of these activities has been within the division, as some have not yet been adopted and others are still at the planning stage. 4.2 Review Key Question 2 What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. Findings Devolution During the design and the initial stages of AfyaInfo, the activity did not anticipate the complexities posed by devolution.27 The activity had to contend with and conform to the new and restructured government management system, i.e. from one national unit to 47 devolved government units. This new structure, introduced nearly 21 months post-ward, slowed down the uptake of activities between March, 2013 and December, 2013. Leadership and Management Respondents noted that lack of leadership, from both the MOH and the activity, is hampering the implementation of HIS activities.28 There was lack ofinsufficient support from the government and the secretariat “did not have much clout.”29 The informants claimed that although the relevant technical capacity exists within the local training institutions, the MOH structure hinders progress in project activities. Respondents also noted that, at times, AI implements HIS-related activities without involving the Ministry—e.g. they go to the counties without involving the Division of HIS. One respondent noted, “We need to work like the GOK is being supported by Al rather than Al being supported by the GOK.”30 Other respondents noted the lack of transparency around the budget and the work plan; that AfyaInfo didn’t share project documents to help with planning; and that “there wasn’t any clear process or directives” and “we understand as we work.”31 Further, MOH respondents believe there is generally a mismatch between annual work plans (AWPs) and the priorities of AI, with HIS staff expressing dissatisfaction with the implementation of activities as AI priorities are addressed first. Within AI, there is an occasional lack of decision space among the Output Leads and poor coordination between AI and GOK partners in planning and field work.32 Inadequate Funding to Support the Legal Framework and Strategic Plan AfyaInfo supported the production of the first “Health Information Strategic Plan”, which set forth ten objectives at an estimated expense of $75 million for the period 2013–2018. The GOK’s HIS AWPs that are guided by the original HIS Strategic Plan 2009–2014 have nominal target budgets, and are not fully funded in the national budget. AfyaInfo has supported the creation of a Health Law that provides for health information systems, which has core buy-in from the governmental 27 AI staff KII; Consortium members RGDs. 28 MOH RGD; HIS RGD. 29 CDC KII. 30 MOH HIS staff SGD. 31 MOH HIS staff SGD. 32 AI and Consortium partners KII and FGD. 16 IBTCI: Mid-Term Performance Review of AfyaInfo stakeholders interviewed. The National Health Bill is currently under consideration in Parliament. If passed, among other things, it would provide the legal backing for roles and responsibilities for NHIS at various levels. “The health bill talks of HIS, it will cement everything else when it comes out.”33 Management Systems Challenges As is typical in many activities, the AfyaInfo Performance Monitoring Plan (PMP) and associated indicators were developed early in the activity life cycle. While the PMP anticipated measures of progress, including a mix of outcome and output/process indicators across the three outputs areas, it did not anticipate the delays that would eventually occur. The activity reported that some measurement constraints were not fully understood at the time, limiting the activity’s ability to actually measure indicators as they may have been originally intended. Most of the indicators in the PMP are of qualitative nature and have not been quantified. It is difficult to measure or demonstrate the extent to which the tasks have been achieved over time, especially if the tasks are rescheduled over a period of time across years, as was the case for many AfyaInfo interventions. Although indicators for specific activities 1.1 to 1.334 are quantitative in nature, unfortunately, they unfortunately do not give an accurate measure of the actual quality (accuracy, timeliness and completeness) of data and system strengthening aspects as they mostly refer to the reporting rates in the DHIS2 and the number of places the DHIS2 system is being used, rather than on the uninterrupted use of the system. However, given the complex nature of this activity and the emphasis on utilizing a systems perspective, the activity has tried to complement its PMP with more detailed performance reporting in both narrative and other forms, such as quarterly and annual reports and tracking/monitoring of detailed implementation plans. There are key limitations in many of the PMP indicators about how one might interpret the level and extent of success. For example, the activity tries to answer Output 2, Indicator 2.9 “Quarterly print and electronic materials on health information and its usefulness available and being produced and distributed at all levels” with a simple “Yes” or “No”, whereas this requires clarity such as how many of the scheduled products were actually delivered, their timeliness and usage. Under Output 3, Indicator 3.6 “NHIS stakeholder coordination mechanisms developed, in place and functioning” is again answered with a simple “Yes” or “No”, yet this requires further clarity such as which mechanisms, their levels, and most importantly their actual functionality and effectiveness (See Annex 11 for PMP). The DHIS2 has been adopted (including buy-in by the new counties leadership) and widely used as the national standard data reporting system. However, the MTR FGDs and RGDs found that many stakeholders seek to see further enhancement and implementation of DHIS2 as the unified and integrated “one” national system for most health information. The respondents also claimed that the “Health Records Information Officers are yet to embrace community data as part of their work”35 since health facility data is often prioritized over community data. Other respondents noted data management tools are not readily available at all levels, data tools for collecting MCUL data were not disseminated and the available tools are not standardized.36 33 MOH HIS staff SGD. 34 1.1 Percentage of health facilities where HIS is in use for at least 24 months uninterrupted before sign-off; 1.2 Percentage of CUs where HIS is in use for at least 24 months uninterrupted before sign-off; 1.3 Percentage of facilities reporting complete and accurate data as required by facility-based programs in the health sector through NHIS 12 months after system deployment. 35 MOH RGD. 36 DHIS, CHRIOs and SDPs FGD/RGDs; IBTCI. May 2014. Data Quality Assessment conducted in February 2014. 17 IBTCI: Mid-Term Performance Review of AfyaInfo Although technical capacity exists within the local training institutions, the respondents claimed poor ICT infrastructure, unreliable electricity and poor internet connectivity as well as the lack of coordination of ICT-related activities are impeding service delivery. Another hindrance mentioned is that the ICT function is not under the MOH, therefore ICT staff in the MOH are not employees of MOH and report to another Department. Conclusions Leadership and Management The senior management of AfyaInfo faces a legacy of having been ineffective in finding common ground with MOH counterparts. This apparent disparity between perceptions in leadership and activity management between AI and the MOH is hampering efforts to transform the HIS into a robust, fully operational and effective system. Inadequate Funding to Support the Legal Framework and Strategic Plan While AI activities are not supported with funding by the MOH, there is inadequate funding from the MOH to support the successful implementation of its HIS strategic Plan 2009–2014 objectives. Given that the MOH and AI work plans are supposed to be harmonized, this could have contributed to the slowing down of activities in Years 1 and 2. Unless sufficient funding is allocated to implement the strategy successfully, challenges will remain in reaching the activity’s goals. Management Systems Challenges The new devolved government structure introduced demands on the project that were not earlier envisaged in the Health Sector Strategic Plan for HIS (2009–14) and most certainly was the cause of some delay in implementation. The planned activities in the PMP are too ambitious and ambiguous, thus limiting the project’s ability to realize them over the life of the project and to measure outcomes accurately. This has significant overall effect on both the outcome of the activity and the actual measure of success. The availability of data collection tools is outside the scope of this activity. However, until tools are readily available at all levels and versions harmonized37, managing data for accuracy and timeliness will continue to pose a management issue. Additionally, until ICT infrastructure and coordination of ICT-related activities are fully functioning, it will be difficult to meet Output 1 of the activity: “establish a strong, unified, and integrated web-based host country-owned and managed NHIS that generates quality data used at all levels to improve health service delivery.” 4.3 Review Key Question 3 What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support MOH in sustaining the use of NHIS? Findings In answer to this question, responses from participants in the FGDs, RGDs, SGDs and KIIs regarding their thoughts on sustainability strategies are noted below: 37 Tools have been revised at various times. However, not all facilities use current versions; therefore there are a variety of versions of tools being used. 18 IBTCI: Mid-Term Performance Review of AfyaInfo 1. Senior officials38 view capacity building in the use of the NHIS and in the establishment of LKM platforms as a critical requirement for country ownership; 2. AI is working towards ensuring national and county government and academia have appropriate departments and relevant capacity; 3. Passage of the proposed National Health Bill will give legal backing to the HIS, which is deemed as needed by the MOH. It also lays out the roles and responsibilities of stakeholders at various levels towards the HIS, which will help in providing clarity in the long-term implementation of HIS; 4. The MOH ICT infrastructure should be strengthened, scaled up and upgraded to current technology. One respondent noted, “The ministry needs to have a cloud. The infrastructure is there but for it to be used, they need to create a cloud so all other systems can go to the cloud. Partners are spending a lot of money to use third-party clouds”39; 5. Sustainability also depends on active participation and increased engagement by Kenyan institutions as well as private entities. The UON’s involvement provides a mechanism for sustainability. The university says it currently has 30 students carrying out research on adoption, system integration, adding and removing interfaces; 6. There needs to be support for research in computing and informatics at the Bachelors, Masters and PhD levels in education; 7. Curriculum development in collaboration with the Kenya Medical Training College and Kenyatta University should ensure that future changes to the development of NHIS and DHIS will be taught at the same time to avoid re-training of people in the field; 8. Some respondents noted a need for USAID to engage the private sector—e.g. Xavier, Denisoft and Savannah Informatics—in their funding mechanism; other respondents noted that as long as USAID funds the development of its systems through its developing partners, there is zero incentive for the GOK to engage the private sector; 9. To achieve sustainability in technical development, AI needs to engage major technical vendors—e.g. HP, IBM—to work on ICT infrastructure networks, client devices and computers, and mobile service provider like Safaricom to lower the cost of recurrent internet access expenditures; 10. Advocacy is needed to increase awareness and sensitization on the importance and value of HIS and health data with governors, county executives and county assemblies for buy-in. Conclusions The original concept of AfyaInfo provides the ingredients for sustainability and AfyaInfo’s inputs on IT and capacity building are cumulative and will likely have long-term benefits. However, the MTRT concludes that for the most part, the MOH has not taken ownership of the process in developing the NHIS. Further, beyond the appreciation of why a NHIS is valuable, there is an insufficient sense of leadership, responsibility or vision within the MOH for how to achieve sustainability. Until the GOK/MOH believe they are leading the process and are willing to allocate sufficient funding to operationalize the DHIS fully in the long term, while decreasing their dependence on donors; the important gains made by this activity will not be sustainable. 38 Sr. Health Informatics Advisor; Assistant Chief Clinical Officer; Assistant Chief of Pharmacy; DCCO. 39 KII CDC 19 IBTCI: Mid-Term Performance Review of AfyaInfo 4.4 Review Key Question 4 What are the key recommendations on the strategic programmatic and management directions that the Mission should consider for the mid-course changes on the contract? The MTRT provides the following recommendations per key question and conclusions based on the findings. Key Question 1 What progress has the activity made in addressing the key challenges and gaps of NHIS (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009–2014)? Functioning HIS; Complete and Accurate Reporting 18 Months after System Deployment With the MOH in the lead and AfyaInfo providing support, determine feasible priorities for the remaining length of the project towards: 1) achieving the integration and interoperability of all systems, 2) addressing the issues surrounding interruptions in the use of HIS (connectivity) as well as the completeness and accuracy of data (tools) and 3) ensuring continuous updating of the MFL/MCUL subsystems in DHIS2 by granting a select group within the MOH administrative rights to the system. Working closely together, prepare an integrated work plan to achieve these priorities outlining responsible parties, human and financial resources needed (and who will be responsible for each) as well as a timeline to complete implementation. Conduct monthly progress report meetings with key stakeholders charged with implementing the work plan, during which stakeholders keep each other accountable and on task to meet the deliverables within the timeframes outlined in the work plan. Management and Phasing out of KePMS It could be that merging KePMS with DHIS2 is not feasible based on PEPFAR reporting requirements. USAID might consider putting the phasing out of KePMS on hold at least until such time that successful and timely progress towards the work plan recommended above is documented and PEPFAR indicators stabilize. ICT Service Desk The MTRT recommends that Abt and USAID determine why there are discrepancies in opinion as to the efficacy of the Service Desk. If perceptions on either side are incorrect, take aggressive steps to correct these misperceptions. AfyaInfo should ensure that all users have access to the Service Desk’s website as well as the phone numbers and information on hours of availability. Functional Learning and Knowledge Management System Much of AfyaInfo’s work, at the higher levels, has been about process, including platforms and actors, with perhaps less attention is paid to the content and use of health data. Case studies or case examples that document how health actually improves from better use of data may be an ingredient to demonstrate the value of an accurate and reliable integrated data system. AI should realign resources if necessary and accelerate support in the operationalization of a robust LKM system led by the MOH at all levels, with particular attention paid to the entire county-level workforce, starting with the point of data generation and entry. 20 IBTCI: Mid-Term Performance Review of AfyaInfo HMIS Division Institutional Capacity In concert with the MOH, AI should assess the efficacy of the institutional capacity building done to date, determine gaps, prioritize addressing the gaps and design a joint work plan to address the gaps, including outlining responsibilities for action and timeline for completion. Key Question 2 What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. Leadership and Management USAID should review with AI leadership the findings regarding the perceptions of the MOH towards the in-country leadership and management of the program, and determine what corrective action is needed to establish a relationship conducive to meeting AfyaInfo’s goals during the remaining period of implementation. To improve the relationship further between AI and the MOH, consider revitalizing the Oversight Committee with the MOH taking the lead, supported by a USAID focal person, as well as AI and training institution representatives with a clear mandate to harmonize the AWPs and monitor the progress and execution of activities. USAID can play a critical role in facilitating a new dialogue to help reboot the relationship (See section Future Directions for more information). Inadequate Funding to Support the Legal Framework and Strategic Plan USAID might consider working with the MOH to lobby for the passage of the Health Bill as well as for Parliament to allocate sufficient funding to implement the strategic plan. AI, with support from USAID, should work with the MOH at the national and county levels to determine adequate but feasible funding levels to support the implementation of DHIS2 and activities to attach these funds to. If funding is not forthcoming from the MOH to uphold their commitments to DHIS2, USAID might consider scaling back activities overall through the remainder of the project. Indeed, without MOH committing to budget funding for these activities now and in the long term, the sustainability of the HIS without significant donor assistance is questionable. Management Systems Challenges Given the significant setbacks AfyaInfo has seen so far, its milestones should be reconsidered in collaboration with USAID and the MOH, as well as the measures of its overall progress (long-term outcomes) for the activity as a whole. In the process, clear linkages should be established between the PMP and the activity’s results framework, as well as more relevant and useful interim measures of performance (output and outcome levels). Key Question 3 What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support MOH in sustaining the use of NHIS? 1. AI should ensure the MOH is in a leadership role, that activities are implemented as per MOH priorities and that the MOH owns the process moving forward with AI in a support role only. 21 IBTCI: Mid-Term Performance Review of AfyaInfo 2. AI should support the MOH to increase advocacy at all levels to ensure stakeholders understand not only the importance of a robust and accurate HIS, but also the vital role accurate information can play in day-to-day management and strategic planning. 3. USAID and/or AI should work with the MOH to determine budget requirements for each activity and require the MOH to budget funds towards sustaining each activity in escalating amounts, to ensure sustainability upon handover to the MOH. 4. To achieve sustainability in technical development, AI should support the MOH to engage the private sector in addressing on-going ICT infrastructure needs and a mobile service provider to lower the costs of recurrent connectivity expenditures. 5. FUTURE DIRECTIONS AND PRIORITIES Under the new Constitution of Kenya 2010, 47 counties now operate their distinct health programs, without formal documents and/or directional brief yet outlining the detailed institutional roles and responsibilities of the national and county levels.40 As such, working together under the overall guidance and oversight of a coordinating entity, the AfyaInfo project along with USAID and the GOK should consider course adjustments to fulfill the goals of a modern, integrated NHIS system, which should now orient toward capacity building at county-level offices. Looking ahead, there are many opportunities for progress through collaboration. For example, ongoing initiatives by other partners on HIS curriculum and mobile solutions offer great opportunities for cross-sharing and economies of scale. This review observed a trend toward renewed excitement and expectation, while also a sense of urgency. This renewed hope, coupled with speedy, prioritized and efficient implementation of key recommendations made in this report, can potentially lead to greater promise and commitment toward better coordination, higher efficiency, greater involvement, improved management and refined strategy—all ingredients for a much more successful next 21 months of the AfyaInfo activity, with greater results toward a stronger NHIS. There appears to be an agreement among all key stakeholders on the importance of a unified and integrated NHIS. USAID, with its partners, should consider support to DHI/M&E management capacity to make evidence-based budget requests for indigenous NHIS strengthening. Whether a revitalized coordinating entity is called the Project Oversight Coordination Committee (POCC) or something else, the function is vital.41 As part of the strategic approach referenced in the recommendations above, the POCC could support the following tasks: 40 The Fourth Schedule of the Constitution of Kenya 2010 – Distribution of Functions between National Government and County Governments and the widespread discussions on the draft Health Bill (currently before Parliament) indicate that responsibilities of the national government in the health sector (MOH) will focus primarily on health policy; standards, quality assurance and regulation; capacity building; coordination; and national health referral facilities, while the county governments will be responsible for county health services. 41 According to the Head of NHIS, this committee existed and held two meetings, then disbanded. He believes revitalizing this committee is important. 22 IBTCI: Mid-Term Performance Review of AfyaInfo  Ensure effective and inclusive dialogue among all stakeholders to foster consensus building on all key activities;  Effectively analyze and devise plans that further align the work of AfyaInfo with the new Kenya Health Bill, once the latter is approved;  Ensure that the activities are part of GOK (budget) planning (e.g. AWP) and support the overall health sector goals and priorities at both national and county levels;  Ensure that any activities supported outside the AWP are relevant, demand-driven and fully aligned with GOK plans and priorities;  Define, verify and track progress on agreed timelines, roles and responsibilities for the implementation of all activities;  Provide overall guidance, manage and coordinate implementation of all activities;  Provide approval/clearance for all work programs and activities, deliverables, milestones and changes in project scope;  Ensure appropriate and effective accountability mechanisms are in place for all parties;  Monitor and ensure reporting on progress toward targets and expenditures in a timely and transparent manner;  Develop and steer long-term sustainability strategies for the NHIS;  Provide an avenue through which major project-related conflicts and disagreements can be resolved. Full-time staff for the committee can keep its work proceeding in between its inter-agency meetings. At a more formal level, the coordination should include, among others, representatives from at least: Preventive Services; Health Standards, Quality Assurance and Regulation; Clinical Services; Administration Services; Health Informatics Monitoring & Evaluation; MOH PPP Unit; and the Council of CECs for Health. This oversight function will be more vital in the years ahead than it was in the past, to address the increasingly difficult issues of integration and interoperable procedures, data definitions and database design and use. If properly structured, this function can shift to a more ‘government-led’ and ‘government-owned’ process for improving the NHIS. 23 IBTCI: Mid-Term Performance Review of AfyaInfo ANNEX 1: AFYAINFO CONSORTIUM, STAKEHOLDERS AND PARTNERS AFYAINFO Consortium AFYAINFO Partnership Key Contributions Relationship Abt. Associates Overall project management, client and stakeholder relations, strategic leadership and technical direction of all project output areas and subcontractors AFYAINFO Prime Implementing Partner (international) Centers for Health Solutions (CHS) Development, testing, and implementation of HIS training, quality assurance, and mentorship program AFYAINFO Partner (local): mainly focusing on output 2 activities (human capacity) ICF International Integration of community-level data into the NHIS; management of the KePMS and support of PEPFAR partners in using the KePMS for reporting SAPRs and APRs until a unified NHIS is fully functional, and the KePMS is transitioned into it AFYAINFO Partner (international): mainly focusing on Output 1 activities (CHIS, DQA, and KePMS) Kenya Medical Training College (KMTC) Review of KMTC Health Records and Information Officers certificate and diploma curricula (conduct Training Needs Assessment, review curricula, revise curricula, and disseminate to internal and external stakeholders) AFYAINFO Partner (local): mainly focusing on Output 2 activities (pre-service training - HRIOs certificate and diploma curriculums) Knowing Limited Development and maintenance of the NHIS’s web-based MFL/MCUL databases AFYAINFO Partner (local): mainly focusing on Output 1 activities (IT/software, systems development) Training Resources Group (TRG) Strengthening of DivHIME’s leadership, management, and coordination structures (national, county) AFYAINFO Partner (international): mainly focusing on Output 3 activities (organizational development) University of Nairobi (UoN) Assistance to the MOH in training local partners, based on their competitive advantages, to become providers for NHIS maintenance, development, and training. UoN is also temporarily hosting the servers until the data is transitioned into a cloud AFYAINFO Partner (local): mainly focusing on Output 1 activities (systems support, maintenance, and development) 24 IBTCI: Mid-Term Performance Review of AfyaInfo AFYAINFO Stakeholders and Partners Partner/ Stakeholder Relationship Activity Start Date Output Division of Health Information, Monitoring and Evaluation (DivHIME) Primary Stakeholder Output 1: HIS infrastructure and systems development to create a unified NHIS; Output 2: Strengthen the MOH’s Learning and Knowledge Management structures; Output 3: Building capacity of DivHIME’s leadership and management necessary to drive NHIS unification effort (national, county) and sustain it over time June 2011 Output 1, Office of the Director of Medical Services Stakeholder Support to Dept. ICT, DivHIME, regulatory bodies, Community Health Services Unit, HSSF Unit, Division of Health Standards and Quality Assurance, and Malaria Control Program June 2011 Output 1, 2 and 3 Office of the Director of Public Health and Sanitation Stakeholder Support to the Dept. ICT, DivHIME, regulatory bodies, Community Health Services Unit, NSSF Unit, Division of Health Standards and Quality Assurance, and Malaria Control Program June 2011 Output 1, 2 and 3 County Departments of Health Stakeholder NHIS infrastructure deployment; NHIS trainings for CHMTs, data managers, county community health services focal persons, County HIS Capacity Building Assessment March 2013 Output 1, 2 and 3 AIDS, Population, and Health Integrated Assistance (APHIA+) USG Implementing Partners Harmonization of support for HMIS work plans and activities among USG partners June 2012 Output 1, 2, and 3 Department of Information and Communication Technology (Dept. ICT) Stakeholder HIS infrastructure and systems development to create a unified NHIS June 2011 Output 1 25 IBTCI: Mid-Term Performance Review of AfyaInfo Division of Health Standards and Quality Assurance Stakeholder Building of MFL Regulatory Module and integrate with regulatory databases December 2012 Output 1 Community Health Services Unit Stakeholder Integration of community-level data into the NHIS June 2011 Output 1 Health Sector Service Fund (HSSF) Stakeholder Upgrading of the MOH FIS and integration of it into the NHIS; conducted systems assessment October 2012 Output 1 CDC Emory USG Implementing Partner Building of MFL Regulatory Module and integration of it with regulatory databases; conduct systems assessment October 2012 Output 1 Malaria Control Program Stakeholder Systems assessment October 2012 Output 1 Kenya Medical Supplies Agency Stakeholder Systems assessment October 2012 Output 1 (KEMSA) World Health Organization (WHO) Development Partner Updating of MFL Health Facility Geocodes Using SARAM Data August 2013 Output 1 PEPFAR Implementing Partners USG Implementing Partners Management of KePMS and support of PEPFAR partners in using it for SAPR and APR until unified NHIS is fully functional, and KePMS is transitioned into it June 2011 Output 1 Health Regulatory Boards and Councils Stakeholder Building of MFL Regulatory Module and integration of it with regulatory databases including Pharmacist and Poisons Board, Kenya Medical Laboratories Technicians and Technologists Board, Medical Practitioners and Dentist Board, Clinical Officers Council, Nursing Council of Kenya, Radiation Protection Board December 2012 Output 1 National AIDS Control Council (NACC) Stakeholder Integration of community-level data into the NHIS June 2011 Output 1 Fanikisha USG Implementing Partner Integration of community-level data into the NHIS June 2011 Output 1 26 IBTCI: Mid-Term Performance Review of AfyaInfo Kenya Medical Training College (KMTC) Stakeholder Review of KMTC Health Records and Information Officers certificate and diploma curricula (conduct Training Needs Assessment, review curricula, revise curricula, and disseminate to internal and external stakeholders) June 2012 Output 2 Kenyatta University Stakeholder Review of Kenyatta University’s BSc Health Records and Information Management curriculum (conduct institutional assessment and market survey, draft faculty capacity development strategy and infrastructure improvement plan, undertake curriculum design and development, and disseminate to internal and external stakeholders) December 2012 Output 2 Human Resources Developme nt Division Stakeholder Implement a system for HIS training tracking; Conduct NHIS trainings (national, subnational levels) using the standard NHIS training materials; Develop HIS Capacity Building Plan January 2013 Output 2 HSSF Stakeholder DHIS2 training for county HSSF accountants January 2013 Output 2 Funzo Kenya USG Implementing partner Institutionalization of HIS trainings through regional hubs June 2013 Output 2 27 IBTCI: Mid-Term Performance Review of AfyaInfo Measure Evaluation USG Implementing Partner Prioritize, map and harmonize CHIS indicators in DHIS2 June 2013 Output 2 Christian Health Association of Kenya (CHAK) Stakeholder HIS training using standardized curriculum; DHIS2 and MFL national TOTs training January 2013 Output 2 Kenya Conference of Catholic Bishops (KCCB), formerly Kenya Episcopal Conference (KEC) Stakeholder HIS training using standardized curriculum; DHIS2 and MFL national TOTs training; DHIS2 and MFL refresher training January 2013 Output 2 Internation al Center for AIDS Care and Treatment and Programs (ICAP) Stakeholder DHIS2 and MFL refresher training March 2012 Output 2 28 IBTCI: Mid-Term Performance Review of AfyaInfo Human Resources Developme nt Division Stakeholder Develop and roll-out DivHIME’s training programs (short, medium, long-term); HIS training committee meetings February 2012 Output 3 Capacity Kenya USG Implementing Partner Working with other USAID partners (Capacity Kenya) develop a roll out plan to address sector capacity needs October 2012 Output 3 29 IBTCI: Mid-Term Performance Review of AfyaInfo ANNEX 2: SCOPE OF WORK SCOPE OF WORK MID TERM REVIEW: AFYAINFO PROJECT May 12, 2014 30 IBTCI: Mid-Term Performance Review of AfyaInfo PROJECT DETAILS: Project Name: AfyaInfo Project Implementing Partner: Abt Associates Inc. Contract Number: IQC #: AID – GHH – 1 – 00 – 07 – 00064 TASK ORDER #: AID – 623 – TO – 11- 00005 Project COR: Washington Omwomo Life of the Project: July 1, 2011 – June 30, 2016 Total Funding: $ 32,802,647 Million Period of Project to be evaluated: June 1, 2011 – May 31, 2014 Type of Evaluation: Performance Evaluation (Mid-Term Review) Completed Evaluation by: IBTCI ATTACHMENT: Annex I-Checklist for Assessing Evaluation Reports A. Background/Project Overview/Problem Statement: The AfyaInfo project supports the GOK to design and build a single unified and web￾based health information system. The project further supports GOK in institutionalizing data quality assurance practices at all levels, resulting in a high degree of completeness and accuracy of program data at health facility and community levels. The complete and accurate program data will inform program planning and decision making; ultimately helping to improve the health of Kenyans. It is however important to note that the project has been very slow in its implementation of specifically tasks 1 and 2 activities. The determination of the causal factors for this slow implementation will provide information especially on engagement approaches with government on such projects in future. A.1.1. Program Goal The strategic goal of AfyaInfo project is aligned to the USAID Implementation Framework 2010 - 2015 with the goal “sustained improvement of health and well-being for all Kenyans”. Specifically, this project responds to Result Area 2: Health Systems Strengthened for Sustainable Delivery of Quality Services (USAD/Kenya Implementation Framework 2010 – 2015). This activity contributes to the strategic objective and the strategic goal for the entire health portfolio as shown in the results framework below. 31 IBTCI: Mid-Term Performance Review of AfyaInfo Results Framework Strategic Goal: Sustained improvement of health and well-being for all Kenyans Strategic Objective: Improved health outcomes and impact through sustainable country-led programs and partnerships Result 1: Strengthened leadership, management and governance for sustained health programs Result 2: Health systems strengthened for sustainable delivery of quality services Result 3: Increased use of quality health services, products and information Result 4: Socialdeterminants of health addressed to improve well￾being of targeted communities and populations Cross-Cutting Elements Whole Market Innovation Gender-Focus Youth-Focus Equity The project specific results are detailed out on the contract, for more information check the activity contract. A.1.2. Program Objective The objective of AfyaInfo project is to support the GOK/Ministry of Health to develop, implement and manage a single, integrated national web-based national health information system (NHIS). A.1.3 Program Activities The program activities are organized in the form of tasks and sub-tasks with specific deliverables, acceptable quality and performance standards expressed for every task. Task 1: Establish a strong, unified and integrated web-based host country owned and managed national health information system that generates quality data used at all levels to improve health service delivery, with a 100% coverage of the counties and at least 80% coverage of health facilities and 80% coverage of community units in every county, by September 2013. 32 IBTCI: Mid-Term Performance Review of AfyaInfo Sub-tasks (Specific Activities): • Conduct a comprehensive systems requirements analysis and produce a costed requirements analysis plan. • Establish an IT infrastructure (hardware, software and user technical services including policies and protocols) capable of supporting development, deployment and maintenance of a unified and integrated web-based national health information system. • Take up management of Kenya Program Monitoring System (KePMS) and support USG PEPFAR partners in using this system for reporting Semi Annual Program Results (SAPR) and Annual Program Results (APR) results until such a time that a one unified and integrated web-based host country national health information system is fully functional, and KePMS is transitioned into it. • Integrate community health information system, Community Based Program Activity Report (COBPAR system) and KePMS into one strong unified and integrated web￾based national health information system. • Establish a functional national data warehouse (databank) with the appropriate data storage capacity, data confidentiality and data security for every user type. • Develop appropriate performance indicators for assessing the progress. Task 2: Based on the geographic coverage of task 1 (100% coverage of counties), establish a functional Government of Kenya (GOK)-managed learning and knowledge management system that improves the culture of information generation, knowledge capturing and information use by September 2015. Sub-task (Specific Activities): • Develop GOK-managed learning and knowledge management system for the health sector. • Conduct training needs assessment for MoH staff on management of learning and knowledge management system, produce training needs assessment report. • Conduct capacity building (including trainings on specific technical areas) programs to develop institutional and human capacity to launch and manage the learning and knowledge management agenda in the health sector. • Develop a range of appropriate information products, create demand for these products and establish relevant public awareness and dissemination forums and systems to ensure use of these information products. 33 IBTCI: Mid-Term Performance Review of AfyaInfo • Develop appropriate performance indicators and benchmarks for assessing the progress. Task 3: Establish a functional HMIS division that is capable of passing a USAID pre-award responsibility determination assessment on leadership and management, financial and procurement capability. Sub-tasks (Specific Activities); • Develop and implement appropriate capacity building programs to strengthen management and coordination structures based on already existing policies and governance structures. • Develop and implement appropriate capacity building programs to strengthen financial, technical and human resources management systems. • Develop appropriate performance indicators for assessing the yearly progress. A.2. PROBLEM STATEMENT: In August 2010, USAID/Kenya conducted a comprehensive assessment of the National M&E system and National Health Information System. The assessment was meant to help in designing a mechanism that would work with the Ministry of Health to address the identified gaps. The assessment key findings, conclusions and recommendations were organized around four thematic areas: Data Collection, Quality, and Access; Technology, Processes, Protocols, and the Human Interface; Policy and Organizational Development, and Management; and Information Products, Data use, and Knowledge Management. Please refer to the report: USAID/Kenya: Assessment of National M&E and National HMIS, August 2010 for a detailed description of the problem statement to which this project was designed to respond. A.2.1. Development Hypothesis If the Government of Kenya (GOK)/Ministry of Health develops, implements and manages a single, integrated web-based national health information system (NHIS), then there will be increased use of strategic information for program management, policy-making and decision making in the Kenyan health sector. 34 IBTCI: Mid-Term Performance Review of AfyaInfo B. STATEMENT OF WORK B.1. Mid-Term Review (MTR) Purpose The overall purpose of the mid-term review is to assess progress made in implementation of this contract in the three task areas listed above in A.1.3. Specifically, the MTR seeks to determine the factors that have facilitated or hindered the implementation of planned activities in the first 3 years. The review will further explore strategies for sustaining the key results at the national, county, sub-county and at health facility levels. B.2. Audience and Intended Use The primary audience for the evaluation is US Government Agencies (DOD, CDC and USAID) and the Government of Kenya. Secondarily, the report is intended for Abt Associates, USAID/Washington and other interested implementing partners (NGOs). It is expected that the results of this MTR will inform decisions on 1) the number of counties that the project should cover with IT infrastructure, 2) the number of counties to be covered with technical assistance on informational system organizational capacity development, 3) the appropriate approaches to implementing activities under this contract, and 4) the appropriate sustainability strategies that would ensure long-term use of the national integrated national information system. These programmatic and management decisions will be used to guide the implementation of project activities in the remaining years of project lifecycle. B.3. MTR Objectives and Key Review Questions B.3.1. The overall objective for this MTR is to: 1. Assess if the project is on-track to achieving expected results by finding out; a. What is working well? What does the project needs to do more? b. What is not working well? What does the project needs to do less? 2. Identify the strategies that national and county governments can put in place to sustain the use of NHIS. 3. Provide recommendations and directions related to each of the above that will be translated into a concrete action plan by the project management; USAID and GOK. Specific objectives: 1. Review the project’s implementation strategy/approaches for every task and their appropriateness in strengthening the national Health information System. 35 IBTCI: Mid-Term Performance Review of AfyaInfo 2. Review the projects achievements towards targeted key milestones, with emphasis on the acceptable quality and performance standards as stipulated in all the three tasks on the contract. 3. Identify possible constraints that have contributed to the slow implementation rate of the key project deliverables. 4. Explore with key stakeholders, including the national and county governments, the sustainability strategies for the national health information systems and its supportive sub-systems achieved through the project. 5. Propose recommendations for future programmatic directions for the project including specific strategies on key areas of performance concerns. B.3.2 MTR Key Questions The review seeks to answer the following questions; 1. What progress has the project made in addressing the key challenges and gaps of National Health Information System (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009 - 2014)? 2. What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. 3. What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support MOH in sustaining the use of NHIS? 4. Based on the key findings and conclusions drawn from questions 1 – 3, what are the key recommendations on the strategic programmatic and management directions that the Mission should consider for the mid-course changes on the contract? 36 IBTCI: Mid-Term Performance Review of AfyaInfo B.4 MTR Design and Data Collection Methods The Contractor is expected to review and refine the proposed MTR design (methodology and data collection methods) including the type of evidence and data analysis methods on the “Matrix: Illustrative Methodological Approaches for Consideration by the MTR team” as included in Section B.4.5 in this SOW and submit to USAID as part of the proposal for approval. It is expected that the refined methodology/tools will generate the highest quality and most credible evidence that corresponds to the questions being asked. It is expected to propose sound social science practices and tools, and explain to the extent possible how the proposed methods/tools would minimize the evaluator biases. As much as possible, MTR team will be provided with all key project documents and national HIS policy/guidelines to give a firm background. It is expected that a review of these documents will help the team in forming unbiased opinions in the course the process. Key project documents will include but not limited to: 1. HMIS Project Contract 2. Project annual work plans to date 3. Project PMP/work plan matrices 4. Quarterly Reports, Annual reports, other deliverables in the form of reports that are relevant 5. Reports/deliverables from consultants, TDYs from home office 6. National HIS Policy 7. Health Strategic Plan for HIS 2009 - 2014 8. USAID/Kenya: Assessment of National M&E and National HMIS 9. USAD/Kenya Implementation Framework 2010 – 2015 B.4.1 Proposed MTR Design As part of mid-term performance review process, the Contractor shall consider the following information in its design: A mixed method of different qualitative designs is suggested. Desk reviews of key project documents and relevant GOK/HIS policies and guidelines; roundtable discussions, policy dialogue and expert consultation forums on key thematic areas will be the main approaches. As much as possible these meetings/roundtable discussions will be held in either IBTCI and/or GOK-run conference institutions such as Kenya School of Government and/or Kenya School of Monetary Studies. The MTR team shall review and refine the proposed data collection methods, type of evidence and data analysis methods; and submit to USAID as part of the proposal for approval. 37 IBTCI: Mid-Term Performance Review of AfyaInfo B.4.2 Data Collection Methods 1) The team will review and analyze the key project documents: contract, work plans, quarterly progress reports, annual reports, key and relevant deliverables by outputs. This will be supplemented by virtual communication and brief technical meetings between the team members and USAID/Kenya Office of Population and Health (OPH) to help prepare them for working together. 2) Conduct Key Informant Interviews (KIIs): • USAID/Kenya OPH technical staff • Directorate of Policy Planning, M&E and Health Informatics (DPP, M&E/HI) • Division of M&E & Health Informatics (DM&E/HI) • County Health Executives, County M&E/HIS lead persons • Health Systems IPs (ITEC & FUTURES) • Umbrella Mechanisms (CHAK & KEC) • Service Delivery Partners (2 CDC-funded, 2 USAID-funded and 1 DOD) • Development Partners supporting HIS work (CHAI, DANIDA, WHO, CDC) • MOH/National Program (NASCOP, DOMC, Department of Family Health) • MOH HQ ( HRD, ICT, Standards & Regulatory, HSSF) • Regulatory Boards/Council (KMPDB, Nursing Council, Pharmacy) • KEMRI/CDC 3) HIS Expert Roundtable Discussions on: 1) National and County Governments Ownership of the Integrated National Health Information System; 2) Sustainable Solutions to Managing Integrated Systems Evolutions; and 3) Industry experts from national schools of computing and health informatics (University of Nairobi & Strathmore). 4) Roundtable Consultations/Discussions with M&E/HMIS advisors, lead persons from national implementation mechanisms, service delivery mechanisms, and development partners. Focus on implementation approaches, strategies for strengthening coordination/collaboration engagements and sustainability of NHIS infrastructure. 5) Roundtable Policy Dialogue with national and county governments on the sustainability of the national health information systems and its related sub-systems 38 IBTCI: Mid-Term Performance Review of AfyaInfo delivered by the project. 6) A Systems Review of the District Health Information System and its sub-systems such as health commodities, CHIS, and COBPAR and assess the completeness and accuracy of the data being reported. 7) Organize one national key findings, conclusions and recommendations validation forum. A selection of key HIS stakeholders will come together to validate the key findings, conclusions and recommendations from the review for developing consensus for the key strategic shifts if there are any for the project. B.4.3 Data Analysis Methods The MTR team will review all performance management information available, and all reports produced by the many contractors and TDYs from the project’s home office. All relevant national HIS policy and guidelines shall also be reviewed by the team to contribute information for triangulation. Detailed synthesis and analysis is expected of the MTR team to support the key findings, conclusions and recommendations that would come out of the review. Content, comparative and analytical analysis techniques of the qualitative data from key informant interviews, roundtable discussions, expert consultations, and roundtable policy dialogue are some of the suggested analysis methods. Use of basic statistics to present data on graphs and charts is also expected. An analysis of data from data quality audits conducted by the national government, the project and/or implementing partners is expected to demonstrate the progress made on the overall completeness and accuracy of data coming out of DHIS. The MTR team is advised that USAID’s information quality standards in ADS 578 apply to this review. Data should be disaggregated by gender at the outcome and output levels, and geographic area (province and county; urban vs. rural; facility vs. community) where appropriate. B.4.4 Methodological Strengths and Limitations USAID’s Evaluation Policy states that any methodological strengths and limitations are to be communicated explicitly in SOWs. Some examples of methodological strengths and limitations include: • Strengths:  Roundtable approach combines strategic and practical questions and would help MTR team to explore and determine most appropriate responses for the questions.  Roundtable approach is a proven methodology for engaging with the policy makers in discussing issues that require high level policy decision making in a timely fashion. 39 IBTCI: Mid-Term Performance Review of AfyaInfo • Limitations:  Short implementation timelines not allowing for data collection at health facility level on key areas of focus.  The level change and stability of national GOK/Ministry of Health and County Health teams is likely to make roundtable discussions not very productive. 40 IBTCI: Mid-Term Performance Review of AfyaInfo Illustrative Methodological Approaches for Consideration by the MTR team Key Evaluation Question Type of Evidence Methods Source Sampling/ Selection Data analysis 1.What progress has the project made in addressing the key challenges and gaps of National Health Information System (as outlined in contract document); to be able to support health care delivery at facility and community levels Comparative/ Analytic Desk reviews Data Abstraction KII Roundtable Discussions • Project’s M&E system • Contract, Work Plans, Quarterly/Annual Progress Reports, • Project Staff • Health Informatics & M&E Division, Purposive / Systematic Random Purposive / Systematic Random Trend analysis on reported results against targets on the Acceptable Quality & Performance Standards Content analysis to understand challenges in meeting targets and revisions to targets. Content analysis on causal factors on agreements/work plans discrepancies, adoption of new evidence/strategies Directorate Policy Planning, M&E & Health Informatics, CHMTs Content analysis on strategic approaches, what works, what doesn’t work, mid-course revisions for better performance 2. What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. Comparative/ Analytic Desk Reviews Roundtable Discussions Roundtable expert • Project documents, • DPP, M&E & HI; DM&E, HI, CHMTs • NASCOP, DOMC, DRH • University of Purposive Random Purposive Content analysis to understand implementation challenges, categorize by implementation approaches and management systems related. Content analysis to identify strategies to overcome the implementation challenges. Content analysis of interviews data to consultations, SSI/KII Strathmore University • Project Staff management effectiveness, challenges, opportunities and threats 41 IBTCI: Mid-Term Performance Review of AfyaInfo 3. What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the key results? Comparative/ Analytic Roundtable Policy Dialogue, KII • DPP, M&E and HI, DDM&E, HI, County Health Executives, Purposive Analyze proposed sustainability strategies by the National Government, County Governments, Local Training Institutions, Development Partners and Implementing partners. 4. What role can development partners play to support MOH in sustaining these investments? Comparative/ Analytic Comparative/ Roundtable expert consultations, KII KII Roundtable consultations, KII • Local Universities (UON, Strathmore) • Chief of Party (ITEC, FUTURES) ICAP, CHAK, Kenya Catholics Secretariat (KCS), AMREF, ITEC, Purposive Triangulation of sustainability strategies by different categories, development of advocacy strategies on how engage national and county governments on implementation of sustainability models proposed. Analytic FUTURES, A+ (2) Roundtable discussions • WHO, DANIDA, CDC, Content analysis of interviews to develop sustainability strategies, advocacy strategies 5. What are the key recommendations on the strategic programmatic and management directionsthat the Mission should consider for the mid-course changes on the contract? Comparative/ Analytic MTR Team Data Synthesis Meetings • Raw and synthesized data from the triangulated different data sources Purposive Triangulated, well-balanced and practical recommendations on programmatic and management issues. 42 IBTCI: Mid-Term Performance Review of AfyaInfo B.4.5. KEY PERSONNEL MTR Evaluation Team Composition The two-person team shall include: 1) a Senior Expatriate Monitoring and Evaluation Specialist, as defined in the IQC, with strong expertise in evaluating health systems, specifically information systems and who will act as the team leader; and 2) a Senior Local Monitoring and Evaluation Specialist, as defined in the IQC, specializing in health systems, health information systems, and M&E. The entire team must be external to USAID and all team members shall be required to provide a written disclosure of conflicts of interest. Team leader must be an expatriate while the other member must be a Kenyan citizen. Evaluation Management IBTCI will provide overall direction to the MTR team; avail all the key project documents, provide all the logistical support required to perform this MTR. IBTCI/MTR team shall be responsible for arranging all roundtable discussions, Key Informant Interviews (KII) and booking meeting places. IBTCI is responsible for quality control and delivery of the required report as agreed to by USAID. IBTCI shall be responsible for arranging all domestic travel and hotel arrangements for the selected county health executives listed below. Selected Counties The following are the selected counties from which county health executives will be invited: Region Counties Rift Valley Uasin Gishu Kericho Western Kakamega Busia Nyanza Homa Bay Siaya Coast Kwale Kilifi Central Nyeri Kirinyaga Eastern Machakos Kitui Nairobi Nairobi North Eastern Mandera 43 IBTCI: Mid-Term Performance Review of AfyaInfo While subject to change with the acceptance by both parties, it is envisioned that all MTR team members will be in Kenya the entire duration of the evaluation’s in-country component. The team leader will be provided a total of up to a maximum of 5 days, in addition to the 4 weeks of fieldwork, in the U.S. to ensure the completion and transmission of the final report as well as the closure of any outstanding matters. C. PERIOD OF PERFORMANCE The period of performance for the MTR is scheduled for June, 2014. The MTR will take a maximum of four weeks. An additional five (5) days will be added to the period of performance to complete the final report after the MTR is complete. The total period of performance for both the MTR and the final report is 35 days. D. DELIVERABLES D.1.Specific Deliverables IBTCI will provide tentative dates and timelines for the following deliverables to be approved by USAID: 1. Briefings: The MTR team will provide two in-country briefings to USAID/Kenya, the first one at entry and second at exit. Additional bi-weekly debriefings will be provided by the IBTCI team to USAID team. At the exit briefing, the evaluators will make an in-country presentation to USAID on the main findings at the end of the in-country reviews and analysis. 2. Work plan: The MTR team will provide a detailed work plan to USAID at the conclusion of the MTR Planning Meetings (MPMs) and before commencing the evaluation work. The work plan will outline the technical approaches that will be undertaken and the methods to be used. It will be approved by USAID before the MTR fieldwork starts. 3. Proposal/Methodology: The methodology for collecting and analyzing the data will be prepared before and/or during the MPM, and approved by USAID before commencing the MTR fieldwork. Illustrative methods have been proposed. 4. Final debriefing: A detailed written outline will be submitted and an oral debriefing will occur reporting to USAID/Kenya on the conclusions that are strongly supported with well-grounded findings and practical, evidenced based recommendations that are linked to the every conclusion. 44 IBTCI: Mid-Term Performance Review of AfyaInfo 5. Draft Report: The first draft of the MTR report will be submitted before the MTR team departs Kenya and after the final debriefing so that comments can be incorporated into the said report. 6. Final Report: The final MTR report will be submitted to USAID/Kenya using single￾line spacing, font size 11, and not more than 25 pages within 5 working days after the MPR team receives comments from USAID/Kenya. 7. Upon final approval of the content by USAID/Kenya, IBTCI will have the report edited and formatted. The final report will be submitted both electronically and in hard copy. Four hard copies of the report will be provided to USAID/Kenya. In addition, all the raw data will be submitted to USAID on CD labeled “AfyaInfo MTR Raw Data” for future reference. Once USAID approves the final report, IBTCI will submit it and all the MTR-related information products to the Development Experience Clearinghouse (DEC) as provided for in the IQC. 8. Organize one national dissemination forum to present key and finalized findings, conclusions, recommendations and strategic directional shifts (if any) to the national key stakeholders. D.2.Final Report Format The format for the MTR Final Report required by D.1.6 above shall be a maximum of 25 pages not including annexes. The report format should be restricted to Microsoft products and 12-point font should be used throughout the body of the report, with 1” page margins. Four bound hard copies shall be submitted, and an electronic copy in MS Word. In addition, all data collected by the evaluation will be provided to USAID in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the project or the evaluation. If the report contains any potentially procurement sensitive information, a second version report excluding this information will be submitted (also electronically, in English) for dissemination among stakeholders and on the DEC. IBTCI is responsible for ensuring that the final evaluation report includes all criteria listed in Appendix 1 of USAID’s Evaluation Policy: 1. Executive Summary—concisely state the most salient findings and recommendations (3 pg); 2. Table of Contents (1 pg); 3. Introduction—purpose, audience, and synopsis of task (1 pg); 4. Background—brief overview of development problem, USAID project strategy and activities implemented to address the problem, and purpose of the evaluation (2-3 pg); 45 IBTCI: Mid-Term Performance Review of AfyaInfo 5. Methodology—describe evaluation methods, including constraints and gaps (1 pg); 6. Findings/Conclusions/Recommendations—for each evaluation question (10-13 pg); 7. Issues—provide a list of key technical and/or administrative, if any (1pg); 8. Future Directions (1 - 2pg); 9. Annexes —that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. These include references to bibliographical documentation, meetings, interviews and focus group discussions. D.3. QUALITY EVALUATION REPORT IBTCI is expected to review USAID’s requirements and expectations on the draft and final reports as detailed on the “Checklist for Assessing Evaluation Reports”, see Annex I. It is important to note that USAID will subject the structure and content of the report to the parameters outlined on the checklist and will use this as a basis for accepting and/or rejecting the reports. D.4 THREATS TO VALIDITY IBTCI is required to manage the MTR team and guard against any possible threats to validity of findings, conclusions, and recommendations drawn from the qualitative methods. Any conclusion drawn from the qualitative data sources must be supported by well-grounded body of evidence that is triangulated and confirmed. It is therefore expected that IBTCI will take the MTR team through the parameters outlined on the USAID’s “Checklist for Reducing Threats to Validity for Qualitative Methods”. E.PROPOSED COST • The proposed magnitude for this SOW is between $100,000 – 150,000. 46 IBTCI: Mid-Term Performance Review of AfyaInfo Annex I: Checklist for Assessing Evaluation Reports EVALUATION REPORT CHECKLIST - V1.0 Title of Study Being Reviewed: Main Implementer(s): Reviewer: Date of Review: GOOD PRACTICE ELEMENTS OF AN EVALUATION REPORTi Keyed to USAID’s 2011 Evaluation Policy EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments STRUCTURE OF THE REPORT 1. Does the evaluation report have a cover sheet attached indicating the type of evaluation conducted (e.g. performance evaluation or impact evaluation) and general design? 2. If a performance evaluation, does the evaluation report focus on descriptive and normative evaluation questions? 3. If the evaluation report uses the term “impact evaluation,” is it defined as measuring the change in a development outcome that is attributable to a defined intervention (i.e. impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual)? 4. Regardless of the type of evaluation, does the evaluation report reflect use of sound social science methods? 5. Does the report have a Table of Contents (TOC)? 6. Do Lists of Figures and Tables follow the TOC? 7. Does the report have a Glossary of Terms? 7.1 Are abbreviations limited to the essential? 8. Is the date of the report given? 9. Does the body of the report adhere to the 20 page guide? 10. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 11. Does the report’s presentation highlight important information in ways that capture the reader’s attention? 12. Is the report well written (clear sentences, reasonable length paragraphs, no typos, acceptable for dissemination to potential users)? 13. Does the evaluation report focus on the essential issues concerning the key 46 IBTCI: Mid-Term Performance Review of AfyaInfo EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments questions, and eliminate the “nice to know”, but not essential information? 14. Does the evaluation report discuss any issues of conflict of interest, including the lack thereof? 15. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? EXECUTIVESUMMARY 16. Does the evaluation report begin with a 3- to 5-page stand-alone summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? 17. Does the Executive Summary concisely state the main points of the evaluation? 18. Does the Executive Summary follow the rule of only saying what the evaluation itself says and not introducing new material? INTRODUCTION 19. Does the report introduction adequately describe the project? 19.1. Does the introduction explain the problem/opportunity the project was trying to address? 19.2. Does the introduction show where the project was implemented (physical location) through a map? 19.3. Does the introduction explain when the project was implemented? 19.4. Are the “theory of change” or development hypotheses that underlie the project explained? (Does the report specify the project’s inputs, direct results (outputs), and higher level outcomes and impacts, so that the reader understands the logical structure of the project and what it was supposed to accomplish?) 19.5. Does the report identify assumptions underlying the project? 19.6. Does the report include sufficient local and global contextual information so that the external validity and relevance of the evaluation can be assessed? 19.7. Does the evaluation report identify and describe any critical competitors to the project that functioned at the same time and in the project’s environment? 19.8. Is USAID’s level of investment in the project stated? 19.9. Does the evaluation report describe the project components funded by implementing partners and the amount of funding? 20. Is the purpose of the evaluation clearly stated? 21. Is the amount of USAID funding for the evaluation indicated? 22. Are all other sources of funding for the evaluation indicated as well as the amounts? 23. Does the report identify the evaluation team members and any partners in the evaluation? 24. Is there a clear statement of how the evaluation will be used and who the intended users are? 25. Are the priority evaluation questions presented in the introduction? 26. Does the evaluation address all evaluation questions included in the Statement of Work (SOW)? 26.1. Are any modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline indicated in the report? 26.2. Is the SOW presented as an annex? 26.3. If so, does the annex include the rationale for any change with the 47 IBTCI: Mid-Term Performance Review of AfyaInfo EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments written sign-offs on the changes by the technical officer? SCOPE AND METHODOLOGY 27. Does the report provide a clear description of the evaluation’s design? 27.1. Is a design matrix or similar written tool presented in an annex that shows for each question/subquestion the measure(s) or indicator(s) used to address it, the source(s) of the information, the type of evaluation design, type of sampling if used, data collection instrument(s) used, and the data analysis plan? 28. Does the report state the period over which the evaluation was conducted? 29. Does the report state the project time span (reference period) covered by the evaluation? 30. Does the evaluation report indicate the nature and extent of consultation on the evaluation design with in-country partners and beneficiaries? 31. Does the evaluation report indicate the nature and extent of participation by national counterparts and evaluators in the design and conduct of the evaluation? 32. Does the report address each key question around which the evaluation was designed? 33. Is at least one of the evaluation questions directly related to gender analysis of outcomes and impacts? 33.1. Are data sex-disaggregated? 34. In answering the questions, does the report appropriately use comparisons made against baseline data? 35. If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness? 35.1. As appropriate, does the report include financial data that permits computation of unit costs and analysis of cost structure? 36. Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an annex)? 36.1. Are all tools (questionnaires, checklists, discussion guides, and other data collection instruments) used in the evaluation provided in an annex? 36.2. Does the evaluation report include information, as appropriate, on the pilot testing of data collection instruments? 36.3. Does the evaluation report include information, as appropriate, on the training of data collectors? 37. Are all sources of information properly identified and listed in an annex? 38. Does the evaluation report contain a section describing the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups,small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc)? 39. Does the evaluation report indicate the evaluation methodology took into account the time, budget, and other practical considerations for the evaluation such as minimizing disruption and data burden? 40. Does the report have sufficient information to determine if the evaluation team had the appropriate methodological and subject matter expertise to conduct the evaluation as designed? 41. If an impact evaluation was designed and conducted, does the evaluation report indicate that experimental methods were used to generate the strongest evidence? Or does the report indicate that alternative methods for assessing impact were utilized and present the reasons why random assignment strategies were not feasible? 48 IBTCI: Mid-Term Performance Review of AfyaInfo EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments 42. Does the evaluation report reflect the application and use to the maximum extent possible of social science methods and tools that reduce the need for evaluator-specific judgments? 43. Does the evaluation scope and methodology section address generalizability of the findings? ANALYSIS 44. Are percentages, ratios, cross-tabulations, rather than raw data presented, as appropriate? 45. When percentages are given, does the report always indicate the number of cases used to calculate the percentage? 45.1. Is use of percentages avoided when the number of cases is small (<10)? 46. Are whole numbers used or rounding-off numbers to 1 or 2 digits? 47. Are pictures used to good effect? 47.1. Relevant to the content 47.2. Called out in the text and placed near the call-out 48. Are charts and graphs used to present or summarize data, where relevant? 48.1. Are the graphics easy to read and simple enough to communicate the message without much text? 48.2. Are they consistently numbered and titled? 48.3. Are they clearly labeled (axis, legend, etc.) 48.4. Is the source of the data identified? 48.5. Are they called out in the text and correctly placed near the call-out? 48.6. Are the scales honest (proportional and not misleading by virtue of being “blown-up”)? FINDINGS 49. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 49.1. As appropriate, does the report indicate confirmatory evidence for FINDINGS from multiple sources, data collection methods, and analytic procedures? 50. Are adequate data provided to address the validity of the “theory of change” or development hypothesis underlying the project, i.e., cause and effect relationships? 51. Are alternative explanations of any observed results discussed, if found? 52. Are unplanned results the team discovered adequately described? 53. Are opinions, conclusions, and recommendations kept out of the description of FINDINGS? CONCLUSIONS 54. Is there a clear distinction between CONCLUSIONS and FINDINGS? 55. Is every CONCLUSION in the report supported by a specific or clearly defined set of FINDINGS? 56. Are the CONCLUSIONS credible, given the FINDINGS the report presents? 57. Can the reader tell what CONCLUSIONS the evaluation team reached on each evaluation question? RECOMMENDATIONS 58. Are RECOMMENDATIONS separated from CONCLUSIONS? (Are they highlighted, presented in a separate section or otherwise marked so that the reader sees them as being distinct?) 59. Are all RECOMMENDATIONS supported by a specific or clearly defined set of FINDINGS and CONCLUSIONS? (Clearly derived from what the evaluation team learned?) 49 IBTCI: Mid-Term Performance Review of AfyaInfo EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments 60. Are the RECOMMENDATIONS practical and specific? 61. Are the RECOMMENDATIONS responsive to the purpose of the evaluation? 62. Are the RECOMMENDATIONS action-oriented? 63. Is it clear who is responsible for each action? 64. Are the RECOMMENDATIONS limited/grouped into a reasonable number? LESSONSLEARNED 65. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 66. Are the LESSONS LEARNED highlighted and presented in a clear way? 67. Does the report indicate who the lessons are for? (e.g., project implementation team, future project, USAID and implementing partners, etc.) BOTTOM LINE 68. Does the evaluation report give the appearance of a thoughtful, evidence￾based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 69. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? 70. Is the evaluation report structured in a way that will promote its utilization? 71. Does the evaluation report explicitly link the evaluation questions to specific future decisions to be made by USAID leadership, partner governments and/or other key stakeholders? 72. Does the evaluation report convey the sense that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? REPORTDISSEMINATION 73. Have all evaluation team members signed a statement attesting to a lack of conflict of interest, or describing and existing conflict of interest relative to the project being evaluated? 74. Was the Report Submitted to the Development Experience Clearing House (DEC)? 75. Has a dissemination plan been developed for this report? 76. Is the report widely shared to interested stakeholders? DEFINITIONS: Performance evaluation: focuses on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. Impact evaluation: measures the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a ―treatment‖ or a ―control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. 50 IBTCI: Mid-Term Performance Review of AfyaInfo Theory of change: A tool to design and evaluate social change initiatives. It is a blueprint of the building blocks needed to achieve long-term goals of a social change initiative. Development Hypothesis: Identifies causal linkages between USAID actions and the intended Strategic Objective (highest level result). External Validity: The degree to which findings, conclusions, and recommendations produced by an evaluation are applicable to other settings and contexts. Findings: Empirical facts collected during the evaluation Conclusions: Interpretations and judgments based on the findings Recommendations: Proposed actionsfor management. i In addition to the USAID 2011 Evaluation Policy, good practices in evaluation reporting have also been drawn from: Morra Imas, Linda and Ray C. Rist. 2009. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC.: The World Bank. Scriven, Michael. 2005. Key Evaluation Checklist. Stufflebeam, Daniel L. 1999. Program Evaluations Metaevaluation Checklist. 51 IBTCI: Mid-Term Performance Review of AfyaInfo ANNEX 3: CONSULTANT CVS TARIQUL KHAN TEAM LEADER SUMMARY OF QUALIFICATIONS Decisive, action-oriented, and results-focused team leader, program/project manager, and international development expert with over nineteen years of experience. Senior advisor, technical expert, and World Bank program/project manager and mission leader to a number of developing countries. Solid operational experience and expertise in: sustainable strategic approach, planning and processes for ‘country-owned’ and ‘country-led’ development solutions; multi-donor/stakeholders initiatives; Public health policy, strategy, planning, service delivery, systems strengthening, and information system (HIS); sectoral & multi-sector program/project design and performance assessment; PPP; performance management and evaluation; capacity assessment and building; institutional readiness and strengthening for development; strategic and management information; results-based management (RBM), strategic planning & management, implementation design, and monitoring and evaluation (M&E); statistical/development data collection, management, analysis and dissemination. Recipient of numerous awards in recognition of outstanding performances, leadership, effective stakeholder/partnership buildup and management, commitment to high quality work and deadlines, strong client, negotiating, diplomatic, and team skills. Proven record of flexibility, integrative and creative thinking, multitasking and mentoring abilities, exceptional work ethic, high personal integrity, and recognized ability to work as a leader and member in complex, multi-cultural team environments. In addition to over 4 years as a senior advisor/consultant/team leader working for USAID, NGOs, and Private Sector, 15 years of progressive work experience (including leadership & management) at the World Bank. EDUCATION • The Johns Hopkins University, School of Advanced International Studies, Washington D.C., U.S.A. Master of International Public Policy | GPA 3.8 • The George Washington University, Washington D.C., U.S.A. Project Management, Survey of Economics | GPA 4.0 • University of the District of Columbia, Washington D.C., U.S.A. B.S. in Computer Science (minor: Mathematics); Summa Cum Laude | GPA 3.9 • Southeastern University, Washington D.C., U.S.A. Computer Information Systems | GPA 3.6 WORK HISTORY CONSULTANCY & ADVISORY WORK: SEPTEMBER, 2008 TO PRESENT • Team Leader: Community Care (HIV) and Orphan & Vulnerable Children (OVC) Program Design; Country Owned Program Design Specialist; GH TECH Bridge (USAID/Malawi Office of Public Health), USA • Team Leader: Development of USAID/K Office of Public Health (OPH)’s M&E Strategy and Performance Management Plan (PMP); IT Shows, Inc. (USAID/Kenya Office of Public Health), USA 52 IBTCI: Mid-Term Performance Review of AfyaInfo • Team Leader: Nutrition and HIV/AIDS Program; Country Owned Program Design Specialist; IT Shows, Inc. (USAID/Kenya Office of Public Health), USA • Team Leader: Assessment of National Health Information System (HIS), Namibia; GH Tech Bridge Project, Development & Training Services, Inc. (USAID/Namibia Office of Public Health), USA • Team Leader: In-Country Assessment & Country-Owned & Led Program Design for National Health Service Delivery; IT Shows, Inc. (USAID/Kenya Office of Public Health), USA • Lead Consultant : Country Owned Development Strategy and Institutional Capacity Assessment/strengthening Expert; Bureau for Global Health, USAID/Washington, USA • Lead Consultant : Country Owned and Led Approach for HMIS Strengthening; Bureau for Global Health, USAID/Washington and USAID/Nigeria, USA • Team Leader : In-Country Assessment & Country-Led Program Design Specialist; IT Shows, Inc. (USAID/Kenya Office of Public Health), USA • Team Leader : Health Policy, Financing, and Private Sector Assessment and Country Owned Program Design Specialist; IT Shows, Inc. (USAID/Kenya Office of Public Health), USA • Lead Consultant : M&E and HMIS Expert: Country Owned and Led Program Design; CAMRIS International, Inc. (Global Health Technical Assistance Project, USAID), USA • Team Leader & Senior Advisor (International): Strategic Planning & Results; Onneyshan (NGO: poverty, gender and human rights), Dhaka, Bangladesh INTERNATIONAL CIVIL SERVANT: THE WORLD BANK, WASHINGTON D.C., U.S.A. 1993 - 2008 Positions: • Team Leader & Senior Operations Officer; - Team Leader & Senior Information Officer • Team Leader & Information Officer; - Information Analyst; - Projects Assistant RELATED PROFESSIONAL EXPERIENCE USAID/KENYA (OFFICE OF PUBLIC HEALTH) • Designed and Developed OPH’s multi-year M&E Strategy and Performance Management Plan (PMP), aligned with both USAID and GOK’s strategic plans and priorities • Conducted Rapid Assessment, Designed and Developed the five year Program for the Nutrition and HIV/AIDS Program focused on country-owned and country-led principles • Designed and developed a Guideline & Toolkit for development and organization of in￾country Assessment, Recommendations, and country-owned Program Design for USAID Support • Conducted assessment of and developed recommendations for the Kenya National Health Policy/Financing/Private Sector environment • Designed and Developed five year Program Descriptions (to be resulted into an RFP) for USAID support for the Kenya National Health Policy/Financing/Private Sector Program focused on country-owned and country-led principles 53 IBTCI: Mid-Term Performance Review of AfyaInfo USAID (GLOBAL HEALTH TECH PROJECT) • Conducted Rapid Assessment, Designed and Developed the five year Program for USAID/Malawi Community Care (PLHIV) and OVC Program focused on country-owned and country-led principles • Conducted assessment of and developed system wide recommendations for the Namibia National Health Information Systems (HIS) • Conducted assessment of and developed recommendations for the Kenya National Monitoring and Evaluation (M&E) and Health Management Information Systems (HMIS) • Designed and Developed a five year Program Descriptions (resulting into an RFA) for USAID support for the Kenya National Health Sector M&E Program and National Health Management Information Systems (HMIS) Program focused on country-owned and country￾led principles. THE WORLD BANK • Multi-disciplinary team leader and Program/Project Manager for over the ten years providing overall strategic direction and management for teams of professionals ensuring that the portfolios, both in the headquarter and on the ground, were managed in a sound and effective manner and the various projects and programs achieved their stated goals/results • Led, advised, trained, and provided hands-on support to other team leaders, task managers, various sectoral, and multi-sectoral teams (including health) in the World Bank and in developing countries in planning, strategy formulation, partnership formulation and stakeholder management, change management and organizational development, program design, implementation and improved operational effectiveness – all with a clear focus on results and results based Monitoring and Evaluation (M&E); resulting in many individual program and project plans, Country Assistance Strategies (CAS), Regional Strategies and Corporate Action Plans, and Poverty Reduction Strategies (PRS) with results-based strategic designs and management, a clearer definition of desired outcomes, and more focused implementation of projects. • Managed the Africa Regional (53 countries and 20 sub-regional groups) Data Program and Center operations including providing expert oversight to the African regional development data and processes for decision making, publications / public access, and lead a Live Database team including staffs and consultants for numerous years to support 47 operations country teams which resulted in significantly improved quality, access, and timely availability of data • Lead new operations to support multi-donor, sector-wide approaches to statistical capacity building and enhancement in several developing countries • Managed a number of new Trust Funds to strengthen statistical capacity in a number of African countries to improve tracking of development results • Presented/promoted a new approach to high officials concerning data collection, production and mining; spearheaded a team for installation and training of a statistical system in various African ministries, several regional and international development institutions and development banks resulting in improved measuring and monitoring of results, significant time-saving in finding data and substantially lowered the costs of producing major institutional publications and other economic reports 54 IBTCI: Mid-Term Performance Review of AfyaInfo • Routinely mentored and coached staff on career path and professional development both within and outside of own team. USA EXPORT INC. • Advise and provide hands-on support to high level staff to modify organizational strategic business goals, restructure and improve areas of operations and approaches, and develop outcomes and results targets, and identify indicators to monitor targets/results. • Lead a team and provided overall guidance in formulation and ongoing implementation of the five years Business Plan; administer results-based management as well as operation and monitoring. • Recommend and propose on existing monitoring and assessments of business projects and investment areas, along with recalibration of measures to maintain viability and profitability. COUNTRY EXPERIENCE WORK Bangladesh, Iran and Saudi Arabia, Botswana, Cote d'Ivoire, Ethiopia, Ghana, Kenya, Malawi, Mozambique, Namibia, Nigeria, South Africa, Sudan, Swaziland, Tanzania, and Zambia.United States RACHEAL MACHARIA TEAM MEMBER EXPERIENCE 2012 May –August Africa USAID East Africa Conducting an End of Project Evaluation for a health project in seven countries in Africa (Burundi, Djibouti, Kenya, Mozambique, Rwanda, Uganda and Zambia) Design of question guide, data collection, data analysis and report writing 2012 April- May Kenya Kimetrica Research and developing a training module for defining and organizing project results (target setting, understanding logical frameworks) 2012 Mar -Present UNV Online Nigeria Chairing the Monitoring an Evaluation committee Guide the overall Monitoring & Evaluation (M&E) strategy and implementation of activities within the organization Assist in setting up M&E systems and ensuring it is implemented effectively 2011 Jun - Dec Kimetrica Kenya 55 IBTCI: Mid-Term Performance Review of AfyaInfo Information management for an M&E software through the design of project management , monitoring and evaluation components Analysis and write up of statistical tabulations and cross tabulations 2011 Jan – Feb USAID- Office of Population and Health Kenya Reviewed existing Health Management Information Systems/Monitoring and Evaluation tools for GOK (MoMS & MoPHS) and USAID APHIA II implementing partners programs Developed and designed program management M&E tools and formats for USAID APHIAplus programs 2010 Sept- Nov Kimetrica Kenya Evaluated the usability of an M&E software system Developed training materials for Results based M&E Facilitated a training of trainers workshop on Results based M&E for users from Somalia and Ethiopia 2010 Apr-Aug ICF MACRO USAID APHIA II Evaluation Kenya KePMS data review, cleaning and system update Conducted Indicator review and pilot testing of the National HIV/AIDS Community based reporting tool Planned and coordinated two national workshops in liaison with the Ministry of Health and development partners. 2009-2010 Koka Koimburi USAID National M&E Program Kenya Monitored and conducted spot checks of the Post Enumeration Survey enumerator and supervisor trainings and data collection that was conducted by the Kenya National Bureau of Statistics (KNBS) Provided technical support to strengthening the Community M&E system of the National Aids Control Council (NACC), through meetings, capacity assessments and field trips Organized for a workshop for the review of the National COBPAR (Community Based Program Activities Reporting) tool Monitored three Demographic Surveillance Sites on program planning, implementation and reporting 2004-2006 Catholic Relief Services Kenya Monitoring & Evaluation officer Designed Management of Information System for a USAID fund Child survival Grant project. Designed monitoring tools and conducted Qualitative and Quantitative evaluation studies for a USAID funded Integrated Health and Agriculture project. 56 IBTCI: Mid-Term Performance Review of AfyaInfo Designed a template using EPI Info for health facility data entry and analysis Conducted quarterly field visits to project areas in Tana River, Mbeere, and Suba Provided feedback and mentoring after reviewing partners quarterly reports Conducted monitoring and evaluation trainings for our partners from which the partners left with monitoring plans as outputs. IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 57 ANNEX 4: SIGNED ATTESTATIONS OF NON-CONFLICT OF INTEREST Name Tariqul Khan Title Senior Expatriate M&E Specialist - Consultant Organization International Business and Technical Consultants Inc. Evaluation Position? Team Leader Team member Evaluation Award Number(contract or other instrument) ESPS Task Order #4 USAID Project(s) Evaluated AfyaInfo implemented by Abt. Associates Contract No: AID￾623-TO-11-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12/1/2014 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 58 Name Rachel Macharia Title Senior Local M&E Specialist - Consultant Organization International Business and Technical Consultants Inc. Evaluation Position? T eam Leader Team member Evaluation Award Number ESPS Task Order #4 USAID Project(s) Evaluated AfyaInfo implemented by Abt. Associates Contract No: AID- 623- TO11-00005 I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 12/2/2014 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 59 ANNEX 5. DETAILED METHODOLOGY Purpose of this Mid-term Performance Review (MTR) During fist three years of the activity implementation key tasks 1 and 2 were characterized with slow performance and therefore USAID/Kenya/OPH commissioned a MTR to determine the factors that have facilitated or hindered the implementation of planned activities in the first three years and explore strategies for sustaining the key results at the national, county, sub-county and at health facility levels. The primary audience for this MTR is US Government Agencies (DOD, CDC and USAID) and the Government of Kenya. Secondarily, the report is intended for Abt. Associates, USAID/Washington and other interested implementing partners. Time Period of MTR The MTR work commenced on July 24, 2014. This included a home-based desk review phase and an in￾country field phase (August 26 to September 12). The USAID In-Brief occurred on August 20, with the Mid-Brief on September 09, and the Out-Brief on September 17. The draft report was submitted on November 12. The evaluation was conducted in Nairobi – within the USAID’s ESPS Office. Stakeholders and Selection Method The MTR team has identified multiple stakeholders from which respondents for this MTR were selected. All stakeholders were mapped into the following nine groups: 1. AFYAINFO Resource Partners (e.g. Abt. Associates, KMTC, ICF International, Knowing Limited, Training Resource Group, University of Nairobi) 2. Central Government Institutions (e.g. DHIM&E) 3. Regional Government Representatives (CHEO and SCHRIO from 14 counties) 4. Governmental and Quasi-Governmental Institutions (e.g. Pharmacy and Poisons Board, Kenya Medical Practitioners and Dentists Board, Nursing Council of Kenya) 5. National MOH Programs (e.g. NASCOP, DFH, NACC) 6. Development Partners (WHO, CDC, DANIDA and CHAI) 7. Other USG IPs Supporting Kenya Health System (e.g. CAPACITY, ITEC, FUTURES) 8. Umbrella Mechanisms (e.g. Christian Health Association of Kenya) 9. USG Funded Program Service Delivery Partners (e.g. APHIAPlus, CDC Emory) Broken down further: Resource Partners: Abt. Associates (Prime), Center for Health Solutions (CHS), ICF International; Kenya Medical Training Collage (KMTC), Knowing Limited (KL), University of Nairobi (UN) and Training Resource Group (TRG). Central Governmental Institutions: Directorate of Policy and Planning, M&E and Health Informatics (DPP, M&E/HI), Division of M&E and Health Informatics (DM&E/HI), Office of the Director of Medical Services (DMS), Office of Director of Public Health and Sanitation (DPH&S), Department of Information and Communication Technology (DICT), Division of Health Standards and Quality Assurance (DHS&QA), Division of Human Resource Development (DHRD) and Health Sector Service Fund (HSSF). Regional Government: County Health Executive Officers (CHEO) and Sub-County Health Records and Information Officers (SCHRIO)42 of the Usin Gishu, Kericho, Kakamega, Busia, Homa Bay, Siaya, Kwale, Kilifi, Nyeri, Kirinyaga, Machakos, Kitui, Nairobi and Mandera counties. 42 Inclusion of SCHRIOs is highly recommended in the list of stakeholders and is proposed by the ESPS team. IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 60 Governmental and Quasi-Governmental Institutions: Pharmacy and Poisons Board (PPB), Kenya Medical Laboratory Technicians & Technologists Board (KMLTTB), Kenya Medical Practitioners and Dentists Board (MPDB), Clinical Officers Council (COC), Nursing Council of Kenya (NCK) and Radiation Protection Board (RPB). National Programs of the MOH: National AIDS and STI Control Program (NASCOP), Department of Family Health (DFH), National AIDs Control Council (NACC) and Division of Community Health Services (CHS). Development Partners Supporting HIS: WHO, CDC, DANIDA and CHAI. Implementing Partners Supporting Kenya Health System: ITEC and FUTURES. The representatives of this organizations are also members of the System Enhancement Committee and ME Subcommittee established by the AfyaInfo. Umbrella Mechanisms: Kenya Conference of Catholic Bishops (KCCB) and Christian Health Association of Kenya (CHAK). USG Funded Program Service Delivery Partners: APHIAplus, AMPATHPlus and Emory. Due to the limited number of organizations unified in each target group randomization of organizations was not applicable. The evaluation team used a census approach to ensure that all organizations are involved in MTR. Data Collection Approaches The MTR team applied a mixed methods approach for the MTR including document review; HISs review, RGDs, FGDs, SGDs, KIIs and quantitative data review collected within the framework of AfyaInfo by the activity prime implementing partner and outsourced companies and organizations. The methods were used linked to, and assist evaluation team to answer, the key questions and sub￾questions (please see at the end of this narrative a Data Collection Matrix). The applied methodology based on the SOW as well as on broad context of health information system development in Kenya and the specific context in which AfyaInfo operates. Document Review: The document review provided the MTR team with background information on AfyaInfo’s key tasks and corresponding specific activities over time and serve as a major source to shape the approach of the MTR. Specific ways in which the document review supported subsequent components of the MTR includes: drafting of data collection guides, determining the characteristics and adequacy of existing lists of beneficiaries (to ensure homogeneity of RGD and FGD participants) and finalize the MTR Implementation Work Plan. RGDs, FGDs and SGDs: The MTR team organized and facilitated five RGDs and four FGDs with between three to nine participants and seven SGDs with between two to seven participants. In total through group discussions the evaluation team interviewed 86 participants. The group discussions were conducted with AfyaInfo stakeholders involved in the development of unified NHIS. More specifically: (i) MOH senior officials representing of, the: Directorate of Policy and Planning, (DPP), Division of Monitoring and Evaluation & Health Informatics (DME/HI), Office of the Director of Medical Services (DMS), Office of Director of Public Health and Sanitation (DPHS), Department of Information and Communication Technology (DICT), Division of Health Standards and Quality Assurance (DHSQA), Division of Human Resource Development (DHRD) and Health Sector Service Fund (HSSF); (ii) MOH officials representing of national health programs: National AIDS and STI Control Program (NASCOP), Department of Family Health (DFH), National AIDs Control Council (NACC) and Division of Community Health Services (DCHS). For group discussions, the availability of respondents was a challenge, despite repeated and follow-up telephone calls conducted by the ESPS. This was due to people traveling from regional locations, and IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 61 the busy nature of high-level stakeholders. For persons traveling from a distance, ESPS organized the logistics and paid for travel, lodging and per diems according to policies and protocols established by USAID. The group discussions provided an opportunity for the MTR team to meet with a greater number of AfyaInfo’s consortium member organizations, USG funded service delivery partners and county health executives and Sub-county data managers. It also gave the MTR team comparative perspectives on AfyaInfo to date. The RGDs and FGDs were organized by the ESPS team and was held at IBTCI office in Nairobi, Kenya. To keep participants engaged throughout the entire session the ESPS organize substantial tea for participants during 30 minute break. The SGDs were organized by the ESPS team and was held at the MOH, CHAK, UoN, WHO and ICF. Key Informant Interviews: The selection of the KIIs was somewhat complex, due to the multifaceted nature of AfyaInfo. The MTR team conducted 22 face-to-face in-depth KIIs. The MTR team conducted six KIIs, two with former and four with current heads of the divisions and departments of the MOH; seven KIIs with Abt. Associates and two with consortium member organizations (KL and TRG) representatives; two with USAID/Kenya and one KII per each development partner (DANIDA, CDC and CHAI); one with KeMSA and one with KNBS. The respondents of the KIIs were selected purposively. Data Collection Steps: The MTR team applied a two-step approach of data collection for this review. Step I: During the initial phase of data collection, the MTR team conducted 16 group discussions with AfyaInfo all stakeholders. The MTR team was assisted by two note-takers. The attitudes, perceptions and opinions of group discussion participants were document by two note-takers attending each individual group discussion. As an end-product, within 24 hours since completion of group discussion, the MTR team received two individual reports per each group discussion. Step II: The MTR team collected significant information on AfyaInfo’s achievements, challenges and trends from step 1 and therefore, Step II was focused on in-depth investigations in particular areas of AfyaInfo interventions which may not have been readily evident through data collected during group discussions. The MTR team used one standardized guides for RGDs and FGDs. To conduct in-depth interviews with key informants the MTR team developed a KII guide. One unified guide of KII was developed for all target groups. Data Analysis Prior to processing the data, the MTR team reviewed collected data to ensure all findings are properly recorded and documented. In this regard, after completion of each RGD/FGD session each member of the MTR team reviewed notes captured by two note-takers and cross-checked through the MTR team’s notes. MTR team analyzed data collected using a process in which quantitative and qualitative data analysis strategies are connected to determine and understand key findings. In this way, the analysis of data collected from each method is conducted independently. Once the analysis was completed for each method, the MTR team checked how the analysis and findings of each method did inform and strengthen the other. This method produced conclusions to inform on-going and future programming. Limitations The breakdown by stakeholder RGD/FGDs was as follows: 29% of targeted MOH health department representatives participated; 83% of regulatory bodies (1 missed); 50% of County Health Executive Officers (8 missed); 86% of Sub-County Health Records Information Officers; 88% of USAID SDPs (1 missed); 86% of resource partners (1 not based in Nairobi, therefore, telephone interview was conducted). IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 62 To ensure a representative coverage of the MOH respondents, time period allocated to conduct all RGD/FGDs and KIIs (12 calendar day), was extended to 16 days. Within additional four days, the ESPS contacted ‘missed’ individuals for the opportunity to attend SGDs. As a result of this, the MTR team conducted three SGDs and interviewed seven representatives of MOH HIS Unit and two representatives of Division of Nutrition. In addition the MTR team organized SGDs with: UON (4 participants), CHAK (3 participants), MEDS (3 participants), WHO (2 participants) and IFC (2 participants). IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 63 DATA COLLECTION MATRIX REVIEW KEY QUESTION 1: What progress has the project made in addressing the key challenges and gaps of National Health Information System (as outlined in Project Contract, National HIS Policy and Health Sector Strategic Plan for HIS 2009 - 2014)? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH DATA ANALYSIS SOURCE METHOD METHOD 1.1 To what extent is the project succeeding in establishing a unified, web￾based NHIS producing and promoting the use of quality data? Descriptive & Analytical DPP, M&E/HI, ICTD, NACC, KeMSA, CHSU, SCHRIOs, RPs & USAID IPs Document review RGD, FGD & KII, Systems review Purposive, As appropriate Trend and overall progress in establishment of unified system, its effectiveness for multiple stakeholders 1.1.1What is the progress in integrating all health service information systems such as disease surveillance, CHIS, MIAS, COBPAR, KePMS and various Dept. of FH systems into the NHIS? Descriptive & Analytical DPP, M&E/HI, ICTD, NACC, KeMSA, CHSU, SCHRIOs, RPs & USAID IPs Document review RGD, FGD & KII, Systems review Purposive. As appropriate Trend and overall progress in establishment of unified system, its effectiveness for multiple stakeholders 1.1.2 What is the progress in integrating management information such as commodities, finance and HRIS into the NHIS? Analytical DPP, M&E/HI, ICTD, NACC, KeMSA, CHSU, SCHRIOs, RPs & USAID IPs Document review RGD, FGD & KII, Systems review Purposive, As appropriate Trend and overall progress in establishment of unified system, its effectiveness for multiple stakeholders 1.1.3 What is the progress in integrating key (summarized) population based statistics (vital statistics, surveys, census, i.e. KNBS data) into the NHIS? Analytical DPP, M&E/HI, ICTD, NACC, KeMSA, CHSU, SCHRIOs, RPs & USAID IPs, KNBS Document review RGD, FGD & KII, Systems review Purposive, As appropriate Trend and overall progress in establishment of unified system, its effectiveness for multiple stakeholders 1.1.4 Is there an active and effective linkage/coordination between NHIS & NIMES? If so, how? If not, what are the impediments? Analytical DPP, M&E/HI, ICTD, RPs, MDP(NIMES) Document review RGD, FGD & KII, Systems review Purpose, As appropriate Trend and overall progress in establishment of unified system, its effectiveness for multiple stakeholders IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 64 1.1.5 How effective has the project been in the interim management of the KePMS and supporting the PEPFAR partners SAPR and APR requirement? Analytical USAID IPs, DPP, M&E/HI & RPs Document Review, RGD, FGD & KII Purposive Trend and overall progress 1.1.6 Is the overall data quality of NHIS and its major components satisfactory for policy discussions/formulation and decision-making process (both for management and patient services)? Analytical USAID IPs, DPP, M&E/HI & RPs Document review RGD, FGD & KII, Systems review Purposive Analysis on methods used by the project to obtain, verify and report on key performance indicator data 1.1.6.1 How reliable is the key performance indicator (milestone) data reported by the project? Please describe with examples and evidences Descriptive & Analytical RPs, M&E/HI Document review As appropriate Analysis on methods used by the project to obtain, verify and report on key performance indicator data 1.1.6.2 How consistent and concurrent is the DHIS data with other key systems including the KePMS? Analytical USAID IPs, DPP, M&E/HI & RPs FGD,RGD & Document Review Purposive As appropriate Content analysis and comparative analysis of baseline findings with follow up DQA (currently in progress) 1.1.7 What is the current status of ICT infrastructure development at the central, county and sub-county levels towards having the capability to support development, deployment and maintenance of the NHIS? Descriptive & Analytical DPP, M&E/HI, ICTD, CHEOs & SCHRIOs RGD, FGD & KII Purposive Content analysis on HIS infrastructure developed at the central, county and sub-county levels and challenges 1.1.8 What is the current status of the Master Facility List (MFL)/MCUL? How often is MFL/MCUL updated? Descriptive & Analytical WHO, HS&QA Division, Health RBs and Councils & RPs RGD, FGD & KII, Systems Review Purposive Fact-based judgmental and content analysis on progress 1.1.10 Has the MFL/MCUL regulatory module developed, geocodes updated and integrated in regulatory database? If not, why? Descriptive & Analytical WHO, HS&QA Division, Health RBs and Councils & RPs RGD,FGD & KII, Systems Review As appropriate Fact-based judgmental and content analysis on progress IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 65 1.1.11 What is the progress in improving collaboration and coordination among various project stakeholders? Descriptive All project stakeholders RGD,FGD & KII, Systems Review purposive As appropriate Content analysis on progress to improve collaboration/ coordination among stakeholders and challenges 1.1.11.1 Does the M&E/HID have better representation in the various key coordination frameworks? If not, why? Descriptive All project stakeholders RGD. FGD & KII Purposive Fact-based judgmental on practical examples (case-by-case) which proves better representation of HIMED in 1.2 What is the current progress in establishing of functional GOK managed Learning and Knowledge Management (LKM) system that is improving the culture of information generation, information dissemination and use? Descriptive & Analytical DPP, M&E/HI, HRD &RPs FGD, RGD & KII. Document Review Purposive ,As appropriate Trend and overall progress in establishment of LKM system and its effectiveness 1.2.1 To what extent and how have the TNA findings been addressed in the newly developed LKM system? Analytical HRD, HSSF, KMTC, KU,RPs, CHEOs, SCHIROs & USAID IPs RGD, FGD & KII Purposive Content analysis on targeting and training needs at the central and county levels 1.2.2 What and how effective have the capacity building initiatives launched by the project been (both institutional and human)?Explain Analytical HRD, HSSF, KMTC, KU, RPs, CHEOs, SCHIROs & USAID IPs RGD, FGD & KII Purposive Content analysis on effectiveness of different training methods (short-medium-long) 1.2.3 Is the TWG focused on LKM functional? If so, how effective is it? Descriptive HRD, HSSF, KMTC, KU, RPs, CHEOs, SCHIROs & USAID IPs RGD, FGD & KII Purposive Content analysis on progress of capacity building of health workers 1.2.4 Are the HIS trainings institutionalized? Analytical HRD, CHAK, KCCB, KEC & FUNZOKenya RGD, FGD & KII Purposive Content analysis on development and roll-out of standardized curriculums by the training institutions IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 66 1.2.5 What and how frequently are information products being produced and disseminated by the LKM system for various stakeholders at different levels to help improve overall operation, planning, decision making and performance management Analytical DPP, M&E/HI, HRD & RPs RGD, FGD & KII Purposive Content analysis on progress achieved by projects 1.2.6 What specific activities/ opportunities provided by the project have promoted the importance of and raised awareness for a culture shift and strengthen the demand for and use of information at all levels Analytical DPP, M&E/HI, HRD & RPs RGD, FGD & KII Purposive Content analysis on progress achieved by projects 1.3 How effective has the project been in strengthening the M&E/HID toward its establishment as a functional and effective organizational entity capable of developing, implementing, and managing a unified and integrated NHIS? Analytical DPP, M&E/HI, HRD &RPs RGD, FGD, KII & Document review Purposive Content analysis on progress achieved by the project to strengthen the capacity of M&E/HID 1.3.1 To what extent the project activities and capacity building initiatives have helped strengthening the management and coordination structures, and the financial, technical, and HR management systems of the M&E/HID? Analytical DPP, M&E/HI, HRD &RPs RGD, FGD, KII & Document review Purposive Content analysis on progress achieved by the project to strengthen the capacity of M&E/HID 1.3.2 Overall, does M&E/HID have the required capacity to pass an institutional capacity assessment/audit? Why or why not? Analytical DPP, M&E/HI, HRD &RPs RGD,FGD,KII Purposive SWOT analysis on HIMED institutional capabilities IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 67 REVIEW KEY QUESTION 2: What are the main implementation challenges? Review the implementation approaches and the management systems in place and determine the extent to which they have affected implementation of key activities. REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH DATA ANALYSIS METHOD SOURCE METHOD 2.1 What components and project aspects are not working well and why? Analytical DPP, M&E/HI, HRD, RPs & USAID/Kenya RGD, FGD & KII Purposive Content analysis on the project implementation challenges 2.2 How effective has the method of implementation been so far? Should the project continue implementing interventions with the same approach and at the same level, or with changes during its final project years? Analytical DPP, M&E/HI, HRD, Health RBs, RPs & USAID/Kenya RGD, FGD & KII Purposive Analysis on adequacy of system adopted by the consortium members (planning, management, decision-making) 2.3 How has the policy environment (laws, regulations etc.) been conducive or non-conducive for the effective implementation of the project? Analytical DPP, M&E/HI, HRD, Health RBs, RPs & USAID/Kenya RGD, FGD & KII Purposive Analysis on external factors and threats and mitigation mechanisms adopted by the consortium? 2.4 If and how effective has the project been toward achieving the desired/required level of coordination with the GOK Institutions? If not, why? Analytical MOH QRs, DPP,M&E/HI, HRD, RPs & USAID/Kenya, RGD, FGD & KII Purposive Analysis on coordination mechanisms established by the project with GOK institutions 2.5 To what extend has the Project achieved the desired/ required level of coordination with other USG service delivery partners? Analytical CDC (2), USAID (2) & DOD (1), ITEC, Futures, RPs & USAID/Kenya RGD, FGD & KII Purposive Analysis on coordination mechanisms established by the project with USAID IPs 2.6 To what extent and how has the project been effective to increase the demand for and use of data and Analytical DMS, DPH&S, DPP, M&E/HI, HRD, RPs & USAID/Kenya RGD, FGD & KII Purposive Content analysis on current demand of data needs IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 68 information by decision-makers? REVIEW KEY QUESTION 3: What are some of the sustainability strategies that the national and county governments could use to ensure long-term use of the NHIS? What role can development partners play to support MOH in sustaining the use of NHIS? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH DATA ANALYSIS METHOD SOURCE METHOD 3.1 What new regulations/laws have been advocated by the Project to improve political feasibility? Descriptive MOH HQ, Health RBs & Councils, DPP, M&E/HI & RPs RGD, FGD & KII Purposive Content analysis on new regulations/ laws supported by the project 3.2 If and to what extent (including timeliness) the new regulations/laws were adopted by the ministries and other relevant government agencies? If not, what were the impediments to that? Descriptive MOH HQ, Health RBs & Councils, DPP, M&E/HI & RPs Purposive Analysis on process of adoption the new regulations/laws supported by the project 3.3 Have the newly adopted regulations and laws improved or hindered project performance? If so in which area(s), ways? Analytical MOH HQ, Health RBs & Councils, DPP, M&E/HI & RPs RGD, FGD & KII Purposive Content analysis on impact of adoption the new regulations/ laws on project performance 3.4 To what extent and how effectively has the MOH allocated match contributions according to the sustainability strategy (addendum to AWP PY1)? Descriptive MOH HQ, CHEOs, SCHRIOs & RPs RGD, FGD & KII As appropriate, Purposive Fact-based judgmental and trend analysis on co-financing of the project initiatives by GOK 3.5 Is there effective coordination with other ministries/departments where services or information is needed to fully implement the program and have an Analytical MOH HQ, DPP, M&E/HI & RPs, USAID IPs RGD, FGD & KII Purposive Analysis on process of adoption the new regulations/laws supported by the project IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 69 effective NHIS? 3.6 Is there increasing GOK resources for funding technology acquisition, supplies, and maintenance? If so, in which areas and to what extent? Descriptive & Analytical MOH HQ, DPP, M&E/HI, RPs, CHEOs & SCHRIOs RGD, FGD , KII, Document review As appropriate Fact-based judgmental and trend analysis on co-financing of the project initiatives by GOK 3.7 To what and extent and how efficiently is GOK absorbing the existing key NHIS staffs and/or hiring new ones? Descriptive M&E/HID, HRD, RPs, CHEOs & SCHRIOs RGD, FGD & KII Purposive Content analysis on staff health workers turn-over at the central, county and sub-county levels 3.8 Is there an effective plan and active effort for program specific parallel systems funded by the DPs to be dissolved and integrated with the NHIS? If so, which, how and what timeline? Descriptive MOH HQ, USAID/Kenya & RPs RGD, FGD & KII Purposive Feasibility analysis on parallel projects if any to identify linkages and improved impact 3.9 How effective is the coordination among the DPs for NHIS related activities and investments? Descriptive MOH HQ, USAID/Kenya & RPs RGD, FGD & KII Purposive Assess efficiency of coordination mechanism among DPs with similar or complementary objectives 3.10 What has been the role of the local universities and other private sector entities toward the sustainability, evolution, and maintenance of NHIS? Descriptive & Analytical UON, Strathmore &RPs RGD, FGD & KII Purposive, As appropriate Content analysis on role and level of involvement of local universities and relevant private sector representatives in sustaining and further evolution of NHIS 3.11 What is the level of involvement been by the county government to ensure sustainability, evolution and maintenance of NHIS? Analytical CHEOs, SCHRIOs, RPs & USAID IPs FGD & KII Purposive Content analysis on role and level of involvement of county government in sustaining and further evolution of NHIS IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 70 ANNEX 6: BIBLIOGRPAPHY 1. AfyaInfo List of systems, sub systems and stakeholders, June 2014 2. AfyaInfo Quarterly Progress Reports for Year1 3. AfyaInfo Quarterly Progress Reports for Year2 and Year3 4. AfyaInfo: YR1-YR3 PMP Tracking Matrix 5. AfyaInfo Annual Performance Report Year 2: 01 July 2012–30 June 2013, October 31, 2013 6. AfyaInfo: PowerPoint: The Methodology for Consistency and Concurrency of data in DHIS2 and KePMS-K2D, June 2014 7. AfyaInfo: Report on the Training Needs Assessment for the Kenya Medical Training College HRIO curriculum review, January 2013 8. AfyaInfo. K2D Baseline Report, September 2012 9. AfyaInfo (Div-HIS). Organizational Management and Assessment Report, November 2011 10. AfyaInfo. Mid Term Review: AfyaInfo Project Evaluator Briefing Book, June 2014 11. AfyaInfo. Paper Data Quality Audits 2010 versus DQA 2014 We may want to drop as AID provided Comment on pg. 12 MTR report 12. AfyaInfo. K2D Baseline Report, September 2012. 13. Kenya National Health Information System Project: AfyaInfo Work plan for Year 1, July 2011 14. Kenya National Health Information System Project: AfyaInfo Sustainability Strategy- for Year 1 Work plan Addendum, January 2011 15. Kenya National Health Information System Project: AfyaInfo Approved Work plan for Year 2 June 2012, 16. Kenya National Health Information System Project: AfyaInfo Revised Work plan for Year 3, February 2014 17. Kenya National Health Information System Project: AfyaInfo Year 1 Annual Performance Report (01July 2011-30 June 2012), October 2012 18. Kenya National Health Information System Project: AfyaInfo Year 2 Annual Performance Report (01July 2012-30 June 2013), October 2013 19. Kenya National Health Information Project: AfyaInfo Year 3 Quarter 2 (Y3Q2) Progress Report (01 October – 31 December 2012), January 2013 20. Kenya National Health Information Project: Year 3 Quarter 2 (Y3Q2) Progress Report (01 January – 31 March), January 2013 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 71 21. Kenya National Health Information Project: AfyaInfo. Infrastructure Deployment Framework, January 2014 22. Kenya National Health Information Project. Review of Kenya Health Information System Legal Framework. 2012 23. Kenya National Health Information Project: AfyaInfo Year 3 Quarter 1Progress Report (01July – 30 September 2012), October 2013 24. Ministry of Medical Services and Ministry of Public Health and Sanitation. Health Sector Health Information System Policy 2010 – 2030 25. Ministry of Medical Services and Ministry of Public Health and Sanitation. Kenya Health Policy 2012-2030 (final version) 26. Ministry of Medical Services and Ministry of Public Health and Sanitation. Health Sector HIS Strategic Plan for Health Information System 2009-2014 27. Ministry of Medical Services and Ministry of Public Health and Sanitation. Div HIS, National Data Quality Assessment Report, October 2010 28. Ministry of Medical Services and Ministry of Public Health and Sanitation. Div HIS, National Data Quality Assurance Protocol, October 2013 29. Ministry of Medical Services and Ministry of Public Health and Sanitation. Div-HIS & department of ICT Infrastructure Improvement Roadmap, February 2013 30. MOH and AfyaInfo: HIS Infrastructure Rollout Strategy, July 2014 31. Republic of Kenya. Draft Health Bill, 2014 32. Strathmore Business School Consultants: Training Needs Analysis of the DivHIS, January 2013 33. USAID/KENYA: Assessment of National Monitoring and Evaluation and Health Management Information Systems. August 2010 34. USAID East Africa. Signed Contract between USAID and ABT Associates, May 2011 35. USAID Kenya /ESPS Proposal for AfyaInfo Mid-Term Review. July 2014 36. USAID/MOH E-Bulletin, March 8, 2013. Vol. 2.8 37. International Business & Technical Consultants Inc., Evaluation Services and Program Support: Data Quality Assessment, Final Report May 8th, 2014 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 72 ANNEX 7. DATA COLLECTION TOOLS 1. Name and designation (Have an attendance list with Name, organization, designation, email address, telephone number, and gender) 2. RGD/ FGD for ___________________________________Duration of FGD/RGD ___________ 3. Gender # of male__________ # of female _____________ Key Reminders to the Facilitator/interviewer: 1. The key is to facilitate and lead rather than direct. 2. The purpose is to get EVERYONE involved in conversation and participate in the discussion. 3. Maintain a non-judgmental approach to participants and their viewpoints. 4. Try to solicit input from less vocal members (if some people just do not want to talk, that’s okay). 5. Questions requiring opinions and judgments should follow factual questions, after some level of trust has been established and the atmosphere is more conducive to candid replies. 6. Although we will have interviewer questions lined up and in a certain order, let’s not be afraid to deviate. It is entirely possible that a person may start talking and end up answering any number of questions without specifically being asked. It is also likely that someone may introduce a subject not included in the questions -- let them talk (within reason!). The whole point is to allow the person to tell their story, including their particular knowledge, opinions, and experiences. Give them the space to say what they need to say. If the person deviates completely from the topic, then do pull them back by referring to the questions. 7. Essential to create a safe environment! Before the meeting, list the following key ground rules on a flip chart: a. Maintain confidentiality b. Participate as much as possible c. Ask questions as they come up d. Turn off cell phones and pagers e. Respect other opinions f. Don’t interrupt; let others finish speaking before you begin Review these ground rules with the group and ask if there are any additional rules people would like to add. Facilitator’s welcome, introduction and instructions to participants Welcome and thank you for accepting the invitation to take part in this focus group. You have been asked to participate as your knowledge, experience, and point of view are important. I realize you are busy and I appreciate your time. Introduction: Introduce ourselves. As you are aware, we are conducting the mid-term evaluation of the AfyaInfo Project. We are very interested in learning about your perception and experience with the AfyaInfo Project including what has and what has not worked well, key activities, results, AFYAINFO: MID-TERM REVIEW FOCUS GROUP/ROUNDTABLE GROUP DISCUSSION (FGD/RGD) GUIDE - GENERIC IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 73 sustainability issues, and future direction relevant to the establishment of a unified and well integrated web-based NHIS. The focus group discussion will take no more than two hours. Ask participants to introduce themselves by stating their name and their background or relationship as it pertains to the NHIS. Consent/Anonymity: I would like to assure you that the discussion and release of information will be anonymous. The notes of the focus group will contain no information that would allow individual subjects to be linked to specific statements. You should try to answer and comment as accurately and truthfully as possible. Review Questions 1.1 To what extent is the project succeeding in establishing a unified and integrated web-based NHIS producing and promoting the use of quality data? 1.1.1 What is the progress in integrating all the key health service information systems such as disease surveillance, CHIS, MIAS, COBPAR, KePMS, and various Dept. of FH systems into the NHIS? 1.1.2 What is the progress in integrating management information such as commodities, finance, and HRIS into the NHIS? 1.1.3 What is the progress in integrating key (summarized) population based statistics (vital statistics, surveys, census, i.e. KNBS data) into the NHIS? 1.1.4 Is there an active and effective linkage/coordination between NHIS & NIMES? If so, how? If not, what are the impediments? 1.1.5 How effective has the project been in the interim management of the KePMS and supporting the PEPFAR partners SAPR and APR requirement? 1.1.6 Is the overall data quality of NHIS and its major components satisfactory for policy discussions/formulation and decision-making process (both for management and patient services)? 1.1.6.1 How reliable is the key performance indicator (milestone) data reported by the project? Please describe with examples and evidences. 1.1.6.2 How consistent and concurrent is the DHIS data with other key systems including the KePMS? 1.1.7 What is the current status of the ICT infrastructure development at the central, county and sub-county levels towards having the capability to support development, deployment, and maintenance of the NHIS? 1.1.8 What is the current status in development and enforcement of new policy guidelines and legal frameworks? 1.1.9 What is the current status of the Master Facility List (MFL)/MCUL? How often is MFL/MCFL updated? 1.1.10Has the MFL regulatory module developed, geocodes updated and integrated in the regulatory database? If not, why? 1.1.11 What is the progress in improving collaboration and coordination among various project stakeholders? 1.1.11.1Does the M&E/HID have better representation in the various key coordination frameworks? If not, why? 1.2 What is the current progress in establishing a functional GOK managed Learning and Knowledge Management (LKM) system that is improving the culture of information generation, dissemination, and use? 1.2.1 To what extent and how the TNA findings been addressed in the newly developed LKM system? 1.2.2 What and how effective have the capacity building initiatives launched by the project been (both institutional and human)? Explain. 1.2.3 Is the TWG focused on the LKM functional? If so, how effective is it? 1.2.4 Are the HIS trainings institutionalized? IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 74 1.2.5 What and how frequently information products are being produced and disseminated by the LKM system for various stakeholders at different levels to help improve overall operation, planning, decision making and performance management 1.2.6 What specific activities/opportunities provided by the project have promoted the importance of and raised awareness for a culture shift and strengthen the demand for and use of information at all levels 1.3 How effective has the project been in strengthening the M&E/HID toward its establishment as a functional and effective organizational entity capable of developing, implementing, and managing a unified and integrated NHIS? 1.3.1 To what extent the project activities and capacity building initiatives have helped strengthening the management and coordination structures, and the financial, technical, and HR management systems of the M&E/HID? Overall, does the M&E/HID have the required capacity to pass an institutional capacity assessment/audit? Why or why not? 2.1 What components and project aspects are not working well and why? 2.2 How effective has the method of implementation been so far? Should the project continue implementing interventions with the same approach and at the same level, or with changes during its final project years? 2.3 How has the policy environment (laws, regulations etc.) been conducive or non-conducive for the effective implementation of the project? 2.4 If and how effective has the project been toward achieving the desired/required level of coordination with the GOK Institutions? If not, why? 2.5 To what extend has the Project achieved the desired/ required level of coordination with other USG service delivery partners? 2.6 To what extent and how has the project been effective to increase the demand for and use of data and information by decision-makers? 3.1 What new regulations/laws have been advocated by the Project to improve political feasibility? 3.2 If and to what extent (including timeliness) the new regulations/laws were adopted by the ministries and other relevant government agencies? If not, what were the impediments to that? 3.3 Have the newly adopted regulations and laws improved or hindered project performance? If so in which area(s), ways? 3.4 To what extent and how effectively has the MOH allocated match contributions according to the sustainability strategy (addendum to AWP PY1)? 3.5 Is there effective coordination with other ministries/departments where services or information is needed to fully implement the program and have an effective NHIS? 3.6 Is there increasing GOK resources for funding technology acquisition, supplies, and maintenance? If so, in which areas and to what extent? 3.7 To what extent and how efficiently GOK is absorbing the existing key NHIS staffs and/or hiring new ones? 3.8 Is there an effective plan and active effort for program specific parallel systems funded by the DPs to be dissolved and integrated with the NHIS? If so, which, how and timeline? 3.9 How effective is the coordination among the DPs for NHIS related activities and investments? 3.10 What has been the role of the local universities and other private sector entities toward the sustainability, evolution, and maintenance of the NHIS? IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 75 3.11 What has the level of involvement been by the county government to ensure sustainability, evolution and maintenance of the NHIS? Key Overarching Strategic Questions: 1. In your opinion, what are some of the key strengths of the NHIS, especially resulted from the AfyaInfo project interventions (i.e., what’s working well?) 2. In your opinion, what are some of the key weaknesses or challenges of the system (i.e., what’s not working so well?) 3. What are the main implementation challenges? Are the current management systems adequate and effective? Why, please explain. 4. How would you describe the leadership and management of both GOK and the AfyaInfo project IPs in respect to the NHIS? 5. How is the policy environment conducive or not conducive to the successful and effective implementation of the project and the NHIS overall? 6. How effective have the performance indicators been in tracking results in the key areas of the project? 7. What are some of the priorities that should be pursued in both in the next two years and in the longer run to sustain current achievements and use of NHIS at all levels? 8. How or in what areas do you think AfyaInfo investment in the next two years can do the most good? Conclusion • Thank you for participating. This has been a very useful discussion • Your opinions will be a valuable asset to the review • I would like to remind you that any comments featuring in this report will be anonymous • Are there any questions? • Before you leave, please make sure you have completed the attendance sheet. • Reminder to the note taker: Please, write your report based on the results of the focus group. Please remember to maintain confidentiality of the participating individuals by not disclosing their names (indicate their organization instead). IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 76 1. Name and designation (Record interviewee(s) Name, organization, designation, email address, telephone number, and gender) 2. KII for ___________________________________Duration of Interview__________ 3. Gender # of male__________ # of female _____________ Key Reminders to the Facilitator/interviewer: 1. The key is to facilitate and lead rather than direct. 2. Begin the interview with a minute or two of general conversation. 3. The purpose is to get the person(s) involved in conversation and participate in the discussion. 4. Maintain a non-judgmental approach to the interviewee(s) and his/her viewpoints. 5. Questions requiring opinions and judgments should follow factual questions, after some level of trust has been established and the atmosphere is more conducive to candid replies. 6. Questions should be simply worded, kept short, and phrased in the vernacular. Generally, they should be phrased to elicit detailed information, not just a simple yes or no answer. 7. Although we will have interviewer questions lined up and in a certain order, let’s not be afraid to deviate. It is entirely possible that a person may start talking and end up answering any number of questions without specifically being asked. It is also likely that someone may introduce a subject not included in the questions -- let him/her talk (within reason!). The whole point is to allow the person to tell his/her story, including their particular knowledge, opinions, and experiences. Give them the space to say what they need to say. If the person deviates completely from the topic, then do pull them back by referring to the questions. Interviewer’s welcome, introduction and background to respondent(s) Welcome and thank you for accepting the invitation to take part in this meeting. You have been asked to participate as your knowledge, experience, and point of view are important. We realize you are busy and very much appreciate your time. Introduction: Introduce ourselves. As you are aware, we are conducting the mid-term evaluation of the AfyaInfo Project. We are very interested in learning about your perception and experience with the AfyaInfo Project including what has and what has not worked well, key activities, results, sustainability issues, and future direction relevant to the establishment of a unified and well integrated web-based NHIS. This discussion will take between one to two hours. Ask respondent(s) to introduce themselves by stating their name and their background or relationship as it pertains to the NHIS. Consent/Anonymity: I would like to assure you that the discussion and release of information will be anonymous. The notes of this meeting will contain no information that would allow individual subjects to be linked to specific statements. You should be open, honest and try to answer and comment as accurately and truthfully as possible. Review Questions 1.4 To what extent is the project succeeding in establishing a unified and integrated web-based NHIS producing and promoting the use of quality data? 1.4.1 What is the progress in integrating all the key health service information systems such as disease surveillance, CHIS, MIAS, COBPAR, KePMS, and various Dept. of FH systems into the NHIS? 1.4.2 What is the progress in integrating management information such as commodities, finance, and HRIS into the NHIS? AfyaInfo: Mid-Term Review Key Informant Interview (KII) GUIDE - GENERIC IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 77 1.4.3 What is the progress in integrating key (summarized) population based statistics (vital statistics, surveys, census, i.e. KNBS data) into the NHIS? 1.4.4 Is there an active and effective linkage/coordination between NHIS & NIMES? If so, how? If not, what are the impediments? 1.4.5 How effective has the project been in the interim management of the KePMS and supporting the PEPFAR partners SAPR and APR requirement? 1.4.6 Is the overall data quality of NHIS and its major components satisfactory for policy discussions/formulation and decision-making process (both for management and patient services)? 1.4.6.1 How reliable is the key performance indicator (milestone) data reported by the project? Please describe with examples and evidences. 1.4.6.2 How consistent and concurrent is the DHIS data with other key systems including the KePMS? 1.4.7 What is the current status of the ICT infrastructure development at the central, county and sub-county levels towards having the capability to support development, deployment, and maintenance of the NHIS? 1.4.8 What is the current status in development and enforcement of new policy guidelines and legal frameworks? 1.4.9 What is the current status of the Master Facility List (MFL)/MCUL? How often is MFL/MCFL updated? 1.4.10Has the MFL regulatory module developed, geocodes updated and integrated in the regulatory database? If not, why? 1.4.11 What is the progress in improving collaboration and coordination among various project stakeholders? 1.5 Does the M&E/HID have better representation in the various key coordination frameworks? If not, why? What is the current progress in establishing a functional GOK managed Learning and Knowledge Management (LKM) system that is improving the culture of information generation, dissemination, and use? 1.5.1 To what extent and how the TNA findings been addressed in the newly developed LKM system? 1.5.2 What and how effective have the capacity building initiatives launched by the project been (both institutional and human)? Explain. 1.5.3 Is the TWG focused on the LKM functional? If so, how effective is it? 1.5.4 Are the HIS trainings institutionalized? 1.5.5 What and how frequently information products are being produced and disseminated by the LKM system for various stakeholders at different levels to help improve overall operation, planning, decision making and performance management 1.5.6 What specific activities/opportunities provided by the project have promoted the importance of and raised awareness for a culture shift and strengthen the demand for and use of information at all levels 1.6 How effective has the project been in strengthening the M&E/HID toward its establishment as a functional and effective organizational entity capable of developing, implementing, and managing a unified and integrated NHIS? 1.6.1 To what extent the project activities and capacity building initiatives have helped strengthening the management and coordination structures, and the financial, technical, and HR management systems of the M&E/HID? 1.6.2 Overall, does the M&E/HID have the required capacity to pass an institutional capacity assessment/audit? Why or why not? 2.1 What components and project aspects are not working well and why? 2.2 How effective has the method of implementation been so far? Should the project continue implementing interventions with the same approach and at the same level, or with changes during its final project years? 2.3 How has the policy environment (laws, regulations etc.) been conducive or non-conducive for the effective implementation of the project? 2.4 If and how effective has the project been toward achieving the desired/required level of coordination with the GOK Institutions? If not, why? IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 78 2.5 To what extend has the Project achieved the desired/ required level of coordination with other USG service delivery partners? 2.6 To what extent and how has the project been effective to increase the demand for and use of data and information by decision-makers? 3.1 What new regulations/laws have been advocated by the Project to improve political feasibility? 3.2 If and to what extent (including timeliness) the new regulations/laws were adopted by the ministries and other relevant government agencies? If not, what were the impediments to that? 3.3 Have the newly adopted regulations and laws improved or hindered project performance? If so in which area(s), ways? 3.4 To what extent and how effectively has the MOH allocated match contributions according to the sustainability strategy (addendum to AWP PY1)? 3.5 Is there effective coordination with other ministries/departments where services or information is needed to fully implement the program and have an effective NHIS? 3.6 Is there increasing GOK resources for funding technology acquisition, supplies, and maintenance? If so, in which areas and to what extent? 3.7 To what extent and how efficiently GOK is absorbing the existing key NHIS staffs and/or hiring new ones? 3.8 Is there an effective plan and active effort for program specific parallel systems funded by the DPs to be dissolved and integrated with the NHIS? If so, which, how and timeline? 3.9 How effective is the coordination among the DPs for NHIS related activities and investments? 3.10 What has been the role of the local universities and other private sector entities toward the sustainability, evolution, and maintenance of the NHIS? 3.11 What has the level of involvement been by the county government to ensure sustainability, evolution and maintenance of the NHIS? Key Overarching Strategic Questions: 1. In your opinion, what are some of the key strengths of the NHIS, especially resulted from the AfyaInfo project interventions (i.e., what’s working well?) 2. In your opinion, what are some of the key weaknesses or challenges of the system (i.e., what’s not working so well?) 3. What are the main implementation challenges? Are the current management systems adequate and effective? Why, please explain. 4. How would you describe the leadership and management of both GOK and the AfyaInfo project IPs in respect to the NHIS? 5. How is the policy environment conducive or not conducive to the successful and effective implementation of the project and the NHIS overall? 6. How effective have the performance indicators been in tracking results in the key areas of the project? 7. What are some of the priorities that should be pursued in both in the next two years and in the longer run to sustain current achievements and use of NHIS at all levels? 8. How or in what areas do you think AfyaInfo investment in the next two years can do the most good? Conclusion • Thank you for participating. This has been a very useful discussion • Your opinions, knowledge, and insights will be a valuable asset to the review • I would like to remind you that any comments featuring in this report will be anonymous • Are there any questions? Reminder to the note taker: Please, write your report based on the results of the KII. Please remember to maintain confidentiality of the participating individual by not disclosing their names (indicate their organization instead). IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 79 ANNEX 8: SCHEDULE, ORGANIZATIONS AND NUMBER OF RGD AND FGD PARTICIPANTS ROUNDTABLE & FOCUS GROUP DISCUSSION SUMMARY DATE RGD RESPONDENTS VENUE M F TOTAL August 26 MOH NHIS Senior Management 2 2 4 ESPS Office August 26 MOH National Program representatives 1 2 3 ESPS Office August 27 Regulatory Bodies & national councils 1 4 5 ESPS Office August 28 County Health Executive Officers 2 5 7 ESPS Office August 29 Sub-County Health Records and Information Officers 3 10 13 ESPS Office SUB-TOTAL 9 23 32 DATE FGD RESPONDENTS VENUE August 25 Consortium Partners 1 8 9 ESPS Office August 25 PEPFAR Service Delivery Partners 2 5 7 ESPS Office August 27 USAID Implementing Partners 2 5 7 ESPS Office August 28 Abt. Associates 3 5 8 ESPS Office SUB-TOTAL 8 23 31 TOTAL 17 46 63 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 80 ANNEX 9: SCHEDULE, ORGANIZATIONS AND NUMBER OF SGD PARTICIPANTS DATE FOLLOW UP RESPONDENTS VENUE ORGANIZATION NAME M F TOTAL September 1 MOH, Health Information Systems (HIS) Unit Dr. Martha Muthami Nancy Amayo Francis Gikundi Gladys Echesa Charles Kinutia Patrick Warutete Roberty Wathondu 4 3 7 Afya House, HIS Unit September 2 Christian Health Association of Kenya (CHAK) Dr. Samuel Mwenda Vincent Kiarie Faith Irene Wagaki 2 1 3 CHAK Office September 2 Division of Nutrition Gladys Mugambi Eunice 2 2 DoN Office September 4 University of Nairobi Oliver Munyao Dr. Dan Orwa Elisha Opiyo John Gichangi 4 4 Chiromo Campus September 4 ICF International Edward Kunyanga Erastus Marugu 2 2 ICF Office K-Rep Plaza September 5 Ministry of Development and Planning (MEDS) Mr. Samson Machuka Chief Economist Senior Economist 3 3 MEDS Office September 8 World Health Organization (WHO) Dr. Humphrey Karamagi Hillary Kipruto 2 2 WHO Offices TOTAL 17 6 23 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 81 ANNEX 10: SCHEDULE, LIST AND NUMBER OF KEY INFORMANT INTERVIEWS DATE FOLLOW UP RESPONDENTS VENUE ORGANIZATION NAME M F TOTAL September 2 Division of M&E & Health Informatics Dr. Soti 1 1 Afya House September 5 Knowing Limited Gilbert Kanjama 1 1 ESPS September 5 Div. M&E & Health Informatics Jeremiah Mumo 1 1 Afya House September 10 Abt Associates Dr. Salome Ngata David Muturi 1 1 2 Abt Associates Office, Royal Ngao House September 12 Abt Associates Alex Njau Dr. Martin Osumba 2 2 Abt Associates Office, Royal Ngao House September 12 Abt Associates Jim Setzer Nick Oyugi 2 2 Abt Associates Office, Royal Ngao House September 12 Abt Associates Mary Scott 1 1 Skype conference SUB-TOTAL 8 2 10 DATE SAMPLED RESPONDENTS VENUE ORGANIZATION NAME M F TOTAL August 30 Former NASCOPE & HMIS Dr. Kimanga 1 1 EKA Hotel September 1 Former Head - MOPHS Dr. S. K. Shariff 1 1 Conference call September 2 CDC James Kwach 1 1 CDC September 3 Clinton Health Access Initiative (CHAI) Mr. Gerald Macharia 1 1 CHAI Offices September 3 MOH Health Management Information Systems (HMIS) Dr. Charles Nzyoka 1 1 Afya House September 4 Kenya Medical Supplies Agency (KEMSA) Samwel Wataku 1 1 KEMSA Office September 8 DANIDA Rhoda Njuguna 1 1 DANIDA Office September 8 Kenya National Bureau of Statistics, Civil Registration Dept. (KNBS) Mr. Judy Kilobi 1 1 KNBS office September 10 MOH Public Private Partnership Dr. Samuel Were 1 1 Afya House, HSR office September 11 USAID Rene A. Berger Washington Omuomo 2 2 USAID Offices September 11 Training Resource Group Mr. Fred Rosensweig 1 1 Teleconference SUB-TOTAL 10 2 12 TOTAL 18 4 22 IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 82 ANNEX 11: YEAR 3 AFYAINFO PMP TRACKING Indicator Achievements Targets Evidence/ Documentation YR1 YR2 YR3 YR3 YR4 YR5 Jul-Sep Oct-Dec Jan￾Mar Apr -Jun 1.1 % of health facilities where health information system is in use for at least 24 months uninterrupted before sign off 91.3%[2] 93.7%[9] 93.6%[17] 86.7%[30] 80% 80% 80% DHIS2 MOH 711 Data for Project Years 1 and 2 through to Y3Q2 that is Disaggregated by County and Ownership 1.2 % of community units where health information system is in use for at least 24 months uninterrupted before sign off 46.3%[3] 55%[10] 52.3%[18] 44.4%[31] 80% 80% 80% DHIS2 MOH 515 Data for Project Years 1 and 2 through to Y3Q2 that is Disaggregated by County 1.3 % of facilities reporting complete and accurate data as required by facility based programs in health sector through HMIS 12 months after system deployment 91.3%[4] 93.7%[11] 93.6%[19] 86.7%[32] 80% 80% 80% DHIS2 MOH 711 Data for Project Years 1 and 2 through to Y3Q2 that is Disaggregated by County and Ownership 1.4 # of independent health sector data/ information systems integrated into single web-based HMIS 2[5] 4[12] 4[20] 4[33] 6 6 6 Kenya Standards and Guidelines for e-Health Systems Interoperability; Screen Shots of DHIS2 Forms (HSSF, In-Patient and KQMH) [https://hiskenya.org]; NHIS Systems Assessment Report; NHIS Desk Review Report; DHIS2 Reports 2.1.1 Functional TWG created/ supported to lead all learning and knowledge management activities and policy dialogue NO YES YES YES YES YES YES Health Sector M&E Framework and Guidelines; Health Sector M&E Technical Working Group TOR; Consultant to Support M&E IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 83 Framework and Guidelines Development TOR 2.1.2 Stakeholder information needs identified NO YES YES YES YES YES YES DDIU Assessment Report and Strategy; Health Manager’s HIS Data and Training Needs Forum Report; Stakeholder Assessment Report 2.1.3 Develop health communication strategy in collaboration with and to meet needs of stakeholders at all levels NO NO NO NO YES YES YES 2.1.4 Develop learning and knowledge management system for use and deployment at all levels NO NO NO NO YES YES YES HIS Capacity Building Plan, Schedule for County CHIS/ HIS Trainings (YR3); Training Management Information System (Years 1 to Year 3); HIS Training Quality Assurance Concept Note; NHIS Mentorship Program Concept Note 2.1.5 Define DQI/ DQA strategy for institutionalization within the MOH NO NO NO YES YES YES YES DQA Protocol; Data Quality Audit Concept Note 2.1.6 % of planned capacity building activities in information use for audiences at all levels carried out N/A[7] 100% N/A[21] 100% 100% 100% 100% HIS Capacity Building Plan, Schedule for County CHIS/ HIS Trainings (YR3); Training Management Information System (Years 1 to Year 3) 2.2 % counties with functional learning and knowledge management system in use for at least 24 months uninterrupted before sign off N/A N/A[13] N/A[22] N/A[34] 80% 100% 100% 2.3 % of health facilities with functional learning and knowledge management system in use for at N/A N/A N/A[23] N/A[35] 75% 80% 80% IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 84 least 24 months uninterrupted before sign off 2.4 % of community units with functional learning and knowledge management system in use for at least 24 months uninterrupted before sign off N/A N/A N/A[24] N/A[36] 75% 80% 80% 2.5 % of national, regional and district level public awareness and dissemination forums in use N/A N/A[14] N/A[25] N/A[37] 50% 60% 70% 2.6 % of counties producing quarterly print and electronic materials on health information N/A N/A[15] N/A[26] N/A[38] 40% 60% 80% 2.7 % of facilities producing quarterly print and electronic materials on health information N/A N/A N/A[27] N/A[39] 20% 40% 60% 2.8 % of community units producing quarterly print and electronic materials on health information N/A N/A N/A[28] N/A[40] 20% 40% 60% 2.9 Quarterly print and electronic materials on health information and their usefulness available and being produced and distributed at all levels NO YES YES YES YES YES YES Kenya Health Sector Performance Factsheet Apr-Jun 2013; Kenya Health Sector Performance Factsheet Dec 2012; Kenya Health Sector Performance Factsheet Sep 2012; Factsheets Dissemination Emails; Annual Health Sector Statistics Report 2013; Annual Health Sector Statistics Report 2011; DDIU Assessment Report and Strategy; MOH HIS Factsheet June 2013; Health Congress Concept Note; County Strategic and Annual Planning Templates and Guidelines IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 85 2.10 Existence of reliable and up-to￾date web based public health information database (including MFL) NO[8] NO NO YES YES YES YES MFL and DHIS2 Logs 3.1 Ability of DivHIS to pass an institutional capacity assessment/ audit on management and coordination, organizational leadership and governance structure, financial and procurement N/A N/A[16] N/A[29] N/A[41] YES YES YES 3.1.1 Policy, planning and legal framework for NHIS reviewed YES YES YES YES YES YES YES Review of the Legal Framework for a NHIS in Kenya; Review of the HIS Policy 2010-2030; Review of the HIS Strategic Plan 2009- 2014 3.1.2 Recommendations for revision of NHIS policy planning and legal framework submitted YES YES YES YES YES YES YES HIS Strategic Plan 2013-2018; Kenya HIS Policy 2013 3.2.1 DivHIS organizational strengthening needs assessed YES YES YES YES YES YES YES Division of HIS Organizational Management and Assessment Report; Institutional Review of the HIS; County HIS Capacity Building Needs Assessment Reports 3.2.2 DivHIS organizational strengthening plan developed YES YES YES YES YES YES YES Division of HIS Organizational Strengthening Action Plan (Part of the Division of HIS Organizational Management and Assessment Report); Overarching Recommendations of the Institutional Review of the HIS; County HIS Capacity Building Needs Assessment Reports 3.2.3 NHIS/ DivHIS leadership and management competencies identified and developed YES YES YES YES YES YES YES SOW for Organizational Strengthening Initiatives; TNA of Division of HIS; The AfyaInfo Effective Leadership and IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 86 Management Training Workshop: Training Report; County HIS Capacity Building Needs Assessment Reports 3.3.1 NHIS/ DivHIS management systems (planning, human resources, financial management, procurement, communication/ advocacy etc.) strengthened/ developed YES YES YES YES YES YES YES Division of HIS Organizational Strengthening Action Plan (Part of the Division of HIS Organizational Management and Assessment Report); HIS Strategic Plan 2013- 2018; Kenya HIS Policy 2013; Draft Health Information Code of Practice; Justification for Long￾Term Training Support to Division of HIS by AfyaInfo; The AfyaInfo Effective Leadership and Management Training Workshop: Training Report; The Draft NHIS Resource Mobilization Strategy; The Draft NHIS Advocacy Strategy; County HIS Capacity Building Needs Assessment Reports 3.4 NHIS institutional and organizational architecture at national and sub-national levels defined and developed NO YES YES YES YES YES YES Committee Report: Functional Structures for Division of HIS 3.6 NHIS stakeholder coordination mechanisms developed, in place and functioning YES YES YES YES YES YES YES HIS Stakeholders Coordination Strategy; AfyaInfo Strategy for USG partners Engagement and Collaboration; HIS Stakeholder Coordination Meeting Minutes 3.8 NHIS/ DivHIS short term, medium term and long term staffing requirements identified and appropriate plan developed (for YES YES YES YES YES YES YES Concept Note: HMIS Human Resources Gap Analysis; Preliminary HR Needs for NHMIS; Draft JDs: HMIS Advisor and ICT IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 87 implementation by Capacity Project) Project Manager [2] This value is from DHIS2 reporting on reporting rate for MoH Form 711 for the month of June 2012. [3] This value is from DHIS2 reporting on reporting rate for “community health extension worker summary report” for June, 2012. [4] This value is from DHIS2 reporting on completeness for MoH Form 711 for the month of June 2012. [5] The MFL and DHIS systems have been linked using the API developed by AfyaInfo. [7] Capacity building plan not yet developed pending results of stakeholder mapping and needs assessments activities. [8] The DHIS platform and the MFL are both available web-based databases. Efforts are ongoing to improve the quality of the data they contain and to integrate them and other relevant public health information data bases. [9] DHIS2 reporting rate for MOH Form 711 for May 2013 as per 22 July 2013. [10] DHIS2 reporting rate for CHEW Summary for May 2013 as per 22 July 2013. [11] DHIS2 reporting rate for MOH Form 711 for May 2013 as per 22 July 2013. [12] DHIS2, HSSF, in-patient subsystem and KQMH. [13] LKM data for sub national levels (county, health facility and community unit) will be available after county HIS assessment that is scheduled for Q1 of Year 3. [14] Dissemination forums data for national and sub national levels (regional and district) will be available after county HIS assessment that is scheduled for Q1 of Year 3. [15] Production of quarterly print and electronic materials on health information for sub national levels (county, health facility and community unit) will be available after county HIS assessment that is scheduled for Q1 of Year 3. [16] Data on capability of DivHIS to pass USAID institutional capacity assessment/ audit will be available end of Year 3. [17] DHIS2 reporting rate for MOH 711 for August 2013 as IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 88 per 21 October 2013. [18] DHIS2 reporting rate for CHEW Summary for August 2013 as per 21 October 2013. [19] DHIS2 reporting rate for MOH 711 for August 2013 as per 21 October 2013. [20] DHIS2, HSSF, in-patient subsystem and KQMH. [21] NHIS trainings scheduled from Y3Q2. [22] Measurement of Indicator 2.2 to begin in the subsequent Quarters, prior to 24 months before sign off. [23] Same as Indicator 2.2 above. [24] Same as Indicator 2.2 above. [25] Measurement of Indicator 2.5 will begin in the subsequent Quarters (in Y3Q3). [26] Measurement of Indicator 2.6 will begin in the subsequent Quarters (in Y3Q3). [27] Same as Indicator 2.6 above. [28] Same as Indicator 2.6 above. [29] Data on capability of DivHIS to pass USAID Institutional Capacity Assessment/ Audit will be available end of Year 3. [30] DHIS2 reporting rate for MOH 711 for November 2013 as per 22 January 2014. [31] DHIS2 reporting rate for CHEW Summary (MOH 515) for November 2013 as per 22 January 2014. [32] DHIS2 reporting rate for MOH 711 for November 2013 as per 22 January 2014. [33] DHIS2, HSSF, in-patient subsystem and KQMH. [34] Measurement of Indicator 2.2 will start in Y3Q3. [35] Same as Indicator 2.2 above. IBTCI: Mid-Term Performance Review of AfyaInfo Annexes 89 [36] Same as Indicator 2.2 above. [37] Measurement of Indicator 2.5 will start in Y3Q3. [38] Measurement of Indicator 2.6 will start in Y3Q3. [39] Same as Indicator 2.6 above. [40] Same as Indicator 2.6 above. [41] Organizational Assessment of the DivHIME by the 36th month after the award will establish the unit’s capability to pass the USAID Institutional Capacity Assessment/ Audit.