MIDUnited States -TERM PERFORMANCE REVIEW OF FUNZO KENYA FINAL REPORT USAID KENYA ESPS DECEMBER 2, 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Dr. Martina Nicolls, Mr. William Kiarie and Dr. Teresa Kinyari of International Business & Technical Consultants Inc. Evaluation Services and Program Support (ESPS) in Kenya. MID-TERM PERFORMANCE REVIEW OF USAID FUNZO KENYA December 2, 2014 Award No: AID-623-I-13-00001 Prepared for Trisha Savage United States Agency for International Development/Kenya C/O American Embassy United Nations Avenue, Gigiri P.O. Box 629, Village Market 00621 Nairobi, Kenya Prepared by International Business & Technical Consultants Inc. 8618 Westwood Center Drive Suite 400 Vienna, Virginia, 22182 DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ACKNOWLEDGMENTS We would like to thank the staff of the Ministry of Health (MOH) of the national government of Kenya and the county health departments, regulatory boards and councils, and training institutions for their cooperation and access to health professionals and related personnel for the mid-term review of the FUNZOKenya project. We would also like to thank the staff of the USAID/Kenya Office of Population and Health (OPH), and IBTCI Evaluation Services and Program Support (ESPS) Office in Nairobi, for their logistical support and guidance. Special thanks must be given to the note-takers, Stephanie Gacheru and Eunice Were. Thanks are also due to the FUNZOKenya prime Implementing Partner, IntraHealth International Inc., and their resource partners and collaborators. We are grateful to all stakeholders and beneficiaries for their cooperation and input. Martina Nicolls, FUNZOKenya MTR Team Leader William Kiarie, FUNZOKenya MTR Human Resource Development Specialist and Evaluator Teresa Kinyari, FUNZOKenya MTR Public Health Specialist and Evaluator ACRONYMS AND ABBREVIATIONS AEF Afya Elimu Fund APHIAplus AIDS, Population and Health Integrated Assistance ART Anti-Retroviral Therapy BDS Bachelor of Dental Sciences BSc Bachelor of Science CCSV Clinical Care of Sexual Violence CHA Community Health Assistant CHD County Health Department CHEW Community Health Education Worker COC Clinical Officers Council CPD Continuous Professional Development CHMT County Health Management Teams CHSU Community Health Strategy Unit DSRS Dept. Standard Regulatory Services DHSQAR Directorate of Health Standards, Quality Assurance and Regulations e-L e-learning EMNC Essential Maternal & Newborn Care EMONC Emergency Maternal Obstetric & Neonatal Care ESPS Evaluation Services and Program Support FBO Faith Based Organization FGD Focus Group Discussion GLUK Great Lakes University of Kisumu GOK Government of Kenya HCW Health Care Workers HELB Higher Education Loans Board HPMT Health Profession Management Tool HRD Human Resources Development HRH Human Resources for Health HRIO Health Records Information Officer HTC HIV, Testing & Counseling HTD Health Technical Departments ICT Information & Communication Technologies iHRIS Integrated Human Resource Information System IMCI Integrated Management of Child Illness IR Intermediate Result IST In-Service Training KHF Kenya Healthcare Federation KII Key Informant Interview KMPDB Kenya Medical Practitioners & Dentists Board KMLTTB Kenya Medical Laboratory Technologists & Technicians Board KMTC Kenya Medical Training College KNDI Kenya Nutritionists & Dietetics Institute KRCHN Kenya Registered Community Health Nurse KSN Kijabe School of Nursing KU Kenyatta University LAPM-FP Long Acting Permanent Methods of Family Planning M&E Monitoring and Evaluation MEDS Mission for Essential Drugs & Supplies MLS Medical Laboratory Sciences MLT Medical Laboratory Technologist MNCH Maternal and Neonatal Child Health MOE Ministry of Education MOH Ministry of Health MPH Masters in Public Health MTR Mid-Term Review NASCOP National AIDS & STI Control Program NCK Nursing Council of Kenya NHHRDTWG National Health Human Resource Development TWG OPH Office of Population and Health PEPFAR President's Emergency Plan for AIDS Relief PHOTC Public Health Officers and Technicians Council PMTCT Prevention of Mother to Child Transmission PNA Performance Needs Assessment PPB Pharmacy & Poisons Board PPP Public-Private Partnership PST Pre-Service Training QA Quality Assurance R4D Results for Development Institute RB Regulatory Bodies RGD Roundtable Group Discussion RH Reproductive Health RP Resource Partner SOW Scope of Work TNA Training Needs Assessment TOT Training of Trainers TWG Technical Working Group UNC University of North Carolina UON University of Nairobi TABLE OF CONTENTS EXECUTIVE SUMMARY ................................................................................................................... I 1. INTRODUCTION ..................................................................................................................... 1 2. BACKGROUND ........................................................................................................................ 1 3. METHODOLOGY ..................................................................................................................... 2 4. EVALUATION QUESTIONS & REPORT STRUCTURE ...................................................... 3 5. FINDINGS, CONCLUSIONS & RECOMMENDATIONS ...................................................... 3 5.1 Q1: IMPACT OF ACTIVITY DESIGN, ACHIEVEMENTS, OPPORTUNITIES & THREATS ........................... 3 5.2 Q2: COORDINATION & COLLABORATION MECHANISM .................................................................. 15 5.3 Q3: PERCEPTIONS OF MOH AND SERVICE DELIVERY IMPLEMENTING PARTNERS ........................... 17 5.4 Q4: INNOVATION & SUSTAINABLE HRH MODELS ........................................................................... 18 6. KEY STRATEGIC FUTURE DIRECTIONS ...........................................................................21 ANNEX 1: SCOPE OF WORK.......................................................................................................22 ANNEX 2: CONFLICT OF INTEREST DISCLOSURES .............................................................31 ANNEX 3: REFERENCES ...............................................................................................................52 ANNEX 4: EVALUATION METHODOLOGY .............................................................................55 ANNEX 5: DATA COLLECTION TOOLS ...................................................................................63 ANNEX 6: LIST OF PERSONS INTERVIEWED AND FOCUS GROUP PARTICIPANTS ....69 ANNEX 7: MARGINALIZED AREAS IN KENYA ........................................................................73 ANNEX 8: REGIONAL HUBS AND SATELLITE SITES ............................................................74 ANNEX 9: SUMMARY OF FUNZO KENYA TARGETS & ACHIEVEMENTS .........................75 ANNEX 10: CURRICULA & E-LEARNING MODULES REVIEWED/DEVELOPED ................77 ANNEX 11: STATUS OF CPD GUIDELINES & CORE CURRICULA .......................................79 Tables TABLE 1: SUMMARY OF MID-TERM ACHIEVEMENTS FOR INTERMEDIATE RESULT 1 ........................................ 4 TABLE 2: LIST OF PRIORITY HEALTH CADRES FOR TRAINING SUPPORT ............................................................. 4 TABLE 3: SCHOLARSHIP AND LOAN STUDENTS BY CADRE AND GENDER .......................................................... 5 TABLE 4: CONTRIBUTIONS TO AEF FOR PRE-SERVICE STUDENT LOANS ............................................................ 5 TABLE 5: SCHOLARSHIPS AND LOAN BY MARGINALIZED COUNTY AND GENDER .............................................. 6 TABLE 6: SUMMARY OF MID-TERM ACHIEVEMENTS FOR INTERMEDIATE RESULT 2 ........................................ 8 TABLE 7: HEALTHCARE WORKER IN-SERVICE TRAINING ACHIEVEMENT PER ANNUAL TARGETS ...................... 9 TABLE 8: SUMMARY OF MID-TERM ACHIEVEMENTS FOR INTERMEDIATE RESULT 3 ...................................... 10 TABLE 9: SUMMARY OF MID-TERM ACHIEVEMENTS FOR INTERMEDIATE RESULT 4 ...................................... 12 i EXECUTIVE SUMMARY BACKGROUND AND METHODOLOGY The USAID-funded FUNZOKenya (FUNZO/K) is a five-year $40 million activity (February 2012 – February 2017) implemented by IntraHealth International as the prime partner in collaboration with seven resource partners.1 FUNZO/K was established to address the Government of Kenya’s (GOK) vulnerabilities in its health workforce training. USAID’s Office of Population and Health (OPH) commissioned an independent mid-term review (MTR) of the activity with the objective to assess FUNZO/K’s progress toward its strategic goal to improve access to and quality of health workforce training through four intermediate results (IR). The MTR has determined activity successes and challenges and has provided recommendations to USAID to improve implementation for the remainder of FUNZO/K’s period of performance (POP). The FUNZO/K MTR was guided by five evaluation questions, provided in the Statement of Work (SOW): 1) To what extent has the activity’s design facilitated the achievement of the activity’s mid-term mandates, targets, outcomes, and milestones? With regard to the activity’s design, what are the main constraints and/or weaknesses, opportunities and threats that need further attention? 2) To what extent is the current coordination and collaboration mechanism among key stakeholders (service delivery implementing partners, relevant GOK ministries and training institutions) effective in achieving program objectives? 3) To what extent is the activity perceived as relevant, responsive and useful to the main stakeholders (MOH and service delivery implementing partners) in addressing the challenge of having skilled and knowledgeable healthcare providers at facility level? 4) What sustainable and innovative Human Resources for Health (HRH) system-related models is FUNZO/K currently implementing? What do the Ministry of Health (MOH) and regulatory boards think of their long-term use? 5) What are the key programmatic and management recommendations that the mission could consider for mid-course changes to the current program design? The mid-term review team (MTRT) was comprised of three experts, one international and two Kenyan professional consultants. The MTR was conducted in September and October 2014 using qualitative and quantitative methods, including a document review, key informant interviews (KIIs), roundtable group discussions (RGDs) and focus group discussions (FGDs). The consultants interviewed 46 key informants, and held three roundtable group discussions and 12 focus group discussions involving 92 respondents. Methods used included the following: content analysis of FUNZO/K and health-related documents; interview and focus group content analysis; a strengths, weaknesses, opportunities and threats (SWOT) analysis; gender-based analysis; and multiple triangulation approaches to enable comparative analysis of data from various sources. In some cases the FUNZO/K baseline data was insufficient for the consultants to conduct extensive comparisons with activity reports and mid-term data. KEY FINDINGS The findings are reported by each of four MTR evaluation questions and, where applicable, also by result areas and FUNZO/K’s cross-cutting strategies: financing, public-private partnerships, technology, and accountability. 1 Please see Annex 1, SOW for list of partners and responsibilities. ii Q1: IMPACT OF ACTIVITY DESIGN, ACHIEVEMENTS, OPPORTUNITIES &THREATS Result 1 – Increased Number of New Healthcare Workers Trained: FUNZO/K developed and launched a forecasting publication in 2013 called Health Workforce Forecast Kenya to project the type and number of health workers Kenya needs to produce to adequately meet the health care needs of the population. In the first year, FUNZO/K awarded 341 pre-service training scholarships against a target of 350. In the second year, the activity introduced a loan scheme, the Afya Elimu Fund (AEF), run by the Higher Education Loans Board (HELB). In addition to administering the scheme, HELB has injected KES150 million into the fund and the HELB contribution into the AEF now exceeds that of USAID. FUNZO/K significantly exceeded its target of the number of pre-service students receiving financial support and by end of September 2014, AEF loans had been awarded to 2,191 students against a target of 600. Forty-five percent of the scholarship recipients were from marginalized counties, while only 5% of the loan students were from marginalized counties. The proportion of students from marginalized counties declined significantly – from 45% to 5% on transitioning from the scholarship to the loan scheme. Result 2 – Supporting Health Workers Training Needs: FUNZO/K conducted a comprehensive national Training Needs Assessment (TNA) in 2012 and established a regional training model (eight regional training hubs and seven satellite centers drawn from well￾established public and private training institutions) to manage and deliver training. The activity has trained 7,595 Health Care Workers (HCWs), exceeding the mid-term target of 7,000. The regional training hubs are functioning well with the exception of the North Eastern region that is not yet operational due to a lack of adequate tutors. The activity also started a training of trainers’ program targeting provision of pedagogy skills to tutors; 87 tutors have been trained to date. Result 3 – Capacity of Training Institutions Enhanced: Fifteen training Institutions (TIs) were assisted to develop business plans to support resource mobilization. The capacity of ten TIs to review and develop curricula was strengthened. TIs were also provided with skills to convert existing courses into e-versions and also develop new e-courses. Ten modules were converted into e-learning mode against a mid-term target of three to six modules. Under the activity’s public-private partnership (PPP) initiative, Kenyatta University has been supported to develop a PPP program with an international fund (Abraaj) to help the university establish a mid-level health training institution in Nairobi. FUNZO/K support has enabled targeted TIs to review their management processes and to adopt more commercially-oriented approaches that have enabled them to enhance operational efficiencies, reduce costs and generate additional revenue. Result 4 – Regulatory Bodies (RBs) Strengthened to Enhance Training Demand: FUNZO/K has worked will all seven medical RBs in Kenya to strengthen their capacity to regulate medical training and practice. All seven have developed continuous professional development (CPD) guidelines and core curricula that are at various stages of development and implementation. Q2: COORDINATION & COLLABORATION MECHANISM Service delivery implementing partners (SDPs) indicated that they would prefer to have more collaboration and involvement in FUNZO/K training interventions.2 To facilitate and improve inter-organizational and inter-regional collaboration FUNZO/K initiated the GOK-led National Health Human Resource Development Working Group (NHHRDWG) monthly meetings. The NHHRDWG enabled the activity to align all interventions under each result within one overarching working group and create an oversight and endorsement mechanism among relevant government departments, regulatory bodies and councils, the eight regional training hubs and the 2 Drawn from RDPs FGDs respondents iii private sector. FUNZO/K has supported RBs to conduct joint inspection of facilities for the accreditation of training providers, scheduled by the Kenya Medical and Dentists Practitioners Board (KMPDB). Q3: PERCEPTIONS OF MOH AND SERVICE DELIVERY IMPLEMENTING PARTNERS3 The majority of respondents interviewed stated that FUNZO/K’s mandate is relevant and the organization has a competent, committed and accessible leadership and management team that has forged close working relations with stakeholders and is making a positive change in strengthening training capacity and systems in the health sector. MOH participants were of the view that FUNZO/K’s activities were aligned with those of the MOH. SDPs reported the activity’s training plans are largely based on gaps they identify. However, they were of the view that FUNZO/K is not adequately responsive to emerging training needs. They also felt that as they are on the ground and more in touch with evolving training needs, they should have a greater role in training. Q4: INNOVATION & SUSTAINABLE HRH MODELS FUNZO/K supported the RBs in the development of short continuous professional development (CPD) guidelines linked to registration and re-licensure. However, there are no linkages between the CPD points and in-service training data held in the iHRIS Train. FUNZO/K also assisted in the development and implementation of cross-cadre CPD guidelines and supported seven RBs to review or develop their core curricula outlining minimum content and competencies essential to guarantee professional license. By the mid-term, the activity has provided assistance to 21 TIs (from a total of 26) to conduct student indexing in accordance with RB guidelines and has supported the conversion of existing or newly developed curriculum modules into open, distance, and e-learning (ODeL). The activity aided in the development of the iHRIS Train database to manage all training data. CONCLUSIONS Q1: IMPACT OF ACTIVITY DESIGN, ACHIEVEMENTS, OPPORTUNITIES & THREATS Result 1 – Increased Number of New Healthcare Workers Trained: The MOH has an approved forecasting system, but it is yet to be rolled out and is not being used to influence health worker production trends. The shift from a scholarship program to a loan scheme has mobilized significant additional funding especially from HELB that has enabled FUNZO/K to exceed its target for number of loans issued. However it is not clear how the activity is measuring its progress towards its intended result of increasing admission rates given the fact that loans are only awarded to students that are already admitted for training. Result 2 – Supporting Current Health Worker Training Needs: FUNZO/K has established an inclusive, collaborative training network. This network has strengthened formal linkages between CHDs and training institutions that were previously weak or limited. Overall, the in-service regional training model is functioning well and is regarded as a model for increasing the capacity of training institutions. Result 3 – Strengthened Capacity of Training Institutions: FUNZO/K’s efforts to enhance resource mobilization, technical and financial management capacity of TIs are beginning to have a positive impact. Significant progress has been made in one PPP. If this is finalized, it will mobilize substantial private sector funding to support mid-level training in a public university. Other potential PPPs are yet to materialize. 3 Results of FGDs and RGDs held with various stakeholders during the MTR. iv Result 4 – Regulatory Bodies Strengthened: Extensive progress has been made towards the development of standardized continuous professional development guidelines, curricula, and standards, as well as the accreditation of training providers. Q2: COORDINATION & COLLABORATION MECHANISM FUNZO/K has established effective coordination and collaboration mechanisms including the NHHRDWG and regional training hubs. The establishment of regional training hubs provides a mechanism that supports sustainability of inter-regional coordination. While the activity has attempted to facilitate communication between all levels of the health training chain, there is still disconnect in terms of planning (national level), training (national level), and the recruitment of health workers (county level). Q3: PERCEPTIONS OF MOH AND SERVICE DELIVERY IMPLEMENTING PARTNERS Whereas MOH is satisfied with FUNZO/Ks support and is of the view that this support is aligned with identified national priorities, service delivery implementing partners feel that FUNZO/K is not adequately responsive to emergent training needs and that training course offerings may be too broad and this could compromise the impact of its training. Q4: INNOVATION & SUSTAINABLE HRH MODELS A comprehensive regulatory framework for CPD training of health professionals is in place and has been adopted by the MOH. TIs have increased their capacity to convert modules into e￾learning delivery modes to increase admissions and facilitate cost effectiveness. The use of existing and well established training institutions in developing a regional training model strengthens sustainability of the in-service training result area. The use of HELB to administer and fund mid￾level health training is both innovative and sustainable. FUNZO/K supported development of iHRIS Train database but its data is not being fully used due to limited access and lack of database interoperability. RECOMMENDATIONS Q1: IMPACT OF ACTIVITY DESIGN, ACHIEVEMENTS, OPPORTUNITIES AND THREATS Result 1 – Increased Number of New Healthcare Workers Trained:  Review the administration of the loan scheme to ensure equity especially for students from marginalized counties.  Explore the potential for closer alignment of loans with identified priority cadres.  Quantify and document how access to financing for pre-service training is increasing enrollment and completion rates. Result 2 – Supporting Current Health Worker Training Needs:  Support county health departments to conduct training needs assessments (TNAs) at the county level.  Conduct annual post-training assessment in 2015 and 2016.  Operationalize the North Eastern regional training hub. Result 3 – Strengthened Capacity of Training Institutions:  Accelerate efforts to ensure that training institutions adopt the TNA output and policies, guidelines and standards developed throughout FUNZO/K.  Roll out the clinical placement model and mentorship guidelines to training institutions. v  Develop mechanisms for formalizing secondment of MOH or county staff to support pre￾service training, including the supervision of students during clinical placement.  Quantify impact of business strategies designed to increase training admissions and set clear targets around the expansion of admission capacity. Result 4 – Regulatory Bodies Strengthened:  Accelerate full implementation of the newly developed CPD guidelines and core curricula.  Develop an online portal for regulatory guidelines, standards, regulations, and core curricula.  Continue strengthening iHRIS and improve data interoperability with other HRH databases.  Review the forecasting model/dashboard to ensure it is used to influence pre-service admission trends.  Delineate more clearly the roles of FUNZO/K and HRH Capacity Bridge in developing the training and workforce forecasts. Q2: COORDINATION & COLLABORATION MECHANISM  Include the six public, private and faith-based medical colleges in marginalized counties for networking opportunities.  Include service delivery implementers in the delivery of in-service training, and in the conversion and adoption of modules for alternative delivery modes. Q3: PERCEPTIONS OF MOH AND SERVICE DELIVERY IMPLEMENTING PARTNERS  FUNZO/K should continue working closely with MOH.  Review the role of service delivery implementing partners in training. One option is to create two categories of training programs, one that should be offered by service delivery partners while the other should continue being managed by FUNZO/K. Another option is to have service delivery implementing partners coordinate all training with FUNZO/K focusing on building the training capacity, developing a network to deliver training and strengthening regulation and quality assurance. Q4: INNOVATION & SUSTAINABLE HRH MODELS  Continue to support iHRIS Train and expand access to and use of the data for planning and integration/interoperability with other HRH databases in Kenya. KEY STRATEGIC FUTURE DIRECTIONS With devolution, counties are requesting support to conduct TNAs and have yet to adopt forecasting, predominantly because it needs an integrated interoperable database for the management of training, human resource and workforce data. Therefore FUNZO/K should consider strengthening county level management, especially for the coordination of in-service training that meet county-specific health needs. FUNZO/K’s major achievements have been based on establishing a buy-in of frameworks, systems and mechanisms for regulations and quality assurance. Most are in the early development stages, draft form or recently completed. FUNZO/K should focus on accelerating their implementation for the remaining years. FUNZO/K should establish an e-portal to give stakeholders access to these newly created national documents, particularly as many have been developed for the first time in the nation’s history. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 1 1. INTRODUCTION USAID/Kenya’s Office of Population and Health (OPH) signed a five-year $40 million Cooperative Agreement on February 24, 2012, with IntraHealth International Inc. to implement a project entitled National Training Mechanism in Kenya, known as FUNZOKenya (FUNZO/K),4 which concludes in February 2017. IntraHealth works with seven resource partners.5 USAID/Kenya/OPH commissioned a mid-term review (MTR) with the aim of determining factors that enhance or impede FUNZO/K’s progress toward its strategic objective of ensuring increased access to and quality of sustained health workforce training for Kenyans. The MTR’s overall objective was to review implementation approaches for each strategy and determine the extent to which they will achieve their intended outcomes, identify improvements and provide recommendations based on findings that will mitigate weaknesses and sustain its strengths. The primary audience and users of the MTR report are the US Government Agencies – USAID, Centers for Disease Control (CDC) and Prevention, and Department of Defense (DOD) – and the Government of Kenya (GOK). The report will assist USAID/Kenya/OPH, IntraHealth and resource partners to refine and improve activities for the remaining 2.5 years through actionable recommendations. 2. BACKGROUND USAID Strategy: FUNZO/K is aligned with the USAID Framework 2010-2015 goal of “sustained improvement of health and well-being for all Kenyans,” and responds to Result Area 2, “Health systems strengthened for sustainable delivery of quality services” (Annex 1: Scope of Work). Development Problem: The activity was designed to address the GOK’s vulnerabilities in its health workforce training. These include a fragmented healthcare system, inability to meet the demand for healthcare workers resulting in critical staff shortages in all cadres (especially nurses and clinical officers), unequal distribution of healthcare workers in urban and rural areas, and the need to address community health needs. Key health education providers offering pre-service education6 were constrained by funding shortfalls. The Ministry of Health’s (MOH) Department of Human Resources Development (HRD) provided support for regional in-service training (IST) of health workers, but had few staff to focus on continuous professional development (CPD). Regulatory Boards (RBs) and councils, by statute, approve pre-service training (PST) curricula, accredit health providers, review and update core curricula, and regulate practice within their respective cadres. However, there were no clear processes for updating curricula, standardizing registration and accreditation or improving registration databases. In addition, the private health sector was not fully embraced by the population and GOK. Development Hypothesis and Strategies: The FUNZO/K development hypothesis (or If￾Then Logic, or Theory of Change) is: If the MOH expands and improves pre-service trainings, develops and implements innovative models for in-service trainings, and strengthens the capacity of pre-service educational institutions then there will be increased capability to plan for, recruit, train, regulate and retain adequate and appropriate human resources for health at all levels of the health system.7 The development hypothesis was formulated from IntraHealth’s technical framework that builds on lessons learned from previous health activities in Kenya. It is based on the premise that four drivers of change, if effectively mobilized, will result in a more responsive, sustainable health training system. The four drivers of change (known as FUNZO/K’s strategies) are: (1) Financing - financial incentives for training and cost-recovery models of training delivery, (2) Public-private partnerships - partnerships with private-sector health training and service delivery institutions and non-health private sector such as technology corporations and financial institutions, (3) Technology - existing and new, adaptive and scalable technologies to expand training relevancy, consistency, accessibility and quality for instruction, faculty development, planning and management, and regulation, and (4) Accountability – responsiveness of regulatory 4 Cooperative Agreement (2012), p1; Funzo means training in Kiswahili. 5 List and responsibilities of resource partners can be found in Annex I. 6 Such as the Kenya Medical Training Colleges (KTMC) that train 90% of new health workers nationally in its 28 campuses. 7 MTR SOW. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 2 bodies to the health needs of the communities they serve, the health workers enrolled in training and the health system. Results: FUNZO/K is required to measure its progress toward their development hypothesis through four results (R) stated in their Cooperative Agreement:8 Result 1: Supporting Increased Number of New Health Workers Trained Result 2: Supporting Current Health Workers Training Needs Result 3: Strengthened Capacity of Training Institutions Result 4: Regulatory Bodies Strengthened to Enhance Training Demand 3. METHODOLOGY Evaluation Services Program Support (ESPS) fielded the mid-term review team (MTRT) consisting of three independent consultants: an international team leader with evaluation experience, and two local evaluators with human resources for health and public health expertise (Annex 2: CV’s and Conflict of Interest Disclosures). They were assisted by two ESPS note-takers. The consultants reviewed relevant project documents provided by the activity and USAID/Kenya before commencing data collection, enabling the consultants to prepare data collection tools. Data Collection: Based on the document review (Annex 3: Bibliography) and the five evaluation questions, the consultants refined the methodology and respondent selection criteria (Annex 4) and data collection tools for each stakeholder group (Annex 5) that were discussed with USAID/Kenya during the in-brief on September 19, 2014. Data collection in Nairobi took place over 25 days from 18 September to 16 October. This consisted of key informant interviews (KII), roundtable group discussions (RGD), and focus group discussions (FGD) with representatives from the following stakeholder categories: resource partners (RP), USAID collaborators, training institutions (TI), regulatory bodies (RB), and health technical departments (HTD), including the Ministry of Health (MOH) and county health departments (CHD). Seven RGD/FGDs were conducted with stakeholder groups with four to nine respondents (Annex 6). ESPS supported respondents from FUNZO/K targeted regions to participate in the MTR in Nairobi. The MTRT, with an ESPS Public Health Specialist, split into two teams to conduct FGDs and KIIs with beneficiaries in regional areas in Nairobi, Kisumu, Nakuru, and Tenwek, selected by USAID based on the extent of FUNZO/K interventions. The beneficiaries included pre-service trainees (PST) studying health courses in training institutions and existing HCWs undertaking in-service training (IST). The MTR selected professions in shortage from four cadres – nurses, clinical officers, health records information officers (HRIO), and medical laboratory technologists (MLT). Respondents worked in, studied in or lived within proximity of the venues or studied/worked in supported institutions within the same region. In total, 15 RGD/FGDs (92 respondents) and 46 KIIs were carried out (Annex 6). Data Analysis: Data analysis included content analysis of FUNZO/K and health-related documents, KII, RGD/FGD content analysis, gap analysis, SWOT analysis, gender-based analysis, and multiple triangulation approaches (Annex 4). ESPS will organize one national dissemination forum to present key findings, conclusions, recommendations and proposed strategic direction shifts. Limitations: Data collection occurred in Nairobi and in three regional locations – Kisumu, Nakuru, and Tenwek, selected due to the extent of interventions in and around the locations. These locations were also in three of FUNZO/K’s eight regional training hubs – Central, Nairobi, and Nyanza (Annex 8) where pre-service students and in-service healthcare workers could travel to attend the interview venues, such as training institutions. However, these regions may not be representative of the eight regional hubs in which FUNZO/K worked because they were not 8 USAID Cooperative Agreement #AID-623-A-12-00011, February 2012. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 3 marginalized counties. The respondents in the regions were pre-selected randomly from lists of activity-supported individuals. ESPS staff telephoned respondents before the day as well as on the day of the interview or FGD. Despite this, some respondents did not attend, nor did the contingency respondents, due to either transport issues, traffic delays, inability to leave work, or inability to leave their practical on-the-job rural attachment session. Some FUNZO/K and/or health-related quantitative data useful for the MTR were not available. This included the number of online scholarship and loan applicants, limiting the capacity to determine the online success rate and the acceptance rate. Data on pre-service student admissions and graduation statistics, to determine the extent to which the activity’s interventions had increased pre-service admissions, was also not available. FUNZO/K baseline data was insufficient for the consultants to conduct comparisons with activity reports and mid-term data to gauge progress over time. Therefore, progress was measured against its intended targets and results. 4. EVALUATION QUESTIONS & REPORT STRUCTURE The MTR report is structured around the first four evaluation questions (EQ) in the Scope of Work (SOW). EQ 5 relates to recommendations and is answered throughout the report. Each question is addressed, where appropriate, in terms of FUNZO/Ks four results and their four strategic drivers of change (financing, public-private partnerships, technology, and accountability). Evaluation Questions: 1. To what extent has the project’s design facilitated the achievement of FUNZO/K’s mid-term mandates/targets/outcomes/milestones? With regard to the activity’s design, what are the main obstacles and/or weaknesses, opportunities and threats that need further attention? 2. To what extent is the current coordination and collaboration mechanism among key stakeholders (service delivery implementing partners, relevant GOK ministries and training institutions) effective in achieving FUNZO/K’s objectives? 3. To what extent is the activity perceived as relevant, responsive and useful to main stakeholders (MOH and service delivery implementing partners) in addressing the challenge of having skilled and knowledgeable healthcare providers at facility level? 4. What sustainable and innovative Human Resource for Health (HRH) system-related models is FUNZO/K currently implementing? What does the MOH and regulatory boards/councils think of their long-term use? 5. What are the key programmatic and management recommendations that the mission could consider for mid-course changes to the current program design? 5. FINDINGS, CONCLUSIONS & RECOMMENDATIONS 5.1 Q1: IMPACT OF ACTIVITY DESIGN, ACHIEVEMENTS, OPPORTUNITIES & THREATS EQ 1: To what extent has the activity’s design facilitated the achievement of mid-term mandates/targets/outcomes? With regard to the activity’s design, what are the main obstacles and/or weaknesses, opportunities and threats that need further attention? 5.1.1 FINDINGS RESULT 1 – INCREASED NUMBER OF NEW HEALTH CARE WORKERSTRAINED FUNZO/K has three interventions to support the increase of new health workers: (1) a health workforce forecasting system, (2) expansion of admissions capacity of institutions, and (3) a funding scheme. Table 1 summarizes the activity’s mid-term achievements. ACHIEVEMENT HIGHLIGHTS International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 4 Table 1: Summary of Mid-Term Achievements for Intermediate Result 1 RESULT 1: SUPPORTING INCREASED NUMBER OF NEW HEALTH WORKERS TRAINED FUNZO/Ks aims:  to have a training database in place, updated, and accessible to MOH and training institutions,  to increase coordination across pre-service stakeholders to facilitate a national & regional inter￾university/training institution interdisciplinary council, and  to create an enabling environment for pre-service students to access training funds. Progress: a. Produced Health Workforce Forecast Kenya: A Reference Report, September 2013 (forecasting system). b. Developed draft policies and strategies for use of forecasting system. c. Identified operational barriers for use of the forecasting system. d. Established the National Health Human Resource Development Working Group (NHHRDWG). e. Developed Terms of Reference for NHHRDWG. f. Working in collaboration with the private sector alliance as well as a resource partner to develop a training database accessible to training institutions. g. Pre-service scholarships issued. Mid-term target 350, Issued 341 (97% of target) – 183 to Males (54%) and 158 to Females (46%). h. Pre-service scholarship scheme transitioned to a loan scheme at the end of Year 1 (existing scholarships will continue to end of students’ courses). i. Pre-service loans issued. Mid-term target 600, Issued 2,191 (365% of target) – 1,187 to Males (54%) & 1,004 to Females (46%). Progress Against FUNZO/K Strategies  Financing – Scholarship program and loan scheme for student funding for health education and training in place.  Public-Private Partnerships – Discussions have commenced between an investor and a training institution, facilitated by FUNZO/K, to establish a new mid-level training institution in Nairobi. After Year 1 the Higher Education Loans Board (HELB) administered a student loan scheme through the Afya Elimu Fund (AEF).  Technology – FUNZO/K is developing a dashboard and database to capture pre-service data from training institutions (under Result 4). Applications for funding are available online.  Accountability – FUNZO/K monitors and verifies the existence of students in training institutions, reviews their academic records, and reconciles tuition fees. Measures for monitoring progress are planned for 2016/2017. FORECASTING & PRE-SERVICE EDUCATION FUNZO/K developed and launched a forecasting publication in 2013 called Health Workforce Forecast Kenya, in conjunction with the MOH’s Human Resource Department (HRD). It aimed to assist the MOH-HRD by enabling the government to determine which cadres of the health profession were experiencing a shortfall and to influence production trends for those priority cadres (Table 2). For example, the report indicated that the number of nurses will need to increase by more than 50% to meet World Health Organization (WHO) norms, and that there was a significant shortage of clinical officers.9 Stakeholders had low levels of awareness of the forecasting report, and many said they had not seen or used the document. The MOH-HRD has yet to disseminate the forecasting policy. Table 2: List of Priority Health Cadres for Training Support FOCUS CADRES FOR 2012/2013 (in alphabetical order) Clinical Officers (CO) Dentists General Practitioners (GP) Health Records Information Officers (HRIO) Medical Laboratory Technologists (MLT) Nurses Nutritionists Pharmaceutical Technologists Pharmacists Physiotherapists Source: Health Workforce Forecast Kenya, September 2013 9 Health Workforce Forecast Kenya, September 2013, p8 and p12. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 5 FUNZO/K awarded student scholarships before the publication of the forecasting report and policy. Of the 341 scholarships awarded in Year 1, nursing students comprised 60% of scholarships while students in clinical courses comprised 20%. Of the 2,048 loans disbursed in September 2014, nursing students comprised 41% of loans and clinical students comprised 22% of loans. Within cadres, 54% of nursing students with scholarships are female and 46% are male, whereas 60% of nursing students with loans are female and 40% are male. In clinical studies, 29% of scholarship students are female and 71% are male; 34% of clinical students with loans are female and 66% are male (Table 3). Table 3: Scholarship and Loan Students by Cadre and Gender CADRE SCHOLARSHIPS LOAN STUDENTS M F TOTAL M F TOTAL Nursing 95 110 205 (60%) 338 508 846 (41%) Clinical Officer 48 20 68 (20%) 301 155 456 (22%) Public Health Officer* 4 3 7 (2%) 132 100 232 (11%) Medical Laboratory Technicians 10 2 12 (3.5%) 100 53 153 (8%) Pharmacy 10 2 12 (3.5%) 88 29 117 (6%) Health Records Information Officer 6 13 19 (5%) 40 43 83 (4%) Radiography* 7 2 9 (3%) 32 16 48 (2%) Nutritionists 3 4 7 (2%) 12 26 38 (2%) Others 0 2 2 (1%) 49 26 75 (4%) TOTAL 183 (54%) 158 (46%) 341 1092 (53%) 956 (47%) 2048 Source: Database provided by IntraHealth Monitoring and Evaluation (M&E) Department, October 10, 2014 (*not a critical cadre) FINANCING AND PUBLIC-PRIVATE PARTNERSHIPS The main strategies to increase access to training funds were the provision of scholarships and loans, and a public-private partnership (PPP) approach to mobilize additional funds. Scholarships were administered by the activity beginning in Year 1, and 341 were awarded (target 350). Currently 167 students (49%) have graduated, 168 are ongoing (49%), and 6 have dropped out (2%).10 At the end of Year 1, the scholarship program ceased, but students on the program continue to receive funding for the duration of their courses. FUNZO/K introduced a loan scheme from Year 2 - the Afya Elimu Fund (AEF), which is run by the government institution Higher Education Loans Board (HELB). FUNZO/K set a mid-term target of 600 loans. Due to the public-private partnership strategy, the activity secured additional funding from health and non￾health private and government organizations above USAID’s allocation (Table 4). Consequently, 2,191 loans were issued (2,048 disbursed) against the target of 600. Currently 529 (26%) students have graduated and 1,517 (74%) students are ongoing. Although gender targets were not set by the activity, 54% of males and 46% of females received scholarships, and 53% of males and 47% of females received loans. This is the first time HELB extended funding to mid-level (diplomas and certificates) health courses. Table 4: Contributions to AEF for Pre-Service Student Loans ORGANIZATION CONTRIBUTION KES (USD$) HELB KES 150 million (USD $1.68 million) with more funding in 2015 (amount to be determined) USAID KES 116 million (USD $1.3 million) with funding continuing in 2015 (TBD) GOK Ministry of Health KES 10 million (USD $112,170) with a commitment of funding until 2017 (TBD) IntraHealth KES 4.25 million (USD $46,270) – a one-off contribution Family Group Foundation KES 1.25 million (USD $14,000) with a commitment period of 10 years (TBD) Rattansi Trust KES 1.25 million ($14,000) with KES 62,500 committed per year until 2017 I&M Bank KES 620,000 (USD $7,000) with similar amount per year to 2017 TOTAL KES 283,370,000 (USD $3,178,575) Source: IntraHealth M&E Department, October 3, 2014 (statistics to September 30, 2014). Conversion rate is USD1=KES89.15 (October 4, 2014), figures are rounded; http://www.xe.com 10 The dropouts include 3 males and 3 females. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 6 TECHNOLOGY AND ACCOUNTABILITY There has not been significant use of technology in respect to Result 1. In terms of the accountability strategy, interventions under this result area have sought to involve training institutions and communities in identifying needy students especially for the scholarship scheme. The HELB loan scheme also considers the social and economic neediness status of students and applicants score additional points if they come from a marginalized county. CONSTRAINTS, OPPORTUNITIES AND THREATS Constraints for beneficiaries: FUNZO/K aims to create an enabling environment for pre￾service students to access funds, but the online application process may constitute a barrier. The activity reported in its 2012 Annual Report that 1,969 people attempted to apply for scholarships online (the only method of application) with only 701 (36%) successfully submitting their application.11 Scholarship students interviewed confirmed difficulties with internet access when applying, taking between two days and two weeks to complete the process with an average of four days. Similar information was not available for loan applications, although HELB indicated to the MTRT that it took time to publicize the loans in 2013, despite high demand and many applications. Loan students told the MTRT that they took between two days and a week to apply online - with an average of three days. However, both groups noted that the form itself was not a problem to use. The majority of loan students interviewed said that, once loans were awarded, disbursement of funds was often delayed. At times, this led to students being denied access to lectures, examinations and/or hostel accommodation. Students also reported that they did not know when fees and upkeep funds were due to be disbursed because HELB did not communicate regular disbursement dates. There are ten established training institutions located in seven marginalized counties (out of 14 marginalized counties). Due to specific criteria for FUNZO/K’s TI selection, the activity supports just one institution in a marginalized county. Opportunities: FUNZO/K is able to disaggregate results by marginalized and non-marginalized counties based on government criteria, but these are not currently shown in their reports.12 When the MTRT disaggregated the scholarship and loans statistics, they found that of the 34113 scholarship students, 45% were from marginalized counties, but only 5% of loan students were from marginalized counties (Annex 7 Marginalized Counties). FUNZO/K indicated that of 161 applicants from marginalized counties, 155 received funding, but this is not reported. FUNZO/K indicated that only USAID funds are tied to marginalized counties (41% of funding, although annual funds fluctuate according to contributions).14 However, there is currently no targeted percentage of funds allocated to students from marginalized counties in line with the percentage of annual USAID funding. Furthermore, there are 147 training institutions in Kenya with ten located in marginalized counties. FUNZO/K’s students are studying in 50 TIs, of which four are in marginalized counties. Therefore, students are from four of the ten institutions in marginalized counties (Table 5).15 Table 5: Scholarships and Loan by Marginalized County and Gender COUNTY STATUS NO. STUDENTS MALE FEMALE TOTAL Scholarship Students in Marginalized Counties 102 51 153 (45%) Scholarship Students in Non-Marginalized Counties 81 107 188 (55%) SCHOLARSHIPS SUB-TOTAL 183 (54%) 158 (46%) 341 11 FUNZO/K Annual Report 2012, p6; data on login attempts captured and reported. 12 Policy On The Criteria For Identifying Marginalized Areas -FY2011-2014 (Feb. 2013) - see Annex 7 for map. 13 Of the 341, FUNZO/K absorbed 92 students from the Capacity Kenya Project, therefore there were 249 new scholarships. 14 Funding is shown in Section 5.1.3 (USAID has provided KES 116 million out of a total of KES 283.4 million = 41%). 15 The 4 TIs are: Garissa MTC in Garissa; Kilifi MTC in Kilifi; Lodwar MTC in Turkana & Mswambweni MTC in Kwale. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 7 COUNTY STATUS NO. STUDENTS MALE FEMALE TOTAL Loan Students in Marginalized Counties 70 42 112 (5%) Loan Students in Non-Marginalized Counties 1021 915 1936 (95%) LOANS SUB-TOTAL 1091 (53%) 957 (47%) 2048 TOTAL 1274 (53%) 1115 (47%) 2389 STUDENTS FROM MARGINALIZED COUNTIES 172 (65%) 93 (35%) 265 Source: IntraHealth M&E Department, October 8, 2014 (statistics to September 30, 2014) In 2011, the activity baseline survey indicated that a total of 8,561 healthcare students were enrolled in 14 surveyed training institutions while 5,597 graduated that year, with 40% from nursing courses.16 No data were available for comparison with baseline figures to assess FUNZO/K’s contribution toward increased admission or retention rates across Kenyan medical training institutions.17 Anecdotal evidence from students indicated that some would have dropped out if they had not received FUNZO/K funding. In the absence of baseline data showing dropout rates due to lack of fees, it is difficult to assess the impact of financial support on course completion rates. Threats: FUNZO/K aimed to provide greater access to scholarships and loans for students in “hard-to-reach areas and for specific cadres,” especially for students from “resource-poor communities such as northern Kenya.”18 However, a criterion to apply for a loan is that students must already be admitted into a health course. Representatives from County Health Departments (CHDs) and TIs indicated that students from resource-poor communities may face barriers in accessing health courses because they may not be aware of their options for study, they may not meet eligibility criteria, and/or distance to a training institution may present a challenge. Activity reports do not currently disaggregate scholarship and loan statistics by resource-poor communities. However, FUNZO/K noted that “a number of high-need cases did not secure loans due to multiple reasons including incomplete information provided,” and instigated an appeal mechanism19 and established an AEF Loan Oversight Committee. AEF is also contrary to Muslim Sharia lending practices as the loans bear interest. County health department staff indicated this may have been a barrier for students from the Northern Kenya where Muslims make up the majority of the population. For example only three students from Garissa and Wajir counties received a loan.20 HELB is exploring the potential to offer Sharia-compliant funding. Eligibility is means-tested and students from arid and semi-arid remote lands have been supported, but HELB indicated that it is not specifically prioritizing students from these regions. Linking the contribution of FUNZO/K’s achievements to increased numbers of healthcare workers in Kenya would need to be carefully considered and reported. The activity’s M&E Plan and annual reports state an intended result as: increased number of new health workers graduating from pre-service training institutions21 whereas the USAID Scope of Work states: supporting increased number of new health workers trained. 22 Output 1.3 states: overall number of students admitted to pre-service training increased.23 There was no information showing if and how FUNZO/K is measuring its success against admission or graduation rates. RESULT 2 – SUPPORTING HEALTH WORKERS TRAINING NEEDS To support existing healthcare workers (in-service) training, FUNZO/K supported a system for identifying training needs, coordinating trainings with minimal disruption to health services 16 FUNZO/K Report of the Rapid Baseline Survey of the National Training Mechanism in Kenya, September 2012, p12. 17 FUNZO/K Annual Report 2013, p11; an end-of-project data collection is scheduled in 2016/2017. 18 Cooperative Agreement (2012), p17 and 24. 19 FUNZO/K Quarterly Report 2, January to March, 2014, p12. 20 Application rates are unknown. 21 FUNZO/K Monitoring and Evaluation Plan, 2013, p8. 22 Scope of Work, July 2014, p3. 23 FUNZO/K Annual Plan 2013, p11. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 8 provision, and evaluating training performance. A summary of achievements is presented in Table 6. ACHIEVEMENT HIGHLIGHTS Table 6: Summary of Mid-Term Achievements for Intermediate Result 2 RESULT 2: SUPPORTING CURRENT HEALTH WORKERS TRAINING NEEDS FUNZO/K’s aims:  to have a functional in-service health training model that reduces absence from service delivery,  to establish a decentralized mechanism to deliver in-service training through 8 regional training hubs, and  to enable the government to evaluate the performance of in-service health training. Progress: a. Produced a Report of the Rapid Training Needs Analysis of the Health Workforce in Kenya, September 2012 (known as the Training Needs Assessment - TNA). b. 10 of the mid-term target of 14 training institutions (71% of target) are using the TNA. c. Established 8 regional training hubs (already met the 2017 target of 8) and coordination mechanism between national and county level governments. d. Established a decentralized mechanism to manage and deliver in-service training through collaborations between county health departments, training institutions, and service delivery facilities. Supported the in-service training of 7,595 (of a mid-term target of 7,000) health care workers (109% of target) – 2,744 to Males (36%) and 4,851 to Females (64%). e. Supported Training of Trainers (TOT) for 87 instructors – 44 Males & 43 Females (no target was set). f. Supported the in-service e-conversion of curricula modules training for 60 training institution staff – 37 Males (62%) and 23 Females (38%). g. Supported the in-service induction of 2,487 nurses using an e-learning module developed by the USAID Capacity Kenya Project – 757 Males (30%) and 1,730 Females (70%) - (no targets were set). h. Supported in-service County Health Management Team (CHMT) training to 51 managers – 22 Males (43%) and 29 Females (57%). i. Conducted a Post-Training Assessment (PTA) of in-service training in March 2014. Progress Against FUNZO/K Strategies Public-Private Partnerships – Discussions have commenced between potential investors and training institutions, facilitated by FUNZO/K, to increase access to training and innovative models of training. Accountability – Conducted a Post-Training Assessment (PTA) of in-service training in 2014. Evaluation of TNA results into the training curricula is planned for 2016/2017. IN-SERVICE TRAINING MODEL & REGIONAL TRAINING HUBS In 2012, FUNZO/K carried out a training needs assessment (TNA). It identified the following priority skills for in-service training support: cervical cancer screening training, HIV prevention of mother-to-child transmission (PMTCT), antenatal care, newborn care, active management of third stage of labor (AMSTL) and HIV pediatric care.24 TNA recommendations included encouraging the use of lower-cost training venues (e.g., from hotels to training institutions), innovative learning methods (e.g., e-learning), and the adoption of a comprehensive training database. FUNZO/K used these recommendations to inform their in-service training model, which included the establishment of regional training hubs to coordinate, manage and deliver the required in-service training. However, the TNA was conducted at the national level and pre-dates the devolution of powers to counties in 2013, in which CHDs now have the mandate to recruit and deploy healthcare workers. FUNZO/K established regional training hubs and satellite centers by partnering with 14 training institutions (Annex 8). The selection of TIs was based on their ability to offer in-service courses to a wide audience and ensure balanced regional coverage. Interviewed stakeholders viewed the regional training model positively, especially the use of well-established, inclusive TIs that involved public, private and faith-based colleges. TI staff indicated that the activity has significantly increased their capacity to offer in-service and pre-service training by building the skills of the faculty and introducing cost-effective training methods. Faculty staff reported that FUNZO/K helped strengthen linkages with CHDs that were previously weak, largely due to the critical role played 24 FUNZO/K Rapid Training Needs Assessment, September 2012 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 9 by FUNZO/K regional hub training managers who facilitated the connections. Faculty staff also said that the managers encouraged closer relations with organizations that supported national health trainings. For example, one TI indicated that the hub manager connected them with the National AIDS & Sexually Transmitted Infection Control Program (NASCOP), which led to access to curricula guidelines, and subsequent in-service training to Health Care Workers (HCWs) in their regional hub. The activity set a mid-term target to train 7,000 existing HCW. The TNA determined the priority areas for training. FUNZO/K’s first year target of 2,000 HCWs was underachieved (812), but the second and third year targets were exceeded to reach a mid-term output of 7,595 (Table 7).25 County health departments and TIs indicated that the achievement of targets was largely due to the efforts of FUNZO/K’s hub managers and the concept of the regional model for in-service training. Details of the topics covered and the numbers trained are provided in Annex 9. It shows that 72% of in-service training is tied to President’s Emergency Plan for AIDS Relief (PEPFAR) funding and is therefore Human Immunodeficiency Virus (HIV)-related, with 19% of training for maternal and neonatal child health and 9% for reproductive health training. Table 7: Healthcare Worker In-Service Training Achievement per Annual Targets ANNUAL TARGET ACHIEVED % ANNUAL TARGET Year 1 (2012) 2,000 812 41% Year 2 (2013) 3,000 3,846 128% Year 3 (2014) 2,000 2,937 147% Year 4 (2015) 3,000 Annual targets for 2015 and 2016 are dependent upon funding Year 5 (2016) 3,000 TOTAL 13,000 7,595 Source: IntraHealth M&E Department, October 3, 2014 (statistics to September 30, 2014) County health departments indicated that the hubs provided a more cost-effective approach to in￾service training by shifting training venues – from hotels to FUNZO/K-supported training institutions. ACCOUNTABILITY FUNZO/K developed a Post-Training Assessment (PTA) tool in October 2013 and used it to conduct a national assessment in March 2014 in conjunction with the MOH, a USAID collaborator and an activity-supported training institution.26 The PTA reported the highest number of trainings for the prevention of mother-to-child transmission of AIDS (attended by 86 in-service HCWs) and for cervical cancer screening (61 HCWs). The PTA documented a “change in service provision with over 60% reporting positive change” such as increased accuracy in diagnosis, reduced waiting time, improved record-keeping, and improved treatment outcomes.27 The PTA recommended the provision of additional topics and the inclusion of monitoring and evaluation (M&E) training so that health facilities could track, monitor and evaluate their service delivery performance. The HCWs interviewed, who had received training in the past year, indicated that they had applied their learning, and had witnessed improved performance. This included improved breastfeeding advice, improved tuberculosis diagnosis, and quicker turnaround times for blood test results (due to improved communication with the blood testing unit). However, all FGD respondents from TIs stated that they are not currently conducting any post-training follow-up. PUBLIC-PRIVATE PARTNERSHIPS 25 FUNZO/K reports the figure of 7,595 in-service health care workers trained which includes 582 achieved by Capacity Kenya before February 2012. If iHRIS is used as the main data source, it may include HCWs trained outside the review period and some trained by other funding mechanisms. 26 Post Training Follow-up Assessment of Kenya Health Workforce Trainings, March 2014. 27 Post Training Follow-up Assessment, March 2014, pviii-ix. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 10 The regional training model has utilized both public and private sector training institutions as hubs, satellites and as resource partners. The activity supported training has also used both public and private sector health facilities as practicum sites. CONSTRAINTS, OPPORTUNITIES AND THREATS The USAID-funded project, Capacity Kenya, conducted a Performance Needs Assessment which documented the inequality toward women in leadership positions in the health system.28 FUNZO/K’s report, State of Medical Education and Training in Kenya produced in collaboration with the MOH, also discussed women in leadership.29 Both documents recommended the need for equal opportunity policies to enable equality in promotional opportunities. Although FUNZO/K’s in-service training is aimed at increasing skills, not promotions, it has developed a management tool and trained 51 staff in county health management teams (CHMT) – 57% of them women. FUNZO/K did not set measurable gender targets in its M&E Plan, but indicated that a review of assessment reports, from the TIs in the regional hubs, would be conducted to determine gender disparities in order to initiate further action.30 Further action has not been required. Nonetheless, FUNZO/K convened meetings with the Nairobi Women’s Hospital School of Nursing, specializing in gender and sexual violence training, resulting in its involvement in the review of nursing curricula and the inclusion of gender and forensic topics for in-service training.31 However, the activity is yet to adopt Performance Needs Assessment (PNA) or Bottlenecks and Best Buys (BBB) recommendations. FUNZO/K has an opportunity to more clearly report its efforts and successes. In FUNZO/K’s reporting, summary data are not adequately encapsulated to provide an immediate grasp of its cumulative achievements in the provision of in-service training. Although FUNZO/K conducted a baseline survey in 2012, it does not enable an effective comparative analysis over time due to missing data. For example, in-service training and gender data are not captured in the baseline dataset. RESULT 3 – CAPACITY OF TRAINING INSTITUTIONS ENHANCED To strengthen the capacity of 26 training institutions by 2017, the activity has three interventions: (1) curriculum development, (2) improved capacity of faculty and clinical mentorship, and (3) upgrading management skills and equipment. A summary of achievements is presented in Table 8. ACHIEVEMENT HIGHLIGHTS Table 8: Summary of Mid-Term Achievements for Intermediate Result 3 RESULT 3: STRENGTHENED CAPACITY OF TRAINING INSTITUTIONS FUNZO/K’s aims:  to have a proportion of training institutions with joint curriculum assessment & review committees, and  to increase resources available to training institutions to improve the health of Kenyans. Progress: a. Supported 15 training institutions (mid-term target was 14 institutions). b. Mobilized resources in 15 training institutions (mid-term target was 17 institutions). c. Increased the capacity of 10 institutions to review or develop curricula (mid-term was 5 institutions). d. Supported the review or development of 30 health courses (no target was set). e. Supported 10 institutions to implement its revised or new curricula (mid-term target was 5). f. Supported the conversion of 10 modules into e-learning mode (mid-term target was 3-6 modules). g. Supported 21 training institutions to conduct student indexing (2017 target is 26). h. Developed a forecasting dashboard (to capture statistical data from training institutions) piloted in 4 counties: Nairobi, Uasin Gishu, Kilifi, and Mombasa. Progress on FUNZO/K Strategies Financing – Promoted and assisted institutions to develop strategies for cost-effective training for sustainability. 28 Capacity Kenya, Report of the Performance Needs Assessment of the Kenya Health Training System, August 2011. 29 Ministry of Health, State of Medical Education and Training in Kenya: Bottlenecks and Best Buys: experiences of eleven health training institutions in Kenya, December 2013 (FUNZO/K). 30 FUNZO/K Performance Management Plan, February 2013, p7. 31 FUNZO 2012 Annual Report, p29. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 11 RESULT 3: STRENGTHENED CAPACITY OF TRAINING INSTITUTIONS Public-Private Partnerships – Discussions have commenced between potential investors and training institutions, facilitated by FUNZO/K, to increase cost-effectiveness of training for sustainability. Technology – Development of database and dashboard to capture training data from institutions so that it is accessible to MOH and institutions for policy and planning decisions (TNA & forecasting). Accountability – Plan to evaluate results of clinical placement & joint curriculum review committees in 2017. CURRICULUM DEVELOPMENT AND E-LEARNING FUNZO/K support has enabled a number of training institutions to review the curricula of many courses that had not been reviewed for years. Some of the training institutions have also developed new courses that they are rolling out. An example is Outspan Medical Training College that has developed a Phlebotomy course that it is offering to participants from throughout the country. A number of training institutions have developed e-courses for pre and in-service training. Mission for Essential Drugs (MEDS), for example, has already rolled out an e-course on commodity management. IMPROVED FINANCIAL MANAGEMENT AND RESOURCE MOBILIZATION Respondents reported that FUNZO/K support has enabled them to review their processes and to adopt more commercially oriented approaches and this has enabled them to enhance operational efficiencies, reduce costs and generate additional revenue. Several training institutions have developed strategic and business plans and also funding proposals that are at different stages of implementation. Nairobi Kenya Medical Training College (KMTC) reported significant savings following the introduction of a commercial catering service. The college has also introduced a new “Nursing Process” course that is deemed to have the potential of generating significant additional income. Training institutions that have been offering FUNZO/K-sponsored courses have used savings generated to improve infrastructure and to procure teaching aids. PUBLIC-PRIVATE PARTNERSHIPS FUNZO/K’s Public- Private Partnerships (PPP) strategy enabled Kenyatta University to connect with an investor (Abraaj) to discuss the establishment of a mid-level training institution in Nairobi to support increased admissions to health courses. FUNZO/K also facilitated strategic plan development with the Kenya Medical Training College, which has 28 campuses across Kenya, to mobilize resources from non-health private partners to support infrastructure development that would enable KMTC to expand its admission capacity. These are long-term strategies that are yet to materialize. CONSTRAINTS, OPPORTUNITIES AND THREATS Constraints: There have been constraints in the development and roll-out of e-courses because of limited Information Technology (IT) infrastructural including lack of IT equipment and challenges with internet access. Another constraint has been the inadequacy of IT skills within training institutions. Opportunities: E-learning presents an enormous opportunity to training institutions to offer existing and new courses to a broader audience and at lower costs. RESULT 4 – REGULATORY BODIES STRENGTHENED To strengthen Regulatory Bodies (RBs), FUNZO/K has five interventions: (1) identification of performance gaps, (2) regulations for curricula, (3) continuous professional development training, (4) a database to capture regulatory information, and (5) quality assurance. A summary of achievements is in Table 9. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 12 ACHIEVEMENT HIGHLIGHTS Table 9: Summary of Mid-Term Achievements for Intermediate Result 4 RESULT 4: REGULATORY BODIES STRENGTHENED TO ENHANCE TRAINING DEMAND FUNZO/K’s aims:  to strengthen performance gaps identification,  to strengthen regulation of curriculum review, development and implementation  to strengthen the link between professional licensing and continuous professional development (CPD)  to support a database system to capture regulatory information, and  to support standardization and Q u a l i t y A s s u r a n c e ( Q A ) of trainings and health care workers performance. Progress: a. Supported 21 training institutions to gain accreditation to provide in-service CPD training (2017 target is 26 training institutions). b. Supported 7 regulatory bodies (out of 8 in Kenya). c. Supported regulatory bodies to develop CPD guidelines (2 disseminated, 3 in draft form). d. Supported development of curriculum standards (3 in draft form). e. Supported student indexing for 2 new regulatory bodies. f. Supported licensing and registration process for doctors and nurses. g. Commenced policy and development of CPD database. Progress on FUNZO/K Strategies Technology – Database to capture professional development training for re-licensure and re-registration. Accountability – Plan to evaluate CPD policy and database in 2016/2017. TRAINING REGULATORY FRAMEWORK FUNZO/K supported seven of the eight regulatory bodies to formulate a Training Regulatory Framework in collaboration with the MOH-HRD and the Directorate of Health Standards, Quality Assurance, and Regulations (DHSQAR). This framework was regarded by all stakeholders interviewed as an exemplary activity designed to facilitate the production of quality assurance standards, standardized curricula, guidelines for professional registration and re-licensure to ensure healthcare workers and healthcare providers meet the highest standards of delivery. Previously, curricula for different cadres were independently devised by various Non￾Governmental Organizations (NGOs) and private organizations, and therefore not standardized. The performance gap identification led to the activity’s support in priority areas such as forecasting, human resource management, database interoperability, training standards and accreditation of training providers, primarily to establish performance and clinical practice norms. The MOH confirmed that these were aligned with the government’s Health Sector Strategic Plan III and Vision 2030. The training regulatory framework has informed the development of core curricula and CPD guidelines including Cross-Cadre CPD guidelines. TECHNOLOGY AND ACCOUNTABILITY Regulatory bodies are currently receiving support to develop and maintain their databases from the CDC/EMORY program. There are ongoing efforts to ensure that the training databases are linked with the FUNZO/K- supported iHIS train. FUNZO/K and CDC/EMORY are supporting regulatory bodies to develop and improve the student indexing system to ensure data on the number of students undertaking different pre-service training courses can be tracked over time until they complete their studies and are registered. CONSTRAINTS, OPPORTUNITIES AND THREATS International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 13 Constraints: Some of the regulatory bodies especially the new one have significant capacity constraints in respect to funding and staffing. Devotion of health services and the health workforce has added a layer of complexity in the regulation of medical practitioners. Opportunities: A comprehensive indexing system for all medical pre-service courses will significantly enhance data accuracy for the number of pre-service students and support longer term health workforce projections. 5.1.2 CONCLUSIONS FUNZO/K has achieved or exceeded its mid-term targets and appears to be on track to achieve its end-of-term targets. The MTRT found that stakeholders at all levels perceived the activity’s design to be grounded in a sound capacity building framework. Result 1 – Increased Number of New Healthcare Workers Trained: The MOH has an approved health forecasting system, but it is yet to be rolled out. FUNZO/K’s allocation of scholarships was not strategically aligned with the forecasting report because the forecasting report had not yet been published. However, due to previous experience in the Kenyan health sector FUNZO/K awarded scholarships to students in courses leading to employment in critical cadres and met the WHO 2012 recommendation to increase nurses by more than 50%. Nursing students comprised 60% of scholarships while students in clinical courses comprised 20%. FUNZO/K administered scholarships and therefore had more control over outcomes. The AEF committee and HELB administered the student loans from Year 2 and this may have resulted in less control over alignment with forecasting data. Nursing students comprised 41% of loans and students in clinical courses comprised 22% of loans. Some students may be disadvantaged by the online funding application process while interest￾bearing loans could deter Muslim students from seeking funding. However, it is not known how many students this has affected. The financing strategy has led to support from private and public sector organizations, thus enabling the activity to significantly exceed its target for issuing loans. However, it is not clear how FUNZO/K is measuring its progress toward its intended result – i.e., are admission or graduation rates rising as a result of its scholarship program and loan schemes? This is because FUNZO/K does not demonstrate how it contributes to increased entry into health courses as students must already be admitted into a course before applying for AEF. Result 2 – Supporting Current Health Worker Training Needs: A TNA was developed at the national level in 2012 and pre-dates the devolution of powers to counties. Currently ten of the targeted 14 training institutions are using the TNA. By establishing eight regional training hubs to manage the delivery of in-service training – and shift it from hotels to training institutions, FUNZO/K has established an inclusive, collaborative regional network. This network has strengthened formal linkages between CHDs and training institutions that were previously weak. However, funding is tied to the President’s Emergency Plan For AIDS Reduction (PEPFAR) and thus 72% of in-service training was Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome (HIV/AIDS)-related. Post-training assessments are not yet conducted at the county level. Overall, the in-service regional training model is functioning well. It is regarded as a model for increasing the capacity of training institutions to conduct training through sharing curricula and training materials and better linkages between training institutions, health facilities and county health departments. Result 3 – Strengthened Capacity of Training Institutions: FUNZO/K has strengthened the capacity of targeted training institutions to review and develop curricula and also convert existing pre and in-service courses and modules into e-versions. Limitations in IT infrastructure and skills constitute significant constraints in rolling out e-courses. The capacity of training institutions to mobilize resources and improve operational efficiencies has also been enhanced International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 14 through the initiatives of FUNZO/K business advisory services. Most of the training institutions have developed comprehensive business plans that are being implemented. Result 4 – Regulatory Bodies Strengthened: Extensive progress has been made toward the development of standardized continuous professional development guidelines, curricula, and standards, as well as the accreditation of training providers. CPD training must be provided by accredited institutions to be recognized; healthcare workers who undertake specified accredited in-service training will then receive credits toward professional registration and licensure. Therefore, strengthening regulatory bodies has been an important strategy to enhance quality of healthcare training. FUNZO/K’s training regulatory framework is a model that could be replicated in other USAID activities. 5.1.3 RECOMMENDATIONS Result 1 – Increased Number of New Healthcare Workers Trained: The MTRT recommends that FUNZO/K review the administration of the loan scheme to clearly define the selection criteria (especially if funding sources have different criteria) and ensure equality of opportunity – i.e., to ensure that students are not disadvantaged by the online application process, students from marginalized areas are adequately represented, and HELB continues exploring a Sharia-compliant funding mechanism for students where interest-bearing loans may be a barrier to application. The activity should explore the potential for closer alignment of loans with critical cadres documented in the current or any future revised forecasting report. FUNZO/K should report the progress of loan results by disaggregated statistics and tables (including marginalized counties) for online application attempts, successful online submissions, and loans awarded against funding type, cadres of study, drop-outs and retention rates. FUNZO/K should track the outcome of the schemes in order to assess its impact on admission and completion rates. The MTRT recommends that FUNZO/K should improve the tracking and documentation of the outcomes of its coordination mechanism (NHHRDWG) and its attempts to enhance the expansion of admissions capacity of TIs to meet health workforce needs. Result 2 – Supporting Current Health Worker Training Needs: FUNZO/K should support county health departments to conduct TNAs at the county level so that they can more directly influence in-service training that leads to skills in areas of need. The MTRT recommends that the activity conducts annual post-training assessments in 2015 and 2016 and continues training MOH, CHD and TIs on monitoring and evaluation to ensure that in-service skills and knowledge are applied and to enable the government to evaluate the performance of in-service health training. FUNZO/K should track and document changes in service delivery indicators (such as service coverage, waiting time for services, and quality of care indicators such as client satisfaction) in facilities and regions that benefit from in-service training. Result 3 – Strengthened Capacity of Training Institutions: The MTRT recommends accelerating efforts to ensure that supported training institutions adopt the TNA and policies, guidelines, and standards developed throughout the FUNZO/K activity. FUNZO/K should accelerate efforts to materialize public-private partnerships for increased admissions capacity, and continue building the capacity of the in-service regional training network by supporting implementation of their business plans and public-private partnership discussions. Finally, FUNZO/K should ensure clinical instructors receive pedagogy skills through Training of Trainers to improve the quality of clinical placement experiences, and to accelerate the clinical placement mentorship scheme. Result 4 – Regulatory Bodies Strengthened: The consultants recommend that FUNZO/K focus on the full implementation of CPD guidelines and core curricula, and facilitate a mechanism for online access, via a portal, to RB documents. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 15 5.2 Q2: COORDINATION & COLLABORATION MECHANISM EQ 2: To what extent is the current coordination and collaboration mechanism among key stakeholders (service delivery implementing partners, relevant GOK ministries and training institutions) effective in achieving the activity’s objectives? 5.2.1 FINDINGS COORDINATION AND COLLABORATION MECHANISM Service Delivery Implementing Partners: As part of FUNZO/K’s strengthening of health training institutions, and the review or development of curricula, service delivery implementing partners were involved in ensuring clinical needs were met, especially for clinical placement instruction and mentorship. Service delivery partners indicated during the MTR that they prefer more collaboration and involvement in FUNZO/K interventions to ensure quality delivery at the facility level. Government Ministries: To facilitate and improve inter-organizational collaboration FUNZO/K initiated the GOK-led National Health Human Resource Development Working Group (NHHRDWG) monthly meetings. It is the forum for addressing and solving critical implementation issues. The NHHRDWG enabled the activity to align all interventions under each result within one overarching working group and create an oversight and endorsement mechanism among relevant government departments, regulatory bodies and councils, the eight regional training hubs and the private sector. One of the main activities at the national level was the identification of performance gaps to facilitate the forecasting report. Another main activity was the identification of database and dashboard barriers to interoperability with databases from relevant government departments responsible for training, human resource, or workforce data. This includes technical working groups from, for example, the Ministry of Finance and the Ministry of Medical Services. FUNZO/K uses this mechanism to gain endorsement for regulations, protocols, industry standards and training requirements. For example, MOH-HRD, other MOD departments, NHHRDWG, RBs and FUNZO/K worked collaboratively to build the capacity of selected training centers from each regional hub to offer e-learning, generate revenues and manage costs, assess investment opportunities through PPPs and prepare plans for funding requests (such as grant applications for equipment upgrades). Training Institutions: Stakeholders interviewed during the MTR, particularly training institution representatives, indicated that “FUNZO/K is the reason we are now able to access the ministry” especially in relation to curriculum reviews and development: “FUNZO/K helped and changed the entire situation.” The curriculum review process was collaborative, and TI representatives confirmed that the process was inclusive. To facilitate inter-regional coordination, the activity established eight regional hubs, each comprising a lead training institution, a satellite center (secondary institution), partners, the government county health departments, and hospitals (identified in Annex 8). Within each hub FUNZO/K appointed a regional training manager. The hubs, through the county health departments, are responsible for overseeing the rollout of in￾service training. All stakeholders confirmed the high functioning collaboration. However, the CHD in Garissa, a marginalized county, indicated that the Northeastern hub required more connection with other regions because it was a disadvantaged, remote county with weaker structures and linkages. CHALLENGES TO COORDINATION AND COLLABORATION FUNZO/K used criteria for selecting lead training institutions in their hubs, such as: existing courses, links to clinical placement sites, ability to meet regulatory standards, regional representation, location, ability to meet training needs and the ability to support innovations in International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 16 teaching delivery methods.32 Therefore, some TIs felt excluded from regional training delivery, and a representative indicated that the roles between resource partners and TIs were not clearly defined. For example, a TI that developed courses, through inter-organizational collaboration, was not chosen to deliver the same courses. Resource partners, service delivery partners and TIs expressed the view that they were not optimally involved in training delivery, despite their regional coverage, networking and health-related expertise. CHDs indicated that there was minimal collaboration during e-learning conversion because TIs were developing niche units competitively rather than collaboratively. DECENTRALIZATION AND DEVOLUTION The activity commenced implementation before the government decentralized administrative powers – therefore the forecasting system and the TNA were developed in 2012 under the provincial structure. Although FUNZO/K is a national health training support initiative, it also designed interventions to strengthen counties. In the first year of devolution, CHDs passed their first budgets, devised strategic plans and recruited their own staff. FUNZO/K placed an advisor within the MOH-HRD from August 2012 to coordinate national interventions. The MOH indicated that despite initial challenges in defining mutual working methods, they appreciated the advisor’s commitment. Respondents reported challenges in coordination due to fragmentation of training, regulation and employment mandates between national and county level institutions. Pre-service training and medical regulation are national mandates while public sector health staff are employed and managed by county governments. Regulatory bodies have discussed the creation of county chapters, and FUNZO/K plans to support the devolution of RBs in Year 4. The Nursing Council of Kenya (NCK), the largest regulatory body, has five regional chapters, but stated that the challenge for decentralized RBs is the cost of office rental.33 At the MTR roundtable discussion RBs raised the potential for cross-cadre regional offices, but the consultants found that generally RBs still operate independently of each other in discharging their mandates. 5.2.2 CONCLUSION The administrative changes brought about by devolution are significant. The establishment of regional training hubs, enhanced by the critical role of the hub managers and the inclusive composition of hubs (county, institutions, hospitals, private institutions and faith-based organizations), provides a mechanism that supports sustainability of inter-regional coordination. While FUNZO/K has attempted to facilitate communication between all levels of the health training chain (which is generally working well for curricula review and development), there is still disconnect in terms of planning (national level), training (national level), and the recruitment of health workers (county level). Overall, service delivery collaborators desired more involvement in training delivery, rather than only for curriculum development or technical advice. RBs also expressed a need to be more closely connected to counties, but cost implications are current challenges. 5.2.3 RECOMMENDATIONS FUNZO/K should focus specifically on the North Eastern regional training hub and associated training institution for strengthening. This would enable the hub to better serve its constituents by strengthening in-service training to improve the skills of HCWs, enabling TIs to improve admissions, and its capacity to develop appropriate curricula, modules for e-learning conversion and improved clinical placements. FUNZO/K should also consider ways to include the six public, 32 FUNZO/K 2012 Annual Report, p29 33 The Nursing Council of Kenya had an internal target for 2014 to establish 8 regional offices. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 17 private and faith-based medical colleges in marginalized counties for networking opportunities.34 The consultants also recommend more inclusion of service delivery implementers in the delivery of in-service training, and in the conversion and adoption of modules for alternative delivery modes. 5.3 Q3: PERCEPTIONS OF MOH AND SERVICE DELIVERY IMPLEMENTING PARTNERS EQ 3: To what extent is FUNZO/K perceived as relevant, responsive and useful to main stakeholders (MOH and service delivery implementing partners) in addressing the challenge of having skilled and knowledgeable healthcare providers at facility level? 5.3.1 FINDINGS The majority of respondents interviewed stated that FUNZO/K’s mandate is relevant and the organization has a competent, committed and accessible leadership and management team that has forged close working relations with stakeholders and is making a positive change in strengthening training capacity and systems in the health sector. PERCEPTIONS OF MINISTRY OF HEALTH FUNZO/K aligns its activities with those of MOH and has seconded one staff to the MOH HRD department. The activity ensures that its training programs are relevant and responsive by using TNA findings and by working closely with MOH technical departments and programs such as NASCOP. FUNZO/K has harmonized its training programs by basing all training on national curriculum and guidelines. FUNZO/K has used NASCOP’s Master Trainer program to develop trainers for the programs it supports. The regional training model has enabled FUNZO/K to respond to training needs at the county level across the country. Development of the core curriculum has been useful in helping training institutions such as Nairobi University review their curricula after many years. Afya Elimu Fund has made a positive difference especially in hard-to￾reach areas. By supporting the National Human Resource for Health Technical Working Group (NHRHTWG) monthly meetings FUNZO/K ensured that the meetings were held regularly. PERCEPTION OF SERVICE DELIVERY IMPLEMENTING PARTNERS FUNZO/K’s training plans are largely based on gaps that service implementing partners identify. However the activity is not responsive to emerging training needs. Service delivery partners are on the ground and are best placed to respond in a timely manner to emergent training needs. Although FUNZO/K has held some planning and coordination meetings with service delivery partners, these meetings are not regular and follow-up is inadequate. The FUNZO/K regional training model is good but needs more resources for it to be fully effective. FUNZO/K training courses are largely aligned to PEPFAR priorities while county governments and health facilities often have different or additional priorities especially maternal and child health. Relative to need, FUNZO/K’s supported training cover a relatively small number of health care workers (an average of 60 per county per year) and are spread over multiple courses and this may reduce their impact. Therefore, there is need for more prioritization targeted at the unique needs of counties. The activity has done a good job in engaging county governments to support health training. Some of the training institutions used by FUNZO/K do not have the requisite technical capacity and skills needed to deliver quality training. Most partners do not have access to information held in training databases developed by the activity. 34 Ortum Mission Hospital in West Pokot; Wamba Mission Hospital in Samburu; St. Luke Catholic Hospital in Kaloleni & North Coast in Kilifi; St. Joseph's Hospital in Kilgoris, Narok; & North Eastern College of Health Sciences in Garissa. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 18 5.3.2 CONCLUSIONS FUNZO/K is working closely and collaboratively with MOH. MOH is satisfied with the support it is receiving from FUNZO/K and is of the view that this support is making a positive difference in strengthening health training. The activity’s training programs are aligned with identified national priorities and are based on approved national curricula and guidelines. In developing its training plans, FUNZO/K takes into consideration gaps identified by service delivery implementing partners, but it is perceived to be not adequately responsive to emergent training needs. FUNZO/K’s training course offerings may be too broad and this could compromise the impact of its training. 5.3.3 RECOMMENDATIONS FUNZO/K should continue working closely with MOH. The role of service delivery implementing partners in training should be reviewed. One option is to create two categories of training programs, one that should be offered by service delivery partners while the other should continue being managed by the activity. Another option is to have SDPs coordinate all training with FUNZO/K focusing on building the training capacity, developing a network to deliver training and strengthening regulation and quality assurance. 5.4 Q4: INNOVATION & SUSTAINABLE HRH MODELS EQ 4: What sustainable and innovative HRH system-related models is FUNZO/K currently implementing? What does the MOH and regulatory boards/councils think of their long-term use? 5.4.1 FINDINGS TRAINING REGULATORY FRAMEWORK FUNZO/K supported seven of the eight RBs to formulate a Training Regulatory Framework (TRF) with the MOH-HRD and the Directorate of Health Standards, Quality Assurance, and Regulations (DHSQAR) under Result 4: regulatory bodies strengthened to enhance training demand. MOH and RBs indicated that this was a major success for FUNZO/K and that it was a highly sustainable framework which has been readily accepted and implemented at the national level. Furthermore, all stakeholders have accepted the framework as a “long overdue” requirement for establishing and guiding long-term system-wide quality improvements across the health training sector. CPD and Cross-Cadre Accredited Guidelines: FUNZO/K supported RBs in the development of continuous professional development (CPD) guidelines linked to registration and re-licensure. Two RBs finalized and disseminated their CPD guidelines in November 2013, but these are not yet operationalized. CPD guidelines are pending finalization for three RBs, with training and accreditation standards in draft form for three RBs, and student indexing is almost completed for two RBs (Annex 11). This process was useful for two newly established bodies: Kenya Nutritionists and Dietetic Institute (KNDI) and Public Health Officers & Technicians Council (PHOTC), as it enabled them to accelerate their response to their constituencies. FUNZO/K supported the MOH-DHSQAR in the development of a National CPD Regulatory Framework. It has been finalized and ratified by stakeholders, and is expected to be disseminated in October 2014. This is the first time all RBs participated to formulate cross-cadre guidelines. Accreditation of Training Providers: FUNZO/K contributed to the formation of a group to accredit cross-cadre CPD providers to ensure quality of health training providers (because RBs do not conduct their own training). RBs were currently waiting for the 47 county health departments to each select a CPD coordinator. The CHD-CPD Coordinator’s role will include facilitating reporting procedures for CPD providers, and assisting in monitoring compliance. The activity International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 19 supported TIs to achieve accreditation status, in accordance with RB guidelines. At mid-term 21 TIs out of a targeted 26 by 2017 have been accredited to provide CPD training. However, CHDs confirmed that there is no timeline for the rollout of implementation. The nursing council is an example of the accreditation process’s success. FUNZO/K assisted NCK to develop their Training and Accreditation Standards (TAS), which is currently being edited. The NCK said it was “a major milestone.” Nursing schools are already using the checklists even though the document has not been disseminated. Core Curricula: FUNZO/K supported 7 RBs to review or develop their core curricula (syllabus) that outlines minimum content and competencies essential to guarantee professional license. Many boards had not revised their CPD curricula for several years. Only one core curriculum developed by a RB has been disseminated. Four RBs will disseminate theirs to accredited providers before the end of 2014. The RBs embraced the improvements to industry standards. Long-established training providers were initially slow to buy-in, but now recognize that core curricula provide essential competencies, while still enabling providers to customize courses to maintain their niche market. The KMPDB indicated that a university is already using their new core curriculum. Curriculum Review/Development: FUNZO/K’s support to ten TIs for curricula review or development (exceeding mid-term target of five institutions) resulted in the review of 17 curricula and the development of ten new curricula responsive to county needs. Of 27 courses, 20 have been rolled out in TIs for implementation (Annex 9). For example, one institution began a new phlebotomy course across Kenya. Only one course was developed using the core curriculum because RBs are yet to finalize their core curricula. It is too early to determine the number of students currently using the new/reviewed curricula as implementation has only recently started. Registration and Licensing: Established RBs had existing student indexing in place. The activity supported the newly established RBs, KNDI and PHOTC to develop guidelines and criteria for student indexing (Annex 11). By mid-term, FUNZO/K has supported 21 TIs (from a total of 26) to conduct student indexing in accordance with RB guidelines. However, there are no linkages between the CPD points and in-service training data held in iHRIS Train. E-LEARNING INNOVATIONS In order for training institutions to become more cost-effective and increase admission capacity, FUNZO/K supported the conversion of existing or newly developed curriculum modules into open, distance, and e-learning (ODeL) activity-supported training on content conversion (Annex 10). FUNZO/K used the USAID-funded Capacity Kenya e-learning module to roll out the FUNZO/K e-induction training (to 2,487 nurses which included 41 master trainers). Following the e-conversion of some of the induction modules, the face-to-face sessions were undertaken over a two-day period, shorter than the previous five days taken using the traditional mode. The cost of e-induction was $113,200 compared to an estimated $328,000 for traditional mode (50%-70% reduction).35 Costs were lower due to the reduction of the number of training days, accommodation and per diems. A total of 60 TI officers have been trained in e-conversion, and 10 courses have been converted (from a 2017 target of three-six courses) and rolled out. One TI (MEDS) has implemented an e-learning unit with 47 participants with six who completed at the end of September 2014.36 The TI indicated that the course fees are $120 compared with $435 for face-to-face training (70% cheaper). Their challenge was the students’ complaints of slow or sporadic Internet access, but overall it was received positively. Another TI has advertised their e￾learning unit and received over 2,000 applicants for selection, with commencement in October 2014. However their greatest challenge is Information, Communication and Technology (ICT) infrastructure and maintenance. Therefore TIs indicated that a blended approach to training (traditional and online) is more realistic. Scholarship and loan students interviewed indicated that 35 FUNZO/K Annual Report, 2012, p23 36 Discussed during an FGD session International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 20 they would embrace e-learning, despite having difficulties with Internet access, but said practical teaching should comprise more units than online units. TIs indicated that FUNZO’s support in e￾learning was appropriate, timely, and a sustainable cost-effectiveness innovation for health teaching and learning. TRAINING FORECASTING FUNZO/K supported the development of Integrated Health Resource Information System (iHRIS) Train database of training information which has been implemented in training institutions. TIs reported that they routinely use the system to input and report their in-service training data. The activity is currently integrating the Master Facility List to link iHRIS Train with the national District Health Information System (DHIS). MOH stakeholders indicated that the system is currently not being used by RBs to support the CPD program to monitor points toward registration and licensure. The system is not currently linked with the national iHRIS to update individual health workers training information, nor does it capture pre-service training data. This data is currently managed by the CDC-Emory program and the two systems lack interoperability. Pre-service data is only available for programs that have a student indexing system. This excludes most post￾graduate training programs. Funzo is continuing discussions with partners, stakeholders, regulatory bodies and TIs to facilitate improved integration and interoperability among data systems. Stakeholders said there was a need to make accurate forecasts for future pre-service training and had concerns that university and college health admission and graduation rates were not yet aligned with workforce needs. FUNZO/K’s interventions to increase the capacity of TIs include their ability to enter their training data into a database, which they and the MOH can access for policy and planning purposes. The MOH noted the interoperability of existing training and human resource databases is a threat to the adoption of the forecasting system. The activity has supported the development of an iHRIS, which aims to be the main database for the MOH-HRD that links education, human resource and workforce information across the health sector for forecasting purposes. The iHRIS train database does not currently link to iHRIS manage, the MOH’s database on human resource data, or with other databases, e.g., those developed by USAID HRH Capacity Bridge37 to capture workforce data. Discussions with FUNZO/K revealed the roles of FUNZO/K and HRH Capacity Bridge are inadequately defined for the purposes of workforce and pre-service training forecasting. 5.4.2 CONCLUSIONS A comprehensive regulatory framework for continuous professional development training of health professionals is in place, but full implementation has not yet occurred. The approach is highly likely to be sustainable because it has been readily adopted by the MOH. Training institutions have increased their capacity to convert modules into e-learning delivery modes to increase admissions and facilitate cost effectiveness, but they require upgraded computing equipment. FUNZO/K developed and launched a forecasting publication in conjunction with the MOH’s HRD to assist them with workforce policy and planning. It identifies cadres of the health profession experiencing a shortfall, thereby requiring future training in those critical cadres (such as nursing and clinical officers). However, there are low levels of awareness of the forecasting report and it is currently not used for planning purposes. The activity supported the iHRIS Train database, which has been implemented in training institutions. It is at the early stages of implementation and its data is not being fully used due to a lack of access for all stakeholders. Hence, iHRIS Train is currently an inventory tool rather than a planning tool to support health forecasting. FUNZO/K is facing a challenge to obtain full functionality, awareness and use of the forecasting system. Regulatory bodies are satisfied with innovative interventions introduced by 37 HRH Capacity Bridge is the Human Resource for Health activity related to the health workforce, such as occupational information. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 21 FUNZO/K including the training regulatory framework, accreditation of training providers and development of CPD guidelines which they view as sustainable. MOH is satisfied with the development of the training regulatory framework but has concerns about limitations of database interoperability that is limiting optimal use of existing training and regulatory information and forecasting. The use of existing and well-established training institutions in developing a regional training model strengthens sustainability of the in-service training result area. The use of HELB to administer and fund mid-level health training is both innovative and sustainable. 5.4.3 RECOMMENDATIONS The MTRT recommend that FUNZO/K accelerates the full implementation of the newly developed CPD guidelines and core curricula, and encourages training institutions to adopt the core curricula and other regulatory body regulations. Regulatory body guidelines, standards, regulations and core curricula should be made readily available online in one integrated and coordinated portal. The consultants recommend that FUNZO/K continues to strengthen iHRIS Train and expands access to and use of the data for planning and integration/interoperability with other HRH databases in Kenya. The activity should also review the forecasting report and system to address the gaps and barriers identified by the MOH. This will ensure its effectiveness in forecasting pre-service education needs to meet the shortfall of critical cadres for workforce planning. FUNZO/K’s mandate for forecasting should be limited to health education and training, and especially forecasting pre-service training trends. FUNZO/K needs to play a greater role in collecting and using forecasting data to influence policy makers, with the aim of achieving an improved balance between training output and national needs. The roles of FUNZO/K and HRH Capacity Bridge in forecasting should be better delineated while enhancing collaboration. This is especially important at the county level because of the challenge arising from counties having the mandate to recruit and deploy healthcare workers, although the responsibility for training remains a national function. 6. KEY STRATEGIC FUTURE DIRECTIONS The government’s move to a decentralized system beginning in March 2013 occurred after FUNZO/K began implementation, and therefore national and county level responsibilities have changed. For example, the recruitment of healthcare workers is the responsibility and mandate of each county, but the training of health workers remains the responsibility of the national government. The forecasting report and the TNA thus pre-date decentralization. With the devolution mandate counties are requesting support to conduct TNAs. FUNZO/K should strengthen county level management, especially for the coordination of in-service training that meet county-specific needs. FUNZO/K’s major achievements have been based on establishing a buy-in of frameworks, systems and mechanisms for regulations and quality assurance. Most of these are in the early stages of development, draft form or recently completed. Acceleration of their implementation should be FUNZO/K’s focus for the remaining years. The activity should establish an e-portal to give stakeholders access to the newly-created documents, particularly as many have been developed for the first time in the nation’s history. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 22 ANNEX 1: SCOPE OF WORK PROJECT DETAILS: Project Name: FUNZOKenya Implementing Partner: IntraHealth International Inc. Agreement Number: AID-623-A-12-00011 Project AOR: Peter Waithaka Life of the Project: February 24, 2012 - February 23, 2017 Total Funding: $ 40,000,000 Period of Project to be Evaluated: February 24, 2012 – May 31, 2014 Type of Evaluation: Performance Evaluation (Mid Term Review) Completed Evaluation by: IBTCI A. Background/Project Overview/Problem Statement: FUNZOKenya project is a national level training mechanism that focuses on strengthening GOK health systems for human resources for sustainable programming. Specifically, the project aims to increase the number of health care workers with updated skills and knowledge in health care provision. This mechanism works closely with Ministry of Health (MOH), the newly-created National Health Training Working Group (NHTWG), training institutions and service delivery partners to ensure that health care workers have needed skills and knowledge to deliver quality health services in HIV and AIDS, malaria, family planning and tuberculosis, and Maternal and Child Health (MNCH). This is a five year activity implemented by IntraHealth as the prime partner with Results for Development, University of North Carolina, Great Lakes University, University of Nairobi, Training Resources Group and Kenya Healthcare Federation active as the sub-partners. A.1.1. Program Goal FUNZOKenya activity is aligned to the USAID implementation Framework 2010-2015 goal of “sustained improvement of health and well-being for all Kenyans”. It responds to the result area 2 of “Health Systems Strengthened for sustainable delivery of quality services” as shown on the results framework below: Results Framework Strategic Goal: Sustained improvement of health and well-being for all Kenyans Cross-Cutting Elements: Whole Market, Innovation, Gender-Focus, Youth-Focus, Equity Result 1: Strengthened leadership, management and governance for sustained health programs Result 2: Health systems strengthened for sustainable delivery of quality services Result 3: Increased use of quality health services, products and information Result 4: Social determinants of health addressed to improve the well-being of targeted communities and populations The strategic goal of the project is to improve access to and quality of health workforce training. The results and the corresponding intermediate results are detailed out on the activity agreement document for your reference, please review the agreement. A.1.2. Development Hypothesis If the Ministry of Health expands and improves pre-service trainings, develops and implements innovative models for in-service trainings, and strengthens the capacity of pre-service educational institutions then there will be increased capability to plan for, recruit, train, regulate and retain adequate and appropriate human resources for health at all levels of the health system. A.1.2. Program Objectives International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 23 The strategic objective of the project is to strengthen human resources for quality health services through strengthened training system. At the design, this activity was planned to implement the following strategies: 1. Financing: Realign financial incentives for training toward rewarding high performance, broker partnerships with non-health private-sector entities to invest in health workers and health worker institutions, and prioritize cost-recovery models of training delivery. 2. Public-private partnerships: Capitalize on the growing number of private-sector training and service delivery institutions and mobilize the non-health private sector such as technology corporations and financial institutions to engage in the health system. 3. Technology: Harness existing technology and introduce new, adaptive and scalable technologies to greatly expand training relevancy, consistency, accessibility and quality— including for instruction, faculty development, planning and management, and regulation. 4. Accountability: Assist Kenyan training institutions and regulatory bodies to be more responsive to the health needs of the communities they serve, the health workers enrolled in training, and the health system—and ultimately to be more responsible for the expected outcomes. A.1.3 Program Activities Over the course of implementation, the activity was planned to implement the following activities within each of the result areas: RESULT 1: Supporting Increased Number of New Health Workers Trained Specific activities: • Strengthen forecasting of Human Resources for Health production. • Support expansion of admission capacity of local training institutions including Kenya Medical Training College and University of Nairobi. • Increase accessibility to loans and scholarships to the rural resource poor applicants (scholarships and loan schemes). RESULT 2: Supporting Current Health Workers Training Needs Specific activities: • Develop system for identifying training needs and gaps. • Strengthen coordination of trainings with minimal disruption to health service provision. • Develop systems for evaluating the performance of training effectiveness for quality and standard assurance. RESULT 3: Strengthened Capacity of Training Institutions Specific activities: • Develop content and courses based on set curriculum. • Improve capacity of faculty and clinical mentorship. • Upgrade training facility management systems and infrastructure. RESULT 4: Regulatory Bodies Strengthened to Enhance Training Demand Specific activities: • Strengthen performance gaps identification • Strengthen regulation of curriculum review, development and implementation • Strengthen the link between professional licensing/retention and trainings for CPD • Support a database system to capture regulatory information • Support standardization and quality assurance of trainings and health workers performance B. STATEMENT OF WORK B.1 Mid-Term Review (MTR) Purpose International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 24 The purpose of this Statement of Work (SOW) is to conduct a Mid-Term Review (MTR) of the FUNZOKenya activity. The overall purpose of the review is to assess the progress made towards the strategic objective of ensuring increased availability of skilled and knowledgeable health care providers. The review seeks to identify significant factors that are facilitating or impeding the delivery of a strengthened health work force capable of providing quality and sustainable health services. In particular, the review is to look at how well the activity has implemented the proposed strategies, what is working and what is not working for every strategy. This will be achieved by identifying strengths and opportunities for leveraging positive results as well as risks and challenges to achieving the proposed results. It is expected that the results of this MTR will inform the development of future work plans including change and/or refinement of implementation strategies for the remaining years. B.2. Audience and Intended Use The primary audience and users of the MTR findings, conclusions and recommendations are the US Government Agencies (DOD, CDC and USAID) and the Government of Kenya. Secondarily, the findings, conclusion and recommendations will further benefit IntraHealth International, USAID/Washington and the global community working in the area of human resources for health in developing countries. It is expected that the results of this mid-term review will inform all decisions on project modifications in terms of scope and coverage of project activities in the remaining years of project implementation. B.3. MTR Objectives and Key Review Questions The overall objective for this MTR is to: 1. Review implementation approaches for every strategy and determine the extent to which the project is on-track in achieving its intended outcomes by finding out: a) what is working well and what does the project need to do more of, and b) what is not working well and what does the project need to do less of? 2. Identify what the project needs to change to achieve expected results and outcomes by the end of the project. 3. Provide recommendations based on key findings and conclusions about any required changes that will make the project improve on the identified weak and sustain areas of strength. Specific objectives: 1. Review implementations approaches for every result area by the four proposed strategies (financing, public-private partnerships, technology and accountability) and determine what works well and doesn’t work well. 2. Discuss and develop practical and evidence-based approaches for every implementation approach found in Item 1 above not to be working well. 3. In particular, assess the activity’s coverage of health care workers with priority in-service trainings as identified by the United States Government (USG) implementing partners. 4. Assess the effectiveness of coordinating and collaborating mechanisms among key stakeholders’ e.g. relevant Government of Kenya (GOK) ministries, service delivery implementing partners, and training institutions in achieving activity objectives. 5. Assess the effectiveness of the activity’s implementation systems and structures (financial, managerial, internal/external coordination & collaboration, Monitoring &Evaluation) in achieving project objectives. Key Review Questions 1) To what extent has the project design facilitated the achievement of the project’s midterm mandates/targets/outcomes/milestones? With regard to the activity design, what are the main constraints and/or weaknesses, opportunities and threats that need further attention? International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 25 2) To what extent is the current coordination and collaboration mechanism among key stakeholders (Service delivery implementing partners, relevant GOK ministries and training institutions) effective in achieving program objectives? 3) To what extent is the project perceived as relevant, responsive and useful to main stakeholders (MOH and service delivery implementing partners) in addressing the challenge of having skilled and knowledgeable health care providers at facility level? 4) What sustainable and innovative HRH system-related models is the activity currently implementing? What does the Ministry of Health and the regulatory boards/councils think in terms of their long-term use? 5) What are the key programmatic and management recommendations that the mission could consider for mid-course changes to the current program design? B.4. MTR Design and Data Collection Methods The Contractor is expected to review and further refine the proposed MTR methodology/data collection tools into well-grounded and appropriate approaches and submit as part of the proposal to USAID for approval. A list of result area-specific detailed questions are included in Annex I, MTR team is therefore encouraged to use this list in coming up with detailed sub-questions as much as possible where necessary. The refined tools are expected to generate the highest quality and most credible evidence that corresponds to the questions being asked above using sound social science practices and tools used in a manner that minimizes the need for evaluator-specific judgments. The following documents will be provided to the MTR team: • Project agreement document • Project annual work plans including M&E Plan/ performance Management Plan (PMP) • Annual & Quarterly Progress Reports • HRH national guidelines/policy documents that the consultants might need • Baseline survey report • Training needs assessment report • Final draft report on post Training Assessment report • Forecasting report • State of Medical Education and Training in Kenya: Bottlenecks and Best Buys: Experiences of eleven health training institutions in Kenya activity design, what are the main constraints and/or weaknesses, opportunities and threats that need further attention? • Regulatory bodies documents on core curriculum and CPD guidelines • Bachelor of Medicine & Bachelor of Surgery Core curriculum by Medical Practitioners & Dentists Board • Bachelor of Dental Surgery Core Curriculum by the Medical Practitioners and Dentists Board • Continuous Professional Development (CPD) Policy Guidelines by Kenya Medical Laboratory Technicians and Technologists Board • Final Draft of Continuing Professional Development for Kenya Nutritionists and Dietitians Institute • Final Draft of Training Standards for Nutritionists and Dietitians in Kenya The Contractor shall consider the following information in its design: a) Evaluation Design A mixed methods approach that has a bias on the qualitative methods with less quantitative methods is recommended for this MTR. The natures of the interventions/strategies lend themselves more to the qualitative methods as opposed to quantitative. Qualitative methods International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 26 shall include in-depth structured interviews, Focus Group Discussions, Roundtable Expert Consultations, Key Informant Interviews, Roundtable Group Discussions, Documents Review and Systems Assessment. Quantitative methods shall include basic statistics such as mean and calculation of service coverage and presentation of data in graphs/charts. b) Data Collection Methods Qualitative methods: MTR team is expected to develop data collection tools for the following methods: • In-depth structured interviews with key staff from MOH’s Human Resources Department at the national, Directorate of Preventive and Promotive Health Services, and Directorate of Clinical Services. • Roundtable expert consultations HRH training models and sustainable organizational capacity development with regulatory board members drawn from Medical and Dentist Board, Nursing Council of Kenya, Clinical Council of Kenya, Laboratory and Pharmaceutical Board, and Nutritionists and Dietetics Council of Kenya. Further the team shall also have Expert Consultations with public sector HRH subject matter experts. • Key informant interviews with training institutions heads and health facility in-charges on the mode of in-service training and pre-service training, institutional capacity enhancement on HRH planning and management • Roundtable Group Discussions with USG service delivery implementing partners drawn from at least 8 counties drawn from all the 8 regions (Eastern, Western, Rift Valley, Nyanza, Coast, Nairobi and North Eastern) on the in-service training of health care workers. All USAID-funded AIDS, Population and Health Integrated Assistance (APHIAplus) mechanisms will be represented at this roundtable group discussion. • Documents review of project documents and this includes quarterly and annual progress reports including performance management plans, relevant reports produced by the activity on Human Resources for Health thematic areas by contractors hired by the activity. • A Systems Review of the innovative HRH system models that the activity has developed. • Focus Group Discussions (FGD) & Key Informant Interviews (KIIs) for beneficiary-level data which requires interviewing 40 pre-service students and 24 in-service health care workers at 4 Kenya Medical Training Colleges (KMTCs) at their respective locations: Nairobi, Nakuru, Kisumu and Tenwek. A random sampling for pre-service students at every KMTC of between 10 - 15 students from any cohort that has benefited from FUNZO program to participate in FGD and thereafter select 2 -3 students from the FGD participants for KII. And, a random sampling for In-Service Health Care Workers in every County Hospital (formerly District Hospitals) of a total of 6 health care workers (2 Nurses, 2 Clinical Officers, 1 Records Officer and 1 Lab Technologist) from a pool of in-service health care workers that have benefited from FUNZO programs. Use comparative and content analysis methods for the FGDs and KIIs. Quantitative methods: • Use of basic quantitative methods such as bar charts/graphs to summarize the achievements of the activity for the last two and half years, to calculate the coverage achieved by the activity for key in-service training of health care workers. • Organize one national key findings, conclusions and recommendations validation forum. A select of key HIS stakeholders will come together to validate the key findings, conclusions and recommendations from the review and for developing consensus for the key strategic shifts if there are any for the project. MTR team is expected to further refine and to improve these methods to better collect data for this review and submit the refined methods as part of the of the proposal to USAID for approval. As much as possible, meetings/roundtable discussions/expert consultations will be held in International Business and Technical Consultants Incorporation (IBTCI) and/or GOK-run International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 27 conference institutions such as Kenya School of Government and/or Kenya School of Monetary Studies. Comprehensive list of respondents for data collection The respondents can be selected from the following list of stakeholders that the Project works with: Training institutions  KMTC Nairobi  KMTC Nakuru  KMTC Garissa  African Medical Research Foundation (AMREF) Training Institute  Tenwek School of Nursing  Mission for Essential Drugs & Supplies  Moi University College of Health Sciences  Maseno University  Maside Muliro University of Science and Technology  Maua School of Nursing  Kenya Methodist University  Kenyatta University  Pwani University  Outspan Medical College  Kijabe School of Nursing  St Joseph’s Nyabondo Medical Training College  Presbyterian University of East Africa Resource partners  Kenyatta University  Strathmore Business School  University of Nairobi College of Health Sciences  University of North Carolina  Great Lakes University of Kisumu  Results for Development Institute  Kenya Healthcare Federation  Higher Education Loans Board Regulatory Bodies  Kenya Medical Practitioners & Dentists Board  Nursing Council of Kenya  Kenya Medical Laboratory Technologists & Technicians Board  Clinical Officers Council  Kenya Nutritionists & Dietetics Institute  Pharmacy and Poisons Board  Public Health Officers Council Health Technical Departments  MOH-HRD  Directorate of Health Standards, Quality and Regulation  Department of Family Health  National AIDS and STI Control Program (NASCOP)  Division of Community Health Services International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 28  County Directors of Health Collaborating Organizations  APHIAplus Partners  Academic Model Providing Access to HealthCare (AMPATH)  Management Sciences for Health- Logistics Management System (MSH-LMS)  MSH-Health Commodity Management  Emory CDC  AfyaInfo  Kenya Conference of Catholic Bishops (KCCB)  Christian Health Association of Kenya (CHAK)  Supreme Council of Kenya Muslims (SUPKEM) c) Data Analysis Methods The main data analysis method shall be through content analysis strongly supported with comparative/analytical and triangulation techniques. Detailed synthesis and analysis is expected of the MTR team to support the key findings, conclusions and recommendations that would come out of the review. Content, comparative and analytical analysis techniques of the qualitative data from key informant interviews, roundtable discussions, expert consultations, roundtable policy dialogue are some of the suggested analysis methods. Use of basic statistics to present data in graphs and charts is also expected. The MTR team shall review the evaluation questions and use their technical expertise to propose a data analysis plan for every evaluation question as part of the proposal to USAID/Kenya/OPH prior to the start of data collection. The MTR team is advised that USAID’s information quality standards in ADS 578 apply to this evaluation. Data should be disaggregated by gender at the outcome and output levels, and geographic area (province and county; urban vs. rural; facility vs. community) where appropriate. d) Methodological Strengths and Limitations USAID’s Evaluation Policy states that any methodological strengths and limitations are to be communicated explicitly in SOWs. Some examples of methodological limitations include: Strengths: o Roundtable approach combines strategic and practical questions and would help MTR team to explore and determine most appropriate responses for the questions o Roundtable approach is a proven methodology for engaging with the policy makers in discussing issues that require high level policy decision making in a timely fashion Limitations: o Short implementation timelines not allowing for data collection at health facility level on key areas of focus. o The level change and stability of national GoK/Ministry of Health and County Health teams is likely to make roundtable discussions not very productive. B.5. KEY PERSONNEL Evaluation Team Composition The two-person team shall include: 1) a Senior Expatriate Monitoring and Evaluation Specialist, as defined in the IQC (Indefinite Quantity Contract), with background on human resources for health (HRH); and 2) a Senior Local Monitoring and Evaluation Specialist, as defined in the IQC, with experience Monitoring and Evaluating Health and HRH is required for this task. The entire team must be external to USAID and all team members shall be required to provide a written International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 29 disclosure of conflicts of interest. Team leader must be an expatriate while the other member must be a Kenyan citizen. Evaluation Management IBTCI will provide overall direction to the MTR team; avail all the key project documents, provide all the logistical support required to perform this MTR. IBTCI/MTR team shall be responsible for arranging all roundtable discussions, Key Informant Interviews (KII) and booking meeting places. IBTCI is responsible for quality control and delivery of the required report as agreed to by USAID. IBTCI shall be responsible for arranging all domestic travel and hotel arrangements for the selected county health executives listed below. C. PERIOD OF PERFORMANCE The period of performance for the MTR is scheduled for June, 2014. The MTR will take a maximum of thirty-five (35) days. An additional five (5) days will be added to the period of performance to complete the final report after the MTR is complete. The total period of performance for both the MTR and the final report is 40 days. D. DELIVERABLES D.1 Specific Deliverables MTR team is required to include tentative dates and timelines for the following deliverables together with the proposal. 1. Briefings: The MTR team will provide two in-country briefs to USAID/Kenya, the first one at entry and second at exit. At the exit brief the team will make an in-country presentation to USAID on the main findings at the end of the in-country reviews and analysis. 2. Work plan: The MTR team will provide a detailed work plan to USAID before commencing the evaluation work. The work plan will outline the technical approaches that will be undertaken and the methods to be used. It will be approved by USAID before the MTR fieldwork starts. 3. Debriefings: A detailed written outline and oral de-briefing of the conclusions supported with well-grounded findings and recommendations will be presented to the USAID/Kenya. 4. Draft Report: The first draft of the MTR report will be submitted before the team departs Kenya and after the final debriefing so that comments can be incorporated into the said report. 5. Final Report: The final MTR report in single line spacing and font size 11, not more than 25 pages will be due at USAID/Kenya within 5 working days after the team receives comments from USAID/Kenya. The team leader will be given up to a maximum of 5 days for the finalization and submission of the MTR report. 6. Upon final approval of the content by USAID/Kenya, IBTCI will have the report edited and formatted. The final report will be submitted both electronically and in hard copy. Four hard copies of the report will be provided to USAID/Kenya. In addition, all the raw data will be submitted to USAID on CD labeled “FUNZOKenya MTR Raw Data” for future reference. Once USAID has approved the final report, IBTCI will submit it together with all MTR-related information products to the DEC as detailed on the activity Contract. 7. While subject to change with the acceptance by both parties, it is envisioned that all team members will be in Kenya the entire duration of the evaluation’s in-country component. 8. Organize one national dissemination forum to present key and finalized findings, conclusions, recommendations and strategic directional shifts (if any) to the national key stakeholders. D.2. Final Report Format The format for the MTR report shall be as follows, and the report shall be a maximum of 25 pages not including annexes. The report format should be restricted to Microsoft products and 12-point font should be used throughout the body of the report, with 1” page margins. Four bound hard copies shall be submitted, and an electronic copy in MS Word. In addition, all data collected by the evaluation will be provided to USAID in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the project or the evaluation. If the International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 30 report contains any potentially sensitive information, a second version report excluding this information will be submitted (also electronically, in English) for dissemination among stakeholders and on the Development Information Clearinghouse. The evaluation team is responsible for ensuring that the final evaluation report includes all criteria listed in Appendix 1 of USAID’s Evaluation Policy. 1. Executive Summary—concisely state the most salient findings & recommendations (3 pp); 2. Table of Contents (1 pg); 3. Introduction—purpose, audience, and synopsis of task (1 pg); 4. Background—brief overview of development problem, USAID project strategy and activities implemented to address the problem, and purpose of the evaluation (2 pp); 5. Methodology—describe evaluation methods, including constraints and gaps (1 pg); 6. Findings/Conclusions/Recommendations—for each evaluation question (10-13 pp); 7. Issues—provide a list of key technical and/or administrative, if any (1–2 pp); 8. Key Strategic Future Directions (2 pp); 9. Annexes —that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. These include references to bibliographical documentation, meetings, interviews, tools used and focus group discussions. D.3 QUALITY EVALUATION REPORT IBTCI is expected to review the USAID’s requirements and expectations on the draft and final reports as detailed on Annex II, “Checklist for Assessing Evaluation Reports”. It is important to note that USAID will subject the structure and content of the report to the parameters outlined on the checklist and will use this as a basis for accepting and/or rejecting the reports. D.4 THREATS TO VALIDITY IBTCI is required to manage MTR consultants and guard against any possible threats to validity of findings, conclusions and recommendations drawn from the qualitative methods. Any conclusion drawn from the qualitative data sources must be supported by well-grounded body of evidence that is triangulated and confirmed. It is therefore expected that ESPS will take the consultants through the parameters outlined on the USAID’s “Checklist for Reducing Threats to Validity for Qualitative Methods”. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 31 ANNEX 2: CONFLICT OF INTEREST DISCLOSURES Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Martina Nicolls Title Ms. Organization IBTCI Evaluation Position? Team Leader Evaluation Award Number (or RFTOP or other appropriate instrument number) ESPS Task Order #5 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) FUNZOKenya implemented by IntraHealth International Inc. # AID-623-A-12-00011 I have real or potential conflict of interest to disclose. _____ Yes ____X__ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: .l. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though Indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) ore being evaluated. 5. Current or previous work experience with an organization that may be seen as an Industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. _____ Yes ______ No I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 25 September 2014 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 32 Disclosure of Conflict of Interest for USAID Evaluation Team Members Name DR TERESA KINYARI MWENDWA Title LOCAL EVALUATOR Organization IBTCI Evaluation Position? Team Member Evaluation Award Number (or RFTOP or other appropriate instrument number) ESPS Task Order #5 USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) FUNZOKenya implemented by IntraHealth International Inc. # AID-623-A-12-00011 I have real or potential conflict of interest to disclose. _____ Yes __X____ No If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: .l. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though Indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) ore being evaluated. 5. Current or previous work experience with an organization that may be seen as an Industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. _____ Yes ______ No I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 33 Signature Date September 25th 2014 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 34 CURRICULUM VITAE Martina Nicolls Key qualifications: Monitoring and evaluation (M&E) expert; database design & management & data audits; gender; child labor; project management; grants management; institutional strengthening; financial & resource planning; education (primary/basic, secondary, higher education/tertiary; vocational; gender equity; teacher training; policy; educational statistics; EMIS); management and evaluation of programs from emergency relief/disaster to recovery, transitional, and development; multi-national, multi-regional & multi-cultural projects Select Professional Experience: Date: April-June 2014 Location: Somalia/Somaliland/Puntland/Galmudug/Mogadishu Company: USAID/IBTCI Position: Team Leader Mid-Term Evaluation of Somali Youth Leadership Initiative: Performance evaluation of program to increase Somalia’s stability through targeted interventions in economic growth, governance, and the reduction of the appeal of extremism - secondary education, employments schemes & youth empowered to participate and contribute positively and productively to society. Date: March 2014 Location: Home-Based Company: Development Afghan Women’s Organization Position: Vocational Technical Advisor Livelihoods for Women in Bamyan Province: Provision of advice on livelihoods training, income generation, agricultural extension, micro-financing and vocational education. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 35 Date: October 2013-January 2014 Location: Sri Lanka Company: USAID-OTI/Social Impact Position: Team Leader / Evaluator Final Evaluation Reintegration & Stabilization in the East and North (RISEN): Performance evaluation for Office of Transition Initiatives to assess achievement of objectives; effectiveness of approach to social cohesion, civic engagement & reintegration; cluster small grants, response to emerging needs, promoting community-level reconciliation, and strengthening local capacity for conflict mitigation. Date: September-November 2013 Location: Sudan Company: USAID/IBTCI Position: Team Leader / Evaluator Final Evaluation Sudan Transition Conflict Mitigation Program: Performance evaluation for Office of Transition Initiatives/Office of Transition & Conflict Mitigation to assess achievement of objectives; effectiveness of approach to conflict mitigation; cluster small grants, response to emerging needs in providing peace dividends, promoting community-level reconciliation, and strengthening local capacity for conflict mitigation; lessons learned to inform future conflict mitigation programming in Sudan. Date: August 2013 Location: Georgia Company: SIDA/Oxfam GB Position: Team Leader / Evaluator Health Rights Monitoring Strategy for the Public Defender’s Office: Developed a monitoring strategy for the PDO Health Rights Unit to effectively protect the health rights of poor and marginalized groups with a special focus on children and youth; Outputs included a stakeholder analysis & the role of civil society organizations, and a short term (1 year) and long term (3 year) monitoring strategy defining implementation links among PDO, government, civil society, and health workers. Date: June-August 2013 Location: Thailand, Cambodia, Vietnam, Lao PDR (Mekong Sub-Region) Company: AusAID/GRM International Position: Team Leader / Child Protection Evaluator Mid-Term Evaluation of Project Childhood: Assessment of the regional program to combat child sexual exploitation in travel and tourism (CSETT), especially girls. Evaluation of Prevention Pillar and Protection Pillar, and their integration, to assess children’s awareness & stakeholder capacity to prevent CSETT; legal frameworks, law enforcement & investigations, UN Office on Drugs International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 36 and Crime (UNODC) & INTERPOL strategies, and cross-border cooperation in South East Asia. Date: January – April 2013 Location: Nepal Company: US Department of Labor/ICF Macro Position: Team Leader / Child Labor Evaluator Final Evaluation: New Path New Steps Program to end child labor/trafficking; assessed design, implementation, management, database, strategic plans, outcomes, & impact (OECD-DAC criteria). Interventions: literacy, pre-vocational, vocational, livelihoods/income generation, scholarships, school safety, child friendly schools, & child friendly local governance. Date: October 2012 – January 2013 Location: Kurdistan Regional Government of Iraq Company: USAID/FHI360 Position: Team Leader / Education & Evaluation Specialist Situational Analysis of Basic Education Conducted a situational analysis to support the objective “Contributing to Regional Peace & Security.” Focused on (1) enrolment and retention; (2) education institutions and management; (3) educators and training; (4) gender/cross-cutting issues; and (5) disputed internal border sites. Date: July-August 2012 Location: Papua New Guinea Company: AusAID/GRM International Position: Higher Education Specialist Final Evaluation of the Group of Eight-University of Papua New Guinea Collaborative Project Evaluation of academic volunteer capacity building program and its impact on teaching and learning, research quality and enhancement, scholarships, and partnership arrangements. Date: November 2011- January 2012 Location: Pakistan Company: USAID/MSI Position: Data Quality Assessor/Auditor Independent Monitoring & Evaluation Contract (IMEC) Data Quality Assessment Review USAID & Implementing Partner’s Performance Management Plans and project output & process indicators; conduct a DQA. Date: October 2011 Location: Kosovo Company: USAID/AED Position: Team Leader / Evaluator International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 37 Final Evaluation: Initiating Positive Change Program Program to build confidence, improve quality of life, economic security & freedom of movement; assessed relevance, effectiveness, efficiency, sustainability & impact (OECD-DAC criteria) of design, implementation & outcomes. Interventions: community forums, civil society & micro-enterprise grants, & media programs. Date: September – October 2011 Location: Afghanistan Company: Danida/Particip-NIRAS Consulting Position: Team Leader / Evaluator Fact-Finding Mission: Education Support Program Formulation of evaluation methodology to assess Danida’s education programs (2003-2010) – Primary Education Program (PEP)/secondary /TVET/ESPA - using OECD-DAC criteria: effectiveness, efficiency, sustainability and impact. Date: September 2011 Location: Nepal Company: US Department of Labor/ICF Macro Position: Team Leader / Child Labor Evaluator Midterm Evaluation: New Path New Steps Program to end child labor/trafficking; assessed design, implementation, management, database, strategic plans, outcomes, & impact (OECD-DAC criteria). Interventions: literacy, scholarships, primary, secondary, vocational, livelihoods/income generation. Date: June 2010 to August 2011 Location: Georgia (Tbilisi) Company: US Dept of State Office of Coordinator of US Assistance to Europe & Eurasia (EUR/ACE) /IBTCI Position: Chief of Party /Monitoring & Evaluation Specialist / Audit Specialist Georgia Monitoring Project Multi-agency, whole-of-government thematic USG evaluations from emergency relief to development; Designed and oversaw implementation of M&E methodologies & PMP plans - indicators, targets, results frameworks, baselines. Congressional reports: cash transfer/budget support; data quality assessment (DQA); (IDP)/refugees, health, disaster assistance; economic growth; maritime security, civil society & democracy small grants. Date: February – May 2010 Location: Mongolia Company: Millennium Challenge Corporation (MCC/USG)/IBTCI Position: Chief of Party / Senior Grants Manager Technical Vocational Education and Training Project – Competitive Grants Program Public & private partnerships; dissemination of best TVET practices; grant/project management; formulated grant administration and tracking manual; capacity building of TVET institutional & Ministry staff – M&E and Financial Reporting guidelines (activity indicators, performance management plans). International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 38 Date: October 2009 – January 2010 Location: Uganda – Karamoja Region (Moroto), Pader, Gulu, and Kitgum Company: USDOL/ICF Macro Position: Team Leader / Evaluation Specialist Midterm Evaluation: Livelihoods, Education & Protection Child labor/trafficking assessment of design, implementation, management, strategic plans, outcomes, impact, database, grants, progress reporting, & cost effectiveness (OECD-DAC criteria). Interventions: primary, secondary & vocational, skills training, livelihoods, literacy, scholarships, & child protection. Workshop on child labor. Date: September-October 2009 Location: Sudan (Darfur) Company: Global Relief Alliance/World Relief Position: Team Leader / Evaluation Specialist Nutrition and Food Security Study Designed & executed anthropometric assessment & household survey to evaluate nutrition (& breastfeeding), health (mortality), water, & food security to guide the multi-sectoral emergency program. Reviewed indicators & synthesized reports for 5 year trend analysis. Date: July 2009 Location: Jakarta, Indonesia (Aceh, East Java, Makassar, Papua) Company: USAID/JBS International Position: Higher Education & Evaluation Specialist Gap Analysis of Pre-Service and In-Service Teacher Training in Basic Education Researched gaps student achievement & quality/relevance of university teacher training programs, scholarship program, strategies & models for ensuring alignment with country’s strategic plans. Date: April - May each year in 2010, 2009, 2008, 2007 Location: Dhaka, Bangladesh Company: European Union/Femconsult Position: Education Specialist; Monitoring & Evaluation Specialist Joint Annual Review Mission of Primary Education Development Program II (PEDP-II) Monitoring 11 donor coordination & progress results; synthesis evaluation reports against goals & objectives. Advise to EU & Ministry of Education on financial reporting/audits, strategic & budget planning, Annual Operation Plans, & Resource Utilization Plans. Data collection & analysis; database (EMIS); performance indicators and Performance Monitoring Plans (PMP). Date: February - March 2009 Location: Afghanistan – Kabul, Heraat, Jalalabad Company: USAID/DevTech Systems International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 39 Position: Team Leader / Monitoring & Evaluation Specialist / Gender Specialist / Finance Specialist Assessment of Activities Attributed to FY2008 Women’s Earmarked Congressional Funding Report: expenditure, equipment, training impact, verification of beneficiaries, & identification of best practices of income generation/livelihoods, health, agriculture, vocational (TVET) & higher education. Date: January - February 2009 Location: India – New Delhi, Mumbai, Chennai, Madurai, Bhopal & Lucknow Company: ILO/USDOL/ICF Macro Position: Monitoring & Evaluation Specialist Final Evaluation of Preventing & Eliminating Child Labor & Human Trafficking in India Reviewed design, implementation, M&E system (& performance indicators), strategic plans, outcomes, impact, and sustainability (OECD-DAC criteria) to assess achievements & cross-cutting factors - gender, human rights/protection, scholarships, educational interventions (primary, secondary and vocational) & income-generation/livelihood activities. Conducted six workshops. Date: September 2008 – January 2009 Location: Kenya Uganda Rwanda and Ethiopia (urban and rural) Company: USDOL/ICF Macro Position: Team Leader / Monitoring & Evaluation Specialist Final Evaluation of Combating Exploitative Child Labor & Human Trafficking through Education in Kenya, Uganda, Rwanda & Ethiopia Together Reviewed design, implementation, M&E system (& performance indicators), strategic plans, outcomes, impact, and sustainability in 4 countries to assess country-specific and regional achievements & cross-cutting issues - gender, human rights/protection, scholarships, educational interventions (primary, secondary and vocational) & income-generation activities. Conducted workshops in each country. Wrote 4 reports + Regional Report. Date: February to April 2008 Location: Juba, Southern Sudan Company: USAID/Winrock Position: Longitudinal Study Team Leader Gender Equity in Education Designed 5-year longitudinal study to measure impact of scholarship initiative to support girls in secondary school & teacher training programs (reviewed indicators and results framework). Prepared training manual and trained MoE and project staff on data collection and entry; devised database & survey; analyzed & presented data. Date: December 2006 - January 2007 Location: Monrovia, Liberia Company: European Commission /ARS Progetti International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 40 Position: Monitoring & Evaluation Expert Midterm Evaluation of EC’s Institutional Support Program Assessed impact & sustainability of design, planning, & finance to improve governance & public sector finance management. Assessed EC’s support to Ministry of Planning, Ministry of Finance, & Auditing Commission; evaluated annual performance management plans & performance monitoring. Assessed impact of audits of airport, ports, forestry, petroleum refinery, maritime affairs, and Central Bank of Liberia. Date: September 2006 Location: Khartoum and Port Sudan, Sudan Company: Danida/NIRAS Consulting Position: Monitoring & Evaluation Expert Revision of UNDP Poverty Alleviation Governance Programme for the Red Sea Ensured Red Sea State Government able to deliver public administration, social & economic infrastructure & participatory, pro-poor, human rights, gender￾sensitive, responsive and transparent services; capacity building & training for Finance Ministry, particularly in data collection for health & education. Date: March –July 2006 Location: Juba, Southern Sudan Company: USAID/AED Position: Chief of Party / Senior Advisor to the Minister of Education/EMIS Advisor EQUIP2, Sudan Technical Advisors Project Policy advice to Minister of Education; capacity building of departments: Quality Promotion & Innovation, Gender and Social Change, and Planning. Advice on EMIS and Annual Education Census: data collection, database management, statistical analysis, & data presentation; science & IT curriculum; adult & vocational education. Date: June - July 2005 Location: Gaborone, Botswana Company: European Union/Commission / Femconsult Position: Statistician Specialist in Education and Training Indicators for Sector Policy Support Program in Education and Training Matrix of sectoral output/outcome indicators with baseline and target values (rationale, risks, data collection & verification); analysis of organizational, policy, & finance data. Ministry of Education Financing process indicators for European Development Fund planning (equity, gender, poverty, HIV/AIDS’s impact on education sector), using SWAP (Sector Wide Approach). Capacity building of ministry staff. Date: July 2003 - May 2004 Location: Baghdad, Iraq (including Erbil, Kerbala, Diwaniyah, Nasiriyah) Company: USAID/Creative Associates International International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 41 Position: Team Leader: Accelerated Learning Program Revitalisation of Iraqi Schools and the Stabilisation of Education (RISE) Project Developed & managed Accelerated Learning pilot program. Surveyed 4,500 out-of-school youth to assess demand and vocational needs & validate interventions; established database with baseline data; established 5 urban & rural interventions. Trained 70 teachers; managed 75 local & 7 international consultants. Date: March 2003 Location: Kathmandu, Nepal Company: Asian Development Bank/SMEC International Position: Teacher Training Specialist Teacher Education Project Conducted research study; evaluated teacher training; conducted capacity needs assessment; documented manual on teacher training; conducted 3 workshops. Date: February 2003 Location: Dhaka, Bangladesh Company: Asian Development Bank/SMEC International Position: Gender Access and Equity Specialist Female Secondary School Assistance Project II Social mobilization - quality management, gender access and equity issues, specific needs of girls in education; survey to test literacy levels. Primary Education Development Program II Input into performance-based financing for 2004-2009 new program as Sector Policy Support; recommendations to strengthen institutional capacity. Date: November 2001 - January 2003 (2 in-country visits per year of 3 months each) Location: Muzaffarabad, Kashmir and Islamabad, Pakistan Company: World Bank/Asian Development Bank/SMEC International Position: Project Director/Management Specialist/Internal Quality Auditor Northern Education Project, Kashmir (WB) Policy reform; teacher training; curriculum development; financial and EMIS development; MoE capacity building; baseline for benchmarking teacher capacity & quality; low-cost & no-cost aids. Multi-Grade Teaching Expert Northern Education Project Produced a teachers’ guidebook on multigrade teaching for primary grades 1-5 (with sample lesson plans, timetables, worksheets and classroom management techniques); trained 80 Master Trainers. Girls Primary Education Development Project, Islamabad (ADB) Project Director for US$78m program (2000-2003) for girls’ primary education by establishing 900 Community Model Schools, 173 teacher resource centres, radio International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 42 instruction, social mobilization, and institutional strengthening. Conducted gender analysis for benchmarking and performance measures. Date: November 1999 - November 2001 Location: Canberra, Australia Company: Leo Major Communications Pty Ltd Position: Director and Principal Communications Consultant Major projects: Centre for Cross-Cultural Research, Australian National University: development of Business Plan to document successful outcomes for continued funding; Commonwealth Dept of Health & Aged Care: corporate capacity building (baseline data; staff & service performance indicators; benchmarking; case studies; evidence-based research; salary & costing models. Date: June 1998 - November 1999 Location: Traralgon, Australia Company: Monash University, Gippsland Campus Position: Campus Manager Divisional oversight of HR; finance; distance education; & curriculum. Managed 250 staff; implemented SAP financial system; finance training; initiated review of Student Union. October 1999: Beijing, Shanghai, Taizhou & Jiaozuo, China Latrobe Shire Education Delegation to Jiangsu & Henan Provinces; established partnerships for student exchange; vocational training; & English Language Centre (IELTS training). Date: June 1997 - June 1998 Location: Adelaide, Australia Company: National Centre for Vocational Education Research Position: National Data Analyst/National EMIS Advisor Publication of national education statistics for Ministerial Council for Education & Training; advised on EMIS; conducted audits of data; publication on women and girls in education; Ministerial White Paper tabled in Parliament on adult/community education (supply & demand), and costing models. Date: 1989 - 1996 Location: Adelaide, Australia Company: Vocational Education and Training Centre Position: Curriculum Development Manager /Research Manager /Accreditation Manager Conducted staff and institutional performance management audits; M&E; benchmarking; data analysis. Implemented policy reform; teacher training & gender mainstreaming. Date: January - May 1986 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 43 Location: Wewak, Papua New Guinea Company: St Benedict’s Teachers’ College Position: Teacher Training Specialist Mathematics, Science & English primary teacher training; conducted workshops for curriculum development for multi-grade primary teaching; supervision and staff appraisal of trainee teachers. Nationality: British/Australian dual citizenship Education: PhD in Educational Administration and Supervision Stamford Hill University, London, UK 2000-2004 Master of Science Communication (Thesis on Educational Statistics) Central Queensland University, Rockhampton, Australia 1994-1997 Bachelor of Education (Majors: Mathematics, English)University of South Australia, Adelaide, Australia 1981-1985 Other studies Graduate Diploma of Middle Eastern & Central Asian Studies (2 subjects) Australian National University 2008 Language skills: Language Reading Spoken Written English (native tongue) 5 5 5 German 4 4 3 French 3 3 2 Urdu 2 2 PUBLICATIONS (Selected) Nicolls, M. (2012) Liberia’s Deadest Ends, Strategic Book Publishing, New York. Nicolls, M. (Mar. 21, 2012) The powerful role of women in peace building, London School of Economics http://www2.lse.ac.uk/intranet/news/informationForStaff/staffStudentsAndAlumni/dailyHeadlines/ 2012/March/21March2012.aspx Nicolls, M. (Feb. 8, 2012) How educational institutions can sustain success in the long term, http://eslarticle.com/pub/articles/other/how-educational-institutions-can-sustain-success-for-the￾long-term-91434.htm Nicolls, M. (Aug 2011) Initiating Positive Change Program – Final Evaluation, US Department of Labor/FHI360 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 44 Nicolls. M. (2011) Independent Midterm Evaluation of Combating Exploitative Child Labor through Education in Nepal: Naya Bato Naya Paila (New Steps New Paths) Project, US Department of Labor/World Education Inc. Nicolls, M. (2010) Kashmir on a Knife-Edge, Strategic Book Publishing, New York. Nicolls, M. (Nov 2010) Pre-Monitoring Review of IDP Data & Information in Georgia, US Department of State/IBTCI Nicolls, M. (Oct 2009) Independent Midterm Evaluation of the Livelihoods, Education, and Protection to End Child Labor in Uganda (LEAP) Project, ICF Macro/USDOL Evans, D., Tate, S., Navarro, R. & Nicolls, M. (August 2009) Teacher Education and Professional Development in Indonesia: A Gap Analysis, USAID. [http://pdf.usaid.gov/pdf_docs/PNADS282.pdf] Nicolls, M. & Carlin, A. (May 2009) Report on U.S. Government Women’s Earmark in Afghanistan for Fiscal Year 2008, Devtech Systems, Inc./USAID Bureau for Economic Growth, Agriculture and Trade, Office of Women in Development. [http://pdf.usaid.gov/pdf_docs/PDACP922.pdf] Nicolls, M. (2009) The Sudan Curse, Strategic Book Publishing, New York Nicolls, M. (April 2008), Independent Midterm Evaluation of IRC: Countering Youth & Child Labor through Education in Liberia and Sierra Leone (CYCLE Project), Macro International/USDOL. Nicolls, M. (2004) A Second Chance: Accelerated Learning in Iraq, Creative Associates International, Washington DC Nicolls, M. (1999) Who uses ACE? Future demand in ACE (Adult and Community Education) - Some Issues, Campbell, A. & Curtin P. (eds), National Centre for Vocational Education Research, Adelaide, Australia Nicolls, M. (May 1998) State Training Profiles: review of statistical documentation for the vocational education and training sector, Journal of Higher Education Policy and Management, Vol.20, No.1, May. Nicolls, M. (1998) State Training Profiles: rural vocational education and training statistics and funding issues, Journal of Institutional Research in Australia, Vol.7, No.1, May Nicolls, M. (1998) Learning for leisure, Australian Training Review, March/April/May, No.26. National Centre for Vocational Education Research 1998, Australian Vocational Education & Training Statistics 1997: at a glance, DMA Branch, NCVER, Adelaide. [co-author] Nicolls, M & Lean, N. (1995) A Proposal for a Data Management Strategy for Vocational Education and Training Statistics, Department for Employment and TAFE, VET Division, Adelaide, Australia Nicolls, M. (1994) Water Quality Management Issues in South Australia, Report for Landcare Australia, Adelaide International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 45 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 46 CURRICULUM VITAE William Kiarie Key Qualifications: International consultant with robust background in capacity building, conducting assessments and evaluations in Human Resource for Health (HRH); leadership and management development, organizational development, HIV/AIDS, and Health Research. Select Related Professional Experience: Date: January – February 2014 Location: Ethiopia Company: MSH for MOH Ethiopia Position: Lead finalization of the Federal HRH Strategic Plan, 2014-2020/ HRH Training of Trainers Date: November-December 2013 Location: Uganda Company: IBTCI for USAID/Uganda Position: Team Leader, Evaluation of the Uganda Capacity Project Date: August 2012 – June 2013 Location: Kenya Company: GIZ Position: Coordinator for Kenyan team for 1 year 4-country HRH fellowship program; supported development of induction handbook for County Health Management Teams Date: January 2013 Location: Ethiopia Company: MSH, Ethiopia HRH Project Position: Lead HRH Trainer Date: February – March 2011 Location: Nigeria Company: MSH/MOH Nigeria Position: Developed workforce plans for 3 States Date: April 2011 Location: Kenya Company: Capacity Project/MOH Position: Conducted national HRH training Date: June 2011 (10 days) Location: Kenya Company: Capacity Project/USAID/MOH Position: Developed and costed work plan for Kenya national HRH strategic plan Date: July 2011 (12 days) Location: Kenya Company: Capacity Project/USAID/MOH Position: Developed and costed national HRH transitional plan to guide MOH on managing the transition to the new constitutional dispensation Date: November-December 2011(15 days) Location: Nigeria Company: Abt Associates/DFID International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 47 Position: Conducted HRH training in 4 States Date: April-May 2010 (27 days) Location: Nigeria Company: MSH/MOH Nigeria Position: HRH training for MOH health managers Date: April-May 2010 (27 days) Location: Nigeria Company: MSH/MOH Nigeria Position: HRH training for MOH health managers Date: Location: June-August 2010 (40 days) Tanzania Company: MSH/Ministry of Health and Social Welfare Position: Designed training materials for TOT training for district HR staff Date: July 2010 (5 days) Location: Kenya Company: Capacity Kenya/MOH Position: Developed competency matrix for HRH managers Date: July – November 2010 (20 days total) Location: Kenya Company: Capacity Kenya/MOH Kenya Position: Developed the inaugural Kenya Annual National HRH Report Date: October-November 2010 Location: Nigeria Company: MSH/MOH Nigeria Position: Trainer, TOT HRH training for MOH managers Date: November 2010 Location: Tanzania Company: MSH/Capacity Tanzania Position: Lead Trainer, HRH District Managers Date: January-February 2008 Location: Kenya Company: CHAK/KEC Position: Lead Consultant to develop HRH action plan for FBO sector Date: March 2008 Location: Ethiopia Company: MSH International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 48 Position: Conducted national HRH assessment Date: January 2007 Location: Kenya (North Eastern province) Company: Capacity Project/USAID Position: Lead Consultant, developed HRH action plan for the province Date: November 2005-June 2007 Location: Southern Sudan Company: Capacity Project Southern Sudan Position: Coordinated capacity-building program for Southern Sudan covering HR baseline assessment; establishment of HRH systems, provided leadership and management support, health workers certification, developed HRH information system, assisted in developing private-public partnerships and national HIV/AIDS strategic plan Date May-October 2007 (60 days) Location: Tanzania Company: USAID Africa Bureau Position: Lead Consultant to assess reproductive health manpower allocation, utilization andmanagement Date: November 2007 Location: North Eastern Kenya Company: Capacity Project/APHIA II/MOH Position: Conducted situational analysis of HRH Date: October 2006 Location: Southern Sudan Company: Capacity Project/USAID Position: Developed national HRH policy for Southern Sudan Date: April 2006 Location: Zanzibar Company: MSH Position: Led comprehensive assessment of HRH for MOH Language Speaking Reading Writing English Excellent Excellent Excellent Kiswahili Excellent Excellent Excellent Kikuyu Excellent Excellent Excellent Education: 1984: B. Pharmacy Degree University of Nairobi 1994: Masters of Business Administration (MBA) United States International University International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 49 TERESA KINYARI MWENDWA MPH CURRICULUM VITAE CURRENT AND RECENT APPOINTMENTS 2009 to date: Lecturer, Masters in Tropical and Infectious Diseases, (MSc TID) University of Nairobi Institute of Tropical and Infectious Diseases (UNITID) 2013: Lecturer, Masters in Internal Medicine and Pediatrics Part 1 – Cell Biology and Immunology 2005 to date: Lecturer, Department of Medical Physiology, School of Medicine, University of Nairobi. 2005 to date: Lecturer, HIV common course in all level 1 students at the University of Nairobi. 2005: Part time lecturer, Aga Khan University Teaching Hospital, BSN 1 course in Medical Physiology. 2002 –2005: Assistant Lecturer, Department of Medical Physiology, School of Medicine, University of Nairobi. 2002-2004: Scholar, International AIDS Research and Training Program (IARTP), University of Washington, Seattle, USA 1998-2002: Project Physician, Pelvic Inflammatory Disease Project, University of Nairobi, Department of Obstetrics and Gynecology in collaboration with the University of Washington, Seattle, USA at the WHO Collaborative Centre for STD and HIV Research and Training . March 1998-October 1998: Medical Officer of Health, Nyeri Provincial General Hospital, Nyeri, Kenya. December 1996-March 1998: Internship (House Officer), Nyeri Provincial General Hospital, Nyeri 1996: Part time lecturer in Anatomy and Physiology with the Registered Clinical Officer Level 1 students at the Kenya Medical Training College, KMTC INVITED LECTURES 2011 to date: Lecturer, Implementation Science Fellowship Track - Update on the National Malaria Control Program goals and objectives in reducing childhood mortality in Kenya RELEVANT CONSULTANCIES Prevalence of acute diarrhea among children aged less than 5 years in Bungoma District-Partners in Applied Technologies in Health(PATH) 2007 (Conducted focus group discussions among caregivers of under5 and health care workers on the challenges and opportunities in the prevention and control of diarrheal diseases; Supervised household survey data collection ) Evaluation of Rheumatic Heart Disease (RHD) sensitization in Nairobi East District 2008 (Danish Heart Foundation/ DANIDA) - Conducted focused group discussions among patients with RHD and health workers treating RHD in health facilities; RHD school health clubs and their teachers and did key informant interviews with chiefs, heads of schools and hospital in charges International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 50 Strengthening health systems to improve adherence performance in health facilities providing ART in Kenya: an intervention study [MSH/NASCOP/ INRUD-KENYA] 2008-2009- Conducted Key Informant Interviews with the Health Facility In-charges and exit interviews on patients receiving ARTs MEMBERSHIP OF SOCIETIES OR ASSOCIATIONS Interim Vice-Chair Public Health Society of Kenya Member Kenya Medical Women Association Member Kenya Medical Association PUBLICATIONS Beatrice O. Ondondo, Robert C. Brunham, William G. Harrison, Teresa Kinyari, Prameet Sheth, Nelly R. Mugo, Craig R. Cohen. Frequency and Magnitude of systemic and mucosal Chlamydia trachomatis IFN-γ responses among highly exposed women. The Journal of Infectious Diseases 2009, 199, 12:1771-1779. William G. Harrison, Teresa Kinyari, Amalia Meier, Robert C. Brunham, Rosemary Nguti, Nelly R. Mugo, Craig R. Cohen: Risk Factors For Incident Neisseria Gonorrhoeae In A Prospective Cohort Of Kenyan Female Sex Workers. The International Journal of Infectious Diseases 2006 Volume 5 Number 2. Craig R. Cohen, Kasra M. Koochesfahani, Amalia S. Meier, Caixia Shen, Karuna Karunakaran, Beatrice Ondondo, Teresa Kinyari, Nelly R. Mugo, Rosemary Nguti, Robert C. Brunham. Immunoepidemiologic Profile of Chlamydia trachomatis Infection: Importance of Heat-Shock Protein 60 and Interferon-γ .The Journal of Infectious Diseases 2005; 192:591–599 Ondondo BO, Brunham RC, Harrison WG, Kinyari T, Sheth PM, Mugo NR, Cohen CR. Frequency and magnitude of Chlamydia trachomatis elementary body- and heat shock protein 60-stimulated interferon gamma responses in peripheral blood mononuclear cells and endometrial biopsy samples from women with high exposure to infection.J Infect Dis. 2009 Jun 15;199(12):1771-9. Kinyari T.N., White E., Sharma A., Morris M., Bukusi E.A., Farquhar C., Ngugi E.N., Cohen C.R. The relationship between sexual partnerships characteristics and condom use among young female sex workers in Korogocho, Nairobi (Poster Presentation at the 16th ISSTDR Congress Amsterdam, the Netherlands 2005) Collaborative Publications with the Ministry of Public Health and Sanitation (Division of Malaria Control-DOMC) 2009: The 2009-2017 National Malaria Strategy development after MPR Phase 1 review 2009: The 2009-2017 National Malaria Monitoring and Evaluation Framework after MPR Phase 1 review 2011: The Community Malaria Case Management Curriculum-Participants and Trainers Manuals RESEARCH GRANTS PI-University of Nairobi, School of Medicine, Deans Committee Research Seed Grant Award–Kshs 180,000 A local sexual network survey of female sex workers and their male clients in Korogocho slums, Nairobi (2009) International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 51 PI-NIH grant through Fogarty International Centre $25,000 (International AIDS Research and Training Program (IARTP)- MPH thesis project at the School of Public Health and Community Medicine ,University of Washington, Seattle(2003). Completed 2004 Co-Investigator and Team Leader - Impact of insecticide resistance on ITN and IRS in Kenya￾a collaborative study between Ministry of Public Health and Sanitation (MPH&S), Kenya Medical Research Institute (KEMRI) and the University of Nairobi (UON) 2011-2014-$2.4 million funded by the Bill and Melinda Gates Foundation(2009). PI-Global Fund Round 10 Malaria Grant-Scaling Up Malaria Interventions for Greater Impact 2012- 2013 through the University of Nairobi Department of Medical Physiology Co-PI Feasibility study on the use of the Fio-net Deki Reader with the malaria RDT in Nyando District (A collaboration between DOMC, UoN and Fionet) March –May 2013 Co-I Feasibility study on the use of the Fio-net Deki Reader with the malaria RDT in Bondo, Rachuonyo, Nyando and Teso district(nested within the Kenya Insecticide Resistance Project) PI- Service Availability Response Assessment Mapping (SARAM) in Nairobi, Kajiado, Kiambu, Murang’a, Nyeri, Kirinyaga and Nyandarua Counties March-April 2013 Epidemiology track lead-UNITID HIV Training Fellowship 2014 -2018 Curriculum Developed Kenya Palliative Health Care Management Curriculum 2010-Trainers and Participants Manual Community Malaria Case Management Curriculum- Trainers and Participants Manuals Curriculum Reviewed Bachelor of Science in Medical Physiology University of Nairobi School of Medicine Masters in Science Medical Physiology University of Nairobi School of Medicine Curriculum in Development (University of Nairobi School of Medicine) Sports Physiology – Certificate, Postgraduate Diploma, Bachelors, Masters and PhD Sports Medicine – Certificate, Postgraduate Diploma, Bachelors, Masters and PhD EDUCATION 1988- 1996, (MBChB) University of Nairobi Medical School, Nairobi, Kenya. 1991 – 1992, (BSc Med Physiol) University of Nairobi Medical School, Nairobi, Kenya. 1999 – 2001, Postgraduate Diploma in STI/HIV Control and Management, University of Nairobi, Nairobi, Kenya. 2002 – 2004, MPH [Epidemiology] University of Washington, Seattle, Washington, USA International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 52 ANNEX 3: REFERENCES Capacity Kenya. Report of the Performance Needs Assessment of the Kenya Health Training System, August 2011(Executive Summary) Commission on Revenue Allocation. Policy on the Criteria for Identifying Marginalized Areas and Sharing Equalization Fund: Financial Years 2011-2014, Government of Kenya, February 2013 County Department of Health. Kisumu County: Health Sector Strategic and Investment Plan 2013- 2017 FUNZOKenya Project. Annual Report, February 24 – September 30, 2012 FUNZOKenya Project. Annual Report, October 1, 2012 – September 30, 2013 FUNZOKenya Project. Assessment of Clinical Placement Capacities and Practices, and Design of Comprehensive Model of Clinical Placement for Kenya (Draft), August 2014 FUNZOKenya Project. Business Strategy Chart/Milestones, Instruments, Interventions & Progress, September 2014 FUNZOKenya Project. Capacity Advisors to Kenya Medical Supplies Agency (KEMSA), no date FUNZOKenya Project. Comprehensive IR3 Technical Approach, accessed September 22, 2014 FUNZOKenya Project. Field Monitoring Tool Kits, M&E Department, September 2014 FUNZOKenya Project. FUNZOKenya Year 2 Detailed Work Plan – Year 1, August 2012 FUNZOKenya Project. FUNZOKenya Work Plan FY2014, October 2013 FUNZOKenya Project. Health Workforce Forecast Kenya: A Reference Report, September 2013 FUNZOKenya Project. Integrated Mentorship Model for Health Professionals (Draft), 2013 FUNZOKenya Project. IntraHealth International Work Plan – Year 1, FUNZOKenya Project, April 2012 FUNZOKenya Project. Monitoring and Evaluation Plan, National Training Mechanism (NTM) in Kenya, November 2012 FUNZOKenya. Post Training Follow-up Assessment of Kenya Health Workforce Trainings 2012, March 2014 FUNZOKenya Project. Quarterly Bulletin, April 2014 FUNZOKenya Project. Quarterly Bulletin, July 2014 FUNZOKenya Project. Quarterly Performance Monitoring Report, Quarter 1, February 24 – June 30, 2012 FUNZOKenya Project. Quarterly Performance Monitoring Report, Quarter 2, July 1 – September 30, 2012 FUNZOKenya Project. Quarterly Progress, Quarter 1, Year 3, October 1 – December 31, 2013 FUNZOKenya Project. Quarterly Progress, Quarter 2, Year 2, January 1 – March 31, 2013 FUNZOKenya Project. Quarterly Progress, Quarter 2, Year 3, January 1 – March 31, 2014 FUNZOKenya Project. Quarterly Progress, Quarter 3, Year 2, April 1 – June 30, 2013 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 53 FUNZOKenya Project. Report of the Rapid Baseline Survey of the National Training Mechanism in Kenya, September 2012 FUNZOKenya Project. Report of the Rapid Training Needs Analysis of the Health Workforce in Kenya, September 2012 FUNZOKenya Project. Technical Brief, September 2013 FUNZOKenya Project. Training Regulatory Framework, a tool for optimizing performance and quality by Regulatory Bodies, accessed September 22, 2014 Institute of Economic Affairs. A Guide for Understanding Decentralization in Kenya: Empowerment and Accountability – the Pillars for a Better Tomorrow, National Council of Churches in Kenya and Diakonia, 2011 KEMSA. Capacity Kenya Performance Monitoring Report, October 1, 2010 – March 31, 2011, Semi-Annual Report, Kenya Medical Supplies Agency, 2011 KEMSA. The Capacity Kenya Project, Strengthening Human Resources for Health, PowerPoint (MIS Advisor), Kenya Medical Supplies Agency, July 2011 KEMSA. The Capacity Kenya Project, Strengthening Human Resources for Health, PowerPoint (Warehouse and Distribution Advisor), Kenya Medical Supplies Agency, July 2011 KMLTTB. Continuous Professional Development (CPD) Policy Guidelines, Kenya Laboratory Technicians and Technologists Board, July 2013 KMTC. Developing Hostels and Renovating KMTC Accommodation Facilities, Draft Concept Note for Public-Private-Partnerships, no date, accessed October 3, 2014 KMTC. Introducing Parallel Programs for KMTC Campuses: A Concept Note for Non-Regular Classes, Draft Concept for Parallel Programs in KMTC, no date, accessed October 3, 2014 KMTC. Out-Sourcing to Improve Admissions, KMTC Out-Sourcing Committee, October 2012 KNDI. Continuing Professional Development (CPD) Guidelines, Kenya Nutritionists & Dieticians Institute, November 2013 KNDI. Training Standards for Nutritionists and Dieticians in Kenya, Kenya Nutritionists & Dieticians Institute, November 2013 Lungai, E. “Sh100m set aside for KMTC expansion,” Daily Nation, www.nation.co.ke, September 22, 2014 Ministry of Health. A Harmonized National Framework and Guidelines for Development and Review of Curriculum for Training of Health Professionals in Kenya (Draft), July 2014 (in conjunction with the Ministry of Higher Education Science and Technology, and the University of Nairobi) Ministry of Health. Human Resources for Health Norms and Standards Guidelines for the Health Sector, Towards Universal Health Coverage: The Kenya Health Strategic and Investment Plan, 2014-2018, August 2014 Ministry of Health. National Continuing Professional Development Regulatory Framework, Directorate of Health Standards Quality Assurance and Regulations, Kenya, September 2014 Ministry of Health. Pharmacy and Poisons Board National Core Curriculum for Bachelor of Pharmacy (B.Pharm) Degree Programme, 2014 edition, Kenya Ministry of Health. Post Training Follow-up Assessment of Kenya Health Workforce Trainings 2012, March 2014 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 54 Ministry of Health. Public Health Officers and Technicians Council – Kenya: Student Indexing Guidelines, April 2014 Ministry of Health. State of Medical Education and Training in Kenya: Bottlenecks and Best Buys: experiences of eleven health training institutions in Kenya, December 2013 (FUNZOKenya Project) Ministry of Medical Services & Ministry of Public Health and Sanitation. Kenya Health Policy 2012- 2030, 2012 MPDB. Bachelor of Dental Surgery Core Curriculum, Medical Practitioners and Dentists Board, 2013 MPDB. Continuous Professional Development (CPD) Guidelines, Medical Practitioners and Dentists Board, November 2013 MPDB. Bachelor of Medicine and Bachelor of Surgery Core Curriculum, Medical Practitioners and Dentists Board, 2013 Nursing Council of Kenya. CPD Guidelines Document, 2014 Nursing Council of Kenya. Training and Accreditation Standards, September 2014 Onudi, O. “Milestones under Business Strategy Advisory Support,” Letter from the Director of the Kenya Medical Training College, Nairobi, February 9, 2013 Presbyterian University of East Africa. Business/Investment Synopsis: Conceptual Synopsis for PUEA Learning Infrastructure (Draft), February 2014 USAID. Cooperative Agreement # AID-623-A-12-00011 - National Training Mechanism (NTN) in Kenya, United States Agency for International Development, February 2012 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 55 ANNEX 4: EVALUATION METHODOLOGY Purpose of Evaluation The purpose of the mid-term performance review was to provide information and recommendations to USAID/East Africa on Funzo’s achievements to date and progress towards its stated results in accordance with the Cooperative Agreement. The primary audience for this mid￾term review (MTR) is the USAID/East Africa mission and the Implementing and Resource Partners who will manage Funzo to its completion. Therefore the MTR report will be used to assist USAID/East Africa and Funzo implementing staff to refine and improve the Project. Evaluation Team The mid-term review team of three evaluation consultants included an international Team Leader and two local evaluators with specialist Human Resources for Health and Public Health expertise. IBTCI/ESPS in Nairobi and headquarters in Washington DC provided logistical and administrative support, quality assurance, and editorial services. Consultant Position Martina Nicolls International Team Leader (TL); M&E Expert William Kiarie HRH Specialist (Local Evaluator) Teresa Kinyari Public Health Specialist (Local Evaluator) Time Period and Geographical Coverage The evaluation commenced on September 8 and concluded by the end of November 2014. This included a home-based desk review phase and an in-country field phase (September 18 to October 16). The USAID In-Brief occurred on September 19, with the Mid-Evaluation Brief on October 7, and the Out-Brief on October 15. The draft report was submitted on October 17 at the departure of the Team Leader from Kenya. The evaluation was conducted predominantly in Nairobi – within the USAID’s Evaluation Services and Program Support (ESPS) Office, with three days in three regional locations at the Kenya Medical Training Colleges in Nakuru and Kisumu, and in the Tenwek School of Nursing. In Nakuru, Kisumu, and Tenwek, respondents from nearby locations (including other colleges, hospitals, and institutions) participated in the FGDs. DATA COLLECTION METHODS The evaluation was conducted in accordance with the 2011 USAID Evaluation Policy38 and ADS 578 for data standards so that reliable data was collected to support evidence-based findings to provide a sound basis for analysis leading to conclusions and recommendations. In order to answer the key evaluation questions, data and information was collected from various sources, such as: Document (Desk) Review USAID provided the evaluators with electronic briefing materials related to Funzo prior to the start of in-country fieldwork for review. The documentation provided the evaluators with background information to inform the types of questions for KIIs and FGDs, and to conduct data checking. The methodological approach to designing the evaluation questions were based around four main inputs in adherence with the 2011 USAID Evaluation Policy: (1) key evaluation questions in the evaluator’s Scope of Work; (2) the Project’s 4 Intermediate Results (IR); (3) a review of Project documents; and (4) questions typical of a mid-term performance evaluation – i.e. descriptive and normative. Performance evaluations focus on descriptive and normative questions: what a particular initiative has achieved; how it is being implemented; how it is perceived and valued; whether expected 38 http://www.usaid.gov/sites/default/files/documents/1868/USAIDEvaluationPolicy.pdf International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 56 results are occurring; and other questions pertinent to the design, management and operational decision-making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual.39 This mid-term performance evaluation asked questions pertaining to before-after comparisons. The evaluation used a non-experimental design since there are no comparison sites. Therefore, the evaluation team assessed Funzo against its baseline data. Implementing a survey for this mid￾term review would have yielded less than rigorous data, and less than actionable findings. This was due to the small levels of activities in some areas, and ongoing trainings. In addition, the Funzo baseline survey was a rapid cross-sectional descriptive survey using 10 surveyors and a purposive convenience sample of training institutions and regulatory bodies. It was a relevant method for a baseline assessment, but it could not be repeated during the mid-term evaluation as USAID was interested in beneficiary and stakeholder perceptions of change, and the number of surveyors required in the proposed timeline was prohibitive. Key Informant Interviews (KIIs) KIIs enabled an in-depth process of inquiry among select stakeholders and beneficiaries. Purposive KII sampling chosen by the mid-term review team included Funzo staff, resource partners, USAID collaborators, MOH health technical department officials, county health department officials, staff from training institutions, officials from regulatory bodies, pre-service beneficiaries and in-service health care workers. The evaluators developed KII interview guideline questions aligned to the SOW’s five key evaluation questions, result area-specific sub-questions, and document review. During KIIs appropriate questions were drawn from the list and customized for specific use. Focus Group Discussions (FGDs) and Roundtable Group Discussions (RGD) Based on the evaluation questions, the Project’s Results Framework, and the desk review, the reviewers prepared the assessment tools (FGD questions guides) before the data collection. The reviewers were assisted by two note-takers. The review team rotated the moderator role (using the FGD question guides) while the other evaluators also took notes. The length of the RGD/FGDs did not exceed two hours. KII and FGD Sampling The ESPS support team assisted with: (i) recruitment of KII respondents and RGF/FGD respondents; (ii) dissemination of agendas and obtaining a written confirmation from invitees; and (iii) the logistics for the group discussions and travel to the regional locations. The ESPS Senior M&E Advisor provided advice to the review team as necessary. For KIIs and FGDs, the evaluators took a purposive sample from the following stakeholder pools: Resource Partners (RP): Kenyatta University; Strathmore Business School; University of Nairobi; College of Health Sciences; University of North Carolina; Great Lakes University of Kisumu; Results for Development Institute; Kenya Healthcare Federation; and Higher Education Loans Board. Health Technical Departments (HTD) – MOH and CHD: MOH-Human Resource Department (HRD); Directorate of Health Standards, Quality and Regulation; Department of Family Health; National AIDS & STI Control Program (NASCOP); Division of Community Health Services; and County Health Departments (CHD). Regulatory Bodies (RB): Kenya Medical Practitioners & Dentists Board; Nursing Council of Kenya; Kenya Medical Laboratory Technologists & Technicians Board; Clinical Officers Council; Kenya Nutritionists & Dietetics Institute; Pharmacy & Poisons Board; and Public Health Officers Council. 39 USAID Scope of Work, July14, 2014, Annex 2, p22 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 57 Training Institutions (TI): Kenya Medical Training College (KMTC) Nairobi; KMTC Nakuru; KMTC Garissa; AMREF Training Institute; Tenwek School of Nursing; Mission for Essential Drugs & Supplies; Moi University College of Health Sciences; Maseno University; Maside Muliro University of Science and Technology; Maua School of Nursing; Kenya Methodist University; Kenya University; Pwani University; Outspan Medical College Kijabe School of Nursing; St. Joseph’s Nyabondo MTC; and Presbyterian University of East Africa. Collaborators/USAID Partners: APHIAplus Partners; AMPATH; MSH-LMS; MSH-Health Commodity Management; Emory CDC; AfyaInfo; Kenya Conference of Catholic Bishops (KCCB); Christian Health Association of Kenya (CHAK); and Supreme Council of Kenya Muslims (SUPKEM). Pre-Service Beneficiaries: Pre-service students (including student loan awardees) trained at the Nairobi, Nakuru, and Kisumu KMTCs and the Tenwek Nursing School. In-Service Beneficiaries: In-service Health Care Workers (HCW) working around the Nairobi, Nakuru, Kisumu and Tenwek locations. The review team had a three-step approach for the collection of data: Step 1: Focus Group Discussions and Roundtable Group Discussions During the initial phase of data collection in Nairobi, the review team conducted 3 RGDs (MOH HTD, CHD, and Regulatory Bodies) and 4 Stakeholder FGDs (2 TI and 2 Collaborators) to total 7 RGD/FGDs. Due to the large number of RPs, TIs and Collaborators (maximum up to 17 per target group) the review team split the FGD in two equal sessions to ensure the maximum number of participants did not exceed 10 per session. The ESPS support team organized the RGD/FGD sessions to be held in their Westlands office, except for the MOH RGD which was held in a hotel near their offices. To keep participants engaged throughout the entire session, the ESPS organized lunch during a 30 minute break. The target RGD/FGD was 9, but was reduced to 7 due to the planned two Resource Partner FGDs being converted to KIIs (mentioned in data limitations). Step 2: Field Trips Field trips were taken to Nakuru, Tenwek, and Kisumu to conduct FGDs and KIIs with pre-service students, in-service HCWs (i.e. beneficiaries), and the heads of the institutions at the KMTCs and associated facilities. Three days were allocated for the field visits in which the team split into 2 teams. However, two days were actually scheduled, due to flight times to Kisumu (thus avoiding an over-night stay). Team A conducted beneficiary FGDs and KIIs in Nairobi and Kisumu, and Team B conducted FGDs and KIIs in Nakuru and Tenwek. An ESPS member assisted with the FGDs to ensure two two-member teams. Consultant Regional Beneficiary FGD Team Martina Nicolls Team A: Nairobi and Kisumu William Kiarie Team A: Nairobi and Kisumu Teresa Kinyari Team B: Nakuru and Tenwek Maxwell Omondi Team B: Nakuru and Tenwek In total the review team conducted 8 regional beneficiary FGDs; 4 FGDs for pre-service students and 4 FGDs for in-service HCWs (one pre-service and one in-service FGD in each of the 4 locations). The total beneficiary respondents International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 58 included 27 pre-service students (from a target of 40) and 23 in-service health care workers (from a target of 24). Step 3: Key Informant Interviews The review team conducted in-depth investigations through face-to-face KIIs in particular areas of intervention which may not have been readily evident through data collected during the RGDs and FGDs. The target was 22: 2 USAID/Kenya officers; 4 TI staff; 2RP; 2 MOH representatives; 2 RBs; 2 USAID Collaborators; and 8 Pre-Service Beneficiary Students. The reviewers conducted a total of 46 KIIs in Nairobi, Nakuru, Kisumu, and Tenwek (listed in Annex 6). During an In-Brief in Nairobi, the reviewers and USAID reviewed the evaluation design, clarified issues, and finalized the methodology and selection sampling process. Organizations: As per the SOW five institutional/organization target groups were defined for the MTR: Resource Partners, Regulatory Bodies, Health Technical Departments (MOH and County Health Directors), Training Institutions, and Collaborators. Due to the limited number of organizations unified in each target group randomization of organizations was not applicable. Individuals: Individuals of each target group actively involved during the past year in Funzo interventions were identified. To select respondents from the target organizations a purposive random sampling method was applied in a three-step selection process: Step 1: Individuals were selected from a list provided to the review team by the prime IP. A pool of individuals was selected for each target group based upon criteria agreed upon by USAID during the In-Brief. Individuals within each pool’s target group had similar characteristics and were eligible (no conflict of interest) and available to participate in the MTR. From the pool, individuals were randomly selected for participation. Step 2: Individuals in each pool were sorted alphabetically. ESPS office support staff recruitment individuals by informing them by telephone that they had been selected through a multi-stage process and were invited to participate. Step 3: The individual was considered to be recruited if during the telephone call she/he confirmed willingness and availability. The recruitment was considered to be complete when the required number of respondents confirmed participation. Upon confirmation, each individual received an email providing the terms of reference (TOR) for their participation with a request to confirm their availability. When participants withdrew, additional participants were selected using the same randomization exercise for replacement. Pre-Service Beneficiaries: A systematic random sampling method was used to select pre￾service students. The ESPS support team developed a master list of all pre-service students in the most advanced cohort (intake of September 2012) that benefited from Funzo from each of the training institutions (Nairobi, Nakuru, Tenwek and Kisumu). The sampling size target for FGDs was 40 students – 10 per region (i.e. 4 FGDs). However 60 students were selected to provide a buffer to replace withdrawn respondents, duplications, or errors in contact details during the recruiting process. Despite the buffer, and follow-up telephone calls, the total number of respondents was 27. This was due to the timing of their clinical placement (attachment) in which students spend time undertaking practical experience in hospitals. In-Service Beneficiaries: A systematic random sampling method was used to select in-service HCWs. Due to overall low numbers of in-service HCWs the review team aimed to increase International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 59 respondents in each cadre (nurses, clinical officers, health records information officers and medical laboratory technologists) by inviting participants from neighboring institutions. Respondents, therefore, did not include all cadres in each FGD. However, they were proportionate in percentage to those trained, and reached a total of 23 respondents from a target of 24. DATA ANALYSIS METHODS Data analysis methods included the following: A. Content Analysis – Content analysis entailed the team’s intensive review of collected KII and FGD data to identify and highlight notable examples of successes (or weaknesses). From the RGD/FGD notes taken by the note-takers (and cross-checked through the review team’s notes), the review team created categories and a hierarchy of categories according to priority cadres and information in order to conceptualize the findings in preparation for analysis. B. Gap and SWOT Analysis – The reviewer used a gap analysis to examine which of the Project’s strategies or activities were delayed, had not commenced as planned, or had not been initially conceived but would affect successful implementation. In addition a Strengths, Weaknesses, Opportunities and Threats (SWOT) analysis was conducted throughout the review. C. Gender Disaggregation – The review team disaggregated the evidence (qualitative and quantitative information and data) by gender where appropriate. D. Triangulation of Data – The evaluators identified, compiled, and triangulated all data and findings from KIIs, RGDs and FGDs, document review, and Project statistics to provide actionable and clear conclusions and recommendations. Analytical triangulation approaches were employed to determine the Project’s outcomes and cumulative results at mid-term (rather than impact, which is largely dependent upon a longer period of time to achieve intended outcomes). Triangulation, a method used to check and establish validity of conclusions by analyzing the responses to evaluation questions from multiple perspectives, such as multiple sources of data, enabled the evaluators to ensure that the findings accurately reflected the situation and were supported by evidence. Triangulation approaches included: Methodological triangulation – At least two methods for data collection were used for the set of comparable evaluation questions. Data source triangulation – The team collected data about the Project through relevant sources, such as Project documentation, government reports, training data, and relevant statistics (see References). Investigator (FGD) triangulation – The team included the note-takers as independent field￾based observers to enable analysis of the data from different perspectives, as they provided an initial, but comprehensive understanding of focus group discussions. At the end of the MTR a validation forum will be organized to present key findings, conclusions and recommendations to select stakeholders to validate the findings and develop consensus for programming shifts. DATA LIMITATIONS Initially the review team estimated that it would conduct 17 RGD/FGDs (9 stakeholder group discussions and 8 beneficiary FGDs) and about 22 KIIs. However, the reviewers conducted 15 RGD/FGDs (7 stakeholder groups and 8 beneficiary groups) and 46 KIIs. Due to traffic constrictions for the week-long Nairobi Agricultural Trade Fair from September 29 to October 5, respondents from two scheduled Resource Partner FGDs arrived separately throughout the day (on September 29) and therefore 8 KIIs were conducted instead. However, rather than limiting information, KIIs enabled in-depth individual interviews which elicited more probing questions and International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 60 comprehensive answers. The traffic congestion in Nairobi and clinical placement (attachment) affected the ability of pre-service to attend focus discussion groups. For roundtable and focus group discussions, the availability of respondents was a challenge, despite repeated and follow-up telephone calls. This was due to people traveling from regional locations, and the busy nature of high-level stakeholders. For persons traveling from a distance, ESPS organized the logistics and paid for travel, lodging and per diems according to policies and protocols established by USAID. The ESPS support team attempted to minimize bias in the sampling process through a systematic random selection of respondents within each stakeholder pool to ensure that each individual had an equal chance of being selected during the review. The interview bias was mitigated by convening daily team debriefs, rolling data analysis, and presentation of RGD/FGD transcripts within 24 hours of the group discussions. However, some individuals cancelled their attendance due to their busy work schedules. To the extent possible, the review team contacted ‘missed’ individuals for the opportunity to attend KIIs, especially where key organizations were not represented. Hence, the number of respondents for RGD/FGDs was 88% of the target, whereas the number of KIIs was over target (46 instead of the targeted 22). The breakdown by stakeholder RGD/FGDs was as follows: 100% of targeted MOH health technical department representatives participated; 94% of training institutes (1 missed); 83% of county health departments (1 missed); 75% of USAID collaborators (2 missed); 75% of resource partners (2 missed); and 100% of regulatory bodies. Given the short time period allocated to conduct all RGD/FGDs and KIIs (7 days), this was expanded to 9 days to ensure a representative coverage of respondents. The reviewers only visited the office of the prime implementing partner, and not the offices of resource partners. A number of factors and trade-offs enabled a high coverage of respondents: (1) six out of seven stakeholder RGD/FGD, and the majority of stakeholder KIIs, were conducted in the ESPS Office reducing the review team’s travel time (at the suggestion of USAID); (2) one RGD – of MOH health technical departments - was conducted in a hotel near the MOH for a breakfast session, enabling 100% of participants; (3) beneficiary FGDs and KIIs were conducted at a convenient KMTC location enabling participants of that organization to attend, as well as participants from surrounding colleges and hospitals to participate (at the suggestion of USAID and included in the Scope of Work); and (4) splitting the three reviewers to enable simultaneous KIIs and FGDs, particularly in regions. DATA COLLECTION FRAMEWORK KEY QUESTION 1: To what extent has the Project design facilitated the achievement of mid-term mandates/ targets/ outcomes? What are the main constraints or weaknesses, opportunities & threats that need further attention? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD 1.1 To what extent has the Project supported the development of a functioning health workforce system that informs policy and planning? Descriptive MOH, TIs, RBs & RPs RGD, FGD, KII & System review Purposive As appropriate Data Analysis: Content analysis on achievements and challenges 1.2 To what extent has the Project increased affordability of marginalized groups in PST? Descriptive Ratio MOH, TIS, RPS & PST MOH, TIS & RPS RGD, FGD, KII Document review Purposive As appropriate Data Analysis: Trend and content analysis on targeting different social groups 1.3 To what extent has the Project succeeded in the development of a cost-effective approach for the expansion of admission capacity of TIs in pre-service training? Descriptive MOH, TIS, RPS & Pre-service students RGD, FGD & KII Purposive Data Analysis: Content analysis of pre-service support through scholarships and loans to expand admissions capacity 1.4 What progress has been made to establish a system for regular Analytical MOH, TIS, RPS, Colls & IST & PST RGD, FGD & KII Purposive International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 61 KEY QUESTION 1: To what extent has the Project design facilitated the achievement of mid-term mandates/ targets/ outcomes? What are the main constraints or weaknesses, opportunities & threats that need further attention? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD identification of training needs & integration into PST & IST? Data Analysis: Content analysis on achievements & challenges in establishment of the forecasting system (pre-service) & Training Needs Assessment (in-service) 1.5 What is the progress for the establishment of a functioning in￾service training model? Descriptive MOH, RBS, TIS, RPS & In-service RGD, FGD & KII Purposive Data Analysis: Content analysis on achievements and challenges (integration of TNA results) 1.6 What is the progress of demand-based PST & IST models? Descriptive MOH, RBS, TIS, RPS & Collab. RGD, FGD & KII Purposive Data Analysis: Content analysis on achievements and challenges 1.6 What progress has been made in the development of cross-cadre accredited CPD courses? Descriptive MOH, RBS, TIS, RPS & Collab. RGD, FGD & KII Purposive Data Analysis: Content analysis on achievements and challenges 1.7 What is the progress towards the development of a QA system for curriculum review and management? Descriptive MOH, RBS, TIS, RPS & In-service trainees RGD, FGD & KII Purposive Data Analysis: Content analysis on achievements and challenges KEY QUESTION 2: To what extent is the current coordination and collaboration mechanism among key stakeholders (USG IPs, relevant ministries and training institutions) effective in achieving project objectives? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD 2.1 To what extent has the Project succeeded in establishing a decentralized mechanism to deliver in-service training (hub)? Analytical MOH, TIS, RPS, Collaborator & In￾service trainees RGD, FGD & KII Purposive Data Analysis: Content analysis on mechanisms in devolution & decentralized system to county levels 2.2 To what extent has the Project succeeded in inter￾organizational collaboration? Analytical MOH, DPS, RPS & Collaborators RGD, FGD & KII Purposive Data Analysis: Content analysis on adopted synergistic approach to avoid duplication between GOK, donors & development partners KEY QUESTION 3: To what extent is the Project perceived as relevant, responsive and useful to main stakeholders (MOH and USG IPs) in addressing the challenge of having skilled and knowledgeable health care providers at facility level? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD 3.1 To what extent has the new health workforce forecast been used for policy dialogue? Descriptive Ratio MOH, TIs & RPs MOH & RPs RGD, FGD & KII Purposive As appropriate Data Analysis: Quantitative analysis of strategic decisions using forecasting system 3.2 Has the iHRIS Plan been adopted and is it updated on a regular basis? Descriptive MOH, Tis, RPs & Collaborators RGD, FGD & KII Purposive Data Analysis: Content analysis on linkage and usage of iHRIS by stakeholders 3.3 Has the curriculum review and development process been rolled out into the training institutions? Descriptive MOH, Tis, RPs, Collab., PST & IST RGD, FGD & KII Purposive Data Analysis: Content analysis on curriculum review and development integration process into the TIs 3.4 How have the joint assessment & curriculum review meetings been integrated into TI practices? Comparative MOH, Tis, RPs & Collaborators RGD, FGD & KII Purposive Data Analysis: Content analysis on curricula joint assessment process, its effectiveness and challenges to integrate into the TIs 3.5 What is the status of pilot-testing the ‘best practices’ of the clinical placement model? Comparative MOH, TIs & RPs RGD/FGD KII Purposive Data Analysis: Content analysis on challenges in meeting of clinical placement model requirements Descriptive RPs Doc. review As appropriate International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 62 3.6 To what extent has the Project supported TIs with infrastructure upgrades (equipment)? Data Analysis: Fact-based Judgmental – type, funding etc. KEY QUESTION 4: What sustainable and innovative HRH system-related models is the activity currently implementing? What does the MOH and the regulatory boards/councils think in terms of their long-term use? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD 4.1Does the health workforce forecasting system include new modules? Descriptive MOH, RB, RP RGD/FGD KII Purposive Data Analysis: Content analysis on forecasting modules 4.2 Has the health workforce forecasting system been institutionalized in MOH? Descriptive MOH, RB & RP RGD/FGD KII Purposive Data Analysis: Content analysis on adopted policies and regulatory frameworks to ensure routine review/update of forecast 4.3 To what extent has the Project succeeded to expend profit and non￾profit sector’s involvement in capacity building of health workers? Analytical MOH, RBs, RPs & IST RGD, FGD & KII Purposive Data Analysis: Content analysis on public-private partnership model adopted by the Project 4.4 To what extent has the Project succeeded in introducing innovative learning models (in-service)? Are they sustainable? Analytical MOH, RBs, RPs & IST RGD, FGD & KII Purposive Data Analysis: Content analysis on efficiency and sustainability of modern leaning models introduced by the Project 4.5 What policies, protocols & regulations have been developed /enforced to guide the integration of TNA results into the curriculum? Descriptive MOH, RBs, RPs, IST & PST RGD, FGD & KII Purposive Data Analysis: Content analysis on adopted practice to ensure sustainability of the Project intervention 4.6 What policies and guidelines have been developed to guide curriculum development review and management and for accreditation of TIs? Descriptive MOH, RBs, RPs RGD, FGD & KII Purposive Data Analysis: Content analysis on adopted practice to ensure sustainability of the Project intervention 4.7 What guidelines have been developed to review & approve CPD courses? CPD points collection system in RBs to issue relicensing/retention? Descriptive MOH, RBs, RPs RGD, FGD & KII Purposive Data Analysis: Content analysis on adopted practice to ensure sustainability of the Project intervention KEY QUESTION 5: What sustainable and innovative HRH system-related models is the activity currently implementing? What does the MOH and the regulatory boards/councils think in terms of their long-term use? REVIEW SUB-QUESTION TYPE OF EVIDENCE DATA COLLECTION SAMPLING OR SELECTION APPROACH SOURCE METHOD 5.1 What is the Project’s contribution in strengthening policy and planning processes? Purposive MOH, RB RGD & KII & Doc. review Data Analysis: Analysis of HRH targets systems-related models 5.2 What are the gaps needed for further attention? Descriptive MOH, RB RGD & KII Purposive Data Analysis: Gap analysis on programming 5.3 To what extent do current challenges threaten sustainability of systems developed by the Project? Descriptive MOH, RB RGD & KII Purposive Data Analysis: SWOT analysis of opportunities and threats International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 63 ANNEX 5: DATA COLLECTION TOOLS TEAM INTRODUCTION FOR FGDs The following is the introduction before commencement of the Round group Discussion (RGD) or Focused Group Discussion (FGD) followed by the RGD or FGD questions that follow: “Thank you for attending this focus group discussion/roundtable. It has been organized as part of the mid-term review of FUNZO/Kenya, a $40 million national health initiative implemented in Kenya from February 2012 to February 2017. FUNZO/Kenya supports the Government’s efforts to make positive changes to health worker education and training systems. The review’s objectives are to find out what has worked well, what hasn’t, the achievements to date, and any impediments to implementing the program. It also aims to determine what needs improving, and how, so that the Project can meet its anticipated results. The ground rules for participation in the focus group discussion are as follows:  First, confidentiality – what we talk about today will not be attributed to anyone but to the discussion as a whole. That’s the reason we are not tape recording the conversation, so that you can be open. Instead, our colleagues here will capture all that is said. No mention of individuals or quotes attributed to an individual will be mentioned.  Second - Please turn off your cell phone, and if taking a call step outside.  Third - Please respect the opinions of others. When responding you can mention that you disagree, politely, and state your opinion. Let others finish before you start speaking. This discussion may jump around from question to question and that’s okay. Please feel free to participate as much as possible, there are no right or wrong opinions because your experience is not the same as everyone else’s experience.  Fourth – Your attendance is voluntary. You may leave the discussion at any time.  Fifth – When you speak during the discussion please announce your organization or whether you are represented as an individual. That’s it regarding ground rules. We’ll start by introducing ourselves.” A: RGD – HEALTH TECHNICAL DEPARTMENTS (MOH and CHD) DISCUSSION QUESTIONS AND ISSUES What was the Project’s role in the development and enforcement of the new health workforce forecasting system? How useful was the system? Do you think the system plays a role in planning and decision-making? What are its strengths or weaknesses? If it has weaknesses are you still willing to keep trying to use the system? What is your role now with regard to the health workforce forecasting system for the second half of the Project? How well did the Project introduce the new system and its protocols and impart the knowledge required? How confident are you that the system and protocols has catalyzed participatory planning and decision making among government agencies? What is the collaboration like between the government, training institutions, and regulatory bodies? What makes it effective/not effective? How can improvements be made? How often, and to what extent, do the regulatory bodies review core curriculum, standards and training norms? International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 64 What’s your view on the efficiency of a decentralized mechanism to deliver in-service training? Are there still gaps in building Human Resource capabilities? What are the gaps? How can they be addressed? Is promoting evidence based (surveys and studies) planning and decision-making processes to improving the current system the best way to make improvements? Is the process succeeding? Is it on track for success? Are there other models that would be more appropriate in the Kenyan context? What’s your view on the acceptance and roll out of the modern teaching and learning practices? How sustainable are they? Does the government support these practices, in what way, and how sustainable? Do the training institutions have the capacity to adopt such modern teaching and learning practices? Are there any issues around the process of allocating scholarships and loans to students in pre￾service clinical training programs? Is it on track? Is it a useful initiative, and in what way? Are the loans going to the right students – is gender and youth considerations taken into account? In what way? To what extent has regional equity been assured in the allocation of scholarships and loans? To what extent has the Project supported the expansion of admissions capacity? How is the Project making a difference in the financial support for new health workers graduating from pre-service training? What’s your view on the quality of graduates – is this process concerned with access, retention, and completion of training at the expense of quality? What were the challenges, or expected challenges, which may threaten the sustainability of the initiatives developed and implemented by the Project? B: RGD – REGULATORY BODIES DISCUSSION QUESTIONS AND ISSUES Can one person per organization, a representative, please tell us the role your specific regulatory body performs in terms of what you performed before the Project, what gaps were identified, and whether you believe these gaps are being addressed? If gaps are not being addressed effectively, how do you think the Project can improve their support so that you can fulfil your roles? What is the collaboration like between all of the regulatory bodies? What makes it effective/not effective? How can improvements be made? What is the collaboration like between the regulatory bodies and the training institutions? What makes it effective/not effective? How can improvements be made? How often, and to what extent, do you review core curriculum, standards and training norms? What are the threats to sustainability of the review process? What is the collaboration like between the regulatory bodies and the Ministry of Health? What makes it effective/not effective? How can improvements be made? What do you think about the relevance and/or effectiveness of cross-cadre Continuous Professional Development training? Do you think gender and/or youth strategies have been considered, and in what way? International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 65 How has the adoption of curriculum reviews or the development of new curricula been understood, accepted and integrated by the training institutions? How has the process of accreditation been accepted – are there any challenges or limitations? To what extent has the working group used the accreditation guidelines, checklist, and calendar to support the training institutions? When we talk about new curriculum, we are also talking about new methods of teaching and learning, such as eLearning – how has that been accepted, and what are the advantages and disadvantages of such modern methods. Will it make a difference in terms of access and/or quality of learning? Now tell me about the usefulness of the database system to capture regulatory information – how has Integrated Health Resource Information System (iHRIS) Train been adopted for data linkages between the training institutions, the regulatory bodies, and the Ministry of Health (the Human Resource Department)? C: FGD – RESOURCE PARTNERS DISCUSSION QUESTIONS AND ISSUES What was your role in the development and enforcement of the new health workforce forecasting system? How useful was the system? Do you think the system plays a role in planning and decision-making? What are its strengths or weaknesses? If it has weaknesses are you still willing to keep trying to use the system? The public-private partnership approaches have been designed to be innovative – how are they innovative? And are there gaps in the model that need further attention? What is the collaboration like between the government, training institutions, and regulatory bodies? What makes it effective/not effective? How can improvements be made? How often, and to what extent, do the regulatory bodies review core curriculum, standards and training norms? How has the curriculum review and development of new curricula been accepted and rolled out in the training institutions? How relevant, effective, and sustainable is the review process? What’s your view on the efficiency of a decentralized mechanism to deliver in-service training? What’s your view on the acceptance and roll out of the modern teaching and learning practices? How sustainable are they? Does the government support these practices, and in what way? Do the training institutions have the capacity to adopt such modern teaching and learning practices? What were the challenges, or expected challenges, which may threaten the sustainability of the initiatives developed and implemented by the Project? D: FGD – TRAINING INSTITUTIONS DISCUSSION QUESTIONS AND ISSUES How useful was the new health workforce forecasting system? Do you think the system plays a role in planning and decision-making? What are its strengths or weaknesses? If it has weaknesses are you still willing to keep trying to use the system? International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 66 Was the Training Needs Assessment conducted? When and over what duration? Do you think it was useful, and how well was it able to identify training needs? Are there still gaps in training for further attention in the curriculum review process? How has the curriculum review and development of new curricula – the process for these – been accepted and rolled out in your training institute? How effectively has the program supported marginalized students through scholarships and/or loans? What are the advantages or disadvantages? How have the joint assessment and curriculum review meetings been integrated into the training institution’s day-to-day practice? What is the collaboration like between the training institutions and the regulatory bodies? What makes it effective/not effective? How can improvements be made? How often, and to what extent, do the regulatory bodies review core curriculum, standards and training norms? Describe your involvement with Business Development. The public-private partnership approaches have been designed to be innovative – how are they innovative? And are there gaps in the model that need further attention? How did you spend (or plan to spend) the savings? To what extent has your training institution been supported with Project infrastructure upgrades? What were the challenges, or expected challenges, which may threaten the sustainability of the initiatives developed and implemented by the Project? E: FGD – COLLABORATORS (USAID CONTRACTORS) DISCUSSION QUESTIONS AND ISSUES What are your views on the health workforce forecasting system developed by FUNZO/Kenya? Are you using it and if yes how are using it and what results are you achieving? How can the system be improved? FUNZO/Kenya has supported the development of a number of in-service training mechanisms including decentralization/hub and the introduction of accredited cross-cadre CPD courses. Give us your views on the effectiveness and any limitations of these mechanisms. What other approaches should be considered that may improve the efficiency and cost effectiveness of in￾service training? How successful has FUNZO/Kenya been in supporting the establishment of a system for regular identification of training needs and their integration into pre- and in-service training curricula? One goal of FUNZO/Kenya has been to promote inter-organizational collaboration to enhance coordination and efficiencies in health education. How effectively has the Project done this? Now tell us about the usefulness of the database system to capture regulatory/training information – how has iHRIS Train been adopted for data linkages between the training institutions, the regulatory bodies, and the Ministry of Health (the Human Resource Department)? What are some challenges that could impact the Project’s performance in the coming years? What ideas do you have on how FUNZO/Kenya can improve its effectiveness in achieving its overall goal of “increasing the number of health care workers with updated skills and knowledge”? International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 67 F: FGD – BENEFICIARIES (PRE-SERVICE STUDENTS) DISCUSSION QUESTIONS AND ISSUES In which ways has FUNZO/Kenya support or helped you enroll and remain in college? What more can be done to improve the support? How well do you feel the pre-service training is preparing you for your professional career? Probe: On completion of your training, in which part of Kenya (or elsewhere) would you like to be deployed? What are the reasons for your choice? What are the sources of funding for your training costs (school fees and other requirements)? Do you face any challenges that could threaten the successful completion of your training? If yes, what are these challenges? What can be done about these challenges? What are your views about the quality of the training that you are receiving in your college? What can be done to improve the quality of training? In which way is your training making use of information communication and technology (ICT) and innovative training approaches? How has this helped your learning? What other comments do you have about your training? G: FGD – BENEFICIARIES (IN-SERVICE HEALTH CARE WORKERS) DISCUSSION QUESTIONS AND ISSUES Tell us about the FUNZO/Kenya supported in-service training that you attended. What was the relevance of this training to your work? Let us know in which way the training has been useful in respect to the following: Enhanced skills for you to perform your tasks? Your Professional Development? Did the training contribute to your CPD points? Career growth? For example did the course lead to a promotion or make you eligible for promotion? Others? Specify_________ To what extent did your participation in the training disrupt service delivery? What are your views about the quality of the in-service training you received in respect to content, trainers, training materials and course delivery? What can be done to improve the quality of the training? In which way did your training making use of information communication and technology (ICT) and/or other innovative training approaches? How did this help in your learning? What are your views on the appropriateness of online courses? What other comments do you have about in-service training? H: KII – GENERAL KEY INFORMANT INTERVIEW GUIDE International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 68 IR 1: Supporting increased number of new health workers trained 1.1 Forecast the number of new health workers production To what extent has the project succeeded in developing a functional health workforce forecasting system that informs policy and planning? How and to what extent has FUNZO/Kenya contributed/not contributed to this process? 1.2 Create an enabling environment for training fees access What is the status of the process of allocating scholarships and loans? What are the issues surrounding the process? Is it on track? Why or why not? How well has the allocation of scholarships and loans succeeded in supporting students from resource poor areas in pre-service clinical training programs? To what extent has regional equity been ensured in the allocation of scholarships and loans to resource poor areas? To what extent has the financial support of students affected the number of new health workers graduating from pre-service training institutions? How have financial institutions and other contributors participated in this process? 1.3 Increase capacity of selected training institutions To what extent has the Project supported the expansion of admission capacity? IR 2: Supporting current health workers training needs 2.1 Training needs identification How has the Training Needs Assessment (TNA) been used by the Project? 2.2 Facilitation of the trainings To what extent has the regional strategy worked in transforming in-service training? What has been the outcome of strategies developed to mobilize funds for in-service training? 2.3 Performance evaluations How has the Post Training Assessment results informed the training strategy? IR 3: Strengthened capacity of training institutions 3.1 Develop content and courses based on set curriculum How has the curriculum review/development process been rolled out into the TIs? How have the joint assessment & curriculum review meetings been integrated into TI practices? 3.2 Improved capacity of faculty and clinical mentorship What is the status of the process of developing and pilot-testing the ‘best practices’ of the clinical placement model? 3.3 Training facility management systems and infrastructure upgraded To what extent has the Project supported training institutions with infrastructure upgrades? IR 4: Regulatory Bodies strengthened to enhance training demand 4.1 Strengthen performance gaps identification To what extent are the regulatory bodies overseeing the improvement of training institutions curriculum based on the gaps identified? 4.2 Strengthen regulation of curriculum review, development and implementation To what extent is the curriculum review working group overseeing the implementation of the core curriculum at the training institutions? To what extent has the working group used the accreditation guidelines, checklist and calendar to support training institutions? 4.3 Strengthen the link between professional licensing/ retention & CPD trainings What is the progress of the regulatory bodies’ cross-cadre CPD training coordination? 4.4 Support a database system to capture regulatory information How has iHRIS Train been adopted for data linkages between TIs, regulatory bodies & HRD￾MOH? 4.5 Support standardization of QA of trainings & health workers’ performance To what extent have Regulatory Bodies reviewed core curriculum, standards and training norms? (Modified from Scope of Work, Annex 1, pp15-17) International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 69 ANNEX 6: LIST OF PERSONS INTERVIEWED AND FOCUS GROUP PARTICIPANTS KEY INFORMANT INTERVIEWS (KII) INFORMANTS POSITION ORGANIZATION USAID Mr. Peter Waithaka Health Human Resource Specialist USAID/Kenya IMPLEMENTING PARTNER, PRIME Dr. James Mwanzia Chief of Party (COP) IntraHealth International Ms. J. Mbiyu-Kinyua Deputy COP/Technical Director IntraHealth International Dr. Joyce Kinaro Assistant Director, M&E IntraHealth International Mr. Peter Milo Assistant Director, Regional Strategies IntraHealth International Ms. Lorie Broomhall Senior M&E Research Advisor IntraHealth International, HQ Ms. Margaret Mwago Senior Manager – Public Sector IntraHealth International Mr. Martin Kinya Technical Advisor IntraHealth International Mr. David Maingi Senior Manager – Training Regulation IntraHealth International Mr. Zacharia Adendi Business Strategy Advisor IntraHealth International Prof. Stephen Okeyo Asst. Director, Systems Strengthening IntraHealth International Mr. Daniel Karenga M&E Manager IntraHealth International Mr. Danielson Onyango M&E Officer IntraHealth International Ms. Naomi Warthura Data Systems Manager IntraHealth International RESOURCE PARTNERS Mr. Abeba Taddese Program Officer, Results for Development (R4D) Results for Development Institute Mr. Joel Lehman Program Officer, R4D R4D Institute Prof. Zipporah Ngumi Resource Manager University of Nairobi (UON) Mr. James Gachari Fund Manager Higher Education Loans Board (HELB) Ms. Francisca Ongecha Lecturer Kenyatta University (KU) Ms. Lucy N. Ngatia Senior Nursing Officer Kenyatta University Dr. Amit N. Thakker Chief Executive Officer/ Public-Private Partnership (CEO/PPP) Advisor Kenya Healthcare Federation (KHF) Ms. Emily Mungai Public-Private Partnership (PPP) Manager KHF, Nairobi Ms. Elizabeth Ochieng Project Officer Great Lakes University of Kisumu Mr. Felix Odhiambo Project Accountant Great Lakes University of Kisumu REGULATORY BODIES Ms. Ida Pam Ombura Education & Continuous Professional Development (CPD) Officer Kenya Medical Laboratory Technicians & Technologists Board Mr. Mikal Ayiro CPD & Accrediting Pharmacist Pharmacy and Poisons Board Mr. Joshua Plekwa Pharmaceutical Inspector Pharmacy and Poisons Board Ms. Sophie Ngugi Head of Standards Department Nursing Council of Kenya Ms. Priscilla Najoli Head of Registrations & Licensing Dept. Nursing Council of Kenya Mr. Daniel Nashali Head of Records Office Nursing Council of Kenya Ms. Eva Wanjiku Head of Information Technology (IT) Department Nursing Council of Kenya Ms. Jostine Mutinda Acting Registrar Nursing Council of Kenya MOH – HEALTH TECHNICAL DEPARTMENTS International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 70 KEY INFORMANT INTERVIEWS (KII) INFORMANTS POSITION ORGANIZATION Ms. Charity Tauta Program Officer Community Health Services Unit (CHSU)/Family Health COUNTY HEALTH DEPARTMENTS Dr. Onyango County Epidemiologist CHD, Kisumu Dr. Charles Ngwalla County Clinical Officer CHD, Kisumu Dr. Benedict Osore County Health Director CHD, Nakuru Dr. Stanley Kiplangat County Health Executive Officer CHD, Tenwek TRAINING INSTITUTIONS Mr. Alloys Musuya Principal KMTC - Kisumu Ms. Diana Mukami Project Manager AMREF Training Institute Mr. Wilson Bii Administrator KMTC - Nakuru Mr. Jackson Masonick Principal Tenwek School of Nursing, Bomet IN-SERVICE Dr. Esther Mbithi Senior Clinician (and FUNZO Trainer) MBagathi District Hospital, Nairobi Dr. Josephine Mwagini Medical Officer MBagathi District Hospital, Nairobi Dr. Bernard Warui Health Administrator Nakuru Provincial General Hospital Ms. Alice Korir Deputy Nursing Officer Tenwek Mission Hospital, Bomet PRE-SERVICE Female Student Clinical Medicine Loan Student KMTC - Kisumu ROUNDTABLE & FOCUS GROUP DISCUSSION SUMMARIES DATE RGD RESPONDENTS VENUE M F TOTAL September 25 Regulatory Bodies 2 2 4 ESPS Office September 26 Ministry of Health 2 3 5 Panafric Hotel September 26 County Health Directors 2 3 5 ESPS Office SUB-TOTAL 6 8 14 DATE FGD RESPONDENTS VENUE September 23 USAID Collaborators (A) 3 4 7 ESPS Office September 24 Training Institutions (A) 5 4 9 ESPS Office September 24 Training Institutions (B) 3 3 6 ESPS Office September 30 USAID Collaborators (B) 5 1 6 ESPS Office October 1 In-Service HCW 2 3 5 Mbagathi Hospital October 1 In-Service HCW 2 4 6 Nakuru PGH October 1 Pre-Service Students 4 1 5 KMTC-Nairobi (0 HELB) October 1 Pre-Service Students 3 6 9 KMTC - Nakuru (2 HELB) October 2 In-Service HCW 3 3 6 Jaramogi Oginga Odinga Referral Hospital (JOORH), Kisumu October 2 In-Service HCW 4 2 6 Tenwek Mission Hospital October 2 Pre-Service Students 6 2 8 Kisumu KMTC (4 HELB) October 2 Pre-Service Students 2 3 5 KMTC - Tenwek (5 HELB) SUB-TOTAL 42 36 78 TOTAL 48 44 92 Note: Pre-Service respondents = loan students (HELB) and scholarship students International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 71 RGD/FGD RESPONDENTS KII RESPONDENTS USAID 1 Prime Implementing Partner 13 Ministry of Health Technical Departments 1 RGD 5 1 County Health Departments (MOH) 1 RGD 5 4 Regulatory Bodies 1 RGD 4 8 Resource Partners 0 FGD 0 10 Training Institutions 2 FGD 15 4 USAID Collaborators 2 FGD 13 0 Pre-Service Beneficiaries 4 FGD 27 1 In-Service Beneficiaries 4 FGD 23 4 TOTAL 15 92 46 RESOURCE PARTNER MAJOR ROLE/RESPONSIBILITY Great Lakes University of Kisumu (GLUK) Training Strategy for Community Health Workers Higher Education Loans Board (HELB) Financier of Scholarship and Loan Scheme Kenya Healthcare Federation (KHF) Private Sector; Scholarship Scheme Kenyatta University (KU) E-Learning Conversion Results for Development Institute (R4D) IHRIS Database and Dashboard Strathmore Business School (SBS) Training Management Development University of Nairobi (UON) Curriculum Development University of North Carolina-School of Nursing (UNC) Training of Trainers (TOT) for Clinical Instructors REPRESENTED ORGANIZATIONS AfyaInfo AMPATHPlus AMREF Training Institute APHIAplus – Imarisha APHIAplus – Kamili APHIAplus – Rift Valley APHIAplus – Western/Nyanza Christian Health Association of Kenya (CHAK) Clinical Officers Council Emory CDC Great Lakes University of Kisumu (GLUK) Kenya Medical Laboratory Technicians & Technologists Board (KMLTTB) Kenya Medical Practitioners and Dentists Board (KMPDB) Kenya Medical Training College (KMTC) – Garissa Kenya Medical Training College (KMTC) – Kisumu Kenya Medical Training College (KMTC) – Nairobi Kenya Medical Training College (KMTC) – Nakuru Kenya Medical Training College (KMTC) – Tenwek Kenya Methodist University (KeMU) Kenya Nutritionists and Dietetics Institute (KNDI) Kenyatta University (KU) Kijabe School of Nursing Management Science for Health (MSH) Maseno University Masinde Muliro University of Science & Technology (MMUST) Maua Methodist Hospital School of Nursing Ministry of Health – Community Health Services Unit (MOH-CHSU) Ministry of Health – County Health Directorate (CHD) - Garissa Ministry of Health – County Health Directorate (CHD) - Meru Ministry of Health – County Health Directorate (CHD) - Nairobi Ministry of Health – County Health Directorate (CHD) - Nyeri International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 72 REPRESENTED ORGANIZATIONS Ministry of Health – County Health Directorate (CHD) – Uasin Gishu Ministry of Health – Department of Family Health (MOH-DFH) Ministry of Health – Directorate of Health Standards Quality & Regulation (DHSQR) CPD Unit Ministry of Health – Human Resource Department (MOH-HRD) Ministry of Health – National AIDS & STIs Control Program (NASCOP) Mission for Essential Drugs & Supplies (MEDS) Nursing Council of Kenya Our Lady of Lourdes Mwea School of Nursing Outspan Medical Training College Pathfinder International Pharmacy and Poisons Board Presbyterian University of East Africa Public Health Officers Council Pwani University St. Joseph Nyabondo Medical Training College Tenwek Mission Hospital, School of Nursing TOTAL REPRESENTATION: (44/50) 88% OF TARGETED RESPONDENTS FOR THE MTR (+ 3 not on original targeted list) IN-SERVICE & PRE-SERVICE FGD ATTENDEES BY CADRES PRE-SERVICE FGD IN-SERVICE FGD Nurses 15 Nurses 8 Clinical Officers 5 Clinical Officers 7 Medical Laboratory Technologists 2 Medical Laboratory Technologists 6 Health Records Information Officers 1 Health Records Information Officers 2 Others (biomedicine, pharmacy, radiology) 4 Others 0 TOTAL 27 TOTAL 23 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 73 ANNEX 7: MARGINALIZED AREAS IN KENYA Kenya has 14 marginalized counties40 (marked in red) identified by the Commission for Revenue Allocation according to the Kenyan Constitution.41 40 http://www.capitalfm.co.ke/news/2013/02/14-counties-to-get-sh3bn-equalisation-cash/ 41 Policy On The Criteria For Identifying Marginalized Areas And Sharing Of The Equalization Fund-FY2011-2014 (February 2013) International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 74 ANNEX 8: REGIONAL HUBS AND SATELLITE SITES REGION REGIONAL TRAINING HUB LEAD TRAINING INSTITUTIONS SATELLITE CENTERS Nairobi Region Kenya Medical Training College (KMTC) AMREF (for e-learning& e-Induction) Mission for Essential Drugs & Supplies (MEDS) University of Nairobi (UON) Rift Valley KMTC Nakuru Tenwek Hospital School of Nursing Moi University, Eldoret Western Maseno University Masinde Muliro Uni. Science & Tech. (MMUST) Nyanza Central Region Kijabe Hospital School of Nursing (KSN) Outspan Medical Training College Eastern Region Kenya Methodist University (KEMU) Maua Methodist Hospital School of Nursing Northeastern Region KMTC Garissa None Coastal Region Pwani University College None TOTAL 8 7 Note: other resource partners for IR3include Kenyatta University, Strathmore Business School, Presbyterian University of East Africa, St. Joseph Nyabondo, and Tawfique Hospital MAP: FUNZO REGIONAL TRAINING HUBS MAP ADAPTED WITH INFORMATION PROVIDED BY INTRAHEALTH Coast Northeastern Nairobi Rift Valley Region Western Eastern Nyanza Central International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 75 ANNEX 9: SUMMARY OF FUNZO KENYA TARGETS & ACHIEVEMENTS INDICATOR TARGET ACHIEVED AT MT PERCENTAGE OF TARGET MT 2017 M F TOTAL % of MT % of 2017 Regional Hubs supported 8 8 8 100% 100% Training Institutions supported 14 26 15 107% 58% Training Institutions using TNA 14 26 10 71% 38% T.I. resources mobilized 17 26 15 88% 58% T.I. with curricula capacity 5 26 10 200% 38% T.I. implementing curriculum 5 26 10 200% 38% T.I. doing student indexing 26 21 81% T.I. accredited to provide CPD 26 21 81% Pre-service Scholarships issued 350 183 158 341 97% Pre-service Loans issued 600 118 7 100 4 2191 365% In-service HCW trained 7000 274 4 485 1 7595 109% E-conversion training 37 23 60 Courses converted online (e-L) 3-6 10 >100% Curricula reviewed/developed 30 TOT for clinical instruction 44 43 87 Nurses Inducted 757 173 0 2487 County Health Management Team (CHMT) Induction 22 29 51 Source: IntraHealth Oct. 3, 2014 (to Sept. 30, 2014). Note: 2014 ‘mid-term’ targets are to end of Year 3 (Year 3 concluded in September 2014). Funzo reports the figure of 7.595 in-service health care workers trained which includes 582 achieved by Capacity Kenya before February 2012. International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 76 IN-SERVICE TRAINING FUNDING SOURCE COURSE TITLE 2014 TARGET TOTAL TRAINED (to September 2014) M F TOTAL PEPFAR Adult Anti-Retroviral Therapy (ART) 428 718 1146 Pediatric HIV 532 804 1336 Prevention of Mother to Child Transmission (PMTCT) 579 1238 1817 HIV Testing & Counseling (HTC) New Algorithm 1135 303 274 577 Commodity Management 154 156 310 Multi Drug Resistance Tuberculosis (TB) 90 81 171 Phlebotomy 6 9 15 Harmonized HIV Curriculum 107 33 74 107 Sub Total 2125 3354 5479 (72%) MNCH Emergency Maternal Obstetric & Neonatal Care (EMONC) 158 419 577 Infant Young Child Feeding (IYCF) 161 401 562 Integrated Management of Child Illness (IMCI) 68 115 183 Essential Maternal and Newborn Care (EMNC) 23 49 72 Focused Antenatal Care 14 16 30 Sub Total 424 1000 1424 (19%) RH/ POPULAT ION Clinical Care of Sexual Violence (CCSV) 19 36 55 Contraceptive Update 16 70 86 Long Acting Permanent Methods (LPM)- Family Planning 23 66 89 Screening for Cervical Cancer 137 325 462 Sub Total 195 497 692 (9%) TOTAL 2744 4851 7595* Source: IntraHealth M&E Department, October 3, 2014 (statistics to September 30, 2014) Note: Funzo reports the figure of 7,595 in-service health workers trained which includes 582 achieved by Capacity Kenya before February 2012 that transitioned to Funzo. Funding sources include: (1) the President’s Emergency Plan for AIDS Relief (PEPFAR), (2) Maternal and Neonatal Child Health (MNCH), and (3) Reproductive Health (RH). International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 77 ANNEX 10: CURRICULA & E-LEARNING MODULES REVIEWED/DEVELOPED INSTITUTION CURRICULA REVIEWED NEW CURRICULA PARTNERS ROLLED OUT / IMPLEMENTED Great Lakes University of Kisumu (GLUK) BSc Community Health BSc Medical Lab. Sciences CUE, Nursing Council of Kenya (NCK), Kenya Medical laboratory Technician and Technologist Board (KMLTTB), Kenya Medical Practitioners and Dentist Board (KMPDB) All BSc Clinical Medicine Kenya Medical Training College (KMTC) H. Dip. Community Health & HIV/AIDS H. Dip. Emergency Care AKH, AFEM, GECC, MOH￾HP, NASCOP H. Dip. EC H. Dip. CH&H/A Basic Dip. Health Promotion Kijabe School of Nursing (KSN) BSc Nursing Dip. Perioperative Nursing CUE, NCK BSc N, KRHCN, Dip. PN KRCHN Dip. Clinical Medicine Maseno BSc Nursing CUE, KMLTTB, NCK All BSc Public Health Maua Moi BSc Nursing CUE, NCK, KMLTTB, KMPDB All B Dental Sciences BSc Physical Therapy BSc Environmental Health Masinde Muliro University of Science and Technology (MMUST) BSc Nursing BSc Midwifery CUE, KMLTTB, NCK BSc N, BSc HPE BSc Medical Lab. Sciences BSc HPE Nyabondo KRCHN CUE, NCK All Dip. Pharmacy Outspan MTC Cert. Phlebotomy Practice for Health Professionals LANCET Kenya, Bachelor of Dental Services (BDS), Clinical Officers Council (COC), NCK, KLMTTB All Presbyterian University of East Africa (PUEA) BSc Nursing BSc Clinical Medicine CUE, COC, NCK, KLMTTB BSc N, BSc OT BSc Occupational Therapy BSc Medical Lab. Sciences Source: IntraHealth M&E Department, October 3, 2014 (H. Dip. = Higher Diploma) International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 78 SUMMARY OF MODULES CONVERTED TO E-LEARNING MODE INSTITUTION COURSE CONVERTED Moi University – Eldoret Statistical Methods in Epidemiology &Human Nutrition for MPH students Maseno University First Aid Masinde Muliro University BSc. Nursing, Pathology Unit Anatomy Kijabe Hospital School of Nursing Pediatric Nursing (Kenya Registered Community Health Nurse Program) Outspan Medical Training College First Aid, Dermatology Unit ICT for Health Workers, Dermatology Unit Mission for Essential Drugs Supply Laboratory Commodity Management Course KMTC (All campuses) Health Records Course- Community Health Module E-Induction Converted an induction course with MOH &hosted at MOH Source: IntraHealth M&E Department, October 3, 2014 International Business & Technical Consultants, Inc. Mid-Term Review of FUNZOKenya 79 ANNEX 11: STATUS OF CPD GUIDELINES & CORE CURRICULA REGULATORY BODY CPD GUIDELINES STANDARDS STUDENT INDEXING Clinical Officers Council (COC) Draft In progress Existing Kenya Medical Lab. Tech. & Technologists Board (KMLTTB) Disseminated In progress Existing Kenya Nutritionists & Dietetics Institute (KNDI) At printer At printer Draft Kenya Medical Practitioners & Dentists Board (KMPDB) Disseminated In progress Existing Nursing Council of Kenya (NCK) Editing Draft Existing Pharmacy & Poisons Board (PPB) In progress In progress Existing Public Health Officers & Technicians Council (PHOTC) In progress Draft Draft Radiation Protection Board (RPB) Not Supported; RPB is under MOH Source: RGDs and KIIs with regulatory bodies REGULATORY BODY STATUS Clinical Officers Council (COC) In progress Kenya Medical Laboratory Technicians & Technologists Board (KMLTTB) Draft Kenya Nutritionists & Dietetics Institute (KNDI) In progress Kenya Medical Practitioners & Dentists Board (KMPDB) Disseminated Nursing Council of Kenya (NCK) Draft Pharmacy & Poisons Board (PPB) Draft Public Health Officers & Technicians Council (PHOTC) In progress Radiation Protection Board (RPB) Not Supported; RPB is under MOH Source: RGDs and KIIs with regulatory bodies