TR-15-119 MEASURE Evaluation PIMA Report on the Midterm Review September 2012–June 2015 December 2015 TR-15-119 MEASURE Evaluation PIMA Report on the Midterm Review September 2012–June 2015 December 2015 ii MEASURE Evaluation PIMA: Report on the Midterm Review Acknowledgments MEASURE Evaluation PIMA thanks David Omollo Owuor, Munaweza Muleji, and Kamden D. Hoff man for their signifi cant contribution to the implementation of the midterm review. Th e project also sincerely thanks all the respondents who gave their time and provided very useful and actionable feedback on the project’s performance and suggestions for the way forward. MEASURE Evaluation PIMA: Report on the Midterm Review iii Executive Summary Th e MEASURE Evaluation Phase III Kenya Associate Award Project (MEval-PIMA), funded by USAID/Kenya, builds sustainable monitoring and evaluation (M&E) capacity of Kenyan health offi cials in evidence-based decision making and improves the eff ectiveness of the Kenyan health system. A midterm review (MTR) aimed to: assess progress in project implementation to date; learn from past eff orts and experiences to improve project implementation for the remainder of the project; and provide lessons learned for building sustainable M&E capacity. Th e fi ndings of the MTR discuss changes in project scope and theory of change; strategic approaches; sector-wide support; support of the National Malaria Control Program (NMCP); support of priority counties of the President’s Malaria Initiative (PMI); support of the programs within the Reproductive, Maternal, Newborn, and Child Health Unit (RMHSU); project implementation rates; and structural and management approaches to the delivery of activities and achievement of the project’s anticipated results. Highlights of the MTR’s Findings Important changes occurred after the project’s inception that infl uenced implementation. Th e project altered its theory of change, which may have an impact on achievement of its outcomes. Th e project has remained true to its fi ve overarching approaches. At the national level, M&E capacity development included both mentoring staff and sending staff members to Ghana for training in M&E. MEval-PIMA established a website to inform and educate the public about the project and M&E capacity building. A recent data demand and use (DDU) mapping exercise by the project revealed that capacity building activities and the development and dissemination of information products have been the most frequently implemented DDU activities in the project to date. Participants in our qualitative interviews did not associate MEval-PIMA with gender mainstreaming. To address sector-wide support, the project used the following strategies to address the main issues identifi ed: (1) advocate leadership in the health sector to support M&E capacity strengthening; (2) enhance stakeholder coordination and advocacy of M&E; (3) develop tools and guidelines for essential M&E processes; and (4) establish platforms for joint sector performance reviews. Support of the NMCP focuses on eight counties where malaria is highly endemic, and addresses the quality, availability, timeliness, and use of routine malaria data. In Kisumu county, supportive supervision strengthened in subcounties: 90 percent of planned supervision visits are carried out now, whereas only half were carried out when the county was established. Support of the RMHSU consisted of strengthened capacity building in DDU; an M&E capacity assessment; and development of a capacity building plan. As for project implementation rates, MEval-PIMA achieved approximately 60 percent of the deliverables annually. Administrative and management policies are in place and are eff ective for project delivery. Th e ICF Nairobi offi ce lacks support from the home offi ce for human resources management; Nairobi staff need additional training in gender and advocacy. Consortium partners need better communication for technical, administrative, and fi nancial issues. Resource allocation and program expenditures for the fi rst three years through quarter two are provided in detail. iv MEASURE Evaluation PIMA: Report on the Midterm Review Recommendations Sector-wide support 1. Orient senior administrators/political leadership in M&E to understand clearly the wider scope of this function, and support M&E strengthening and use of data for decision making 2. Support advocacy of M&E fi nancing, especially by county governments 3. Document counties with eff ective M&E systems and share with decision makers and stakeholders 4. Advocate at the Ministry of Health (MOH) to sponsor staff at national/county levels for M&E courses at local institutions 5. Advocate at the MOH to include M&E as a key course in the Kenya Medical Training College Support of the National Malaria Control Program 1. Advocate that county governments and stakeholders support improved assurance of supply of data collection tools and harmonized data collection tools, data summary tools, and the DHIS 2 2. Support follow-on training to ensure that staff who have been trained are practicing their skills and enhancing their capacity 3. Work with service provision partners to institute data quality assurance processes at the counties 4. Build capacity of local M&E partners in order to sustain M&E capacity building after the project ends Specifi c support of county health management teams a. Build health managers’ skills in using evidence to engage county executive members, members of county assemblies, etc. b. Work with governors to advocate adequate resourcing of the health services c. Orient health workers on national guidelines at the county, subcounty, and health-facility levels d. Establish a credible system for monitoring health commodities supplies at the county level Support of the RMHSU 1. Strengthen partnerships with USAID-funded service-provider partners working at county level to undertake M&E capacity strengthening at the county, subcounty, and facility levels 2. Expand engagement to reach the higher-level decision makers at the county in order to strengthen support required to enhance and sustain county-level M&E capacity strengthening 3. Identify and focus on specifi c counties for M&E capacity building in a comprehensive manner, to demonstrate sustained eff ectiveness in the delivery of expected outcomes MEASURE Evaluation PIMA: Report on the Midterm Review v MEval-PIMA program management 1. Strengthen structure and management for optimal delivery for the remainder of the project 2. Share approved activities and budgets with staff at the beginning of the fi nancial year to ease implementation and support compliance with internal procedures and policies 3. Revisit information, communication, and technology support to the Nairobi offi ce to reduce response/completion time 4. Standardize job description formats as feasible; ensure that all staff fi les have copies of job descriptions 5. Build staff 's capacity to address the project's gender and advocacy needs 6. Revisit deputy chief of party position in the project structure; consider shifting duties to the associate directors 7. Recruit a replacement for the staff person in the position of county M&E capacity building, who is directly responsible for working with the county health management teams 8. Consider clustering county-based staff in teams with a maximum of fi ve people per team for supervision 9. Enhance teamwork and effi cient delivery by changing accountability lines to require that all Nairobi project staff set performance goals, receive daily supervision of their work, and undergo performance evaluation 10. Provide a clear picture to national and regional staff of approved activities and annual budget allocation 11. Ensure that U.S.-based staff members’ expertise is used appropriately in skills transfer to local staff 12. Continue quarterly meetings to share progress and build capacity of staff members; hold quarterly partner calls consistently MEASURE Evaluation PIMA: Report on the Midterm Review vii Abbreviations APHIA Aids, Population and Health Integrated Assistance Program AWP annual work plan BEmONC basic emergency obstetric and neonatal care CB capacity building CBO community-based organization CC county commissioner CDC Centers for Disease Control and Prevention CHAI Clinton Health Access Initiative CHEW community health extension workers CHIS community health information system CHMT county health management team COP chief of party CRO civil registration offi ce CRVS civil registration and vital statistics DCOP deputy chief of party DDU data demand and use DSS Demographic Surveillance System EGPAF Elisabeth Glaser Pediatric AIDS Foundation EmONC emergency obstetric and neonatal care EPR epidemic preparedness and response FP family planning GOK Government of Kenya HIS health information system HRIO health records and information offi cer ICT information, communication, and technology ICC interagency coordinating committee IP implementing partner IR intermediate result KII key informant interview M&E monitoring and evaluation MCU Malaria Control Unit MECAT M&E Capacity Assessment Tool MEASURE Monitoring and Evaluation to Assess and Use Results MEval-PIMA MEASURE Evaluation PIMA MICC Malaria Interagency Coordinating Committee viii MEASURE Evaluation PIMA: Report on the Midterm Review MNCH maternal, newborn, and child health MOH Ministry of Health MSH Management Science for Health MTR midterm review NHSSP National Health Sector Strategic Plan NMCP National Malaria Control Program JHPIEGO Johns Hopkins Program for International Education in Gynecology and Obstetrics OCI organizational capacity index OD organizational development PEPFAR President’s Emergency Plan for AIDS Relief PMP Performance Management Plan PMI President’s Malaria Initiative RGD roundtable group discussions RH reproductive health RMEA regional monitoring and evaluation advisors RMNCH reproductive, maternal, newborn, and child health RMHSU Reproductive and Maternal Health Services Unit SOP standard operating procedure SP strategic plan TOC theory of change TOT training of trainers TWG technical working group UNFPA United Nations Population Fund UNC University of North Carolina UNICEF United Nations Children’s Fund USAID United States Agency for International Development USD United States dollar USG United States government VA verbal autopsy VS vital statistics WHO World Health Organization MEASURE Evaluation PIMA: Report on the Midterm Review ix Table of Contents 1 Background .......................................................................................................... 1 Purpose of the Midterm Review ................................................................................1 Methods ................................................................................................................2 Limitations ..............................................................................................................3 2 Findings ................................................................................................................ 3 Strategic Approaches ..............................................................................................4 Sector-wide Support for Strengthening M&E ..............................................................7 Support of NMCP ...................................................................................................9 3 Conclusions ......................................................................................................... 17 Theory of Change ............................................................................................ 18 Cross-Cutting/Strategic Approaches ................................................................... 18 4 Recommendations ................................................................................................ 20 Sector-wide Support ......................................................................................... 20 Support of the NMCP ....................................................................................... 21 Support of the RMHSU ...................................................................................... 21 MEval-PIMA Program Management ..................................................................... 21 5 Appendices ......................................................................................................... 24 Appendix 1: Action Plans from Field Visits ............................................................ 24 Appendix 2: Deliverables Status for Sector-wide Approaches ................................ 32 Appendix 3: Deliverables Status for Support to NMCP ......................................... 33 Appendix 4: Deliverables Status for Support to RMHSU......................................... 35 Appendix 5: Financial Information ..................................................................... 37 MEASURE Evaluation PIMA: Report on the Midterm Review 1 1 Background Th e MEASURE Evaluation Phase III Kenya Associate Award (AA) Project (MEval-PIMA) is funded by USAID/Kenya to improve the lives of the Kenyan people. Th e project contributes to Result 2 of USAID/Kenya’s results framework: “Strengthened health systems for the sustainable delivery of quality services.” MEval-PIMA’s strategic objective is “to build sustainable M&E capacity of Kenyan health workers to use evidence-based decision making to improve the eff ectiveness of the Kenyan health system.” Th e project began with four intermediate results (IRs). Th ese are shown in Box 1. Box 1: MEval-PIMA Project Intermediate Results IR 1: Improved capacity of targeted programs to identify and respond to M&E information needs at national and subnational levels IR 2: Improved availability and use of quality health information at national and subnational levels IR 3: Improved capacity of local training and research institutions to meet HR [human resources] needs of health M&E professionals IR 4: Strong program management processes and systems to eff ectively achieve the fi rst three IRs Five strategic approaches to assist in eff ective implementation of activities within the IRs: (1) institutionalization of M&E in target organizations; (2) human capacity development in M&E; (3) communication and advocacy of M&E; (4) data demand and use in decision making; and (5) gender mainstreaming. Th e project directed programs at the national, county, subcounty, facility, and community levels focusing on organizational strengthening and improved approaches for dissemination and use of information. Purpose of the Midterm Review The purpose of the midterm review (MTR) was to assess progress made in project implementation to date; learn from past eff orts and experiences to improve project implementation for the remainder of the project; and provide lessons learned for building sustainable M&E capacity. Specifi cally, the team analyzed three outputs under IR1 and IR4. Assessing county-related activities under IR11 and IR22 was removed from this review after a U.S. government (USG) team conducted site visits and the MTR was refocused (see Appendix 1 for details). Th e following are the three outputs analyzed under IR1:  Output KA1-1: Sector-wide support for strengthening M&E  Output KA1-2: Improved capacity of the National Malaria Control Programme (NMCP) to identify and respond to M&E information needs  Output KA1-3: Improved capacity of the Reproductive and Maternal Health Services Unit (RMHSU) to identify and respond to M&E information needs 1 Output 1.7 in the work plan: “Improved capacity of CHMTs [community health management teams] to collect, analyze, and use M&E data for evidence-based decision making.” 2 Outputs 2.2 and 2.3 in the work plans: “Referral monitoring system operationalization and scale-up” and “Strengthened mortality and vital events data systems,” respectively. 2 MEASURE Evaluation PIMA: Report on the Midterm Review Two main questions were examined in the MTR, focusing on the technical and managerial aspects: 1. Is the direction of project implementation aligned with its strategic goal? 2. To what extent is the project structured and managed for optimal delivery? Methods The MTR employed a mixed methods approach. A desk review of the project’s performance management plan (PMP), work plans, annual reports, project theory of change, and other project￾related documents provided the data necessary for quantitative data collection related to project implementation rates. Qualitative data were used to clarify the project’s strategic approaches, its management processes, and its engagement with the benefi ciaries and stakeholders and to assess stakeholders’ perceptions of project performance. Respondents in the qualitative interviews were sampled based on their interaction with the MEval-PIMA project at the national and subnational levels and also on the focus of the project’s support within the Ministry of Health (MOH). Th e respondents were categorized as shown in Box 2. Box 2: Respondents Sampled MEval-PIMA project staff Kenya’s Ministry of Health and related departments County health management team members and county executive member for health USAID: Strategic information, President’s Malaria Initiative, and family health teams United Nations bodies: World Health Organization, United Nations Children’s Fund, United Nations Population Fund Other stakeholders: Management Science for Health; Clinton Health Access Initiative; Afya￾info; Johns Hopkins Program for International Education in Gynecology and Obstetrics; AIDS, Population And Health Integrated Assistance Program; and Kenya Medical Research Institute–U.S. Centers for Disease Control and Prevention. Key informant interviews (KIIs) and roundtable group discussions (RGDs) were held with stakeholders at the national and subnational levels to gather perspectives on the performance of the project and the contribution of the fi ve project strategies to M&E system strengthening. Th e KIIs and RGDs captured perspectives on M&E gaps and emerging issues that might be addressed by the MEval-PIMA project, as well as suggestions for improving the project. Discussion guides for the KIIs and RGDs were developed, reviewed, and approved by MEval-PIMA’s MTR technical working group. Tools were also shared with the mission for input. When face-to-face interviews were impractical, telephone calls or Skype calls were used. Verbal consent was sought from all respondents before each KII or RGD began. Th e RGDs had two to fi ve participants. All RGDs were conducted face-to-face except one, which was completed by Skype. All interviews were audio-recorded. Consultants and note takers discussed the data collected, to fi ll any gaps and to establish common themes. Th e data were entered in data summary matrices to facilitate visualization of perspectives and opinions from the respondents around each of the MTR questions. To the extent possible, quantitative and qualitative data were triangulated to reduce the possibility of bias and ensure that inferences, judgments, and conclusions were as accurate as possible. MEASURE Evaluation PIMA: Report on the Midterm Review 3 Limitations Qualitative data for the NMCP were collected only from the county health management teams (CHMTs) in two counties, as requested by the President’s Malaria Initiative (PMI); this limited representation of the teams’ progress. Further, PMI requested that one of the counties selected should be where the project had not implemented malaria activities, to anticipate future malaria M&E needs. Implementation rates use crude estimates for completion; even so, they provide a good representation of project accomplishments. 2 Findings Th e MTR’s main fi ndings focus on changes in project scope and theory of change; strategic approaches; sector-wide support; support of the NMCP; support of specifi c PMI-supported counties; support of the RMHSU; implementation rates; and structural and management approaches to delivery of key activities and results of the project. An overview follows. Changing environment: Important changes occurred after the project began, and they infl uenced the project’s implementation. Baseline assessments in Year 1 were delayed, owing to security concerns, elections, and a reorganization of the government at the national and subnational levels. Project startup and theory of change (TOC): Th e project experienced some challenges in recruitment of senior staff and delays in funding while PEPFAR reviewed its funding priorities. Th e project experienced noteworthy changes in its TOC that may have an impact on outcomes. Figure 3 shows the original TOC. Th e TOC changed in the following major ways:  PEPFAR shifted away from support for M&E champions to identify and address M&E capacity gaps and enhance the institutionalization of project interventions. Instead, the agency favored strengthening the MOH’s new M&E unit, with the intention that this unit would take the lead in building the capacity of the ministry’s M&E system. Th e approach to develop M&E champions through coaching and mentoring is still being implemented with a lower cadre than originally proposed and is executed mainly at the county level.  When the Kenyan government decreased its commitment to co-fund the establishment of Centers of Excellence for improved community health information systems, USAID halted its own fi nancial commitment for technical assistance through the middle of Year 2.  Because the Kenya Health Policy (2014–2030), outlining the need to strengthen the referral system, was released only a few months after project startup, a functional referral system had not yet been established. Th e project shifted its focus from monitoring referral standards and training staff to helping fi nalize and launch the Kenyan Health Sector Referral Strategy (2014–2018). Th is included health worker orientation on the referral system; support of the referral system technical working group (TWG); development of referral tools; incorporation of referral system strategy indicators in national reporting; mapping, engagement, and mentorship of stakeholders to avoid duplication; and encouragement of referral links among facilities.  IR3 was defunded at the end of Year 2; thus, the project did not carry out activities to improve the capacity of institutions to provide M&E services. Th e project will no longer contribute to strengthening local capacity for developing health professionals with M&E skills in the medium- to long-term. 4 MEASURE Evaluation PIMA: Report on the Midterm Review Figure 3: Project Theory of Change Capacity of health programs to identify and respond to M&E information needs Strengthen M&E leadership and management National system capacity Skills sufficient among workforce Adequacy of M&E training Higher education technical, administrative, and financial capacity Effectiveness of training delivery Higher education capacity to train in M&E Number of students trained in M&E by Kenyan higher education institutions Capacity to provide M&E services Local operational capacity to train in M&E Need for short-term training Need for skills updates over time among existing M&E staff To build Kenya health workers capacity to make evidence-based decisions to improve the Kenyan health system Operationalization of referral monitoring systems Availability of CHIS data Availability of mortality and other vital statistics Availability and use of quality health information at national and subnational levels Strategic Approaches MEval-PIMA emphasizes fi ve cross-cutting issues, or strategic approaches (integration, gender mainstreaming, sustainability, organizational development [OD], and data demand and use [DDU]) in achieving the results of the project.3 Discussions with respondents showed that through these approaches, MEval-PIMA contributed to: (1) thoughtful action around M&E questions in complex public health work (e.g., newborn care); (2) develop￾ment of assessment and monitoring systems, includ￾ing referral and civil registration work; and (3) devel￾opment of metrics for monitoring, evaluation, and capacity assessment. Institutionalization of M&E in target organizations Organizational development is the main approach the project applies. Our goal is to enhance institutionalization of the M&E function, by promoting ownership of, participation in, and sustainability of the M&E function. Th e project has 3 Because cross-cutting approaches are, by defi nition, integrated in all activities, extracting the impact of each approach is diffi cult. Box 1. Examples of OD Contributions  Consensus-building meetings with senior leaders at the MOH  Mapping of M&E stakeholders at the national and county levels  Formation of health-sector M&E TWGs MEASURE Evaluation PIMA: Report on the Midterm Review 5 remained true to the fi ve overarching OD approaches: strengthening the voice and impact of leaders; aligning stakeholders to support M&E capacity strengthening; developing tools and guidelines to support engagement and implementation processes; building coalitions to support M&E reforms; and spreading motivation for change, by creating a pool of internal and external change agents. Capacity development in M&E The tailored M&E capacity-building plans focus on areas in the benefi ciary organization where capacity weaknesses were identifi ed. However, since the M&E cadre has been largely absent in the health sector, trainings to equip staff with M&E skills have been the most visible intervention in capacity building. For instance, staff members were trained in data analysis and reporting and generation of information products such as malaria surveillance bulletins, annual reports, etc. Workshops were conducted, many at county level, to orient staff on new M&E policies, guidelines, and tools. At the national level, capacity development included mentoring staff , by working in tandem with them on M&E tasks. Th e NMCP’s MEval-PIMA staff members benefi ted from an international training on M&E in Ghana. Communication and advocacy MEval-PIMA proposed to use communication and advocacy to enhance institutionalization of M&E at the national and county levels. Th e project’s communications strategy helps inform stakeholders of the progress MEval-PIMA is making toward delivery of the intermediate results, and to share M&E technical knowledge. Th e strategy is inclined more toward project branding through communication of project activities and does not emphasize advocacy for M&E capacity building enough. Th e project’s communication strategy targeted internal and donor audiences to improve the understanding of MEval-PIMA’s role in M&E capacity building. Achievements thus far include the establishment of the MEval-PIMA website—http://www.cpc.unc. edu/measure/pima—to inform and educate the public about the project and about M&E capacity building. Th e site off ers fact sheets, program and county profi les, success stories, and newsletters. Th e project conducts routine monitoring of website hits (requests):19,695 from the site’s launch on May 6, 2013 through June 30, 2015. Th e project also shares M&E technical knowledge with health and M&E audiences through a mailing list and through the community of practice established by MEval-PIMA; approximately 1,000 members are reached through these two modalities. Th rough the MEval-PIMA community of practice, audiences discuss M&E issues and are linked to M&E resources. Data demand and use (DDU) Th is strategic approach is a major pillar in the delivery of the project’s overall goal to build sustain￾able M&E capacity and enhance use of quality health data for evidence-based decisions and program planning. Assessment of the data use contexts was done as part of the M&E Capacity Assessment Tool (MECAT) at the national and county levels. Th e project activities focused on building capacity Box 2. Examples of DDU Products  Malaria bulletins have been developed quarterly.  Sensitization and rollout of RH scorecards has occurred in multiple counties.  Reproductive health program profi les have been produced.  M&E capacity building plans, of which DDU plans are an integral part, have been developed and disseminated at the national level and in several counties. 6 MEASURE Evaluation PIMA: Report on the Midterm Review Box 3. Increasing Birth Registration in Garissa Birth registration is essential to public health, because it allows authorities to estimate the size and scope of epidemics. During a data review exercise in Garissa, it emerged that birth registration was 29.7 percent—way below the national average of 58.4 percent. To address this challenge, collaborative action planning meetings were held with various stakeholders, led by the CHMT and Civil Registration Service. MEval-PIMA supported two subcounties in Garissa in addition to another county, by training local registration agents and placing them in local health institutions, providing data collection tools, and engaging facility health offi cers to register births through a maternal child health strategy initiative. The issuing of free birth certifi cates at the facility meant that the community wouldn’t have to travel long distances to get birth registration certifi cates. This led to notable increases in the number of birth registrations. It also served as an incentive for expectant mothers to deliver in hospital. Other subcounties in Garissa are planning to apply the same intervention to improve birth registration. …I am very happy that my son got his birth certifi cate within a short time. When I told my husband that our son’s birth certifi cate was ready for collection at the hospital, he couldn’t believe it. My husband doesn’t like hospitals. But on that day he accompanied me to hospital to confi rm it was true. Since then, it is easier for me to bring my son to clinic… —Mother of a young son in DDU and improving data availability. A recent DDU mapping exercise by the project revealed that capacity-building activities (i.e., data reviews, performance reviews, trainings, malaria surveil￾lance, reproductive health [RH], basic emergency obstetric and neonatal care [BEmONC], and the Reproductive, Maternal, Newborn, and Child Health [RMNCH] Scorecard) and the development and dissemination of information products have been the project’s most frequently implemented DDU activities to date. Availability of county profi les resulted in users and producers engaging to review data and trends and make informed decisions, thereby promoting accountability. Th e project has documented success stories, such as Garissa County’s improved demand for and use of data, as summarized in Box 3. Participants in the KIIs emphasized the increasing value of data review meet￾ings as an integral part of the system. At the county level, discussions with stakeholders revealed challenges in their capacity to translate access to data into useful action. Organizational development, leadership, and political challenges at the county level also hamper health managers from making bold but necessary decisions. Structured meetings are absent at the county and national levels to address how the institutions involved would work together, leveraging their comparative advantages to optimize results. Th ere was also no discernible eff ort to cascade national discussions and guidelines about partnership and collaboration to the fi eld at county level. Gender mainstreaming MEval-PIMA proposed to integrate gender, emphasizing sex- and age-disaggregated data collection, analysis, and use throughout all activities, in order to facilitate gender-sensitive programming and informed decision making. MEval-PIMA staff received training on gender mainstreaming at the beginning of the project, and gender assessments were carried out for the NMCP as part of fulfi lling MEASURE Evaluation PIMA: Report on the Midterm Review 7 Global Fund proposal development requirements and completing county assessments.4 Trainings in DDU emphasized gender sensitivity. However, it was found that although facility data are sex￾disaggregated, the DHIS 2 is aggregated. Apart from this initial eff ort at gender mainstreaming, the project did not map specifi c gender-mainstreaming activities, and progress in this domain is not monitored in the PMP. However, programs supported by MEval-PIMA, such as the NMCP, seem to be gender-aware as a result of project activities. Interview participants did not associate MEval-PIMA with gender mainstreaming at all; some were surprised that it is one of the project’s strategic approaches. Sector-wide Support for Strengthening M&E Project focus MEval-PIMA’s intention is to develop and enhance sector-wide capacity for M&E planning, coordination, and policy development. According to a review of project documents, general support of the health sector aimed to facilitate development of M&E structures and processes to ensure eff ective coordination and delivery of high-quality data from the community level to the national level. Th e project’s support addresses the following issues:  Weak M&E capacity, which previous health sector assessments identifi ed as a critical gap  Unclear roles, responsibilities, and linkages with other departments in the M&E sector  Lack of a focal point and weak mechanisms in the health sector to coordinate M&E capacity- building eff orts and investments across the sector  Weak operationalization of M&E in Kenya’s Health Sector Strategic and Investment Plan  Inadequate transparency and public accountability in the health sector across the country Th e project used the following strategies to address the key issues: (1) advocate support of M&E capacity strengthening by health sector leadership; (2) enhance stakeholder coordination and advocacy for M&E; (3) develop tools and guidelines for critical M&E processes; and (4) establish platforms for joint sector performance reviews. Relationship building The project gained acceptance by senior leaders, but then reduced this engagement to focus on the technical cadre. Although the project has worked closely with its benefi ciaries (i.e., MOH and related departments and other M&E implementing partners), the working relationships formed for implementation of the project were informal, as indicated by some respondents in the KIIs and RGDs. Differing project expectations Though most partners interviewed agreed with project interventions under MEval-PIMA’s sector-wide support, in several cases, expectations of the partners went beyond the project’s scope. For example, USAID expected MEval-PIMA to support the MOH in the development of policies, national guidelines, and tools, 4 MEval-PIMA supported development of a report on the relationships between gender and malaria in Kenya, which can be found here: http://www.cpc.unc.edu/measure/pima/malaria/gender-and-malaria-in-kenya. …They (MEval-PIMA) are with us as we make our work plans, so they know where to support, but we don’t know much about them …what else they can support. —MOH national staff member 8 MEASURE Evaluation PIMA: Report on the Midterm Review and roll them out to the counties; help standardize the tools and training of health workers; and help counties develop M&E plans. In contrast, the national-level M&E unit of MOH expected also to receive support in developing health sector investment plans, work plans, and guidelines for use at the national and county levels. MEval-PIMA did help the M&E unit draft its annual work plans and used the process to identify the areas where the project could support the M&E unit. Achievement of deliverables At least eleven activities were not included in the work plan. However, these activities were mentioned in the quarterly reports. In most cases, these activities were added later and, therefore, did not have timelines in the work plan. Nine of these activities were in Year 2, and one activity was in Years 1 and 3. Appendix 2 provides a table of deliverables and their annual status. It is the practice of the project to note additional tasks and deliverables in the quarterly report and not amend the work plan to include the new activities. Changes in activities from Year 1 to Year 2 represent better clarity from MOH, USAID, and stakeholders as devolution evolved. Th e deliverables achieved thus far under sector-wide support focused on stakeholder engagement; arranging forums for coordination and data review; and developing key guidelines for M&E. A systematic baseline M&E capacity assessment of the M&E unit at MOH headquarters has yet to be undertaken, though some capacity building activities, such as staff training, are ongoing. Table 1 represents key facilitators and challenges in implementation of sector￾wide support in M&E. Th e main changes in MEval-PIMA’s sector-wide support cited by respondents were:  Strengthened capacity of the M&E unit in the central MOH through improved skills of staff members in data analysis, interpretation, and reporting; in performance review; and in the development of strategic and investment plans  Improved coordination and collaboration facilitated by the establishment of M&E TWGs in 17 counties  Enhanced opportunities for knowledge exchange and data use through support of platforms such as the Annual Health Congress, a learning and performance review forum Most of the respondents believed that the establishment of the M&E unit at the MOH was solely the result of MEval￾PIMA’s work. While MEval-PIMA’s work may have contributed to establishment of the M&E unit, a review of documents from the M&E directorate of the Ministry of Devolution and Planning illustrates that a government policy already existed to establish such units in all ministries. This misunderstanding refl ects the profi ling of MEval-PIMA as the lead player, rather than as a technical assistance partner, in M&E in the health sector at the national level. MEASURE Evaluation PIMA: Report on the Midterm Review 9 Table 1: Facilitators of and Challenges in Implementing Sector-wide Support Facilitators  Strong M&E skills of MEval-PIMA staff  Longstanding working relationship with the MOH through previous projects  Close collaboration with implementing partners (United Nations Population Fund, United Nations Children’s Fund, and the World Health Organization)  Flexibility in a changing environment Challenges  Shifts in MOH staff during devolution, requiring spending more time on relationship building  Weak collaboration between county health departments and MOH, which delayed consensus building on guidelines and policies and also delayed project implementation  Realignment of PEPFAR priorities, resulting in changing funding priorities Support of NMCP Project focus MEval-PIMA’s support of the NMCP is aligned with the revised Kenya Malaria Strategy 2014– 2017, Objective 4: “To strengthen surveillance and M&E systems so that key malaria indicators are routinely monitored and evaluated in all counties by 2017.” MEval-PIMA’s support of the NMCP is a continuation of MEASURE Evaluation Phase III’s eff orts to strengthen malaria M&E, surveillance, and evaluation. Th is prior work included identifi cation of key indicators and dashboards for routine M&E; identifi cation of key data gaps from the HMIS and commodities and laboratory information systems; development of customized tools for data collection; collation and dashboard production; and creation of a national surveillance rollout plan. MEval-PIMA focuses on eight counties where malaria is highly endemic: Kisumu; Homabay; Migori; Siaya; Busia; Bungoma; Kakamega; and Vihiga. Th e main focus is to improve the quality, availability, timeliness, and use of routine malaria data. MEval￾PIMA therefore aimed to support NMCP to strengthen its routine surveillance systems, synthesize data into malaria information products, institutionalize the use of the malaria information acquisition system, and improve malaria data quality through regular data quality audits, in collaboration with malaria stakeholders. Th e baseline M&E capacity assessment of the NMCP conducted in 2013 indicated an organizational capacity index (OCI) score of 65.25 percent,5 far ahead of comparable public health programs in Kenya. Achievement of deliverables Implementation of activities was initially hampered by the movement of government staff and reorganization at the MOH during devolution. In Year 2, delays were related to unclear coordination mechanisms between the county and the national governments. Appendix 3 provides a table of deliverables and their annual status. Table 2 represents key facilitators and challenges in 5 Th e OCI is a composite measure that allows the project to systematically measure outcomes in capacity building towards strengthened M&E. Th e OCI was used to assess baseline and monitor annual change in capacity to strengthen M&E in order to answer the implementation question, “How has the capacity to implement functional M&E systems changed”? Th e higher the number, the stronger the organizational capacity. MEval-PIMA has a long working relationship with the NMCP, and their staff members are committed to delivering on the activities they pick off the NMCP annual work plan. —NMCP staff member 10 MEASURE Evaluation PIMA: Report on the Midterm Review implementation of MEval-PIMA’s support of the NMCP. Th e main changes cited based on the KIIs in support of the NMCP are:  Improved staff skills in data analysis, surveillance, and generation of information products, such as malaria bulletins, which are likely to lead to enhanced data use  Improved coordination and collaboration resulting from the establishment of M&E TWGs in counties  Enhanced opportunity for knowledge exchange and data use through forums such as the Kenya National Malaria Forum Table 2: Facilitators of and Challenges in Implementing NMCP Support Facilitators  Established working relationships with NMCP staff  Project’s commitment to deliver on activities identifi ed in the NMCP annual work plan  MEval-PIMA’s presence in numerous coordinating structures/TWGs  Working in close consultation with partners (AIDS, Population and Health Integrated Assistance Program (APHIA) + Western/MalariaCare)  MEval-PIMA’s participatory approach to training MOH staff to develop information Challenges  Poor coordination of planning and program activities between national and county levels  Low capacity of MOH staff to use routine health information system (DHIS 2), especially at the county level  Inconsistent provision of supportive supervision, particularly in the counties  Weak follow-on supervision and mentoring of facility workers trained on surveillance  Inadequate staffi ng of health facilities in counties, resulting in limited time to devote to M&E tasks  Lengthy MOH decision and implementation cycles, which sometimes stretch the delivery timelines for MEval-PIMA Support of the NMCP to strengthen the capacity of county health management teams Project focus The project intends to build the capacity of health managers at the subnational level to collect, analyze, and use data in the management of health services, to ensure eff ective health system performance. Th e counties have a critical role in the establishment of sustainable and credible M&E systems, because they are responsible for health service delivery at community and facility levels—the source of the data that populate the national health information system (DHIS 2). MEval-PIMA’s regional M&E coordinators provided mentoring and coaching to strengthen practices to ensure the quality of data entry in the DHIS 2; training and building capacity at the county level in data use at point-of-collection and nonroutine data collection (rapid methods for outcome monitoring); and monitoring the fl ow of information through the M&E system from the community level through the national level. To explore the impact on county-level M&E of MEval-PIMA’s participation in national-level interventions, the consultants visited two counties—Busia and Kisumu—at the request of PMI. Th e two counties have high malaria endemicity and high maternal mortality rates. MEval-PIMA conducted baseline M&E capacity assessments in Kisumu, and interventions are underway in that county. Th e project’s support of the eight PMI focus counties has had two phases; Kisumu is in Phase 1 and Busia is in Phase 2. By the time of the MTR, no malaria activities had been MEASURE Evaluation PIMA: Report on the Midterm Review 11 undertaken in Busia. PMI requested that Busia county be included in the MTR for comparison with a county that had received MEval-PIMA support, for an indication of the malaria M&E services that will be needed in future. Table 3 presents the data challenges discovered in both counties and progress to date in Kisumu county. Table 3: Data Challenges and Responses  Kisumu/Busia Data Challenges  MEval-PIMA Activities in Kisumu  Data availability and quality issues  Shortage of updated data collection tools  Data collected with improvised tools  Partner support not well- coordinated  P oor use of data by health service managers  Support supervision is weak  Worked with the CHMT to produce a health strategic and investment plan incorporating an M&E capacity building plan  Baseline assessments  Surveillance training  Health sector strategic and investment planning Kisumu results to date Supportive supervision was strengthened in subcounties, which in turn empowered staff to supervise in the facilities; 90 percent of planned supervision visits are actually carried out now, as compared to half when the county was established. Th e use of data has improved; the county corrected its base population to ensure the accuracy of its service coverage estimates. Because data quality also improved, health facilities are making more accurate forecasts of their health commodity needs, and the number of reported shortages is declining. Th e CHMT hopes that the Kenya Medical Supplies Authority will respond to the county’s use of actual data for ordering supplies, rather than imposing supply caps. Busia results to date Th e CHMT added a county “M&E offi cer” position to the staff establishment to help address data availability and quality issues. Th e team hopes also to establish an M&E unit. Th e team recognized the need for closer supervision of community health workers, to ensure the quality of services and data. Th ese activities took place with MEval-PIMA’s help, but they had been highlighted as gaps in the overall MECAT assessment in 17 counties. Table 4 presents key facilitators and challenges in supporting the CHMTs. Table 4: Facilitators of and Challenges in Implementing CHMT Support  Facilitators  Free maternity care reimbursements from the MOH based on data from the DHIS 2 motivated facility managers to focus on completeness and quality of data entered in the system. Counties established forums to ensure that data in the DHIS 2 refl ect the facility records.  In Kisumu, the county government allocates funds and fuel for supportive supervision.  Challenges  In Busia, funds allocated for supportive supervision can be diverted to other projects, limiting the number of supervisory visits.  Counties are opening new health facilities without proper planning, exacerbating already poor staffi ng levels.  Political and administrative county leaders are still not effectively engaged by the health managers and other stakeholders to adequately resource health services. 12 MEASURE Evaluation PIMA: Report on the Midterm Review Support of RMHSU Project focus MEval-PIMA’s support of the RMHSU contributes to the National Reproductive Health Strategy 2009–2015 objective to enhance the provision of a comprehensive range of essential sexual and reproductive health services through strengthened M&E capacity to increase availability and use of information for evidence-based decision making, planning, and program interventions. Th e project focuses on maternal and newborn health, family planning (FP), and M&E programs of the RMHSU at the national and county levels. Th e RMHSU received support from Phases II and III of MEASURE Evaluation, prior to the start of MEval-PIMA. Th is support consisted of development of an M&E plan, a maternal and perinatal death review database, and mentorship of two national DDU champions. After the project conducted a baseline capacity assessment (with an OCI score of 38.2 percent), the project found issues related to poor routine reporting, poor submission of data on maternal deaths, and limited capacity to analyze reproductive health information. Th erefore, the project focused on the following areas of work:  Build strong partnerships to support planning and coordination of M&E  Conduct M&E capacity building for RMNCH at the national and county levels  Improve the availability and use of strategic information for decision making in RMNCH and FP  Provide technical assistance in M&E to the RMHSU: e.g., reviewing M&E plans Achievement of deliverables Th e project was expected to support the creation of a robust M&E unit to respond to data needs at the RMHSU; orient RMHSU staff on M&E basics to enable the staff to summarize and present their information; and strengthen the M&E unit in the new areas of FP and maternal and child health. Th e deliverables in Year 1 were mainly capacity building and strengthening data demand and use. A MECAT was carried out and a capacity building plan developed. Only one deliverable in Year 2 had planned timelines. Year 2, like Year 1, focused on capacity building in reporting and enhancing participation in coordination forums. In Year 3, the project trained county staff on the RMNCH scorecard and trained county FP teams on commodities reporting. Reproductive health staff in 15 counties were trained to conduct EmONC assessments. Appendix 4 presents a table of deliverables and their annual status. Table 5 shows key facilitators of and challenges in implementing support of the RMHSU. The scorecard allowed visibility in the system of what everyone is doing. For instance, if a county sees red in their scorecard they take ownership and change to improve on their indicators. —USG service delivery partner MEASURE Evaluation PIMA: Report on the Midterm Review 13 Table 5: Facilitators of and Challenges in Implementing Support of the RMHSU Facilitators  Technically competent MEval-PIMA M&E staff committed to delivering in this area of support  Good relationship between MEval-PIMA and RMHSU staff Challenges  Frequent change of key program staff at RMHSU and weak ownership of some achieved deliverables by RMHSU  Limited support from the executive and political leadership for strengthening the M&E function to promote delivery of health services at the county level  Weak capacity of county staff to use routine health information systems (DHIS 2); system perceived as a conduit for submitting information to the national level  Diffi culties coordinating activities between national and county RH programs Implementation rate In estimating the implementation rate for each targeted program, the MTR took into consider￾ation the total number of deliverables planned for the year, and the shift within deliverables among those canceled and new ones that were added during the year. Th e assumption was that all deliverables planned for the year were intended to be delivered by the close of that year. Any deliverable achieved in that year was given a score of 1 point. Any deliverable making good progress but not fully achieved by year end was given a score of 0.5 points. Th is score seemed to be the best way to quantify partially achieved deliverables; however, the degree of completion of the partially achieved deliverables was not calculated. Th e denominator was the total number of planned deliverables and additional deliverables, minus the number of deliverables that were canceled that year. Th e number of deliverables postponed to the following year was retained in the denominator. A percentage score indicative of the rate of achievement of deliverables in the respective year was then calculated, using the following formula: Applying this, the rate of implementation of interventions for sector-wide support over the past two and a half years is as follows:  Year 1 = [(0+0.5)/(1-0)] X 100 = 50%  Year 2 = [(2+0.5)/(5-1)] X 100 = 63%  Year 3 (up to quarter 2) = [(0+1)/(4-0)] X 100 = 25% Th e indicative implementation rates have been similarly estimated for the project’s support of NMCP and RMHSU. All are as listed in Table 6. Rate (%) = [(1 X deliverables achieved) + (0.5 X deliverables partially achieved)] X 100 (total planned deliverables + deliverables added in year – cancelled deliverables) 14 MEASURE Evaluation PIMA: Report on the Midterm Review Table 6: Implementation Rate Program Year 1 Year 2 Year 3 Annual Average Sector-wide 50% 63% 25% 46% Reproductive Health 67% 79% 54% 67% Malaria 43% 71% 70% 61% MEval-PIMA Project Average 53% 71% 50% 58% Th e MEval-PIMA project achieved approximately 60 percent of the deliverables annually. Th e rate was explicably lower in Year 1 across the programs, because time was spent engaging with the benefi ciaries and other stakeholders, fi nalizing project implementation and monitoring plans, and recruiting project staff . Th e rate increased in Year 2 as baseline assessments and trainings were completed. In Year 3, the rate is likely to be higher, given that the program’s implementation rate had already reached 50 percent by the second quarter. Structural and management approach for the delivery of results Management Policies and procedures Administrative, fi nancial, procurement, and human resources management policies are in place. Th e administrative policies currently are eff ective for project delivery. Th e agreement among MEval-PIMA consortium partners is for ICF to manage in-country procurement. Th erefore, procurement processes are characterized by centralized support in ICF/ Nairobi and headquarters, and require staff to initiate procurement with suffi cient lead time to avoid delays. Centralized support creates a bottleneck when a number of concurrent activities are being undertaken that require support from the limited procurement staff . Time zone diff erences also contribute to delays in procurement when U.S.-based support is required for goods and services costing more than $3,000 and a local purchase order cannot be raised. Th e human resources policies for ICF staff are executed remotely by U.S.-based staff . Th e lack of signifi cant local human resources management seems to weaken ICF staff engagement. A review of the policies shows that they adequately address: 1) liquidity and credit risks; 2) project delivery risks; 3) compliance risks; and 4) operational risks. Th e policies are introduced to staff during the orientation process and are enforced. However, during implementation of project activities, there were instances in which technical staff were eager to implement, and overlooked established procedures and required timelines. Activity implementation When planning implementation activities, staff are required to prepare a concept note detailing the activities to be undertaken, which is approved by the activity leads and forwarded to the appropriate associate director. Staff reported that approval at this stage is prone to delay and on MEASURE Evaluation PIMA: Report on the Midterm Review 15 occasion the details in the operation plan are signifi cantly diff erent from those in the work plan as understood by the implementing staff , which led to cancellation of the planned activity. During implementation, the project developed a cascading process to link interventions at the national level to the county level, which created two clusters of staff : one around the associate director for capacity building at the national level and the other around the associate director for capacity building at the subnational level. Th ese clusters speak internally among themselves; information is shared usually by the associate directors and the activity leads, which seems to be working well. In Year 2, a bundle6 of strategic approaches to the benefi ciaries’ activities based on contextual needs led to signifi cantly improved delivery of those activities. Active communication forums—e.g., biweekly management meetings and follow-up meetings on all levels—are in place and operational, including quarterly meetings for all staff members to build staff capacity and to address project-wide issues. Staffi ng Skills gaps Although MEval-PIMA’s staffi ng structure has signifi cant strengths, several weaknesses hamper the project’s progress. Human resources management issues for ICF staff need to be addressed: for example, the lack of human resources management support by the ICF home offi ce for the Nairobi offi ce. Th ere is a lack of strong gender skills. In working with NMCP, the local staff relied on expertise from the U.S. to deliver a gender strategy. However, during the process, there was negligible transfer of skills to local staff . Advocacy skills were also noted as a gap among staff . Lack of understanding by staff inhibited advocacy of M&E. Th e Kenya staff appreciate the backstopping provided by the U.S.-based technical staff and gave that a rating of four7 during the MTR. Co-location At the county level, MEval-PIMA co-located staff mostly with the MOH; MEval-PIMA staff would be hosted by the benefi ciary while MEval-PIMA provided offi ce equipment and furniture for the co-located staff member. Th is arrangement has shortened response time and MEval-PIMA has a better understanding of the implementation dynamics at the frontline. A review of the arrangement demonstrated that it works well when a participatory approach is used by the co￾located staff and there is strong support from the immediate supervisor. In cases where the staff member had previous local experience, the networks and local knowledge they had acquired were useful in accelerating delivery. 6 Th e bundle consisted of OD; DDU; collaboration and partnership; capacity building; sustainability; and gender mainstreaming. 7 1=poor and 5=excellent. Staff members felt that the relationship with supervisors needed the moderating infl uence of local human resources management. Local statutory compliance, allocation of training opportunities, international travel, performance reviews, and grievance handling were some of the areas in which staff felt that the current arrangement of provision of human resources management skills from the offi ce based in the U.S. was a burden. Co-location is the most effective way to deliver support in the counties, but it depends on good management and support. —MEval-PIMA staff member 16 MEASURE Evaluation PIMA: Report on the Midterm Review Job descriptions A review of job descriptions showed inconsistent formats. Th ere was variation in reporting lines; key roles and responsibilities; education, skills, and experience required; internal and external liaison; and job grade. In some instances, the duties and responsibilities currently carried out by staff members were not congruent with those listed in the job descriptions. However, in some cases, job descriptions, roles, and responsibilities do shift in response to project demands. Roles in staffi ng positions Th e deputy chief of party (DCOP) position is expected to support the COP in leading and managing the project. However, the DCOP does not supervise technical staff . Th e DCOP currently oversees the M&E staff in the Nairobi offi ce, the communication and documentation specialist, and the child protection and OVC M&E system team. Th e oversight responsibilities of two of the technical associate directors (national and subnational) are overwhelming, which limits provision of eff ective supervision. Some inconsistencies were noted in actual reporting arrangements and the project’s organogram: e.g., some staff have both a supervisor and a technical/project manager to work with, which was not part of the original plan. Monitoring, evaluation, and learning MEval-PIMA did not seem to take advantage of periodic staff meetings to collectively examine performance and deliverables achieved against the annual work plan and operational plan. Operation of consortium Th e project took into account the comparative advantage of each consortium member:  Palladium: Data demand and use  Tulane University: Engage with institutions of higher learning in building local monitoring and evaluation capacity  Management Sciences for Health: Address organizational issues  ICF: With strong local presence, take lead in implementing the project  University of North Carolina (UNC): Address special studies and project management However, adjusting from the delivery approach under MEASURE Evaluation III to the arrangement under MEval-PIMA required more coherence among partners during implementation. Because UNC (the prime) does not have local presence, ICF leads project implementation, requiring careful navigation and handling of relationships. Th e partners felt that the depth and frequency of communication among them needs more eff ort and diligence. Some partners felt that poor communication and coordination hampered work planning and budgeting processes. Th e partners pointed out that budget reviews carried out during Year 1 of project implementation lacked strong coordination and communication. Financial performance Resource allocation Over the implementation period, 59 percent of resources were allocated for expenditure on activities related to IR1: building MOH capacity. Th e second largest allocation was for IR2, at 32 percent, followed by baseline assessments, at 8 percent. Th e baseline assessments were allocated MEASURE Evaluation PIMA: Report on the Midterm Review 17 resources only for the fi rst two years. IR3 was allocated only 1 percent the resources budgeted during Year 1, Year 2, and Year 3. Appendix 5 displays the most recent project’s resource allocations. Program expenditure Th e project spent less than expected over the fi rst half the implementation period; during Year 1, only 37 percent of the annual budget was spent. A number of factors account for this: among them, the recruitment of staff undertaken that year. In Year 2, implementation expenditures increased to 71 percent of the annual budget. Appendix 5 presents the budget expenditures. Expenditures were aff ected by a delay in the disbursement of funds from USAID in the early part of the year and some changes to the budget. Th e changes arose from a number of meetings and from USAID’s feedback regarding targeting of the proposed expenditure. In Year 3, expenditure at the time of the review had already risen to nearly half (44%) the annual budget and was on a trajectory that would lead to improved budget utilization in Year 4. Comparing expenditure to obligated amounts elucidates the project’s fi nancial performance. Th e total budget for the three years was $22,937,634. Th e total obligated amount from the beginning of the project up to the second quarter of Year 3 was $16,279,225. Th us, a total of 71 percent of the budget was obligated over the period under review. During that period, the project has spent 72 percent of the obligated amount. Appendix 5 presents project use rates versus obligated amounts. An analysis of the expenditure over the period under review shows that 92 percent of consolidated expenditure over the implementation period was passed on to the project partners by UNC. 3 Conclusions A review of the internal environment and the external operating environment highlighted the following key project strengths, weaknesses, opportunities, and threats: Strengths  Well-educated and experienced staff in Kenya  Respected technical expertise pre-dating the MEval-PIMA project  Strong working relationship with the MOH, especially at the national level  Ability to draw on a pool of highly experienced technical experts based in the U.S. Weaknesses  Internal processes are exposed to undue delay as staff at the offi ce in Kenya consult and seek the support of respective administrative staff based in the U.S.  Organizational structure has large span of control for associate directors with instances of slow turnaround times regarding issues raised by staff in lower levels and in regions Opportunities  Devolution has created an opportunity to highlight the need for evidence-based decision making in the health sector  Working closely with counties provides designated geographical areas in which lasting change can occur to improve monitoring and evaluation capacity Threats  Funding and operational challenges in the health sector in Kenya, especially at the county level, are likely to work against the prioritization of M&E at key administrative and political decision-making levels in the counties 18 MEASURE Evaluation PIMA: Report on the Midterm Review More broadly, from a donor and sector-wide perspective, obtaining successful and sustainable systemic results envisioned under the project may be a challenge in the time allotted, given the numerous changes that have occurred. It is prudent to begin to consider a follow-on project to build on success of the current MEval-PIMA project and possibly scale up results. Th e following sections propose answers to the questions that the MTR raised. Theory of Change The main change in the TOC was the defunding of IR3. Th e implications in this shift are that the project will no longer be able to help strengthen local capacity for developing health professionals with M&E skills in the medium to long term. Cross-Cutting/Strategic Approaches As an approach, organizational development had low visibility among KII respondents, though many cited its component interventions contributing to institutionalization. Respondents said that MEval￾PIMA’s approach of undertaking the organizational capacity assessments done at national level with staff in the target programs had built their capacity to undertake such assessments in future. Th e project was also reported to have approached development of health strategic and investment plans, annual work plans, and M&E capacity-building plans in this participatory manner. Inclusion of M&E in the strategic and investment plans and in each annual work plan was cited as an approach that will ensure that M&E is integrated in the organizations’ planning and budgeting processes and thus facilitate adoption of evidence-informed decision making. However, suggestions by several respondents that MEval-PIMA expand its focus to include “the real decision makers” implies that a renewed emphasis is required to work with senior leadership forums to enlist full support for M&E capacity building among the targeted health institutions and programs. KII respondents cited MEval-PIMA’s participatory methods such as conducting joint capacity assessments or evaluations and publishing information products as eff ective approaches to capacity building. At the NMCP, for instance, after working on the fi rst Malaria Bulletin with MEval-PIMA, staff felt confi dent to produce subsequent bulletins and are now doing so with minor editorial support from MEval-PIMA. MEval-PIMA is cited by almost all respondents as the “go-to” partner for capacity building in M&E. Individual capacity building in M&E may also be achieved, by enrolling in M&E courses at local training institutions. It would be worthwhile for the project to consider supporting people to take short courses in M&E to strengthen their M&E capacity. Communication and advocacy can be strengthened to enhance understanding of MEval-PIMA by benefi ciaries and other M&E stakeholders. Further, communication can also be used to lead advocacy for M&E capacity building from a rights-based perspective, targeting especially the senior health￾sector leaders who are key data users and are not interacting with the project closely enough. Given the new constitution’s affi rmation of citizens’ right to health, communication can be used to enhance institutionalization of M&E through advocacy with the citizens (rights holders), and decision makers Decisions are not made at technical level. Having products at this level will not help. It is important to have products that . . . capture the interest of the decision makers. —MOH staff MEASURE Evaluation PIMA: Report on the Midterm Review 19 (duty bearers), highlighting the central role of an eff ective M&E system in improving the quality of health services and achieving good health outcomes. Several recommended improving internal project M&E for the promotion and support of better data demand and use: harmonizing the frequency of the OCI; clarifying the operational defi nition of the indicators “Instances of data use” and “Number of information products”; considering revising “Th e number of national programs implementing M&E capacity building plans”8 to (1) “Diversity of M&E capacity building actions by the supported program,” (2) Number of capacity building benefi ciaries,” or (3) “Number of individuals trained.” Th e existing indica￾tor is desirable but there are no target numbers for each year. Maintaining gender as a cross-cutting issue will ensure that the project identifi es gender disparities in health access, programming, and outcomes, and takes action to address these disparities. In addition, MEval￾PIMA could advocate adjusting the DHIS 2 to ensure that gender disaggregation of collected data is maintained. Sector-wide support Sector-wide support is progressing well in guiding M&E capacity strengthening across the country. Stakeholders feel that more eff ort should be directed to establishing and strengthening M&E units at the county level. Th is includes collaborating with county-level partners and establishing an eff ective M&E infrastructure to facilitate the fl ow of quality data from the community to the national level. In addition, strong engagement with senior leaders in the health sector at national and county levels enhances institutionalization of M&E and adoption of evidence-informed decision making and planning. Support of the NMCP MEval-PIMA’s support of NMCP is progressing well. Production of information materials, including the malaria bulletins, is now led by NMCP staff members. Capacity building in surveillance is using a training of trainers (TOT) approach, building capacity at the county level. However, post-training follow-up and mentorship must be strengthened at the county level to ensure data quality. 8 Th is indicator may not give the best information, as it may remain static once programs begin to implement their M&E capacity-building plans regularly. Effective engagement of counties is most likely to produce sustainable results. For example, Bungoma and Homabay counties are already buying SP [sulfadoxine/pyrimethamine] for malaria in pregnancy. Siaya will be paying community health workers to strengthen community strategy. —USAID There is that issue of achieving by all means. Sometimes [it] can be detrimental. When you do everything to achieve by all means, irrespective of the human relationships with our partners . . . you need to . . . see . . . value in getting everyone on board . . . rather than getting to a target at all costs. —United Nations agency 20 MEASURE Evaluation PIMA: Report on the Midterm Review Support of the RMHSU MEval-PIMA’s support of the RMHSU is progressing well despite frequent staff changes at the unit. Partners and stakeholders are satisfi ed with the results achieved thus far. However, it is apparent that participation by RMHSU staff in project activities could be enhanced to strengthen their ownership of the project’s deliverables. As some respondents mentioned, the project may need to revisit the strict timelines for achieving results and work more closely with RMHSU staff and stakeholders to create feasible timelines, in order to preserve good working relationships. MEval-PIMA structural and management approaches MEval-PIMA’s approach to bundling strategic approaches based on contextual needs underlined an important conceptual position that capacity building is not an end in itself but, rather, a signifi cant part of strengthening health systems in M&E for improved health outcomes. Important elements in staffi ng capability need to be addressed: human resources, gender mainstreaming, and advocacy. Th e DCOP position is not as operational as intended. Th is position should be reviewed in terms of the value it adds to the project. MEval-PIMA did not seem to take advantage of periodic staff meetings to collectively examine performance and deliverables achieved against the annual work plan and operational plan. Such collective review would provide additional impetus for implementation and would allow valuable lessons to be collectively extracted and used to improve implementation. It would also be an opportunity to explore and address challenges in the fi eld. Regular communication and consultation among staff and consortium members during the budget-making process are essential for eff ective implementation of the work plan. Focus is needed on strengthening accountability, enhancing team work, and achieving the project’s deliverables effi ciently. 4 Recommendations Th e project’s implementation is aligned with its overall goals but hampered by deviation from the theory of change at inception. Th ere is potential to improve project delivery over the next two and a half years, by addressing management and structural challenges. Sector-wide Support Given the facilitators and challenges noted, the following are key recommendations for the MEval￾PIMA project: 1. Consider orienting senior administrators/political leaders in M&E so they can understand clearly the wider scope of this function and support M&E strengthening and use of data for decision making 2. Support advocacy of M&E fi nancing, especially by county governments 3. Document several counties with eff ective M&E system strengthening models and share their stories with relevant decision makers and stakeholders 4. Advocate that the MOH at national and county levels sponsor their staff to take M&E courses in local institutions, such as African Medical Research Foundation, Great Lakes University, University of Nairobi, and Daystar University 5. Advocate that the MOH include M&E as a key course in the Kenya Medical Training College MEASURE Evaluation PIMA: Report on the Midterm Review 21 Support of the NMCP Although MEval-PIMA remains focused on strengthening M&E capacity through delivery of technical strategies, to enhance our achievements and visibility, we need to consider engaging more with county governments and liaising with other stakeholders to address some key challenges that impede M&E function. Additional recommendations are: 1. Advocating that county governments and other stakeholders support improved assurance of supply of data collection tools and harmonized data collection tools, data summary tools, and the DHIS 2 2. Supporting follow-on training to ensure staff trained are appropriately putting their new skills in practice 3. Working with service provision partners to institute data quality assurance processes at the counties 4. Identifying local M&E partners to build their capacity in order to sustain M&E capacity building beyond the life of the project Specifi c support of CHMTs Counties need support in promoting health sector guidelines; conducting supportive supervision to ensure that high-quality health services are being provided; and ensuring that the planning cycles in the counties are aligned with the national planning and budget cycle. Although the following suggestions may not fall immediately within the mandate of MEval-PIMA, the project can liaise with service delivery partners to carry them out: 1. Build health managers’ skills in using evidence to engage county executive members, etc. 2. Work with governors to advocate adequate resourcing of the health services 3. Orient health workers on national guidelines rather than the current system of simply distributing the guidelines at the county, subcounty, and health facility levels with limited guidance 4. Establish a credible system for monitoring supplies of health commodities at the county level Support of the RMHSU Stronger participation by the RMHSU and counties in developing and rolling out activities is crucial for benefi ciary ownership of the approaches, tools, and products developed. A shared understanding of the scope of the project among benefi ciaries, partners, and key stakeholders is critical for smooth implementation. Additional recommendations are: 1. Focus on strengthening partnerships with USAID-funded service provider partners working at county level to undertake M&E capacity strengthening interventions at the county, subcounty, and facility levels 2. Expand engagement beyond technical staff to reach county decision makers (governors and leaders of county assembly committees on health and fi nance) to strengthen the executive and political support required to enhance and sustain county-level M&E capacity 3. Identify and focus on specifi c counties for M&E capacity building in a comprehensive manner to demonstrate sustained effectiveness in the delivery of expected outcomes 22 MEASURE Evaluation PIMA: Report on the Midterm Review MEval-PIMA Program Management There is a need to strengthen structure and improve management for optimal delivery over the second half of the project. Th e following are key recommendations based on the MTR’s fi ndings, to improve overall staffi ng and management of the project. Policies and procedures Share approved activities and budgets with staff at the beginning of the fi nancial year to ease implementation and support compliance with internal procedures and policies. Provide readily available copies of policies and procedures for ease of reference by staff members. Support for information, communication, and technology (ICT) Revisit ICT’s support of the Nairobi offi ce to reduce response/completion time. Consider a service delivery agreement to reduce help-desk turnaround time to one hour and procurement lead time to two weeks. Job descriptions Standardize format, taking into account changes that have occurred in each position’s duties and responsibilities, and ensure that a job description is in the fi le of every staff member. Skills gaps Build capacity among staff to address the project’s gender and advocacy needs. Staffi ng Revisit the position of the DCOP in the project structure and consider shifting these duties to the associate directors. Consider developing the position “associate director of advocacy, communication, and knowledge management,” shifting reporting to ensure that this new position has fi ve staff positions to oversee. Th ese positions would be M&E; communication and documentation; child protection and the OVC M&E system; DDU; and OD. Recruitment Recruit someone to the position of county M&E capacity building, who is directly responsible for working with the CHMTs. Supervision of staff in the counties Consider clustering the staff based in counties in fi ve-member teams, should the counties in which MEval-PIMA operates be eight or more. Each of the teams would have one of the county-based staff appointed as the cluster team leader. Th is person would help coordinate and administer the team and report to the associate director for capacity building, subnational programs. MEASURE Evaluation PIMA: Report on the Midterm Review 23 Accountability lines and teamwork Consider enhancing teamwork and effi cient delivery by changing accountability lines to require that all project staff have the following at the Nairobi offi ce level: setting of performance goals; daily supervision of work; and performance evaluation. Unsatisfactory performance needs can be addressed with the management staff at the Nairobi offi ce through the COP, who will work with the consortium partner and the staff member concerned to develop and implement a performance improvement plan. When severe disciplinary or performance issues arise, the project should ask the consortium partner for change of staff or recall of staff . Budgeting Staff members, including those in the region, should have a clear picture of the approved activities and annual budget allocation. During the budgeting process, it is necessary to emphasize to all parties that identifying, quantifying, and budgeting for project activities should precede determination of related labor and travel costs. Where budget changes are necessitated by signifi cant changes in the operating environment, using participatory approaches to decide those changes will make them easier to carry out. Technical backstopping Project leadership should discuss with USAID the added value of technical assistance: when and how often it will be needed during the remainder of the project. It is crucial to ensure that the experience and expertise of the U.S.-based staff members are used appropriately to deliver results and to transfer skills to local staff . Communication The project should continue quarterly meetings to share progress and build the capacity of staff members, using these meetings to collectively extract learning from ongoing implementation and the contributions of fi eld staff . Th ese meetings can also be used to resolve any management issues about which staff may have questions and to give the staff regular updates on the project’s status. Th e circulation of an internal newsletter could share key monthly developments and announce upcoming events and milestones. Critical developments still should be circulated to staff by email from the COP. Consortium Quarterly partner calls should be held consistently. Circulating the agenda in advance for comment would allow issues—especially those regarding relationships among the partners—to be raised and addressed. 24 MEASURE Evaluation PIMA: Report on the Midterm Review 5 Appendices Appendix 1: Action Plans from Field Visits Referral systems strengthening (RSS) action plan notes The following gaps were identifi ed during the USG fi eld visits: 1. Lack of standardization in facility-level referral processes and referral documentation 2. Weak referral monitoring systems at facility level 3. Weak facility-level collaboration and coordination with other implementing partners 4. Failure of the RSS training to reach most of the health workers who had not been trained previously Th e project proposes to use the following strategies to address the gaps and strengthen HIV referral systems at the facility level: 1. Standardize HIV referral processes and documentation, by developing HIV client referral standard operating procedures (SOPs). Th is will involve a review of existing processes at the facility level to choose the best alternatives. NASCOP and HIV stakeholders, such as the facility-level implementing partners, should participate in this, and also in validating the SOPs. Currently, referral processes, referral documentation, and referral tracking vary by facility, depending on the implementing partner supporting the facility. For example, some facilities use partner-developed client referral forms to communicate referrals; in other facilities, referrals are communicated verbally or in writing on a blank piece of paper or a community referral form. Documentation of referrals also varies by facility, as does the linkage to care processes and the tracking of referral completion. Th e SOPs will standardize the referral processes and guide health workers in the referral of HIV clients to care. 2. Support performance monitoring of HIV referral systems and promote referral data use for decision making at the facility and county levels. Th is will be done through continuous appraisal of the facility-level referral processes; review of HIV referral data; promotion of referral data use through facility- and county-level data use forums; as well as capacity building of the CMHTs, the staff of USG implementing partners (IPs), and health workers to monitor the HIV referral systems’ performance. 3. Promote stakeholder engagement and collaboration in the implementation of RSS activities. Facility-level IP staff will be involved in the facility-level RSS activities, including HIV data review and data use, trainings and mentorship, and referral forums. At the county level, the project will support biannual referral forums to discuss county HIV referral data, collaboration, coordination of referrals, and referral challenges. Participants in these county-level forums will be MOH, USG IPs involved in HIV service delivery, and any other stakeholders involved in HIV service delivery. 4. Develop and disseminate HIV services directories. Th e project will support the development of HIV services directories for use at the facility level. Th e directories will provide an inventory of the HIV services available in the county, to facilitate HIV referrals and allow health workers to track referral completion. MEASURE Evaluation PIMA: Report on the Midterm Review 25 5. Build capacity of health workers to strengthen HIV referral systems and monitor their performance. Th e project will develop an abridged version of the RSS curriculum for use, together with the HIV module, to train county-level RSS resource persons. Th e resource person will include the county AIDS and STI coordinators (CASCOs) and staff from the USG IPs involved in HIV service delivery at the county level. Th e resource persons will deliver trainings in MEval-PIMA-supported facility-level continuous medical education (CME) and off er facility￾level on-the-job training and mentorship to health workers involved in HIV service delivery. 6. In addition, MEval-PIMA will support and mentor health workers and the CHMT members to monitor the performance of HIV referral systems and data use, by passing on their skills in HIV referral data collection and the analysis and interpretation of HIV referral indicators. Below is a summary of the proposed activities: 1. Develop (standardize/harmonize available) HIV client referral SOPs. 2. Conduct quarterly review of the facility-level referral processes, referral documentation, and referral completion and linkage to care in the MEval-PIMA RSS sentinel facilities, in collaboration with the facility-level IPs. 3. Develop facility-level referral profi les to inform facility-level performance of the HIV referral systems. 4. Conduct facility-level forums to discuss HIV referral processes, referral documentation, and referral data; develop remedial action plans; monitor implementation of the action plans; and build health workers’ capacity to monitor the referral system. 5. Develop referral directories of HIV services, to facilitate referrals and the tracking of referral completion for inter-facility referrals. 6. Conduct county-level referral forums involving MOH and county-level IPs, in order to review HIV referral indicators, referral processes, and linkage to care, as well as to discuss challenges to collaboration and develop corrective actions. 7. Identify and participate in existing forums to discuss inter-county referrals, such as the Nyanza M&E TWG. 8. Develop an abridged version of the RSS curriculum for trained resource persons to disseminate at CME trainings and other forums. 9. Train resource persons in each county, using an abridged version of the RSS curriculum and the HIV module on HIV referral systems strengthening and performance monitoring. 26 MEASURE Evaluation PIMA: Report on the Midterm Review RSS action plan details Output Area Issue/Gap identifi ed Proposed Solution Specifi c Activities Frequency RSS 1. Lack of standardization in facility-level referral processes and referral documentation (It was observed that referral processes and referral documentation for HIV client referrals had not been standardized across facilities. The referral processes and documentation varied, depending on the IP supporting the facility. The role for PIMA is to take the lead in the standardization processes with the involvement of the IPs, county health departments, and NASCOP.) 1. Development of referral standards including SOPs with the involvement of NASCOP, county health departments, and other implementing partners. 1. Engage NASCOP on the development of HIV clients referral SOPs to ensure buy-in and sustainability Continuous 2. Develop HIV-positive clients referral SOPs, to standardize the process of HIV-positive clients’ referral into care Once 3. Print seed copies of the referral SOPs for the RSS sentinel sites Once 4. Advocate printing of the SOPs and sensitizing health workers to them through NASCOP and other USG implementing partners Continuous 2. Standardization of existing referral tools through a review of the several tools in use and development of a referral/ linkages register and forms that meet MOH and partners’ needs Already done by NASCOP in the revision of registers; there is now a stand-alone linkage/referral register for HIV clients 2. Weak referral monitoring systems at facility level 1. Work with facility-level implementing partners and MOH to avail RMS inputs such as referral forms and registers 1. Advocate printing and dissemination of the NASCOP referral/linkage registers and referral forms Continuous MEASURE Evaluation PIMA: Report on the Midterm Review 27 Output Area Issue/Gap identifi ed Proposed Solution Specifi c Activities Frequency 2. Promote facility-level referral and HIV linkages data use for decision making. 1. Quarterly review of the facility-level referral processes, referral documentation, and referral completion and linkage into care in the PIMA RSS sentinel facilities in collaboration with the facility-level implementing partners Quarterly 2. Development of facility￾level referral profi les to inform facility-level performance of the HIV referral systems. Quarterly 3. Conduct facility￾level forums to discuss HIV referral processes, referral documentation, and data; develop remedial action plans; monitor implementation of the action plans; and build health workers’ capacity on referral system monitoring Quarterly 3. Develop HIV services referral directories to facilitate tracking of referrals for inter-facility referrals. 1. Draft county-specifi c HIV referral services directories Once 2. Disseminate the referral directories Once 3. Update during the biannual county-level referral forums Biannual 28 MEASURE Evaluation PIMA: Report on the Midterm Review Output Area Issue/Gap identifi ed Proposed Solution Specifi c Activities Frequency 3.   Weak facility-level collaboration and coordination with other implementing partners 1. Facility-level stakeholder mapping and engagement 1. Engage facility-level HIV implementing partners through involvement in the mentorship forums and HIV data reviews; share RSS action plans, facility-level referral forums and CMEs Continuous 2. Referral stakeholder forums at the county level led by MOH to discuss referral challenges and HIV linkages data and develop RSS action plans 1. Conduct county-level referral forums involving MOH and county-level IPs to review HIV referral indicators, referral processes, and linkage to care, as well as to discuss challenges and collaboration and develop corrective actions Biannual 3. Need for inter-county referral forums to discuss inter-county referrals 1. Identify and participate in existing forums to discuss inter-county referrals, such as the Nyanza M&E TWG Based on the TWGs' calendars (quarterly) 4.   The RSS training had not reached most of the health workers who had not been trained 1. Use the TOTs and the health workers who were trained as resource persons to cascade the trainings and develop facility-level “training cascade plans,” to ensure that the training reaches a critical mass 1. Develop training dissemination plans to the nontrained staff with the CHMT and the TOTs at county level Once 2. Develop abridged training material for use by the trained facility staff in disseminating the trainings in various forums Once 3. Train resource persons in each county, using the abridged version of the RSS curriculum and the HIV module Once 4. In the RSS sentinel sites, support facility-level CMEs, OJTs, webinars, online courses, etc., to improve training coverage and sensitize health workers on HIV referral system strengthening and performance monitoring Quarterly 29 Civil registration and vital statistics (CRVS) action plan to address gaps identifi ed during the USG site visits Output Area Issue/Gap Identifi ed Proposed Solution Indicative Activities Timelines Strengthen capacity of the MOH staff at the subnational level to demand and use information for decision making 1. There was no evidence of linkage and synergy between the M&E TWGs at the county and national levels 1. Foster cross learning among counties and between the county- and national-level M&E TWG 1. Support joint participation in M&E TWG meetings at national- and county-level meetings 2. Support facility in-charges to attend county-level quarterly M&E TWG meetings 3. Support quarterly inter-county performance and learning forums to share best practices in monitoring and evaluating program performance Quarterly 2. Build capacity for M&E TWG coordination 1. Work with the s/CHMTs to establish a forum that brings together stakeholders at the subcounty and facility levels to address program performance gaps identifi ed during quarterly performance reviews Once 2. There are no strategies to address gaps identifi ed during data quality and performance review meetings 1. Develop a data quality and program improvement plan to support the CHMTs in addressing data quality and program implementation gaps identifi ed during relevant review meetings 1. Work with stakeholders to develop performance improvement plans following data and/or performance reviews 2. Support follow-up meetings with stakeholders 3. Provide TA to CHMTs during supportive supervision 4. Provide TA to standardize data and performance review meetings and support the selection of indicators Quarterly 30 Output Area Issue/Gap Identifi ed Proposed Solution Indicative Activities Timelines 3. Stakeholder engagement at the subcount y and facility levels was found to be weak 1. Establish forums that bring stakeholders at the subcounty and facility level together 1. Develop comprehensive subcounty stakeholder inventories and share with respective service delivery partners (USG and non-USG) 2. Support quarterly health management team meetings to review achievement of service delivery targets 3. Support monthly integrated performance reviews at high￾volume facilities involving key stakeholders 4. There were no strategies to support implementation of skills gained through training at various levels 1. Support the CHMT to monitor skills transfer among trainees 1. Develop indices to measure effectiveness of learning 2. Establish county learning networks and a critical mass of master trainers to provide continuous support for skills transfer 3. Support and involve training alumni during inter-county capacity-building activities 4. Develop a sustainability plan for county/subcounty capacity building efforts Quarterly 5. There was limited evidence of cross￾county learning and skills transfer 1. Establish forums for cross-county learning and exchange 1. Support regional (county-level) annual M&E best practices workshop 2. Support annual inter-county learning and exchange forums, led by training alumni on specifi c areas (RSS, ICD10 coding, data management, and performance reviews) Quarterly/ annually 2. Support efforts to harmonize and standardize tools and approaches to capacity building: strategic plans (SPs), annual work plans (AWPs), and M&E plans 2. Work with partners to disseminate tools and templates for performance reviews, AWPs, and data quality reviews Once 31 Output Area Issue/Gap Identifi ed Proposed Solution Indicative Activities Timelines 6. There was limited evidence of cross￾county learning and skills transfer 1. Develop information products for subcounty and facility levels 1. Work with strategic partners to develop quarterly performance dashboards at the subcounty level and disseminate during quarterly health management meetings to review progress in program outcomes/service delivery 2. Work with strategic partners to develop quarterly performance dashboards at select health facilities Quarterly 2. Strengthen performance review processes at subcounty level 1. Provide TA for data review meetings to monitor program performance/services delivery and develop action plans for program improvement based on fi ndings 3. Develop a process for sharing and disseminating information products 1. Provide TA to review data for program management and improvement at the subcounty level 2. Support CHMTs and subcounty management teams to conduct quarterly integrated supportive supervision 7. There was no evidence of data use at the subcounty and facility levels 1. Develop information products for subcounty and facility levels 1. Work with strategic partners to develop quarterly performance dashboards at the subcounty level and disseminate at quarterly health management meetings to review progress in program outcomes/service delivery 2. Work with strategic partners to develop quarterly performance dashboards at select health facilities Quarterly 2. Strengthen performance review processes at subcounty level 1. Provide TA for data review meetings to monitor program performance/services delivery and develop action plans for program improvement based on fi ndings 3. Develop a process for sharing and disseminating information products 1. Provide TA to review data for program management and improvement at the subcounty level 2. Support CHMTs and subcounty management teams to conduct quarterly integrated supportive supervision 32 MEASURE Evaluation PIMA: Report on the Midterm Review Appendix 2: Deliverables Status for Sector-wide Approaches Year Sector-wide Deliverables Planned Planned Timeline Status at Q1 Q2 Q3 Q4 Year-end 1 Develop a concept note on the institutionaliza￾tion of capacity building and organizational development for M&E units Partially achieved 2 Stakeholder engagement plans for sector-wide stakeholders Not achieved; unclear if canceled A national sector-wide M&E stakeholder group (ICC or TWG) supported Achieved Annual Health Sector Performance Review Report Partially achieved; had no timelines MOH staff trained in core M&E and health sys￾tems competencies Delayed, then stopped altogether due to bud￾get cut Requisite support provided in preparations for the Annual Health Congress Achieved 3 M&E capacity assessment for MOH M&E Unit Partially achieved A tool for tracking fl agship projects Postponed to Q3 Stakeholder maps for health M&E stakeholders in place Partially achieved M&E guidelines and standards Ongoing MEASURE Evaluation PIMA: Report on the Midterm Review 33 Appendix 3: Deliverables Status for Support to NMCP NMCP Deliverables Planned, Year 1 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end Stakeholder engagement report Ongoing M&E capacity assessment report and presentation Achieved Individual-level M&E capacity building plans Rolled over to Year 2 Identifi cation and training of a malaria GIS champion Canceled; had no timelines Organizational M&E capacity building plan Rolled over to Year 2 Document M&E monitoring system (the project MIS at MEval-PIMA) Achieved Data use strategy for Malaria Information Acquisition System (MIAS) Rolled over to year 2 A revised Kenya malaria M&E plan 2013–2017 Ongoing Two malaria surveillance bulletins in 2013 Achieved A concept note and protocol on process and contextual evaluation of the long-lasting insecticide-treated nets (LLIN) campaign Canceled A concept note on the process and contextual evaluation of the rollout of the malaria supportive supervision tool Canceled NMCP Deliverables Planned, Year 2 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end NMCP M&E CB action plan Achieved Review of Kenya National Malaria Strategic Plan 2013– 2017 Ongoing Review of Malaria M&E Plan 2013–2017, which aligns with newly devolved governance systems Ongoing Malaria Control Unit (MCU) DDU plan, including a strate￾gy to revitalize MIAS Achieved Information products that MCU can use to engage exter￾nal stakeholders for advocacy and accountability; specifi c products are: 1. Malaria surveillance bulletins 2. Annual Malaria Report 2012–2013 Achieved 34 MEASURE Evaluation PIMA: Report on the Midterm Review NMCP Deliverables Planned, Year 2 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end Malaria surveillance curriculum package: 1. A curriculum and implementation guide 2. Trainers manual 3. Participants’ manual with PowerPoint slides 4. Training schedule and rollout plan Achieved Mapping of epidemic preparedness and response (EPR) sentinel sites in six counties Canceled EPR data review meetings for six counties Canceled Data‐driven EPR plans at national level and selected coun￾ties Canceled 2nd Kenya National Malaria Forum, 2014 Achieved NMCP Deliverables Planned, Year 3 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end Malaria Surveillance Training Report Partially achieved Q2, completed in Q3 Updated Malaria Surveillance Curriculum package Scheduled for Q4 Data quality improvement activity reports Ongoing Kenya National Malaria Forum Report and Abstract Achieved Malaria surveillance bulletins; Annual Malaria Report 2013–2014; county fact sheets for Migori, Kisumu, and Kakamega Partially achieved (template done) Kenya Malaria Indicator Survey protocol Partially achieved (submitted for ethical approval) Scope of work for impact evaluation; subcontract with local fi rm or consultants for PMI impact evaluation Partially achieved MIAS implementation strategy Achieved Malaria M&E and operations research (OR) TWG minutes Achieved Revised Malaria M&E plan 2009–2017 Achieved Revised national documents with focus on gender Achieved MEASURE Evaluation PIMA: Report on the Midterm Review 35 Appendix 4: Deliverables Status for Support to RMHSU RMHSU Deliverables Planned, Year 1 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end 1. Stakeholder engagement report produced at the M&E TWG Partially achieved 2. M&E capacity-assessment report and presentation made to RMHSU program managers and national RH policymakers Achieved 3. Organizational M&E capacity-building plan developed for RMHSU Achieved 4. Capacity building in report writing for MOH, including NMCP, community health information system, and other divisions Ongoing 5. M&E capacity-building plans for individual RMHSU program staff, developed as appropriate Ongoing 6. Dissemination of RMHSU 2011–2012 annual report Achieved 7. Strengthened advocacy for use of RH data Achieved 8. Documentation of CB for M&E monitoring system, to track progress in CB for the organizational and individual CB plans Achieved RMHSU Deliverables Planned, Year 2 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end 1. RH annual report Achieved 2. Information needs analysis report and a revised M&E plan Partially achieved 3. Data use plan for RMHSU Achieved 4. Report on the audit of Maternal Death Surveillance and Response, with recommendations to strengthen MPSDR Partially achieved 5. Regular and functional national M&E TWG meetings as forums for stakeholder coordination Achieved 6. Contribution to provision of M&E information in FP and MNCH TWG Achieved 7. FP/RH staff trained and provided coaching and mentoring in M&E Achieved 36 MEASURE Evaluation PIMA: Report on the Midterm Review RMHSU Deliverables Planned, Year 2 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end 8. Final report for the baseline and evaluation of basic emergency obstetric and neonatal care (BEmONC) scale‐up Partially achieved 9. Final report and dissemination of the “Free Maternity Services Satisfaction Survey” Partially achieved 10. Final report on the WHO multi‐county MNH survey Achieved 11. Report on Kenya RH scorecard indicators Partially achieved 12. Program briefs Achieved RMHSU Deliverables Planned, Year 3 Planned Timeline Q1 Q2 Q3 Q4 Status at Year-end 1. RMHSU AWP 2015–2016 Ongoing 2. Training curriculum and training reports Partially achieved 3. RMHSU Annual Report 2013/2014 Achieved (but late in Q2) 4. RH information products, such as briefs and bulletins Achieved 5. Revised RH strategic plan Ongoing 6. Emergency obstetric and neonatal care (EmONC) assessment and evaluation tools and reports Partially achieved 7. RH tool and approaches; technical reports Partially achieved 8. RMNCH scorecard in selected counties, data review meetings, other data review and use tools and reports Partially achieved 9. Action plans for strengthening FP commodities reporting and forecasting Achieved (was not in work plan) 10. Assessment reports and updates for maternal and perinatal death surveillance and response and BeMONC Achieved (was not in work plan) 11. Poster/oral presentation at a maternal and child health and FP forum Not yet started 12. FP data review tools Partially achieved MEASURE Evaluation PIMA: Report on the Midterm Review 37 Appendix 5: Financial Information Project resource allocation, 2013–2015 Description Year 1a Year 2b Year 3b Total IR 1.0: Improved capacity of MOH health programs to identify and respond to M&E information needs at national and subna￾tional levels 1,763,407 4,367,547 6,324,759 12,455,713 IR 2.0 Improved availability and use of quality health information at national and subnational levels 5,067,951 - 1,039,341 2,647,405 6,676,015 IR 3.0 Improved capacity of local training and research institutions to meet human re￾sources needs of health M&E professionals 891,239 -578,466 – 312,773 Baseline assessments 925,868 731,020 – 1,656,888 Total 8,648,465 3,480,760 8,972,164 21,101,389 Notes: a) Figures for Year 1 are from work plan for year 2, the version dated March 2014; b) Figures for Year 2 and Year 3 are from the work plan for Year 3, the version dated November 2014. Project budget utilization Budget Expenditure (USD) Year 1 7,057,284 2,664,051 Year 2 7,676,491 5,457,408 Year 3 (to Q2) 8,203,859 3,606,262 Project utilization of obligated amount Obligated Expenditure Annual Cumulative USD USD utilization utilization Year 1 9,729,225 2,664,051 27% 27% Year 2 2,400,000 5,457,408 227% 67% Year 3 (to Q2) 4,150,000 3,606,262 87% 72% Total 16,279,225 11,727,721 72% TR-15-119 MEASURE Evaluation PIMA Carolina Population Center University of North Carolina at Chapel Hill 400 Meadowmont Village Circle, 3rd Floor Chapel Hill, North Carolina 27517 Phone: +1 919-445-9350 | Fax: +1 919-445-9353 Email: measure@unc.edu www.measureevaluation.org MEASURE Evaluation PIMA is funded by the U.S. Agency for International Development (USAID) through associate award 623-12-000002. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government.