AUGUST 27, 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared by of Brian Agbiriougu, Mary Wieczyniski Furnivall, Wilson Kisubi, Ben Okeyo, Ingrid Orvedal, John Paul Oyore, and Management Systems International. FINAL PERFORMANCE EVALUATION AIDS, POPULATION AND HEALTH INTEGRATED ASSISTANCE, PEOPLE￾CENTERED, LOCAL LEADERSHIP, UNIVERSAL ACCESS, AND SUSTAINABILITY (APHIAPLUS) NAIROBI /COAST PROJECT FINAL PERFORMANCE EVALUATION AIDS, POPULATION AND HEALTH INTEGRATED ASSISTANCE, PEOPLE-CENTERED, LOCAL LEADERSHIP, UNIVERSAL ACCESS, AND SUSTAINABILITY (APHIAPLUS) NAIROBI/COAST PROJECT Brian Agbiriougu, Mary Wieczyniski Furnivall, Wilson Kisubi, Ben Okeyo, Ingrid Orvedal, John Paul Oyore, and Management Systems International August 2013 Contracted under No. AID–623–I–12–00001 (Task Order No. AID–623–TO–13– 00018) Kenya Support Program DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms and Other Abbreviations............................................................................................................................. 1 Executive Summary ........................................................................................................................................................... 3 Evaluation Purpose and Questions................................................................................................................................ 8 Project Background........................................................................................................................................................... 9 Methods and Limitations................................................................................................................................................12 Findings, Conclusions, and Recommendations.........................................................................................................14 Lessons Learned...............................................................................................................................................................50 ANNEX A. Evaluation Statement of Work................................................................................................................. 3 ANNEX B. Methodology Update and Data Limitations.........................................................................................35 ANNEX C. Recommendations ....................................................................................................................................62 ANNEX D. Considerations for the Next Five-Year Health Assistance Program ...........................................65 ANNEX E. Commodities Supplied by APHIAplus in the 17 Sites Visited .........................................................69 ANNEX F. Support to Kenyatta National Hospital’s GBV Recovery Center..................................................71 ANNEX G.1. Program Management Plan Data Tables for Nairobi ....................................................................72 ANNEX G.2. Program Management Plan Data Tables for the Coast................................................................89 ANNEX H. Result 4 Indicator Discussion.............................................................................................................. 103 ANNEX I. Data Collection Tools............................................................................................................................. 104 ANNEX J. Statement of Differences........................................................................................................................239 1 ACRONYMS AND OTHER ABBREVIATIONS AED CAP Academy for Education Development Capable Partners Project AIDS Acquired Immune Deficiency Syndrome ANC antenatal care APHIA II AIDS Population and Health Integrated Assistance APHIAplus AIDS Population and Health Integrated Assistance—People-Centered, Leadership￾Focused, Universal Access, and Sustainability Project ART antiretroviral therapy ARV antiretroviral drug ASSIST Applying Science to Strengthen and Improve Systems CBO community-based organization CCC Comprehensive Care Unit CHW community health worker CIPK Council of Imams and Preachers of Kenya CLUSA Cooperative League of the United States of America CME Continuous Medical Education COP Country Operational Plan CQI Continuous Quality Improvement CU community unit DHIS District Health Information System DHMT District Health Management Team DISC drop-in service center EID Early Infant Diagnosis EMTCT Elimination of Mother to Child Transmission ETL education through listening FP family planning FST formal short-term training FSW female sex worker GBV gender-based violence GOK Government of Kenya HCM Health Communications and Marketing HCT HIV counseling and testing HFMT Health Facility Management Team HIS Health Information System HIV Human Immune Deficiency Virus HTC HIV Testing and Counseling IDU intravenous drug user IGA income-generating activity 2 IMCI Integrated Management of Childhood Illnesses IR Intermediate Result KCCB Kenya Conference of Catholic Bishops KEMSA Kenya Medical Supplies Agency KEPH Kenya Essential Packages of Health KGGA Kenya Girl Guides Association KII key -informant interview KQMH Kenya Quality Model for Health Ksh Kenya shillings M&E monitoring and evaluation MARP s most - a t - risk populations MNCH maternal , neonatal , and child health MOH Ministry of Health MSI Management Systems International MSM men who have sex with men MSW male sex worker NARESA Network of AIDS Researchers of Eastern and Southern Africa NASCOP National AIDs and STI Control Program NCCK National Council of Churches Kenya NGO nongovernmental organization OJT o n -the - job training OPH Office of Population and Health ORT Oral Rehydration Therapy OVC orphans and vulnerable children PEPFAR President’s Emergency Plan for AIDS Relief PLHIV p ersons living with HIV PMP Program Management Plan PMTCT Prevention of Mother -to -Child Transmission PSI Population Services International PWP Prevention With Positives PWUD persons who use drugs QA/QI quality assurance/ quality improvement RFA request for applications RH reproductive health SILC Savings and Internal Lending for Communities SOW scope of work STI sexually transmitted infection SUPKEM Supreme Council of Kenya Muslims TA technical assistance TB Tuberculosis URC United Research Corporation USAID United States Agency for International Development VCT voluntary counseling and testing VMMC voluntary male medical circumcision VSLA Voluntary Savings and Loan Association 3 EXECUTIVE SUMMARY This is an independent performance evaluation of the AIDS Population and Health Integrated Assistance— People-Centered, Leadership-Focused, Universal Access, and Sustainability Project (APHIAplus) Nairobi/Coast project, a three-year, $55 million award that began on January 1, 2011, and ends on December 31, 2013, implemented by Pathfinder International. The specific objectives of the evaluation are twofold: a) to conduct an in-depth qualitative assessment of technical and management approaches, coordination with host country and other stakeholders, and support to country health systems; and b) to conduct a review of project achievements based on assigned targets in HIV/AIDS, FP/RH, and Child Survival activities. Findings, conclusions, and recommendations will shape strategies, approaches, and activities for the follow-on program design. The primary audience for the evaluation is USAID/Kenya and the Government of Kenya. Secondly, the report is intended for Pathfinder, USAID/Washington, and other interested implementing partners. In doing so, the evaluation sought to answer the following questions: 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4?* Were the key expected outcomes achieved? If not, why not? 2. What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in Results 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 4. To what extent was the APHIAplus model appropriate for achieving key outcomes under Results 3 and 4 of the Implementation Framework? 5. What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? PROJECT BACKGROUND APHIAplus is one of five regional service delivery mechanisms under the APHIAplus Service Delivery awards. Within the USAID/Kenya Health Implementation Framework 2010–15, the APHIAplus design focused on Result 3, “increasing delivery of quality health services,” and Result 4, “addressing the underlying social determinants of health of this Implementation Framework.” APHIAplus Nairobi/Coast covers Zone 2, which is described as a moderate-need zone that covers urban areas. It includes the whole of the former Nairobi and Coast region (excluding Tana River District, which is covered in the Northern Arid Zone). APHIAplus Nairobi/Coast works with a consortium of partners, subawardees, and national technical assistance mechanisms. Working with the Government of Kenya and its partners, APHIAplus service-delivery projects support marginalized, vulnerable, and underserved populations: youth, most-at-risk populations (MARPs), persons living with HIV (PLHIV), those on antiretroviral drugs, orphans and vulnerable children (OVC) *Result 3, increased use of quality health services, products, and information; Result 4, social determinants of health addressed to improve the well-being of targeted communities and populations. 4 affected by AIDS, women of reproductive age (including pregnant and postpartum women), and highly vulnerable girls, neonates, and infants. The implementation strategy concentrates on achieving two levels of results: 1) at the facility and community levels working through national, provincial, and district GOK structures and 2) with communities through community-based organizations (CBOs). The project works through three packages of interventions (strategies) to achieve its goals and objectives—the Foundation, Most Vulnerable Groups, and Partnerships Packages. EVALUATION DESIGN, METHODS, AND LIMITATIONS The evaluation employed a mix of quantitative and qualitative methods to answer the evaluation questions. Data collection methods included desk review, data extraction, key-informant interviews, site visits, observational checklists, and facility- and community-level group discussion. The data analysis methods included descriptive statistics, data mapping, pattern/content analysis, response convergence/divergence analysis, and triangulation. The team experienced numerous limitations in this assessment, and these limitations made a significant impact on the formation of valid and evidence-based conclusions. There were three major limitations: • The Timing of the Evaluation. This evaluation occurred two and a half years into a three-year project (reduced from five years); data for the evaluation were collected when most activities had only a year or less of data to report, limiting results found mostly to outputs and preliminary outcomes; higher-level outcome and any impact-level (including sustainability) results and trends/progress were almost impossible to assess. • The Project Design and Documentation. The APHIAplus project has undergone significant changes in activities and PMPs, and the structure of work plans. This created two major limitations: 1) indicators and their targets were inconsistent, making tracking of progress and achievements difficult to assess, and 2) it was difficult to deduce linkages connecting results, outcomes, activities, and targets within results found. Leading to difficulty in assessing effectiveness of implementation strategies. • The Primary Data Sampling Frame. The facilities visited were selected by the Mission and were a convenience sample tiered by demand (high/medium/low), skewing sampling toward mid-demand facilitates, as opposed to a stratified random sample to produce statistically valid and representative results. In addition, APHIAplus’s scope includes a wide range of activities. As such, it was difficult to assess all components against project documentation. Thus, the primary data results in this report provide only an indicative snapshot of project performance and cannot be generalized. MAJOR CONCLUSIONS Question 1. Achievement of Results and Outcomes The complicated structure and multiple formats of project documents, make it difficult to fully understand the project’s logic in terms of indicators and how the project intended to demonstrate results. Without a clear causal logic, it is difficult to determine APHIAplus’s contribution to key indicators, outcomes, and results. Overall lack of fidelity between project documents makes progress against work plans and achievements difficult to track. In year 3, the project developed a new Monitoring and Evaluation Plan, which added an exhaustive results chain and logic model that captures outcome indicators. Yet the project has not reported on these new indicators nor incorporated them into the Performance Monitoring Plan in the first quarterly report for year 3. Using project and District Health Information System data to determine contributions of APHIAplus and calculate potential outcome indicators leaves an inconclusive picture of the project’s actual contribution 5 (but in line with findings presented in Question 2) regarding specific project activities and support for various indicators. The limited data presented by the project, because of their short implementation timeframe, limited conclusions about outcomes achieved. Also, significant changes in activities and indicators from year to year limited conclusions and any assessments of strategy effectiveness or contributions to Results 3 and 4. Question 2. Effective Strategies The Foundation Package. Capacity building at the facility level (supply-side support) and activation of community units (demand-side support) were shown to be key strategies to increasing availability of services and contributing to key results. Specifically, integrated service delivery, referrals, and linkages between the facilities and their communities were shown to fill a large gap in HIV care and treatment services. While there are still significant gaps in maternal, neonatal, and child health and a need for continued inputs to maintain service quality, these components of the Foundation Package show promise to be an effective strategy because they are directly support government systems. Results were inconclusive about the extent APHIAplus support for OVC is contributing to reducing vulnerability because of insufficient record keeping on project activities and beneficiaries. The project’s strategy for gender mainstreaming and prevention of gender-based violence is neither clearly articulated nor implemented across activities. Thus, clear contribution to expected project results cannot be determined. The Most Vulnerable Groups Package. Anecdotal evidence suggests that elements of this package were appropriate for increasing demand for services among key populations; community-level health messaging and linking to clients were shown to have positive preliminary results. However, linkages to services between and among MARPS, CBOs, and facilities were not found to be systematic. Also, the effectiveness of evidence-based population targeting and intervention design was found to be weak; accepted best practices such as MARP–friendly providers and facilities were rare. Economic strengthening activities showed early potential to reduce vulnerability of these populations but need a systematic approach and a firmer theory of change to demonstrate results at scale. Economic support to improve health-seeking behavior has significant theoretical merit. However, the large number of both project target populations and activities make this strategy operationally difficult to realize in a systemic way and also make attribution and control for confounders difficult. The Partnerships Package. While there is anecdotal evidence of successes in partnerships within the health sector and with national-level actors, the intended results of this package as described in project documents was not evident during evaluation fieldwork. The project does not adequately measure or document the partnerships or relationships it has developed as part of the package, nor the results that have come from these connections. Also, most partnerships found were with the project’s direct subgrantees—as opposed to links between the project and other funded initiatives or existing networks. Partnerships within the projects’ social determinants activities are lacking. Results such as joint planning and leveraged resources have yet to be seen (although this may be due more to timeframe than to design). It is unclear whether, on the one hand, partnerships that USAID envisioned and on which the effectiveness of this strategy would be based were unavailable or, on the other hand, the project failed to make appropriate connections. Question 3. Use of New Programmatic Evidence APHIAplus’s support to the Government of Kenya in health planning, policy development, and dissemination showed success in enabling APHIAplus–supported health facilities and community units to align with most national health sector policies and guidelines. While APHIAplus implements many 6 community-based activities that are aligned with national and international guidelines, certain gaps were revealed—particularly appropriate targeting of interventions and record keeping. The extent to which internal Quality Assurance/Quality Improvement (QA/QI) activities have been conducted by the project or used to improve programming was not clear during fieldwork. As such, the team cannot make any definitive conclusions regarding the project’s use of internal QA/QI program data. Question 4. Appropriateness of the APHIAplus Model While some underlying elements of the model are theoretically sound, significant operational challenges—such as significant regional differences, vast geographic distances, limited coordination with national mechanisms, and complexities of the management model—faced during implementation have limited any substantial conclusions regarding the model overall. Question 5. Sustainable Activities In general, APHIAplus is designed to work with the Government of Kenya and within existing local systems, as such several project activities have the potential to improve and strengthen local systems (such as establishing/improving data systems and use, strengthening policies/guidelines, supporting the annual work planning sessions). However, to foster sustainability, direct financial and GOK support activities and gap-filling measures must be reconsidered. OVERALL RECOMMENDATIONS* Before the end of the project: • The project should conduct a rapid survey of human resources for health needs among supported facilities to prioritize and target capacity-building efforts to fill gaps before the project ends.. The results of the survey can be provided to Capacity and FUNZO—the national mechanisms that support this type of capacity building—to include in their next work plans as necessary. (Q2) • Continue to support the government and ASSIST (the national technical assistance mechanism responsible) with the rollout of Kenya Quality Model for Health (KQMH) at facilities. (Q2) • The project should conduct rapid assessments of its CU activation activities, PLHIV, youth, and social determinants interventions to understand what has been achieved during the life of the project with recommendations for future investments. (Q2) • Support the GOK to improve patient record keeping and referral tracking by ensuring that patient forms identify the referral source and type of service on each record to facilitate the referral feedback loop. (Q2) • The project should continue to support the development, rollout, and adherence to national guidelines, with continued targeted capacity building to District Health Management Teams (DHMTs) and CUs in evidence-based planning with stakeholders. (Q2/Q3) *Recommendations for beyond project closure and considerations for future design are found in Annexes C.1 and D. 7 • The project should provide high-level technical assistance to its subgrantee partners, in line with rapid assessment findings, and specifically to ensure consistent alignment to national and international guidelines and standards. (Q2/Q3) • APHIAplus should report against the results chain presented in the year 3 M&E Plan, utilizing a selection of new indicators approved by USAID and including all F–indicators that need to be reported to USAID/Washington and Congress. This will enable USAID to determine to some degree APHIAplus’s contribution to Results 3 and 4. (Q1) • USAID should conduct a Data-Quality Assessment of APHIAplus Nairobi/Coast to address reporting issues specifically related to PEPFAR indicators that measure individual exposure to individual and/or small group level interventions. (Q1) 8 EVALUATION PURPOSE AND QUESTIONS PURPOSE This is an independent performance evaluation of the AIDS Population and Health Integrated Assistance—People-Centered, Leadership-Focused, Universal Access, and Sustainability Project (APHIAplus) Nairobi/Coast project (henceforth called APHIAplus in this document). APHIAplus is a cooperative agreement with Pathfinder International and USAID/Kenya’s Office of Population and Health (OPH). The three-year, $55 million award began on January 1, 2011, and ends on December 31, 2013. The purpose of the evaluation is to assess the performance of APHIAplus, primarily for accountability and learning. Its specific objectives are a) to conduct an in-depth qualitative assessment of technical and management approaches, coordination with the host country and other stakeholders, and support to country health systems; and b) to conduct a review of project achievements based on assigned targets in HIV/AIDS, family planning/reproductive health, and child survival activities. QUESTIONS This purpose and objectives of the evaluation will be achieved through answering the following five evaluation questions defined in the scope of work:* 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4?† Were the key expected outcomes achieved? If not, why not? 2. What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in Results 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the health sector? 4. To what extent was the APHIAplus model appropriate for achieving key outcomes under Results 3 and 4 of the Implementation Framework? 5. What sustainable activities in service delivery and healthcare systems were established and/or strengthened at the district health facility and/or community level? Findings, conclusions, and recommendations will shape strategies, approaches, and activities for the follow-on program design. The primary audience for the evaluation is USAID/Kenya and the Government of Kenya. Secondly, the report is intended for Pathfinder, USAID/Washington, and other interested implementing partners. *Annex A includes the full scope of work. †Result 3, increased use of quality health services, products, and information; Result 4, social determinants of health addressed to improve the well-being of targeted communities and populations. 9 PROJECT BACKGROUND APHIAplus is one of five regional service delivery mechanisms under the APHIAplus Service Delivery awards, which were preceded by APHIA 2 projects covering eight Kenyan provinces. The five APHIAplus projects were awarded in January 2011 after an extensive redesign process that began in late 2009 following an assessment of USAID/Kenya’s health portfolio. This process also resulted in the USAID/Kenya Health Implementation Framework 2010–15. The Implementation Framework envisioned the APHIAplus projects as supporting delivery of critical health services at the provincial and district levels while working in coordination with USAID–funded, national-level technical assistance partners.* Within the Implementation Framework, the APHIAplus design concentrated on Result 3 and Result 4. APHIAplus Nairobi/Coast covers Zone 2, described as a moderate-need zone that covers urban areas. It includes the Nairobi and Coast region (excluding Tana River District, covered in the Northern Arid Zone) as these are the two regions with the largest urban settings. The APHIA projects were expected to implement evidence-based, high-impact, cost-effective interventions that considered the following: • Supporting complementary technical areas tailored and designed for the proposed zone • Working through provincial government structures (to ensure coordination) to provide support at district and community levels • Linking USAID’s service delivery support with systems￾strengthening activities • Ensuring rapid deployment, smooth transition, and continuity of essential care services without disrupting critical populations *Request for Applications, USAID/Kenya APHIAplus, Health Service Delivery projects, April 23, 2010. The Project Directly Supports USAID/Kenya’s Implementation Framework 2010–15 Results Result 3. Increased use of quality health services, products, and information 3.1. Increased availability of an integrated package of quality high-impact interventions at community and health facility levels 3.2. Increased demand for an integrated package of quality high-impact interventions at community and health facility levels 3.3. Increased adoption of healthy behaviors 3.4. Increased program effectiveness through innovative approaches Result 4. Social determinants of health addressed to improve the well-being of targeted communities and populations 4.1. Marginalized, poor, and underserved groups have increased access to economic security initiatives through coordination and integration with economic strengthening programs 4.2. Improved food security and nutrition for marginalized, poor, and underserved populations 4.3. Marginalized, poor, and underserved groups have increased access to education, life skills, and literacy initiatives through coordination and integration with education programs 4.4. Increased access to safe water, sanitation, and improved hygiene 4.5. Strengthened systems, structures, and services for protection of marginalized, poor, and underserved populations 4.6. Expanded social mobilization for health 10 • Working within the government health system in delivering health services* • Building strong partnerships with effective Kenyan nongovernmental organizations (NGOs), faith￾based organizations, the private sector, and other implementing partners essential for sustained service delivery Working with the government and other partners, APHIAplus service-delivery projects support marginalized, vulnerable, and underserved populations. These include youth, most-at-risk populations (MARPs), persons living with HIV, those on antiretroviral drugs (ARVs), orphans and vulnerable children affected by AIDS, women of reproductive age (pregnant and postpartum women), and highly vulnerable girls, neonates, and infants. The implementation strategy concentrates on achieving two levels of results—at the facility and community levels working through national, provincial, and district GOK structures and with communities. APHIAplus Nairobi/Coast works with a consortium of the following five partners: • Pathfinder International (the principal implementer) partners with governments, NGOs, and the private sector, to promote best practices and state-of-the-art approaches to family planning and reproductive health. • ChildFund International concentrates on child-centered change that leads to healthy and secure infants, educated and confident children, and skilled and involved youth. ChildFund brings experience in family-centered OVC care and community engagement for children’s health. • Cooperative League of United States of America (CLUSA) works with communities to identify and address health concerns. CLUSA’s training activities build organizational and management capabilities of communities to take responsibility for health services and raise local resources for community health, social, and economic initiatives. • Network of AIDS Researchers of Eastern and Southern Africa (NARESA) conducts HIV research and supports evidence-based policy formulation and program implementation. • Population Services International raises awareness for healthy behaviors, generates demand for health products, and promotes franchised private providers. PSI/Kenya conducts programs in malaria, family planning/reproductive health, HIV/AIDS, and child survival. The partners are supported by numerous local subgrantees. In addition to subgrantees, APHIAplus works with USAID–funded national mechanisms. APHIAplus implements three overarching strategies: the Foundation, the Most Vulnerable Groups, and Partnership Packages. THE FOUNDATION PACKAGE. The Foundation Package (supporting Result 3) of interventions focuses on systems-strengthening activities, which support the foundation of strong service delivery and contribute to sustainable quality improvement and health impact. These consist of key investments required to support local capacity and priority services, including evidence-based best practices and services required for the general population. *The APHIAplus projects were designed to work at level-3 facilities and below. For reference, level 1 is the community unit; level 2 is dispensaries; level 3 is health centers; level 4 is district hospitals; level 5 is county-level referral hospitals; and level 6 is national-level referral hospitals. 11 THE MOST VULNERABLE GROUPS PACKAGE. The Most Vulnerable Groups Package (supporting Results 3 and 4) of interventions targets underserved, poor, or marginalized groups who experience serious barriers to claiming their rights to health and health care. This package consists of tailored approaches and targeted interventions, which respond to the needs of specific groups within the subzones. It concentrates on vulnerable populations and thus addresses barriers to access to Kenya Essential Packages of Health (KEPH) and related health services in facilities and at the community level. THE PARTNERSHIPS PACKAGE. The Partnerships Package (supporting Results 3 and 4) of interventions is described as formalized partnerships and linkages to leverage limited resources and maximize the sustainability of improved health outcomes. Additionally, it ensures appropriate coordination with local- and national-level service providers to build synergies that achieve the anticipated program outcomes. Crosscutting project approaches include service integration, linkages between levels of facility care, facility and community services, gender, and quality assurance/quality improvement. The project intended to “regionalize” or tailor these strategies to respond to the health needs and contexts specific to Nairobi and Coast. 12 METHODS AND LIMITATIONS The evaluation employed a mix of quantitative and qualitative methods to answer the evaluation questions. Data-collection methods included desk review, data extraction, key-informant interviews, site visits, observational checklists, and facility- and community-level group discussion. The data-analysis methods included descriptive statistics, data mapping, pattern/content analysis, response convergence/divergence analysis, and triangulation.* Data collection took place during three and a half weeks in July and August. The team visited 14 facilities (plus two additional sites for instrument piloting), which were purposively selected by USAID in the Statement of Objectives (see Primary Sampling Data Frame, next page, for more details).† To collect data from private and/or faith-based facilities (TUNZA–supported sites), the team visited three additional facilities. At the community level, the team collected data from 32 community-based organizations (CBOs) and 26 client groups that were selected based on representation of the targeted audience, urban/rural settings, and linkage to facilities visited. This data were augmented by KIIs with 79 individuals who included implementing partners; national-, county-, and district-level government officials; national technical assistance partners; and relevant community and network representatives.‡ The evaluation team experienced numerous significant limitations that affected the formation of valid and evidence-based conclusions—namely 1) the timing of the evaluation, 2) the project design, 3) project documentation, 4) the primary data sampling frame, 5) health management information systems, 6) KIIs, and 7) various confounders—each of which we will outline here. Many of the data limitations listed herein are also noted findings that will be further detailed within the appropriate section of the report. TIMING OF THE EVALUATION. This evaluation occurred two and a half years into a three-year project (reduced from five years). Data for the evaluation were collected when most activities had only a year or less of data to report, which limited the results found mostly to outputs and preliminary outcomes; higher-level outcome and any impact-level (including sustainability) results and trends/progress were nearly impossible to assess. This means the evaluation needed to concentrate on the implementation processes, the current status of activities, and early results—limiting an assessment of outcomes. PROJECT IMPLEMENTATION and DOCUMENTATION. APHIAplus has undergone significant changes in activities, PMPs, and the structure of work plans. This created two major limitations: a) indicators and their targets were inconsistent, making tracking of progress and achievements hard to * MSI received a statement of differences with this evaluation from AphiaPlus, which is published here as Annex J. Comments in the statement of differences largely parallel comments receive from AphiaPlus on the draft report. In response to AphiaPlus comments on the draft report, MSI made a substantial number of changes to this final version. A detailed report on specific changes made in response to each comment from AphiaPlus on the draft report were transmitted to the USAID/Kenya Contracting Officer’s Representative (COR) for this contract on September 24, 2014. This transmission is available upon request. †USAID/Kenya OPH provided the sample as part of the Statement of Objectives. The coverage concentrates on two program areas— a) Prevention of Mother-to-Child Transmission and b) Treatment—since facilities that provide these services also provide other health services. Other criteria considered included a) geography, b) availability of a community unit attached to the site, and c) volume of service delivery statistics (a mix of high-, moderate-, and low-volume of service delivery). ‡See Annex B for an updated methodology. 13 assess, and b) it was difficult to deduce linkages connecting results, outcomes, activities, and targets within results found. This led to difficulty in assessing effectiveness of implementation strategies. PRIMARY DATA SAMPLING FRAME. The facilities visited were selected by the Mission and were a convenience sample tiered by demand (high/medium/low), skewing sampling toward mid-demand facilities—as opposed to a stratified random sample to produce more representative results. In addition, APHIAplus’s scope includes a wide range of activities. As such, it was difficult to assess all components against project documentation. Thus, the results in this report provide only an indicative snapshot of the project’s performance and cannot be generalized.* HEALTH MANAGEMENT INFORMATION SYSTEMS. Incomplete data sets and the inability to retrieve data in a timely manner (before tool development) impeded the ability of the evaluation team to design the tools as initially envisioned. The data extraction and analysis was carried out after the fieldwork. KEY INFORMANT INTERVIEWS. Because of the devolution process, many potential key informants have recently been redeployed to new positions. Often individuals were recalled by APHIAplus to meet with the evaluation team, which led to interviews being carried out with former instead of current staff. The evaluation fieldwork took place during the teachers strike, and the team was unable to meet with teachers and school officials or visit schools to observe school-based project activities. CONFOUNDERS. Further attribution was not possible given the number of other actors in the project areas working with similar mandates to APHIAplus, as well as the fact that the project is a follow-on project of past APHIA projects. The focus of the evaluation is therefore on contribution. *Generalizability is a standard statistical statement that refers to nongeneralizability of evaluation data owing to not visiting a statistically significant number of sites. This also applies to interview data. 14 KEY FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS The report will address Question 2 first to present and analyze project inputs and strategies, followed by Question 1, which will determine the extent to which these inputs and strategies contributed to outcomes as measured by project (or external) data. This will be followed by Questions 3, 4, and 5. Conclusions are presented in textboxes to clearly distinguish them from Findings. Recommendations focus on what the project can do before it ends in December 2013. Additional recommendations for USAID future programming and decision-making can be found in Annex C. QUESTION 2. WHAT STRATEGIES EMPLOYED WITHIN THE WORK PLAN WORKED AND WHAT DID NOT WORK FOR SUCCESSFUL IMPLEMENTATION AND ACHIEVEMENT OF KEY OUTCOMES IN RESULTS 3 AND 4? WHY? The team used the three defined packages of interventions—the Foundation Package, the Most Vulnerable Groups Package, and the Partnerships Package—as an organizing principle for the evaluation.* To answer the evaluation questions, it was necessary to link these strategies to Results 3 and 4 and their subsequent Intermediate Results (IRs). The three strategies crosscut both Result areas; all are implemented at the facility and community level. The Partnerships Package was designed to support project activities and achievements within the other two packages and thus is not directly associated with any indicators. Each section following is linked with the appropriate IRs.† Findings and Conclusions for Result 3. Increased Use of Quality Health Services, Products, and Information Foundation Package. To assess the Foundation Package (systems-strengthening activities that support the foundation of strong service delivery and contribute to sustainable quality improvement and health impact), the evaluation team visited 17 facilities,‡ using an observational checklist and conducting key￾informant interviews (KIIs) with facility staff, facility management teams, and DHMTs. APHIAplus initially planned to provide facility support based on individual onsite facility needs identified in the annual work planning process to achieve Intermediate Result 3. APHIAplus models its work plan on the GOK’s agreed-on annual work plan. Therefore this evaluation has used APHIAplus work plans to measure whether APHIAplus has worked to support the government to carry out its plans and fill any identified needs to better enable the GOK to provide quality services.§ Project support was spread across four major health services: HIV/AIDS, tuberculosis, malaria, and maternal, neonatal, and child health. Types of support identified during the evaluation were in line with *This approach was set forth and approved in the Evaluation Methodology. †Strategies are presented clearly in the project description as the main packages of interventions. The project documents do not always link all of the interventions to Results 3 and 4, so the evaluation team has mapped interventions to the appropriate result. ‡USAID selected 14 of the 17 sites. §APHIAplus supports 512 facilities at level 4 and below. A “rationalization” exercise was undertaken in the last quarter of 2011 with the U.S. government and the Kenyan Ministry of Health to distribute facility support across different partners to minimize duplication of technical support. Therefore, some earlier supported facilities were no longer supported by APHIAplus. 15 project planning documents in terms of capacity building of health facility staff and health management teams: formal short-term trainings (FSTs), continuous medical education (CME), and on-the-job training (OJT) in several technical thematic areas (including data use and decision-making). The project also gave support in the form of facility renovations and provision of key supplies and equipment. Major Activity: Capacity Building of Health Facility Staff and Health Management Teams Training. Staff trainings. Trainings were done in a cascade format, whereby clinicians and nurses who attended FSTs were trained as trainers and were responsible for training other staff in the facility. KIIs with all 17 Health Facility Teams confirmed that training subjects responded to needs identified by each respective health facility, and included dissemination of new service delivery guidelines as well as sessions on how to integrate services. (Question 3 will address project support to dissemination of guidelines). Health facility staff at 16 sites reported that capacity-building activities generally were useful and - enabled them to better manage patients through improved documentation and indicator tracking methods, a systemic improvement, which supports IR 3.1: Increased availability of an integrated package of quality high-impact interventions at community and health facility levels. In addition, all facilities visited reported a shift from short- to long-acting methods of contraception based on the latest RH/FP guidelines disseminated by the Project, supporting IR 3.4: Increased program effectiveness through innovative approaches. According to best practice, this shift could exemplify early success in improving service quality. That said, the evaluation team observed some gaps between project work plan targets and achievements in RH/FP* and maternal, neonatal, and child health (MNCH) services. For example, only 1 out of 15 RH/FP clinics visited had received training in Commodity Management, though project work plans cite this type of training as part of the package of support for all facilities, including linking staff with FUNZO (the national capacity building partner) for more formal training programs in this area.† Seven out of 17 MNCH clinics were also found to have gaps in the following trainings slated for all support facilities: emergency obstetric care for prevention and management of pregnancy complications (including postabortion care); early initiation and exclusive breastfeeding; baby-friendly hospital services; immunization; and Integrated Management of Childhood Illnesses. Based on observations and interviews at all facilities visited, APHIAplus prioritized trainings based on funds availability—in all sites visited more emphasis was placed on HIV integration and HIV–related thematic areas, as opposed to MNCH and RH/FP. Four DHMTs interviewed cited that training requests were often denied owing to APHIAplus budget limitations. Direct financial support was provided for training activities such as providing training venues, and lunch and transportation reimbursement, as necessary to make the training possible. Technical support and facilitative supervision. APHIAplus is providing technical support to staff and facilitative supervision to all 17 APHIAplus–supported facilities in conjunction with the DHMTs. Technical support was found in the form of teaching facility staff in the use of job aids and MPH and KEPH protocols and guidelines. Additionally, APHIAplus (through the consortium partner NARESA) used mentoring to build the capacity of staff, through a wide range of clinical and systems topics and *Two health facilities visited by the evaluation team were faith-based and therefore did not provide family planning services (or receive support from APHIAplus in this area). †Additional discussion of the links between APHIAplus and the national level mechanisms is provided under Question 4. 16 techniques, as cited during the key informant interview with the NARESA team lead. As cited in interviews and project documents, the APHIAplus strategy was to build a cadre of in-house mentors as well as developing particular mentors’ skills in clinical areas or systems. Mentoring of facility staff was mentioned in all health facilities visited. Interviewees emphasized mentoring on M&E documentation, HIV, and RH/FP, including mentoring staff in level-3 facilities and above to insert IUCDs. However, interviewees did not display the wide range of topics described in the team’s interview with NARESA. The mentoring system, which is separate from the supervisory activities as described by health facility staff (in all facilities visited), was ongoing and also provided when requested. Documentation from APHIAplus on mentorship activities was limited to data on the number of mentoring visits per subcounty and did not provide detail on topic or skill areas. While more detailed data were described during the NARESA interview, they were not provided for review. In terms of Facilitative Supervision, all 17 DHMTs interviewed cited direct financial support from APHIAplus to conduct supportive supervision trips, and that APHIAplus staff also conducted monthly Facilitative Supervision of VCT/HCT Counselors to ensure adherence to guidelines. In all of the facilities visited, APHIAplus has supported the distribution of Ministry of Health (MOH) data-collection tools, KEPH guidelines, mother-to-child booklets, and other related job aids, as well as provided training updates on the use of these tools. Consistent with the project work plans, interviews at all sites visited stated that APHIAplus supported counselor supervision sessions every month, which helped ensure that all services adhered to standards of confidentiality, counseling, and consent. Interviews and observational assessments of the Comprehensive Care Clinics (CCCs) in all 17 facilities visited revealed that the project supported DHMTs and Health Facility Managements Teams to fill gaps in service provision, particularly at level-2 facilities. Support entailed continuous medical education in key HIV–related thematic areas, facilitation of supplies (consumables and HIV test kits), financial support for PLHIV support groups, and defaulter tracking efforts. Thirteen out of 17 informants at the county level and below (level 3 and below, meaning DHMTs and facility staff) cited key support from APHIAplus (in the form of trainings and technical support) in monthly data use and feedback meetings. These meetings are used to review data and highlight gaps in service provision. Consistent with project work plans, APHIAplus supported the rollout of the new Kenya Quality Model for Health at the facility level.* At the time of this evaluation, KQMH had been piloted in only two sites nationally through the University Research Corporation (URC) Applying Science to Strengthen and Improve Systems (ASSIST) project (a national technical assistance partner). The evaluation team visited one site in Kilifi County where APHIAplus has supported the formation of Quality Assurance/Improvement Teams as per the KQMH policies. The project has worked with URC ASSIST to roll out the KQMH guidelines, making the tools and guidelines available, providing direct financial support for traditional birth attendants and other community unit workers to attend the meetings, supporting the meetings with lunch, and providing TA during the QA/QI meetings. *The team was unable to meet with URC Assist, the technical assistance partner leading the rollout. 17 Conclusion. APHIAplus is providing training to staff, and technical support and supportive supervision to DHMTs and Facility Management Teams to comply with MOH policies and guidelines. There is anecdotal evidence* that the project is making a positive impact on improved patient management, service delivery, and data quality and data use at the facility level where it is taking place. However, training was uneven, with some facilities and some training topics yet to be addressed. As well, some of the support given was direct—training venues and costs; APHIAplus staff facilitating data review and program monitoring meetings; or paying stipends for DHMT managers to conduct supervision visits. While this direct support may have been necessary to meet project targets, it should be acknowledged that it may not truly build systems-level capacity in a sustainable way. This is an early conclusion, however, as it remains to be seen whether these are merely initial efforts that may evolve over time. Major Activity: Facility Renovations. Based on a review of project documents, initially APHIAplus had no renovations mandate, but in an effort to expand availability of high-quality HIV services USAID gave approval for the project to spend up to 10,000 USD per facility to conduct minor renovations at select health facilities. Interviews with the Pathfinder service delivery team and seven DHMTs indicated that the facilities and types of renovations were determined based on onsite needs assessments. Project documents shared with the evaluation team state that 53 facilities have completed or have ongoing renovations supported by APHIAplus (out of 180 initially surveyed).† KIIs and direct observation confirmed that 12 facilities visited had undergone targeted renovations in CCC units, MNCH delivery units, laboratories, pharmacies, and commodity stores. Structured observation showed improved capacity to provide privacy/confidentiality, hygiene, and, most critically, separation of TB patients from general clients to reduce communication of the disease. Also, the evaluation team’s visit to the community corroborated the health facility teams’ perception of the renovations. Three out of 11 community units (CUs) interviewed gave positive feedback about the project’s renovations, noting provision of containers for CCCs, pharmacy equipment, and desks. Respondents credited these provisions with an increase in service demand, saying that, “clients now come to the improved facilities.” At the same time, all 17 facilities reported that they had a shortage of space in which to deliver services and meet client needs, irrespective of having received renovation support previously. Currently the USAID health portfolio does not include additional support for facility renovations or further budget for infrastructure improvements. Conclusion. APHIAplus has provided support in facility renovations, which anecdotally has improved the ability of staff to provide services in accordance with health service guidelines and has led to increased facility use in some areas. However, space limitations cited indicate that the renovation efforts may create an increased expectation for additional renovations among HFMTs and increased demand for services within communities that facilities may still not be prepared to handle without increased support. Major Activity: Provision of Supplies and Equipment. Part of the APHIAplus strategy (as reflected in work plans) was to support the expansion of available quality services by ensuring no stock￾outs of critical supplies and equipment. Interviews with all 17 HFMTs revealed that APHIAplus had provided multiple consumables, though staff cited gaps in provisions of MNCH supplies (see Annex E). Facility observation found anthropometric measurement equipment‡ in 14 out of 17 and newly established Oral Rehydration Therapy (ORT) Corners (a physical space mandated to provide ORT *Anecdotal evidence is a social science term that refers to perceptual data based on a few individual observations. †Renovations Assessment (no date); Infrastructure Improvement; For Better Service Delivery (no date). ‡A more detailed breakdown of supplies to health facilities visited is provided in Annex E. 18 information and supplies in all facilities) in 7 out of 17 project-supported MNCH clinics, and no nutrition corners.* Conclusion. The project delivered critical supplies that enabled delivery of key services in project sites (with notable gaps in MNCH commodities). Major Activity: Support to the Lab Network. Interviews with the 17 HFMTs pointed to various types of support to facility labs in diagnostic capacity, including staff training in proper procedures and documentation in TB and HIV laboratories and External (test) Quality Assurance, and provision of equipment and supplies (even linking with the Capacity project† to provide lab staff in a few cases). Prompted by cited delays in test turnaround time, direct provision support also included the paying for courier services to transport specimens and results between level-4 reference labs and level-3 facilities in 15 out of 17 facilities assessed. During an interview with the evaluation team, the MOH lab coordinator suggested that the lab networking will struggle without donor support. Conclusion. The project was found to have solid results in lab support with training on quality assurance and improved documentation; also, the courier system was found to be an improvement that decreased test turnaround time (and hence anecdotally increased service/testing uptake), but is not sustainable without outside funding. Integration of Key Services. In line with strategies outlined in the work plan in ensuring integration of key services, through the major activities above, APHIAplus project documentation placed emphasis on supporting the integration of key services in accordance with the KEPH. As such, KIIs revealed that all health facility management team members, service providers, and Pathfinder service delivery team members understand the issues around and appreciate the benefits of RH/HIV integration, TB– HIV integration, multipoint HIV counseling, and testing and linkages with HIV care and treatment centers. KIIs with 17 health facility teams revealed that APHIAplus has supported TB–HIV service integration by enhancing defaulter tracing, integrating data collection tools, and providing guidelines. Staff at 15 health facilities confirmed APHIAplus supported integration of HIV, STI, and RH/FP services through training in integrated messaging (e.g., dual protection) and the direct provision of emergency contraceptives and post￾exposure prophylaxis (PEP) and job aids (wall charts and other materials for providers). Health facility staff responses pointed to increases in STI screening, uptake in FP, and improved management of integrated RH/FP services within their facilities. KIIs and observational assessments at all tier levels of health facilities visited revealed that the APHIAplus project is building on APHIA II results by supporting integration of Prevention of Mother-to-Child Transmission (PMTCT) and Early Infant Diagnosis (EID) service provision within MNCH, ANC, and L&D *Nutrition corners are not mentioned in GOK guidelines. †The Capacity project is a national TA partner mandated to work with the national GOK on human resources for health. CERVICAL CANCER SCREENING Screening for cervical cancer was one of the novel introductions APHIAplus made to enhance the reproductive health of clients accessing RH/FP clinics. The project made investments in cervical cancer screening in 15 health facilities visited in terms of provision of key supplies, equipment, and training of staff. Interviews revealed that staff felt better prepared to conduct visual inspection and diagnosis of incipient and late stages of cervical cancer. 19 clinics. Newly rolled-out EMTCT guidelines include a mandate to integrate PMTCT into services provided in MNCH clinics; the project’s year 3 work plan incorporates these guidelines. KIIs with PMTCT staff, however, revealed inconsistent results for the integration of MNCH and FP back into PMTCT services: while screening for and enrollment of HIV–positive pregnant women into PMTCT services occurs regularly at ANC clinics, it seldom occurs at MNCH units. For example, HIV test kits and guidelines were available and observed in only 7 out of 17 MNCH clinics visited by the evaluation team. Additionally, though facility staff at all 17 sites felt the project has strengthened overall EID services, interviews in MNCH clinics cited losing patients to follow-up in the management of HIV– exposed and confirmed positive infants in pediatric HIV care and treatment to be a constant issue. In terms of gender-based violence integration, only 2 of 17 health management teams interviewed mentioned receiving trainings or provision of guidelines and legal documents on post-rape care within RH/FP services.* Conclusion. The Project was found to have provided consistent support for integration of health services in support of KEHP compliance at this point in time: data showed positive early results in some facilities in some services at the outcome level, such as increases in FP usage and improved comprehensive care management. but poorer results in others, such as the unidirectional integration of PMTCT where in year 3 the project has begun to align its work plans with the newly rolled-out EMTCT guidelines. Necessary changes in facility workflows and referrals for full alignment to these new guidelines will likely take time to become habitual for facility staff and thus see the full benefit of increased enrollment. Activation of the Community Strategy. Support for the activation of Kenya’s 2006 Community Strategy, health care delivery at the community level (or Level 1), was built into the APHIAplus Foundation Package as a way to improve community level service delivery and improve linkages to higher-level facilities. Review of the 2012 project PMP showed reporting of 86 “fully functional” CUs in Nairobi and 80 in Coast as a result of APHIAplus efforts by the end of 2012.† Further project documentation indicates a breakdown of functional, semifunctional, and nonfunctional CUs (see Table 1), based on 17 characteristics, but do not clearly define these categories or describe at which point CUs move from one category to another.‡§ Key-informant interviews with APHIAplus project staff indicate that the project is supporting several semifunctional CUs. There are discrepancies between what is reported in the PMP and what project staff interviews and additional documentation show. *There is further discussion of the projects approach and activities related to GBV prevention included in the section looking at activities contributing to Result 4. †USAID defines “fully functional” CUs based on the payment of 2000 Ksh allowance to community health workers (CHWs). ‡Functionality of CUs, APHIAplus, 2013. §This information was provided by APHIAplus (September 13, 2013): “From 2013, fully functional CUs are those where the CHWs are paid a performance‐based Ksh 2,000 monthly stipend. Semifunctional CUs meet GoK criteria for functional CUs but receive Ksh 1,000 ([because of] budgetary constraints). Nonfunctional CUs are those that do not meet the GoK criteria for functionality and do not receive any stipend.” 20 Table 1. Status of Community Units Functionality of Status of Community Units Nairobi Coast Total Functional CUs 59 26 85 Semifunctional CUs 23 54 77 Nonfunctional CUs 4 2 6 Totals 86 82 168 Eight out of 21 key informants in the district and central levels of the Ministry of Health told the evaluation team that the MOH does not currently have resources to take over project support to those CUs and described APHIAplus support to the Community Strategy as essential to its functioning. One key informant stated, “APHIAplus is the only one working on the Community Strategy.” When asked how APHIAplus supported CUs in its official capacity, all surveyed CUs* thought the project did “an adequate job supporting CHWs.” This support included training and supply of identification badges and referral books; some (but not all) received bags, bicycles, and other forms of in-kind support. From year 1 to year 2 targets shifted considerably because of changes in implementation priorities over project years for the two regions (as described in project documents). Table 2 shows that the Project did not meet targets for numbers of trained CHWs† in Nairobi or Coast for either project year. Documentation was not provided on overall training breakdown (including nature and duration of training activities) to CHWs. No information was available regarding CHW performance and contribution to key results over time (e.g., trends in deliveries in facilities). Table 2. Number of Community Health Workers Trained NAIROBI COAST 2011 actual (target) 2012 actual (target) 2011 actual (target) 2012 actual (target) 8,297 (3,650) 4,013 (7,195) 15 (20) 7,593 (4,000) Evidence from APHIAplus project work plans and quarterly reports suggests utilization of monitoring data within CU programming and by community-level health workers. Nine out of 11 CU staff groups interviewed reported that they were able to use their monthly data for decision-making or strategic planning mainly through the “chalkboard” system (MOH 516). The chalkboards were observed in 12 out of 15 level 1–3 facilities. One CU reported having received training on mHealth and eHealth reporting as part of the APHIAplus pilot of this new technology. Seven out of 11 CUs interviewed described receiving project assistance to start IGAs and VSLAs. APHIAplus also assisted with CBO registration and training in proposal writing, to increase funding capacity beyond project support. *CU members were surveyed or interviewed in groups, including CHWs, Community Health Committee members, and other community leaders who are part of the CU. †This indicator refers to training of all CHWs, including youth leaders, peer educators, and the like. 21 Conclusion. APHIAplus has played a central role in activating the Community Strategy in Nairobi and Coast, with early evidence showing facilitation of evidence-based decision-making and facility linkages. But there is little evidence at this time (soon after CU activation) showing outcome-level results, such as increased service demand, linking of clients to services, or any outcomes of IGA/VSLA or CBO support efforts. Beyond the IGA/VSLA efforts for self-financing, the activation of the community strategy is dependent on APHIAplus financial and technical support, raising questions for long-term sustainability. Most Vulnerable Groups Package. This package was designed to increase uptake of health services by marginalized populations with a two-pronged approach: a) targeted outreach and services to these populations and b) reducing barriers such as stigma and violence at the community level. The target groups listed in project documents include FSWs, MSM/MSWs, PWUD (including IDUs), PLHIV, and youths. As described in project documents, each group is targeted with appropriate behavioral interventions, taking into account appropriate guidelines (adherence to guidelines is discussed under Question 3). In terms of tailored services, KIIs and observational assessments revealed that approaches fell into two major camps: community-level and facility-level efforts. At the community level, APHIAplus supported mobile testing services, targeted messaging at public and household levels, HIV–themed CBOs, and increased leveraging of CHWs. At the facility-level, service improvements included specialized provider approaches and integrated service support. Community Level. Major findings of APHIAplus support at the community level included mobile Voluntary Testing and Counseling (VTC) and door￾to-door outreaches in most areas, consistent with project work plans. APHIAplus also directly provided supplies and equipment for community HTC, FP, and STI outreach efforts and engaged CHWs for a variety of tasks, including community messaging, patient identification, and referral and defaulter tracking. Evidence was found of trainings (Output) of CHWs in TB screening and referrals; Early Infant Diagnosis and HIV–Exposed Infants follow-up; refresher/ tailored RH/FP outreach techniques; and defaulter tracking for all service dropouts. The evaluation team observed other outputs of defaulter tracking efforts (daily attendance registers) in all 17 clinics visited. Though, there was inconsistency in evidence on the outcome indicator of improved regimen adherence, as return patients were not marked as either those previously lost to follow-up and returning after outreach or just regular patients who never dropped out. Anecdotal outcome data from interviews with clinic staff at all facilities indicate early success in increasing demand for and retention to all types of HIV–related services, because of increased CHW–targeted outreach, but solid data showing significant increases against a preproject baseline were not yet seen after such a short implementation timeframe. Referral feedback loops (tracing CHW–initiated referrals through the system) were also inconsistent. Evidence of APHIAplus support to HIV–oriented community support also included IGAs/VSLA programs for members: three of four youth groups surveyed reported project support to training in IGA efforts, with one group found to have started its IGA efforts. The intended outcome reported was self-financing PROMISING PRACTICE At Kilifi District Hospital, the CCC nurses reach female partners and children of married MSM by conducting HIV testing in the household post–HTC follow-up with no disclosure of MSM status to the wife. Of the 12 MSM who are married and whose wives were tested, there was 1 discordant couple. Some of the children are positive. Reaching wives and children of HIV–infected MSM through household HTC is a promising innovation because it reaches female sexual partners of married and often hidden MSM, and does not require MSM–status disclosure. 22 and sustainability of these groups’ efforts to reach out to HIV–affected youth after APHIAplus funding ends, but it is too soon for these results to have materialized. Two of these four groups also reported APHIAplus–funded youth center renovations. PMTCT and male involvement in RH efforts were found to have spanned both facility and community levels. At the community level, CHWs were trained in public messaging, household visits, psychosocial support, and defaulter tracing for these topics; stipends to support group leaders/ staff and transportation reimbursements from the project were also reported in all 17 facilities visited. The two Mentor Mother programs* interviewed reported that staff received monthly allowances of up to Ksh 9,000.† Facility Level. At the facility level, staff in 1 facility out of 17 mentioned the use of male champion support groups in the facility as enhancing male involvement in RH/FP. Staff in all clinics reported an uptake in HIV–related services resulting from CHW messaging and outreach efforts in the community, but solid data showing significant increases against a preproject baseline was yet not seen after such a short implementation timeframe. Other facility-level MARP–targeted efforts included establishing youth￾friendly services for RH/FP in 13 out of 15 facilities; output-level results included office equipment and training of youth volunteers in messaging and counseling. Project documents state that youth-friendly services would include support for additional service areas, such as HIV testing and counseling and MNCH, but this was not observed in the field. While APHIAplus–supported CBOs were increasing referrals of MARPs to services (confirmed in KIIs with both CBO’s and clinic staff), health worker training efforts for providing MARPs–friendly services were very inconsistent (this does not include the youth volunteer training in youth-friendly services). Staff interviews in facilities indicated that not all facility staff received this type of training. Additionally, all facility staff interviewed who had been trained as “MSM friendly” providers reported feeling stigma and harassment from colleagues and their communities. According to interviews with health facility staff groups at all 15 APHIAplus-supported RH/FP clinics visited by the evaluation team, there has been marked increase in attendance and demand for RH/FP services. Interviewees believed this was due to intensified community outreaches by trained community health workers, youth-friendly service providers, improved referrals, and education (through community–facility linkages). KIIs with clinic staff at 15 RH/FP clinics revealed that despite the fact that referrals within the health facility and from the community have enhanced RH/FP–HIV/STI integration, measures to ensure a systematic documentation of referral feedback loops did not occur in all health facilities. The evaluation team also surveyed seven MARP CBO’s: one FSW, three MSM, and three PWUD organizations were interviewed. All surveyed PWUD organizations cited challenges, with the harm￾reduction emphasis being only on broader HIV prevention objectives, as many of their clients are still dealing with active addictions. All surveyed FSW and MSM and two out of three PWUD organizations cited the vulnerability of children and youth and lack of supportive programming for reduction of violence and police harassment as major challenges not addressed by APHIAplus–supported *Mentor Mothers groups are used to make sure that HIV positive mothers attend ANC, adhere to drug regimens, are not lost to follow-up, and remain in the system. These group leaders are based at the facility, do not have a government-specified allowance, and are not considered CHWs. †CHWs have an allowance of 2000 Ksh specified in the GOK Community Strategy. 23 programming (HIV prevention objectives). Key-informant interviews with two key population network leaders corroborated the above and told the evaluation team that it is necessary for support to take into account the emergent needs of the populations—for example, the critical issues of violence and legal support, in addition to health and HIV prevention.* Conclusion. The Most Vulnerable Persons Package has shown early anecdotal success with its community outreach efforts in increasing service demand among MARPs, improving CHW services to these groups, and improving systems such as referral systems and defaulter tracing. Some notable gaps in systems strengthening included closure of referral and defaulter tracing loops and full integration of HIV– related services with regular RH and FP services. While CHW success was prominent, provider trainings and improvements in services were found to be limited at this early stage. The effects of support to MARP–related community groups was found to be mixed: while IGA efforts showed promising beginnings and mentor mother/ PMTCT group leaders received direct cash allowance (though these do not match other GOK approved cadre amounts), the demand this support created and its sustainability pose a risk to APHIAplus’s advancements. Additionally, APHIAplus–supported HIV– prevention activities for key populations do not seem to be meeting the basic needs of clients; though these are out of the purview of APHIAplus, as currently designed they are important considerations for future programming. Partnerships Package. This component was meant to formalize partnerships with relevant organizations and projects to leverage support and improve project reach at the facility. Project documents also state that APHIAplus would use existing networks and link with other donor-funded projects to maximize results and leverage support. Question 4 provides a detailed discussion of the linkages with the national TA mechanisms. Fifteen out of 21 health sector stakeholders interviewed cited that APHIAplus has been flexible to respond to GOK priorities and has supported initiatives per the Annual Operational Plans. During interviews, 11 key county- and national-level stakeholders in the health sector cited stakeholder forums supported by APHIAplus as important activities in coordinating and mapping out support from donors and GOK on key health sector activities. GOK officials are still receiving technical assistance from the project in terms of participation in Technical Working Groups and support for policy development and dissemination. APHIAplus supported the Kenya Girl Guides Association (KGGA) with a school-based guiding program life skills curriculum with health messages focusing on primary and secondary abstinence. While KGGA is often constrained by its parent organization with regard to health messaging and interventions, the APHIAplus–supported project with KGGA builds on more than a decade of USAID assistance and is an example of reaching youth within the general population through existing structures for increased reach and sustainability. *While the 2012 MARPs Surveillance Report (NASCOP ) indicates that 18 percent of MSM in Nairobi and 25 percent of MSM in Mombasa are infected with HIV, interviewed key population CBO staff reported varying levels of HIV positivity rates for their clients (for example, Mombasa MSM 3 percent and Kilifi MSM 22 percent). The project does not disaggregate testing data by new/repeat testers or by positive/negative results in documentation reviewed by the evaluation team. No determination can be made on the causes without further documentation and research, but these findings raise questions of appropriate record keeping (at all levels and between levels) as well as highlighting the self-selective nature of these types of groups and how that might affect positivity rates. 24 County-level stakeholders in Kilifi cited significant APHIAplus support in linking PLHIV with Food by Prescription (a World Food Program initiative) involving logistical support, hiring staff, and holding meetings to maintain engagement. A 2010 assessment concluded that Food by Prescription in Kenya was responsible for improving nutritional status and health outcomes of clients.* Twelve out of 17 key informants from the health sector (county-level and below) cited significant support by other donors in a variety of overlapping areas, though only a rough description was provided by key informants without exact funding levels, activities, or mandates. Informants told the evaluation team that to avoid double counting and overlapping support partners were asked to operate in different sites. APHIAplus staff told the evaluation team they are working through existing networks that the project supports as subgrantees (e.g., the Council of Imams and Preachers of Kenya [CIPK] and Omari target Muslim communities in Coast). Established national religious networks met by the team (SUPKEM, NCCK, KCCB) shared work plans that overlapped with the APHIAplus project in terms of implementing similar interventions to similar target groups. Key stakeholders within these networks stated that they have not been engaged by APHIAplus as stakeholders or partners. Conclusion. APHIAplus has done well in linking and developing relationships with GOK health officials; support is well received, and there is early evidence of buy-in from officials. APHIAplus partnerships with KGGA, Food by Prescription, CIPK, and Omari demonstrate the ability of the project to leverage existing structures and projects funded by other donors to reach more beneficiaries or improve health outcomes. That said, there may be other partners and existing networks that APHIAplus could have worked with to implement activities for key populations. In addition, where projects continue to work in the same sites, it is important to note that linking inputs by APHIAplus to outcomes is difficult. Findings and Conclusions for Result 4: Social Determinants of Health Addressed to Improve the Well-Being of Targeted Communities and Populations APHIAplus documentation states that the project will address the social determinants of health largely as a crosscutting, integrated approach. The main emphasis is ensuring that marginalized poor and underserved populations have increased access to economic security initiatives and addressing food insecurity and nutrition to vulnerable populations. Work planning documents state that only the Most Vulnerable Groups Package and Partnerships Package feed into this Result Area. Most Vulnerable Groups Package. Support to Orphans and Vulnerable Children.† At project onset, APHIAplus OVC targets more than tripled—as the project was asked to assume support for OVC from previous projects.‡ All six OVC CBOs met by the evaluation team conducted a household validation survey in 2011 and 2012 to confirm OVC.§ Documentation provided to the evaluation team included a two-page completion report, blank tools, and charts of final numbers. The evaluation team *USAID. 2010. Food by Prescription in Kenya. Available at http://www.aidstar￾one.com/sites/default/files/care_and_support/resources/reports/food_by_prescription/AIDSTAR_FBP_Assessment_Report_Fin al.pdf. †Indicators under IR 4.5 are concentrated on support to improved systems for gender-based violence. ‡These included Track 1 and New Partners Initiative activities. §APHIAplus project documents state that the validation exercise was used to inform the project on OVC numbers (including unenrolled children in OVC households), priority programming needed (birth certificates, VSLA, etc.), and established linkages to the GOK cash transfer system. 25 did not feel that this documentation was appropriately detailed for OVC activities under USAID PEPFAR funding. All six surveyed CBO groups reported that OVC household enrollment is initially conducted by the CHW, and the information is then sent to the OVC focal person, Community Health, and APHIAplus local office for review. KIIs with three county-level and two district-level Children’s Officers made their lack of participation in actual OVC selection clear to the evaluation team. A visit to 10 conveniently selected OVC households confirmed that CHWs participated in the enrollment of OVC households. Feedback from KIIs and CBO groups indicates that many CHWs are caregivers and/or employed by the CBOs working with OVC. Both findings corroborate the project’s documented strategy for using CHWs in OVC activities. In September 2012, APHIAplus partners started digitalizing OVC data in the Orphans Longitudinal Management Information System. But currently each child has a physical file as part of a distribution list.* APHIAplus reports that these lists are filled and categorized under six serves areas, and records are kept in service files. OVC forms are collated to show numbers by gender and various services that the OVC received, per CBO. A conveniently selected sample of 30 files show inconsistencies in documentation and unclear reporting of support, such as incomplete documentation of support provided, multiple children in one file, and lack of monthly reporting forms recently instituted. All six surveyed OVC groups implemented the same services, including parenting skills and support groups, home visits to at-risk and affected families, disclosure and bereavement support, linkages with police and county administration, and VSLA/SILC groups. Four out of six surveyed OVC groups had support groups for OVC living with HIV. Within the six OVC groups, there were variations in number and level of direct and indirect services offered in health, protection, economic strengthening, food security, education, and safe shelter, water, sanitation, and hygiene.† In visits to a convenience, sample of 10 caregivers confirmed support received from APHIAplus. All six OVC groups were appreciative of the critical support received from the project. Conclusion. While there is some evidence of the Household Validation exercise, the project was unable to provide sufficient documentation of the results of the exercise. In addition, there is a potential conflict of interest caused by CHW involvement in the selection and validation of OVC as potential recipients of OVC household support as caregivers or employees of CBOs working with OVC. Anecdotally, APHIAplus support to OVC and their caregivers is well received, but records kept for OVC are inconsistent and lack clear reporting of support. Without proper record keeping it is not possible for the project to fully capture its contribution to reducing vulnerability among OVC and their families. *The evaluation team was told after data collection that the OLMIS was being piloted in a few sites but was not provided with a list of these sites. The project’s First Quarterly Report for 2013 and earlier work plans mention only a workshop provided during the quarter. †Further discussion of OVC programming alignment to guidelines is presented under Question 3. 26 Prevention of Gender-Based Violence and Gender Mainstreaming.* The third-year project work plan identified several gender mainstreaming and other gender and gender-based violence (GBV) activities but does not link these activities to clear strategies or outcomes. The two surveyed Gender Working Groups started during APHIAplus and are tasked to track GBV cases and work with the local provincial administrators, especially chiefs and other NGOs, to facilitate medical and legal support. Each group was trained in VSLA, the Sexual Offenses Act of 2006, and formed groups. Supported community groups implemented the Sita Kimya Campaign, which included 16 days of Activism against Gender Violence . Five out of seven key informants in Coast region reported that the project enabled the establishment of 14 safe spaces in Coast region with the provision of supplies and linkages to health facilities and police. However, safe space staff were not considered by informants to have up-to-date skills to manage GBV/SGBV survivors. Three surveyed CBOs that support OVCs and three PLHIV support groups did not demonstrate a comprehensive understanding of gender or gender mainstreaming in programming. APHIAplus gender staff interviewed stated that they did not feel that APHIAplus programming staff had a solid understanding of gender mainstreaming and thus could not implement activities appropriately. Conclusion. The APHIAplus approach to GBV and gender mainstreaming lacks a well-articulated overall strategy with clearly defined interventions that are aligned to internationals guidance. Project documents are not constructed in a manner that shows a clear relationship between process- and outcome-level indicators or strategies and the activities of which these consist, either for gender mainstreaming or for discrete gender activities. The extent of mainstreaming gender across community programming is unclear. Economic Strengthening Activities. Project documentation shows that APHIAplus has supported 360 VLSA/SILC groups in Coast region, consisting of more than 5,523 members, and 292 VSLAs in Nairobi consisting of 4,397 members. Some were founded before APHIAplus but continue to receive project support. Project documents state that over Ksh 52 million has been disbursed since these groups were formed, and over Ksh 15 million was saved last year. Additionally, project documentation indicates that OVC caregivers, CHWs, youth groups, and PLHIV are members of these VLSA/SILC groups, but documentation on group composition was not provided by the project. All 18 community groups surveyed, including youth, have been trained on organizational strengthening, financial literacy, and leadership skills and through this have formed table-banking activities and reported satisfaction with support in this area, while still citing areas where they would like more support. Issues included the need for more post–IGA training practical support. Community-based organizations that had not received economic strengthening activities reported interest in receiving training. Conclusion. APHIAplus support to economic strengthening initiatives (IGAs and VSLAs) has been well received, but after only two years of project activity it is difficult to quantify any changes or link evaluation evidence to sustained outcomes. Addressing Other Social Determinants of Health. Project documents reported inputs in enhancing value chain addition in farming, linkages to the Ministry of Agriculture for enhanced *Discussion of the support to the Kenyatta Hospital Gender-Based Violence Recovery Center is located in Annex F. 27 agriculture and livestock production, and the formation of producer organizations and cooperative societies. However, limited documentation was provided regarding the nature and breadth of this support to beneficiaries. Interviews with four key informants in the agriculture sector cited support from APHIAplus in providing trainings to PLHIV and other community-based support groups and providing training to groups of CHWs in agricultural techniques. APHIAplus addressed water, sanitation, and hygiene activities largely through OVC and CU programming, as well as through school-based activities. Provision of water tanks to schools, support to school health clubs, promotion and support to hand-washing days, and hygiene promotion (in terms of provision of soap and basins) were described by the one school health official with whom the team was able to meet. * The informant described these activities as valuable to improve hygiene, though gaps remain in sanitation. Conclusion. Overall, activities addressing the social determinants of health are integrated into programming for marginalized poor and underserved populations, and are used to strengthen the sustainability of investments in CUs, youth groups, and PLHIV support groups (as described in earlier sections). However, APHIAplus lacks clear documentation regarding the coverage and scope of support by targeted audiences within targeted counties, and lacks measurement of the robustness of interventions. Partnerships Package. Project documentation indicates that the project would engage district and provincial administrations at the community level as part of the partnerships package to engage all stakeholders and avoid duplication. In contrast, project staff told the evaluation team that they have limited involvement with local administrations. The team did meet one provincial local administrator (chief) in Chasimba who was happy with APHIAplus support, but would like to be more involved and engaged by the project in the future. Two national- and county-level stakeholders interviewed outside of the health sector cited not being involved or aware of APHIAplus activities, and were reportedly unhappy about APHIAplus’s excluding them from discussions and not supporting their activities directly. There was anecdotal evidence and project documentation to show that some OVC were linked to Equity Bank’s Wings to Fly project and have received scholarships for secondary education. Anecdotal evidence also suggests that, because of support from APHIAplus in registering community groups as CBOs, these groups have been able to leverage funds from the Total War Against AIDS project. Conclusion. There is little documentary or perceptual evidence of implementing the Partnership Package to address the social determinants of health. *During the evaluation, teachers were on strike and schools were closed. So the evaluation team was unable to observe or speak with any key informants other than one school health official. 28 Overall Conclusions on Strategies Foundation Package. Capacity building at the facility level (supply-side support) and activation of CUs (demand-side support) were shown to be key strategies to increase availability of services and contribute to key results. Specifically, integrated service delivery, referrals, and linkages between the Facilities and their Communities were shown to fill a large gap in HIV care and Treatment service availability at lower facility levels (i.e., dispensaries/level 1). While there are still significant gaps in MNCH and a need for continued inputs to maintain service quality, these components of the Foundation Package show promise to be an effective strategy in part because of the strategies’ alignment within existing government systems and structures. Results were inconclusive about the extent that APHIAplus support for OVC is contributing to reducing vulnerability because of insufficient record-keeping on project activities and beneficiaries. The project’s strategy for gender mainstreaming and prevention of GBV is not clearly articulated or implemented across activities; hence clear contribution to expected project results cannot be determined. Overall though, there are areas where the project gives direct inputs (e.g., supplies and equipment, lab support, funding of training venues). While this direct support may be necessary to meet targets in the short run, they may not be sustainable once APHIAplus closes, and may not be contributing to systems strengthening, which is the goal of the Foundations Package. In the worst-case scenario, these inputs may even create a demand that cannot be filled after project closure, or possibly even within the project timeframe equally to all project sites. Most Vulnerable Groups Package. Anecdotal evidence suggests that elements of this package were appropriate for increasing demand for services among key populations—community-level health messaging and linking to clients were shown to have positive preliminary results. Improving health systems to MARPS through the use of CHWs, and improving systems such as referral systems and defaulter tracing, helped strengthen the Community Strategy to deliver results. However, linkages to services were not found to be systematic. Also, the effectiveness of evidence-based population targeting and intervention design was found to be weak; accepted best practices such as MARP–friendly providers and facilities were rare. Economic strengthening activities showed early potential to reduce vulnerability of these populations but need a systematic approach and a firmer theory of change to demonstrate results at scale. Economic support to improve health-seeking behavior has significant theoretical merit; however, the large number of both project target populations and activities make this strategy operationally difficult to realize in a systemic way and also make attribution and control for confounders difficult. Partnerships Package. While there is anecdotal evidence of successes in partnerships within the health sector and with national-level actors (such as joint planning, development of annual work plans, technical assistance, policy development and dissemination), the intended results of this package as described in project documents was not evident during evaluation fieldwork. Two good practices for strengthening the effectiveness of the Partnership Package include a) formalizing partnerships and b) building on previously established relationships. The project does not adequately measure or document the partnerships or relationships it has developed, or the results that have come from these connections. Also, most partnerships found were with the project’s direct subgrantees, as opposed to links between the project and other funded initiatives or existing networks. Partnerships within the projects’ social determinants activities are lacking. Results such as joint planning and leveraged resources were yet to be seen (although this may be due more to timeframe than design). It is unclear whether, on the one hand, partnerships that USAID envisioned and on which the effectiveness of this strategy would be based were unavailable or, on the other hand, the project failed to make appropriate connections. 29 Recommendations Before the end of the project: • The project should conduct a rapid survey of human resources for health needs among supported facilities to prioritize and target capacity-building efforts to fill gaps. The results of the survey can be provided to Capacity and FUNZO, the national mechanisms that support this type of capacity building, to include in their next work plans as necessary. (Q2) • Support the GOK to improve patient record keeping and referral tracking by ensuring that patient forms identify the referral source and type of service on each record to facilitate the referral feedback loop. For instance, include a field for those lost to follow-up; this can be done with a simple box that patients check, asking, “Were you referred by a CHW to continue your treatment?” or something similar. • Continue to support the government and URC Assist with the rollout of KQMH at facilities. • Conduct rapid assessments of its CU activation activities, PLHIV, youth, and Result 4 interventions to understand what has been achieved during the life of the project with recommendations for future investments. This will be used as a handover tool for future activities.* QUESTION 1. TO WHAT EXTENT HAS APHIAplus NAIROBI/COAST BEEN EFFECTIVE IN CONTRIBUTING TO ACHIEVING RESULTS 3 AND 4? WERE THE KEY EXPECTED OUTCOMES ACHIEVED? IF NOT, WHY NOT? The discussion under Question 1 concentrates on indicators, performance, and appropriateness based on the PMP and project logic. Inputs and outputs presented in Question 2 will also be considered in answering Question 1. Expected Results are defined by the USAID Implementation Framework, Results 3 and 4, and key expected outcomes are to be based on Intermediate Results. Findings and Conclusions The project has reported against the set of PMP indicators approved by USAID for years 1 and 2, which include a selection of PEPFAR, Kenyan Ministry of Health, and project indicators. Indicators reported by the project do not have baseline measures. Project documents and anecdotal evidence from project staff state that baselines were to be taken from APHIA II end of project data where possible. Between years 1 and 2, the project added and dropped numerous indicators. The majority of indicators changed were project indicators (15 dropped from year 1 and 24 added in year 2); also, at the request of USAID, 7 new PEPFAR standard indicators were added. Documentation from the project explains these shifts as primarily related to changes in programming, shifts in the GOK Annual Operational Plan and attempts to improve reporting. Years 1 and 2 project indicators were designed to measure outputs, with few exceptions. There is no reported monitoring by APHIAplus of the effect of interventions on health or behavioral outcomes among the targeted population. At the same time, the APHIAplus project does not include funding for behavioral measurement. In year 3, the project developed a new M&E plan, which added an exhaustive results chain and logic model that captures outcome indicators. This addresses *A full set of questions to include in each rapid assessment is included in Annex C.2. 30 some of the noted weaknesses in the previous PMP. Yet, the project has not reported on these new indicators nor incorporated them into the PMP in the first quarterly report for year 3.* Each project document (project description, work plans, and quarterly reports) follows a different structure in terms of presenting activities and results under packages of interventions and result areas. A review of 14 subawardee SOWs showed significant difference in formatting and content of SOWs; including varied language on objectives, indicators, and deliverables. Within the subset reviewed, eight SOWs contained only an organizational profile and list of core activities; they did not include objectives, deliverables, indicators, or M&E plans. Anecdotal evidence from key project staff describe these differences in documentation as attributable to a constantly evolving project logic and as attempts to improve reporting and alignment. The evaluation team understands that, through a change in the ADS 203, the USAID/Kenya Mission is currently creating a project logframe for the OPH. Through that process, partner PMPs will become Activity M&E plans that feed into an overall strategic framework. Conclusion. The complicated structure and multiple formats of project documents (including subaward SOWs) make it difficult to fully understand the projects logic in terms of indicators and how the project plans to demonstrate results. Without a clear causal logic, it is difficult to determine APHIAplus’s contribution to key indicators, outcomes, and results. Overall lack of fidelity between project documents makes progress against work plans and achievements difficult to track. Beyond this, though the year 3 M&E approach is a very promising step forward in measuring outcomes, the team is unable to assess the projects contribution toward Results 3 and 4 based on this new approach, as it has been implemented only during the first quarter of this year and as of the writing of this report the new approach described in the year 3 M&E plan has not been reported on. The evaluation team was unable to assess APHIAplus’s influence on behavioral outcomes because of the lack of funding for behavioral measurement. As this is the last year of the project, it is unlikely that the project will be able to display the trends necessary to show true contribution beyond anecdotal evidence. The project set targets based on their work plan and considering the County Operational Plan (COP) targets set by USAID, but as the project was mandated to support the government and community (evident in work plans and project documents), needs often changed while targets remained in place (and unmet or exceeded). As the project overachieved on numerous targets set through the COP process, for year 3 the project requested to use its own targets for these indicators. Between years 1 and 2 in both Coast and Nairobi, the project increased the proportion of indicators meeting targets,† but in year 2 the project was still meeting fewer than 50 percent of indicator targets. *Full PMP data sets are available in Annex G. †To define performance against targets, the team will be guided by USAID’s annual reporting guidelines, which considers within 10 percent above or below set targets to be satisfactory performance. 31 Table 3. Percentage and Number of Targets Met Nairobi Coast YR 1(2011) YR 2(2012) YR 1(2011) YR 2 (2012) Percentage of indicators on target * 15 40 20 42 (out of total number) 10/65 29/73 14/70 33/79 Conclusion. While target setting was clearly a challenge, the project balanced the flexibility of meeting the emerging needs of the GOK and communities with the requirements of USAID. That said, there is a significant issue in that the project still met fewer than 50 percent of targets set as of year 2 and as USAID reports performance in these terms for its annual report to Washington. Partners interviewed reported receiving helpful assistance from APHIAplus with M&E and reporting. However, a review of project documentation onsite with 13 out of 26 community groups indicates that, for PEPFAR indicators that measure individual exposure to individual and/or small group level interventions (including PWP), partners did not collate documentation by client. This does not meet PEPFAR and U.S. government reporting requirements, which mandate indicator reporting by client to minimize duplicate reporting of individuals receiving service. Conclusion. There is lack of documentary evidence that each individual receiving prevention services received a defined package of services as per the requisite PEPFAR indicator definitions. *Ibid. 32 APHIAplus Contribution to Outcome-Level Results. The indicators and data presented below are selected for their ability to demonstrate (however roughly) project contribution toward key indicators, outcomes, and results. To demonstrate any changes in the APHIAplus contribution over time, the evaluation team has used data from the District Health Information System (DHIS) to provide denominators for counties.*The project is targeting a subset of each region. Overall the project supports 512 facilities out of 1,040 facilities in both regions. So though contribution is not expected to be 100 percent, ideally the percentage would hold steady or increase over time as the project refines its interventions and reaches more beneficiaries. Result 3. Increased Use of Quality Health Services, Products, and Information. As described in project documents, these indicators are influenced at project supported facilities through providing capacity building as well as strengthening the Community Unit structures (including the increased use and capacity of CHWs). Under IR 3.1, figure 1 illustrates a decrease in the Percentage of HIV–positive pregnant women who received ARVs to reduce the risk of mother-to-child transmission for Nairobi and Coast by the project (out of the total HIV positive pregnant women for each region).† As presented in Question 2, providers described challenges in enrolling women consistently into PMTCT, especially from MNCH clinics.‡ *In conducting these analyses, the team found some inconsistencies within the DHIS, where data were potentially incomplete or potentially of low quality. The team has made a determination here of where the data can still be reported. †In Nairobi the project supports 78 out of 225 facilities (35 percent) for this intervention. In Coast the project supports 322 out of 388 (83 percent). ‡The denominator is retrieved from the DHIS for the two counties. The DHIS data were incomplete for this indicator in 2011, so the data are displayed only for 2012 and the first quarter of 2013. 33 Figure 1. Percent of HIV–Positive Pregnant Women Who Received Antiretroviral Drugs to Reduce the Risk of Mother-to-Child Transmission Currently, the project reports on the indicator “number of infants tested for HIV at 6 weeks,” which is considered an indicator of the quality of a PMTCT program if reported as a percentage of all infants born to HIV infected mothers. Figure 2 displays a rough percentage using project-reported data.* There has been a slight increase over the life of the project in the proportion of infants tested at 6 weeks overall for both Nairobi (32 percent to 44 percent) and Coast (22 percent to 26 percent). This is corroborated by data presented in Question 2, where providers at all sites felt the project has strengthened overall EID services, interviews in MNCH clinics cited losing patients to follow-up in the management of HIV–exposed and confirmed positive infants in pediatric HIV care and treatment to be a constant issue. *The indicator “Number of HIV–positive pregnant women who received antiretrovirals to reduce the risk of mother-to-child transmission” (a PEPFAR indicator reported on by the project) is being used as proxy for the “Number of infants born to HIV– positive women.” 34 Figure 2. Percentage of Infants Tested for HIV at 6 Weeks Under IR 3.1, in year 2, the project introduced a new treatment indicator, “percent of adults and children known to be alive and on treatment 12 months after initiation of antiretroviral.” This is a PEPFAR Next Generation Outcome Indicator, which measures program quality and client retention.* Figure 3 illustrates little overall change in project performance and shows that the project has consistently underachieved on its target of 95 percent.† Figure 3. Percentage of Adults and Children Known to Be Alive and on Treatment 12 Months After Initiation of Antiretroviral Drugs *PEPFAR. 2013. Next Generation Indicator Reference Guide. †The project PMP identifies 95 percent as the target for this indicator during each project period. 35 Figure 4 presents two project indicators used to demonstrate the level of TB/HIV integrations. Performance has been more or less consistent over the life of the project in Nairobi, while in Coast the project underperformed in the first year but shows consistent performance for 2012 and the first quarter of 2013. Figure 4. Measure of Tuberculosis/HIV Integration Figures 5 and 6 focus on two MNCH indicators for projects activties, which as presented in Question 2 showed significant gaps at project supported sites. The project measures the number of pregnant women attending at least four ANC visits. This indicator falls under IR 3.3: Increased adoption of healthy behaviors in the project’s PMP over all three years of implementation. Figure 5 illustrates slight overall improvement in the projects contribution toward the percentage of women attending at least four ANC visits, but with a downward trend in the first quarter of this year for both Nairobi and Coast.* Increasing the percentage of children under one who are fully immunized falls under IR 3.1: Increased availability of an integrated package of quality high-impact interventions at community and facility level. Increasing the percentage of children under 1 who are fully immunized falls under IR 3.1 of the project’s PMP for all three years of implementation. Figure 6 depicts a spike in year 2 and a dip in the start of year 3 for the percentage of under-1-year-olds fully immunized in sites supported by APHIAplus out of the total number of under-1-year-olds fully immunized in Nairobi and Coast counties.† Based on information from project documents, these indicators would be influenced at project-supported sites through improved provider capacity as well as strengthening the community unit structure. *APHIAplus supports 78 out of 225 sites (35 percent) that reported on this indicator in Nairobi and 322 out 388 sites (83 percent) in Coast. †In Nairobi, APHIAplus supported 68 out of 318 sites reporting on this indicator (21 percent) and in Coast supported 341 out of 404 (84 percent). It should be noted as well that in Nairobi the project reported on this indicator only in the last quarter of 2011 because of project startup issues and that this can explain low performance in 2011. 36 Figure 5. Percentage of Pregnant Women Who Receive at Least Four ANC Visits, APHIAplus Contribution Figure 6. Under 1 Fully Immunized, Percentage of APHIAplus Contribution As described in Question 2, there are many other actors working with similar mandates in the same counties and subcounties as APHIAplus. This can lead to potential misattribution of results to one project or another. For example, the project reports on a number of indicators related to RH/FP for their supported facilities. Observations and interviews at project support sites highlighted that the Tupange Project (a USAID–funded RH/FP project implemented by JHPIEGO) is active in those same sites and is also providing support for the provision of RH/FP services. 37 Conclusion. Using project and DHIS data to determine contributions of APHIAplus and calculate potential outcome indicators leaves an inconclusive picture of the projects actual contribution. The data presented in figures 1 and 2, falling under IR 3.1, illustrate success in improving early infant testing but potential challenges in enrolling women in PMTCT (and providing ARVs), which corroborates findings and conclusions presented in Question 2 regarding the project’s challenges with fully integrating PMTCT, ANC, and MNCH. Figures 3 and 4 demonstrate that the project has had some success across project years in contributing to IR 3.1 in terms of integrating HIV and TB services and in retaining clients on HIV treatment in line with KEPH guidelines. This is corroborated by evidence presented in Question 2 regarding support for integration of TB/HIV as well as overall increased capacity to deliver HIV and TB services. However, as described in Question 2, limited documentation of CHW activities reduces the ability to draw conclusions about projects contribution to increasing demand (IR 3.2) as well as increased provision of services at the community level (IR 3.1). Figures 5 and 6 demonstrate just two of several indicators for the MNCH domain (spanning IR 3.1 and 3.3) but show downward trends, which corroborate anecdotal evidence of the projects inconsistent support in this area described under Question 2. Additionally, the two indicators reported on by the project that have the potential to demonstrate improved service quality in PMTCT (figure 2) and HIV care and treatment (figure 3) have shown slight improvement but limited achievement. But considering the limited time of implementation and collection of data on these indicators, this level of performance is not unexpected. The limited data presented by the project and significant changes in indicators from year to year (hindering comparison over project years) limit conclusions the team can draw about potential contribution to Result 3 overall. As well, direct attribution of APHIAplus inputs to outcomes or results is difficult, given other actors working with similar mandates in the same facilities. Result 4. Social Determinants of Health Addressed to Improve the Well-Being of Targeted Communities and Populations. The project does not report on any outcome indicators under this result area. There are no appropriate external data available to the evaluation team with which to draw out proxy measures or use to demonstrate changes over time because of project inputs. As described above, the year 3 M&E plan does introduce outcome indicators for social determinants activities, but these indicators are not reported on by the project as of the first quarter of this year nor are they included in the current PMP. Annex H presents an illustrative selection of indicators under Result 4 as of the year 3 M&E Plan. As presented in Question 2, there is anecdotal evidence of project activities reaching beneficiaries. Conclusion. The APHIAplus approach to addressing the social determinants of health, as described in project documents, does not show a clear relationship between indicators or strategies and the activities of which these consist. In determining the projects contribution to outcomes under Result 4, it is possible to say only that the project helped facilitate reaching an increasing number of beneficiaries (as described in Question 2). Without data at the household level or behavioral outcomes reported, it is not possible to state the extent to which the project contributed to addressing the social determinants for health or reducing vulnerability. Recommendations Before the end of the project: • APHIAplus should report against the results chain presented in the year 3 M&E plan, using a selection of new indicators approved by USAID and including all F–indicators that need to be reported to USAID/Washington and Congress. This will enable USAID to determine to some degree APHIAplus’s contribution to Results 3 and 4. 38 PROMISING PRACTICE APHIAplus evolved with the international HIV and public health domain by first facilitating the adaptation of the PEPFAR/WHO Elimination of Mother-to-Child Transmission (EMTCT) guidelines into the national Prevention of Mother-to-Child Transmission guidelines through its involvement in the technical working group. The project ensured availability of the new EMTCT guidelines in health facilities (as observed at all 17 facilities visited) and facilitated Continuous Medical Education sessions for facility staff to update the facilities on the guideline changes. • USAID should conduct a Data Quality Assessment of APHIAplus Nairobi/Coast to address reporting issues specifically related to PEPFAR indicators that measure individual exposure to individual and/or small group level interventions. QUESTION 3. TO WHAT EXTENT WERE THE PROJECT’S ANNUAL WORK PLANS AND STRATEGIES THEREIN INFORMED BY NEW PROGRAMMATIC EVIDENCE IN THE HEALTH SECTOR? For the purpose of this evaluation, “programmatic evidence” will be defined as national policies and guidelines* and internal program-generated data.† Findings and Conclusions Alignment to Policies and Guidelines. At the Facility Level. As described in program documents, APHIAplus has been involved in the development of health policy at the national level as part of the Partnerships Package. This was corroborated through interviews with national bodies, such as National AIDS and STI Control Program (NASCOP), the National AIDS Control Council and the National Council for Population and Development, which revealed that APHIAplus provided important technical inputs into the development of national guidelines and policies such as the new ART guidelines, EMTCT guidelines, VMMC guidelines, the 2012 Population Policy, HRH strategy and 2012 RH Integration Strategy. This was made possible by their involvement in national technical working groups. To ensure better alignment to KEPH and other national guidelines, the project has worked with DHMTs in their annual work planning process, determining priority activities in accordance with the existing KEPH national guidelines.‡ Additionally, the project provided capacity building to staff and health management teams, which included updates and refreshers on national guidelines. APHIAplus staff conducted monitoring visits and supported DHMTs to conduct supportive supervision visits with the goal of ensuring standardization and consistency of staff in meeting these guidelines. Further, interviews with health facility staff verified that APHIAplus supported the printing and distribution of these policy documents and guidelines.§ As described in Question 2, interviews and observations consistently showed availability and adherence to appropriate guidelines across all facilities visited. While observations in 13 out of 17 MNCH clinics revealed that KEPH guideline documents were available, in 10 of these clinics, Integrated Management of Childhood Illnesses (IMCI) guidelines (an important component of MNCH) were outdated or absent. Additionally, as per project documents, APHIAplus has *International guidelines will be used where national guidelines are unavailable. Per the methodology, the threshold for the issue of guidelines/policies to be adhered to in work plans is July 2012. †As agreed during initial briefing with USAID. ‡APHIAplus staff cite additional ways in which work plans and strategies are informed by new programmatic evidence to include national-level forums, policy dissemination (described here), as well as internal guidance from USAID. §Additional discussion is included in Question 2. 39 focused much of its support at the community level and working through supported CUs. As described in Question 2, the MOH does not currently have resources to activate CUs and key informants in the health sector described APHIAplus support to the Community Strategy as essential to its functioning. Conclusion. The project’s emphasis on support to the GOK in health planning, policy development, and dissemination has enabled APHIAplus–supported GOK health facilities and CUs to align to most health sector policies and guidelines. However, with noted gaps in MNCH—IMCI particularly—guideline availability is still a concern and program challenge. Additionally, the project’s involvement in providing technical inputs and support to national guidelines development and dissemination has enhanced the GOK’s capacity to institutionalize new evidence promptly and allowed new policies to be incorporated into the project’s work plans with its support to DHMT planning processes, as demonstrated by the example of the absorption of the new EMTCT guidelines. At the Community Level. Based on a checklist of interventions against appropriate PEPFAR and national guidelines, * the 14 partner organizations interviewed by the evaluation team implemented activities that met national/international guidance for their target audience in many areas, though gaps were still observed as described below. For key population activities (focused on MSM, FSW, PWUD, PLHIV) main areas of alignment observed across 14 groups surveyed included consistent condom distribution, though a significant gap was in the distribution of lubricants (for seven key population group visited all sites described issues with stock-outs). All groups visited provided community-based outreach conducted as per appropriate guidelines (in appropriate places†) and linked to services in facilities or drop-in service centers (DISCs) for STI prevention, screening and treatment, HTC, and ART. Observation revealed gaps in referrals or linkages to education/counseling for FP and drug use in all groups. In alignment with PEPFAR guidelines, IEC materials for HIV prevention were consistently available at all 17 groups, though the 7 groups that targeted key population subsets (FSW, MSM, or PWUD) did not have IEC materials that targeted issues specific to those groups. Current IEC materials are sourced from UNFPA, NASCOP, UNODC, and NACADA. NASCOP leads the design of IEC materials for HIV prevention in Kenya to ensure consistency of messaging. Additionally, the evaluation team observed that while the seven groups that targeted key population subsets (FSW, MSM, or PWUD) had activities targeting clients through outreach efforts (for FSWs and MSWs), the purposeful targeting of sexual partners (with a focus on discordant couples and emotional partners as per PEPFAR guidelines) was not observed in any group, with the exception of the promising practice described in the discussion under Question 2. Five of the seven groups working with PLHIV did target sexual partners through hosting support groups for discordant couples, counseling, and other varying types of education. In reviewing APHIAplus youth activities in comparison with PEPFAR and national guidance,‡ the evaluation team found that *Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR. Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR. Comprehensive HIV Prevention for People Who Inject Drugs, Revised Guidance, issued in July 2011 by PEPFAR. National Guidelines for HIV/STI Programs for Sex Workers issued in September 2010 by NASCOP. †For example, bars and sex dens for MSM and FSWs. ‡Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. 40 • All four surveyed youth groups used messaging that reinforced wider social and behavioral change and communication messages (e.g., G–Pange, Shuga) in line with national guidelines. • Youth groups employing ETL tactics used session guides for messaging in line with PEPFAR guidelines, but review of a convenient sample of session guides indicate that some themes are incomplete (e.g., VMMC), only one set included PWP messages, and there were no sessions/ themes on drugs and alcohol. • Although session themes might consist of 3 to 11 sessions, the project measured participation in 2 or 3 sessions as the minimum standard. • The two surveyed PLHIV youth support groups had programs tailored to age (children, preteens, teens, and youth) with age-appropriate activities and messages, demonstrating alignment with PEPFAR guidance. While subpartners described training in M&E (as described under Question 1), all subpartners stated during interviews that they would have liked to have received more technical assistance in HIV/AIDS and key populations programming. One subpartner responsible for a significant amount of work with key populations cited not having received any technical training from the project.* Conclusion. APHIAplus key populations (FSWs, MSM/MSWs, PWUD, PLHIV, and youth) partners implement many activities that are aligned to national guidelines and international guidelines, but there are still gaps in appropriate targeting of interventions and provision of necessary services and commodities, which limit the effectiveness of project-supported activities and inputs described in Question 2. Of the six surveyed OVC groups, all offered some services that aligned to national and international standards† in their package of services. Under health, all six groups provided parenting skills and support groups, home visits to at-risk and affected families, and disclosure and bereavement support that aligned to guidance. But gaps were observed with regard to RH services, HIV prevention, and RH services for adolescent OVC, including limited linkages to RH/FP services.‡ All six groups reported strong linkages to facilities for child health interventions such as nutrition-related services, vitamin A supplements, and deworming, and all six supported home visits by CHWs. For PLHIV OVC, there were support groups and discussion of HIV-prevention, adherence support, and linkages to care. Under the protection element, all six groups provided linkages in alignment with guidance—linking groups to police, county administration, Children’s Department, and Area Advisory Councils. While all six groups assisted caregivers and OVC in obtaining birth certificates, only one group assisted on inheritance issues, three conducted some trainings and community outreach on GBV, and only two provided paralegal follow-up, police referrals, and post-exposure prophylaxis for cases of abuse. Within the economic strengthening element, all groups offered support for the creation of VSLA/SILC groups or IGAs among caregivers and had received training from APHIAplus. Four out of six groups had started or continued VSLA/SILCs, and five of six were observed to have started or continued IGAs. Under the food security element no group was observed to provide linkages to farmers’ groups and cooperatives to scale up food productive for caregivers. Under the education element five out of six groups work with the Ministry of Education to *A short discussion of support provided in organizational development through Fanikisha will be described under Question 4. †Guidance for Orphans and Vulnerable Children Programming, issued in July 2012 by PEPFAR. Framework for the National Child Protection System, issued in November 2011 by the National Council for Children’s Services. ‡Only four offered on menstruation-focused counseling; one addressed other RH/HIV issues including MCP, early pregnancy, and family planning; and one focused HIV prevention only to OVC living with HIV. 41 ensure all school-age OVC stay in school. Within the safe shelter and WASH element, five groups of six provided hygiene education. In line with current OVC guidance in Kenya, APHIAplus has supported URC ASSIST in rolling out Quality Improvement (QI) teams focused on tracking OVC in communities and improved reporting.* The evaluation team was able to meet with one QI team leader in Kilifi who cited training from URC ASSIST and logistical support from APHIAplus since 2012. All five GOK Children’s Officers interviewed by the team cited continued need for improved M&E within OVC programming and tracking, as well as foreseen challenges with continuing QI teams without donor support. No surveyed OVC group (out of six) conducted a census of households within CUs to establish a clear baseline of need. Only one OVC CBO reported using the 2011 Child Status Index to determine whether a child has improved as a result of support. There was no documentation provided of the Child Status Index and how it is used to measure progress or prioritize resources. Additionally, documentation provided by APHIAplus does not include graduation criteria outlined in PEPFAR guidance to stage/prioritize households and assistance inputs. Conclusion. A review of project documents and KIIs with consortium partners and OVC implementing partners indicates that APHIAplus OVC programming has gaps in aligning to PEPFAR and national standards in the following areas: • A comprehensive household assessment process of all children within a defined geographic area against graduated definitions of vulnerability • Records by child that fully capture the nature of direct and indirect support over time against identified needs • The application of the Child Status Index to measure progress over time Internal Project Quality Improvement and Quality Assurance Activities. Based on a review of project work plans (including the initial project description), Continuous Quality Improvement (CQI) is cited to be central to APHIAplus, in which CQI processes and elements will be integrated into supervision and data collection tools, so that project implementation can be assessed and addressed on a continuous basis at all levels and for all interventions. Project documents and statements from APHIAplus staff cite examples of the projects’ own internal quality-assurance model, tools, and activities in areas where projects are being implemented outside of the healthcare system. Some examples are project team meetings to discuss data trends and issues and to make programmatic recommendations, training needs assessments, and self-assessments for various beneficiaries. The team was unable to observe any of these activities or validate these processes as no documentation was provided. Conclusion. The extent to which internal QA/QI activities have been conducted by the project or used to improve programming was not clear during fieldwork. As such the team cannot make any definitive conclusions regarding the projects’ use of internal program data. Internal QA/QI would help the project identify issues in implementation early on and make shifts in programming on an as needed basis before small issues become large issues, especially in community activities supported by the project outside of the healthcare system. *Ministry of Gender, Children, and Social Development. 2012. Minimum Service Standards for Quality Improvement of Orphans and Vulnerable Children Programs. Nairobi, Kenya. 42 Recommendations For the rest of the project: • Continue to support the development, rollout, and adherence to national guidelines, with continued targeted capacity building to DHMTs and CUs in evidence-based planning with stakeholders. • Provide targeted technical assistance to subgrantee partners implementing activities under APHIAplus to support improved alignment to national and international guidelines. QUESTION 4. TO WHAT EXTENT WAS THE APHIAplus MODEL APPROPRIATE FOR ACHIEVING KEY OUTCOMES UNDER RESULTS 3 AND 4 OF THE IMPLEMENTATION FRAMEWORK?* Each component of the model as defined in the project RFA is dealt with separately below. Findings and Conclusions Geographic Zoning. The APHIAplus model is predicated on geographic zoning based on unique challenges that included urban slums and marginalized, poor, underserved, and most-at-risk populations. Nairobi and Coast were placed together in the original USAID design because of similar HIV epidemic patterns and urbanization.† The APHIAplus project description discusses that the project will tailor interventions to these similarities as well as address differences in other health patterns, specifically variations in malaria prevalence, vaccination coverage, child stunting, and deliveries by skilled birth attendants.‡ APHIAplus catered for some differences by zonal targeting (Malaria, VMMC, IGAs) as seen in their PMP and indicator targets; however, there is little project documentation of strategic targeting programmatic inputs or interventions for these areas in the two regions. Beyond issues considered in the RFA, project description or work plans, Nairobi is experiencing very high rural–urban migration rates, rapid expansion of informal settlements, and facility access and usage challenges based on historical and long-out-of-date urban planning patterns and growth management. These types of issues were raised by key informants as particular challenges in Nairobi. Although Coast is also experiencing urban growth, it is not to the same scale as Nairobi, and the region has wide swathes of low-density rural populations and traditional cultures.§ Project documents reviewed do not address these more emergent issues or describe how these regional differences may have limited the projects ability to meet targets. During interviews, APHIAplus staff cited challenges of larger distances, both between Nairobi and Coast (as well as within the Coast region itself), which made project management and coordination more challenging and often less efficient. *Recommendations for Question 4 are captured for future design considerations in Annex D. †RFA no. 623–10–000009, USAID/Kenya APHIAplus, Health Service Delivery Projects, 2010. ‡APHIAplus project description, 2010. §Republic of Kenya, Ministry of Planning and National Development. 2010. Mombasa District Strategic Plan 2005–10. Demographic Research Circular migration patterns and determinants in Nairobi slum settlements. 43 Conclusion. While the idea of geographic zoning based on similarities was sound in terms of programming, the particular grouping of Nairobi and Coast by USAID in the RFA was not found to be appropriate. Though similarities in HIV epidemic patterns and urbanization do exist, there are significant differences that do not seem to have been identified or taken into account in placing the two regions together. In terms of APHIAplus Nairobi/Coast, grouping Nairobi and Coast together may have hindered appropriate strategic planning to deal with emerging issues in the interest of keeping the project a cohesive whole. It remains to be seen whether geographic zoning has been successful in other APHIAplus projects. Vast geographic distances between and within the two regions may have posed unforeseen challenges in implementation and oversight; the challenge of distance may have contributed to a reduced ability to meet targets or achieve expected results (as described under Questions 1 and 2). Integrated Approach for Poor, Marginalized, and Vulnerable Populations. The integrated approach uses a technical focus that helps address the underlying health care issues of people living in urban slums and marginalized, poor, underserved, and most-at-risk populations in the geographic zones, specifically focusing on OVC and MARPS among other vulnerable populations through an integrated approach to high-quality technical assistance to the health system and communities. As described in project documents, the project planned to use three strategies (the Foundation Package, the Most Vulnerable Groups Package, and the Partnerships Package) to implement interventions to strengthen the health care system (at the community and facility level) and support improved use of services by vulnerable populations (through demand creation and addressing barriers to access related to the social determinants of health). As described in Question 2, through the Foundation Package the project was able to provide technical assistance and support to improve service delivery at the facility level and use the Community Strategy to target members of the general population and increase demand for health care through trained CHWs. Through the Most Vulnerable Groups Package, the project targeted MARPs, PLHIV, OVC, and youth with packages of defined services within communities and facilities, as well as addressing the social determinants of health through additional economic strengthening activities for a wider net of vulnerable groups (e.g., women and rural households). While interviews and observations at all facilities anecdotally showed that project interventions improved availability of key services and increased demand and use by the general population, use of services by key populations was still reportedly low as described in Question 2. Interviews and observations related to project activities implemented at the community level highlighted success stories and anecdotal evidence of increased use of community health services for key populations (DISCs, support groups, etc.) as well as initiation of IGAs or VSLAs. There was limited anecdotal evidence linking these successes to increased use of facility health services or reduced vulnerability of key populations. Many APHIAplus staff interviewed described feeling “stretched too thin”—in terms of implementation of the wide range of activities described in their work plans—to meet the expectations of USAID as related to project inputs and out. Conclusion. The integrated model of interventions described in the APHIAplus project documents is theoretically sound, but with the current resources available and the scale at which APHIAplus is being implemented, this model is inappropriate and unable to achieve its expected results. 44 National Mechanisms. The APHIAplus model is predicated on the assumption that the APHIA implementing partners will receive timely and high-quality technical assistance/complementary programming from national-level technical assistance partners. Project inception and planning documents describe maximizing partnerships and linkages with national TA partners to fully meet their mandate. These mechanisms work at the national level in areas such as training and staffing, health management information systems and other information management, commodity supply, and laboratory assistance. The majority of stakeholders interviewed said they felt a two-tiered approach was appropriate, though four out of six national TA partners interviewed have only recently started to engage with APHIAplus or begun implementation of their own activities. All national TA partners and APHIAplus staff cited both successes of the partnerships as well as challenges. Linkages between APHIAplus and national TA partners such as KEMSA and Capacity have enabled APHIAplus to facilitate supply of RH/FP and HIV–related commodities to all 17 clinics as required and ensure placement of RH/FP counselors, nurses, and other cadres of staff in a few visited clinics. Interviews with PMTCT service providers revealed that APHIAplus directly facilitated the supply of rapid test kits for PMTCT between the KEMSA and the corresponding health facilities to the point that clinical service providers believed that APHIAplus were the ones procuring these commodities. Additionally, interviews with key stakeholders confirmed that APHIAplus is supporting several national TA partners to better manage and use data, including collaborating with AfyaInfo in DHIS version 2.0 use, MEASURE Evaluation in the Community-Based Health Information System, and Futures Group in the Electronic Medical Records (IQ care) pilots.* All national TA partners interviewed cited challenges with unclear differentiation of mandates and the need for more formal agreements between all APHIAplus projects and individual national TA partners. Only Futures and APHIAplus HCM (two national TA partners) described joint planning for activities where mandates overlap or are meant to build on each other. APHIAplus staff cited one example of overlapping mandates as being requested to provide data-collection tools to facilities (which was considered to be the role of AfyaInfo). While the project has responded to this gap, it has created unexpected budgetary implications. Fanikisha, the national TA mechanism that provides organizational development support and training to CSOs, has just begun to conduct wider trainings to groups of CSOs and has not yet worked in either the Coast or Nairobi regions. Evaluation team members were told during these interviews that while APHIAplus projects have a position on Fanikisha’s committee of advisors responsible for selecting CSOs for training, the projects have yet to sign MOUs delineating individual roles and responsibilities. Fanikisha has been working with 10 national CSOs, which they fund for service delivery, a few of these fall within the Nairobi/Coast project—including WOFAK, with which the evaluation team met in Mombasa—but a complete list was not provided to the evaluation team. Discussions with the APHIAplus health service delivery team in Mombasa and seven DHMTs revealed that there are still challenges in communicating training needs between APHIAplus and FUNZO (the national TA partner that coordinates long-term substantive trainings for healthcare workers) and responsiveness of FUNZO to deliver requested trainings. Additionally, all national TA partners and APHIAplus staff described issues of timing, where mechanisms were supposed to be in place to support each other but were not. Examples include FUNZO starting late to support training of facility staff and *Key stakeholders refers to individuals interviewed at facilities visited, APHIAplus staff, staff at national TA mechanisms, and selected GOK health sector informants. 45 the Leadership, Management and Governance Project, which is supposed to provide leadership and governance training to Health Management Teams (DHMTs and at facilities) but started only a year ago. All national TA partners suggested that a national coordinating mechanism was needed to improve communication and coordination between the national TA partners and all APHIAplus projects. Conclusion. Using the two-tiered approach with APHIAplus supported by national TA partners (and vice versa) is an appropriate model, but significant challenges to effective implementation include coordinated timing of program implementation, planning, and accountability. It is difficult to assess whether the Fanikisha project will achieve expected results, as the subset of CSOs targeted is so small and the wider training has yet to take place in Nairobi or Coast. When national mechanisms were not able to deliver as planned, APHIAplus employed short-term strategies to fill gaps. Management Structure. APHIAplus model uses an implementing consortium as its management model. APHIAplus staff conveyed in interviews that the consortium partners (and staff) provided specific and appropriate expertise, which resulted in synergistic innovations and relationships. Anecdotally, the consortium structure allowed sharing of resources, support of one another in the field (e.g., shared vehicles, supportive supervision) and co-location in project offices for prime contractors and the majority of subcontractors. During fieldwork, beneficiaries and stakeholders interviewed referenced APHIAplus but did not identify individual consortium partners when discussing support from the project, hinting at external perceptions of APHIAplus as one organization. Based on interviews with consortium partners, there are conflicting opinions on how well coordination is working, * but there is anecdotal evidence that the project has implemented recommendations from the internal midterm review, which cited the need to improve coordination through restructuring project teams and planning by cluster. During interviews with APHIAplus staff, all cited some initial challenges in coordination and consortium management but say that now the project holds weekly meetings with the entire staff (using video conferencing to include offices in the Coast), which has improved coordination. Interviews with consortium partners also revealed that distance posed challenges in management of the two regions. Conclusion. While the consortium management structure allows for projects to bring on partners with specific expertise needed, the complexity of this type of management model compounded issues in the early stages of the project in terms of coordination. Additionally, while the consortium management structure is appropriate in terms of the technical needs for a project of this type, it will be important to consider the time it takes for a consortium such as this to work out the kinks and begin to function well together. Building on Prior USAID Investments. The model assumes that APHIAplus builds on the successful interventions of APHIA II, thus maximizing USAID’s investments. As reported by APHIAplus, no comprehensive list or specific direction was provided by USAID on which activities to carry over.† *Initial challenges cited by APHIAplus staff included budget transparency issues, issues of communication between prime contractor and subcontractors (including some staff suggesting the need for a limit on the number of partners in the consortium), challenges of bringing on the right partner for the task and challenges of keeping staff. †Some items were passed over from APHIA II, Coast, including an OVC database and information about subgrantees. As APHIA II in Nairobi was managed by Pathfinder, there was not a “handover” as such. 46 While some promising practices from APHIA II Coast, as identified by Family Health International, were not continued (e.g., workplace programs, the CVD and HIV program), the project did build on clinical services introduced or strengthened under APHIA II. In Nairobi, APHIAplus built on relationships with subgrantees, such as KGGA, and continued to strengthen clinical services provided under APHIA II. At the facility level, the transition from APHIA II to APHIAplus was perceived to be seamless by facility staff and health sector key informants interviewed, with few gaps in HIV services reported. All seven DHMTs reported gaps in commodity availability and provision of stipends to CHWs during the transition, but there was consistent support for HTC and Care and Treatment. APHIAplus has continued to work with some but not all community organizations developed under APHIA II and other projects (e.g., AED CAP). Additionally, there were reported disruptions in subgrantee-provided services to MARPs, youth, and other key population groups. Across the project, subgrantees described gaps in funding of 6 to 20 months. For example, one subgrantee (who implemented FSW and MSM DISCs in Coast) reported a one-year gap between the close of APHIA II subawards and receiving the new APHIAplus subaward. During this time the subgrantee had to let go of staff, scale down activities, and suspend peer educators and condom distribution. Another example is the nine-month gap in support the Kenyatta National Hospital Gender-Based Violence Recovery Center. Additionally, while USAID cited forums for sharing of best practices, the evaluation team was unable to corroborate this during fieldwork. Additionally, as cited in interviews with subgrantees, several had only recently started project-supported activities. For example, Mentor Mothers groups had both started in May 2013, and FIDA had just started APHIAplus supported activities in April 2013. APHIAplus subgrantee contracts will end in September 2013, when the project begins its closeout process. Conclusion. APHIAplus was able to build on investments from APHIA II at the health facility level, where only minor disruptions were noted. There were more significant gaps at the community level. APHIAplus did not build on all prior USAID investments in Nairobi/Coast, and as no specific direction was provided by USAID, it is unclear to what extent activities were dropped because of changes in APHIAplus’s mandate. Additionally, it seems there was limited sharing of best practices and lessons learned from previous investments in other regions in Kenya. It is worrying that many project-supported subgrantees have just started activities under their subawards, leaving them only five or six months to conduct project-funded activities before APHIAplus closeout begins and subawards end. This poses a problem for future projects to build on USAID’s investments under APHIAplus Nairobi Coast. Overall Conclusion While the APHIAplus model seems theoretically sound, significant operational issues and challenges faced during implementation have limited its effectiveness. 47 QUESTION 5. WHAT SUSTAINABLE ACTIVITIES IN SERVICE DELIVERY AND HEALTHCARE SYSTEMS WERE ESTABLISHED AND/OR STRENGTHENED AT THE DISTRICT HEALTH FACILITY AND/OR COMMUNITY LEVEL?* To answer this question, the evaluation team used the definition of sustainability as captured in the USAID Policy Framework 2011–15 (p. 32), which states: “Sustainability is achieved when host country partners and beneficiaries are empowered to take ownership of development processes, including financing and maintaining project results and impacts beyond the life of the USAID project.” Findings and Conclusions Support to Healthcare Systems and Facilities to Ensure Quality Service Delivery. Key informants at the national and facility levels revealed that APHIAplus has supported strengthening of nationally adopted Health Information Systems (HIS) in Kenya. The project has done this by collaborating with AfyaInfo, Measure Evaluation, and Futures Group International to implement and/or pilot (build capacity and roll out) sustainable technological innovations, such as District Health Information System 2.0 (for routine HIS), Community-Based Health Information System, and IQ Care Electronic Medical Records (Patient Management), respectively. Conclusion. Supporting the government and national partners to implement nationally adopted health information systems that would improve the GOK’s capacity to collect, manage, retrieve, and report data more efficiently is a pathway to strengthening the overall health system of the country. Interviews with all health facility teams and DHMTs revealed that APHIAplus established and supported a process of building capacity of facility staff through a systematic cascade of training activities. Staff who attend formal short-term trainings organized by APHIAplus staff are in turn responsible for providing these trainings to their colleagues through CME sessions or OJTs. Many of the health facility staff commented only on the duration of the CMEs—the fact that they were only three days long—thus truncating some topics in the schedule. At all sites visited staff who had received training through the cascade structure appeared knowledgeable on training topic.† Additional support from APHIAplus in the rollout of KQMH and policy development and dissemination has helped ensure that quality services are delivered (as described in Questions 2 and 3). Interviews with all seven DHMTs revealed that the project supported inculcating the process of supportive supervision within the government’s system. Additionally, interviews revealed that DHMT members felt facilitation of and participation in supportive supervision visits has strengthened their ability to monitor and work toward ensuring the quality of service delivery activities within the GOK system by identifying and addressing needs as they arise. Interviews with 15 (out of 21) key informants in the health sector described feeling ownership and a clear understanding of the importance of supportive supervision and other functions of the DHMTs, citing that activities would continue when APHIAplus ended as long as resources were available. Among *Recommendations for Question 5 are captured for future design considerations in Annex D. †Notably, the evaluation did not include a formal knowledge assessment of individual staff. 48 stakeholders, there was limited understanding of what the healthcare budgets would look like for counties after devolution has taken place. Conclusion. The project’s approach to capacity building meets best practices in terms of using the cascade system for dissemination of information, and the project has strengthened government systems through its input to the supportive supervision process. Anecdotally, these methods have helped improve quality service delivery at the facility level and staff trained through the system seemed knowledgeable on relevant topics. It is unclear how the government will sustain these activities without donor support, but (anecdotal evidence suggests) ownership has increased. Community Strategy. As observed by the evaluation team, the project has assisted the government in implementing the Community Strategy, by supporting the establishment and empowerment of functional community units. Also, anecdotal evidence reveals that the implementation of the community strategy has strengthened community health facility linkage, thereby increasing access in the long term to essential health services by the general population (as described in Question 2). At the same time, activation is dependent on APHIAplus financial support, indicating potential issues after project closure. 49 Conclusion. APHIAplus support for the implementation of the Community Strategy has assisted the government in institutionalizing community units as part of its healthcare system and revitalized the demand for essential health services in catchment areas visited by the evaluation team. Although APHIAplus has helped CUs become potentially more sustainable through the use of IGAs and VSLAs (as described in Question 2,) there is a real risk that the achievements of semifunctional or fully functional CUs will disappear once donor support is finished without full incorporation into the healthcare system. Overall Conclusion In general, because of the project’s emphasis on working within existing government and community￾level healthcare systems, activities have the potential to improve and strengthen local systems and structures. While systems and structures may not be sustainable as they now stand, there is anecdotal evidence of increased ownership at the government and community levels for these systems and structures. 50 LESSONS LEARNED 1. Establishing the purpose and objectives of each strategy or ‘package’ of interventions is closely linked to successful implementation. Where there was little or no clarity of purpose or clearly articulated strategy, implementing partners and other stakeholders struggle to deliver effective interventions. 2. Changing track midstream, in terms of work planning, PMPs and activities, must be accompanied by an attendant monitoring mechanism to ensure that progress can still be measured; it is not possible to identify what progress has been made when indicators are inconsistent across the implementation period. 3. Applying standard monitoring and reporting mechanisms will enable the donor to pick up areas of concern, or inconsistencies between strategy and activity, at a point sufficiently early to intervene. 4. In order to accurately track project outcomes and results, clear baseline assessments must be done to provide a ‘starting point’ for key indicators. 5. Clear direction during project handover is essential to ensure key interventions are continued and successes built upon in successive follow-on activities. 6. A clear sustainability initiative built into the project design at the inception phase, especially as relates to sub￾grantee activities, will help to focus partners on ensuring the longevity of the activities. Strengthening record keeping, developing strong partnerships and identifying possible funding sources for the future are all critical here, but they also need to be linked to clearly defined activities to avoid ambiguity or possible failure to implement. 7. Utilizing best practices and aligning interventions to current international/national guidance is essential for achieving expected results and appropriate scale on interventions. 8. Formalized sharing between projects that engage similar populations and focus on similar health issues is essential to build on context-appropriate best practices for public health interventions. Page left intentionally blank ANNEXES - CONTENTS ANNEX A. Evaluation Statement of Work.................................................................................................................3 ANNEX B. Methodology Update and Data Limitations.........................................................................................35 ANNEX C. Recommendations ................................................................................................................................. 62 ANNEX D. Considerations for the Next Five-Year Health Assistance Program ........................................ 65 ANNEX E. Commodities supplied by APHIAplus in the 17 Sites Visited..........................................................69 ANNEX F. Support to Kenyatta National Hospital’s GBV Recovery Center ............................................... 71 ANNEX G.1. Program Management Plan Data Tables for Nairobi ................................................................. 72 ANNEX G.2. Program Management Plan Data Tables for Coast Province ................................................... 89 ANNEX H. Result 4 Indicator Discussion............................................................................................................ 103 ANNEX I. Data Collection Tools........................................................................................................................... 104 ANNEX J. Statement of Differences........................................................................................................................239 ANNEX A. EVALUATION STATEMENT OF WORK EXECUTIVE SUMMARY..................................................................................................................................................3 1. BACKGROUND INFORMATION AND DEVELOPMENT HYPOTHESIS .............................................3 Identifying Information.......................................................................................................................................4 Development Context ......................................................................................................................................4 1.1.1 Problem or Opportunity Addressed................................................................................4 1.1.2 Target Areas and Groups...................................................................................................5 Intended Results..................................................................................................................................................6 A.1.1. Program Goal and Objectives..............................................................................................................6 Approach and Implementation ........................................................................................................................7 2. EVALUATION RATIONALE .............................................................................................................................11 2.1 Evaluation Purpose................................................................................................................................11 Audience and Intended Use ...........................................................................................................................12 2.3 Evaluation Questions............................................................................................................................12 3. EVALUATION DESIGN AND METHODOLOGY............................................................................................13 3.1 Evaluation Design ..................................................................................................................................13 3.2 Data Collection Methods....................................................................................................................13 3.3 Data Analysis Methods.........................................................................................................................14 3.4 Methodological Strengths and Limitations.......................................................................................15 4. EVALUATION PRODUCTS....................................................................................................................................15 5. TEAM COMPOSITION.............................................................................................................................................17 6. EVALUATION MANAGEMENT.............................................................................................................................18 A–3 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Executive Summary APHIAplus Nairobi/Coast is a cooperative agreement (Agreement Number 623–A–11–00009) with Pathfinder International and is managed by USAID/Kenya’s Office of Population and Health (OPH). The award, which is for $55 million, began on January 1, 2011 and ends on December 31, 2013. This project is one of five regional service delivery mechanisms under the APHIAplus Service Delivery awards. The APHIAplus service delivery mechanisms were preceded by APHIA II projects that covered all the eight provinces of Kenya. The five APHIAplus projects were awarded in January6 of 2011 after an extensive redesign process that began in late 2009 following an assessment of USAID/Kenya’s health portfolio, which resulted in a new USAID/Kenya Health Implementation Framework 2010-2015. APHIAplus stands for AIDS Population and Health Integrated Assistance – People-centered, Leadership￾focused, Universal access and Sustainability. The Implementation Framework envisioned these projects supporting delivery of critical health services at the provincial and district levels while working in coor￾dination with USAID-funded national-level technical assistance partners. The evaluation will be conducted by Management Systems International (MSI) and is intended primarily for accountability and learning. The main objective of the evaluation is to assess the performance of the APHIAPlus Nairobi/Coast. The two specific objectives include: (1) To conduct an in-depth qualitative assessment of technical and management approaches, coordination with host country and other stake￾holders and support to country health systems; and, (2) To conduct a review of project achievements based on assigned targets in HIV/AIDS, FP/RH and Child Survival activities. The evaluation seeks to answer the following questions: 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 4. To what extent was the APHIAplus model appropriate for achieving key outcomes un￾der Result 3 and 4 of the Implementation Framework? 5. What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? The data collection and analysis uses a mix of methods to answer the questions. Data collection meth￾ods include: desk review, key stakeholder interviews, and group discussions. Data analysis will employ various techniques, comprising summary statistics, comparison analysis, pattern/content analysis, re￾sponse convergence/divergence analysis, and mix methods integration/findings synthesis, as appropriate for the data collected. The six-person evaluation team consists of three international team members (including the Team Lead￾er) and three national team members, with a range of experience and expertise related to the health sector and monitoring and evaluation. The team size and composition reflects the expansive nature of the program and USAID/Kenya’s interest in meeting approaching commitments for designing the next program. A–4 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 1. Background Information and Development Hy￾pothesis Identifying Information · Program: Office of Population and Health · Project: APHIAplus Nairobi / Coast · Award Number: AID-623-A-11-00009 · Award Dates: January 1, 2011 – December 31, 2013 · Funding: $55 million · Implementing Organization: Pathfinder · Project COR/AOR: Jerusha Karuthiru · Type of Evaluation: Final Performance Evaluation · Period to be evaluated: January 1, 2011 – April 2013 Development Context 1.1.1 Problem or Opportunity Addressed USAID/Kenya has a long history of supporting the population and health sector in Kenya. Under the 2006-2010 APHIA II program, USAID/Kenya and its partners reported gains in the area of HIV/AIDS (treatments, care, support and prevention), TB, Child Survival, Malaria, and increased use of family plan￾ning services as well as integration of reproductive health and HIV services. These advancements were supported by PEPFAR funding, which listed Kenya as a USG priority country. A final assessment of the APHIA II provided insights for the design of a five-year USAID/Kenya Imple￾mentation Framework for the Health Sector, which highlighted several opportunities for future invest￾ments, particularly in advancements made in HIV/AIDS and Family Planning. USAID/Kenya and the Gov￾ernment of Kenya (GOK) recognized a need to continue high impact health results balanced with health systems strengthening for long-term sustainability. This corresponded with a shift from “emergency” response in PEPFAR I to a technical assistance approach of identifying structures and systems for sus￾tainable solutions in PEPFAR II. This re-focus embodied in APHIAplus would rely on multiple actors, in￾cluding public and private sector at national, provincial, district and community levels, to work together to meet the national health objectives. APHIAplus began at a pivotal time for Kenya, building momentum from an enabling policy environment framed around Kenya Vision2030, which expressly aims to provide “equitable and affordable quality health services to all Kenyan in order to continue to reverse negative health trends.” A–5 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 1.1.2 Target Areas and Groups APHIAplus is structured to include Zonal Service Delivery Projects, whereby the program clustered tar￾get areas by geographic and trend commonalities. APHIAplus Nairobi/Coast covers Zone 2, which is described as a moderate need zone that covers urban areas. It includes the whole of Nairobi and Coast Province (excluding Tana River District – which is covered in the Northern arid zone) as these are the two regions with the largest urban settings. USAID/Kenya reported supporting over 700 health facilities in this area. These facilities range from large district hospitals to small health centers. APHIAplus Nairobi/Coast’s level of engagement ranges among the various sup￾ported facilities, from full support and technical assistance to holding a few trainings. The charts below show the specific project sites intended for examination by the evaluation as selected by USAID/Kenya. Section 3.2 describes in more detail the selection process. A–6 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal a. Intended Results A.1.1. Program Goal and Objectives The Activity Approval Document from March 2010 includes a Results Framework for APHIAplus as shown below: Strategic Goal: Sus- tained improvement of health and well- being for all Kenyans Assistance Objective: Improved health outcomes and impact through sustainable country-led programs IR1: Strengthened leadership, manage- ment and governance for sustained health programs IR2: Health systems strengthened for sustainable delivery of quality services IR3: Increased use of quality health ser- vices, products and information IR4: Social determinants of health addressed to improve the well- being of marginalized, poor and under- served populations A–7 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal For the APHIAplus Nairobi/Coast evaluation, Intermediate Results 3 and 4 will be considered as the overall objec￾tives of the program. The program also has a very comprehensive results chain for Results 3 and 4 that specifically links those results to outcomes, outputs, inputs, with corresponding indicators that are collected and reported. b. Approach and Implementation APHIAplus Nairobi/Coast is a cooperative agreement (Agreement Number 623-A-11-00009) with Pathfinder In￾ternational and is managed by USAID/Kenya’s Office of Population and Health (OPH). The award began on Janu￾ary 1, 2011 and ends on December 31, 2013. This project is one of five regional service delivery mechanisms un￾der the APHIAplus Service Delivery awards. The APHIAplus service delivery mechanisms were preceded by APHIA II projects that covered all the eight provinces of Kenya. The five APHIAplus projects were awarded in January of 2011 after an extensive redesign process that began in late 2009 following an assessment of USAID/Kenya’s health portfolio, which resulted in a new USAID/Kenya Health Implementation Framework 2010- 2015. In issuing the RFA for APHIAplus Service Delivery awards that covered four geographic zones, a swap was made between the Total Estimate Cost (TEC) for APHIAplus Nairobi/Coast and that of a separate project, APHIAplus Rift Valley, resulting in APHIAplus Nairobi/Coast ending up with a TEC of US$55 million. Discussions were held between various USAID offices, including Regional Legal Advisors, the Front Office, Regional Financial Management Systems, and OPH. Two options were considered: 1) increase the TEC of APHIAplus Nairobi/Coast or 2) reduce the period of performance of the award. The Mission opted for the second choice and reduced the period of per￾formance of the APHIAplus Nairobi/Coast project to three years instead of five. APHIAplus stands for AIDS Population and Health Integrated Assistance – People- centered, Leadership-focused, Universal access and Sustainability. The Implementa￾tion Framework envisioned these projects supporting delivery of critical health ser￾vices at the provincial and district levels while working in coordination with USAID￾funded national-level technical assistance partners. The framework below graphically captures the program architecture for the entire APHIAplus program. APHIAplus Nairobi/Coast falls under one of Service Delivery Components. USAID/Kenya expected that in each APHIAplus zone, activities would include evidence-based high impact, cost-effective interventions that took into consideration the following:1 · Supporting complementary tech￾nical areas tailored and designed for the proposed zone(s); 1 These are APHIAplus assumptions and a critical part of a sustainability analysis. We will test these assumptions by looking for evidence on whether they held true or not during the course of Project implementation. Being that these assumptions largely address alignment with local systems and systems strengthening, the results of this analysis will greatly inform the sustainability question of this evaluation. A–8 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal · Working through provincial government structures (to ensure coordination) to provide support at dis￾trict and community levels; · Linking USAID’s service delivery support with USAID’s systems strengthening activities; · Ensuring rapid deployment, smooth transition and continuity of essential care services to critical popula￾tions (e.g. anti-retroviral therapy (ART) clients, orphans and vulnerable children (OVC), etc.) without dis￾ruption; · Coordinating vertically, throughout the Kenyan government structure, and horizontally, with other US Government and Kenyan partners; · Working within the government health system in the delivery of health services; and · Building strong partnerships with effective Kenyan NGOs, faith based organizations (FBOs), the private sector, and other implementing partners essential for sustained service delivery. Working with the government and other partners, APHIAplus Service Delivery projects support marginalized, vul￾nerable and underserved populations including youth, most-at-risk-populations (MARPs), people living with HIV (PLWHs) and those on ARVs, orphans and children affected by AIDS; women of reproductive age (pregnant and post-partum women), highly vulnerable girls, neonates and infants. The implementation strategy focuses on achiev￾ing two levels of results – at the facility and community level. The project works through the GOK structure at the national, provincial, district and community levels. APHIAplus Nairobi/Coast reported four cross-cutting themes in the program: i) gender; ii) youth; iii) whole market-approach; and, iv) equity. APHIAplus Nairobi/Coast follows the tenets of the Global Health Initiative (GHI) and international mandates; the￾se include the following principles, as captured in the Program Description in the Request for Proposals: 1. Assure country-led, country-owned and country-managed 2. Align Kenyan, USG and development partner strategies 3. Invest in leadership, capacity and systems for long-term sustainability 4. Maximize a client-centered approach through integration of services and systems 5. Increase involvement of the private sector in health care delivery 6. Ensure strategic collaboration and coordination 7. Manage for results with mutual accountability The AHPIAplus works with a consortium of partners. Partners are co-located in the same office in Nairobi to enable coordination and enhance management. The following organizations constitute the partnership: · Pathfinder International (principal implementer) partners with governments, NGOs, and the private sec￾tor, to promote best practices and state-of-the-art approaches in Family Planning and Reproductive Health. Pathfinder’s comprehensive approach to HIV/AIDS programming promotes synergies among key intervention areas including prevention, stigma reduction, ART, HBC, and PMTCT. · Child Fund International is committed to child-centered change that leads to healthy and secure infants, educated and confident children, and skilled and involved youth. ChildFund brings global and national ex￾perience in family-centered OVC care, and community engagement for children’s health. · CLUSA works with communities to identify and address health concerns. CLUSA’s training activities build organizational and management capabilities of communities to take responsibility for health services and raise local resources for community health, social and economic initiatives. · Network of AIDS Researchers of Eastern and Southern Africa conducts HIV research and supports evi￾dence-based policy formulation and program implementation. NARESA promotes interdisciplinary re￾search and provides technical leadership in HIV, and is a pioneer of PMTCT in Kenya. · Population Services International uses innovative, culturally-sensitive approaches to raise awareness for healthy behaviors, generate demand for health products, and promote franchised private providers. PSI/Kenya conducts programs in malaria, Family Planning/Reproductive Health, HIV/AIDS and child surviv￾al. A–9 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal The partners are supported by a number of local sub-grantees. In addition to sub-grantees, APHIAplus Nairo￾bi/Coast works with National Mechanisms (listed in Annex E). Through basic program literature and conversations with USAID and APHIAplus Nairobi/Coast a reasonable de￾velopment theory emerges: By providing technical assistance to the government of Kenya to improve health care systems sustainably, the demand by Kenyans of health services will increase, leading to greater utilization. During the evaluation, this development theory will be further defined and validated. c. Existing Data The following is a list of collected documentation shared by USAID. This should not be considered exhaustive for the purposes of the document review of the evaluation. Documents Collected: APHIAplus Workplans · Year 1: Jan – Dec 2011 · Year 2: Jan – Dec 2012 · Year 3: Jan – Dec 2013 · Organogram 2011 · Organogram 2012 Monitoring Materials · PMP 2011 · PMP 2012 · USAID · M&E Plan, Work Plan Year 2, January 2012 – December 2012 · M&E Plan for Aphiaplus Nairobi – Coast Health Service Delivery Project, Nov 2012 · M&E Plan 2011, May 31, 2011 · DQA Report for Nairobi Province ART sites Modifications/Related reports · Modification 1: Feb 2, 2011 · Modification 2: Sept, 15, 2011 · Modification 3 · Modification 4 · Request for Applications (RFA), USAID/Kenya APHIAplus, Health Service Delivery projects April 23, 2010. · USAID/Kenya 5 year Implementation Framework for the Health Sector (2010-2015) AIDS, Population and Health Integrated Assistance (APHIA II) Assessment Report, November, 2009 A–10 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal APHIAplus (Service Delivery Component) Activity Approval Document, March, 2010 · M&E/HMIS Assessment; APHIA II Nairobi presentation July, 2010. · APHIA II Coast overview (2006 – 2010); End of project meeting); Dec 15, 20110. · APHIAplus Nairobi – Coast Project Renovation Needs Assessment Report, September, 2011. USAID’s Global Health Strategic Framework; Better Health for Development: FY 2012 – FY 2016. Progress/Annual Reports · APHIAplus Nairobi-Coast Y1 Q1 to Q4 Progress Reports (Jan – Dec, 2011) · APHIAplus Nairobi-Coast Y2 Q1 to Q4 Progress Reports (Jan – Dec, 2012) · APHIAplus Nairobi-Coast, 2011 and 2012 Site Monitoring reports · APHIA II Coast Quarterly Report Jul – Sept, 2009, Oct – Dec, 2009, Jan – Mar, 2010 and Apr – Jun, 2010. · APHIA II Central End of project Report, 6.8.2011 · APHIA II Nairobi End of Project Report 15.6.2011 · APHIA II Nairobi – Central End of Project Report Aug, 2006 – Feb, 2009 Other Reference materials · Kenya Census data – 2009; May, 2009. · Taking the Kenya Essential Packages for health to the Community: A strategy for the Delivery of Level One Services; MOH, June 2006. · Compendium of Indicators for Evaluating Reproductive Health program: Measure Evaluation Manual Series, No. 6 – August 2002. · Kenya Health Information System Policy · Kenya Indicator and Standard Operating Procedure manual for Health Sector, may 2008 · Kenya National M&E Framework for HIV/AIDS and STI Control, October 2010 · National Reporting Framework of Indicators: The Vision 2030 first medium –term Plan (2008 - 2012; Ministry of State for Planning, national Development and Vision 2030. · Kenya Demographic and Health Survey 2008 -2009; June 2010. · Kenya District Profile - Population Estimates 2009; May, 2009. · Kenya Health System Assessment Report 2010; August, 2010. · Kenya Service Provision Assessment Report, 2010. · The Millennium Development Goals Report; UN, 2010. · New Generation Indicators Reference Guide; Version 1.1, August 2009. · Kenya AIDS Indicator Survey 2007; Final Report Sept, 2009. A–11 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal · Overview of Community Health services in Kenya; Presentation by Ken Oruenjo. · National Monitoring the Building Blocks of Health Systems; a Handbook of Indicators and their Measurements Strategies; WHO 2010. · The Kenya 2007 HIV/AIDS Estimates; an Interim Projected HIV Prevalence and Incidence Trends for 2008 to 2015: NASCOP/NACC, July, 2009. · Kenya HIV Prevention Response and Modes of Transmission Analysis; KNACC, March 2009. · Kenya National AIDS Strategic Plan 2009/10 – 2012/13. · National Health Sector HIV Report 2009; Progress with the National Health Sector Response: National AIDS/STI Control Program, November 2010. · Procedure Manual for HIV Data Management, November 2010 · CHW AIM: A Toolkit for Improving Community Health Worker Programs and Services Guide, march 2011. · Reproductive Health Communication Strategy 2010 – 2013, Ministry of Public Health and Sanitation, Govern￾ment of Kenya October 2009 · National Reproductive Health and HIV and AIDS Integration Strategy, Ministry of Public Health and Sanitation and Ministry of Medical Services, Government of Kenya, August 2009 · Strategy for Improving the Uptake of Long-acting and Permanent Methods of Contraception in the Family Planning Program July 2008 – June 2012, Ministry of Public Health and Sanitation, Government of Kenya, 2008 · Priority High Impact interventions required to accelerate progress towards attainment of MNCH and nutri￾tion targets both at community and facility level, Ministry of Public Health and Sanitation, Government of Ken￾ya, March 2010 · National Family Planning Costed Implementation Plan 2010-2015, Ministry of Public Health and Sanitation, Government of Kenya, August 2011 · Summary Report on Family Planning in Kenya, USAID/Kenya, September 2009 · National Road Map for Accelerating Attainment of the MDGs Related to Maternal and Newborn Care in Ken￾ya, Government of Kenya, August 2010 2. Evaluation Rationale 2.1 Evaluation Purpose MSI will conduct the Final Performance Evaluation of the APHIAplus Nairobi/Coast program. This is a required evaluation. The main objective of the evaluation is to assess the performance of the APHIAPlus Nairobi/Coast. The two spe￾cific objectives include: 1. To conduct an in-depth qualitative assessment of technical and management approaches, coordination with host country and other stakeholders and support to country health systems. 2. To conduct a review of project achievements based on assigned targets in HIV/AIDS, FP/RH and Child Survival activities. A–12 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal The period of evaluation is the program start date to April 2013. 1.2 Audience and Intended Use This evaluation will detail what has worked, what has not worked and why, for APHIAplus Nairobi/ Coast. Rec￾ommendations will shape strategies, approaches and activities for the follow-on program design. The primary audience for the evaluation is USAID/Kenya and the Government of Kenya. Secondarily, the report is intended for Pathfinder, USAID/Washington and other interested implementing partners (NGOs). 2.3 Evaluation Questions The evaluation seeks to answer the five following questions: 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Re￾sults 3 and 4? Were the key expected outcomes achieved? If not, why not? Answering Question One will require examining all the components of the program implementation, including the causal logic from plans, activities, results, outcomes, and higher-level results, and whether or not they have been achieved. In doing so, the findings shall also indicate whether the implementation (Work Plans) was aligned to the Program Description. 2. What strategies employed within the workplan worked and what did not work for successful im￾plementation and achievement of key outcomes in Result 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? The Request for Applications referred to evidence-based high impact interventions, particularly with respect to increasing access to clean water, modern contraceptive use and decreasing maternal, neonatal, and under-five mor￾tality. Further, Pathfinder International stated that APHIAplus supported facilities to review monitoring and re￾porting data to inform work planning, and identification and prioritization of issues to address. Beyond reporting data, this question seeks to answer whether any external programmatic literature, national or international in￾formed workplans and strategies. 4. To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? Question Four focuses on the model employed by APHIAplus. USAID/Kenya has stated the main characteristics of interest of this model to be assessed in the evaluation: (i) zonal distribution; (ii) integration in the OVC commu￾nity; and, (iii) management structure. 5. What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? Question Five focuses on sustainability, which has been defined as models/activities that remain operational largely without donor support--monetary or technical. This relates to country ownership and focuses particularly on bal￾ancing supply with demand, examining how APHIAPlus Nairobi/Coast has supported/ prepared district- and com￾munity-level partners in project sites to meet the increase in demand for quality services. This should examine what interventions (strategies, activities, protocols, and methods) have been adopted and institutionalized by the GOK in terms of national resourcing (funding, staffing, equipment, etc.) and incorporation into agency plans, poli￾cies, budgets, protocols, etc. to continue interventions beyond USAID support. Gender will be viewed as a cross-cutting theme to be explored where appropriate throughout answering the eval￾uation questions. The evaluation team is expected to be responsive to USAID's dual expectations for treating A–13 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal gender appropriately: (a) gathering sex disaggregated data, and (b) identifying gender differential participation in/benefits from aspects of the program where differences on this basis are possible. In answering the questions, the evaluation team will put attention towards highlighting relevant lessons intended for the future follow-on program. One to two pages of the report will be dedicated specifically to lessons learned (see Section 4.2 for specifics on the report outline). 3. Evaluation Design and Methodology 3.1 Evaluation Design The final performance evaluation will be retrospective and prospective, looking backward to examine the change from the beginning of the project until now, as well as maintaining an eye towards learning for future programming. APHIAplus is part of a continuum of programming in the health sector that started with APHIA I and continued with APHIA II. The evaluation will need to distinguish the ways in which APHIAplus differed from previous pro￾gramming and how these distinctions led to or contributed to achievement of proposed results. The evaluation team is expected to use well-developed data collection and analysis methods to address each of USAID’s evaluation questions. A preliminary version of a matrix for associating data collection and analysis meth￾ods with evaluations questions (Getting to Answers) is provided in Annex C. This matrix shares with the evalua￾tion team the initial thinking about appropriate methodological choices. The evaluation team is expected to review and refine this methodology, or suggest higher quality alternatives that could be employed at no additional cost beyond what USAID has allotted for this evaluation. Details the evaluation team adds to this preliminary plan for gathering and analyzing data on each evaluation question shall be submitted to USAID for review/approval as part of the evaluation team’s Methodology and Workplan (Section 4.1). 3.2 Data Collection Methods Some key aspects of the data collection are the following: Document Review • A preliminary audit of the documents listed in Section 1.5 has been completed. The evaluation team will be expected to review documentation provided by APHIAplus Nairobi/Coast and USAID on the program, and secondary research collected (particularly from other major health-related partners – WHO, UNICEF, UNAIDS, etc). An instrument will be developed to codify and organize data from the document review for analysis as per answering the evaluation questions. Given the large amount of quantitative data, the team will need to determine a system or database to collate and consolidate relevant statistical data for future review and analysis. It is expected that the evaluation team will present initial findings from the document review as part of the Team Planning Meeting (Section 4.1) in the beginning of the evaluation. Site Visits • Fourteen facilities (plus an additional one for instrument piloting) were purposively selected to provide the evaluation team an opportunity to observe various sites to help contextualize data collected as well as meet with service providers. The sample of sites is not large enough to be generalized, but should give the evaluation team an indicative snapshot of program implementation at the beneficiary level. • USAID/Kenya OPH provided the sample as part of the Statement of Objectives. The coverage focuses on two program areas: PMTCT and Treatment since facilities that provide these services also provide other health services. Other criteria considered included: (i) geography, (ii) availability of a community unit at￾tached to the site, and (iii) volume of service delivery statistics (a mix of high, moderate, and low volume of service delivery). Key Informant Interviews A–14 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal · Key informant interviews will be essential to understand the process, information flows, decision-making and operational engagement of the program with the Ministry of Health, facilities and communities. The evaluation team will interview a purposively selected set of key stakeholders, including USAID, APHIAplus partners, national mechanisms, health-related agencies, community leaders, health service providers and GOK (national and district – county level). As there are multiple stakeholders with distinct roles in the program, a semi-structured interview tool will be developed for different types of stakeholders to ensure adequacy of questions and comparability across interviews. Group Discussions (GDs) · The GDs are intended for community groups linked to the facilities. Depending on the availability, target groups should cover youth, MARPS, PLWHs, women of reproductive age (pregnant and post-partum women), etc. One to two Discussion Groups will be scheduled linked to the facility visit. 3.3 Data Analysis Methods Given the scale of the program, size of the team, amount of data available and to be collected, it is crucial for the evaluation team to design a data analysis plan at the onset, whereby the tools and instruments used for data collec￾tion will feed into the data processing and synthesis of findings, and reporting can quickly be undertaken. The plan allocates two weeks for data analysis and processes given the large quantity of data expected from the field work. Comparison Analysis · Comparison analysis will be employed at four different levels: planned versus actual; before and after; sta￾tistical comparison/validation; and actual versus best practice. The first three levels most directly relate to answering Question One, which will need to look at what was planned, carried out and how that con￾tributed to achieving higher level results. Statistical comparison/validation will help support (or counter) findings based on monitoring data. Comparison to best practice is most relevant in examining methods and models as a source to gauge effectiveness and appropriateness. Summary Statistics · In assessing the PMP data and indicators, summary statistics will be used to analyze quantitative data and demonstrate achievement across the results framework. Pattern/Content Analysis · For qualitative data, patterns will be examined so that comparisons can be made between respondents and contexts to determine the pattern of effect of APHIAplus Nairobi-Coast. This will involve broad pat￾terns and a more detailed examination of how different respondents answered the same question, e.g., different community groups, government stakeholders, etc. Response Convergence/Divergence Analysis · The team will review data collected to determine where there is significant response convergence from the varied stakeholders and beneficiaries. Where divergence is found, the team will follow-up to better understand the context and reasons for divergence in facts, perceptions or opinions. Mixed Methods Integration/Findings Synthesis · Using a mixed methods approach, data from various methods will be integrated to arrive at findings. Where different methods produce conflicting evidence, the evaluation team will, to the extent possible, double back to examine why these data conflict, as well as weight the data from the various methods in terms of strength in validity and reliability. A–15 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 3.4 Methodological Strengths and Limitations Examining impact or achievement of higher level results may prove challenging. By the time of the evaluation data collection, the three-year program will still have not even reached the mid-point of its third year. Pathfinder has noted that the first year focused primarily on the start-up of the program, signifying that full program implementa￾tion will only have taken place for a year and one half. While this perception will be reviewed during the evalua￾tion, it should be noted that if start up did, indeed, take one year, there may be some data limitations. Attribution to APHIAplus specifically may also be difficult as the program has served as a follow-on program to past APHIA programs. Further, support such as technical assistance and systems strengthening require time to take root and produce concrete changes. The evaluation and its findings and conclusions, therefore, may need to focus both on the processes used by the partner and the status of systems strengthened or established by the program. While the program has made a significant effort to develop a comprehensive results framework and chain, with multiple levels of indicators associated with precise activities, the data source may not be reliable. The program uses information from government health facility reporting. The issues with reporting accurate and reliable data are well-known by Pathfinder and USAID/Kenya. While APHIAplus may be working to improve M&E systems and reporting throughout its program, limitations remain. Data from the PMP and the M&E system will be used as a data source, but weighed and supplemented with data from external sources as well as primary qualitative data collected during the course of the evaluation. The beneficiary population includes underage OVCs and other sensitive groups. Issues of confidentiality and ethics will need to be considered in developing the data collection tools and instruments. Best practice and technical expertise from the evaluation team will be drawn upon to address this topic. Having such a large team simultaneously collecting qualitative and quantitative data requires to a greater degree of oversight and management to ensure quality. In order to minimize risk of issues with the data at the analysis stage, one or two MSI technical staff will accompany different team members at the beginning of data collection (5-7 days maximum) to ensure quality data collection across the team. Doing so also supports the team leader in coordinat￾ing members as they collect data across Nairobi and the Coast. 4. Evaluation Products 4.1 Deliverables This revised proposal to USAID/Kenya is due by May 30, 2013. Given the need for technical review and approval, and the contracting process, MSI assumes for the initial purpose of creating an indicative schedule, that the con￾tract will be awarded on June 10, 2013 and the evaluation start date will be June 17, 2013. The evaluation team will be responsible for delivering the following products on the proposed schedule below. The schedule will be shifted accordingly if the Task Order is approved on a later date. June 24-July 5 Team Planning Meeting (TPM) & Data Collection Tool Pre-testing: The TPM will be held in MSI offices once the evaluation team is in country. It is expected that USAID and Pathfinder will be engaged with this process. The second week of this process will focus on pre-testing the tools developed at local facilities (large and small) to ensure appropriateness and precision of the tool and data collection. Two sites are considered to capture the different dynamics and structures of the sites. The outcomes of this meeting include: · Presentation of the initial findings of the document review by evaluation question; · Clarification of team members' roles and responsibilities; · Establishment a team atmosphere, share individual working styles, and agree on proce￾dures for resolving differences of opinion; A–16 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal · Review of the final evaluation questions; · Review and finalization of the assignment timeline and share with USAID; · Development of data collection and analysis methods, instruments, tools, and guidelines; · Test data collection instruments at one to two facilities (this is built into the schedule given USAID/OPH experience in previous assessments regarding poor tool design) · Review and clarification of any logistical and administrative procedures for the assign￾ment; · Development of a preliminary draft outline of the team's report; and · Assignment of drafting responsibilities for the final report. July 8 Workplan and Methodology: During the TPM, the team will prepare a detailed work plan which will include the methodologies (evaluation design, tools) and operational workplan to be used in the evaluation. This will be discussed with and approved by USAID prior to implementation. Aug 8 & 9 Presentation with USAID and Partners: The evaluation team will present the major findings of the evaluation to USAID and partners in a PowerPoint presentation. The presentation will follow a similar structure to the final report and present major findings, conclusions, and recommenda￾tions. The first presentation will be given to partners, followed by a presentation to USAID and GOK the following day will have an opportunity to comment and provide input as part of the presentation. The team will consider the comments and revise the draft report accordingly, as appropriate. Aug 9 Initial Analysis Report & Data Processed: To facilitate USAID/Kenya’s future planning and deci￾sion-making, an initial analysis report and processed data will be shared. The report will focus on emerging findings and preliminary conclusions. The processed data will also be shared for USAID to review and use. The draft evaluation report is expected to build and elaborate on the infor￾mation presented in the initial analysis report. Aug 27 Draft Evaluation Report: A draft report will be submitted to MSI prior to team leader departure. The written report should clearly describe findings, conclusions, and recommendations, fully sup￾ported by triangulated evidence. The report will synthesize and present information in an easy- to-digest style. USAID will provide comments on the draft report within two weeks of submis￾sion. Sept 25 Final Evaluation Report: The team will submit the final report that incorporates the team re￾sponses to Mission comments and suggestions. The format will adhere to the standard reporting guidelines listed in 4.2. The team shall provide to USAID a weekly report of ongoing activities during the course of the evaluation describ￾ing the process, any issues encountered, and relevant emerging findings. The evaluation report will adhere to USAID Evaluation Policy and as such all raw quantitative data will need to be shared with USAID. Qualitative data will also be shared, if specifically requested by USAID. 4.2 Reporting Guidelines The format for the evaluation report shall be as follows, and the report should be a maximum of 30 pages not in￾cluding annexes. The report format should be restricted to Microsoft products and 12-point font should be used throughout the body of the report, with 1” page margins. An electronic copy in MS Word shall be submitted. In addition, all data collected by the evaluation shall be provided to USAID in an electronic file in an easily readable format; organized and fully documented for use by those not fully familiar with the project or the evaluation. If the report contains any potentially procurement sensitive information, a second version report excluding this infor￾mation shall be submitted (also electronically, in English). Below represents a guideline for the report structure. a. Executive Summary—concisely state the most salient findings and recommendations (3 pg); A–17 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal b. Table of Contents (1 pg); c. Evaluation Purpose and Evaluation Questions—purpose, audience, and synopsis of task (1 pg); d. Project Background—brief overview of development problem, USAID project strategy and activities im￾plemented to address the problem, and purpose of the evaluation (2-3 pg); e. Evaluation Design, Methods, Limitations—describe evaluation methods, including constraints and gaps (1 pg); f. Findings/Conclusions/Recommendations—for each evaluation question (10-15 pp); g. Lessons Learned—any pertinent lessons for the overall purpose and audience of the evaluation (1-2 pp); h. Annexes that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. These include references to bibliographical documentation, meetings, interviews and group discussions. 5. Team Composition The evaluation team will be composed of six evaluators – three international team members (including the team leader) and three national team members. The composition of the team sought to match experiences and exper￾tise in the following areas: Public Health, M&E for public health, Kenya public health systems, systems strengthen￾ing, social determinants for health, evaluation, data collection and analysis methods, practitioner experience, and community health research. CVs for the Team Leader and Members can be found in Annex F. International Evaluation Team Leader · Evaluation design experience, including the selection of data collection methods on a question-specific ba￾sis and development of a detailed data analysis plan. · Superior writing ability, including evidence of an ability to structure evaluation reports in a way that logi￾cally and transparently lays out empirical findings, conclusions and recommendations in relation to evalua￾tion questions. · Health-related social science research skills and statistics, including strong data visualization skills. · Knowledge of HIV/AIDS-related programming and M&E. · Field experience that includes at least some but not necessarily all of the following: o Survey research experience, including development of structured and semi-structured interview and/or observation instruments, by hand and using mid-size survey aids such as EPI Info from CDC; sample size determination for specific confidence levels and confidence intervals; selection and use of other survey data (and other large data set) entry and analysis software (SPSS or alternative). o Structured or semi-structured group interviews and/or focus groups, including the creation of written instruments and transcripts for same; content analysis and other techniques for coding and trans￾forming group/open-ended data into analyzed information; qualitative data analysis software (NVIVO or alternative) and/or case study documentation experience; transformation of qualitative data into a quantitative form that can be merged with other quantitative data in a mixed methods analysis. o Applied experience with non-experimental theory testing techniques for examining program/project effects (baseline reconstruction, outcome mapping, outcome harvesting, general elimination method (modus operandi), contribution analysis, lagged regressions and/or others). · Advanced degree in International Development, Economics, Political Science, Statistics or related social science field preferred. International Technical Team Member · A Master’s Degree in public health, research methodology, social science or a related field. · 8 + years of professional experience in implementing, monitoring and evaluating health-related develop￾ment programs, with extensive evaluation experience. · Experience working on USAID health-related evaluations and development projects. · In depth understanding of high-impact health interventions and health system strengthening. · Health-related social science research skills and statistics, including strong data visualization skills. A–18 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal · Demonstrated written communications skills, especially in drafting evaluations, assessments and reports, required. · Familiarity with USAID Forward quality evaluation standards and requirements. National Technical Team Member · A Master’s Degree in monitoring and evaluation, research methodology, health, social science or a related field. · 5+ years of professional experience in implementing, monitoring and evaluating health-related develop￾ment programs in Africa, with extensive evaluation experience. · Experience working on USAID health-related evaluations and development projects. · In depth understanding of high-impact health interventions and health system strengthening. · Health-related social science research skills and statistics, including strong data visualization skills. · Demonstrated written communications skills, especially in drafting evaluations, assessments and reports, required. · Familiarity with USAID Forward quality evaluation standards and requirements. 6. Evaluation Management 6.1 Logistics USAID/Kenya will provide input through an initial in-briefing to the evaluation team, identify key documents, and assist in introducing the evaluation team to the implementing partner. It will also be available for consultations regarding sources and technical issues with the evaluation team during the evaluation process. MSI will assist in arranging meetings with key stakeholders identified prior to the initiation of field work. USAID/Kenya will provide an introductory letter to partners, including the GOK, to facilitate scheduling interviews. The evaluation team will be responsible for arranging other meetings as identified during the course of the evaluation. It will advise USAID/Kenya of any meetings with the Government of Kenya and seek advice from USAID/Kenya on whether they choose to participate. MSI is responsible for arranging vehicle rental and drivers as needed for site visits around Nairobi and the field. MSI will also provide hotel arrangements office space, internet access, printing, and photocopying. It will also make all payments to vendors directly after team members arrive in country. 6.2 Scheduling Work is to be carried out over a period of approximately fifteen weeks, beginning on or about June 17, 2013 with document review. Evaluators will deploy to Kenya o/a June 22 and field work will be completed the week of July 27, 2013. It is expected that the team will be divided up to mini-teams of two to collect data in the field. A de￾brief and an initial analysis report will be submitted o/a September 9, 2013. A final report will be submitted and the evaluation will conclude o/a September 25, 2013. The scheduling considers the number of stakeholders, communi￾ty groups, and a rough allocation two to three days per facility location depending on size. Exact scheduling and division of labor will be reviewed during the Team Planning Meeting and presented in the Methodology and Workplan. 6.3 Budget A budget will be provided separately. A–19 ANNEX A:AID-623-TO-13-000JJ A.1. BACKGROUND Project Name: APHIAplus Nairobi Coast Implementing Partner: Pathfinder Agreement Number: AID-623-A-11-00009 Project COR/AOR: Jerusha Karuthiru Life of the Project: January 1, 2011 - December 31, 2013 Total Funding: $55 million A.1.1. Program Goal Background APHIAplus Nairobi/Coast APHIAplus Nairobi/Coast is a cooperative agreement (Agreement Number 623-A-11-00009) with Pathfinder In￾ternational and is managed by USAID/Kenya’s Office of Population and Health (OPH). The award began on Janu￾ary 1, 2011 and ends on December 31, 2013. This project is one of five regional service delivery mechanisms un￾der the APHIAplus Service Delivery awards. The APHIAplus service delivery mechanisms were preceded by APHIA II projects that covered all the eight provinces of Kenya. The five APHIAplus projects were awarded in January of 2011 after an extensive redesign process that began in late 2009 following an assessment of USAID/Kenya’s health portfolio which resulted in a new USAID/Kenya Health Implementation Framework 2010- 2015. In issuing the RFA for APHIAplus Service Delivery awards that covered four geographic zones, a swap was made between the Total Estimate Cost (TEC) for APHIAplus Nairobi/Coast and that of a separate project, APHIAplus Rift Valley, resulting in APHIAplus Nairobi/Coast ending up with a TEC of US$55 million. Discussions were held between various USAID offices, including Regional Legal Advisors, the Front Office, Regional Financial Management Systems, and OPH. Two options were considered: 1) increase the TEC of APHIAplus Nairobi/Coast or 2) reduce the period of performance of the award. The Mission opted for the second choice and reduced the period of per￾formance of the APHIAplus Nairobi/Coast project to three years instead of five. APHIAplus stands for AIDS Population and Health Integrated Assistance – People-centered, Leadership-focused, Universal access and Sustainability. The Implementation Framework envisioned these projects supporting delivery of critical health services at the provincial and district levels while working in coordination with USAID-funded national-level technical assistance partners. The framework divided Kenya into five zones based on specific needs and characteristics of each zone as follows: · Zone 1 –Nyanza and Western Provinces: a high -need zone as determined by the high HIV prevalence, high fertility and infant mortality ratios. In addition these provinces are in a malaria endemic zone. · Zone 2 –Nairobi and Coast Provinces: considered a moderate need zone with large urban settings. Tana River district is not included under this zone. · Zone 3 –Rift Valley, another moderate need zone. This zone excludes Turkana and Samburu districts in the north. · Zone 4 - considered a transitioning zone and covers Central and Eastern Provinces but omits the districts of Marsabit, Moyale and Isiolo. · Zone 5 –the Northern Arid Zone, covering all North Eastern Province and including: Turkana and Samburu in Rift Valley; Marsabit, Moyale and Isiolo in Eastern Province; and Tana River in Coast. A–20 USAID/Kenya expected that in each APHIAplus zone, activities would in￾clude high impact, cost-effective inter￾ventions that took into consideration the following: · Supporting complementary technical areas tailored and designed for the proposed zone(s); · Working through provincial government structures (to ensure coordination) to pro￾vide support at district and community levels; · Linking USAID’s service deliv￾ery support with USAID’s systems strengthening activi￾ties; · Ensuring rapid deployment, smooth transition and conti￾nuity of essential care services to critical populations (e.g. an￾ti-retroviral therapy (ART) clients, orphans and vulnerable children (OVC), etc.) without disruption; · Coordinating vertically, throughout the Kenyan government structure, and horizontally, with other US Government and Kenyan partners; · Working within the government health system in the delivery of health services; and, · Building strong partnerships with effective Kenyan NGOs, faith based organizations (FBOs), the private sector, and other implementing partners essential for sustained service delivery Working with the government and other partners, APHIAplus Service Delivery projects would support marginal￾ized, vulnerable and underserved populations including youth, most-at-risk-populations (MARPs), people living with HIV (PLWHs) and those on ARVs, orphans and children affected by AIDS; women of reproductive age (pregnant and post-partum women), highly vulnerable girls, neonates and infants. The project works through the GoK struc￾ture at the national, provincial, district and community levels. APHIAplus is part of a larger US Mission effort, col￾laborating with other stakeholders. A.1.2. Program Objectives of APHIAplus The result areas of USAID/Kenya’s Results Framework (above) to which the APHIAplus Service Delivery projects contribute are: Result 3: Increased use of quality health services, products and information Result 4: Social determinants of health addressed to improve the well-being of targeted communities and popula￾tions A.2. STATEMENT OF OBJECTIVES A.2.1. Evaluation Purpose Under this task order, MSI shall conduct a final Performance Evaluation of the APHIAplus Nairobi Coast program. Findings and recommendation will inform USAID, the GOK and other stakeholders on what has worked, what has not worked and why. Recommendations will also shape strategies, approaches and activities for the follow-on program design. The period of evaluation is April 2013. A–21 A.2.2. Evaluation Objectives and Key Evaluation Questions The main objective of this final evaluation is to assess the performance of the APHIAPlus Nairobi Coast Project. The specific objectives are: i) To conduct an in-depth qualitative assessment of technical and management approaches, coordination with host country and other stakeholders and support to country health systems. ii) To conduct a quantitative review of project achievements based on assigned targets in HIV/AIDS, FP/RH and Child Survival activities. The following questions are proposed and shall be refined by MSI in consultation with the technical team for use during the interviews: 1. Approach i) Was implementation according to the original proposal? If not, what was different? ii) What is the perception of GOK/Ministry of Health, project staff and other key stakeholders of the ability of the integrated APHIAplus model in approach, geographic coverage and management structure to achieve key outcomes under result 3 and 4 of the 2010-2015 framework? iii) To what extent were project activities aligned with the Government of Kenya (GoK) priorities as outlined in National Strategic Plans/Annual Work Plans/Annual Operational Plans over the per￾formance period? 2. Project Outputs and Outcomes: i) To what extent has the APHIAplus Nairobi/Coast project contributed to increased use of quality health services, products and information on a sustainable basis? Were the key expected out￾comes achieved against what was proposed? ii) What strategies worked and what did not work for successful implementation and achievement of key outcomes in Result 3 and 4 of the Implementation Framework? What lessons can be learnt from the project and key stakeholders’ experience to contribute to future design of similar projects? What do GOK/other stakeholders perceive the value of APHIAplus model/activities to be? What should be continued and/or dropped? iii) What sustainable models/ activities in service delivery and health care systems were established, built and/or strengthened at the district, health facility and/or community level towards improved and sustainable health care delivery? B.1. PERIOD OF PERFORMANCE The period of performance of this task order will start from May 6, 2013. The evaluation will take place in in Nai￾robi and the Mombasa Coastal area where the APHIAPlus program has been operating. C.1. ACCOUNTING AND APPROPRIATION DATA BBFY EBFY Fund OP Prog Area A–22 Dist Code Prog Elem SOC Amount $ ANNEX: Table 1: USAID/Kenya Zonal Descriptions by Need Maps of proposed sites Annex: Table 1: USAID/Kenya Zonal Descriptions by Need High Need Moderate Need (With Urban Areas) Moderate Need Rift Valley “Transitioning” Northern Kenya (Arid Lands) Nyanza Western Nairobi Coast Rift Valley Central Eastern North East- ern Population (mil￾lions) 5.2 4.5 3.2 3.1 8.9 3.9 5.5 1.5 Total Fertility Rate (/woman) 5.4 5.6 2.8 4.8 4.7 3.4 4.4 5.9 HIV Prevalence (%) 14.9 5.4 8.8 8.1 6.3 3.6 4.6 0.8 Population At-Risk for Malaria – Moderate/High Endemicity 2.84 m 2.56 m -- 120,000 30,000 -- -- -- TB- All Cases Prev (/100,000 pop) 419.1 217.5 580.9 367.4 202.9 276.3 278.4 173.0 Health Facility Births (%) 44.2 25.3 89.4 44.4 32.9 73.0 42.8 17.3 Neonatal Mortali￾ty Rate (/1000 LB) 39 24 48 44 30 31 31 33 Infant Mortality Rate (/1000 LB) 95 65 60 71 48 42 39 57 Under Five Mor- tality Rate (/1000 LB) 149 121 64 87 59 51 52 80 Child Stunting (% < 2SD) 30.9 34.2 28.5 39.1 35.7 32.4 41.9 35.2 Major Funding Source to Zone HIV/Malaria HIV HIV/Malaria HIV HIV FSI/Water Additional Fund￾ing Sources TB-Nyanza, FP, MCH, WASH TB, MCH, WASH TB, FP, MCH FP FP-Eastern FP, MCH, HIV Potential Funding Level ++++ +++ +++ +++ ++ + SOURCE: KAIS 2007, DHS 2008/9, KNBS – 2009 Population Projection; DMC data 2009; TB 2009; MoDoNKOAL Vision 2030 Annex Proposed Evaluation Sites A–23 ~~~JQ I KENYA Constituency Boundary Courl:y Bo...-.daries Highwoy Patk 2.1%-5.0% 5.1%-7.0% 7.1% .. 9.0o,(, Over So. not imply official endorsement or acceptance by the U.S. Government. A–24 ~~~JQ I KENYA """ unda11yi """" Sub ""dis~ ict Hospital (vVundanyi Community UniQ OPH Project Evaluation-Coast Health Facilities _) __..-'------ • County Hea.dque.rt111rs • Other Towns 0 Health Faellity County 80111daries Roads Highw.-y Major Road Park 2.1%-5.0% 5.1%-7.0% 7.1% .. 9.0o,(, Over So,<, ~' '~- Source: NACC 2011 Estiniat~ Prepared by: USAIO KEH YA GIS Facility Nairobi. February 2013 KWALE iliilfililj illl j 0 10 20 30 Kilometers The boundaries and names used on this map d:> not imply official endorsement or acceptance by the U.S. Government. A–25 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Annex B: Schedule The following schedule is proposed to complete the evaluation: APHIAplus N-C Eval Kenya Support Program June 2013 Sun Mon Tue Wed Thu Fri Sat 9 10 11 12 13 14 15 16 17 18 19 20 21 22 Int’l team members arrive in NBO 23 24 25 26 27 28 29 30 July 1 2 3 4 5 6 A–26 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 7 8 Meeting with USAID/ Approval of Method￾ology and Workplan 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 August 2013 Sun Mon Tue Wed Thu Fri Sat 28 29 30 31 Aug 1 2 3 4 5 6 7 8 Debriefing to Partner 9 Debriefing to USAID & GOK & Initial Analysis 10 A–27 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 Draft Report to USAID 28 29 30 31 Sun Mon Tue Wed Thu Fri Sat Sept 1 2 3 4 5 6 7 8 9 10 Comments from USAID 11 12 13 14 15 16 17 18 19 20 21 A–28 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 22 23 24 25 Final Report to USAID 26 27 28 A–29 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Annex C: Getting to Answers Program or Project: APHIAplus Nairobi / Coast Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling or Selection Ap￾proach Data Analysis Meth￾ods Data Source(s) Method 1. To what extent has the program met its goal: to build the capacity of the people and member as￾sociations of Laikipia to manage their natural re￾sources, including range￾lands, water and forests? Yes/No LWF and USAID documents Secondary Re￾sources Beneficiaries Key Stakeholders (LWF, USAID, Dutch Embassy, KFS, KWS, WRUA, Water Municipali￾ties, Land Owners, Mpala Research Center, etc.) Document Review Group Discussions KIIs KIIs purposively selected for rele￾vancy to project User Groups for GDs will be pur￾posively selected across water, for￾est, rangeland consideration to covering a range of geographical contexts; best and worst case group; and years of activ￾ity. Pattern/Content Analysis Comparison Analysis (Before and After) Response Conver￾gence/Divergence Analy￾sis Mix Methods Integra￾tion/Findings Synthesis X Description X Comparison2 X Explanation3 2. To what extent are the various monitoring sys￾tems established as part of this program effective and transferable? 3. To what extent is the monitoring information used in decision-making at the forum and com￾munity level? Yes/No LWF and USAID documents Secondary Re￾sources (on Moni￾toring Systems; out￾side best practices) Beneficiaries Key Stakeholders (LWF, USAID, Mpala Research Center, KenWeb, other users of the Document Review KIIs GDs Comparison Analysis (to best practice/outside systems) Pattern/Content Analysis Response Conver￾gence/Divergence Analy￾sis Mix Methods Integra￾tion/Findings Synthesis X Description X Comparison X Explanation 2 Comparison – to baselines, plans/targets, or to other standards or norms 3 Explanation – for questions that ask “why” or about the attribution of an effect to a specific intervention (causality) A–30 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Evaluation Questions Type of Answer/ Evidence Needed Methods for Data Collection Sampling or Selection Ap￾proach Data Analysis Meth￾ods Data Source(s) Method monitoring infor￾mation) 4. What external factors played a significant role in affecting, positively or negatively, the underlying challenges the project sought to address, and what was done to adjust the project design to those factors? (e.g. policy environment, British Ar￾my movement, drought, conflict, etc.) Yes/No LWF and USAID documents Secondary Re￾sources Beneficiaries Key Stakeholders (LWF, USAID, Dutch Embassy, KFS, KWS, WRUA, Water Municipali￾ties, Land Owners, Mpala Research Center, etc.) Document Review Group Discussions KIIs Pattern/Content Analysis Response Conver￾gence/Divergence Analy￾sis Mix Methods Integra￾tion/Findings Synthesis X Description Comparison X Explanation 5. In which ways has the program made a differ￾ence to women, men and youth through its inter￾ventions on water, for￾ests and rangelands? Yes/No LWF and USAID documents Secondary Docu￾mentation Beneficiaries Key Stakeholders (LWF, USAID, Dutch Embassy, KFS, KWS, WRUA, Water Municipali￾ties, Land Owners, etc.) Document Review Group Discussions KIIs Pattern/Content Analysis Comparison Analysis (Before and After) Response Conver￾gence/Divergence Analy￾sis Mix Methods Integra￾tion/Findings Synthesis X Description X Comparison X Explanation A–31 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Annex D: Criteria to ensure the quality of the evalua￾tion report A–32 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Annex E: Collaboration between APHIAplus Nairo￾bi/Coast and national mechanisms National Mechanism How do we coordi￾nate What do they do What do we do What do we do together 1. AfyaInfo Direct coordination between RMU and AfyaInfo (monthly) Support HIS and CBHIS in standardization and har- monization of tools Support in development and continuous updating of DHIS Support in development of training curricula Updating of facility and community code (MFL and M-CUL) Orientations on DHIS Entry, cleaning and data anal￾ysis through DHIS Inclusion of AWP in the DHIS Establishing a joint work plan Plan and coordinate intensively 2. FUNZO Coordination at program level They connect ad- hoc with the provin- cial team Organize the training Identify with the province people that need to be trained-training assessment Ensure post training follow up Identify the ToTs 3. Capacity Quarterly meetings at program level Recruit for the ministry at province/county Supervise Train Support in recruitment- during interviews There is a memo- randum between the two partners 4. Assist-URC Quarterly meeting at program level Developed a joint implementation framework Provide technical support to A+ on perationalization of KQMH Training of county team Co-locate one staff in coast Train staff Document best practices Monitor and evaluate activi￾ties Joint funding for activities 5. MSH (LDP) At program level Based on MoU Co-located Provide support to MoH (MOPHS) Connect with the people trained (QI, lab, mentorship, community, SD) Plan a county sup￾port strategy A joint work plan is in place A–33 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal National Mechanism How do we coordi￾nate What do they do What do we do What do we do together 6. HCSM Quarterly meetings Joint work plan Put in place mechanisms (Nationally) then move to the partner SD managers take on board active monitoring, problem spotting at facility level Redistribution of commodi￾ties Attend District level com- mittee meetings Monitor and report on it (reporting rates) Change the world Advocacy Coordinate in the province 7. KENYA PHARMA Provide ARVs to the pro- gram 8. KEMSA Coordination for agents Coordination at province level Currently no formal coordination Quarterly visits to facilities Currently no formal coordi￾nation 9. SCM Program level com- munication on a regular basis Provide ART, Lab moni￾toring reagents and HIV test kits Act as link between facilities and SCM Redistribution Linking with HCSM for commodity reporting Improving storage space Joint field visits with provincial teams 10. FANIKISHA Regular coordination meetings at program level Focal person in the TWG Identify grantees Build capacity of national level grantees Sharing grant man- agement practices 11. Measure (M&E) Main USAID contact for M&E/SI technical support to the MOH Structures Support to Div. CHS, DRH, and Div. HIS, M&E Systems establish￾ments Responsive TA in SI Member of the TWGs on CHS K2DTWG A–34 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal National Mechanism How do we coordi￾nate What do they do What do we do What do we do together 12. Futures Group MoU and coordina- tion teams Develop software on EMR Scale up of EMR Support infrastructure Adopt software that have been developed Complimentary equipment Data reconstruction and follow up Joint planning Problem solving A–35 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal ANNEX B. METHODOLOGY UPDATE Schedule of Facilities, Community Groups, and Individuals Met During the Evaluation Week 1 Clinical Team Community Team KII Team Tuesday, 9 July 2013 St. Bridget’s Dispen￾sary Mbagathi District Hospital Little Sisters of St. France: OVC Kenya Girl Guides Associa￾tion: Youth Community Strategy Commit￾tee: Mbagathi Project Director, NARESA Ministry of Education, Quality Assurance County Director Children’s Services – NBO County Health Director (for￾mer PMO MoMS) – NBO County Health Director (for￾mer PMO MoPHS) – NBO Wednesday, 10 July Mbagathi District Hospital KENWA: PLHIV SAPTA: PWID Mbagathi PLHIV Support Group Dep. Director, Reproductive Heatlh County Director, Gender and Social Development MOH, Epidemiologist and QI AfyaInfo, COP Thursday, 11 July Ngong Road Health Center Ngando Community Unit Kabiru Youth Group Haki Community-Based Or￾ganization: OVC Education through Listening Session with Youth County Public Health Nurse County Training Coordinator District Gender and Social De￾velopment County Community Focal Point Friday, 12 July Meeting with DHMT Kangemi Health Center Westlands Health Kangemi Health Center Community Unit Kagemi Youth Group Kagemi OVC and Mama Groups Pathfinder COP Capacity COP PASCO – NBO County County Lab Tech (NBO) A–36 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Center Deep Sea Project: Communi￾ty-Based Prevention with Posi￾tives Week 2 Clinical Team Community Team KII Team Monday, 15 July Meeting with the County Health Management Team, Kilifi District Hos￾pital The Omari Project: PWUD Meeting with Mentor Mothers Meeting with PLHIV sup￾port group at Kilifi District Hospital Former PMO for COAST Kilifi County Nursing Officer County Director for Children’s Services County Agriculture Officer County Nursing Officer APHIAplus Prevention Team Tuesday, 16 July Mtepeni Dispensa￾ry MSM peer educators at Kilifi District Hospital Kilifi Township CHW Self- Help Group and OVC OVC support group for HIV-infected children Kidutani Mawamba: eco￾nomic livelihoods Usaidizi Nyumbani: OVC School Heatlh Coordinator County Cooperatives Officer APHIAplus Livelihood Team Gender and Social Development Officer APHIAplus RMU team APHIAplus Gender Officer Wednesday, 17 July Mtwapa Health Center Meeting with CHWs in Chasimba Youth activities in Chasimba Meeting with PLHIV sup￾port group Meeting with PMTCT sup￾port group AMREF Chasimba Local Chief and Elders Gender and Social Development Coordinator Thursday, 18 July Meeting with the Mombasa Country Director and County Health Management Team Mlaleo Facility Reach Out: PWUD City Council of Mombasa Kaderhoi Dispensary: PWUD Kisauni Drop In Center: FSW Former DMOH Kilifi DCO Children’s Officer County Agriculture Officer FIDA Mombasa WOFAK A–37 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Sita Kimya demonstration: GBV Dream Achievers: youth group Discussion with the Gender Working Group Former Provincial Nursing Officer Friday, 19 July Coast Provincial General Hospital Meeting with ICRH Meeting on Family Matters and Gender Meeting with PMTCT and Adolescent Support groups Mombasa HIV Clinic: MSM Healthy Choices II: youth intervention Meeting with Male Champi￾ons Kilindi DHMT Nutrition Officer Kaloleni SOLWODI RH Trainer CIPK Week 3 Clinical Team Community Team KII Team Monday, 22 July Meeting with the County Health Management Team, Wundanyi Sub- District Hospital OVC Caregivers in Wundanyi TAYFCAD Youth Group PLHIV Male Support Group Wundanyi CU Male Champions Network District Children’s Officer Wundanyi Voi – District Youth Officer District Agriculture Extension Of￾ficer – Wundanyi APHIAplus Deputy Director RH Coordinator Community Strategy Focal Person Tuesday, 23 July Mbale Health Cen￾ter HEART NGO Sagalla Youth Group Sagalla Economic Livelihood Group Rauka Group (TBA, PMTCT) Voi Hospital Nursing Officer District Cooperatives Officer – Taveta Sauti ya Wanawake, chairlady District Agriculture Extension Of￾ficer, Sagala APHIAplus Taita Taveta County A–38 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Cluster Team Wednesday, 24 July Travel to Nairobi Thursday, 25 July Dandora II Health Center Ngaira Rhodes Dispensary Kaloleni Unique Support Group for PLHIV Bar Hostess Association for MSM Bar Hostess/HOYMAS – heads of organizations Male Champions for Family Plan￾ning in Kenya (NBO) FHI360 Country Director Friday, 26 July Mukuru Kwa Reu￾ben Facility Reuben Discordant Couple Support Group Reuben HBC – CHWs Reuben OVC Reuben GBV Working Group Kenyatta National Hospital GBV Services HCM Project PSI FANIKISHA Saturday, 27 July Coast PMO (former) Week 4 KII Monday, 29 July County HMIS Director of Children’s Services NACC NASCOP CLUSA Outreach Services Team CMMB Service Delivery Team Crosscutting/Gender Team HSS Team MSH LGM NCPD Childfund MEASURE Futures – EMMR Tuesday, 30 July USAID SUPKEM NCCK Wednesday, 31 July Thursday, 1 July KCCB Breakdown of Facilities and Community Groups A–39 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Facilities Public Facilities 15 Private/Religious 2 Community Groups (By Target Popula￾tion) TOTAL = 35 OVC 6 PLHIV 7 Youth 4 FSW 1 MSM 3 PWUD 3 Community Unit groups (CHC/CHW) 11 Breakdown of Key Informant Interviews Sector/Focus Area National County Sub-County Totals Health 4 17 4 25 Consortium Teams 14 15 Networks 4 5 National TA 7 7 Gender 3 1 4 OVC 1 2 2 5 Result 4 (Ag, Ed, Coop, Youth) 1 Ed 2 Ag, 1 Coopera￾tives 1 Ed, 1 Youth, 2 Coopera￾tives 2 Ed, 2 Ag, 3, Coop, 1 Youth = 8 Other 4 4 Subgrantees 6 6 TOTAL: 79 A–40 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal METHODOLOGY AND WORKPLAN—JULY 4, 2013 A. INTRODUCTION Project Summary: The AIDS Population and Health Inte￾grated Assistance – People-Centered, Leadership-Focused, Universal Access, and Sustainability Project (APHIAplus) Nai￾robi/Coast is a cooperative agreement (Agreement Number 623–A–11–00009) with Pathfinder International. The agree￾ment is managed by USAID/Kenya’s Office of Population and Health (OPH). The three-year, $55 million award began on January 1, 2011, and ends on December 31, 2013. This project is one of five re￾gional service delivery mechanisms under the APHIAplus Service Delivery awards. The APHIAplus service delivery mecha￾nisms were preceded by APHIA II pro￾jects that covered all eight provinces of Kenya. The five APHIAplus projects were awarded in January of 2011 after an ex￾tensive redesign process that began in late 2009 following an assessment of USAID/Kenya’s health portfolio. This re￾design process also resulted in a new USAID/Kenya Health Implementation Framework 2010–2015. The Implementation Framework envi￾sioned these projects supporting delivery of critical health services at the provincial and district levels while working in coor￾dination with USAID-funded national-level technical assistance partners.* Within the Implementation Framework, the APHIAplus design focused on Result 3: Increasing Delivery of Quality Health Services and Re￾sult 4: Addressing the underlying social determinants of health of this Implementation Framework. USAID/Kenya expected that in each APHIAplus zone, activities would include evidence-based high im￾pact, cost-effective interventions that took into consideration the following:† * Request for Applications (RFA), USAID/Kenya APHIAplus, Health Service Delivery projects April 23, 2010. † Evaluation Scope of Work; These are APHIAplus assumptions and a critical part of a sustainability analysis. We will test these assumptions by looking for evidence on whether they held true or not during the course of Project implementation. This project directly supports USAID/Kenya’s Im￾plementation Framework 2010-2015 Results: Result 3: Increased use of quality health services, products, and information 3.1. Increased availability of an integrated package of quality high-impact interventions at community and health facili￾ty levels 3.2. Increased demand for an integrated package of quality high-impact interventions at community and health facili￾ty levels 3.3. Increased adoption of healthy behaviors 3.4. Increased program effectiveness through innovative ap￾proaches Result 4: Social determinants of health addressed to improve the well-being of targeted communities and populations 4.1. Marginalized, poor, and underserved groups have in￾creased access to economic security initiatives through coordination and integration with economic strengthen￾ing programs 4.2. Improved food security and nutrition for marginalized, poor, and underserved populations 4.3. Marginalized, poor, and underserved groups have in￾creased access to education, life skills, and literacy initia￾tives through coordination and integration with educa￾tion programs 4.4. Increased access to safe water, sanitation, and improved hygiene 4.5. Strengthened systems, structures, and services for pro￾tection of marginalized, poor, and underserved popula￾tions 4.6. Expanded social mobilization for health A–41 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal • Supporting complementary technical areas tailored and designed for the proposed zone(s) • Working through provincial government structures (to ensure coordination) to provide sup￾port at district and community levels • Linking USAID’s service delivery support with USAID’s systems strengthening activities • Ensuring rapid deployment, smooth transition and continuity of essential care services without disruption to critical populations (e.g. anti-retroviral therapy (ART) clients, orphans, and vul￾nerable children (OVC) • Coordinating vertically, throughout the Kenyan government structure, and horizontally, with other US Government and Kenyan partners • Working within the government health system in the delivery of health services • Building strong partnerships with effective Kenyan NGOs, faith based organizations (FBOs), the private sector, and other implementing partners essential for sustained service delivery Working with the government and other partners, APHIAplus service delivery projects support margin￾alized, vulnerable, and underserved populations, including most-at-risk-populations (MARPs); people liv￾ing with HIV (PLHIV) and those on ARVs; orphans and vulnerable children (OVC) affected by AIDS; women of reproductive age (pregnant and post-partum women); and highly vulnerable girls, neonates, and infants. The implementation strategy focuses on achieving two levels of results: 1) at the facility and community levels working through national, provincial, and district GOK structures and 2) with com￾munities. APHIAplus Nairobi/Coast reported four crosscutting themes in the program: 1) gender, 2) youth, 3) whole market-approach, and 4) equity. APHIAplus Nairobi/Coast follows the tenets of the Global Health Initiative (GHI) and international man￾date. These include the following principles, as captured in the Program Description in the Request for proposals:* 1. Ensure country-led, country-owned, and country-managed 2. Align Kenyan, U.S. government, and development partner strategies 3. Invest in leadership, capacity, and systems for long-term sustainability 4. Maximize a client-centered approach through integration of services and systems 5. Increase involvement of the private sector in health care delivery 6. Ensure strategic collaboration and coordination 7. Manage for results with mutual accountability APHIAplus Nairobi/Coast works with a consortium of partners. Partners are co-located in the same office in Nairobi to enable coordination and enhance management. The following organizations consti￾tute the partnership: • Pathfinder International (principal implementer) partners with governments, NGOs, and the private sector to promote best practices and state-of-the-art approaches in Family Planning and Re￾productive Health. Pathfinder’s comprehensive approach to HIV/AIDS programming promotes syn￾ergies among key intervention areas including prevention, stigma reduction, ART, HBC, and PMTCT. Being that these assumptions largely address alignment with local systems and systems strengthening, the results of this analysis will greatly inform the sustainability question of this evaluation. *Request for Applications (RFA), USAID/Kenya APHIAplus, Health Service Delivery projects April 23, 2010. A–42 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal • ChildFund International is committed to child-centered change that leads to healthy and secure infants, educated and confident children, and skilled and involved youth. Child Fund brings global and national experience in family-centered OVC care, and community engagement for children’s health. • CLUSA works with communities to identify and address health concerns. CLUSA’s training activi￾ties build organizational and management capabilities of communities to take responsibility for health services and raise local resources for community health, social, and economic initiatives. • Network of AIDS Researchers of Eastern and Southern Africa conducts HIV research and supports evidence-based policy formulation and program implementation. NARESA promotes inter￾disciplinary research and provides technical leadership in HIV and is a pioneer of PMTCT in Kenya. • Population Services International uses innovative, culturally-sensitive approaches to raise awareness for healthy behaviors, generate demand for health products, and promote franchised pri￾vate providers. PSI/Kenya conducts programs in malaria, Family Planning/Reproductive Health, HIV/AIDS, and child survival. The partners are supported by a number of local sub-grantees. In addition to sub-grantees, APHIAplus Nairobi/Coast works with USAID-funded national mechanisms. The project implements three overarching strategies: 1. The Foundation Package of interventions comprises key support to delivery of Kenya Essential Package of Health (KEPH) across Zone 2 and continues to be implemented in all sub-zones. These systems’ strengthening activities provide the foundation of quality service delivery and contribute to sustainable quality improvement and related health impact. KEPH contains evidence-based best prac￾tices and services required for the general population. Support to efficiently and effectively deliver KEPH is thus critically important to the achievement set national health outcomes as outlined in the National Health Strategic Plan. 2. The Most Vulnerable Groups Package of investments targets underserved, poor, or marginal￾ized groups who experience serious barriers in claiming their rights to health and health care. The most vulnerable groups’ package consists of tailored approaches and targeted interventions that are responsive to the needs of specific groups within the sub-zones. This package, which focuses on vul￾nerable populations, thus addresses barriers to access to KEPH and related health services. 3. The Partnerships Package features formalized partnerships and linkages to leverage limited re￾sources and maximize the sustainability of improved health outcomes. This package ensures appro￾priate coordination with local and national level service providers in order build synergies to achieve the anticipated program outcomes. The project favors rapid acceleration and scale-up of essential health promotion, prevention, and curative service coverage for the general population and for vul￾nerable, marginalized, and underserved groups. The improved service delivery mandate is being im￾plemented by emphasizing sustained quality improvement, integration, and leveraging of existing re￾sources and programs for maximum impact. Crosscutting project approaches include service integration, linkages between levels of facility care, facil￾ity and community services, and quality assurance/quality improvement. The project has “regionalized” these strategies to respond best to the health needs and contexts of Nairobi and Coast. A–43 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Evaluation Summary The purpose of the evaluation is to assess the performance of the APHIAPlus Nairobi/Coast and it is intended primarily for accountability and learning. The two specific objectives include: 1. To conduct an in-depth qualitative assessment of technical and management approaches, coor￾dination with host country and other stakeholders, and support to country health systems. 2. To conduct a review of project achievements based on assigned targets in HIV/AIDS, FP/RH, and Child Survival activities. This purpose and objectives will be achieved through answering the following five evaluation questions (more detail on approaches can be found in the Evaluation Design section below):* 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in Result 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 4. To what extent was the APHIAplus model appropriate for achieving key outcomes under Re￾sult 3 and 4 of the Implementation Framework? 5. What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? Recommendations will shape strategies, approaches and activities for the follow-on program design. In addition, the evaluation team will highlight relevant lessons intended for any future health activities with one to two pages of the report dedicated specifically to lessons learned. The primary audience for the evaluation is USAID/Kenya and the Government of Kenya. Secondarily, the report is intended for Path￾finder, USAID/Washington and other interested implementing partners. In order to answer the five evaluation questions a number of issues will be important to identify. The team has taken each question below and described considerations, definitions, frameworks, and other issues for the purpose of clarity. B. EVALUATION DESIGN Overall Framework for Analysis The team will use the following five points as a framework for addressing and critically answering all of the evaluation questions. These points will ensure that all cross-cutting issues are addressed within each question and throughout the analysis. * Evaluation Scope of Work, in the Evaluation Design section below Frameworks, considerations and lines of inquiry will be examined in detail for each question A–44 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Through the answers to the five evaluation questions the team will as￾sess if strategies and approaches successful in: 1. Contribution to Result 3 and 4 outcomes and performance against targets 2. Achieving the three overarching strategies (Foundation Package, Most Vul￾nerable Populations Package, Partnerships Package) 3. Addressing technical issues specific to the health themes (e.g., Family Plan￾ning, Maternal, and Child Health) 4. Addressing cross-cutting approaches: integration, linkages, quality assur￾ance, and quality improvement (QA/QI), gender,* innovation, regionaliza￾tion, equity, and government and community ownership of the program 5. Establishing and/or strengthening systems and processes at the district health facility and/or community level to foster sustainability Evaluation Question 1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? Considerations: For the purpose of this evaluation, the team has defined “effectiveness” as performance against targets, coverage of services by targeted population, and general alignment of results to the specific level of fund￾ing (as details become available) and to project documents (RFA, program description, work plans, quar￾terly reports). • For performance against targets, the team will be guided by USAID’s annual reporting guidelines, which considers 10 percent above and below set targets to be satisfactory performance • For coverage by target population, the team will be guided by Universal Access standards (80 percent) • For alignment of results to level of project funding and project documents, the team will map/compare funding stream data, project results (PMP results against targets), and project re￾ported/planned activities For the purpose of this evaluation “outcome indicators” will be defined as indicators that align to USAID Results 3 and 4 and their intermediate results. The team will include appropriate PEPFAR New Genera￾tion Indicators and maternal, newborn, and child health (MNCH), tuberculosis (TB), and family planning (FP) indicators under this definition (as reported by the project). Target populations and key services for further analysis may include some or all of the below: • MNCH and pregnant women • ART, TB/HIV, care (including pre-ART services) coverage, and PLHIV * Gender will be viewed as a cross-cutting theme to be explored where appropriate throughout answering the evaluation ques- tions. The evaluation team will be responsive to USAID's dual expectations for treating gender appropriately through: a) gather￾ing sex disaggregated data and b) identifying gender differential participation in/benefits from aspects of the progr am where differences on this basis are possible. A–45 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal • TB and entire population • FP and women of reproductive age • Service coverage and MARPs (sex workers [SWs], men who have sex with men [MSM], and people who inject drugs [PWID]) • OVC services by OVC due to HIV • HIV testing and counseling (HTC) for those 15–49 years of age To answer the evaluation question, the team will use nine quarters of performance data in the following analyses: • Initial analysis of performance against “regionalized“ targets for APHIAplus indicators as report￾ed in their Performance Monitoring Plans (identifying indicators which have been reported against since year 1 and are still active in year 3). • Initial analysis of trends using APHIA II data where indicators have continued through to APHIAplus. • Analysis of performance on indicators to funding levels, project results (PMP results against tar￾gets), and project reported/planned activities. Additionally, the team will conduct pattern analyses and frequencies of data gathered from key informant and stakeholder interviews (including facility and community-based checklists) to triangulate data from all sources and tools to determine convergence/divergence of findings and validate conclusions. As appropriate data is available, the team will also conduct the following: • Analysis of demand trends and use of specific services per target audiences (first by project defini￾tion and then by national/ international guidance) • Analysis of coverage of services (coverage within population, addressing gap/need) with a focus on key populations and services (as defined by the project) Evaluation Question 2: What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in Result 3 and 4? Why? Considerations For the purpose of this evaluation “successful implementation” will be defined as meeting the basic package as defined by the Kenyan Essential Packages for Health (KEPH) and other national guidelines (supplementing with international guidelines as necessary) and demonstrated responsiveness to each of the cross-cutting approaches. The team will assess the project’s defined packages as the three overarching strategies within the pro￾ject result areas: • Foundation (clinical) package (Result 3) • Most vulnerable population package (Result 3 and 4) • Partnership package (Result 3 and 4) To answer this evaluation question, the team will: • Conduct a mapping/comparison of program documents (strategies/activities indicated in the RFA, PD, work plans) to performance against targets. A–46 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal • Conduct pattern analysis and frequencies of data gathered from key informant and stakeholder in￾terviews (including facility and community-based checklists) to triangulate data from all sources and tools to determine convergence/divergence of findings and validate conclusions. Evaluation Question 3: To what extent were the project’s annual work plans and strategies therein in￾formed by new programmatic evidence in the Health Sector? Considerations For the purpose of this evaluation “programmatic evidence” will be defined as policies, guidelines, inno￾vations and evidence-based interventions (EBIs), and internal program-generated data. The threshold for the issue of guidance is July 2012 for program to be accountable to using guidance for work plans and implementation as of April 2013.* As such, this programmatic evidence may include • PEPFAR guidelines • Normative bodies guidelines (e.g., UNAIDS, WHO) • Issues/modification of national policy and guidelines • The project’s own monthly, quarterly, semi-annual, annual data reports, QA/QI data and research, and assessments • Lessons learned, best practices, and EBIs (internal and external) • Evaluations/assessments (e.g., the APHIA II assessment and end of project reports) • Kenyan innovations (e.g., Women’s Economic Empowerment and Justice Initiative [WEIJI]) • Urban-focused research To answer this question, the team will • Conduct a mapping of program documents (strategies/activities indicated in the RFA, PD, work plans) to national and international guidelines and policies. This exercise will also indicate project’s strategic planning cycle/process. This mapping will be linked to findings from Evaluation Q1 and Q2. • Analysis of any quality improvement/assurance data captured by project (as available). • Conduct pattern analysis and frequencies of data gathered from key informant and stakeholder in￾terviews (including facility and community-based checklists) to determine if and how data was used for decision-making at the facility or CBO/community level. • Triangulate data from all sources and tools to determine convergence/divergence of findings and validate conclusions. Evaluation Question 4: To what extent was the APHIAplus model appropriate for achieving key out￾comes under Result 3 and 4 of the Implementation Framework? Considerations For the purpose of this evaluation, the team understands the APHIAplus model to be outlined as fol￾lows, and as described in the RFA. * As per initial discussion with USAID A–47 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal APHIAplus focuses on achieving Result 3 and Result 4 from the USAID Five Year Implementation Framework for Health by targeting assistance at the county level and below, and in level 3 facilities and below. The APHIAplus model: • Is predicated upon geographic zoning based on unique challenges that included urban slums and marginalized, poor, underserved, and most-at-risk populations • Utilizes a technical focus that helps to address the underlying health care issues of people living in urban slums and marginalized, poor, underserved, and most-at-risk populations in the geo￾graphic zones, specifically focusing on OVC and MARPS among other vulnerable populations through an integrated approach to high quality technical assistance to the health system and communities. • Is predicated upon the assumption that the APHIA implementing partner will receive timely and high quality technical assistance/complementary programming from national-level technical assis￾tance partners. APHIAplus also works through the following principles: a transition from emergency response to a development model and a renewed focus on country-led, country-managed, and country- owned. The model assumes that APHIAplus builds on the successful interventions of APHIA II, thus max￾imizing USAID’s investments. APHIAplus uses an implementing consortia as its management model. To answer this question the team will • Utilize the mapping exercise conducted as part of Evaluation Questions 1, 2, and 3. • Conduct pattern analysis and frequencies of data gathered from key informant and stakeholder in￾terviews (including facility and community-based checklists) to determine if and how the APHIAplus model characteristics (as listed above) met the needs of beneficiaries. • Triangulate data from all sources and tools to determine convergence/divergence of findings and validate conclusions. The evaluation team acknowledges that the findings and conclusions for this question may be largely based on the triangulation of perceptual data from stakeholders. Evaluation Question 5: What sustainable activities in service delivery and health care systems were es￾tablished and/or strengthened at the district health facility and/or community level? Considerations: The APHIAplus RFA* defines sustainability as the following: Assure country-led, country-owned, and country-managed leadership and ownership as the main drivers of sustainability. To this end, USAID/Kenya promotes the tenets of the 2005 Paris Dec￾laration which encourages countries to define and manage their development policies and strat￾egies. Working closely with the GOK, other donors, and its own partners, the USAID/Kenya program will support country ownership and enable long-term country capacity to plan and *Request for Applications (RFA), USAID/Kenya APHIAplus, Health Service Delivery projects April 23, 2010. A–48 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal manage and evaluate high impact health service delivery program. This will involve assuring close alliances with the GOK, fully engaging civil society to assure that health services meet the needs of people, and expanding involvement of private for-profit commercial sector, private institu￾tions and organizations, and not-for-profit private sector institutions, including professional as￾sociations, nongovernmental organization (NGOs), faith-based organizations (FBOs), and com￾munity-based organizations (CBOs). In the USAID Strategic Framework 2011–15* (page 32), USAID states: Sustainability is achieved when host country partners and beneficiaries are empowered to take ownership of development processes including financing and maintaining project results and im￾pacts beyond the life of the USAID project. For the purpose of this evaluation, the team will focus analyses of sustainability on service delivery and health care system processes, which were established or strengthened at the district health facility and/or community level.† Framework and lines of inquiry  Based on team’s mapping exercise (under Evaluation Q1-3), conduct comparison analysis of re￾ported sustainability to any observed/crosschecked/validated measures/activities  Define trace indicators and compile/collect data as possible through interviews and checklists. Example indicators might be:  DHMT performance  Provider performance  Community engagement in county strategic planning  Conduct pattern analysis and frequencies of data gathered from key informant and stake￾holder interviews (including facility and community-based checklists) to measure against trace indicators.  Triangulate data from all sources and tools to determine convergence/divergence of findings and validate conclusions. Using the considerations presented in USAID’s newly released Design Guidance, the team will provide recommendations for how to incorporate measures and structures for sustainability into future activi￾ties. C. DATA COLLECTION METHODS Data collection methods will involve a mix of quantitative and qualitative methodologies in order to gather key data and information. Methods will include: desk review, data extraction, key informant in￾terviews, semi-structured group interviews, and structured observation checklists/site visits. *USAID Strategic Framework 2011–15 (2011), page 32. †As per USAID guidance during initial meeting. A–49 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Table 1: Data collection methods for the five evaluation questions Data Collection Methods Evaluation Questions Desk Review 1, 2, 3, 4, 5 Data Extraction 1 Key Informant Interviews 1, 2, 3, 4, 5 Semi-structured group interviews 1, 2, 3, 4, 5 Structured Observation/checklists 1, 2, 3, 4, 5 Desk Review will involve a review of documentation provided by APHIAplus Nairobi/Coast and USAID on the program, and secondary research collected (particularly from other major health-related partners such as WHO, UNICEF, and/or UNAIDS).Through the initial audit of these documents, the team will present a set of initial findings in line with the evaluation questions during the Team Planning meeting (June 26, 2013). These initial findings will guide the rest of the methodology development pro￾cess. Data Extraction will be used on both national data (KHIS) and project data (indicator data as well as Quality Assurance and Quality Improvement data). The team will extract the necessary data elements including indicators, results, targets and prepare datasets for analysis (see initial data analysis plan on un￾der Evaluation Question 1). Key informant interviews will be essential to understand the process, information flows, decision- making, and operational engagement of the program with the Ministry of Health, facilities and communi￾ties and other stakeholders. The evaluation team will interview a purposively selected set of key stake￾holders, including USAID, APHIAplus partners, national mechanisms, health-related agencies, community leaders, health service providers, and GOK (national, district, and county levels). As there are multiple stakeholders with distinct roles in the program, a semi-structured interview tool will be developed for different types of stakeholders to ensure adequacy of questions and comparability across interviews. The team discussed the issue of standardizing the language of the questionnaires to ensure that each team is getting the answers in the same way. In addition, the team discussed how to make sure that there are sufficient points to triangulate data from multiple sources (e.g., from national-level and county￾level GOK stakeholders, from national non-governmental stakeholders, and from community leaders). Semistructured group interviews will be used to better understand the function of management teams at the county and community levels as well as community groups and all levels of health workers who have received mentoring or training through the project. Group interviews will be conducted with small homogenous groups (e.g., Community Health Workers, most vulnerable population groups) or heterogeneous groups (e.g., County Health Management Teams, CBO management) See Annex C for an illustrative list.  Facility based interviews will include District Health Management Teams, Community Health Workers, Health Facility Management Team, Community Health Committee, Facility In-Charges and Department Heads (as appropriate to facility tier), including the Tunza network.  Community based interviews will be selected within parameters of timing/logistics and linkages to facilities pre-selected, and include a representation of key targeted audiences (e.g., youth, MARPS, PLHIV, women of reproductive age) at both the CBO level and client level. The team will also select a rural/urban mix as appropriate for each target audience. A–50 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Structured observation will involve two separate checklists to be used at health facilities and with community groups visited. The tools will be based on current GOK guidance (or international stand￾ards) as appropriate. • Facility Services Checklist will be used in health facilities to assess service delivery elements, training and other support provided by APHIAplus • Key Populations Services Checklist will be used with CBOs and community-delivered services to assess service delivery for key populations (as part of the group interview) Sampling Fourteen facilities (plus two additional sites for instrument piloting) were purposively selected to pro￾vide the evaluation team an opportunity to observe various sites to help contextualize data collected as well as meet with service providers. USAID/Kenya OPH provided the sample as part of the Statement of Objectives. The coverage focuses on two program areas: PMTCT and Treatment since facilities that provide these services also provide other health services. Other criteria considered included: 1) geog￾raphy, 2) availability of a community unit attached to the site, and 3) volume of service delivery statistics (a mix of high, moderate, and low volume of service delivery). The list of facilities and community groups to be visited can be found in Annex B. The sample of sites is not large enough to be generalized, but should give the evaluation team an indicative snapshot of pro￾gram implementation at the beneficiary level. D. DATA ANALYSIS METHODS Given the scale of the program, size of the team, amount of data available and to be collected, it is cru￾cial for the evaluation team to design a data analysis plan at the onset, whereby the tools and instru￾ments used for data collection will feed into the data processing and synthesis of findings, and reporting can quickly be undertaken. The plan allocates two weeks for data analysis and processes given the large quantity of data expected from the fieldwork. Table 3. Data analysis methods for the five evaluation questions Data Analysis Methods Evaluation Questions Descriptive Statistics (frequencies, trend analysis, cross tabula￾tions, pivot tables) 1, 2, 3, 4, 5 Mapping (Comparison) Analysis 1, 2, 3, 5 Pattern/Content Analysis 1, 2, 3, 4, 5 Response Conversion/Diversion Analysis 1, 2, 3, 4, 5 Triangulation 1, 2, 3, 4, 5 Descriptive Statistics: In assessing the project PMP data and indicators, national level data and project QI/QA data, descriptive statistics will be used to analyze quantitative data and demonstrate achievement across the results framework. Other types of descriptive statistics, such as frequencies, trend analysis (time-series), cross tabulation and pivot tables will be used to describe data in the most appropriate way. For data gathered from structured portions of interviews and checklists frequencies and other de￾scriptive statistics will be used for analysis. Mapping (Comparison) Analysis: Mapping (comparison) analysis will be employed at six different levels. A–51 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal • Planned versus actual (project proposal, work plans, and budget breakdown and reported re￾sults) when looking at project achievement of expected outcomes (Q1) • Comparison of before and after (when baseline information is available) when looking at project achievements and trends over time (trend analysis above) (Q1) • Statistical comparison/validation will help support (or counter) findings based on monitoring data (Q1) • Planned versus reported when looking at strategies for successful implementation (Q2) • Actual versus best practice when looking at project strategies and interventions aligning with new programmatic evidence and current guidance (Q3) • Planned versus actual/achieved when looking the appropriateness of the APHIAplus model for achieving key results (Q4) • Comparison and synthesis of definitions and descriptions across documents to determine defini￾tions for key terms on which to build further comparison and analysis of collected data (Q4 and Q5) Pattern/Content Analysis: For qualitative data, patterns will be examined so that comparisons can be made between respondents and contexts to determine the pattern of effect of APHIAplus Nairo￾bi/Coast. This will involve broad patterns and a more detailed examination of how different respond￾ents answered the same question, e.g., different community groups, government stakeholders. The team compared each of the interview/observational tools against the data analysis tools. The team then made minor revisions to the interview/observational tools to ensure that the data requested—and the language in which questions within the tools are framed—meets the requirements of the triangula￾tion process and yet is phrased in a manner appropriate to the specific type of stakeholder. After discus￾sion, the team did not change the language in the interview/observational tools regarding stakeholders’ opinions of improvement. This is because we expect respondents to express opinions of needed im￾provement based on their specific knowledge of the thematic area in question. Also, the pretesting demonstrated that the questions are worded in a manner that achieves the desired responses. In gen￾eral, it is important to note that the format of a semi-structured discussion guide is designed to elicit responses with some degree of flexibility whilst maintaining a certain level of structure. Response Convergence/Divergence Analysis: The team will review data collected to determine where there is significant response convergence from the varied stakeholders and beneficiaries. Where divergence is found, the team will follow-up to better understand the context and reasons for diver￾gence in facts, perceptions or opinions. Triangulation: Using a mixed methods approach, data from various methods will be integrated to ar￾rive at findings. Where different methods produce conflicting evidence, the evaluation team will, to the extent possible, double back to examine why these data conflict, as well as weight the data from the var￾ious methods in terms of strength in validity and reliability. E. Strengths and Limitations of Methodology Data limitations will include the availability of current population levels statistics at various levels for de￾termination of coverage and service use statistics to inform the evaluation. The availability and timeliness of other data and documentation requested will also be a limitation. The sample of facilities was drawn for the team by USAID, while this may not be a direct limitation it should be noted that it may introduce some bias. The small sample size for health facilities and commu- A–52 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal nity groups will also limit the evaluation’s ability to draw generalizable conclusions, but should give the evaluation team an indicative snapshot of program implementation at the beneficiary level. Attribution to APHIAplus specifically will be difficult as the program has served as a follow-on program to past APHIA programs. Further, support such as technical assistance and systems strengthening re￾quire time to take root and produce concrete changes. The evaluation and its findings and conclusions, therefore, will need to focus both on contributions towards systems and sustainability. A–53 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal ANNEX A: Getting to Answers Program or Project: APHIAplus Nairobi/Coast End of Project Evaluation Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appro￾priate) Methods for Data Collection, (e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey) * Sampling or Se￾lection Ap￾proach, (if one is need￾ed) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis) Data Source(s) Method 1. To what ex￾tent has the APHIAplus Nai￾robi/Coast been effective in con￾tributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? Project data￾base/USAID data￾base/APHIA II da￾ta/national data (HMIS) Data extraction Data extraction (Focus on key interventions and key target popula￾tions as determine by initial data run) Site visits (purpos￾ive selection of 14 sites, criteria in SOW) KII (purposive sample) Semi-structured group interviews (Within parame￾ters of tim￾ing/logistics and linkages to facili￾ties pre-selected, sampling will look at representation * Descriptive statistics (Trend analysis, cross tabs, frequency distributions) *Mapping of activities/outcomes - Comparison Analysis *Pattern/Content Analysis *Convergence/Divergence Analy￾sis *Triangulation of KII, data, doc￾uments X Description Project documents (RFA, PD, work plans and Quarterly re￾ports, etc.) Desk review X Comparison† Key informants (USAID, Project Consortium, GOK – national/county, Civil society, LIPS) KII Key Stakeholder groups (CBOs, HFMTs, DHMTs, CHC, CHW, men￾tored cadres) Semi-structured group inter￾views Observation/site vis￾its (14 facilities, CBOs, community sites) Structured observation check￾lists * Data from evaluations are a deliverable and methods should indicated how data will be captured (e,g., USAID requires a transcript for focus groups). † Comparison – to baselines, plans/targets, or to other standards or norms A–54 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appro￾priate) Methods for Data Collection, (e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey) * Sampling or Se￾lection Ap￾proach, (if one is need￾ed) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis) Data Source(s) Method of key targeted audiences (most vulnerable popula￾tions) and ru￾ral/urban mix) 2. What strate￾gies employed within the workplan worked and what did not work for success￾ful implementa￾tion and achievement of key outcomes in Result 3 and 4? Why? Project documents (RFA, PD, work plans and Quarterly re￾ports) Desk review Site visits (purpos￾ive selection of 14 sites, criteria in SOW) KII (purposive sample) Semi-structured group interviews (see above) *Mapping of activities/outcomes - Comparison of planned and re￾ported activities *Triangulation of data *Pattern/Content Analysis *Convergence/Divergence Analy￾sis *Descriptive Statistics (Frequen￾cies from checklists and struc￾tured portions of interviews) X Description X Comparison Key informants (USAID, Project Consortium, GOK – national/county, Civil society, LIPS) KII X Explanation Observation/site vis￾its (14 facilities, CBOs, community sites) Structured observation check￾lists Key Stakeholder groups (CBOs, HFMTs, DHMTs, CHC, CHW, men￾tored cadres) Semi-structured group inter￾views 3. To what ex￾tent were the project’s annual work plans and Guidelines and plan￾ning documents (GOK, USAID, inter￾national); Project Desk review Data extraction (Focus on key interventions and key target popula- *Mapping/comparision of strate￾gies/interventions to ‘program￾matic evidence’ A–55 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Evaluation Questions Type of Answer/ Evidence Needed (Check one or more, as appro￾priate) Methods for Data Collection, (e.g., Records, Structured Observation, Key Informant Interviews, Mini-Survey) * Sampling or Se￾lection Ap￾proach, (if one is need￾ed) Data Analysis Methods (e.g., Frequency Distributions, Trend Analysis, Cross-Tabulations, Content Analysis) Data Source(s) Method strategies therein informed by new programmatic evidence in the Health Sector? documents (RFA, PD, work plans and Quar￾terly reports) tions as determine by initial data run) Site visits (purpos￾ive selection of 14 sites, criteria in SOW) KII (purposive sample) *Descriptive Statistics (Frequen￾cies from checklists and struc￾tured portions of interviews) *Triangulation of data *Pattern/Content Analysis *Convergence/Divergence Analy￾sis X Description Observation/site vis￾its (14 facilities, CBOs, community sites) Structured observation checklist X Comparison Project QI/QA data Data extraction /desk review X Explanation Key informants (USAID, Project Consortium, GOK – national/county, Civil society, LIPS) KII A–56 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 4. To what ex￾tent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementa￾tion Framework? Project documents (RFA, PD, work plans and Quarterly reports, etc.), background documents, etc. Desk review KII (purposive sample) Semi-structured group interviews (see above) *Mapping /comparison of ‘model’ to strategies and out￾comes/results (as articulated in Q1-3) and compari￾son/description of key definitions – ie the Model *Triangulation of data *Pattern/Content Analysis *Convergence/Divergence Analy￾sis *Descriptive Statistics (Frequen￾cies from checklists and struc￾tured portions of interviews) X Description Key informants (USAID, Project Consortium, GOK – national/county, Civil society, LIPS) KII X Comparison Key Stakeholder groups (CBOs, HFMTs, DHMTs, CHC, CHW, mentored cadres) Semi-structured group in￾terviews X Explanation Observation/site visits (14 facilities, CBOs, community sites) Structured observation checklists 5. What sustaina￾ble activities in service delivery and health care systems were established and/or strength￾ened at the dis￾trict health facility and/or communi￾ty level? Background documents (project documents, etc.) and project data Desk review Site visits (purpos￾ive selection of 14 sites, criteria in SOW) KII (purposive sample) Semi-structured group interviews (see above) *Descriptive Statistics (Frequen￾cies from checklists and struc￾tured portions of interviews) *Comparison/description of key definitions -‘sustainability’ *Pattern/Content Analysis *Convergence/Divergence Analy￾sis *Triangulation of data X Description Key informants (USAID, Project Consortium, GOK – national/county, Civil society, LIPS) KII Key Stakeholder groups (CBOs, HFMTs, DHMTs, CHC, CHW, mentored cadres) Semi-structured group in￾terviews X Explanation Observation/site visits (14 facilities, CBOs, community sites) Structured observation checklists A–57 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal ANNEX B: Work Plan/Site Visit Schedule Kenya Sup￾port Program Evaluation of APHIAplus Nairobi–Coast June 2013 Sun Mon Tue Wed Thu Fri Sat 16 17 Desk review 18 Desk review 19 Desk review 20 Desk review 21 Desk review 22 IO arrives 23 BO arrives from Nakuru 24 TPM: Introduction and review of SOW 25 TPM: Meeting with USAID, Pathfinder (IP) 26 TPM: Desk review and G2A 27 TPM: Work on methodology, work plan, and tools 28 TPM: Work on methodology, work plan, and tools Share draft methodol￾ogy work plan with MSI 29 TPM: Work on methodology, work plan, and tools 30 July 1 TPM: Finalize data collection tools BA arrives in the evening 2 TPM: Pre-test in Nairobi 3 TPM: Pre-test in Nai￾robi 4 TPM: Refine tools, and finalize roles and responsibilities Submit Final Method￾ology to MSI (Holiday) 5 TPM: Finalize team itinerary and logistics MSI shares final meth￾odology with USAID 6 A–58 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal July 2013 Sun Mon Tue Wed Thu Fri Sat 7 8 Meeting with USAID at 8:30 – 11:30: ap￾proval of methodology and workplan , Revi￾sions (PM) Vehicles: 2 9 Data Collection: Team 1: St. Bridgets (AM) Mbagathi (PM) Team 2: Community Team 3: KIIs Vehicles: 3 10 Data Collection: Team 1: Mbagathi (all day) Team 2: Community Team 3: KIIs Vehicles: 3 11 Data Collection: Team 1: Ngong (all day) Team 2: Community Team 3: KIIs Vehicles: 3 12 Data Collection: Team 1: Westlands (AM) Kagemi (PM) Team 2: Community Team 3: KIIs End of week report Vehicles: 3 13 14 USAID participation in site visits this week (Cherry-OPH) 7 Team member travel to Kilifi Flight for 7, Vehi￾cles from MBA to KIL 15 Data Collection: Team 1: Kilifi (all day) Team 2: Community Team 3: KIIs Vehicles: 2 Accommodation: 7 in Kilifi 16 Data Collection: Team 1: Kililfi Team 2: Community Team 3: travels to Mombasa for KIIs Vehicles: 2 Accommodation: 4 in Kilifi, 3 in Momba￾sa 17 Data Collection: Team 1: Matepeni Team 2: Community in Matepeni Team 3: KIIs in Mom￾basa Vehicles: 2 Accommodation: 7 in Mombasa 18 Data Collection: Team 1: Mtwapa (AM) Coast PGH (PM) Team 2: Community In Mtwapa Team 3: KIIs in Mom￾basa/Mtwapa Vehicles: 2 Accommo￾dation: 7 in Mombasa 19 Data Collection: Team 1: CPGH (all day) Team 2: Community in Mombasa Team 3: KIIs in Mom￾basa, end of week report Vehicles: 2 Accommodation: 7 in Mombasa 20 Team Meeting Vehicles:2 Accommodation: 7 in Mombasa A–59 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal 21 22 7 team members move to Voi Vehicles: 2 Accommodation: 7 in Voi 23 Data Collection: Team 1: Wundanyi (all day) Team 2: Community Team 3: KIIs Vehicles: 2 Accommodation: 7 in Voi 24 Data Collection: Team 1: Mbale (all day) Team 2: Community Team 3: KIIs Vehicles: 2 Accommodation: 7 in Voi 25 Data Collection: Drive to Mombasa for flight to Nairobi Vehicles: 2 26 Data Collection: Team 1: Dandora (AM) Ngaira (PM) Team 2: Community Team 3: KIIs Vehicles: 3 27 Data Collection: Wrap up of pending issues End of week report Vehicles: 2 A–60 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal August 2013 Sun Mon Tue Wed Thu Fri Sat 28 29 Data Analysis 30 Data Analysis 31 Data Analysis Aug 1 Data Analysis 2 Data Analysis 3 4 5 F/C/R Workshop 6 Data Analysis 7 Dry run presentation Draft of Initial Analysis Report to MSI 8 Debriefing to Partner 9 Debriefing to USAID & GOK Submission of Initial Analysis Report & Raw Data to USAID 10 11 12 Writing of the draft report 13 Writing of the draft report 4 members last day 14 Writing of the draft report (TL & IO) 15 Writing of the draft report (TL & IO) 16 Writing of the draft report (TL & IO) 17 Draft report to MSI IO to leave for US 18 19 MSI review draft, pro￾vide comments 20 Team fix the draft report 21 Team fix the draft report 22 MSI Review/edit 23 MSI Review/edit 24 25 26 MSI Review/edit 27 Draft Report to USAID 28 29 30 31 September 2013 A–61 Use or disclosure of data contained on this sheet is subject to the restriction on the title page of this proposal Sun Mon Tue Wed Thu Fri Sat 1 2 3 4 5 6 7 8 9 10 Comments from USAID 11 Team address the comments 12 Team address the comments 13 Team address the comments 14 15 16 Team address the comments 17 Final Draft sent to MSI 18 MSI to review final draft 19 Draft with Editor 20 Draft with Editor 21 22 23 Draft with Editor 24 Final Review my MSI 25 Final Report to USAID (morning) 26 27 28 A–62 ANNEX C. RECOMMENDATIONS Annex C.1. Recommendations for USAID The evaluation team recommends that USAID extend the APHIAplus Nairobi/Coast mechanism for a full five years, as it will allow the following: • Time for the Mission to design a robust five-year assistance program in health that is aligned with the CDCS and new Constitution. • All of the APHIAplus mechanisms to end concurrently for a country-wide transition to a new assistance program. • Substantial disruptions in service delivery and activities to be mitigated through subawards and by local implementing partners. • New awards to be in place 3–6 months before the close of all APHIAplus, which is especially important for work at the county level and below. If USAID decides to extend APHIAplus Nairobi/Coast for two more years, USAID should work with the project to sharpen work plans for the final two years. In doing so, all Result 3 and 4 activities should be aligned with targeted audiences and determine the focus of activities for Year 4 of the project work plan. Evidence gathered on coverage and scale from the series of rapid assessment conducted by the end of 2013 should be used in the decision-making process. Specifically, USAID should ensure APHIAplus: 1. Conducts a rapid survey of human resources in health needs among supported facilities in order to prioritize and target capacity building efforts to fill gaps before the project ends. The results of the survey can be provided to Capacity and FUNZO for inclusion in their upcoming work plans as necessary. 2. Conducts rapid assessments of its CU activation activities, PLHIV, youth activities, and Result 4 interventions to understand what has been achieved during the life of the project with recom￾mendations for future investments. 3. Consolidates and measures CU/PLHIV economic strengthening activities and determines its con￾tribution to the improved performance of CUs and support groups. 4. Improves facility provider training in MARP–related friendly service approaches, including en￾gagement of facility leadership and links with CHW community-level sensitization to remove barriers to provision of services. 5. Continues to advocate and support the development, roll-out, and adherence to national guide￾lines and targeted capacity building to DHMTs and CUs in evidence-based planning with stake￾holders, including continued support for the roll out of KQMH at facilities. 6. Ensures internal project QA/QI tools and processes are in place to ensure quality programming. 7. Provides high-level technical assistance to their sub-grantee partners in line with rapid assess￾ment findings and ensures consistent alignment to national and international guidelines and standards, specifically. A–63 8. Sustains IGAs focusing on agricultural initiatives. To do this, USAID needs to foster collabora￾tion within its own offices, such as with ABEO, as well as involve other line Ministries in the GoK, such as Agriculture, Livestock / Fisheries Development and Cooperatives 9. Reports against the results chain presented in their Year 3 M&E Plan, using a revised and re￾duced logical framework and PMP approved by USAID to include key outcome level indicators supported by appropriate process indicators. The evaluation team recommends prioritizing the APHIAplus M&E plan once the Mission creates the new PAD, harmonizing it with the project logframe, and creating an activity logframe to clarify casual logic and make outcome indicators attributable to APHIAplus. 10. Uses an approved USAID template for sub-partner SOWs, which includes key elements (objec￾tives, indicators, M&E plans, etc) in order to streamline future documentation. USAID should: 1. Conduct a Data Quality Assessment of APHIAplus Nairobi/Coast to address reporting issues specifically related to PEPFAR indicators that measure individual exposure to individual and/or small group level interventions. 2. Conduct an external review of the APHIAplus Nairobi/Coast OVC component. Suggested ques￾tions to be investigated include:  How are resources prioritized within the project?  How are households selected for OVC assistance and what are the systems and pro￾cesses for a disinterested selection?  What resources does each child receive and when, and how is this documented?  How can future U.S. government support to OVC in Nairobi and Coast be strength￾ened and aligned to the PEPFAR OVC Guidance issued in 2012? 3. Move all social protection, child protection, and household economic strengthening activities to a dedicated award when feasible, and use it to transition between APHIAplus and the next five- year assistance program to pilot integrated implementation models that have measureable link￾ages to facility and community services. C.2. Recommendations on Rapid Assessment Details for APHIAplus By the end of 2013: APHIAplus should conduct a rapid assessment of its CU activation and intervention to understand what has been achieved during the life of the project with recommendations for future investments. This will be used as a hand-over tool for future activities. Suggested questions include: • Within each targeted sub-county/constituency, how many CU were fully, semi, and just starting to be activated? • What is the nature and duration of CHW training and distribution of in-kind support? What are the gaps? • How have CHWs improved health in the community (e.g., delivery uptake in faculties over time)? • What is the status of economic strengthening activities for CHWs, and how robust is each activ￾ity? A–64 APHIAplus should conduct a rapid assessment of its PLHIV–focused interventions to understand what has been achieved during the life of the project with recommendations for future investments. This will be used as a hand-over tool for future activities. Suggested questions include: • Within each targeted county and for each targeted age band, what is the population denomina￾tor for PLHIV? • What is the nature and extent of CPWP services/models that APHIAplus implemented for PLHIV (by age/special interest)? • What are the CPWP gaps, and how can interventions better meet national and international standards? • What is the status of economic strengthening activities for PLHIV and how robust is each inter￾vention? APHIAplus should conduct a rapid assessment of its youth-focused interventions to understand what has been achieved during the life of the project with recommendations for future investments in health communication among youths in Nairobi/Coast. This will be used as a hand-over tool for future activi￾ties. Suggested questions include: • Within each targeted county/constituency and for each targeted age band, what are the popula￾tion denominator and the main health issues? • What interventions has APHIAplus implemented for each targeted age band and what is the population coverage, intensity/dosage for each set of health messages, and cross-cutting themes? • What are the gaps in these interventions and messages, and how can interventions better incor￾porate integrated health messaging? • What is the status of economic strengthening activities for youths and how robust are they? • How can future youth-focused interventions be strengthened to improve coverage, intensity, dosage, sustainability, and more nuanced targeting? APHIAplus should conduct a rapid assessment of its Result 4 interventions to discern what has been achieved during the life of the project with recommendations for future investments. Suggested ques￾tions include: • Within each targeted county/constituency, what are the nature, targeting, and results of specific Results 4 activities per targeted population? What are the gaps? • What is the status of economic strengthening activities by targeted audience and how robust is each activity? • How have Result 4 activities improved health in the community? • How likely will Results 4 activities be sustained after the end of project? How will activities be supported by generated resources (e.g., CU support activated for OVC support). A–65 ANNEX D. CONSIDERATIONS FOR THE NEXT FIVE-YEAR HEALTH ASSISTANCE PROGRAM The evaluation team has some additional recommendations to consider based on evaluation findings and conclusions. • Conduct a collaborative Needs Assessment on the weakest aspects of all APHIAs (using eval￾uation and assessment data). The Assessments must include local stakeholders (implementers and beneficiary representatives), and the latest industry research/best practices. The data will be the foundation for the redesign of APHIA and its packages of intervention. • Based on the Needs Assessments, conduct one-week Collaborative Planning Workshops with key stakeholders, including local implementers, representatives from beneficiary groups/target populations, government, to inform design and cultivate ownership. The key to a successful analysis is to involve both facility-level providers and managers who are most attuned to the needs, but also to key decision makers up the supply and resource chain: District to Cen￾tral level authorities who allocate resources. Donors and private sector players (including IGA, VSL, Community-insurance orgs and for-profit suppliers like pharmacies) are also essential as they are invari￾ably a key part of the equation. Once assessed, the critical next step is a Collaborative Planning Work￾shop to identify short- and long-term actions within the jurisdiction and capacity of each stakeholder to remedy immediate and systemic needs towards the goal of full range and quality services that are sus￾tainably resourced (this is usually a combination of short-term donor assistance while public systems establish IGAs, insurance pools, or budget lines for needed cash, staff and supplies). Specific considerations include: Improving the APHIAplus model (Q4/Q5) • USAID should reconsider direct financing and gap-filling approaches to focus on sustainable in￾puts of strengthening the health system based on best practice. More sustainable approaches in￾clude systems strengthening components to resource management, even if they include a short- term provision of cash and supplies. These components include activities such as a Stakeholder Analyses (to identify key players) and collaborative Needs and Resource Assessments of not on￾ly services the population needs (disease burden rates and facility ratios, etc.), but delivery sys￾tems analysis of key elements including Workforce, Financing (public and private), and infrastruc￾ture (facilities, information systems and supply chains). If USAID decides to use a similar model in the future, it should: • Focus on strategies that meet the unique needs of Nairobi and Coast, but through implementa￾tion models that are not necessarily co-joined. • Ensure that consortium partners have a clear coordination strategy • Ensure that mechanisms are in place to share technical and programmatic best practices/lessons learned from all areas in Kenya • Allow overlap during transition to reduce service disruptions A–66 If USAID decides to use a similar model with regard to national and sub-national mechanisms, it should ensure: • A coordinating secretariat or more regular coordination meetings between sub-national pro￾jects. • Formal MOUs between national mechanisms and sub-national projects, which delineate areas for collaboration and responsibilities, including budget implications. Strengthening service delivery and enhancing community strategy (Q2) • For facility-based interventions, future projects should deliver a minimum standardized frame￾work of interventions to improve consistency across project-supported facilities, counties, and regions. This package should include support for the integration of services and targeted support for MNCH. • Youth Friendly Services should be expanded to include HTC and MNCH services. This can be done by including youth-focused sessions on safe motherhood, sexual health/abuse and STI screening (including HIV) within provider CMEs/OJTs/FSTs as well as within capacity building ef￾forts for youth friendly desk volunteers. Efforts should also be targeted to asking the right ques￾tions, appropriate referrals and confidentiality for youth who come for services. • USAID should strategize with national lab support mechanisms (KEMRI and KEMSA) to deter￾mine the best way forward in addressing the sustainability of the current lab networking system and improving the turn-around time for delivery of results. Some ideas include implementing an electronic system to facilitate test results to labs or building the capacity of laboratories to per￾form tests on site. • USAID-supported IGA and VSLA activities should be set up and supervised in a manner that en￾sures that supported CHWs will continue to contribute to the ongoing implementation of the community strategy when government budgets are unable to support it. • USAID should employ a capacity approach with the MOH whereby activated CUs are assumed into GOK budgets in a phased manner. Refocusing the integration of community-based interventions targeting youths, and mar￾ginalized and vulnerable populations to address key health priorities and behaviors (Q2) Youths  Future youth projects need to ensure national mechanisms update all youth materials and cur￾ricula, or USAID should mandate non-national level projects to do this themselves, particularly when new guidance and policy are issued.  In partnership with the GOK, USAID should commission rapid assessments of youth interven￾tions to help ascertain programmatic and cost effectiveness. Recommendations will help USAID to determine its future assistance for youth-focused interventions, including choices regarding the scale of coverage and depth of interventions within targeted geographic areas. Key Populations A–67  Future CPWP, PLHIV, and key population programming should strive to meet international and national standards, and utilize global best practices while tailoring initiatives to the Kenyan con￾text and Community Strategy. A suggested checklist for new program design with key popula￾tions should include:  Size estimates and behavioral surveys for MARPs  State-of-the-art technical/organizational capacity building of Kenyan CBOs to implement key populations activities with renewed focus on partner targeting and PWP  Further segmenting and tailoring interventions to key vulnerable populations (e.g., un￾derage FSW, married MSM, children of PWUD, and populations with overlapping risk behaviors such as FSW, IDU, and MSM prisoners)  Meaningful integration of RH/FP, advocacy and legal rights, MNCH, and child protection into programming  Roll out drug addiction treatment services and MAT in accordance with the continuum of prevention, care, treatment, and reintegration  Incorporation of alcohol awareness and reduction interventions with referrals to addic￾tion treatment services  Expanded community and law enforcement sensitization and advocacy  Building provider capacity in key population-friendly services in CCCs and other facility services using a "whole-of-facility" approach  The inclusion of income generating activities and alternative employment activities for key populations  Behavioral measurements  Sharing best practices between APHIAplus regions  Adapting promising models from other regions (e.g., Flying Squads in Asia to address harassment and protection)  Stronger linkages to Children’s Services and underage FSWs, MSWs, and MSMs • Future key population partners should disaggregate HIV testing results for key populations by new/repeat testers, and use data to inform programming (e.g., whether MSM interventions are reaching most-at-risk segments within MSM populations, or the aggressive expansion of PWP services for infected MSM). • Future USAID assistance in HIV prevention should focus on achieving saturation-level coverage within defined geographic areas where key populations have demonstrated significant contribu￾tions to HIV incidences. Assistance should build NASCOP’s ability to measure, monitor, and co￾ordinate programs and quality. • For future projects, USAID should further investigate promising best practices that have the po￾tential for scale up. These include:  Reaching wives and children of HIV-infected MSM through methods such as household HTC and couples-based PWP services. Of particular interest is the identification of dis￾cordant couples.  MNCH services for PWUD. • USAID should assess Mentor Mothers and how it can better align with the Community Strategy and/or become more self-sustainable as a program. • For future projects, USAID should develop or provide more specific guidance on creating and sustaining partnerships as envisioned in the APHIAplus RFA and early project documents. These A–68 partnerships should include consultations with established networks, including religious net￾works such as SUPKEM, NCCK and KCCB; and key population networks as they exist. Reconsidering the approach to integrated child protection, social protection, and house￾hold economic strengthening aligned to national and PEPFAR guidelines (Q2) • Social protection, child protection, and household economic strengthening activities should have a dedicated separate award. • Future support for OVC should align to national and PEPFAR guidance, particularly with regard to how OVC support is embedded within wider child protection, social protection, and house￾hold economic strengthening initiatives. • Future USAID-funded health projects should include comprehensive gender and GBV strategies and detailed gender analyses in their annual work plans. PMPs should include indicators that clearly demonstrate alignment to proposed gender and GBV strategies and interventions with process and outcome level indicators. These should include indicators to measure the process and results of gender mainstreaming. • Future USAID support for public sector GBV services should further align with national best practices such as measurable linkages to legal aid, law enforcement, and child and social services. • Future USAID-funded projects with Result 4 activities should include comprehensive strategies in their annual work plans based on detailed analyses of vulnerability and need. • Future USAID support should work to continue sustaining IGAs, focusing on agricultural initia￾tives. To do this, USAID needs to foster collaboration within its own offices, such as with ABEO, as well as involve other line Ministries in the GoK, such as Agriculture, Livestock / Fish￾eries Development and Cooperatives. Improving project monitoring and reporting of results (Q1) • USAID should require that future projects define and report against indicators at all levels of the project’s causal logic chain, with an emphasis on higher levels so that projects can demonstrate true results; this should be aligned with the Mission’s new PAD. Future USAID activities ad￾dressing behavior, particularly for key populations, should include behavioral measurements. • Future sub-grantee SOWs should be standardized and aligned with project objectives, and in￾clude M&E indicators and reporting requirements. • Based on the selection of outcome indicators and the new PAD, USAID should define a set of baseline assessments necessary to demonstrate indicator trends and commission them as neces￾sary. The rapid assessments recommended in the previous section can serve as a basis for this. • USAID should require future projects use internal QA/QI systems to ensure that implementa￾tion issues are dealt with in a timely fashion. A–69 ANNEX E. COMMODITIES SUPPLIED BY APHIAPLUS IN THE 17 SITES VISITED Commodity supplied by Aphiaplus* Type Total 1 Gloves 17 2 Male condoms piece 1500 3 pieces of penis model piece 30 4 HIV test kits facilities 17 5 Timer for testing in the Lab facilities 1 6 Flow charts facilities 17 7 Referral forms facilities 17 8 VCT reception cards facilities 17 9 Waste paper bags facilities 17 10 Tables facilities 10 11 Chairs facilities 10 12 Filing cabinets facilities 10 13 Bins - proper waste management facilities 17 14 Tools, IEC material facilities 17 15 refurbished the chairs facilities 1 16 TV and DVD set facilities 1 17 Cooker facilities 1 18 Speculum facilities 15 19 Couches facilities 2 20 Disinfection bucket facilities 1 21 LEEP machine facilities 10 22 Lugols iodine facilities 10 23 Acetic acid facilities 10 24 Cryotherapy facilities 10 25 Registers facilities 17 26 Cabinets for storage facilities 10 27 DBS transportation supplies facilities 1 28 Patient files facilities 5 29 file holders facilities 5 30 BP machine facilities 12 31 Defaulter tracing registers facilities 17 32 Computers pieces 5 33 Computers cabinets facilities 1 34 wheelchair facilities 1 35 stretchers facilities 1 36 Drip stands facilities 1 37 Weighing scale facilities 10 38 Sphygmomanometer facilities 2 39 Thermometer facilities 1 40 MUAC tapes facilities 7 41 Height board facilities 7 A–70 42 Vacutainers facilities 3 43 Needles and holders facilities 3 44 Cool box for samples facilities 3 45 Laboratory request forms facilities 3 46 Delivery book for samples facilities 3 47 Tables facilities 3 48 Microscope facilities 1 49 Bunsen burner facilities 1 50 IEC Materials facilities 17 51 Air conditioner facilities 1 53 Ultrasound scan machine facilities 1 A–71 ANNEX F. SUPPORT TO KENYATTA NATIONAL HOSPITAL’S GBV RECOVERY CENTER The project supports Kenyatta National Hospital’s (KNH) GBV Recovery Center that provides GBV services including HTC, EC, PEP, post-rape health services and counseling, GBV outreaches, survivor support groups, rehabilitation, and some linkages to legal action and law enforcement. KNH staff report that community outreaches result in almost instantaneous demand creation, particularly after meeting with schools. Barriers to services identified by staff include daytime hour services, costs for imaging ser￾vices, the manual registration service, and the intake of GBV clients in all wards which leads to lost and under-identified cases. The KNH GBV Recovery Center provides critical services to Kenyans from all parts of the county. The reported surge of patients after community and school-based outreach sessions suggest that there is still considerable unmet demand for services. The GBV Recovery Center is unable to meet fully the needs of clients due, in part, to structural barriers such as day time hours, the admittance of staff directly admit￾ted to hospital wards, and a laborious manual registration/reconciling process. Thus, some clients who are unaware of KNH's GBV services do not access services. Although the GBV Recovery Center is a GOK initiative, the GOK has not included costs in the KNH budget. KNH staff reported that as such the GBV Recovery Center is unsustainable and staff are advo￾cating with the MOH to assume costs. This includes assuming project-paid staff in the KNH payroll and advocating for free imaging services for GBV clients (similar to the free imaging disaster clients receive). The main challenge is the sustainability of the GBV Recovery Center. Services are 100 percent depend￾ent upon donor funding. RECOMMENDATIONS: USAID should employ a capacity approach with KNH/MOH whereby GBV Recovery Center costs are assumed into GOK budgets in a phased manner. USAID should reconsider support for the development of a 24/7 One-Stop GBV Recovery Center in KNH. USAID can help advocate with other donors for physical construction. Similar to its work in placing a CU in Mbagati District Hospital, future USAID support should detail CHWs to the GBV Recovery Center to expand GBV community outreach and recovery support groups. A–72 ANNEX G.1. PROGRAM MANAGEMENT PLAN DATA TABLES FOR NAIROBI NAIROBI PEPFAR indi￾cator id or other type of indicator Indicator name Baseline Year 1 target Year 1 results Percent achievement Year 2 tar￾gets Year 2 re￾sults Percent achievement Prevention Prevention Sub Area 1: PMTCT P1.1.D Number of pregnant women with known HIV status (includes women who were tested for HIV and received their re￾sults) 21,666 21,991 34,529 157 72,000 77,046 107 P1.2.D. MIN Number of HIV positive pregnant women who received antiretrovirals to reduce risk of mother-to- child transmission 1,223 1,231 1,693 138 3,400 3,832 113 Prevention Sub Area 4: Injection and Non-injection drug use P4.1.D PR Number of injecting drug users (IDUs) on opioid substitution therapy 0 3 0 0 0 0 Prevention Sub Area 5: Male Circumcision P5.1.D MIN Number of males circum￾cised as part of the mini- 0 52 169 325 2,400 3,168 132 A–73 mum package of MC for HIV prevention services, by age <15 0 5 0 0 148 196 132 15 to 24 0 18 93 517 1,342 1,771 132 >25 0 29 76 262 910 1,201 132 Prevention Sub Area 6: Post Exposure Prophylaxis P6.1.D MIN Number of persons pro￾vided with post-exposure prophylaxis, by exposure type 0 100 2,352 2352 2,400 1,790 75 Occupational TBD TBD 123 400 98 25 Rape/sexual assault vic￾tims TBD 100 1,856 1856 1,200 388 32 Other non-occupational TBD TBD 373 800 1,304 163 Prevention Sub Area 7: Prevention with People Liv￾ing with HIV (PwP) P7.1.D MIN Number of People living with HIV/AIDS (PLHIV) reached with a minimum package of Prevention with PLHIV (PwP) inter￾ventions 32,001 35,488 36,954 104 53,000 38,053 72 Prevention Sub Area 8: Sexual and Other Risk Re￾duction P8.1.D PR Number of the targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum stand￾ards required, by sex and age 0 20,500 137,348 670 267,412 296,543 111 Male 0 3,500 36,495 1043 86,156 94,982 110 10-14 0 0 7,945 29,477 27,106 92 15-19 0 3,500 11,802 337 16,026 19,633 123 A–74 20-24 0 5,018 18,360 22,387 122 25+ 0 0 11,730 22,293 25,856 116 Female 0 17,000 100,853 593 213,102 201,561 95 10-14 0 9,310 38,753 35,889 93 15-19 0 1,700 23,456 1380 28,273 31,037 110 20-24 0 15,069 26,462 31,342 118 25+ 0 53,018 87,767 103,293 118 P8.2.D Number of targeted pop￾ulation reached with indi￾vidual and/or small group level preventive interven￾tions that are primarily focused on abstinence and/or being faithful, and are based on evidence, and/or meet the mini￾mum standards required 0 20,000 30,872 154 80,000 74,520 93 Male 0 10,000 14,883 149 32,966 30,611 93 10-14 0 5,000 7,945 159 29,090 27,054 93 15-19 0 5,000 6,938 139 3,876 3,557 92 20-24 0 0 0 0 25+ 0 0 0 0 Female 0 10,000 15,989 160 46,917 43,909 94 10-14 0 5,000 9,310 186 38,258 35,840 94 15-19 0 5,000 6,647 133 8,658 7,962 92 20-24 0 32 0 107 25+ 0 0 0 0 P8.3.D PR Number of targeted pop￾ulation reached with indi￾vidual and/or small group level preventive interven￾tions that are based on evidence and/or meet the minimum standards re￾quired, by MARP type and sex 0 6,250 2,372 38 27,400 19,292 70 SW 0 4,800 1,388 29 18,500 12,093 65 A–75 Male 0 100 0 0 0 0 Female 0 4,700 1,388 30 18,500 12,093 65 IDU TBD 0 25 1,200 1,761 147 Male 0 0 20 917 1,367 149 Female 0 0 5 283 394 139 MSM 0 700 212 30 4,700 1,601 34 Other vulnerable popu￾lations (Matatus) 0 750 747 100 3,000 3,837 128 Male 0 700 576 82 2,500 3,672 147 Female 0 50 171 342 500 165 33 P8.4.D Number of condom ser￾vice outlets 250 290 116 350 384 110 Prevention Sub Area 11: Testing and Counseling P11.1.D MIN Number of individuals who received Testing and Counseling (T&C) ser￾vices for HIV and re￾ceived their test results, by sex and age 0 132,800 218,321 164 240,000 209,085 87 Male 0 54,448 99,499 183 120,000 92,378 77 <15 0 5,312 7,097 134 12,000 11,828 99 15+ 0 49,136 92,402 188 108,000 80,550 75 Female 0 78,352 118,822 152 120,000 116,707 97 <15 0 5,312 6,755 127 10,000 15,677 157 15+ 0 73,040 112,067 153 118,000 101,030 86 Prevention Sub Area 12: Gender P12.1.D Number of people reached by an individual, small-group, or communi￾ty-level intervention or service that explicitly addresses norms about masculinity related to HIV/AIDS, by sex and age 0 4,000 1,590 40 60,000 40,981 68 Male 0 2,000 739 37 56,600 38,659 68 A–76 0-14 0 0 0 889 602 68 15-24 0 1,000 253 25 29,338 19,864 68 25+ 0 1,000 496 50 26,870 18,193 68 Female 0 2,000 841 42 3,400 2,322 68 0-15 0 0 0 799 546 68 15-24 0 1,000 295 30 666 455 68 25+ 0 1,000 546 55 1,934 1,321 68 P12.2.D Number of people reached by an individual, small group, or communi￾ty-level intervention or service that explicitly addresses gender-based violence and coercion related to HIV/AIDS, by sex and age 0 4,000 15,373 384 46,556 47,972 103 Male 0 2,000 8,522 426 23,278 21,631 93 0-14 0 0 833 5,819 2,380 41 15-24 0 1,000 3,878 388 5,820 8,196 141 25+ 0 1,000 3,811 381 11,639 11,055 95 Female 0 2,000 6,851 343 23,278 26,340 113 0-15 0 0 635 5,819 1,366 23 15-24 0 1,000 2,333 233 5,820 11,501 198 25+ 0 1,000 3,883 388 11,639 13,473 116 P12.3.D Number of people reached by an individual, small-group, or communi￾ty level intervention or service that explicitly addresses the legal rights and protection of women and girls impacted by HIV/AIDS, by sex and age 0 4,000 5,454 136 11,000 6,930 63 Male 0 2,000 2,210 111 4,400 3,167 72 0-14 0 0 44 1,100 562 51 15-24 0 1,000 340 34 1,100 913 83 25+ 0 1,000 1,826 183 2,200 1,692 77 A–77 Female 0 2,000 3,244 162 6,600 3,763 57 0-14 0 0 57 1,650 1,159 70 15-24 0 1,000 390 39 1,650 1,096 66 25+ 0 1,000 2,797 280 3,300 2,388 72 P12.4.D Number of people reached by an individual, small-group, or communi￾ty level intervention or service that explicitly aims to increase access to income and productive resources of women and girls impacted by HIV/AIDS, by sex and age 0 122 179 147 6,540 3,174 49 Male 0 61 0 957 0-14 0 0 0 0 98 15-24 0 0 1 0 191 25+ 0 0 60 0 668 Female 0 122 118 97 6,540 2,217 34 0-14 0 0 0 0 381 15-24 0 38 1 3 1,962 154 8 25+ 0 84 117 139 4,578 1,682 37 CARE Care Sub Area 1: 8.1 Number of OVC served by OVC programs 0 71,000 69,364 98 80,500 80,264 100 8.1.a Number served in 3 or more core areas 0 35,500 57,441 162 74,435 78,561 106 Male TBD 13,135 28,617 218 37,830 39,996 106 Female TBD 22,365 28,824 129 36,606 38,565 105 8.1.b Number served in 1 or 2 core areas 0 35,500 11,923 34 6,065 1,703 28 Male TBD 14,555 5,707 39 329 1,069 325 Female TBD 20,945 6,216 30 2,796 634 23 Care Sub Area 1: Clinical Care (Includes OVC) A–78 C2.1.D MIN Number of HIV positive adults and children re￾ceiving a minimum of one clinical service, by sex and age 0 55,000 49,948 91 54,000 61,510 114 Male 0 19,500 17,506 90 21,600 21,626 100 <15 TBD 2,750 1,632 59 2,160 1,946 90 15+ TBD 16,750 15,874 95 19,440 19,680 101 Female 0 35,500 32,442 91 32,400 39,884 123 <15 TBD 2,750 1,623 59 1,620 2,072 128 15+ TBD 32,750 30,819 94 30,780 37,812 123 C2.2.D MIN Number of HIV positive persons receiving cotrimoxazole prophylax￾is, by sex and age 0 35,488 46,458 131 35,488 36,648 103 Male 0 14,550 16,120 111 14,550 13,299 91 <15 TBD 1,420 1,250 88 1,420 1,243 88 15+ TBD 13,130 14,870 113 13,130 12,056 92 Female 0 20,938 30,338 145 20,938 23,349 112 <15 TBD 1,420 1,015 71 1,420 1,305 92 15+ TBD 19,518 29,323 150 19,518 22,046 113 C2.3.D MIN & PR Number of HIV positive clinically malnourished clients who received therapeutic or supple￾mentary food TBD TBD 0 0 0 C2.4.D MIN TB/HIV: Percent of HIV positive patients who were screened for TB in HIV care or treatment setting TBD 70 0 0 70 90 129 C2.5.D MIN TB/HIV: Percent of HIV positive patients who were screened for TB in HIV care or treatment (pre-ART or ART) who started TB treatment TBD 80 0 0 95 95 100 A–79 Care Sub Area 5: Support Care C5.1.D MIN Number of eligible clients who received food and/or other nutrition services, by age 62,875 66,932 106 <18 52,875 66,932 127 18+ 9,000 0 0 Pregnant/Lactating 1,000 0 0 C5.2.D MIN Number of eligible chil￾dren provided with shel￾ter and care-giving 55,703 48,641 87 <18 55,703 48,641 87 18+ 0 0 C5.2.D MIN Number of eligible chil￾dren provide with health care services 54,257 51,927 96 <18 54,257 51,927 96 18+ 0 0 C5.4.D MIN Number of eligible chil￾dren provided with edu￾cation and/or vocational training 44,401 47,905 108 <18 44,401 47,783 108 18+ 0 122 C5.5.D MIN Number of eligible adults and children provide with protection and legal ser￾vices 77,500 76,204 98 <18 75,500 75,304 100 18+ 2,000 900 45 C5.6.D MIN Number of eligible adults and children provided with psychosocial, social, or spiritual support 80,500 80,030 99 <18 78,500 78,230 100 18+ 2,000 1,800 90 A–80 TREATMENT Treatment Sub Area 1: ARV Services T1.1.D Number of adults and children with advanced HIV infection newly en￾rolled on ART, by age and sex TBD 1,396 4,056 291 4,200 4,829 115 Male TBD 558 1,533 275 1,680 1,780 106 <1 TBD 5 0 0 15 7 47 <15 TBD 51 125 245 153 131 86 15+ TBD 502 1,408 280 1,512 1,649 109 Female TBD 803 2,523 314 2,520 3,049 121 <1 TBD 5 0 0 15 8 53 <15 TBD 75 83 111% 60 128 213 15+ TBD 723 2,440 337% 2,430 2,921 120 Pregnant/lactating TBD 35 0 0% T1.2.D Number of adults and children with advanced HIV infection receiving antiretroviral therapy (ART, Current), by sex and age 17,470 18,692 21,085 113 26,000 22,935 88 Male 6,988 7,477 8,040 108 10,000 8,971 90 <1 52 53 0 0 65 24 37 <15 648 695 723 104 935 713 76 15+ 6,288 6,729 7,317 109 9,000 8,258 92 Female 10,482 11,215 13,045 116 16,000 13,964 87 <1 53 53 0 0 65 11 17 <15 995 1,069 731 68 835 694 83 15+ 9,084 9,738 12,314 126 14,952 13,270 89 Pregnant women 350 355 116 33 148 208 141 T1.3D Percent of adults and children known to be alive and on treatment 12 months after initiation of 95 83 87 A–81 antiretroviral therapy <15 95 83 87 >15 95 83 87 Health System Strengthening Health System Strengthening Sub Area 1: Laboratory H1.1.D MIN Number of testing facili￾ties (laboratories) with capacity to perform clini￾cal laboratory tests 7 7 8 114 9 9 100 Programmatic Indicators Monitoring and Evaluation PR Number of quarterly RDQA in supported sites and implementing part￾ners targets 0 12 38 317 20 26 130 PR Number of quarterly data feedback sessions with partners 30 31 103 System strengthening PR Number of DHMT/PHMT members trained on COPE 0 30 0 0 PR Number of Community Units trained on Com￾munity COPE 0 20 0 0 PR Number of DHMT mem￾bers oriented on DDIU (Data Demand & Infor￾mation Use) 0 24 33 138 PR Number of facilities, IPs, CUs, CBOs holding quar￾terly meetings to discuss feedback on service quali￾ty 0 38 Number of facilities ap- 0 9 6 67 A–82 plying COPE Number of stakeholder meetings conducted at community, district and provincial level 0 72 65 90 Number of mentoring reports received 0 108 289 268 PR Number of functional community units IN 0 103 86 83 PR Number of health facili￾ties supported 0 204 187 92 Result 3: Increased use of quality health services, products, and information 3.1: Increased availability of an integrated package of quality high-impact interventions at community and facility level PR Number of IP/CU trained on OVC QI standards 3 10 333 14 27 193 PR Number of community workers trained targets 3650 8297 227 7195 4013 56 HIV/TB MIN Number of Infants tested for HIV at 6 weeks TBD 526 544 103 1,700 1,556 92 MIN Number Infants tested for HIV at 12 weeks TBD 157 348 222 1,020 816 80 Number of individuals counseled and tested for HIV at MARP clinics or drop-in centers 10,000 13,303 133 MIN Number of Infants pro￾vided with ARV prophy￾laxis 1,000 1,046 1,762 168 3,400 3,526 104 MIN Number of TB patients who received HIV CT and test results TBD 9,000 5,928 66 5,000 6,276 126 PR Number of HBC clients served, by sex 11,164 11,164 12,309 110 14,000 15,052 108 Male 2,688 2,688 3,079 115 3,500 3,694 106 A–83 Female 8,476 8,476 9,230 109 10,500 11,358 108 PR Number of couples coun￾seled and tested for HIV 10,833 15,616 144 25,000 10,421 42 Counseled 10,833 16,203 150 25,000 10,834 43 Tested 10,833 15,593 144 25,000 10,421 42 Both HIV positive 300 409 136 220 360 164 Discordant 300 711 237 900 620 69 MNCH Number of children de￾wormed at least once in a year 50,000 142,047 284 3.1.6-10 Number of children un￾der 5-years of age who received Vitamin A from USG-supported programs (including OVC) 15,000 30,022 200 76,000 73,866 97 Number of Children who have received Measles vaccine by 12 months 18,000 20,891 116 36,000 40,024 111 MIN Pregnant women receiv￾ing two doses of Intermit￾tent Presumptive Therapy (IPT2) 0 732 3.1.6-8 Number of children less than 12 months of age who received DPT3 from USG-supported programs 22,000 19,838 90 40,000 42,280 106 Number of children <1 year fully immunized 22000 8419 38 36,000 39,666 110 Number of Pregnant women supplied with LLITNs 0 0 Number of Long Lasting Insecticide Treated Nets (LLITN) distributed preg￾nant women and children <5 0 0 3.1.6-26 Number of people 900 875 97 A–84 trained in mater￾nal/newborn health through USG-supported programs RH/FP PR 3.1.7-4 Couple Years of Protec￾tion (CYP) in USG- supported program at project-supported facili￾ties 0 48,000 87,207 182 120,000 148,625 124 Number of people trained in FP/RH with USG funds 90 905 1006 410 605 148 Males 0 347 210 205 98 Females 0 458 200 400 200 Number of USG-assisted service delivery points providing FP counseling or services 77 87 113 90 87 97 Number of service deliv￾ery points reporting stock-outs of any contra￾ceptive commodity of￾fered by the SDP 3 0 0 3.1.7-6 Number of counseling visits for FP/RH as a re￾sult of USG assistance 7,000 62,773 897 70,000 109,652 157 Number of USG-assisted service delivery points experiencing stock-outs of specific tracer drugs 3 0 0 MIN Number of WRA receiv￾ing FP commodities 3.2: Increased demand for an integrated package of quality high-impact interventions and community and facility levels PR Number of individuals reached through small 6,000 4,535 76 1,000 0 0 A–85 group discussions on health excluding HIV (disaggregated by age and sex) PR Number of community education sessions con￾ducted (community dia￾logue days, ETL sessions, drama, etc.) 20,000 14,607 73 3.3: Increased adoption of healthy behaviors PR Number of community dialogue days held on health topics 36 82 228 MIN Number of pregnant women attending at least 4 ANC visits 15,000 22,154 148 60,000 39,062 65 MIN MCH 3.1.6-11 Number of deliveries with a skilled birth at￾tendant (SBA) in USG- assisted programs 9,000 26,350 293 40,000 37,085 93 3.1.6-46 Percent of births deliv￾ered by caesarean section 0% MIN Number of households using ITNs 3.4: Increased program effectiveness through innova￾tive approaches PR Number of male champi￾ons trained for FP/RH, GBV, VMMC 90 80 89 PR Number of support groups for vulnerable populations (MARPs, PLHIV, youth) formed/ strengthened Result 4: Social determinants of health addressed to improve the well-being of the community, especially marginalized, poor, and underserved populations 4.1: Marginalized, poor, and underserved groups have increased access to economic security initiatives through coordination and integration with economic strengthening programs A–86 PR Number of households linked to household eco￾nomic strengthening initi￾atives 15 3729 24860 PR Number of children trained on basic financial literacy 300 322 107 2,000 620 31 PR Number of households trained on VSL 13,000 9,111 70 Number of households that have initiated an in￾come generating activities 3,000 3,724 124 Number of CBOs linked to micro-finance institu￾tion (MFI) 12 12 100 4.2: Improved food security and nutrition for margin￾alized, poor, and underserved populations PR Number of individuals receiving nutrition litera￾cy education 3,000 5,832 194 PR Number of nutrition edu￾cation sessions conducted by CHEWs, CHWs, farmers' groups Number of farmer groups formed and strengthened Number of producer organizations formed and linked to marketing net￾works 4.3: Marginalized, poor, and underserved groups have increased access to education, life skills, and literacy initiatives through coordination and integration with education programs PR Number of supported schools with children's health and/or rights clubs 20 16 80 40 23 58 4.4: Increased access to safe water, sanitation, and improved hygiene Number of individuals reached through theater 20,000 23,066 115 A–87 sessions on safe water and hand washing 3.1.6-3 MCH Liters of drinking water disinfected with USG- supported, point-of-use treatment products 5,000 5,000 21,621,100 432,422 56,000,000 27,278,700 49 MIN Number of households treating water MIN Number of households with functional pit latrines 4.5: Strengthened systems, structures and services for protection of marginalized, poor and underserved populations PR Number of district and provincial GBV working groups established 1 2 2 100 PR Number of teachers trained in: 30 90 300 PR Number of functional GBV working groups supported 6 6 100 PR Number of male champi￾ons networks supported 3 3 100 4.6: Expanded social mobilization for health PR Number of special events conducted (MOYA youth week, Malezi Bora/BF weeks, etc) 4 9 225 8 13 163 PR Number of Shuga, GATE and Gjue events held 4 7 175 PR Number of adoles￾cents/youth reached through drama outreach￾es in secondary schools on RH/FP, GBV, VMMC 20,000 9,987 50 PR Male 10,000 5,039 50 PR Female 10,000 4,948 49 A–88 A–89 ANNEX G.2. PROGRAM MANAGEMENT PLAN DATA TABLES FOR COAST PROVINCE COAST PROVINCE Indicator name Baseline Year 1 target Year 1 results % achieve￾ment Year 2 targets Year 2 results % achieve￾ment YR 3 ac￾tive Prevention Prevention Sub Area 1: PMTCT P1.1.D Number of pregnant women with known HIV status (includes women who were tested for HIV & received their results) 99,408 104,378 96,500 92% 100,000 109,084 109% Yes P1.2.D. MIN Number of HIV positive preg￾nant women who received antiretrovirals to reduce risk of mother-to-child transmission 4,482 5,562 4,011 72% 4,200 4,063 97% Yes Prevention Sub Area 4: Injection and Non-injection drug use P4.1.D PR Number of injecting drug users (IDUs) on opioid substitution therapy 0 50 - 1,200 0 0% No Prevention Sub Area 5: Male Circumcision P5.1.D MIN Number of males circumcised as part of the minimum package of MC for HIV prevention services, by age N/A N/A - 1,660 0 0% Yes <15 N/A N/A - 300 0 0% Yes 15 to 24 N/A N/A - 800 0 0% Yes >25 N/A N/A - 560 0 0% Yes A–90 Prevention Sub Area 6: Post Exposure Prophylaxis P6.1.D MIN Number of persons provided with post-exposure prophylaxis, by exposure type 350 368 859 234% 5,555 1,173 21% Yes Occupational 150 158 145 92% 719 169 24% Yes Rape/sexual assault victims 150 158 511 324% 3,407 518 15% Yes Other non-occupational 50 53 203 387% 1,429 486 34% Yes Prevention Sub Area 7: Prevention with People Living with HIV (PwP) P7.1.D MIN Number of People living with HIV/AIDS (PLHIV) reached with a minimum package of Preven￾tion with PLHIV (PwP) interven￾tions 8,000 8,400 22,396 267% 48,869 47,073 96% Yes Prevention Sub Area 8: Sexual and Other Risk Reduction P8.1.D PR Number of the targeted popula￾tion reached with individual and/or small group level preven￾tive interventions that are based on evidence and/or meet the minimum standards required, by sex and age 0 35,000 99,095 283% 339,000 334,945 99% Yes Male 0 15,000 42,937 286% 116,000 144,122 124% Yes 10-14 0 0 3,416 15,000 4,592 31% Yes 15-19 0 8,000 29,368 367% 16,000 24,558 153% Yes 20-24 0 0 0 32,000 42,375 132% Yes 25+ 0 7,000 10,153 145% 53,000 72,597 137% Yes Female 0 20,000 56,158 281% 223,000 190,823 86% Yes 10-14 0 0 5,349 40,000 4,498 11% Yes 15-19 0 10,500 31,563 301% 33,000 26,267 80% Yes 20-24 0 0 0 46,000 45,067 98% Yes 25+ 0 9,500 19,246 203% 104,000 114,991 111% Yes P8.2.D Number of targeted population reached with individual and/or small group level preventive interventions that are primarily 0 7,000 21,168 302% 75,000 71,590 95% A–91 focused on abstinence and/or being faithful, and are based on evidence and/or meet the mini￾mum standards required Male 0 3,000 9,866 329% 35,000 29,150 83% 10-14 0 3,000 7,871 262% 20,000 4,592 23% Yes 15-19 0 0 1,995 15,000 24,558 164% Yes 20-24 0 0 0 0 0 Yes 25+ 0 0 0 0 0 Yes Female 0 4,000 11,302 283% 40,000 42,440 106% Yes 10-14 0 4,000 8,176 204% 25,000 4,498 18% Yes 15-19 0 0 3,126 15,000 26,267 175% Yes 20-24 0 0 0 0 0 Yes 25+ 0 0 0 - 0 Yes P8.3.D PR Number of targeted population reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the mini￾mum standards required, by MARP type and sex 0 8,850 24,064 272% 42,000 53,507 127% Yes SW 0 2,800 18,704 668% 28,160 33,082 117% Yes Male 0 280 4,352 1554% 3,000 1,563 52% Yes Female 0 2,520 14,352 570% 25,160 31,519 125% Yes IDU 0 150 286 191% 4,840 8,416 174% Yes Male 0 135 263 195% 3,400 6,299 185% Yes Female 0 15 23 153% 1,440 2,117 147% Yes MSM 0 500 1,851 370% 7,000 10,983 157% Yes Other vulnerable populations (Matatus) 0 5,400 3,223 60% 2,000 1,026 51% Yes Male 0 5,000 1,642 33% 1,800 872 48% Yes Female 0 5,000 2,108 42% 200 154 77% Yes P8.4.D Number of condom service out￾lets 380 400 414 104% 500 503 101% Yes Prevention Sub Area 11: Testing and Counseling P11.1.D MIN Number of individuals who re- 463,916 536,628 331,058 62% 360,000 386,014 107% Yes A–92 ceived Testing and Counseling (T&C) services for HIV and re￾ceived their test results, by sex and age Male 198,345 229,052 131,745 58% 156,814 159,223 102% Yes <15 39,668 45,618 30,342 67% 37,362 24,382 65% Yes 15+ 158,677 183,434 101,403 55% 119,452 135,016 113% Yes Female 265,571 307,577 199,313 65% 203,186 226,791 112% Yes <15 53,116 61,083 37,317 61% 56,638 29,279 52% Yes 15+ 212,455 246,493 161,996 66% 146,547 197,613 135% Yes Prevention Sub Area 12: Gender P12.1.D Number of people reached by an individual, small-group, or community-level intervention or service that explicitly addresses norms about masculinity related to HIV/AIDS, by sex and age 0 186,300 258 0.14% 78,600 69,136 88% Yes Male 0 24,300 123 0.51% 76,000 65,960 87% Yes 0-14 0 7,290 0 0.00% 0 18 Yes 15-24 0 4,860 0 0.00% 26,828 3,609 13% Yes 25+ 0 12,150 123 1.01% 49,172 62,333 127% Yes Female 0 162,000 135 0.08% 2,600 3,176 122% Yes 0-15 0 50,220 0 0.00% 0 30 Yes 15-24 0 43,740 0 0.00% 1,826 2,230 122% Yes 25+ 0 68,040 135 0.20% 750 916 122% Yes P12.2.D Number of people reached by an individual, small group or community-level intervention or service that explicitly addresses gender-based violence and coer￾cion related to HIV/AIDS, by sex and age 0 204,800 1,773 0.87% 45,225 26,814 59% Male 0 41,600 620 1.49% 22,612 22,388 99% Yes 0-14 0 7,540 122 1.62% 375 1,953 521% Yes 15-24 0 14,060 208 1.48% 12,139 10,970 90% Yes 25+ 0 20,000 290 1.45% 10,098 9,465 94% Yes Female 0 163,200 1,153 0.71% 22,613 30,311 134% Yes A–93 0-15 0 50,520 69 0.14% 606 2,399 396% Yes 15-24 0 44,240 383 0.87% 11,861 14,486 122% Yes 25+ 0 68,440 701 1.02% 10,146 13,426 132% Yes P12.3.D Number of people reached by a individual, small-group, or com￾munity level intervention or service that explicitly addresses the legal rights and protection of women and girls impacted by HIV/AIDS, by sex and age 0 186,300 1,721 0.92% 16,000 15,601 98% Male 0 24,300 566 2.33% 4,920 5,429 110% Yes 0-14 0 7,290 122 1.67% 244 1,160 475% Yes 15-24 0 4,860 208 4.28% 2,191 2,449 112% Yes 25+ 0 12,150 236 1.94% 2,484 1,819 73% Yes Female 0 162,000 1,155 0.71% 11,080 10,172 92% Yes 0-14 0 50,220 69 0.14% 702 1,588 226% Yes 15-24 0 43,740 383 0.88% 4,228 3,804 90% Yes 25+ 0 68,040 703 1.03% 6,352 4,769 75% Yes P12.4.D Number of people reached by a individual, small-group, or com￾munity level intervention or service that explicitly aims to increase access to income and productive resources of women and girls impacted by HIV/AIDS, by sex and age 0 40,000 52,549 131% 40,000 9,884 25% Male 0 11,924 24,780 208% 11,924 4,219 35% Yes 0-14 0 980 0 0% 980 0 0% Yes 15-24 0 5,052 21,975 435% 5,052 512 10% Yes 25+ 0 5,892 2,805 48% 5,892 3,707 63% Yes Female 0 28,076 27,769 99% 28,076 5,665 20% Yes 0-14 0 1,964 0 0% 1,964 0 0% Yes 15-24 0 11,508 24,258 211% 11,508 729 6% Yes 25+ 0 14,604 3,511 24% 14,604 4,936 34% Yes CARE A–94 Care Sub Area 1: 8.1 Number of OVC served by OVC programs 79,849 79,849 93,054 117% 80,000 79,791 100% Yes 8.1.a Number served in 3 or more core areas 39,471 39,471 40,343 102% 72,025 75,103 104% Yes Male 38681.6 38,682 38,238 99% 35,945 37,522 104% Yes Female 789.42 789 2,105 267% 36,080 37,581 104% Yes 8.1.b Number served in 1 or 2 core areas 40,378 40,378 52,711 131% 7,975 4,688 59% Yes Male 39570.4 39,570 41,955 106% 4,469 2,778 62% Yes Female 807.56 808 10,756 1,332% 3,506 1,910 54% Yes Care Sub Area 1: Clinical Care (Includes OVC) C2.1.D MIN Number of HIV positive adults and children receiving a mini￾mum of one clinical service, by sex and age 66,000 69,300 85,802 124% 77,025 101,080 131% Male 26,400 27,720 28,749 104% 26,907 33,494 124% Yes <15 5,280 5,544 4,751 86% 4,602 5,010 109% Yes 15+ 21,120 22,176 23,998 108% 22,305 28,484 128% Yes Female 39,600 41,580 57,053 137% 50,118 67,586 135% Yes <15 7,920 8,316 4,646 56% 4,398 5,018 114% Yes 15+ 31,680 33,264 52,407 158% 45,720 62,568 137% Yes C2.2.D MIN Number of HIV positive persons receiving cotrimoxazole prophy￾laxis, by sex and age 62,056 65,159 66,946 103% 63,472 47,336 75% Yes Male 24,822 26,063 23,077 89% 23,468 15,175 65% Yes <15 4,964 5,212 2,661 51% 3,521 2,317 66% Yes 15+ 19,858 20,851 20,416 98% 19,947 12,858 64% Yes Female 37,234 39,096 43,869 112% 39,983 32,161 80% Yes <15 7,447 7,819 2,476 32% 3,281 2,342 71% Yes 15+ 29,787 31,276 41,393 132% 36,702 29,819 81% Yes C2.3.D MIN & PR Number of HIV positive clinical￾ly malnourished clients who received therapeutic or supple￾mentary food 7,000 7,350 - 5,220 10,529 202% Yes C2.4.D MIN TB/HIV: Percent of HIV posi- TBD 40% 35% 88% 90% 97% 108% Yes A–95 tive patients who were screened for TB in HIV care or treatment setting C2.5.D MIN TB/HIV: Percent of HIV positive patients who were screened for TB in HIV care or treatment (pre-ART or ART) who started TB treatment TBD 40% 96% 240% 100% 100% 100% Yes Care Sub Area 5: Support Care C5.1.D MIN Number of eligible clients who received food and/or other nu￾trition services, by age 46,000 50,820 110% Yes <18 40,250 50,820 126% Yes 18+ 3,833 - Yes Pregnant/Lactating 1,917 0 0% Yes C5.2.D MIN Number of eligible children pro￾vided with shelter and care- giving 60,000 68,119 114% Yes <18 59,063 68,119 115% Yes 18+ 937 0 0% Yes C5.2.D MIN Number of eligible children pro￾vide with health care services 70,000 70,845 101% Yes <18 69,063 69,908 101% Yes 18+ 937 937 100% Yes C5.4.D MIN Number of eligible children pro- vided with education and/or vocational training 75,000 74,499 99% Yes <18 74,000 74,277 100% Yes 18+ 1,000 222 22% Yes C5.5.D MIN Number of eligible adults and children provide with protection and legal services 60,000 41,712 70% Yes <18 59,683 41,512 70% Yes 18+ 317 300 95% Yes C5.6.D MIN Number of eligible adults and children provided with psycho￾social, social, or spiritual sup- 81,000 74,499 92% Yes A–96 port <18 80,000 73,562 92% Yes 18+ 1,000 937 94% Yes TREATMENT Treatment Sub Area 1: ARV Services T1.1.D Number of adults and children with advanced HIV infection newly enrolled on ART, by age & sex 6,656 7,076 6,948 98% 6,700 6,447 96% Yes Male 2,662 2,793 2,040 73% 1,923 2,185 114% Yes <1 111 116 0 0% 100 19 19% Yes <15 421 440 359 82% 325 275 85% Yes 15+ 2,130 2,237 1,681 75% 1,498 1,910 128% Yes Female 3,029 3,183 4,131 130% 3,821 4,338 114% Yes <1 126 135 0 0% 100 23 23% Yes <15 479 503 310 62% 301 283 94% Yes 15+ 2,424 2,545 3,821 150% 3,420 4,055 119% Yes Pregnant/lactating 965 1,100 777 71% Yes T1.2.D Number of adults and children with advanced HIV infection receiving antiretroviral therapy (ART) (Current), by sex and age 38,675 50,361 27,297 54% 35,180 28,682 82% Male 17,404 24,892 9,277 37% 11,953 9,576 80% Yes <1 724 6,031 0 0% 500 103 21% Yes <15 2,752 4,846 1,402 29% 1,731 1,298 75% Yes 15+ 13,928 14,015 7,976 57% 9,722 8,278 85% Yes Female 20,641 24,807 17,747 72% 23,227 19,106 82% Yes <1 885 2,562 0 0% 500 89 18% Yes <15 3,363 5,750 1,403 24% 1,732 1,274 74% Yes 15+ 16,393 16,494 16,344 99% 20,178 17,832 88% Yes Pregnant women 630 662 302 46% 817 154 19% Yes T1.3D Percent of adults and children known to be alive and on treat￾ment 12 months after initiation of antiretroviral therapy 95% 81% 85% A–97 <15 95% 81% 85% Yes >15 95% 81% 85% Yes Health System Strengthening Health System Strengthening Sub Area 1: Laboratory H1.1.D MIN Number of testing facilities (la￾boratories) with capacity to perform clinical laboratory tests 15 16 15 94% 15 15 100% Yes Programmatic Indicators Monitoring and Evaluation PR Number of quarterly RDQA in supported sites and implement￾ing partners targets 0 4 1 25% 4 8 200% Yes PR Number of quarterly data feed￾back sessions with partners 0 4 13 325% Yes System strengthening PR Number of DHMT/PHMT members trained on COPE 0 6 8 133% Yes PR Number of Community Units trained on Community COPE 0 6 0 0% Yes PR Number of DHMT members oriented on DDIU (Data De￾mand & Information Use) 90 90 198 219% Yes PR Number of facilities, IPs, CUs, CBOs holding quarterly meet￾ings to discuss feedback on ser￾vice quality 0 45 62 137% Yes Number of facilities applying COPE 0 85 22 26% Yes Number of stakeholder meet￾ings conducted at community, district and provincial level 0 40 75 188% Yes Number of mentoring reports received 0 792 797 101% Yes PR Number of functional communi￾ty units IN 0 90 80 89% Yes A–98 PR Number of health facilities sup￾ported 0 412 414 100% Yes Result 3: Increased use of quality health services, products and information 3.1: Increased availability of an integrated package of quality high-impact interventions at community and facility level PR Number of IP/CU trained on OVC QI standards 30 0 0% 21 21 100% Yes PR Number of community workers trained targets 20 15 75% 4,000 7,593 190% Yes HIV/TB MIN Number of Infants tested for HIV at 6 weeks 2,441 2,563 872 34% 1,787 1,043 58% Yes MIN Number Infants tested for HIV at 12 weeks 459 482 742 154% 1,787 866 48% Yes Number of individuals coun￾seled and tested for HIV at MARP clinics or drop-in centers 10,000 9,974 100% Yes MIN Number of Infants provided with ARV prophylaxis 3,000 2,654 88% 3,700 3,571 97% Yes MIN Number of TB patients who received HIV CT and test re￾sults 7,000 6,601 94% 10,000 7,378 74% Yes PR Number of HBC clients served, by sex 15,154 15,912 15,939 100% 15,000 16,144 108% Yes Male 4,052 4,255 3,849 90% 4,483 4,351 97% Yes Female 11,102 11,657 12,090 104% 10,517 11,793 112% Yes PR Number of couples counseled & tested for HIV 12,000 16,402 137% 12,000 19,001 158% Yes Counseled 12,000 16,884 141% 12,000 19,131 159% Yes Tested 12,000 16,402 137% 10,000 19,001 190% Yes Both HIV positive 300 522 174% 400 388 97% Yes Discordant 300 604 201% 600 841 140% Yes MNCH Number of children dewormed at least once in a year 513,813 539,504 278,144 52% 300,000 183,666 61% Yes 3.1.6-10 Number of children under 5 360,244 160,748 302,386 188% Yes A–99 years of age who received Vita￾min A from USG-supported programs (including OVC) Number of Children who have received Measles vaccine by 12 months 78% 82% 83% 102% 110,034 106,984 97% Yes MIN Pregnant women receiving two doses of Intermittent Presump￾tive Therapy (IPT2) 80,892 115,445 95,410 83% 75,176 85,658 114% Yes 3.1.6-8 Number of children less than 12 months of age who received DPT3 from USG-supported programs 23,433 30,463 37,161 122% 88,468 103,911 117% Yes Number of children <1 year fully immunised 99,846 105,059 56,748 54% 87,723 100,772 115% Yes Number of Pregnant women supplied with LLITNs 100,000 48,717 49% Yes # of Long Lasting Insecticide Treated Nets (LLITN) distribut￾ed pregnant women and chil￾dren <5 141,918 149,014 74,476 50% 250,000 245,412 98% Yes 3.1.6-26 Number of people trained in maternal/newborn health through USG-supported pro￾grams TBD 2,860 1,174 41% 4,000 345 9% Yes RH/FP PR 3.1.7-4 Couple Years of Protection (CYP) in USG-supported pro￾gram at project-supported facili￾ties 17,500 18,375 98,288 535% 140,000 217,642 155% Yes No. of people trained in FP/RH with USG funds 4,000 1,206 30% Yes Males 500 473 95% Yes Females 3,500 733 21% Yes No. of USG-assisted service delivery points providing FP counseling or services 341 341 347 102% 360 352 98% Yes No. of service delivery points reporting stock-outs of any con- 662 302 46% 0 0 Yes A–100 traceptive commodity offered by the SDP 3.1.7-6 Number of counseling visits for FP/RH as a result of USG assis￾tance 310,125 339,935 81,933 24% 110,000 237,840 216% Yes Number of USG-assisted service delivery points experiencing stock-outs of specific tracer drugs 0 0 Yes MIN Number of WRA receiving FP commodities 451,287 488,851 210,830 43% Yes 3.2: Increased demand for an integrated package of quality high-impact interventions and community and facility levels PR Number of individuals reached through small group discussions on health excluding HIV (dis￾aggregated by age and sex) 5,130 9,695 9,648 100% 500 0% Yes PR Number of community educa￾tion sessions conducted (com￾munity dialogue days, ETL ses￾sions, drama, etc.) 20,000 29,764 149% Yes 3.3: Increased adoption of healthy behaviors PR Number of community dialogue days held on health topics #DIV/0! Yes MIN Number of pregnant women attending at least 4 ANC visits 48,771 65,106 34,972 54% 52,216 45,151 86% Yes MIN MCH 3.1.6-11 Number of deliveries with a skilled birth attendant (SBA) in USG-assisted programs 51,062 73,604 43,072 59% 57,220 56,599 99% Yes 3.1.6-46 Percent of births delivered by caesarean section 11% 662 302 46% Yes MIN Number of households using ITNs TBD 20,000 24,472 122% Yes 3.4: Increased program effectiveness through innovative approaches PR Number of male champions trained for FP/RH, GBV, VMMC Yes A–101 PR Number of support groups for vulnerable populations (MARPs, PLHIV, youth) formed/ strengthened 82 198 241% Yes Result 4: Social determinants of health addressed to improve the well being of the community, especially marginalized, poor and underserved populations 4.1: Marginalized, poor and underserved groups have increased access to economic security initiatives through coordination and integration with economic strengthening programs Yes PR Number of households linked to household economic strength￾ening initiatives 3,730 15,947 428% Yes PR Number of children trained on basic financial literacy 15 55 367% 105 0 0% Yes PR Number of households trained on VSL 10,000 1,822 18% Yes Number of households that have initiated an income gener￾ating activities 50,000 49,532 99% Yes Number of CBOs linked to mi￾cro-finance institution (MFI) 2,250 2,348 104% Yes 4.2: Improved food security and nutrition for marginizalized, poor and underserved populations PR Number of individuals receiving nutrition literacy education 5 1 20% Yes PR Number of nutrition education sessions conducted by CHEWs, CHWs, farmers' groups 1,500 48 3% Yes Number of farmer groups formed and strengthened 1,000 949 95% Yes Number of producer organiza￾tions formed and linked to mar￾keting networks 75 75 100% Yes 4.3: Marginalized, poor and underserved groups have increased access to education, life skills, and literacy initiatives through coordination and in￾tegration with education program PR Number of supported schools with children's health and/or rights clubs 45 0 0% 300 352 117% Yes A–102 4.4: Increased access to safe water, sanitation and improved hygiene Number of individuals reached through theatre sessions on safe water and hand washing 27,000 10,588 39% 30,000 52,440 175% Yes 3.1.6-3 MCH Liters of drinking water disin￾fected with USG-supported, point-of-use treatment products TBD 120,000 26250600 21876% 45000000 7861200 0 175% Yes MIN Number of households treating water 6,000 65,895 1098% Yes MIN Number of households with functional pit latrines 20,000 24,716 124% Yes 4.5: Strengthened systems, structures and services for protection of marginalized, poor and underserved populations PR Number of district and provin￾cial GBV working groups estab￾lished 4 4 100% Yes PR Number of teachers trained in: 120 0 0% Yes PR Number of functional GBV working groups supported 9 6 67% Yes PR Number of male champions networks supported 2 2 100% Yes 4.6: Expanded social mobilization for health PR Number of special events con￾ducted (MOYA youth week, Malezi Bora/BF weeks, etc) 4 9 225% 10 14 140% Yes PR Number of Shuga, GATE and Gjue events held 4 7 175% Yes PR Number of adolescents/youth reached through drama out￾reaches in secondary schools on RH/FP, GBV, VMMC Yes PR Male Yes PR Female Yes A–103 ANNEX H. RESULT 4 INDICATOR DISCUSSION Result 4: Social determinants of health addressed to improve the well-being of targeted communities and populations 4.1. Marginalized, poor, and underserved groups have increased access to economic security initiatives through coordination and integration with economic strengthening programs In Year 2, the project reports on the number of households that have initiated an income generating activity. There is no denominator of total households in the project’s catchment area. The Year 3 M&E plan describes another level of indicators that include the percent of households who have grown their assets through income generating activities (as an intermediate outcome indicator), using the indicator data described above as a numerator. The project has not yet reported on this new indicator. 4.4. Increased access to safe water, sanitation, and improved hygiene In Year 1, the project reports on the number of children reached through theater sessions on safe wa￾ter and hand washing. In Year 2, this indicator was dropped and replaced by the number of individuals reached through theater sessions on safe water and hand washing. The Year 3 M&E plan includes the behavior indicator number of households with soap and water at a hand washing station commonly used by family members, but the project had not operationalized or measured this indicator at the time of this evaluation. 4.5. Strengthened systems, structures, and services for the protection of marginalized, poor, and under￾served populations In Year 1 and Year 2, the project reports on the number of district and provincial GBV working groups established. Though the project increased the number of working groups established, it was unable to meet targets for Year 2 in the Coast.* The Year 3 M&E Plan describes indicators for measuring function￾ality or population measures such as the number of GBV working groups holding regular meetings and percentage of project districts with systems and services for the protection of vulnerable households. The project has not operationalized these indicators or measured these indicators as of the first quarter of Year 3. 4.6. Expanded social mobilization for health As of Year 2, the project only measures one indicator under this Intermediate Result—the number of special events conducted (e.g., MOYA youth week, Malezi Bora). Though the project did not meet tar￾gets in either year, there was an increase from Year 1 to Year 2 in the number of events supported. * See Annex for detailed information about indicators. A–104 ANNEX I. DATA-COLLECTION TOOLS KII Guide – National Level GOK Officials: DRH, NACC, MOH, NASCOP, Division of Community Health Services, MOGCSD, NACADA Name: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–105 KII Guide – National Level GOK Officials: 1. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/ meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: A–106 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–107 3. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, social protection, child protection, and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 4. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues in your area? Ask for precise examples. A–108 Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 5. Policy alignment is important for quality service delivery and cohesion. Thinking about the GOK and other (like PEPFAR or the World Bank) policies that are most relevant to your work: a. Which policies do you see as important for the APHIAplus program to consider/ incorporate into its efforts? List all mentioned. b. Which of these is APHIAplus incorporating well, and which not so well? c. What challenges are arising due to any poor alignment? (Response may be N/A) d. How can these challenges be corrected? (Response may be N/A) Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 6. Can you tell us what the transition between APHIA II and APHIAplus was like? For example, was the transi￾tion process smooth or had challenges? Do you have any recommendations on how to improve such a transi￾tion process? 7. Now I’d like to ask about how well APHIAplus served its target populations. Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Slums/ Coast Urban What has worked well and what are challenges? Are there ways to improve services to this population? A–109 Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Low income / Coast Rural What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Middle Class /Coast arid, semi-arid What has worked well and what are challenges? Are there ways to improve services to this population? Most vulnerable populations (name those you mean) What has worked well and what are challenges? Are there ways to improve services to this population? Local Implementing Partners (community based organizations) What has worked well and what are challenges? Are there ways to improve services to this population? Other: What has worked well and what are challenges? Are there ways to improve services to this population? Other: A–110 Populations Excellent, little need for changes Adequate Needs major revision DK/NA What has worked well and what are challenges? Are there ways to improve services to this population? 8. Did the APHIAplus consortium structure best support the needs of community, facilities, and government bodies? 9. If future support were available, what could be done to better address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 10. In your opinion, how has APHIAplus strengthened heath care facilities and government or community pro￾cesses? Do you think this has fostered sustainability? 11. Has the Government started to contribute more resources to the areas of service delivery or support that APHIAplus is supporting in its targeted districts? If not, why not? 12. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? 13. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! KII Guide – County Level GOK Officials: County Director of Health County Children’s Officer District Social Development Officer District Quality Assurance Officer A–111 District Development Officer Name: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–112 KII Guide – County Level GOK Officials: 14. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 15. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: A–113 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–114 16. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, child protection, social protection and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–115 17. We would like to ask your opinion on how the APHIAplus project supported you and your team in your official capacity. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–116 18. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues in your area? Ask for precise examples. 19. In your position, have you seen any evidence that APHIAplus supported facilities and/or CBOs have been able to use their monthly (or other time frame) data for decision-making or strategic planning? 20. If you receive support or funding from other donors, can you tell me who they are and briefly what they do? Remind that the project started after January 1 2011. Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 21. Policy alignment is important for quality service delivery and cohesion. Thinking about the GOK and other (like PEPFAR or the World Bank) policies that are most relevant to your work: a. Which policies do you see as important for the APHIAplus program to consider/ incorporate into its efforts? List all mentioned. b. Which of these is APHIAplus incorporating well, and which not so well? c. What challenges are arising due to any poor alignment? (Response may be N/A) d. How can these challenges be corrected? (Response may be N/A) Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 22. Did you or your colleagues receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. What have you been able to do differently as a result of receiving support from APHIAplus? A–117 23. Now I’d like to ask about how well APHIAplus served its target populations in your area. Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Slums/ Coast Urban What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Low income / Coast Rural What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Middle Class /Coast arid, semi-arid What has worked well and what are challenges? Are there ways to improve services to this population? Most vulnerable populations (name those you mean) What has worked well and what are challenges? Are there ways to improve services to this population? Local Implementing Partners (community based organizations) What has worked well and what are challenges? Are there ways to improve services to this population? Other: A–118 Populations Excellent, little need for changes Adequate Needs major revision DK/NA What has worked well and what are challenges? Are there ways to improve services to this population? Other: What has worked well and what are challenges? Are there ways to improve services to this population? 24. Did the APHIAplus consortium structure best support the needs of your community, facilities, and govern￾ment bodies? 25. If future support were available, what could be done to address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 26. Now I’d like to ask about the sustainability of the APHIAplus efforts, even though it’s only been a short time. For the purpose of our discussion, I’d like to focus on any service delivery or health care processes or sys￾tems, put in place or strengthened at the district health facility and/or community level.* Which of the following APHIA plus program elements has your ministry/County successfully adopted into your regular operating procedures? * As per USAID guidance during initial meeting A–119 Key activities from the APHIAplus work plan adopted by County Begun adoption Fully adopted Not adopted at all a. Element: Details/ explain level of adoption: b. Element: Details/ explain level of adoption: c. Element: Details/ explain level of adoption: d. Element: Details/ explain level of adoption: 27. From your perspective, what has helped and what has hindered County adoption of these elements? 28. From what you’ve seen, to what extent has the community taken ownership of these elements? A–120 Key activities from the APHIAplus work plan adopted by County Begun adoption Fully adopted Not adopted at all e. Element: Details/ explain level of adoption: f. Element: Details/ explain level of adoption: g. Element: Details/ explain level of adoption: h. Element: Details/ explain level of adoption: 29. From your perspective, what has helped and what has hindered community adoption of these elements? 30. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? 31. Are there any other comments or observations that you would like to make about APHIAplus? A–121 Thank you for your participation! A–122 KII Guide – Nongovernmental Stakeholders: PLHIV networks, religious associations, key population associations, FIDA, SOLWODI Name: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–123 KII Guide – Non-Governmental Stakeholders 32. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 33. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/ meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: A–124 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–125 34. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, social protection, child protection, and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 35. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues in your area? Ask for precise examples. A–126 Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 36. Policy alignment is important for quality service delivery and cohesion. Thinking about the GOK and other (like PEPFAR or the World Bank) policies that are most relevant to your work: a. Which policies do you see as important for the APHIAplus program to consider/ incorporate into its efforts? List all mentioned. b. Which of these is APHIAplus incorporating well, and which not so well? c. What challenges are arising due to any poor alignment? (Response may be N/A) d. How can these challenges be corrected? (Response may be N/A) Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 37. Can you tell us what the transition between APHIA II and APHIAplus was like? For example, was the transi￾tion process smooth or had challenges? Do you have any recommendations on how to improve such a transi￾tion process? 38. Now I’d like to ask about how well APHIAplus served its target populations that your organization serves. Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Slums/ Coast Urban What has worked well and what are challenges? Are there ways to improve services to this population? A–127 Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Low income / Coast Rural What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Middle Class /Coast arid, semi-arid What has worked well and what are challenges? Are there ways to improve services to this population? Most vulnerable populations (name those you mean) What has worked well and what are challenges? Are there ways to improve services to this population? Local Implementing Partners (community based organizations) What has worked well and what are challenges? Are there ways to improve services to this population? Other: What has worked well and what are challenges? Are there ways to improve services to this population? Other: A–128 Populations Excellent, little need for changes Adequate Needs major revision DK/NA What has worked well and what are challenges? Are there ways to improve services to this population? 39. Did the APHIAplus consortium structure best support the needs of community, facilities, and government bodies? 40. If future support were available, what could be done to better address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 41. In your opinion, how has APHIAplus strengthened government engagement targeted to the populations your organization serves? 42. In your opinion, how has APHIAplus strengthened the engagement of the people you work with in health and health care? 43. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? 44. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! KII Guide – USAID Name: Date: Location (Office/Facility or Community Name): A–129 Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–130 KII Guide - USAID 1. What are your key roles and responsibilities for the APHIAplus program? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. We would like to ask your opinion on how the APHIAplus project supported health care facilities in align￾ing to KEPH guidelines/ meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–131 Element Excellent, little need for changes Adequate Needs major re￾vision Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–132 Element Excellent, little need for changes Adequate Needs major re￾vision 45. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, social protection, child protection, and use of other services. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? A–133 Element Excellent, little need for changes Adequate Needs major re￾vision 46. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues overall? Ask for precise examples. Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 47. Policy alignment is important for quality service delivery and cohesion. Thinking about the GOK and other (like PEPFAR or the World Bank) policies that are most relevant to your work: a. Which policies do you see as important for the APHIAplus program to consider/ incorporate into its efforts? List all mentioned. b. Which of these is APHIAplus incorporating well, and which not so well? c. What challenges are arising due to any poor alignment? (Response may be N/A) d. How can these challenges be corrected? (Response may be N/A) Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 1. At the end of APHIA II, were there particular interventions/strategies implemented in Coast region that USAID felt were particularly successful or innovative, and merited scale up or continuation? 2. Can you tell us what the transition from APHIA II to APHIAplus was like from the perspective of the donor? In particular, can you tell us about: a. What was the process for the closeout of APHIA II to the start up of APHIAplus b. What sort of information, tools, and resources were shared? i. Between FHI360 and USAID, and how ii. Between FHI360 and the new implementing consortium, and how c. What were the successes of this transition? d. What were the challenges of this transition? i. If there were challenges, how did FHI360 overcome them? 3. Do you have any suggestions for the future to make transitions like this more successful and to maximize and build on gains when moving from one project to the next or one implementer to the next? A–134 4. Are there any challenges that you have been encountered over the course of the project that may have effect￾ed implementation? 48. Now I’d like to ask about how well APHIAplus served its target populations. Populations Excellent, little need for changes Adequate Needs major revision DK/NA Nairobi Slums/ Coast Urban What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Low income / Coast Rural What has worked well and what are challenges? Are there ways to improve services to this population? Nairobi Middle Class /Coast arid, semi-arid What has worked well and what are challenges? Are there ways to improve services to this population? Most vulnerable populations (name those you mean) What has worked well and what are challenges? Are there ways to improve services to this population? Local Implementing Partners (community based organizations) What has worked well and what are challenges? A–135 Populations Excellent, little need for changes Adequate Needs major revision DK/NA Are there ways to improve services to this population? Other: What has worked well and what are challenges? Are there ways to improve services to this population? Other: What has worked well and what are challenges? Are there ways to improve services to this population? 49. Did the APHIAplus consortium structure best support the needs of community, facilities, and government bodies? 50. If future support were available, what could be done to better address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 51. In your opinion, how has APHIAplus strengthened heath care facilities and government or community pro￾cesses? Do you think this has fostered sustainability? A–136 52. Has the Government started to contribute more resources to the areas of service delivery or support that APHIAplus is supporting in its targeted districts? If not, why not? 53. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! A–137 KII Guide – Consortium partners Pathfinder International, NARESA,CLUSA, Population Services International, Child Fund Name: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–138 KII Guide – Consortium Partners 1. Within the APHIAplus consortium, please tell us about your role. a) What are the key project activities that you have been involved? b) What are your (as a partner in the consortium) key areas of interventions in the program? c) Who are your target groups and beneficiaries? d) Tell us more about the implementation process and approach so far Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? A–139 Element Excellent, little need for changes Adequate Needs major re￾vision What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? A–140 Element Excellent, little need for changes Adequate Needs major re￾vision What needs to be done to improve? 3. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, social protection, child protection, and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? A–141 Element Excellent, little need for changes Adequate Needs major re￾vision What needs to be done to improve? 4. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues in your area? Ask for precise examples. 5. In your position, have you seen any evidence that APHIAplus supported facilities and/or CBOs have been able to use their monthly (or other time frame) data for decision-making or strategic planning? 6. How have you as a program management team/ consortium used internal program data and external data for program decision making or strategic planning. Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 7. Policy alignment is important for quality service delivery and cohesion. Thinking about the APHIA Plus project you have been implementing: a. Which policies do you see as important for the APHIAplus program to consider/ incorporate into its efforts? List all mentioned. b. Which of these has APHIAplus incorporated well, and which not so well? Give Examples. c. What challenges are arising due to any poor alignment? (Response may be N/A) d. How can these challenges be corrected? (Response may be N/A) Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? A–142 8. What is your opinion on the APHIA plus model? Conduct a SWOT analysis as they relate to Clinical, Com￾munity and Partnership packages? Strength Weakness Opportunities Threats Clinical package Community Package Partnership Package 9. Tell us about the overall coordination and management of the APHIAplus Nairobi Coast consortium? What worked well and What did not? 10. Can you tell us what the transition between APHIA II and APHIAplus was like? For example, was the transi￾tion process smooth or had challenges? Do you have any recommendations on how to improve such a transi￾tion process? 11. Did the APHIAplus consortium structure best support the needs of community, facilities, and government bodies? 12. If future support were available, what could be done to better address the health needs of the populations in Nairobi and Coast? A–143 Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 13. In your opinion, how has APHIAplus strengthened heath care facilities and government or community pro￾cesses to foster sustainability? 14. Has the Government started to contribute more resources to the areas of service delivery or support that APHIAplus is supporting in its targeted districts? If not, why not? 15. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? 16. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! A–144 KII Guide – National TA Partners Afya Info, Capacity, ICF Macro,FANIKISHA, MSH Name: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification/ Tech area: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting a review of the APHIA PLUS program implemented in Nairobi and Coast, from Jan 2011 up to now. The aim of the review is to better understand the project, looking at performance, successes and challenges, and lessons learned for the future. In this regard, you have been selected to represent your Ministry based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your region/technical ar￾ea, and will be used to plan future intervention programs in this region/area and in the country. I would like to ask you some questions related to the APHIA PLUS Program in this region/ technical area. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have questions at this point about this discussion? A–145 KII Guide – National TA Partners: 1. Please explain in detail what is your program as a Technical Partner- National Mechanism? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. Please describe your working relationship with APHIA plus Nairobi Coast. 3. What has worked well? What have been the challenges? 4. In your opinion, are there ways to strengthen the working relationships between APHIAplus Nairobi/ Coast and the national technical assistance mechanism? 5. In your opinion, how well did APHIAplus Nairobi/Coast respond to changing priorities, policies, and emerging issues in your area? Ask for precise examples. Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 6. Did your project exist before January 2011? If yes , Can you tell us as a TA partner what the transition be￾tween APHIA II and APHIAplus was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? 7. In your opinion, what has worked well with the APHIAplus model? Probe for: zoning; national versus decen￾tralized responsibilities; building on previous USAID investments; regionalization; integration; linkages; quality assurance; gender; innovation. 8. In your opinion, what have been the challenges with the APHIAplus model? A–146 9. If future support were available, how would you recommend a future assistance model for health? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 10. In your opinion, how has APHIAplus strengthened heath care facilities and government or community pro￾cesses? Do you think this has fostered sustainability? 11. If APHIAplus support ended tomorrow, which elements or results of your Technical support to the Health sector would continue? If yes give examples of mechanisms put in place for continuity 12. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! A–147 Key-Informant Interview Guide FHI360 Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–148 As you know, the activities implemented in the former Coast region of Kenya under APHIA II were implemented by your organization, FHI360. Another consortium won the follow-on award for the APHIAplus in the Coast re￾gion. The APHIAplus model was intended to build on the successful interventions of APHIA II, thus maximizing USAID’s investments. As such, we have a few questions for you. 5. At the end of APHIA II, were there particular interventions/strategies implemented in Coast region that FHI360 felt were particularly successful or innovative, and merited scale up or continuation? Probe with examples from FHI360’s end of project report including: • Cluster model for rural populations • CVD/HIV integration initiative • The Gold Star Network & CSO development & workplace interventions • Result 4 activities • Use of data and research • Measurable improvements from EOP data (e.g.; CYP; MARP coverage) 6. Can you tell us what the transition from APHIA II to APHIAplus was like from the perspective of an organiza￾tion that was departing from its work in Coast region? In particular, can you tell us about: a. What was the process for the closeout of APHIA II to the start up of APHIAplus b. What sort of information, tools, and resources were shared? i. Between FHI360 and USAID, and how ii. Between FHI360 and the new implementing consortium, and how c. What were the successes of this transition? d. What were the challenges of this transition? i. If there were challenges, how did FHI360 overcome them? 7. Do you have any suggestions for the future to make transitions like this more successful and to maximize and build on gains when moving from one project to the next or one implementer to the next? 8. Do you have any questions or suggestions for us? Thank you so much for your assistance. A–149 Group Interview Guide Health Facility Management Team (HFMT) and District Health Management Team (DHMT) Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–150 54. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 55. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Fill in the responses into the Clinical Services Assessment 56. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, child protection, social protection and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? A–151 Element Excellent, little need for changes Adequate Needs major re￾vision What went well? What were challenges? What needs to be done to improve? 57. We would like to ask your opinion on how the APHIAplus project supported you and your team in your official capacity. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: A–152 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 58. In your position, have you seen any evidence that APHIAplus supported facilities and/or CBOs have been able to use their monthly (or other time frame) data for decision-making or strategic planning? 59. If you receive support or funding from other donors, can you tell me who they are and briefly what they do? Remind that the project started after January 1 2011. Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 60. Did you or your colleagues receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. What have you been able to do differently as a result of receiving support from APHIAplus? 61. If future support were available, what could be done to address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–153 62. Now I’d like to ask about the sustainability of the APHIAplus efforts, even though it’s only been a short time. Which of the following APHIA plus program elements has your area of operation successfully adopted into your regular operating procedures? Key activities from the APHIAplus work plan adopted by County Begun adoption Fully adopted Not adopted at all i. Element: Details/ explain level of adoption: j. Element: Details/ explain level of adoption: k. Element: Details/ explain level of adoption: l. Element: Details/ explain level of adoption: 63. From your perspective, what has helped and what has hindered the adoption of these elements? 64. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? A–154 65. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! Group Interview Guide Community Unit (C) Community Health Committee (CHC) and Community Health Workers (CHWs) Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–155 66. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 67. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Capacity building of service delivery providers or staff What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Service Integration What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? A–156 Element Excellent, little need for changes Adequate Needs major re￾vision What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 68. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, child protection, social protection and use of other services . What was done well or what might have been challenges? A–157 Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 69. We would like to ask your opinion on how the APHIAplus project supported you and your team in your official capacity. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Element: A–158 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 70. In your position, have you seen any evidence that APHIAplus supported CUs, CHCs and CHWs have been able to use their monthly (or other time frame) data for decision-making or strategic planning? A–159 71. If you receive support or funding from other donors, can you tell me who they are and briefly what they do? Remind that the project started after January 1 2011. Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 72. Did you or your colleagues receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. What have you been able to do differently as a result of receiving support from APHIAplus? 73. If future support were available, what could be done to address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 74. Now I’d like to ask about the sustainability of the APHIAplus efforts, even though it’s only been a short time. Which of the following APHIA plus program elements has your area of operation successfully adopted into your regular operating procedures? Key activities from the APHIAplus work plan adopted by County Begun adoption Fully adopted Not adopted at all m. Element: Details/ explain level of adoption: n. Element: A–160 Details/ explain level of adoption: o. Element: Details/ explain level of adoption: p. Element: Details/ explain level of adoption: 75. From your perspective, what has helped and what has hindered the adoption of these elements? 76. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? 77. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! Group Interview Guide Health Facility Management Team (HFMT) and District Health Management Team (DHMT) Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: A–161 Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–162 78. Are you familiar with the APHIAplus project? Describe how the project started after January 1 2011 . Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 79. We would like to ask your opinion on how the APHIAplus project supported health care facilities in aligning to KEPH guidelines/meet the needs of OVC and vulnerable children. What was done well or what might have been challenges? Fill in the responses into the Clinical Services Assessment 80. We would like to ask your opinion on how the APHIAplus project supported communities in improving their health care, child protection, social protection and use of other services . What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Implementation of the Community Strategy – (e.g. strategic/ resource planning, demand creation) What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? A–163 Element Excellent, little need for changes Adequate Needs major re￾vision What went well? What were challenges? What needs to be done to improve? 81. We would like to ask your opinion on how the APHIAplus project supported you and your team in your official capacity. What was done well or what might have been challenges? Element Excellent, little need for changes Adequate Needs major re￾vision Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? Element: A–164 Element Excellent, little need for changes Adequate Needs major re￾vision What exactly did APHIAplus do to support this? What went well? What were challenges? What needs to be done to improve? 82. In your position, have you seen any evidence that APHIAplus supported facilities and/or CBOs have been able to use their monthly (or other time frame) data for decision-making or strategic planning? 83. If you receive support or funding from other donors, can you tell me who they are and briefly what they do? Remind that the project started after January 1 2011. Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 84. Did you or your colleagues receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. What have you been able to do differently as a result of receiving support from APHIAplus? 85. If future support were available, what could be done to address the health needs of the populations in Nairobi and Coast? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–165 86. Now I’d like to ask about the sustainability of the APHIAplus efforts, even though it’s only been a short time. Which of the following APHIA plus program elements has your area of operation successfully adopted into your regular operating procedures? Key activities from the APHIAplus work plan adopted by County Begun adoption Fully adopted Not adopted at all q. Element: Details/ explain level of adoption: r. Element: Details/ explain level of adoption: s. Element: Details/ explain level of adoption: t. Element: Details/ explain level of adoption: 87. From your perspective, what has helped and what has hindered the adoption of these elements? 88. If APHIAplus support ended tomorrow, which elements or results, if any, would you say would continue on? Why or why not? A–166 89. Are there any other comments or observations that you would like to make about APHIAplus? Thank you for your participation! A–167 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: General Population Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–168 FOR ORGANIZATIONS IMPLEMENTING PROGRAMS WITH SEGMENTS OF THE GENERAL POPULATION This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR, and the Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. 90. When was the project initiated? Month: ____________ Year: _________________ 91. Are you familiar with the APHIAplus project? 92. Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. Please note if and how support built on previous investments from APHIA II, if applicable. ) 93. If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 94. What are the segments of the general population that you target in your programs? 95. For each of these targeted populations, what is your catchment area? 96. Do you have an estimate of the size of the population in your catchment area, by targeted population? If yes, what is it? 97. For each targeted population, what is your target number to reach for the year? a. In the past 12 months, for each targeted population what is the number you have reached? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? A–169 98. For each targeted population, what type of interventions do you carry out? (Ask the partner to describe the activities, check all that they mention that apply, prompt for gaps, and confirm by observation) (In general, assess for the use of appropriate channels reaching intensity/dosage, the use of EBIs in the selec￾tion of interventions) HIV prevention activities targeted to segments within the general population Targeted population #1: ________________________________________ Element Yes/No (Reported) Yes/No (Observed) Notes HTC with emphasis on new testers Consistent condom use with all non-cohabiting, non-marital sexual part￾ners Partner reduction and faithfulness Empower women to nego￾tiate safer sex Male role modeling Integrated HIV/ RH/ FP, MNCH for women Linkages and referral from HTC to HIV care and treatment STI Prevention, Screening, and Treatment Linkages and referrals to PMTCT Linkages and referrals to VMMC Linkages and referrals to post-rape care and PEP Condom distribution and promotion Linkage to other health and non-health services (e.g.; IGAs, VSLAs) Where are these interventions implemented? A–170 Targeted Population #1: _____________________ Yes/No (Reported) Notes Interventions that are imple￾mented within an existing network/ organizations (e.g.; in churches, mosques) Interventions that are imple￾mented in a public venue (e.g.; market place) Community Center School Other: Other: What materials to you distribute to clients (check all that apply)? Targeted Population #1 Yes/No (Re￾ported) Notes Condoms Lubricants Pamphlets Referrals to clinics Other (Specify) None In your opinion, what interventions worked well? Which ones faced challenges? 99. For each targeted population, what are the interventions and messages? (Ask the partner to describe the in￾terventions , check all that they mention that apply, prompt for gaps, and confirm by observation as possible) A–171 HIV prevention activities targeted to segments within the general population Targeted population #2: ________________________________________ Element Yes/No (Reported) Yes/No (Observed) Notes HTC with emphasis on new testers Consistent condom use with all non-cohabiting, non-marital sexual part￾ners Partner reduction and faithfulness Empower women to nego￾tiate safer sex Male role modeling Integrated HIV/ RH/ FP, MNCH for women Linkages and referral from HTC to HIV care and treatment STI Prevention, Screening, and Treatment Linkages and referrals to PMTCT Linkages and referrals to VMMC Linkages and referrals to post-rape care and PEP Condom distribution and promotion Linkage to other health and non-health services (e.g.; IGAs, VSLAs) 100.Where are these interventions implemented? Targeted Population #2: _____________________ Yes/No (Reported) Notes A–172 Interventions that are imple￾mented within an existing network/ organizations (e.g.; in churches, mosques) Interventions that are imple￾mented in a public venue (e.g.; market place) Community Center School Other: Other: What materials to you distribute to clients (check all that apply)? Yes/No (Re￾ported) Notes Condoms Lubricants Pamphlets Referrals to clinics Other (Specify) None In your opinion, what interventions worked well? Which ones faced challenges? 101.Describe how these projects reinforces these messaging through multiple channels and please give examples: 102.How do you know that each individual supported under your program receives the defined package of ser￾vices and not just a few of the services? (prompt for referral systems and type; case management; tracking and follow up) 103.We are interested in how you track your clients. Can we ask a few questions about that? (ask the partner to show you the monitoring data and assess how the partner has measured - for the entire period of support - P8.1.D Number of the target population reached with individual and/or small group level HIV prevention interven￾tions that are based on evidence and/or meet the minimum standards required A–173 104.How do you know that the program is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials 105.Gender is an important element of APHIAplus. How have you addressed gender in your program a. Prompt for technical interventions that address needs of the targeted population and gender, ad￾dressing male norms and behaviors, improved equitable access to services, referrals to GBV services and legal protection 106.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 107.Did you project receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Were there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 108.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 109.Can you explain to us how the government and the community has supported or taken ownership of these interventions? A–174 110.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 111.Do you have any questions for comments for us? Thank you very much for your assistance. A–175 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: General Population Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–176 FOR CLIENTS IMPLEMENTING PROGRAMS WITH SEGMENTS OF THE GENERAL POPULATION This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR, and the Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 1. What type of interventions do you receive from the program? (Ask what services are provided, and then prompt. Check all that apply and confirm by observation as feasible, then afterwards assess what is provided by APHIAplus) (In general, assess for the use of appropriate channels reaching intensity/dosage, the use of EBIs in the selec￾tion of interventions) Targeted Population: _____________________ Yes/No (Reported) Notes Interventions that are imple￾mented within an existing network/ organizations (e.g.; in churches, mosques) Interventions that are imple￾mented in a public venue (e.g.; market place) Community Center School Other: Other: 2. Where are these interventions implemented? Targeted Population: _____________________ Yes/No (Reported) Notes Interventions that are imple￾mented within an existing network/ organizations (e.g.; in churches, mosques) Interventions that are imple￾mented in a public venue (e.g.; market place) Community Center A–177 School Other: Other: 3. For each targeted population, what are the interventions and messages? (Ask the partner to describe the in￾terventions , check all that they mention that apply, prompt for gaps, and confirm by observation as possible) HIV prevention activities targeted to segments within the general population Targeted population: ________________________________________ Element Yes/No (Reported) Yes/No (Observed) Notes HTC with emphasis on new testers Consistent condom use with all non-cohabiting, non-marital sexual part￾ners Partner reduction and faithfulness Empower women to nego￾tiate safer sex Male role modeling Integrated HIV/ RH/ FP, MNCH for women Linkages and referral from HTC to HIV care and treatment STI Prevention, Screening, and Treatment Linkages and referrals to PMTCT Linkages and referrals to VMMC Linkages and referrals to post-rape care and PEP Condom distribution and promotion Linkage to other health and non-health services (e.g.; IGAs, VSLAs) 4. What materials do you receive from the project? Yes/No (Re- Notes A–178 ported) Condoms Lubricants Pamphlets Referrals to clinics Other (Specify) None 5. In your opinion, what is working well? What have been challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 6. Do you have any recommendations to make the program better? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 7. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention? 8. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 9. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–179 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement MSM programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–180 FOR ORGANIZATIONS IMPLEMENTING PROGRAMS WITH MSM This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR. 112.When was the project initiated? Month: ____________ Year: _________________ 113.Are you familiar with the APHIAplus project? 114.Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. 115.If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 116.What is your catchment area? 117.Do you have an estimate of the size of the MSM population in your catchment area? If yes, what is it? 118.What is your target number of MSM to reach for the year? b. In the past 12 months, what is the number of MSM you have reached? 119.What is your target number of sexual partners of MSM to reach this year, by type of partner? a. In the past 12 months, what is the number of MSM partners you have reached, by type of partner? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 120.What type of interventions do you carry out targeting MSM? (Ask the partner to describe the activities, check all that they mention that apply, prompt for gaps, and confirm by observation) The HIV/STI Package of Services for MSM and Their Sex Partners Element Yes/No (Reported) Yes/No (Observed) Notes A–181 The HIV/STI Package of Services for MSM and Their Sex Partners Element Yes/No (Reported) Yes/No (Observed) Notes Community-based Out￾reach Distribution of Condoms and Condom-Compatible Lubricants HIV Counseling and Test￾ing Active Linkage to Health Care and ART Targeted IEC STI Prevention, Screening, and Treatment Other 121.Where is the intervention being implemented? (Check all that apply) Yes/No (Reported) Yes/No (Observed) Notes Bar Brothel Clinic Community Center Household Mobile Unit Street-Based Other (Specify) 122. In your opinion, what interventions worked well? Which interventions faced challenges? 123.What materials to you distribute to clients (check all that apply)? A–182 Condoms Yes/No (Re￾ported) Notes Lubricants Pamphlets Referrals to clinics Other (Specify) None 124.How does your program support HIV positive and HIV negative clients? (Assess for clearly defined interven￾tions for HIV positive clients including linkages to community and/or facility-based PWP services) Culm # tested HIV posi￾tive Cumulative # in PWP services (e.g. support groups) Year One Year Two Year Three Assessment of HIV Activities with MSM and Their Sexual Partners 125.Does your organization carry out any interventions that target sex partners of MSM? No______ Yes______ What type of sex partners? Casual Male Partners Regular Male Partners Female Partners 126.What type of interventions do you carry out with sex partners of MSM? (Ask the partner to describe the ac￾tivities, check all that they mention that apply, prompt for gaps, and confirm by observation) Interventions for Sex Partners of MSM Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Other : A–183 Interventions for Sex Partners of MSM Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Other: 127.How do you know that each individual supported under your program receives the defined package of ser￾vices and not just a few of the services? (prompt for referral systems and type; case management; tracking and follow up) 128.We are interested in how you track your clients. Can we ask a few questions about that? (ask the partner to show you the monitoring data and assess how the partner has measured - for the entire period of support - P8.3.D Number of MARP reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required) 129.How do you know that the program for MSMs is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials 130.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 131.Did you project receive support under APHIA II? If yes: A–184 a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Where there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 132.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 133.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention targeting MSM? 134.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 135.Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–185 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: MSM Clients Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–186 FOR MSM THAT PARTICIPATE IN APHIAPLUS-SUPPORTED PROGRAMS This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 10. What type of interventions do you receive from the APHIAplus-supported program? (Ask what services are provided, and then prompt. Check all that apply and confirm by observation) The HIV/STI Package of Services for MSM and Their Sex Partners Element Yes/No (Reported) Yes/No (Observed) Notes Community-based Out￾reach Distribution of Condoms and Condom-Compatible Lubricants HIV Counseling and Test￾ing Active Linkage to Health Care and ART Targeted IEC STI Prevention, Screening, and Treatment Other Assessment of HIV Activities with MSM and Their Sexual Partners 11. Does the organization carry out any interventions that target your sexual partners? No______ Yes______ What type of sex partners? Casual Male Partners Regular Male Partners Female Partners 12. What types of interventions are carried out with your sexual partners? (Check all that apply and confirm by observation) A–187 Interventions for Sex Partners of MSM Type of Partner_____________________________ Yes/No (Reported) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of MSM Type of Partner_____________________________ Yes/No (Reported) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 13. In your opinion, what is working well? What has challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 14. Do you have any recommendations to make the program better for MSM and their sex partners? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–188 15. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for MSM and their sex partners? 16. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 17. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–189 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement non-OVC Result 4 programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–190 FOR ORGANIZATIONS IMPLEMENTING NON- OVC RESULT FOUR ACTIVITIES 136.When was the project initiated? Month: ____________ Year: _________________ 137.Are you familiar with the APHIAplus project? 138.Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. 139.If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 140.What is your catchment area? 141.By targeted population, what is your target number to reach for the year? c. In the past 12 months, by targeted population what is the number you have reached? 142.What is your target number of to reach this year by targeted population? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 143.By targeted population, what type of interventions do you carry out? (Ask the partner to describe the activi￾ties, check all that they mention that apply) Targeted Population: __________________________________________________________ Element – Economic Strengthening Yes/No (Reported) Notes Sustainable IGAs Support VSLA/ SILC groups Other: A–191 Element – Food Security Farmer’s groups and coopera￾tives to scale up food produc￾tion Other: Element – Education, Life Skills, and Literacy Yes/No (Reported) Notes Support to stay in school Vocational training Other: Element – Safe Water, Sani￾tation, Hygiene Hygiene education Distribution of soap, water tanks, purifiers Other: Element – Others: Other: Other: For each targeted audience, what type of support do you give (check all that apply)? Yes/No (Re￾ported) Notes Scholastic support (school fees, books, pencils, uniforms, etc) Shelter and care items (bedding, clothing, soap, cooking utensils) A–192 Point of use water treatment Other (Specify) Other (Specify) None 144.For each targeted population, please describe how beneficiaries are linked into HIV prevention, care, and treatment activities, and other health-related activities and services supported under AHPIAplus. 145.In your opinion, what interventions worked well? Which ones faced challenges? Why? 146.How do you know that the program is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials 147.Gender is an important element of APHIAplus Nairobi/Coast . How have you addressed gender in your pro￾gram a. Prompt for technical interventions that address needs of the targeted population and gender, ad￾dressing male norms and behaviors, improved equitable access to services, referrals to GBV services and legal protection 148.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? A–193 Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 149.Did you project receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Where there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 150.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 151.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention? 152.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 153.Do you have any questions for comments for us? Thank you for your assistance. A–194 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement non-OVC Result 4 programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc). Based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–195 FOR CLIENTS RECEIVING NON- OVC RESULT FOUR ACTIVITIES 154.When was the project initiated? Month: ____________ Year: _________________ Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 18. What type of interventions do you receive from the program? (Ask what services are provided, and then prompt. Check all that apply and confirm by observation as feasible, then afterwards assess what is provided by APHIAplus) Targeted Population: __________________________________________________________ Element – Economic Strengthening Yes/No (Reported) Notes Sustainable IGAs Support VSLA/ SILC groups Other: Element – Food Security Farmer’s groups and coopera￾tives to scale up food produc￾tion Other: Element – Education, Life Skills, and Literacy Yes/No (Reported) Notes Support to stay in school Vocational training Other: Element – Safe Water, Sani￾tation, Hygiene Hygiene education Distribution of soap, water tanks, purifiers A–196 Other: Element – Others: Other: Other: For each targeted audience, what type of support do you give (check all that apply)? Yes/No (Re￾ported) Notes Scholastic support (school fees, books, pencils, uniforms, etc) Shelter and care items (bedding, clothing, soap, cooking utensils) Point of use water treatment Other (Specify) Other (Specify) None 19. In your opinion, what is working well? What have been challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 20. Do you have any recommendations to make the program better? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–197 155.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention? 156.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 157.Do you have any questions for comments for us? Thank you for your assistance. A–198 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement OVC programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc.) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–199 FOR ORGANIZATIONS IMPLEMENTING OVC ACTIVITIES This tool is based on the and the Guidance for Orphans and Vulnerable Children Programming, issued in July 2012 by PEPFAR, and the Framework for the National Child Protection System, issued in November 2011 by the National Council for Children’s Services. 158.When was the project initiated? Month: ____________ Year: _________________ 159.Are you familiar with the APHIAplus project? 160.Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. 161.If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 162.What is your catchment area? 163.How do you define OVC? 164.Considering the tremendous needs for OVC in Nairobi and the Coast, please tell us how you prioritized the use of your funding? (probe for selection of districts/ communities; selection criteria; HES criteria (e.g.; families in destitution, families that struggle to make ends meet; families ready to grow). 165.What is your target number of OVC to reach for the year? d. In the past 12 months, what is the number of OVC you have reached? 166.What is your target number of OVC to reach this year? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 167.What type of interventions do you carry out? (Ask the partner to describe the activities, check all that they mention that apply and verify by observation) A–200 Services for OVC and Caretakers Element – Health Yes/No (Reported) Yes/No (Observed) Notes Parenting skills and support groups Home visits to at-risk and af￾fected families inclusive of early childhood development (growth monitoring, deworming, vitamin A, immunization, and Malezi Bora Campaign For adolescent OVC, HTC re￾ferrals and RH services, HIV prevention knowledge For HIV-infected OVC , adher￾ence support and support groups Disclosure and bereavement support, and other psycho-social services Other: Element – Protection Yes/No (Reported) Notes Legal support and succession planning Linkages with the policy, county administration, Children’s De￾partment, Area Advisory Coun￾cils Sensitive OVC, caregivers, LIPs, and CUs on the Sexual Offenses Act Other: Element – Economic Strengthening Yes/No (Reported) Notes Sustainable IGAs for OVC Support VSLA/ SILC groups, referrals to cash transfers Provide TA at the county, dis￾trict, and community level to budget and provide OVC ser￾vices Other: Element – Food Security Farmer’s groups and coopera- A–201 tives to scale up food produc￾tion for caregivers Other: Element – Education, Life Skills, and Literacy Work with the MOE to ensure that all school-age OVC stay in school Provide scholastic support (books, pens, uniforms, school fees) Provide vocational training for older OVC Other: Element – Shelter Safe Water, Sanitation, Hygiene Hygiene education Distribution of soap, water tanks, purifiers Distribution of bedding, soap, clothing, cooking utensils Other: Element – Others: Other: Other: 168.In your opinion, what interventions worked well? Which ones faced challenges? Why? 169.How do you know that each individual supported under your program receives the appropriate package of services as per their needs and not just a few of the services? (prompt for referral systems and type; tracking and follow up) We are interested in how you track your clients. Can we ask a few questions about that? (ask the partner to show you the monitoring data and assess how the partner has measured - for the entire period of support - P1.1.D Number of eligible adults and children provided with a minimum of one care service (UNIQUE individual) A–202 170.What type of in-kind support do you give to OVC (check all that apply)? Yes/No (Re￾ported) Yes/No (Ob￾served) Notes Scholastic sup￾port (school fees, books, pencils, uniforms, etc) Shelter and care items (bedding, clothing, soap, cooking utensils) Point of use wa￾ter treatment Fees for voca￾tional training Other (Specify) Other (Specify) None 171.How do you ensure that the intended OVC receives this type of in-kind support? (observe systems) 172.We talked a little about this before, but please tell us how does your program support HIV positive and HIV negative OVC and caregivers. (Assess for clearly defined interventions for HIV positive clients including linkag￾es to community and/or facility-based PWP services, the use of support groups, coverage) OVC Culm # tested HIV posi￾tive Cumulative # in PWP services (e.g. support groups) Year One Year Two Year Three Caregiver Culm # tested HIV posi￾tive Cumulative # in PWP services (e.g. support groups) Year One Year Two Year Three 173.How do you know that the program for OVC is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials A–203 174.Gender is an important element of APHIAplus Nairobi/Coast . How have you addressed gender in your pro￾gram a. Prompt for technical interventions that address needs of the targeted population and gender, ad￾dressing male norms and behaviors, improved equitable access to services, referrals to GBV services and legal protection 175.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 176.Did you project receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Where there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 177.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? A–204 Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 178.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention? 179.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 180.Do you have any questions for comments for us? Thank you for your assistance. A–205 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: OVC and Caretaker Clients Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–206 FOR OVC AND CARETAKERS THAT PARTICIPATE IN APHIAPLUS ACTIVITIES This tool is based on the and the Guidance for Orphans and Vulnerable Children Programming, issued in July 2012 by PEPFAR, and the Framework for the National Child Protection System, issued in November 2011 by the National Council for Children’s Services. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 1. Tell us your situation before APHIA plus OVC program and after the intervention of the APHIA PLUS OVC program (before and after scenario). (Probe on the interventions listed below) Element – Health Yes/No (Reported) Notes Parenting skills and support groups Home visits to at-risk and affected families inclusive of early childhood development (growth monitoring, deworming, vitamin A, immunization, and Malezi Bora Campaign For adolescent OVC, HTC referrals and RH services, HIV prevention knowledge For HIV-infected OVC , adherence support and support groups Disclosure and bereavement sup￾port, and other psycho-social ser￾vices Other: Element – Protection Yes/No (Reported) Notes Legal support and succession plan￾ning Linkages with the policy, county ad￾ministration, Children’s Department, Area Advisory Councils Sensitive OVC, caregivers, LIPs, and CUs on the Sexual Offenses Act Other: Element – Economic Strengthen￾ing Yes/No (Reported) Notes Sustainable IGAs for OVC Support VSLA/ SILC groups, referrals to cash transfers Provide TA at the county, district, and community level to budget and provide OVC services A–207 Other: Element – Food Security Farmer’s groups and cooperatives to scale up food production for care￾givers Other: Element – Education, Life Skills, and Literacy Work with the MOE to ensure that all school-age OVC stay in school Provide scholastic support (books, pens, uniforms, school fees) Provide vocational training for older OVC Other: Element – Shelter Safe Water, Sanitation, Hygiene Hygiene education Distribution of soap, water tanks, purifiers Distribution of bedding, soap, cloth￾ing, cooking utensils Other: Element – Others: Other: Other: 2. Have you ever received this type of in-kind support (check all that apply)? Yes/No (Re￾ported) Yes/No (Ob￾served) Notes Scholastic sup￾port (school fees, books, pencils, uniforms, etc) Shelter and care A–208 items (bedding, clothing, soap, cooking utensils) Point of use wa￾ter treatment Fees for voca￾tional training Other (Specify) Other (Specify) None 3. What are your feelings in the manner in the partnership between the local organization that is supporting you, OVC, School and community work?(Probe for)  Written agreement and content if available and known  Steps that were involved in the initiation of the partnership  Mobilization of the different groups  Recruitment procedures of the OVC in the program  Distribution of supplies  Participation of the OVCs in the program and partnership  Training  Linkages, networking and collaboration 4. What were the good things that the community and APHIAplus OVC program did to improve your life as OVC care giver. (Probe for) a. Material b. Non material 5. What were the challenges faced by the APHIAplus OVC program and how did they address them? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 6. Do you have any recommendations to make the program better for OVC and caretakers? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–209 7. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for OVC and caretakers? 8. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 9. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement PLHIV programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–210 FOR ORGANIZATIONS IMPLEMENTING PROGRAMS WITH PLHIV This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR, and the Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. 181.When was the project initiated? Month: ____________ Year: _________________ 182.Are you familiar with the APHIAplus project? 183.Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. 184.If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 185.What is your catchment area? 186.Do you have an estimate of the size of the PLHIV population in your catchment area? If yes, what is it? 187.What is your target number of PLHIV to reach for the year? e. In the past 12 months, what is the number of PLHIV you have reached? 188.What is your target number of sexual partners of PLHIV to reach this year? b. In the past 12 months, what is the number of sexual partners of PLHIV partners you have reached, by type of partner? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 189.What type of interventions do you carry out targeting PLHIV? (Ask the partner to describe the activities, check all that they mention that apply, prompt for gaps, and confirm by observation) The HIV/STI Package of Services for PLHIV, Sex Partners, and Families A–211 Element Yes/No (Reported) Yes/No (Observed) Notes HTC of sexual partners and family members Interventions for HIV dis￾cordant couples (e.g.; risk reduction counseling, con￾dom distribution, referrals to PMTCT, referrals to VMMC for HIV-negative male partners) Sexual risk reduction coun￾seling and on-going sup￾port, with client-driven prevention goal with rele￾vant peer support activities Support of safe disclosure to sexual partners and fam￾ily members Alcohol use assessment and counseling Drug use assessment and referrals to services STI Prevention, Screening, and Treatment Family planning and safer pregnancy counseling Condom distribution and promotion Adherence counseling and support Linking and retaining indi￾viduals into HIV care and treatment services Linkage to other health and non-health services (e.g.; IGAs, VSLAs) 190.How and where is the intervention being implemented? (Check all that apply) Yes/No (Reported) Yes/No (Observed) Notes A–212 Support groups Clinic Community Center Household Mobile Unit Referrals to services Community-based or other forms of case management Other 191.What materials to you distribute to clients (check all that apply)? Yes/No (Re￾ported) Notes Condoms Lubricants Pamphlets Referrals to clinics Other (Specify) None Assessment of HIV Activities with PLHIV and Their Sexual Partners 192.Does your organization carry out any interventions that target Sex partners of PLHIV? No______ Yes______ What type of sex partners? Casual Regular Partners 193.What type of interventions do you carry out with sex partners of PLHIV? (Ask the partner to describe the activities, check all that they mention that apply, prompt for gaps, and confirm by observation) Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes A–213 Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 194.In your opinion, what interventions worked well? Which ones faced challenges? Why? 195.How do you know that each individual supported under your program receives the defined package of ser￾vices and not just a few of the services? (prompt for referral systems and type; case management; tracking and follow up) 196.We are interested in how you track your clients. Can we ask a few questions about that? (ask the partner to show you the monitoring data and assess how the partner has measured - for the entire period of support - P7.1.D Number of People Living with HIV reached with a minimum package of PwP interventions A–214 In order to count under this indicator, PLHIV must have received at last visit (in a clinic/facility-based or commu￾nity/home-based program) the following interventions that constitute the minimum package of PwP: • Assessment of sexual activity and provision of condoms (and lubricant) and risk reduction counseling (if indicated) • Assessment of partner status and provision of partner testing or referral for partner testing • Assessment for STIs and (if indicated) provision of or referral for STI treatment and partner treatment • Assessment of family planning needs and (if indicated) provision of contraception or safer pregnancy counseling or referral for family planning services • Assessment of adherence and (if indicated) support or referral for adherence counseling • Assessment of need and (if indicated) refer or enroll PLHIV in community-based program such 197.How do you know that the program for PLHIV is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials 198.Gender is an important element of APHIAplus. How have you addressed gender in your program? a. Prompt for technical interventions that address needs of the targeted population and gender, ad￾dressing male norms and behaviors, improved equitable access to services, referrals to GBV services and legal protection 199.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 200.Did you project receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Where there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 201.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 202.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention targeting PLHIV? A–215 203.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 204.Do you have any questions for comments for us? Thank you very much for your assistance. A–216 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: PLHIV Clients Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–217 FOR PLHIV THAT PARTICIPATE IN APHIAPLUS-SUPPORTED PROGRAMS This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR, and the Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 21. What type of interventions do you receive from the APHIAplus-supported program? (Ask what services are provided, and then prompt. Check all that apply) Element Yes/No (Reported) Notes HTC of sexual partners and family members Interventions for HIV dis￾cordant couples (e.g.; risk reduction counseling, con￾dom distribution, referrals to PMTCT, referrals to VMMC for HIV-negative male partners) Sexual risk reduction coun￾seling and on-going support, with client-driven preven￾tion goal with relevant peer support activities Support of safe disclosure to sexual partners and fami￾ly members Alcohol use assessment and counseling Drug use assessment and referrals to services STI Prevention, Screening, and Treatment Family planning and safer pregnancy counseling Condom distribution and promotion Adherence counseling and support Linking and retaining indi￾viduals into HIV care and treatment services A–218 Element Yes/No (Reported) Notes Linkage to other health and non-health services (e.g.; IGAs, VSLAs) Assessment of HIV Activities with PLHIV and Their Sexual Partners 22. Does the organization carry out any interventions that target your sexual partners No______ Yes______ What type of sex partners? Casual Regular Partners 23. What types of interventions were carried out with your sexual partners? (Check all that apply and confirm by observation) Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration A–219 and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 24. In your opinion, what is working well? What has challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 25. Do you have any recommendations to make the program for PLHIV, their sex partners, and families better? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 26. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for PLHIV, their sex partners, and families? 27. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 28. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–220 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: PLHIV Clients Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–221 FOR PLHIV THAT PARTICIPATE IN APHIAPLUS-SUPPORTED PROGRAMS This tool is based on the Technical Guidance on Combination HIV Prevention issued in May 2011 by PEPFAR, and the Technical Considerations for FY 2012 Country Operational Plans issued in August 2011 by PEPFAR. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 29. What type of interventions do you receive from the APHIAplus-supported program? (Ask what services are provided, and then prompt. Check all that apply) Element Yes/No (Reported) Notes HTC of sexual partners and family members Interventions for HIV dis￾cordant couples (e.g.; risk reduction counseling, con￾dom distribution, referrals to PMTCT, referrals to VMMC for HIV-negative male partners) Sexual risk reduction coun￾seling and on-going support, with client-driven preven￾tion goal with relevant peer support activities Support of safe disclosure to sexual partners and fami￾ly members Alcohol use assessment and counseling Drug use assessment and referrals to services STI Prevention, Screening, and Treatment Family planning and safer pregnancy counseling Condom distribution and promotion Adherence counseling and support Linking and retaining indi￾viduals into HIV care and treatment services A–222 Element Yes/No (Reported) Notes Linkage to other health and non-health services (e.g.; IGAs, VSLAs) Assessment of HIV Activities with PLHIV and Their Sexual Partners 30. Does the organization carry out any interventions that target your sexual partners No______ Yes______ What type of sex partners? Casual Regular Partners 31. What types of interventions were carried out with your sexual partners? (Check all that apply and confirm by observation) Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of PLHIV Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach HTC, couples HTC, and disclosure support Promotion, Demonstration A–223 and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 32. In your opinion, what is working well? What has challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 33. Do you have any recommendations to make the program for PLHIV, their sex partners, and families better? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 34. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for PLHIV, their sex partners, and families? 35. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 36. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–224 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: PWID Clients Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–225 FOR PWID THAT PARTICIPATE IN APHIAPLUS-SUPPORTED PROGRAMS This tool is based on the Comprehensive HIV Prevention for People who Inject Drugs, Revised Guidance, issued in July 201 by PEPFAR. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 37. What type of interventions do you receive from the APHIAplus-supported program? (Ask what services are provided, and then prompt. Check all that apply and confirm by observation) The HIV/STI Package of Services for PWID Element Yes/No (Reported) Yes/No (Observed) Notes Community and peer- based Outreach Drug Dependence Treat￾ment or Therapy HIV Counseling and Test￾ing ART for PWID living with HIV Condoms and condom- compatible lubricants for PWID and their sexual partners IEC for PWID and their sexual partners Vaccination and diagnosis of viral hepatitis Prevention, diagnosis, and treatment of TB STI Prevention, Screening, and Treatment Other Assessment of HIV Activities with PWID and Their Sexual Partners 38. Does the organization carry out any interventions that target your sexual partners? No______ Yes______ What type of sex partners? Casual Partners Regular Partners A–226 39. What types of interventions are carried out with your sexual partners? (Check all that apply and confirm by observation) Interventions for Sex Partners of PWID Type of Partner_____________________________ Yes/No (Reported) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of PWID Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 40. In your opinion, what is working well? What has challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 41. Do you have any recommendations to make the program better for MSM and their sex partners? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? A–227 42. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for PWID and their sex partners? 43. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 44. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance A–228 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: CBOs that implement SW programs Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–229 FOR ORGANIZATIONS IMPLEMENTING PROGRAMS WITH SEX WORKERS This tool is based on the National Guidelines for HIV/STI Programs for Sex Workers issued in September 2010 by NASCOP. 205.When was the project initiated? Month: ____________ Year: _________________ 206.Are you familiar with the APHIAplus project? 207.Please discuss the nature and type of support you have received from APHIAplus Nairobi/Coast. (If the project started after January 1 2011 and/or if the project receives funding of other sources, clearly define the nature and type of support that can be directly attributed to APHIAplus. 208.If you receive funding from other donors, can you tell me who they are and briefly what they do? Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 209.What is your catchment area? 210.Do you have an estimate of the size of the sex work population in your catchment area? If yes, what is it? 211.What is your target number of sex workers to reach for the year? f. In the past 12 months, what is the number of sex workers you have reached? 212.What is your target number of sex partners to reach this year? c. In the past 12 months, what is the number of sex workers partners you have reached, by type of partner? Evaluation Question #2: What strategies employed within the workplan worked and what did not work for suc￾cessful implementation and achievement of key outcomes in Result 3 and 4? Why? Evaluation Question #3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the Health Sector? 213.What type of interventions do you carry out targeting sex workers? (Ask the partner to describe the activi￾ties, check all that they mention that apply, prompt for gaps, and confirm by observation) The HIV/STI Package of Services for Sex Workers and Their Sex Partners Behavioral Components of the HIV/STI Package of Services Element Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out- A–230 reach Risk Assessment, Risk Re￾duction Counseling and Skills Build￾ing Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Screening and Treatment for Drug and Alcohol Abuse Biomedical Components of the HIV/STI Package of Services Yes/No (Reported) Yes/No (Observed) Notes HIV Testing and Counseling STI Screening and Treat￾ment TB Screening and Referral to Treatment HIV Care and Treatment Reproductive Health Ser￾vices Family Planning  Post-Abortion Care Services  Cervical Cancer Screening Emergency Contraception Post-Exposure Prophylaxis Structural Interventions Yes/No (Reported) Yes/No (Observed) Notes 100% Condom Use Pro￾gramme Services to Mitigate Sexual Violence Support to Expand Choices Beyond Sex Work Additional Components of the HIV/STI Package of Services Yes/No (Reported) Yes/No (Observed) Notes A–231 Psycho-Social Support Family and Social Services Other Components 214.Where is the intervention being implemented? (Check all that apply) Yes/No (Reported) Yes/No (Observed) Notes Bar Brothel Clinic Community Center Household Mobile Unit Street-Based Other (Specify) 215.What materials to you distribute to clients (check all that apply)? Yes/No (Re￾ported) Notes Condoms Lubricants Pamphlets Referrals to clinics Other (Specify) None 216.How does your program support HIV positive and HIV negative clients? (Assess for clearly defined interven￾tions for HIV positive clients including linkages to community and/or facility-based PWP services) Culm # tested HIV posi￾tive Cumulative # in PWP services (e.g. support groups) A–232 Year One Year Two Year Three Assessment of HIV Activities with Sex Workers and Their Sexual Partners 217.Does your organization carry out any interventions that target Sex partners of SWs? No______ Yes______ What type of sex partners? Clients Regular Partners 218.What type of interventions do you carry out with sex partners of sex workers? (Ask the partner to describe the activities, check all that they mention that apply, prompt for gaps, and confirm by observation) Interventions for Sex Partners of Sex Workers Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services Interventions for Sex Partners of Sex Workers Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 219.In your opinion, what interventions worked well? Which ones faced challenges? Why? A–233 220.How do you know that each individual supported under your program receives the defined package of ser￾vices and not just a few of the services? (prompt for referral systems and type; case management; tracking and follow up) 221.We are interested in how you track your clients. Can we ask a few questions about that? (ask the partner to show you the monitoring data and assess how the partner has measured - for the entire period of support - P8.3.D Number of MARP reached with individual and/or small group level preventive interventions that are based on evidence and/or meet the minimum standards required) 222.How do you know that the program for SWs is meeting the needs of your clients? a. Prompt for needs assessment, quality assurances standards, continuous monitoring with results in￾forming improved performance, the development and application of QA/QI tools b. Observe QA/QI tools and materials 223.With APHIAplus support, has your program developed or implemented any aspects to the program that you would consider to be particularly innovative? What are these? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 224.Did you project receive support under APHIA II? If yes: a. Can you tell us what the transition was like? For example, was the transition process smooth or had challenges? Do you have any recommendations on how to improve such a transition process? b. Where there changes in your SOW or funding level when your support moved to APHIAplus? c. What have you been able to do differently as a result of receiving support from APHIAplus? 225.What do you think needs to be done to make your program as strong as you think it should be, if future sup￾port were available? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 226.Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention targeting sex workers? A–234 227.If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 228.Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. A–235 Group Interview Guide and Observation Checklist Most Vulnerable Populations Package: SW Participants Name of Interviewee: Date: Location (Office/Facility or Community Name): Position/Title: Time: County/Region: Profession/Qualification: Gender: Female/Male Name of interviewer: Hello. My name is _______________________and this is my colleague _________. We are part of a team conducting an end-of-project performance evaluation of the APHIAplus program implemented in Nairobi and Coast. The evaluation is investigating the implementation of the project from January 1, 2011 up to now. The aim of the end-of-project evaluation is to better understand the project, looking at performance, successes and chal￾lenges, and lessons learned for the future. In this regard, you have been selected to represent (name the specific ministry/ organization/ etc) based on your position, knowledge and expertise. Your participation in this process is VOLUNTARY and the information you provide will be useful in finding out the status of APHIAplus activities and support in your (name region/ ar￾ea) and will be used to plan future intervention programs in this (name region/ area) and in the country. I would like to ask you some questions related to APHIAplus Nairobi/Coast as per your area of expertise. There are no right or wrong answers. Please be as honest and specific as possible so that we may best understand your perspective on the project’s work. We will be taking notes during the discussion. This will help us to ensure that we captured all the important in￾formation discussed. All the information you give will be used to prepare the End of Project Evaluation report for USAID. The report will include a synthesis of information we gather from all stakeholders, data and documents, but findings will not be linked to specific names. If you have any questions about the evaluation, you are free to ask us now or contact us later. Do you have ques￾tions at this point about this discussion? Thank you. Then let us proceed. A–236 FOR SEX WORKERS WHO PARTICIPATE IN APHIAPLUS-SUPPORTED PROGRAMS This tool is based on the National Guidelines for HIV/STI Programs for Sex Workers issued in September 2010 by NASCOP. Evaluation Question #1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 45. What type of interventions do you receive from the program? (Ask what services are provided, and then prompt. Check all that apply and confirm by observation as feasible, then afterwards assess what is provided by APHIAplus) The HIV/STI Package of Services for Sex Workers and Their Sex Partners Behavioral Components of the HIV/STI Package of Services Element Yes/No (Reported) Notes Peer Education and Out￾reach Risk Assessment, Risk Re￾duction Counseling and Skills Build￾ing Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Screening and Treatment for Drug and Alcohol Abuse Biomedical Components of the HIV/STI Package of Services Yes/No (Reported) Notes HIV Testing and Counseling STI Screening and Treat￾ment TB Screening and Referral to Treatment HIV Care and Treatment Reproductive Health Ser￾vices Family Planning  Post-Abortion Care Services  Cervical Cancer Screening Emergency Contraception A–237 Post-Exposure Prophylaxis Structural Interventions Yes/No (Reported) Yes/No (Observed) Notes 100% Condom Use Pro￾gramme Services to Mitigate Sexual Violence Support to Expand Choices Beyond Sex Work Additional Components of the HIV/STI Package of Services Yes/No (Reported) Yes/No (Observed) Notes Psycho-Social Support Family and Social Services Other Components Assessment of HIV Activities with Sex Workers and Their Sexual Partners 46. Does the organization carry out any interventions that target your sexual partners? No______ Yes______ What type of sex partners? Clients Regular Partners 47. What types of interventions are carried out with your sexual partners? (Check all that apply and confirm by observation) Interventions for Sex Partners of Sex Workers Type of Partner_____________________________ Yes/No (Reported) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services A–238 Interventions for Sex Partners of Sex Workers Type of Partner_____________________________ Yes/No (Reported) Yes/No (Observed) Notes Peer Education and Out￾reach Promotion, Demonstration and Distribution of Male and Female Condoms and Water-Based Lubricants Male Circumcision Access to HIV/STI services 48. In your opinion, what is working well? What have been challenges? Evaluation Question #4: To what extent was the APHIAplus model appropriate for achieving key outcomes under Result 3 and 4 of the Implementation Framework? 49. Do you have any recommendations to make the program better for SWs and their sex partners? Evaluation Question #5: What sustainable activities in service delivery and health care systems were established and/or strengthened at the district health facility and/or community level? 50. Can you explain to us how the government and the community has supported or taken ownership of this in￾tervention for SWs and their sex partners? 51. If APHIAplus support ended tomorrow, would this program still continue? Why or why not? If yes, please explain that systems or processes that were put in place to help the program continue. 52. Are there any other comments or observations that you would like to make about the APHIAplus-supported project? Thank you very much for your assistance. 1 FINAL PERFORMANCE EVALUATION STATEMENT OF DIFFERENCE AWARD NUMBER: USAID/KENYA RFA NO: 623-10-000009 PREPARED FOR: AOR: MS. JERUSHA KARUTHURI USAID /KENYA C/O AMERICAN EMBASSY UNITED NATIONS AVENUE, GIGIRI P.O. BOX 629, VILLAGE MARKET 00621 NAIROBI, KENYA PREPARED BY: MRS. EILEEN MOKAYA CHIEF OF PARTY SUBMITTED ON FEBRUARY 28, 2014 PATHFINDER INTERNATIONAL INTERNATIONAL LIFE HOUSE, MEZZANINE FLOOR MAMA NGINA STREET PO BOX 48147 -00100 GPO, NAIROBI, KENYA WWW.PATHFINDER.ORG USAID KENYA -APHIAPLUS NAIROBI-COAST HEALTH SERVICE DELIVERY PROJECT 1 ANNEX J. STATEMENT OF DIFFERENCES 2 Table of Contents 1. Introducing the Statement of Difference.................................................................................... 3 2. Framework for the Statement of Difference.............................................................................. 4 3. Statement of Difference at the level of the Main Findings.......................................................... 5 4. Statement of Difference at the level of Specific Evaluation Objectives ....................................... 6 5. Statement of difference on Methods and Limitations. ............................................................... 6 TIMING OF THE EVALUATION..................................................................................................... 6 PROJECT IMPLEMENTATION and DOCUMENTATION.................................................................. 7 PRIMARY DATA SAMPLING FRAME. ........................................................................................... 8 HEALTH MANAGEMENT INFORMATION SYSTEMS...................................................................... 8 KEY INFORMANT INTERVIEWS.................................................................................................... 8 CONFOUNDERS.......................................................................................................................... 9 6. Statement of difference with regard to Validation of Data......................................................... 9 7. Statement of Difference at the Level of the Five Evaluation Questions....................................... 9 Question 1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? .............. 9 Question 2: What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in results 3 and 4? Why? ............ 19 Question 3 : To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the health sector? .................................................................. 27 Question 4: To what extent was the APHIAplus model appropriate for achieving key outcomes under results 3 and 4 of the implementation framework. ............................................................ 28 Question 5: What sustainable activities in service delivery and healthcare systems were established and/or strengthened at the district health facility and/or community level? ............. 28 DISCLAIMER : The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. 3 1. Introducing the Statement of Difference The essential features, components and contextual frameworks of the APHIAplus Nairobi Coast Service Delivery Project have been well summarized in the project background section of the Final Performance Evaluation and are those that one finds almost textually in the RFA, in Pathfinder International’s Proposal, in the work plans and the quarterly reports, briefs and technical papers. What is made less explicit is a set of features that are equally obvious and that have retained the attention of the APHIAplus Nairobi Coast team from the very beginning of the project. APHIAplus Nairobi Coast is a great example of a multi-dimensional, multi-scale, multi-component and value laden configurable system that operates in a rather unstable and changing environment. The system that it interacts with is a source of very difficult scientific challenges for observing, understanding, reconstructing and predicting its multi-scale dynamics1 . In addition to being a classical service delivery project, APHIAplus Nairobi Coast is an active exploration of implementation strategies that make the guiding principles of the Global Health Initiative the core principles of a technical assistance project to a host country. The evaluation team states that APHIAplus implements three overarching strategies: the Foundation Package, the Most Vulnerable Groups Package, and Partnership Packages (page 5). The project team has no objection that the evaluation team frames APHIAplus’ strategies in reference to the original proposal and in reference to the narrative of the year three work plan. But it is fully incorrect to state that these three packages represented the overarching strategies of APHIAplus or to identify this package concept as the overarching strategies “used in the work plans”. It is correct to say that after having been awarded the grant, project management did not refer in any way to the package concept in the work plan of year one and the work plan of year two. The packages re-surfaced only in year 3 work plan based on the recommendation of USAID to all APHIAplus projects to “return to their original proposals and to deliver what was promised in these well written proposals”. It is correct to say that in lieu of the overarching strategies mentioned, the project choose adherence to the principle based approach of the Global Health Initiatives to be a core strategy, accompanied by a health systems strengthening strategy, different program specific strategies (HIV/AIDS, MCH, FP/RH) and a monitoring and evaluation strategy each of which was reviewed in successive work plans. The evaluation team never mentioned the overarching health systems and health systems strengthening strategy, which referred intensively to the national health systems strengthening framework, but more importantly, also constantly, to the international literature in this subject area, as part of the evidence based approach of the project. The documents submitted to the evaluation team enable it to identify how year by year the articulation of these strategies in the work plan became more elaborate, mainly in response to USAID’s explicit request to clearly articulate such strategies. 1 This is freely quoted from Complex systems science as a new trans disciplinary science. Paul Bourgine, Diplomacy and Foreign Affairs, December 2013. 4 The reason for not using the package concept was twofold: first, they are difficult to operationalize and second, since one has to break them down in relevant sub-packages, it is more economical and sustainable to adopt the packages of the host country - when they exist - and to adopt the overarching frameworks of the host country, rather than using concepts and vocabularies that are expensive to entertain and that are without marginal benefit to the project. When management re-introduced the packages in November 2012, their inclusion had no major impact on the work plan: they did not affect the work plan structure nor the monitoring and evaluation structure. 2. Framework for the Statement of Difference Purpose, specific objectives and evaluation questions are the framework that guides the inclusion of observations in this statement of difference. The purpose: 1. to assess the performance of APHIAplus, primarily for accountability and learning. The specific objectives: 1. to conduct an in-depth qualitative assessment of technical and management approaches, coordination with the host country and other stakeholders, and support to country health systems 2. to conduct a review of project achievements based on assigned targets in HIV/AIDS, family planning/reproductive health, and child survival activities. The evaluation questions: 1. To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 2. What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in Results 3 and 4? Why? 3. To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the health sector? 4. To what extent was the APHIAplus model appropriate for achieving key outcomes under Results 3 and 4 of the Implementation Framework? 5. What sustainable activities in service delivery and healthcare systems were established and/or strengthened at the district health facility and/or community level? 5 3. Statement of Difference at the level of the Main Findings The evaluation team has clearly stated the purpose of the evaluation, i.e. how the evaluation outcome should satisfy a prime concern for accountability and how it should feed into a learning process that is built in USAID’s management cycle. Our statement is that this purpose has not been attained. As Prime, Pathfinder International is entitled to know how the external evaluation team rates this project: was it money well spent, is the GOK a beneficiary of high quality technical assistance, can one say that this project is loyal to the USAID Implementation Framework and to the Partnership Agreement, to the PEPFAR funding and to the principles of the Global Health Initiatives. APHIAplus Nairobi Coast is a multi-dimensional, multi-component and multi-level project. Without addressing these multiple facets one reduces it to an activity based project. Under the section lessons learned the evaluation team proposes eight lessons. If we locate the eight lessons learned in such multi-dimensional framework, we see only one dimension being addressed: the project management dimension. No major lesson about service delivery, no lesson about addressing social determinants, no lessons about sustainability in health systems, nothing about strategies. Lessons learned (shortened) What is it about? 1. Establishing the purpose and objectives of each strategy or ‘package’ of interventions is closely linked to successful implementation. Articulation of strategy 2. Changing track midstream, in terms of work planning, PMPs and activities, must be accompanied by an attendant monitoring mechanism. Reflecting change in monitoring systems 3. Applying standard monitoring and reporting mechanisms will enable the donor to pick up areas of concern, or inconsistencies… Tools for project oversight 4. In order to accurately track project outcomes and results, clear baseline assessments must be done … Importance of baselines 5. Clear direction during project handover is essential to ensure key interventions are continued and successes built upon … Start up and close out procedures 6. A clear sustainability initiative built into the project design at the inception phase, especially as relates to sub-grantee activities, will help to focus partners on ensuring the longevity of the activities… Addressing sustainability in sub granting 7. Utilizing best practices and aligning interventions to current international/national guidance is essential … Importance of evidence based design 8. Formalized sharing between projects that engage similar populations and focus on similar health issues is essential … Knowledge management 6 4. Statement of Difference at the level of Specific Evaluation Objectives There is no clear-cut conclusion provided on the technical and management approaches, coordination with the host country and other stakeholders, and support to country health systems. Nor is there a clear statement made on what has been achieved with regards to the targets set in HIV/AIDS, Family Planning/Reproductive Health, and Child Survival activities. We expect here statements at a higher level than the activity level, i.e. at the experiential and conceptual level. We express our different view with regard to the level from which material from the evaluation questions is explored. We have no answer to the question whether our technical approach is sound, whether our management approaches satisfactory, whether our coordination with host country meets standards imposed by the Paris Declaration, and is our support to country health systems in line with Kenya and USG standards. Nor is the simple question answered with regard to targets that the project set and its achievements against targets. 5. Statement of difference on Methods and Limitations The evaluation team states that it “experienced numerous significant limitations that affected the formation of valid evidence based conclusions” Page 4. “ The evaluation team experienced numerous significant limitations that affected the formation of valid and evidence-based conclusions—namely 1) the timing of the evaluation, 2) the project design, 3) project documentation, 4) the primary data sampling frame, 5) health management information systems, 6) KIIs, and 7) various confounders—each of which we will outline here. Many of the data limitations listed herein are also noted findings that will be further detailed within the appropriate section of the report.” TIMING OF THE EVALUATION “… Data for the evaluation were collected when most activities had only a year or less of data to report, which limited the results found mostly to outputs and preliminary outcomes; higher-level outcome and any impact-level (including sustainability) results and trends/progress were nearly impossible to assess. This means the evaluation needed to concentrate on the implementation processes, the current status of activities, and early results—limiting an assessment of outcomes. The evaluation team finds that given that “most activities that had only a year or less of data to report” it encountered substantial restrictions (e.g. “nearly impossible”) of its ability to explore projects achievements within the relevant frame of the input/output/outcome/impact model. 7 The project team states that it is absolutely incorrect that “most activities had only a year or less to report”. It is true that the first quarter report had little performance data to report, that the second quarter reported well on services but still little on prevention activities, but from the third quarter onwards the project was effectively on track with intensive activity reporting. It is also true that over the three year timeline the effective start of some activities shifted to a later period than planned, or did not take place at all. And it is also true that the project did not hesitate to introduce new activities, even early in the third year, when based on decision making processes involving often MOH, USAID, Partners and Project in any combination, adjustments to the work plan were needed. This overall pattern of phasing in and phasing out activities, cannot be summarized by the evaluation’s team’s statement “most activities had only a year or less data to report”. The correct statement should be that the timing of the evaluation had no fundamental impact on the ability of the evaluation team to fully explore project achievements fully exploring different facets suggested by the input/output/outcome/ impact model. PROJECT IMPLEMENTATION and DOCUMENTATION Page 4 and page 12. “APHIAplus has undergone significant changes in activities, PMPs, and the structure of work plans. This created two major limitations: a) indicators and their targets were inconsistent, making tracking of progress and achievements hard to assess, and b) it was difficult to deduce linkages connecting results, outcomes, activities, and targets within results found. This led to difficulty in assessing effectiveness of implementation strategies.” The evaluation team labels changes in activities, PMPs and work plan structure as “significant” and attributes to such “significant changes” the existence of two issues which would have limited the ability of the evaluation team to assess the effectiveness of the implementation strategies. The first issue is inconsistency of indicators and targets, the second issue is the difficulty to “deduce linkages connecting results, outcomes, activities, and targets within results found”. The project team states that it is incorrect to qualify the changes in activities, PMP and work plan as “significant” and of such a degree of significance that they ultimately affect the evaluation team’s ability to assess effectiveness of implementation strategies. APHIAplus monitors the metrics of work plan activities (similar or identical versus new activities, year by year) and prudent stability is the ground rule. PMPs were subject of alterations, and got better indeed, year by year, while work plan structure was adjusted purposefully, year by year, to what we call full alignment with GOK strategic and planning frameworks. We state that the annual adjustments were sensible and purposeful and were a reflection of better articulation of different implementation strategies. As such these changes are contributing to rather than limiting the assessment of the effectiveness of implementation strategies. Whereas we raise no objection against having found evidence of inconsistency at indicator and target level - see limitation a)-, we absolutely find no place for an inconsistent statement like limitation b) in this report section. In addition, we state that with regard to documentation, our internal evaluations have always identified documentation as an inbuilt weakness of the project. It has broad consequences at any stage in the project cycle. 8 PRIMARY DATA SAMPLING FRAME Page 12. The facilities visited were selected by the Mission and were a convenience sample tiered by demand (high/medium/low), skewing sampling toward mid-demand facilities—as opposed to a stratified random sample to produce more representative results. In addition, APHIAplus’s scope includes a wide range of activities. As such, it was difficult to assess all components against project documentation. Thus, the results in this report provide only an indicative snapshot of the project’s performance and cannot be generalized. The project team disagrees with the statement about the sampling frame. We do not know with precision the involvement of the evaluation team with the constitution of the sampling frame and the selection of the sites, but note that the same evaluation team describes the site sampling in different ways in the inception report and in the final report. It was understood that the site visits would provide an opportunity to provide context to the findings from other sources. It would be technically impossible to obtain a stratified random sample of the given size that would be more representative results. The smallness of the sample size would prevent any data to be generalizable to the total population. HEALTH MANAGEMENT INFORMATION SYSTEMS. “Incomplete data sets and the inability to retrieve data in a timely manner (before tool development) impeded the ability of the evaluation team to design the tools as initially envisioned. The data extraction and analysis was carried out after the fieldwork.” The evaluation team describes how the design of tools was frustrated by issues that have to do with data and data retrieval. We state that this statement is extremely imprecise and leaves the reader guessing what is implied. While the heading is “health management information systems” and leaves it to the reader to guess whether the title refers to the HMIS of the MOH or to information systems in general or information systems of the project, the argument made leaves it completely open if the problem lies with the project or with the project context. The project team states that given the imprecision identified above we consider that no statement of difference is required that would relate the statement of the evaluation team to project related issues. On the other hand we do not agree with a statement that data would be incomplete and impossible to retrieve. We refer to the well-articulated monitoring and evaluation strategy (of each single year) of the project and to the insight it provides into the rather challenging data environment in which APHIAplus Nairobi Coast has to operate. Our statement of difference relates to the perceived difficulty to get data. It needs particular skills indeed, but it is not at all impossible to achieve within the timeline of an evaluation mission. KEY INFORMANT INTERVIEWS Page 13. “Because of the devolution process, many potential key informants have recently been redeployed to new positions. Often individuals were recalled by APHIAplus to meet with the evaluation team, which led to interviews being carried out with former instead of current staff. The evaluation fieldwork took place during the teachers strike, and the team was unable to meet with teachers and school officials or visit schools to observe school-based project activities”. 9 The project team uses the statement of the evaluation team on key informant interviews as an opportunity to express our disagreement with the weight given to such considerations. The above statement affects only the school programs and a simple statement that school programs have not been reviewed would be sufficient CONFOUNDERS “Further attribution was not possible given the number of other actors in the project areas working with similar mandates to APHIAplus, as well as the fact that the project is a follow-on project of past APHIA projects. The focus of the evaluation is therefore on contribution.” The evaluation team refers to the reality that large scale projects such as APHIAplus Nairobi Coast do not operate in a partner vacuum, and that projects that base their monitoring and evaluation on opposite assumptions will encounter plenty of conceptual problems. The project team expects the evaluation team to be fully familiar with those issues, since project design and interpretation of the concept of “results” has taken this reality fully in account. This position was documented from year one onwards in the work plan. We therefore state our difference: this “confounders” statement has no relevance in a section that describes the “numerous or significant limitations that affected the formation of valid and evidence-based conclusions by the evaluation team”. 6. Statement of difference with regard to Validation of Data The project team identifies a weak data validation system as a serious limitation of the evaluation. While the concept of external and independent evaluation is valid, it assumes that the evaluation team applies clear frames of reference that avoid arbitrariness, especially when context and context interpretation interferes with data interpretation. Given the size of the team, the nature of the project and the time allocated to the evaluation a stronger system of validation of data should have been put in place. In August the project team has forwarded a 29 page Consolidated Comments on Draft of Final Performance Evaluation to the evaluation team through the AOTR. Although many error comments were used in one way or another, substantial differences of fact and interpretation remain. 7. Statement of Difference at the Level of the Five Evaluation Questions Question 1: To what extent has the APHIAplus Nairobi/Coast been effective in contributing to achieving Results 3 and 4? Were the key expected outcomes achieved? If not, why not? 10 Page 29 Findings and Conclusions. “ Between years 1 and 2, the project added and dropped numerous indicators. The majority of indicators changed were project indicators (15 dropped from year 1 and 24 added in year 2); also, at the request of USAID, 7 new PEPFAR standard indicators were added. The evaluation team states that indicator changes were numerous and contribute to difficulties in tracking achievements. The project team list those indicator changes and states that these changes do not affect the logic or flow of project activities. Project Indicator Changes: clarifications provided to evaluation team Indic. No. Indicator Reason for change of the indicator after year 1. RESULT 3: 3.1: Increased availability of quality health services, products an information RH/FP 3.1.6-46 Percent of births delivered by caesarean section Dropped a from Year 2 onwards because not retained in the program log frames MIN Number of WRA receiving FP commodities This was an AOP Indicator tracked only by Coast province in year 1) 3.4: Increased program effectiveness through innovative approaches PR Number of male champions trained for FP/RH, GBV, VMMC This was an innovation in year 1, but in yr. 2 it was adjusted to reality and changed and moved to sub result 4.6 Expanded social mobilization for health: · Number of male champions networks supported. PR Number of support groups for vulnerable populations formed or strengthened (MARPs, PLHIV, youth) Dropped as from year 2 since it was an innovation in year 1 RESULT 4: 4.1: Marginalized, poor and underserved groups have increased access to economic security initiatives through coordination and integration with economic strengthening programs PR Number of households linked to household economic strengthening initiatives From year 2 onwards the indicator was broken down into: · Number of households trained on VSL · Number of households that have initiated an income generating activities. · Number of CBOs linked to micro-finance institutions 4.2: Improved food security and nutrition for marginalized, poor and underserved populations PR Number of individuals receiving nutrition literacy education From year 2 onwards this indicator was merged and reported as: · Number of community education sessions conducted (community dialogue days, ETL sessions, drama, etc.) PR Number of nutrition education sessions conducted by CHEWs, CHWs, farmers' groups 11 Project Indicator Changes: clarifications provided to evaluation team Indic. No. Indicator Reason for change of the indicator after year 1. 4.6: Expanded social mobilization for health PR Number of Shuga, GATE and Gjue events held From year 2 onwards, this indicator was merged and reported as : · Number of special events supported (MOYA youth week, Malezi Bora/BF weeks, etc.) PR Number of adolescents/youth reached through drama outreaches in secondary schools on RH/FP, GBV, VMMC There was change of strategy from year 2 onwards: instead of targeting secondary schools using drama outreaches the project targeted the community units. Hence this indicator was dropped. It existed already under community strategy. SYSTEMS STRENGTHENING PR Number of DHMT/PHMT members trained on COPE It was dropped due to the adoption of KQMH that MOH launched. COPE is a subset of the KQMH. PR Number of Community Units trained on Community COPE It was dropped due to the adoption of KQMH that MOH launched. COPE is a subset of the KQMH. PR Number of DHMT members oriented on DDIU (Data Demand & Information Use) This indicator was tracked and reported by FHI at the national level since they were funded to train and orient DHMT Members on DDIU. This avoid duplication of roles Page 30. Question 1 Conclusion Par 1. “The complicated structure and multiple formats … Without a clear causal logic, it is difficult to determine APHIAplus’ s contribution to key indicators, outcomes, and results. ..Overall lack of fidelity between project documents makes progress against work plans and achievements difficult to track. Beyond this, though the year 3 M&E approach is a very promising step forward in measuring outcomes, the team is unable to assess the projects contribution toward Results 3 and 4 based on this new approach, as it has been implemented only during the first quarter of this year and as of the writing of this report the new approach described in the year 3 M&E plan has not been reported on.” The project team states that the causal logic diagram concept was presented in the project proposal, that it was integrated in the monitoring and evaluation plan of year one, that it was used in the joint work planning exercise of year one, that by June 2011 several of these causal logic diagrams were fully developed and that they were used for internal programming in order to clarify for staff the intervention logic for each of the components. They were fully worked out in the second half of 2012 as part of managements’ resolve to strengthened outcome oriented programming for each intervention. On the other hand, it is correct to say that the causal logic diagrams imposed a data collection burden which was clearly beyond what the project could afford given the intense monitoring workload already imposed on the project. Page 30-31 Question 1 Findings and Conclusions Par. 4. “Between years 1 and 2 in both Coast and Nairobi, the project increased the proportion of indicators meeting targets, but in year 2 the project was still meeting fewer than 50 percent of indicator targets. 12 The project team had already provided feedback to the evaluation team about the use of wrong primary information to evaluate the proportion of targets that were met (Consolidated Comments on the Draft of the Performance Evaluation), but the final document presents the same incorrect data above as the draft document. In order to arrive at this count one cannot plainly add indicators printed out in the PMP. Some of them indeed represent repetitions. Such a table should have been discussed ( see section on Data Validation added under 5. Methods and Limitation). For YR 1 and YR 2 the proportion of targets that were met at the 80 % level are presented below. A similar table, this time calculating those that met at the 100 % level, is displayed next. Percentage and Number of Targets Met at the 80% level Nairobi Coast YR 1 YR 2 YR 1 YR 2 % indicators on target 85% 83% 62% 85% (out of total number) 49/58 54/65 40/65 63/74 Percentage and Number of Targets Met at the 100 % level Nairobi Coast YR 1 YR 2 YR 1 YR 2 % indicators on target 84% 71% 56% 66% (out of total number) 48/58 46/65 37/65 49/74 Page 32. APHIAplus Contribution to Outcome-Level Results , Par 3 “ Under IR 3.1, figure 1 illustrates a decrease in the Percentage of HIV–positive pregnant women who received ARVs to reduce the risk of mother-to-child transmission for Nairobi and Coast by the project (out of the total HIV positive pregnant women for each region)…” Table 3. Percentage and Number of Targets Met Nairobi Coast YR 1(2011) YR 2(2012) YR 1(2011) YR 2 (2012) Percentage of indicators on target 15 40 20 42 (out of total number) 10/65 29/73 14/70 33/79 Conclusion. While target setting was clearly a challenge, the project balanced the flexibility of meeting the emerging needs of the GOK and communities with the requirements of USAID. That said, there is a significant issue in that the project still met fewer than 50 percent of targets set as of year 2 and as USAID reports performance in these terms for its annual report to Washington. 13 Figure 1. Percent of HIV–Positive Pregnant Women Who Received Antiretroviral Drugs to Reduce the Risk of Mother-to-Child Transmission The evaluation team presents graphs for two periods of Nairobi and Coast province each, in which three phenomena are communicated: drop of the indicator over the two intervals, lower levels of the indicator for Nairobi Province and more pronounced drop of the indicator for Nairobi. The project team states that the primary data are inaccurate. They do not reflect the data of the facilities that are supported by the project and are in contradiction with the validated quarterly data, the semiannual report (SAR) and the annual report (AR). The evaluation team also compares a first quarter figure of one year with an annual figure of a previous year while four quarters in that year have varying patterns. In such situations the unit of comparison should be the quarter, and the relevant comparison is the quarter of the most recent year with preceding quarters. From a data visualization point of view it is clear that the graphs produced by the evaluation team depict with great efficiency exclusively downward trends and as such mislead the viewer about the statistical information that should relate to the project. The properly scaled and dimensionalized statistical information available tells a story that cannot be understood without the proper contexts (see graphic below) : 1. In 2011 the project improves detection of HIV+ pregnant women and minimally improves prophylaxis rate ( Q1-Q3) ; 2. Sudden increase of pregnant women due to “rationalization” of facilities between CDC and USAID supported projects in Q4 (the number of facilities that offer PMTC increases in APHIAplus NC) and major efforts to improve overall prophylaxis rate ; from Q5 till Q9 stable pattern of ANC attendance and pregnant women ANC HIV+ , with high rate of prophylaxis achieved (not maximal). The data show a dip in Quarter 8 ( or the fourth quarter of 2012). This phenomenon was well studied and reported to USAID , since analysis by APHIAplus Nairobi Coast demonstrated that this was the effect of the nurses strike that affected the health services in all provinces of Kenya and thus all APHIAplus projects (5) (graph based on DHIS data not shown here). The latter information was provided to the evaluation team. 14 Figure : Total Nairobi Coast data for HIV + pregnant women who receive ARVs at ANC. Figure: Percent of HIV + pregnant women who receive ARVs at ANC The evaluation team has correctly copied the feedback on the facility coverage given by the project team, but has put this information in a footnote without correcting the data so that they correspond to the facilities that the project reports on. Below is the relevant facility coverage NAIROBI COAST INDICATOR Total sites that reported on the indicator A+ supported Sites % of sites supported by A+ Total sites that reported on the indicator A+ supported Sites % of sites supported by A+ Figure 1. Percent of HIV–Positive Pregnant Women Who Received Antiretroviral Drugs to Reduce the Risk of Mother-to-Child Transmission 225 78 35% 388 322 83% Figure 2. Percentage of Infants Tested by HIV at Six Weeks 123 46 37% 192 156 81% Figure 5. Percentage of Pregnant Women Who Receive at Least Four ANC Visits. 225 78 35% 388 322 83% Figure 6. Under One Fully Immunized. 318 68 21% 404 341 84% 0 500 1,000 1,500 2,000 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 ANC HIV+ Mothers given ARV prophylaxis at ANC 0 10 20 30 40 50 60 70 80 90 100 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 15 Page 35 Figure 4 . “ Figure 4 presents two project indicators used to demonstrate the level of TB/HIV integrations. Performance has been more or less consistent over the life of the project in Nairobi, while in Coast the project underperformed in the first year but shows consistent performance for 2012 and the first quarter of 2013” Figure 4. Measure of Tuberculosis/HIV Integration The evaluation team present charts that overemphasize the underperformance of Nairobi Province. The project team: states that in fact both provinces reach high level performance (see chart below) , with Coast definitely having the highest level of the indicator. 0 0.2 0.4 0.6 0.8 1 2011 2012 2013 2012 2013 Coast Nairobi % of HIV+ screened for TB in HIV setting % of HIV+ screened for TB in HIV setting who started treatment 16 Page 36 Figure 5. “ Figure 5 illustrates slight overall improvement in the projects contribution toward the percentage of women attending at least four ANC visits, but with a “downward trend in the first quarter of this year for both Nairobi and Coast” Figure 5. Percentage of Pregnant Women Who Receive at Least Four ANC Visits, APHIAplus Contribution The evaluation team looks at the share that women, who attend four antenatal care visits that are reported in the the M&E system of the project supported facilities, represent out of the total number of reported 4th antenatal care visits of the whole province (project +non project). The project team states that this is a relatively insignificant indicator when used in isolation to document outcomes observed in the project area. This indicator is sensitive to changing contexts. Indeed, the observed 2013 Q1 drop of the indicator, pointed out by the evaluation team to show declining performance, does not say anything about the project performance. The drop in this evaluation team indicator is due to a phenomenon that combines the aftermath of a nurses strike in Q4 2012 and elections in Q1 2013 and Rapid Results Initiative in 2012 which affect patterns of survice delivery and services use that such indicator is too sensitive to (see also our comment for figure 1). The full information and interpretation of this dip was available and made available to the evaluation team. The project team states that data validation and correct interpretation are neglected areas of this evaluation exercise. . 17 Page 36. Figure 6 . “ Figure 6 depicts a spike in year 2 and a dip in the start of year 3 for the percentage of under-1-year-olds fully immunized in sites supported by APHIAplus out of the total number of under-1-year-olds fully immunized in Nairobi and Coast counties. Figure 6. Under 1 Fully Immunized, Percentage of APHIAplus Contribution The evaluation team wants to demonstrate the relative contribution of the sites supported by the project to the total volume of children < 1 year fully immunized. The project team states that a coverage indicator would better demonstrate the eventual difference between sites supported or not supported by the project. In addition, the data are completely wrong and do not correspond in any way to the reported and validated indicator data of the project. Year 1 reported and validated data are 179,426 children and Year 2 data are 264,161 children. Both figures are substantially higher than data in the graph. The evaluation team had access to the datasets of the project. Page 37. Conclusion Par 2: “ The limited data presented by the project and significant changes in indicators from year to year (hindering comparison over project years) limit conclusions the team can draw about potential contribution to Result 3 overall. As well, direct attribution of APHIAplus inputs to outcomes or results is difficult, given other actors working with similar mandates in the same facilities. “ The evaluation team has created a few indicators in order to complement the indicator set that the project uses and has identified patterns which can be systematically be interpreted in a negative manner, i.e. reduction, smallness, fluctuation. The project team disagrees fully with how the evaluation team explores the answer to the evaluation question: to what extent was the project effective in contributing to the achievement of results 3 and 4. Project management has translated its vision, perception and understanding of what will measure the achievement of the results through the monitoring and evaluation framework and through the performance indicators which are approved by USAID every single year. From a performance management viewpoint establishing whether set targets were achieved is the first essential exercise and interpreting or commenting on those trends is the immediate next step. The 18 evaluation has not made an attempt to do this in depth and has presented an non validated summary metric in which the project team does not recognize itself. The project team has calculated those metrics based on the validated project data and states that the answer to question one is very positive. The evaluation team does not answer the questions why certain outcomes were not achieved. The answers could have come from the consulted sources of information ( see evaluation methodology). Here we suggest a few that the team should have found very easily: the project team struggled with establishing some sound technical capabilities as a technical assistance partner; the project team understands with hindsight that it was often ambitious and not tactical, i.e. too many simultaneous smart objectives; there was often slowness in launching and successfully starting up critical interventions; often eyes were not set on the targets and time between ending a process, looking at data and steering the intervention in the right direction was not properly paid attention too, etc. It is only after such critical analysis that one would expect the evaluation team to explore the data and the information from different angles. For this, the evaluation team should have used the right data and maybe have worked in collaboration with the project team instead of doing such analysis after the field work: the project team observes a great ignorance and negligence of context and the evaluation team contributes to the phenomenon that it decries in APHIAplus N-C indicator framework, i.e. continuously changing indicators. The project team has done plenty of such little data explorations throughout the project period and identified intriguing issues which were fed straight into programming or internal evaluation: sustainability questions, gender based patterns and fluctuations in use of care, reporting trends, decisions based on wrong data, effects of events outside the project sphere on project performance data; effect of demand creation at community level on service uptake, ability of the community strategy to create changes at scale etc. Many of these issues are found in reports or could have been identified by the evaluation team if it would have held a few discussions around data. Finally, we confirm that the hard evidence for the multiple questions that arise as soon as targeted performance is reached is relatively scant. And this is clearly a very important point to be considered in large projects like APHIAplus: monitoring should be prioritized from the very beginning and the resources should be firewalled so that no compromise should be made with regard to data, documentation and knowledge generation. 19 Question 2: What strategies employed within the work plan worked and what did not work for successful implementation and achievement of key outcomes in results 3 and 4? Why? Page 15 Staff trainings, Par 3. “That said, the evaluation team observed some gaps between project work plan targets and achievements in RH/FP and maternal, neonatal, and child health (MNCH) services. “ The evaluation team identifies correctly the deviation from planned training/orientation targets but does not accurately represent the clarifications that were given. The project team states that all clinical staff trainings that follow established MOH Training Curriculums are excluded from APHIAplus authorized activities and are under the planning and implementation responsibility of the national mechanism FUNZO. Annual work planning of year one and two anticipated intense FUNZO activities. In reality, effective FUNZO implementation started much later. All training subjects mentioned by the evaluation team belong to that category. For priority needs, APHIAplus was allowed to organize short training courses ( max 3 days), but this could not be used as a mechanism to shortcut FUNZO training initiatives. It is correct that skills related to constant improvement of HIV/TB programs (prevention, care and treatment) were prioritized in response to the performance pressure. Page 15. Technical support and facilitative supervision, Par 1. “Documentation from APHIAplus on mentorship activities was limited to data on the number of mentoring visits per Sub County and did not provide detail on topic or skill areas. While more detailed data were described during the NARESA interview, they were not provided for review. “ The evaluation team qualifies the available documentation of the mentorship activities as being limited to the number of visits. The project team: states that this is inaccurate for two reasons: (1) the quarterly reports, made available to the evaluation team prior to the field visits, contain synthesis information of the topics and skill areas. In addition, (2) facility assessment reports (close to 100) are available and on file while monthly mentorship visits by the more than 100 mentors are documented and stored electronically. During the evaluation, a protocol was established for the follow-up and documentation of information requests. No such request for follow up on mentorship reports is registered. 20 Page 17. Conclusion, Par.1 “.. There is anecdotal evidence that the project is making a positive impact on improved patient management, service delivery, and data quality and data use at the facility level where it is taking place. “ The evaluation team describes the evidence it was confronted with as anecdotal evidence and in response to the project teams comment on the use of this qualifier of the evidence observed, puts the following definition in a footnote: “Anecdotal evidence is a social science term that refers to perceptual data based on a few individual observations”. The project team disagrees 100 % and states that the intervention areas such as service delivery, patient management and data quality and use that it evaluated are not in the domain of social sciences but in the domain of clinical and patient care where the evidence is the individual patient record, the register, the aggregated report, the supervisory report or the mentorship report. These documents are available in each facility, management unit or project archive and were accessible to the evaluation team. Page 17. Conclusion continued Par 1. While this direct support may have been necessary to meet project targets, it should be acknowledged that it may not truly build systems-level capacity in a sustainable way. This is an early conclusion, however, as it remains to be seen whether these are merely initial efforts that may evolve over time The evaluation team: states that the direct support was maybe necessary in order to meet project targets and points to negative effect on capacity building. The team repeats that point in the summary of question 2 under foundation package. The team does not elaborate on this in the summary recommendations. The project team states that the evaluation team has identified a question that is of extreme importance for the APHIAplus model (the concept of USAID and the implementation strategy chosen by the project). By pointing it out in an intermediate level conclusion and omitting it in the summary conclusions, and more importantly in the lessons learnt, the evaluation team fails to tackle fundamental evaluation questions. The project team states that these are fundamental “strategy” questions and that their exploration should help understand the role of government, USAID and Implementing Partners in shaping program strategies in such a sophisticated technical assistance model as APHIAplus. Page 17. Facility Renovations. Conclusions : APHIAplus has provided support in facility renovations, which anecdotally has improved the ability of staff to provide services in accordance with health service guidelines and has led to increased facility use in some areas. The evaluation team has acknowledged and integrated the different facts-based comments of the project team on the Facility Renovations section of the draft of the report and qualifies the evidence as anecdotal. The project team states that APHIAplus N-C was right in negotiating with USAID to allocate resources for at least a few (> 50) limited renovations in order to address service delivery capacity improvement comprehensively because in most of these places it became a critical variable in 21 improving service use. The sites visited confirm these positive trends and provide weight to routinely reported examples of such improvements. The chain of evidence in the sites visited are hard facts. Page 18. Provision of supplies and Equipment. Conclusions “ The project delivered critical supplies that enabled delivery of key services in project sites (with notable gaps in MNCH commodities).” The evaluation team describes notable gaps in MNCH commodities. The project team states that the complete list of supplies, commodities and equipment was provided to the evaluation team and that according to evaluation methodology description, the field visits should provide context to the data available at the project level. The table in annex E downplays the evidence of the support given at project level. The evaluation team does not state if the gaps are compared with project listed commodities or if the gaps are due to the fact that the needs are larger than what the project can provide. For the project team, the observed gaps document the huge needs of the facilities. Page 17. The lab network. Conclusions “ … the courier system was found to be an improvement that decreased test turnaround time (and hence anecdotally increased service/ testing uptake), but is not sustainable without outside funding”. The evaluation team qualifies evidence of the impact of the lab network on service uptake and testing uptake as anecdotal. The project team rejects this incorrect analysis. It does not rely on or takes into account the monthly reports that are tracked in registers accessible to the evaluation team and documented in quarterly reports accessible to the evaluation team, and that allow to demonstrate the critical role of this lab networking strategy to ensure that facilities that have no labs (most of them) can effectively participate in care and treatment and in prevention of mother to child transmission and in particular in the scale up of pediatric care. To the project team these are facts in support of effectiveness of the lab network as alternative solution to lab services when facilities effectively have no labs. Page 18 Integration of Key Services. Conclusions “poorer results in others, such as the unidirectional integration of PMTCT” The evaluation team qualifies as “poor results” the observed service organization that is in line with guidelines prevailing in year 1 and 2 but that is an area to be worked on afresh given the recently disseminated eMTCT guidelines. The project team considers that the description by the evaluation team of the successful outcomes of efforts towards greater integration of services in observed facilities are indication that the integration strategy described in the work plan and reported on in quarterly reports worked. The fact that the evaluation team recognizes that achieving EMTCT outcomes as per GOK guidelines will take some time, supports our statement. 22 Page 20. Activation of the Community Strategy, Par. 2 “ Table 2 shows that the Project did not meet targets for numbers of trained CHWs in Nairobi or Coast for either project year. Table 2. Number of Community Health Workers Trained NAIROBI COAST 2011 actual (target) 2012 actual (target) 2011 actual (target) 2012 actual (target) 8,297 (3,650) 4,013 (7,195) 15 (20) 7,593 (4,000) The evaluation team observes that transparently produced CHW training data (see quarterly reports and comments) have a peculiar pattern and qualifies this as the project not meeting targets in either project year. The sentence suggests that these non-reached targets are the ones that “changed considerably from year one to year 2” The project team 100 % disagrees with this sentence. The evaluation team corrected its original draft statement based on the comments given by the project, but did only cosmetic work while leaving the grammar untouched so that the main idea conveyed is that the project did not reach targets.. The project team has provided a written correction in the consolidated comments on draft of final performance evaluation (p 12). The project team restates the same factual interpretation: the figures are correct and reflect program priorities and not program underperformance. In 2011, Nairobi builds on achievements of Pathfinder led APHIA II Nairobi and accelerates Community Strategy expansion beyond set targets, while Coast has priorities set on overall start-up of project in Coast. In 2012, Nairobi, based on DHMT priorities, slows down the planned quarterly expansion rate of the CS in favor of strengthening existing CUs, while Coast decides to expand at a set rate of one CU per district per quarter, before being forced to slow down given the now imposed cost attached to being “functional CU”. It should be noted that this PMP indicator in table 2 refers to all community workers trained (CHWs, peer educators, youth leaders, etc.). Page 20. Activation of the community Strategy. Par 1. “ Documentation was not provided on overall training breakdown (including nature and duration of training activities) to CHWs. No information was available regarding CHW performance and contribution to key results over time (e.g., trends in deliveries in facilities)”. The evaluation team attributes to inexistence of data, its inability to describe the effect that the support to the Community Strategy has. The project team has given data on the impact of community strategy, on the effect of defaulter tracing with CHWs, on care of OVC, on assisted deliveries on immunization outcomes and on family planning. In addition, payment of allowances to CHWs payments to CHW’s are based on performance and performance has to be documented before payments are made. With regard to 23 breakdown of training information, information is provided in Quarterly Reports and was provided separately to the evaluation team. Page 21 Community Strategy Conclusion: “ But there is little evidence at this time (soon after CU activation) showing outcome-level results, such as increased service demand, linking of clients to services, or any outcomes of IGA/VSLA or CBO support efforts. Beyond the IGA/VSLA efforts for self-financing, the activation of the community strategy is dependent on APHIAplus financial and technical support, raising questions for long-term sustainability. ” The evaluation team fails to find evidence of effects of the community strategy at outcome level and adds a parenthesis that suggests that it would not expect to find such outcomes within a 2.5 year interval. For the project team low level (CUs or facilities) and higher level (district or province) DHIS data demonstrate increased coverage for key health interventions and validate the choice of the project to strongly support to the Community Strategy . Increased demand for services is for instance already documented in January–March 2012 Quarterly Report. The evaluation team points to improved outcomes elsewhere in the report. The conclusion statement also ignores discussions which were held with the evaluation team about sustainability issues surrounding the Community Strategy and which pointed to the need to apply a health system perspective that involves at least four of its building block, i.e. Service Delivery, Human resources for Health, Financing, and Governance. Page. 22 Most Vulnerable Groups Package / Community level, Par 2. “The two Mentor Mother programs interviewed reported that staff received monthly allowances of up to Kshs 9,000 The evaluation team describes the mentor mother at the same level as CHWs. The project team comments that unlike CHWs, mentor mothers (see PMM Program) operate at facility level (for reasons of confidentiality). Their allowance is specified in the national PMMP guideline. Page 22. Facility Level, Par 1. “..Staff in the clinics uptake in HIV–related services resulting from CHW messaging and outreach efforts in the community, but the small sample size and short APHIAplus timeframe limits this finding to anecdotal evidence.” The evaluation team qualifies their findings as anecdotal evidence The project team refers to the evaluation methodology and the role of the field visits. Findings of the field visit should help contextualize the information found in other data sources. Quarterly reports, Semi Annual Progress Reports and Annual Progress Reports, rigorously verified data sources, document improved service uptake at large scales. The findings by the evaluation team are positive pointers to the validity of these already data quality tested reports. This does not fit in the definition of anecdotal. 24 Page 22 Facility Level, Par 3. “ . Health worker training efforts for providing MARPs–friendly services were very inconsistent “ The evaluation teams points to inconsistent training packages The project team has provided the documentation on the provider training, and made the manuals available. Trainings for MARPs are tailored to MARPs subtype, using NASCOP approved manuals, guidelines and trainers, Page 22 Facility level , Par 4 “ All surveyed FSW and MSM and two out of three PWUD organizations cited the vulnerability of children and youth and lack of supportive programming for reduction of violence and police harassment as major challenges not addressed by APHIAplus–supported programming (HIV prevention objectives). The evaluation team provides the viewpoint of the interviewed organizations, without adding the context that was provided. The project team provides the information that this programming is effectively done and that one needs to look at depth and intensity of the intervention, rather than at the issue of interventions being present or not present. SW and MSM peer educators were trained as paralegals in 2012. They refer peers for SGBV incidences to Gender Based Recovery Centre at CPGH. FSW peer educators participated in 16 days of activism against GBV, an activity that is documented in project and partner quarterly reports. Page 25 Most Vulnerable Groups Package / Support to Orphans and Vulnerable Children, Par 1. . “The evaluation team did not feel that this documentation was appropriately detailed for OVC activities under USAID PEPFAR funding” The evaluation team refers to the validation exercise as if it was intended to be a needs assessment. The project team states that the purpose of the validation survey was not to serve as a needs assessment to identify OVC activities under PEPFAR funding. The US Guidance for OVC programming describes those mandatory activities already in detail. The objective of the validation survey included: · To validate the existence of the OVC handed over to APHIAplus Nairobi-Coast · To establish the households participating in the GOK supported Cash Transfer · To establish household income and the resources available at household level. The validation exercise fully addressed these objectives, and this is the report that was given to the evaluators. It should be noted that there was no standard format of reporting that the project was supposed to follow. The project produced a report that was sufficient in addressing the objectives of the validation process. Page 27 Most Vulnerable Groups Package / Support to OVC, Conclusion “ In addition, there is a potential conflict of interest caused by CHW involvement in the selection and validation of OVC as potential recipients of OVC household support as caregivers or employees of CBOs working with OVC” 25 The evaluation team concludes from information gathered that there is a possible conflict of interest The project team states that this conclusion is informed by inaccuracies. CHWs are volunteers within the CBOs and are not employed. They do not determine nor distribute services to OVC. The few who are caregivers are themselves part of vulnerable households or have taken in orphaned children to live in their HHs. Working with CHWs is part of the project strategy to ensure sustainable community structures at the grassroots where OVC are found. Initial OVC enrolled in the APHIAplus Nairobi project were transitioned from APHIA II, Track 1s and NPI projects. The validation exercise ascertained the existence, the number, location and economic status of these OVC households. No OVC were selected. Over the project period, additional enrollment has been as a result of enrolment of OVC in already existing OVC households and exits that are triggered by relocation. The OVC who replace those who have exited are identified by the LIPs as per enrolment criteria, and verified by the project then enrolled for support. District children Officers also refer vulnerable households to the project, upon which the OVC in those households get enrolled. Page 27, Most Vulnerable Groups Package / Support to OVC, Par 2 “ .. “A conveniently selected sample of 30 files shows inconsistencies in documentation and unclear reporting of support, such as incomplete documentation of support provided, multiple children in one file, and lack of monthly reporting forms recently instituted.” The evaluation team analyzes different aspects of documentation of service delivery and interprets tools incorrectly. The project team restates the comments already provided to the evaluation team: in year 1 and 2, reporting of OVC services was consolidated in distribution lists, filed under the 6 services areas. Records were kept in service files. Another file contained the master list of children enrolled per CBO. These files are available in the various IPs. In addition, each OVC has a file which contains their details. CHWs collate services that OVC have received in the month in OVC 1 form. All the OVC 1 forms are then collated further to show the number by gender and the various services OVC received per IP/CBO. These reports are what the project consolidates to enter into the KePMS. After the initiation of OLMIS in Sept 2012, the monthly assessment form and the services tracking form were rolled out. These forms are expected to replace the distribution lists. They are filled monthly and quarterly in the individual OVC files. But because OLMIS is yet to be fully operational, the project is running both processes (using distribution lists as well as using monthly assessment and service tracking forms)”. The inconsistencies perceived in reporting may have been due to lack of understanding on the processes by the evaluators. The observation of “multiple children in one file” is clearly confusion. The evaluation team confound service tracking forms (which have many children, per service area) with OVC files (which contain details per individual OVC). Page 28 “APHIAplus support to economic strengthening initiatives (IGAs and VSLAs) has been well received, but after only two years of project activity it is difficult to quantify any changes or link evaluation evidence to sustained outcomes.” 26 The Project team states that this conclusion fails to appreciate the link between household economic strengthening and the other six services provided directly to OVC. HES is key to sustainability of the six services because it empowers the OVC caregivers to respond to OVC needs. Quarterly reports describe the extent the extent to which OVC caregivers empowered through HES are responding to various OVC needs without necessarily depending on APHIAplus support and the evaluation team should have been able to establish this evidence qualitatively. Page 26 Prevention of GBV and Gender Mainstreaming , Par 1 “ The third-year project work plan identified several gender mainstreaming and other gender (GBV) activities but does not link these activities to clear strategies or outcomes.” The project team states that it has a two tier gender strategy, referred to in the above sentence. Gender is a cross-cutting issue and outcomes are mediated through different indicators and cannot be traced in isolation of the respective service delivery dimension indicators. Gender mainstreaming is evident through indicators reported at different service delivery points (PMTCT; FP, MNCH; social determinants; BCC etc.) which are all gender disaggregated. Data for gender indicators in the different service delivery areas were availed to evaluators and none of these quantitative data has been analyzed. Page 29 Prevention of GBV and Gender Mainstreaming. Par 2 “Three surveyed CBOs that support OVCs and three PLHIV support groups did not demonstrate a comprehensive understanding of gender or gender mainstreaming in programming” The evaluation team questions programming skills of CBO’s. The project team states that the structure of the evaluation process does not render itself to capturing demonstration rather than information. Groups implied here are village groups, hence the understanding of concepts and practices like gender mainstreaming can hardly be demonstrated in one-hour interviews. They are observable through group membership, beneficiary profiles, type of benefits and services provided and data disaggregation during reporting as availed to the evaluation team. Page 26 Prevention of GBV and Gender Mainstreaming, Par 2. “..APHIAplus gender staff interviewed stated that they did not feel that APHIAplus programming staff had a solid understanding of gender mainstreaming and thus could not implement activities appropriately.” The project team states that in-house training on gender mainstreaming and gender programming took place and targeted all relevant staff. The statement was later contextualized by gender staff and this was conveyed to the evaluation team. Page 26 Conclusion “..The APHIAplus approach to gender lacks a well￾articulated overall strategy with clearly defined interventions that are aligned to internationals guidance” The project team states that implied ambiguity of gender strategy in the project is unfounded. As a cross-cutting issue (RFA, Proposal, Work plans), gender outcomes cannot be traced in isolation of 27 the respective service delivery dimension indicators. Data for gender indicators in the different service delivery areas is was availed to evaluators. Gender mainstreaming is a structural component of programming; involves capacity-building and change in practices. The gender strategy employs targeted capacity-building, public advocacy and mobilizations and cross-sectional institutional networking towards creation and establishment of gender safe spaces. The ultimate aim of this strategy is to make the project intervention area a gender safe space in which mainstreaming is an integral and non-negotiable aspect of programming, monitoring and reporting activities. Capacity improvement interventions targeted staff and implementation partners: trainings, CMEs and workshops (see various reports and data). GBV trainings were conducted for DHMTs and covered priority topics: e.g. trauma management, post-rape HIV prevention and management, etc. Interventions adhere to national and international standards and have been dynamic, making the strategy to be rebranded gender and rights to accommodate the paradigm shift on gender and health issues beyond GBV or the male-female conception of gender. Page 26 Economic Strengthening Activities, “ documentation on group composition was not provided by the project.” The project team states that the detailed lists of VS&L were provided as documentations to the evaluation team, including the VS&LA training manual. Question 3: To what extent were the project’s annual work plans and strategies therein informed by new programmatic evidence in the health sector? Page 38. Alignment to policies and Guidelines Par 1 “..While observations in 13 out of 17 MNCH clinics revealed that KEPH guideline documents were available, in 10 of these clinics, Integrated Management of Childhood Illnesses guidelines (an important component of MNCH) were outdated or absent” The evaluation team makes use of the feedback info that the project provided in the Consolidated Comments on the Draft of the Final Performance Evaluation, but does not really contextualize the issue properly. The project team states that IMCI guidelines are availed at national level and distributed by the project. Alternatively, they are photocopied and provided during IMCI trainings. But there has not been any IMCI training (mandate of FUNZO), and the project cannot cover all in OJT. However, various aspects of IMCI are covered in CMEs. In some facilities the guidelines are taken away by staff and not replaced. The project is not allowed to reprint guidelines. Page 39. Alignment to policies and guidelines –conclusion; “ However, with noted gaps in MNCH—IMCI particularly—guideline availability is still a concern and program challenge 28 The project team corrects the impression coming from the report and states that the observed weakness in program area MNCH is not a problem of alignment with policies or work plans but an issue of gaps in the allocation of dedicated program resources to particular program areas. Page 40 At the community level. Bullet 4 “ Although session themes might be comprised of 3–11 sessions, the project measured participation in 2 or 3 sessions as the minimum standard. “ The project team does not agree with the statement on 2 or 3 sessions as minimum standard. Individual and small-group level prevention interventions with multiple contacts are effective in reducing HIV transmission risk behavior and need to be delivered with fidelity to the appropriate populations – includes compliance with intended number of sessions - is an important component of comprehensive HIV prevention strategies. Number of sessions varies depending on the program. E.g. HCII: 4 sessions; Shuga : 1-3 sessions; FMP : 5 sessions; ETL : 2-3 sessions. Question 4: To what extent was the APHIAplus model appropriate for achieving key outcomes under results 3 and 4 of the implementation framework. Page 46 Building on USAID investments cont. Par 1. While some promising practices from APHIA II Coast, as identified by Family Health International, were not continued (e.g., workplace programs, the CVD and HIV program), the project did build on clinical services introduced or strengthened under APHIA II. The project team states that CVD and HIV was not in APHIA II’s portfolio, but was a separate project, integrated in APHIA II Coast. Question 5: What sustainable activities in service delivery and healthcare systems were established and/or strengthened at the district health facility and/or community level? Page 48 Conclusion . It is unclear how the government will sustain these activities without donor support, but (anecdotal evidence suggests) ownership has increased. The project team states that to sum up the evidence gathered by desk review, data exploration, key informant interviews, semi-structured group interviews and structured observations/checklists as ‘anecdotal’ suggests that from data collection to data analysis and synthesis of findings very little weight is given to qualitative data. APHIAplus has analyzed quantitative data to answer such questions, and demonstrated that dependency, in particular financial dependency, is what characterizes the fragility of the achievements in the APHIA II and APHIAplus programs. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523