REPORT APHIAPLUS WESTERN KENYA MID-TERM REVIEW 5 SEPTEMBER 2013 REPORT Technical report - APHIAplus Western Mid-term Review report 2 Contents Acronyms List 3 Executive Summary 5 1.0 Introduction 6 2.0 Methods 6 3.0 Findings 9 4.0 Conclusions and Recommendations 29 5.0 Annexes 35 Annex 1. Target Outcomes from Cooperative Agreement 35 Annex 2. Terms of Reference (ToR) for the Mid-term Review 37 Annex 3. Field Syntheses Notes 39 Annex 4. Top Level Findings Summary Sheet 40 Annex 5. Persons Interviewed 51 Annex 6. Evaluation Tools 53 Annex 7. Team Members 74 Annex 8. Thematic Framework 77 Annex 9. Documents Consulted 78 REPORT Technical report - APHIAplus Western Mid-term Review report 3 Acronyms List AIDS Acquired Immunodeficiency Syndrome ANC Antenatal Care APHIA AIDS, Population and Health Integrated Assistance ART Antiretroviral Therapy ARVs Antiretroviral Drugs ASOs AIDS Service Organizations BCC Behaviour Change Communication CBOs Community-Based Organizations CPI Client–Provider Interaction CQI Continuous Quality Improvement CWC Child Welfare Clinic DG Demand Generation DHMT District Health Management Team EC Emergency Contraception FBO Faith-Based Organisation FGDs Focus Group Discussions FGM Female Genital Mutilation FP Family Planning GPS Global Positioning Systems HF Health Facility HIV Human Immunodeficiency Virus HCT HIV counselling and testing HCW Health Care Worker IEC Information, Education & Communication IUD Intrauterine Device KAP Knowledge Attitude and Practices KDHS Kenya Demographic and Health Survey KII Key Informant Interview LAM Lactation Amenorrhea Method M&E Monitoring and Evaluation MNCH Maternal, Newborn, and Child Health MIS Management Information Systems MOH Ministry of Health MOM Ministry of Medical Services MOPHS Ministry of Public Health & Sanitation MORPS Most at Risk Populations NGO Nongovernmental Organisation PAC Postabortion Care PHMT Provincial Health Management Team REPORT Technical report - APHIAplus Western Mid-term Review report 4 PLWHA Persons Living with HIV and AIDS PNC Postnatal Care PNF Postnatal Family Planning PMTCT Prevention of Mother-to-Child Transmission QC Quality of Care QTA Quality Technical Audit RDC Research Determination Committee REC Research Ethics Committee RH Reproductive Health SDH Social Determinants of Health SDP Service Delivery Point SRH Sexual and Reproductive Health STI Sexually Transmitted Infections TB Tuberculosis TBA Traditional Birth Attendant VCT Voluntary Counselling and Testing VMMC Voluntary Male Medical Circumcision WASH Water, Sanitation and Hygiene YFS Youth-Friendly Service REPORT Technical report - APHIAplus Western Mid-term Review report 5 Executive Summary This report documents a mid-term review of the APHIAplus project, a five-year effort funded by the US Agency for International Development (USAID) to improve heath in two provinces in western Kenya. The review assessed progress against goals of the Government of Kenya and USAID/Kenya and evaluated the project’s contributions towards increasing access to services, improving service delivery, enhancing data utilisation, and achieving other desirable outcomes. Evaluation methods included trend analysis of performance data and key informant interviews and focus group discussions with stakeholders, implementers, and project staff. Findings suggest that the project is performing well in some output areas and enjoys positive perceptions of its basic support inputs. At the same time, however, the project is not meeting some performance targets. Additionally, perceptions of how the project operates, responds to technical challenges, and supports and collaborates with stakeholders indicate that a county-specific approach is needed to achieve higher levels of performance in counties performing adequately and to reverse negative trends in counties experiencing a drop in performance. Recommendations to improve performance include:  Strengthen county-driven strategies with targeted action plans to increase supply of and demand for high-quality services.  Ensure consistency in measurement of indicators at the county, district, and community level; upgrade current information system management practices to make data available more quickly for decision-making; and confirm that all project outcomes are being tracked and reported on annually.  Enhance systems, processes, and capacity to improve coordination and communication related to Mpesa payments, vendor work orders, and other important procurement functions.  Strengthen emphasis on regular engagement with key stakeholders to ensure they are part of planning and long-term solution building.  Hone the capacity-building approach to ensure adequate investment in key partners and stakeholders and to ensure the project increases the number of skilled technical experts, managers, and leaders in Kenya.  Work towards sustainability by joining with partners at the community and facility levels to develop continuous improvement activities and exit strategies. Although this mid-term review has a number of inherent limitations, it provides a reliable overall assessment of project performance and stakeholder perceptions of performance. Recommendations based on this review warrant further discussion and may help to guide modifications in strategy, planning, and implementation to improve outcomes in the second half of the project. REPORT Technical report - APHIAplus Western Mid-term Review report 6 1.0 Introduction The AIDS, Population, and Health Integrated Assistance Zone 1 Project—commonly known as APHIAplus Western Kenya—is a five-year effort (2011 through 2015) to improve health in Kenya’s Nyanza and Western provinces. It is funded by the US Agency for International Development (USAID). PATH leads a consortium—including the Elizabeth Glaser Pediatric AIDS Foundation, Jhpiego, and World Vision—that is implementing the project. The APHIAplus implementation framework has two main result areas that contribute to USAID/Kenya’s strategic objective to “improve health outcomes and impacts through sustainable country-led programs and partnerships.” Result 3 focuses on improving the availability of high-impact, high-quality health services at the facility and community levels, increasing demand for those services, and improving health-seeking behaviours. Result 4 addresses social determinants of health with the objective of improving the health and well-being of marginalized, poor, and underserved households, families, and children. It focuses on economic security, food and nutrition security, education, life skills and literacy, safe water, sanitation and hygiene, and social mobilization for health. PATH staff led an APHIAplus mid-term review (MTR) to help strengthen project strategies and implementation and accelerate progress towards expected outcomes and deliverables. The MTR was especially intended to inform planning for project years 4 and 5. Key MTR objectives were to evaluate: 1. APHIAplus achievements against Government of Kenya (GoK) and USAID/Kenya objectives and mandates. 2. Project contributions to strengthening access to, utilisation of, and demand for high-quality services. 3. Contributions to cross-cutting service delivery components and strategies. 4. Contributions to improving data quality and data utilisation. 5. Relationships and support related to coordination with the Ministry of Health (MOH) and other stakeholders. 2.0 Methods The MTR employed a mixed-method approach that was conducted in three phases. Phase 1 was a desk review of trends in 19 high-priority indicators (see Table 1) selected by the project’s senior management team, as outlined in the project Performance Monitoring Plan. The review examined these trends in key project service areas from project inception (January 2011) through March 2013. This phase culminated in a Kisumu-based workshop (June 27, 2013) with key management, technical, and county leadership to help interpret the performance data. REPORT Technical report - APHIAplus Western Mid-term Review report 7 Table 1. Key indicators used to assess core service delivery areas. Antiretroviral therapy No. of individuals newly initiating antiretroviral therapy during the reporting period No. of individuals receiving antiretroviral therapy at the end of the reporting period HIV testing and counseling services No. of individuals who received counseling and testing services Prevention of mother-to-child transmission (PMTCT) No. of pregnant women who were tested and knew their test results No. of women provided with antiretroviral (ARV) prophylaxis for PMTCT No. of HIV-exposed infants provided with ARV prophylaxis No. of infants tested for HIV at 6 weeks Care No. of individual patients newly enrolled into HIV care within reporting period No. of positive persons currently on cotrimoxazole Antenatal care (ANC) No. of women attending at least 1 ANC visit No. of women attending 4 ANC visits Child survival No. of infants fully immunized No. of children 12 to 23 months of age who received measles vaccine by 12 months of age Community strategy Percentage of Community Health Units that report complete and accurate data for all program areas monthly Community prevention No. of target audience reached by interventions focused on abstinence No. of individuals reached by interventions beyond abstinence/faithfulness No. of most-at-risk population members reached by interventions that meet minimum standards Orphans and vulnerable children (OVC) No. of OVC supported with 3 or more benefits Voluntary medical male circumcision No. of males circumcised as part of minimum package for HIV prevention services In phase 2, project leadership and the review team identified critical service delivery areas and technical themes for further exploration during semi-structured interviews, group discussions, and site visits with key stakeholders and implementers in two purposively selected, representative counties in each province. Selection criteria emphasized county cohesion and project stability (i.e., a consistent REPORT Technical report - APHIAplus Western Mid-term Review report 8 history of project presence since inception, overlap in project-supported service delivery areas, counties with limited site shuffling). Stakeholder participants included officials from the Ministry of Health and Ministry of Education, implementing partners, service providers, and field agents. The assessment team also visited select clinical facilities to observe service quality, access, utilization, and data accuracy and completeness. The purpose of the interviews and group discussions with partners and stakeholders was to better understand how APHIAplus was performing in specific areas and to identify areas where the project could improve the support it provides. The focus was on technical assistance to achieve the following objectives:  Expand service provision.  Increase demand for services.  Integrate provision of services at both community and facility levels.  Improve data quality and data utilisation.  Reduce gender-related barriers among vulnerable groups to increase access to services.  Unify youth-focused service provision across service delivery areas. Phase 3 was an internal management review to determine whether perceptions of project staff were consistent with perceptions of key stakeholders. Data were collected through ten one-on-one interviews with project managers and technical leaders and one focus group discussion. The analysis of data from phases 2 and 3 was guided by a health systems strengthening framework based on specific themes: (1) planning; (2) financing; (3) training/mentoring; (4) supplies/commodities/equipment; (5) support supervision; (6) information/data; and (7) working relationships. The analysis identified emerging strengths and challenges, and comparisons were made across counties, government administrative units, and organizational levels. Findings are presented by service delivery areas covered in the assessment. The MTR gathered both quantitative and qualitative information. Although the review was not intended to be rigorous research that might be generalizable across every county, facility, or community that APHIAplus serves, it was designed to provide strong evidence on project performance as well as on key drivers behind the performance. REPORT Technical report - APHIAplus Western Mid-term Review report 9 3.0 Findings Objective 1. APHIAplus achievements against GoK and USAID/Kenya objectives and mandates Performance against overall project targets Performance varied across key indicators. In a few cases, performance exceeded annual overall targets. One example was performance related to the number of individuals who received HIV testing and counselling services test results. Whereas the overall target through March 2013 was 1,290,124 individuals, the project actually reached 1,365,715 people, exceeding the overall target by about 75,000. Performance also exceeded targets for the number of people reached by interventions focused on abstinence and the number of individuals in most-at-risk populations (MARPS) reached by interventions meeting minimum standards. For some indicators, however, the project was not reaching targets. For example, it was not meeting performance targets for the number of pregnant women who were tested and given their test results. The number actually reached was 348,749, or about 10% below the target of 386,900. Likewise, the number of pregnant women with HIV who received prophylaxis for prevention of mother￾to-child transmission (PMTCT) was 16,291, or 32% below the target of 23,913. The project however had put in place strategies towards achievement of these targets. The project’s inability to achieve some mid-term performance targets was discussed at length during the June 2013 participatory review of the indicator data. Stakeholders noted some internal factors that hindered the project’s ability to meet targets, such as:  Inadequate use of data. Most technical personnel cited the need for more data dashboards to allow teams to drill down, investigate, and use the data.  Constraints in delivering core interventions due to payment and procurement delays because of budget constraints. Some team members felt that activities that could have increased demand for and utilization of services were delayed or not implemented because of funding or procurement delays. For example, staff felt trend dips in OVC benefits received could be attributed to delays in school fee processing or delayed procurement of items. Stakeholders also noted a number of external factors influencing the project’s ability to meet targets. Some examples include:  Targets may not reflect the epidemiological and demographic shifts at the province/county level.  In Nyanza province, because of the rationalization of districts and sites for US Government (USG) partners, APHIAplus took on lower-volume sites and gave away key higher-volume sites.  The 2012 nurses strike caused a drop in performance due to a low volume of service provision. REPORT Technical report - APHIAplus Western Mid-term Review report 10  Problems with procurement of HIV rapid test kits were beyond the control of the project and had a severe impact on service delivery. Different trends in Nyanza and Western in key performance indicators The two provinces demonstrated different trends in key performance indicators (See Annex 2). With a few exceptions, trends were generally better in Nyanza than in Western. Differences between the two provinces were especially striking for the monthly average number of individuals who received HIV testing and counselling services test results, a number that was generally rising in Nyanza and decreasing in Western (Figure 1). Nyanza was over-performing by about 125,000, whereas Western was under its target by more than 50,000. REPORT Technical report - APHIAplus Western Mid-term Review report 11 Similarly, performance indicators for utilization of antenatal care (ANC) (Figure 2) and PMTCT services (Figure 3) showed improvements in Nyanza but declines in Western. Some of the differences might be attributable to the loss of key high-volume sites in Bungoma and Busia counties in Western, and private facilities in Western may have inconsistently reported data. Figure 2. Number of women receiving at least one ANC visit, Nyanza (top) and Western (bottom) provinces. Figure 3. Number of women receiving PMTCT services, Nyanza (top) and Western (bottom) provinces. Figure 1. Number of people receiving HIV counselling and testing each month, Nyanza (top) and Western (bottom) provinces. REPORT Technical report - APHIAplus Western Mid-term Review report 12 For HIV counselling, testing, and treatment indicators, the trends are complex. Although counselling and testing for HIV appeared to be rising in Nyanza and decreasing in Western (see Figure 1), the number of newly enrolled individuals on antiretroviral drugs (ARVs) was decreasing over time in both provinces (Figure 4). Also, the number of individuals on ARVs was decreasing in Nyanza but steadily increasing in Western (Figure 5). Figure 4. Number of people newly initiated on ARVs, Nyanza (top) and Western (bottom) provinces. Figure 5. Number of people currently enrolled on ARVs, Nyanza (top) and Western (bottom) provinces. REPORT Technical report - APHIAplus Western Mid-term Review report 13 The documented changes in Nyanza are actually misleading at the provincial level. After accounting for a USAID mandate that the APHIAplus project take on new lower-volume facilities in September 2012 while cutting project support to existing higher-throughput facilities, Nyanza actually had an increasing trend over time for both the number of individuals currently on ARVs and individuals on cotrimoxazole. In Western province, it is unclear what accounts for the reduction in testing and newly enrolled individuals on ARVs and the increase in currently enrolled individuals and individuals on cotrimoxazole. Further inquiry is recommended. Indicators for the percentage of fully immunized children and children receiving a second dose of measles vaccine suggested an increasing trend in Nyanza. In Western, however, the trend was falling, with extreme peaks and valleys presumably matching vaccine availability at facilities. Monthly data fluctuations (sharp dips and peaks) were evident in nearly all indicator trend lines. While normal fluctuations are to be expected, other factors appeared to be at play. Missing or inconsistent monthly reporting was relatively common, especially in private health facilities supported by the project. Service delivery interruptions also caused dips in volume. For example, a nurses strike from December 2012 to February 2013 led to delays in vaccine delivery that affected monthly child immunization figures, and delays in the delivery of HIV test kits in Western province were observed to affect service provision during 2013. Nevertheless, the differing performance of Western vs. Nyanza is not easily explained. In nearly all cases, when unusually high or low data points were removed or adjusted, the deeper underlying trend remained about the same. The one exception is where APHIAplus was asked to change sites in Nyanza. This change resulted in a number of high-volume sites being removed from the trend line and correlates to a reduction in the performance in some indicators. REPORT Technical report - APHIAplus Western Mid-term Review report 14 Objective 2. APHIAplus contributions to strengthening access to, utilisation of, and demand for high-quality services During phases 2 and 3, The MTR team gathered information on stakeholder and implementer perceptions on what the project has contributed towards improving service delivery and meeting other project objectives. This section summarises findings from a number of service delivery areas: support towards the Community Strategy (CS); reproductive health (RH); HIV and AIDS; maternal, newborn, and child health (MNCH) and nutrition; malaria; tuberculosis (TB); and orphans and vulnerable children (OVC). Support for the Community Strategy All categories of respondents recognized that the project’s direct support for the community strategy (CS) was exemplary. Stakeholders acknowledged that APHIAplus has supported establishing or reviving Community Units (CUs), including providing direct support to community health extension workers (CHEWs) and community health workers (CHWs). APHIAplus also supported CUs to hold dialogue and action days and this too was acknowledged by CUs. APHIAplus was reported to have supported orientation and specialised training for CHWs and CHEWs and provided some CHWs with reporting tools, branded t-shirts, bags, tapes for measuring mid-upper arm circumference (MUAC), bicycles (which in some areas double as village ambulances), reference books and guidelines, and a monthly stipend of KES2,000 (US$231 ). “APHIAplus has helped in the formation and implementation of the community strategy….APHIAplus has provided 610 bicycles and also motorbikes and supported community dialogue meetings by providing lunches and transport. Overall, APHIAplus is leading in increasing sustainability at CUs.” – Kakamega CHMT member Several District Health Management Team (DHMT) members suggested additional APHIAplus support for CUs. CU members and health care workers recommended stipend support to Community Health Committee (CHC) members to whom CHW members report. It was reported that stipend support to CHWs had created dissent with CHC members who did not receive any even though the support provided was in line with MOH policy. “Some CUs are not supported by APHIAplus, and this is de-motivating the CHWS and CHCs in those CUs. APHIAplus should come in and support all uniformly.” – Bungoma DHMT member The most cited benefit of APHIAplus CS support was the recognition of CHWs as the health link at the community level and improved community health outcomes as a result of their intervention. 1 An exchange rate of US$1 = KES86.11, the rate as at 01-Sep-13, has been used throughout the report. REPORT Technical report - APHIAplus Western Mid-term Review report 15 “[There is] increased access to health information since the CHWs are community members who live among them. The CHWs have gained recognition as an authority in health-related matters unlike before the CU was established.” – Nyamira CU representative “Latrine and immunization coverage has greatly improved since the CU began operations.” – Bungoma CU staff member CHW support was also credited with increasing demand for RH and MNCH services. “Antenatal care has been enhanced with CHWs making sure every pregnant mother comes to the facility for four visits.... The CHWs… ensure no loss to follow-up and adherence. The CHWs are involved in service promotion and maintenance. ” – Kisumu health facility representative “In the last year no woman has given birth in the village. They all come to the facility.” – Kisumu health facility representative Because of APHIAplus-supported training, CHWs reported that they could identify community members suffering from diseases such as malaria and TB, following which they refer them to the nearest health facility. CHWs, however, suggested having additional kit items, such as pain killers and jigger medication, to manage basic ailments as well as drugs to manage uncomplicated malaria. They also suggested additional training in malaria case management, nutrition, and home-based care and recommended that CU training be tailored to the health afflictions affecting their area. CHEWs and CHWs reported that the airtime allowance and monthly stipend motivated them to perform their duties. C/DHMTs and health facility-based staff also acknowledged this financial support as essential to the basic functioning of the CU, noting that even though it did not represent the true value of the CHWs’ time, it went a long way in showing their contribution was appreciated. Every focus group and interview also revealed, however, that the CHWs’ stipend was almost always paid late, sometimes several months in arrears, sometimes not paid at all. The mode of payment is Mpesa, a popular mobile money transfer system. CHEWs and CHWs indicated that delayed payments made it impossible to associate stipends paid out to specific months of service. “Do away with Mpesa....[Use] cash for stipends.” – Nyamira DHMT member Stakeholders across the board suggested that the amount of the stipend be increased, at the very least doubled. There seemed to be misunderstanding about the purpose of the stipend which, in contrast to current MOH donor policy on community strategy support, respondents seemed to view as a ‘salary’ rather as than compensation for expenses incurred. REPORT Technical report - APHIAplus Western Mid-term Review report 16 CHWs did appreciate the bicycles for enhancing mobility. CHWs serving hilly or rocky terrain suggested provision of motorcycles, at least one per CU. Reproductive health Facilities reported that they had received consumables (iodine, cotton wool) and training for cervical cancer screening. Some CHWs reported being trained on family planning (FP) methods, and some indicated that they carried condoms for distribution. Health workers attributed a reduced incidence of teenage pregnancies to APHIAplus support. Health care workers also reported a need for additional training in FP, particularly insertion of implants. However, RH was not perceived as a strong area of APHIAplus support among MOH staff interviewed. Several project SMT members noted that USAID PEPFAR funds for RH activities are not significant. HIV/AIDS (HTC, PMTCT, and care and treatment) APHIAplus’ HIV/AIDS services support, especially for HIV testing and counselling (HTC), is an area in which all stakeholders were pleased and satisfied. It is also a service delivery package in which USAID PEPFAR funding is significant. Stakeholders reported many APHIAplus accomplishments, such as (1) established a laboratory network to support CD4, EID, and VL services; (2) assisted in the distribution of proficiency test samples for HTC; (3) provided CD4 specimen collection bottles and supported their transportation; (4) employed peer educators (PEs) and HTC counsellors who also support client loss and defaulter tracing; and (5) facilitated support groups for people living with HIV and AIDS (PLWHA). Stakeholders also reported that APHIAplus supports CCC training for health care workers on PMTCT and ART. Facility staff and DHMT members mentioned that APHIAplus provides furniture and other point-of-use equipment for HIV and AIDS services. “The support from APHIAplus is credited with everything that the facility is delivering today. All the services to the community are attributed to the training and supervisory support. The consistency of commodities supplied by APHIAplus ensures that service is uninterrupted. The lab support has enabled the rollout of a comprehensive HIV management at the facility.” – Kisumu health facility staff member The perceived benefits from APHIAplus’ HIV/AIDS support were numerous. It was apparent that demand for HIV and AIDS services, HTC, PMTCT, and care and treatment had increased, coupled with an improvement in the quality of services offered, including addressing the problem of stigma. The benefits of having a laboratory support network stood out amongst all interviewed, having contributed to enhanced access to laboratory services. Health care workers also indicated that they were motivated by the APHIAplus reward system, having been given trophies for marked improvements in service provision. REPORT Technical report - APHIAplus Western Mid-term Review report 17 “The lab support network has assisted in providing diagnostic support. People can get tested and also have their CD4 monitored without having to travel to Kisumu or the private facility at St. Monica.” – Kisumu health facility staff member Respondents were also asked to comment on four specific APHIAplus interventions related to HIV/AIDS services, as outlined below. The project regards these as ‘promising innovations’. Reminder system for HIV clients on antiretroviral treatment: This system typically involves peer educators keeping a record of patients’ appointments and referring through CHWs, and calling or sending texts to remind patients a few days prior. Even though all facilities had used the system, at the time of the MTR exercise, most were not actively using it. The primary challenge indicated was insufficient airtime to contact clients in view of the large number of clients registered with some health care workers, who reported needing to use personal airtime for calls. “The facility takes the clients’ telephone numbers and reminds them of their next clinic visit, though there are no facility phones and credit is not provided. The facility wishes APHIAplus would buy facility phones and provide credit to facilitate this service.” – Kisumu health facility worker At the same time, the Kisumu facility health worker quoted above also felt that the real driver of adherence is the strength of the CU-health facility linkage. The reminder system is just one more tool health that facilities are being encouraged to employ. TheraSim® model for coaching and mentoring: TheraSim® is a computer-based self-learning program to support health care worker knowledge and skills. The content is in modular form and is case-based, and it allows users to assess knowledge levels (by scoring) and provides a knowledge resource for continuous learning and reference. Although many respondents were not aware of what TheraSim® could do, some attested to classroom training for some facilities and noted that inter-facility mentorship had occurred. It is noteworthy is no one mentioned anything about the quality of learning and coaching/mentoring. Uliza! Clinician’s HIV Hotline for real-time clinical consultations: Uliza! is a telephone helpline to support health workers troubleshoot difficult clinical cases in real time. Users call in to a toll-free number that has HIV experts available to answer questions 24 hours a day. Started by FACES in 2008, it is now a collaborative project supported by three partners: FACES, AMREF, and APHIAplus. In one county, a few health care workers had either heard of Uliza! or had mentioned using it, but overall, it was not a widely known or used resource among most interviewed health care workers. Performance-based financing (PBF): PBF is a system initiated by APHIAplus that bases HIV and AIDS services financial support to outputs rather than inputs (i.e., the number of clients tested per HTC counsellor). Each counsellor has a target of up 15 clients per month. Because PBF was piloted in only REPORT Technical report - APHIAplus Western Mid-term Review report 18 two of the 44 districts that APHIAplus supports, and these districts were not represented in the MTR sampling area, most interviewed health care workers were not aware of the PBF arrangement. Tuberculosis Among health facility staff and DHMT members, APHIAplus was credited with increasing demand for TB services and the number of people being screened. Stakeholders identified defaulter tracing of TB clients as the primary achievement from APHIAplus support to TB programmes. APHIAplus was also credited with providing TB training for health care workers, especially on integration of HIV and TB services. APHIAplus also provided TB test kits and supported transportation of sputum specimens. Respondents also noted an important caveat. APHIAplus supports very few facilities that provide TB screening services. This is significant in that the APHIAplus catchment zone had 28% of the country’s total TB cases, with most coming from Nyanza province. DHMT representatives and health care workers recommended increasing the number of supported facilities. Maternal, newborn, and child health (MNCH) and nutrition Respondents indicated that MNCH services have greatly improved as a result of APHIAplus support. They said that MNCH services were well supported and sufficiently integrated. Also, health care workers had received project-supported training that they believe has contributed to declines in maternal morbidity and mortality by getting more expectant women to attend ANC, deliver at a health facility, and attend postnatal care (PNC). Furthermore, many CHWs felt that APHIAplus-supported training had enabled them to sensitise women on safe MNCH practices through home visits. One DHMT member appreciated project support in organizing EPI outreaches. Attitude changes were also reported by some CUs, where partner involvement and support in MNCH was said to have begun or improved. “ANC has been enhanced, with CHWs making sure every pregnant mother comes to the facility for four visits. APHIAplus supported ANC services fully.” – Kisumu health facility staff member Stakeholders also said that APHIAplus helped to improve child health indicators. For example, one CU discussant observed that facility deliveries in their catchment area had increased from 0 to 10 per month, immunization coverage had risen from less than 50% to 70%, and ANC services, not previously accessed, stood at 70%. There was also a general perception that vaccination compliance and coverage had improved as a result of APHIAplus support. DHMTs and health care workers lauded CHWs’ efforts to ensure mothers visited the Child Welfare Clinic (CWC). “The death rate of children of under 5 years has been reduced as a result of APHIAplus training on child care.” – Kisumu CU representative Stakeholders also recognized and appreciated APHIAplus nutrition support, primarily training. Health care workers reported they were better able to assess nutrition status because of APHIAplus REPORT Technical report - APHIAplus Western Mid-term Review report 19 training. CHWs also appreciated the MUAC tapes provided by APHIAplus, which enabled them to make basic child nutrition needs assessments. During one CU focus group discussion, participants noted that since APHIAplus has been providing support, the community responsiveness to nutritional needs of children, expectant mothers, and HIV persons has improved. These discussants also mentioned that kitchen gardens were initiated and are now being replicated through APHIAplus support. There were a few challenges with regard to MNCH. Some facilities pointed out the need for renovation or expansion because they had limited infrastructure to enable them provide high-quality MNCH services. CHWs in one CU were also concerned that they had not received their promised ‘reward money’, which supports earlier observations of inconsistencies in CHW stipend expectations. They indicated that CHEWs had informed them that through APHIAplus support, they would receive KES300 (US$3) for every expectant mother they referred to the health facility for delivery. Orphans and vulnerable children (OVC) Local implementing partners (LIPs) supporting the APHIAplus OVC programme reported they were receiving funds to pay fees and to purchase school uniforms, bedding, and sanitary pads. They also received funds to provide psychosocial support through activities such as life skills training. One Area Education Officer, for example, commented on the usefulness of the life skills training supported by APHIAplus, as the training enables OVC providers to better address problems children face in Kisumu’s slum neighborhoods. To enhance economic and food security, APHIAplus supported training of OVC caregivers on how to set up kitchen gardens, and families were reported to be benefiting from vegetable sales. In Kakamega, through CAMP CBO, the food security of OVC households was improved through support for establishing greenhouses, and funds from crop sales were then used to buy items such as soap and books for OVC. All LIPs indicated delays in fee payments as the biggest challenge they faced with APHIAplus OVC support. In some instances, their fee proposals were also underfunded. “School fees delays have been a challenge. For example, first-term fees were delayed for six months. How can this be explained to the OVC households? Further, they had applied for Ksh. 9 million and only received Ksh. 3.2 million.” – Kisumu LIP representative Some staff from community-based organizations (CBOs) included in the assessment expressed dissatisfaction with the perceived “top-down approach” of APHIAplus support. Several described their working relationship with APHIAplus as problematic, and they accused APHIAplus of making programmatic decisions without their involvement or participation. REPORT Technical report - APHIAplus Western Mid-term Review report 20 “The APHIAplus top-to-bottom approach is a challenge and cannot work in an OVC setting. There is need to first understand the needs on the ground and then intervene.... They do not have direct contact with the senior leadership of the project. This does not auger well, particularly if they want to provide feedback on their work to the higher level. The OVC validation exercise flopped because APHIAplus did not involve CBOs in the planning and execution. They cannot rely on the data!” – Kisumu LIP representative Malaria Overall, respondents said little about APHIAplus malaria support. DHMT members did report that malaria supervision was taking place through integrated disease surveillance in the community. Some CU CHWs and CHEWs reported having received malaria case management training. Malaria indicators were said to have improved in project-supported CUs because more people were using LLITNs. However, respondents pointed out that APHIAplus has not supported LLITN distribution. Rather, this intervention is undertaken by a USAID-supported national mechanism implementing partner through another mechanism that is not coordinated through APHIAplus. 3. APHIAplus contributions to cross-cutting service delivery components and strategies Cross-cutting service delivery components and strategies as outlined in the APHIAplus implementation framework include equity, generation of demand for services, and methods to address gender and youth issues. Equity Although stakeholders viewed equity from the perspective of youth and gender interventions that are equitable, the cooperative agreement, results frameworks, annual work plans, and reports did not list any equity performance indicators or reporting on equity in design or service delivery. Generation of demand for services Respondents were asked to describe how they were increasing demand or service uptake and the kind of support provided by APHIAplus. They were also asked to describe how they were integrating services and how their referral network and defaulter tracing mechanisms worked, as well as the kind of support the project provided to achieve integration. Targeted APHIAplus support to C/DHMTs, health facilities, and CUs—particularly in regard to financing, orientation training and mentorship, supplies, procurement, equipment, data systems, and supportive supervision—was acknowledged as having improved demand for services. Some of these services were not being offered or were of poor quality prior to project inception. “[APHIAplus has] assisted in developing quality of service—for example, the WPQAS approach which helps in troubleshooting problems leading to improved services.“ REPORT Technical report - APHIAplus Western Mid-term Review report 21 – Kakamega CHMT member C/DHMTs and health care workers said that engagement of local administration helped to increase demand, noting that chiefs and assistant chiefs play a key role in demand creation through barazas since they also double up as members of Health Facility Management Committees (HFMCs). APHIAplus-supported PEs were also creating awareness and linking communities with health facilities. C/DHMTs and health care workers observed that a strong CU almost always correlated positively with reported high demand for services. The sensitisation provided by CHEWs and CHWs in the community was reported is an important link in health service delivery. Additionally, educational talks at health facilities informed waiting clients of available services. Outreach services provided by health facilities—and supported by APHIAplus—also contributed to demand generation through recruitment of new clients or follow-up of existing clients. Facility staff said their free services and waiver system, intended for those who could not afford to pay for services or who were vulnerable (such as OVCs), also aided in increasing demand for services. Service charters displayed at all facilities were reported as a passive strategy for demand generation. Referral networks and defaulter tracing: Referral strategies are centred in the Community Strategy. When CHWs make home visits, once need is determined, they refer clients to the nearest facility. They use referral forms that APHIAplus trained them to use and follow up with the facility to ensure the client arrived. CHWs also countercheck this by making follow-up household visits and confirming the facility visit by looking at the client’s clinic card. The referral network reportedly improved demand for services, especially ANC visits, immunisation, and delivery services. “The CU linkage with the facility has acted as the most effective referral network. The CHWs refer the client to the facility and call the facility in-charge and later follow up to ensure that the referral is an effective one. This has effectively closed the gap for loss to follow-up.” – Kisumu CU member CHWs said that clients in some cases choose to visit health facilities other than those they have been referred to because of personal preferences, geographical challenges, or stockouts of drugs or supplies. They were also concerned that some health care workers disregard the referral forms and do not sign them. Shortages of referral forms and ambulances were noted as crippling challenges. APHIAplus-supported PEs at health facilities are the point persons for defaulter tracing. All PEs keep a diary, provided by APHIAplus, containing contact details for clients. This information is used by both PEs and CHWs to trace defaulters. This system seems to work well, though stakeholders said it could be improved if all CHWs had mobile phones and airtime. The bicycles that APHIAplus supplied to CHWs have proved very useful for defaulter tracing. Cough monitors who monitor TB clients said the stipend they receive from APHIAplus has motivated them to carry out their work. REPORT Technical report - APHIAplus Western Mid-term Review report 22 “The CHW is the backbone of the fight against loss to follow-up. The CHWs are aware of the status of all those they refer and they keep a very tight leash on them.” – Kisumu health facility representative Challenges listed included a suboptimal number of CHWs relative to the catchment population. It was also reported that PEs usually receive their monthly stipend late. Because the amount of airtime allocated to PEs was considered insufficient, they did not always make reminder calls or send SMS messages. “The catchment population served by the facility is very large, and one CU is unable to meet community needs. Although the standard acceptable ratio of CHW to household is 1:50, some CHWs have as many as 162 households. APHIAplus is launching two new CUs to curb this need. In addition, the project is planning to procure phones for facilities to be used exclusively for follow￾up.” – Kakamega health facility representative Integration of service delivery: HIV/AIDS, TB, and MNCH services are generally well integrated, according to stakeholders. Vitamin A supplementation and deworming services are also integrated, particularly during Malezi Bora, as supported by APHIAplus. APHIAplus has also supported integration of services by providing relevant training for health care workers, supplying relevant screening kits, supporting DHMT and health facility quarterly meetings that discuss service integration, and supporting integrated outreach activities. “Through the community unit [CHWs and other CHCs], the community has been mobilised to attend outreach services where HTC, malaria, MCH, and TB screening and treatment is done, and curative services are provided. APHIAplus has provided trainings, transport [bicycles], lunches, stipends, and materials.” – Bungoma DHMT representative “APHIAplus has provided support in strengthening integration through training and mentorship. The facility staff conduct monthly evaluations to assess the quality of care and utilization.” – Kisumu health facility representative Nonetheless, some DHMT staff felt that integration of services, though extensively discussed at quarterly meetings, was not being operationalized. For example, they noted that referral networks supported by APHIAplus were providing referral tools that were not being completed appropriately. Health care workers also felt that infrastructure limitations hampered the provision of or compromised the quality of integrated services. Stockouts of drugs and supplies also limited service integration, particularly during outreach activities. All cadres of health care workers recommended additional training on integration of services. CHWs, in particular, were observed not to understand integration concepts. Importantly, though, some of these challenges fall within the scope of the APHIAplus mandate, such as addressing coordination issues at quarterly meetings supported by the project or REPORT Technical report - APHIAplus Western Mid-term Review report 23 strengthening referral networks. However, other challenges, such as stockouts, are not within the scope of APHIAplus to directly support. Gender-related issues Respondents were asked to describe how APHIAplus had supported them to address gender￾related barriers that may prevent people from receiving needed services. CHMTs, DHMTs, and health care workers said that APHIAplus supported a number of gender initiatives, such as local radio programmes featuring discussions on gender issues, training for DHMTs and health care workers on gender-based violence (GBV) and post-rape care, and support for health facilities to make their services gender-sensitive, such as by setting aside rooms for attending to rape victims. Stakeholders also noted that APHIAplus supported a number of action days addressing GBV. “There are more women CHWs than men, and they are able to seek out gender issues. APHIAplus has also trained retired midwives in the community, and they can to assist in gender issues.” – Kakamega CHMT member APHIAplus gender support significantly contributed to awareness on gender-related issues, according to respondents. “Health facilities that are supported by APHIAplus have been receptive of clients referred from the GBV centre. They have also been trained on compliance. So far, APHIAplus has supported 1,240 victims in referral, counselling, and establishment of support groups.” – Kakamega LIP representative Even though they acknowledged APHIAplus support to address GBV and rape, most facility-level health care workers did not seem to understand the government’s policy and guidelines on gender mainstreaming at the health facility, how gender affects service delivery, and how to integrate gender￾sensitive mechanisms into their work. Two exceptions were health facilities in Kisumu and Kakamega, where staff indicated that APHIAplus had sensitised them on gender and were attempting to implement gender mainstreaming strategies. “The facility staff have been sensitised on gender matters. APHIAplus supports a men’s support group and is considering doing the same for an organized group of women. This has addressed stigma, which otherwise would have been a barrier to seeking services.” – Kisumu health facility representative At the CU level, most stakeholders admitted that the gender mainstreaming policy was alien to them. Some CUs indicated that they had not received any support to address gender-related barriers but welcomed the idea. A small number of CUs indicated they had received gender training. REPORT Technical report - APHIAplus Western Mid-term Review report 24 “There is limited support from APHIAplus. However, the CUs have reduced GBV in the community through community sensitizations in barazas and other community gatherings….Rape cases that were unreported are now being reported as a result of the sensitizations.” – Nyamira CU member In Kakamega, CU members felt that this kind of support would help them to better implement the national policy and to develop and implement interventions to address gender-related concerns in the community, primarily by sensitising men on the need to seek skilled medical care. They did observe that the ratio of men to women who seek health services was disproportionate, possibly a result of the fact that health promotion initiatives have focussed on women. “Health services are more accessed by women than men, and health-seeking behaviours of men are low and wanting. Male CHWs are being used to reach fellow men. APHIAplus has also trained the health facility staff on rape, drug abuse, and legal issues. All of these efforts support the community better.” – Kakamega health facility representative Youth focus Through the APHIAplus life skills education (LSE) programme, the project has supported health talks for youth in schools. Some DHMT staff attributed reduced cases of septic abortions to this intervention. Youth are also being empowered through APHIAplus support to initiate IGAs, according to stakeholders. APHIAplus is also supporting youth-focused health promotion and HIV prevention through Magnet Theatre, an innovative communication tool, as mentioned by several local implementing partners (LIPs) supported by the project. In most health facilities, lack of infrastructure and materials was reported as the key challenge limiting establishment of ‘full on’ youth-friendly services (YFS). This also seems to have limited APHIAplus’ ability to support facilities to establish the same. One Kakamega DHMT said they had written a proposal to APHIAplus for support of YFS but had not received a response by a year after submission. Overall, based on the C/DHMT and health care workers interviews, it was apparent that having a YFS focus within government health facilities was not an immediate priority. “There was an attempt to introduce YFS, but this has not succeeded. APHIAplus supported youth activities, such as sports and games, by providing HIV testing. A few staff were sensitised on supporting youth at the PGH, but due to shortages of staff it has been impossible to maintain the right staff for YFS.” – Kakamega CHMT member Stakeholder reported many challenges that commonly plague youth including drug abuse, unsafe sex, teenage pregnancies, and unsafe abortions. In response, CU members reported that CHWs provide male condoms to male youth. They further indicated that female youth ask CHWs and health facilities for female condoms because they are shy or unable to negotiate with their partners for use of male condoms. They also recommended that YFS at health facilities or community rehabilitation REPORT Technical report - APHIAplus Western Mid-term Review report 25 centres be managed by youth. They observed that youth often do not come out to express their issues but open up when targeted or approached. The greatest difficulty appears to be limited resources and commitment within the health ministries needed to support YFS. 4. APHIAplus contributions to improving data quality and utilisation C/DHMTs and HCWs were asked to describe the quality of the data that they generate at their health facilities on a monthly basis. Responses were mixed. Some teams indicated that it was satisfactory, and others indicated that it was poor. Various data quality issues were cited by respondents and included incomplete monthly data reports, manually filled in reports that are not legible, double reporting, and poor storage practices. “Data generated from the facility is of good quality as all staff members are involved in the process. Each department compiles its data, and then it is merged and analyzed before submission.” – Kakamega health facility representative The quality of data is not very good. Data quality is compromised by lack of proper understanding of the indicators. At the CCC where the in-charge has been directly involved, the quality is good. – Bungoma health facility representative Respondents were also asked to describe what they do with the data collected, how it is utilised by health care workers, and how satisfied they were with this process. In all quarters, APHIAplus￾supported training was lauded for enhancing understanding of the tools as well as for using data for decision-making. APHIAplus was also credited with providing continuous mentorship and supply of tools where the MOH had shortfalls. Where there were challenges, they were discussed at APHIAplus￾supported monthly data meetings with DHMTs. These monthly meetings were considered valuable and essential to the data improvement process. “The facility has a quality improvement committee that shares and analyses the data, and when they have a challenge, they call APHIAplus and discuss it together for better guidance. APHIAplus has been very helpful and comes regularly and sometimes calls.” – Kisumu health facility representative “Data has issues but APHIAplus has been supporting review meetings. Health care workers have been trained in data analyses.” – Bungoma CHMT member APHIAplus is supporting the chalkboard-based Community Strategy health management information system (CHMIS) that CHWs are using to gauge their performance against set indicators, and the system is updated monthly. The chalkboard is public and accessible by all community members. Data review meetings with CHWs and CHCs, supported by CHEWs, are conducted monthly. These meetings determine planning for dialogue and action days. REPORT Technical report - APHIAplus Western Mid-term Review report 26 5. APHIAplus relationships and support for coordination with the MOH and other stakeholders When asked to describe their working relationship with APHIAplus, most respondents described it as good. APHIAplus financial and technical support for planning and implementation of activities as well as supportive supervision contributed to this description. “The relationship is excellent. It is good despite the bureaucracy.” – Bungoma CHMT member Nonetheless, some respondents said that delays in planning, financing, and supportive supervision had compromised their views of their relationship with APHIAplus, indicating that these delays had led them to postpone or eliminate certain activities. Education officers in Nyamira reported that their working relationship with APHIAplus had deteriorated over time as APHIAplus involved them less in decision-making than in the past. For example, several Western Kenyan stakeholders specifically mentioned that relationships were better during the APHIA II Western project (2005–2010). Even APHIAplus-supported meetings and workshops, which were held regularly in 2011-2012, had become scarce, according to some respondents. “It was very good initially, but of late there have been challenges”. – Nyamira AEO The evaluation team observed that the relationship between the MOH and APHIAplus did not seem to be demand-driven. It would seem the MOH is implementing what APHIAplus ‘directs’ it to do rather than asking for technical support for what needs to be done or having significant input on the direction of health programming. Comments from the DHMT, CHMT, and county ministry officials suggest a similar undertone and concern that the mandate of APHIAplus trumps partner needs or expectations and has surpassed possibilities or practicalities. Comments also suggest that APHIAplus support, even where it was provided within expected limits, was usually considered insufficient. APHIAplus has, however, continuously capacitated C/DHMTs and stakeholders in the anticipation that this will contribute to establishment of sustainability mechanisms. Nonetheless, it was clear from interviews that a dependency syndrome has developed. Management review The management review focused on getting an internal signal from a sample of project technical and management teams as to how they perceive the project is progressing. Key questions that the management review set out to answer included:  What are APHIAplus’ critical inputs or strengths of the current technical and management approach of the project.  Key challenges faced by technical and management personnel. REPORT Technical report - APHIAplus Western Mid-term Review report 27 Critical inputs/strengths Most project management staff interviewed highlighted APHIAplus support to DHMTs as a critical input and strength. They especially cited assistance given with tracking indicators and data and developing annual work plans and action plans. They noted that key resources such as training and mentoring were appreciated across local implementing organizations. There was recognition across the technical and management units that the approach to establishing community units was starting to take hold and that efforts with OVC were reaching targets. Technical and management leads mentioned that integration within the project was getting stronger. Procurement and finance systems were enhanced, or gaps were being addressed. Much of this integration was still new, but teams felt they were already planning better at the county level. Contacts with key stakeholders at the county and district level were becoming more coordinated, and there was a growing recognition of the efforts and role of the county coordinator. Technical and management leads identified a number of innovations that have been critical project inputs, as outlined in Table 3. Half of the innovations mentioned are PATH-led youth-focused HIV prevention interventions (SIO, RAVE, SWAG, Magnet Theatre). Table 3. Critical project innovations identified by technical and management leads. Innovation Description Boda Boda The Boda Boda Ambulance initiative is an affordable referral option for pregnant women in labour. Splash Inside Out (SIO) This introduces health activism at work by conducting informal health education/peer education sessions. RAVE Reaching Adolescents via Health Education. Use of the DHIS2 data for decision￾making District Health Management System 2 is open-source software that allows users to access aggregated health data. Use of Mpesa to pay CHW/Vs Mpesa allows payments to be made mobile-to-mobile. SWAG This is an initiative within the APHIAplus Facebook page that provides youth opportunities to explore and share knowledge and experiences related to FP/RH, VMMC, and TB. Magnet theatre as a channel that can be used by all technical health areas This is interactive community theatre that addresses and promotes education on health topics such as HIV and sex. Integrating technical streams of work such as MCH/RH/HIV in integrated RH camps These are community outreach events in target locations where MCH/RH/HIV services are offered free of charge. REPORT Technical report - APHIAplus Western Mid-term Review report 28 Challenges Many respondents said the complexity and size of APHIAplus represented an ongoing significant challenge. Another common theme was a need for better coordination and communication. Both external and internal communication/coordination were considered sporadic and often inconsistent. Also, skills and execution around planning were cited as a challenge that the project needed to address more consistently. As one interviewee stated, “There are some who are planners, while others plan only a few days in advance.” Although coordination was improving at the county level through use of county workplans and introduction of the project’s electronic DASHboard to monitor workplans monthly, most respondents discussed the need for more formal linkages and collaboration. Some progress reporting was occurring at the county level but was not fully used to inform work planning or target setting across the project as a whole. Additionally, some technical advisors still preferred to schedule visits through their technical liaisons in the counties rather than coordinate visits through the county coordinator. When activities are not synchronized at the county level, some interviewees felt the project was missing opportunities for routine communication with DHMT or other district stakeholders and potentially undermining the APHIAplus brand. In some cases, technical staff were pulling activities from the work plan but not fully integrating their specific opportunities with other technical areas. A reoccurring example was training. Some participants, for example, noted that RH, malaria, and nutrition training could be integrated and conducted at the same time rather than trying to coordinate three different trainings. Another example was the need for more opportunities to conduct technical planning involving both country coordinators and STAs, whereas these groups currently meet only when sharing the results for the quarterly report. Additionally, interviewees highlighted a need for more uniform use of planning and communication tools. Although many meetings are scheduled, they are often cancelled because of last -minute requests from the COP, DCOPs, USAID, or MOH. Two other reoccurring themes were a lack of good measurement and information management and tracking systems, especially for the provision of OVC benefits, procurement systems, and M&E. Respondents did say, however, that they saw project leadership taking active steps to improve these systems and processes. In some cases, such as the planned MIS system for OVC and BCC, interviewees did not know the status of the planned improvements. The constant need for new staffing and staff turnover was also identified as a key internal challenge. Several leaders expressed frustration with the length of time needed to hire new personnel. As one interviewee stated, “It takes anywhere from 4 to 6 months to hire someone.” In some specific technical areas, including HIV prevention, as well as functional areas such as community strategy and M&E, project staff expressed that APHIAplus does not have all the skills REPORT Technical report - APHIAplus Western Mid-term Review report 29 needed to address the challenge of helping the MOH expand services and increase demand. A few participants noted that a challenge ahead is to figure out better ways to synchronize APHIAplus’ traditional model of working through facilities with a more community-centric model that would require other skillsets more aligned with facilitating community strategy. Although many respondents felt that measurement on the whole was improving for the project, they felt an ongoing need for better mechanisms to understand the results of the data collected, improved feedback loops and co-analysis of project data, and improved understanding of USAID targets. They also noted that the project needs to be better rationalized, including the setting of denominators at a county or sub-country level to better understand performance. Finally, the project needs better ways of capturing analytics that link activities to the intended outputs. Some respondents mentioned that they cannot easily show how more outreach activities or training may be contributing to service delivery trends. 4.0 Conclusions and recommendations Conclusions and recommendations based on the midterm review fall into six thematic areas: 1. Targeted implementation strategy: The aims and specific geographic, technical, and demographic areas where APHIAplus can most effectively make a difference during the remaining two years of the project. 2. Consistent measurement and learning: Data collection and measurement systems, tools, and processes. 3. Coordination/communication: The process flow, integration, sequencing, and steps of sharing information as well as the mechanisms of dialogue, exchange, and push and pull of information. 4. Collaboration: The mechanisms by which people come together, establish norms and relationships, and build trust both internally and externally. 5. Capacity: The mechanisms through which the project trains currently available talent, brings in new talent, or outsources talent to support technical and functional areas. 6. Sustainability: The vision and plan for maintaining the advances gained through the current assistance and institutionalizing best practices. Each of these thematic areas is discussed in detail below. 1. Targeted implementation strategy. Although much of the project’s work has been shifted to ensure that teams operate in a more integrated fashion, the project must be nimble enough to use different tactics to address different performance issues in Nyanza and Western provinces. There are unexplained or only partially explained differences between the two provinces in key indicator trends in areas such as ANC, PMTCT, and immunization. If APHIAplus is to meet performance targets for USAID, it must more fully understand these trends and identify county-level strategies to improve REPORT Technical report - APHIAplus Western Mid-term Review report 30 performance. Subsequent recommendations (see below) focus on specific tactics that the project team may use. A first step will be to identify specific counties, facilities, and communities that need:  New activities/approaches.  A different frequency or consistency of existing activities and approaches. There is a critical need to clarify the process by which project technical and managerial staff and county leads understand how targets are set annually in relation to the scope of proposed work and concurrent activities, identify root causes of low performance, select and target activities to change performance, and consistently monitor performance moving forward. Given the number of geographic shifts USAID has asked the project to make, the effects of these shifts need to be appropriately reflected in targets the project is asked to reach. More rigorous forecasting at the catchment or district levels should be considered to allow the management and technical teams to discuss current performance and needed recalibrations to USAID targets. 2. Consistent measurement and learning. The MTR found a need to increase emphasis on continuous monitoring, evaluation, and learning. Ensuring data quality and accuracy is currently a multistep process that requires ad hoc corroboration with both technical and county staff. Systems need to be prioritized and implemented to ensure the project can illustrate its contribution to expected outcomes. Monitoring has been the most consistent for service delivery at the facility level, but there is still a need for better interpretation of results and reporting of insights. The analysis and interpretation of results need to increasingly be completed in conjunction with district health teams and project county teams. Stakeholder perceptions and indicator data were sometimes contradictory. For example, stakeholders in some areas reported favourable trends in demand for HTC and ART services, but data from specific facilities and districts showed either flat or even decreasing demand. This may indicate that APHIAplus needs to redouble its efforts to help DHMTs and facilities see their data and use it effectively for decision-making and review the rationalization of targets at the county, district and catchment level.. Key recommendations related to measurement and learning include:  Redouble efforts to ensure consistent follow-up with MOH M&E officers at the district level and with DHMTs.  Establish more consistent internal monthly reporting and data corroboration with technical and county leads.  Complete the OVC and BCC integrated MIS and merge these data with DHIS2 facility data to provide an accurate picture of the overall impact of the project.  Import data into visualization dashboards and continue to refine the data to understand key project indicators as well as other analytics.  Train key M&E staff in the implementation and use of key data visualization software to ensure dashboards can be reproduced and easily shared with staff. REPORT Technical report - APHIAplus Western Mid-term Review report 31  For a high-priority set of service delivery indicators, establish denominators to understand whether trends make sense in relation to targets set by USAID.  By using existing data and available baseline results, explore the data more thoroughly to understand: o The variation in prevalence of HIV in different geographic regions and among different subpopulations, such as youth, and the comparative transmission and resistance rates (as applicable) in the counties where APHIAplus works. o Anthropologic cultural and social information on practices, beliefs, and norms contributing to health and health-seeking behaviours, such as factors influencing behaviours to seek ANC services and skilled delivery, gender inequities, and inclusion/exclusion of youth.  Increase the number of people who can access key indicator data and use it for decision￾making.  Establish data, programme, and resource structures that support regular, substantive feedback mechanisms to improve the effectiveness and efficiency of service delivery and management systems. Institutionalise organisational learning through use of MIS combined with visualization tools such as Tableau.  Share lessons learned (negative as well as positive) in forums and in ways that are useful to other implementers in Kenya, particularly other APHIA projects, and in the health and development sector. Document the process of CQI and its uptake as a best-practice model that other APHIAs and programmes can use. Although APHIAplus reports contain a lot of good ‘lessons learned’, change resulting from these lessons was not evident. 3. Coordination/communication. Interviewees typically noted that the project could be more consistent in its coordination and communication, both internally and externally. Given that APHIAplus must work through intermediaries to accomplish its work, the project needs to find more systematic mechanisms to both set up work and communicate expectations and limitations related to the work. Interviews suggested some areas of misalignment between the perceptions of stakeholders and those of project staff. One example is the Mpesa program. A number of stakeholders said this program was not working at an optimal level and gave examples of missed or late payments. By contrast, project staff said the system is now running well with few missed/late payments, though they acknowledged that there were initially some problems. Although the MTR did not attempt to verify whether the Mpesa payments happened regularly, the discrepancy highlights the need for either further improvements in the coordination of Mpesa or simply better and more regular communication with Mpesa recipients. A second example is the confusion around the CHW stipend given by APHIAplus, which is being inconsistently applied. A key recommendation is to develop a communication plan that includes clear messages on stipend payments to project partners at all levels. Payments should then consistently applied and monitored jointly by the project and MOH staff to avoid further confusion and ill will towards the project. REPORT Technical report - APHIAplus Western Mid-term Review report 32 Internally, gaps in coordination and communication may be undermining solid technical approaches or leading to missed opportunities to sequence technical activities focused on the same audience. Gaps or inconsistencies in coordination and communication with DHMTs were reflected in both internal and external interviews. Some key short-term solutions can be implemented to remedy these shortcomings. Key recommendations to improve coordination and communication are outlined below:  Develop and implement an internal communication/engagement plan with regular planned intervals of communication and scheduled visits that are communicated in advance.  Institute more consistent joint technical trips to key stakeholders for planning, support supervision, training, or mentorship.  Institute a consistent and inclusive mechanism for county-level technical planning that includes project technical leads and STAs.  Ensure more consistent coordination and communication around the approach to youth￾friendly services and gender.  Explore how the TB and Social Determinants of Health Teams work with county coordinators and county teams to integrate their work. Both teams were cited as positive models of integration.  Develop a communication plan for stipend payments to CHWs and ensure the plan includes a clear set of messages on stipend payments and on current MOH policy on CHW stipends. Then consistently apply the payment scheme, with joint monitoring by MOH and project staff. 4. Collaboration. The MTR also highlighted that the project’s approach is not always collaborative and at times is perceived as directive, especially to key district-level stakeholders. The project model for collaboration needs to be defined and emanate more clearly from county leadership. Variability of site performance may be consistently related to counties and technical areas that are concentrating efforts in honing their collaborative approach. Key recommendations in this area are:  Identify personnel at the deputy director level who will champion improved collaboration.  Work with county coordinators to understand which stakeholders require more collaboration.  Design county-level engagement strategies and activities focused on improving stakeholder collaboration and building sustainable capacity within key stakeholder and implementation groups. 5. Capacity-building. The project’s work to build capacity requires a more systematic approach. The current mentorship model is not well defined and is not seen as a priority, especially for those who are supposed to be ‘mentoring’ others. Although the value of trainings and mentoring is recognised by counterparts, the project cannot easily show a plausible contribution of these outputs to REPORT Technical report - APHIAplus Western Mid-term Review report 33 improvements in the performance of counties, facilities, or CUs over time. The approach needs to be more intentional and planned and linked to performance tracking of health facilities and CHWs. Key recommendations:  Establish a project champion from the SMT to help lead improvements in the APHIAplus capacity-building approach.  Engage internal/external technical assistance to develop a two-year capacity-building plan with key partners that includes a clear set of objectives, technical areas to be covered, and milestones. The plan should build on the technical strengths of the project team to continue to mentor and enhance skills.  Establish formal mentorship relationships with the health system throughout the project and contextual systems, set criteria, and conduct regular follow-up to understand whether skills and knowledge are enhanced and used.  When feasible, explore more ‘piloting’ and ‘demonstration’ opportunities, where innovative systems are tested and validated. 6. Sustainability. Findings from this assessment confirm that APHIAplus is not a viable long￾term model to meet future needs for technical assistance, financial support, and supportive supervision. This should raise concern because it is unclear who would play the APHIAplus role if the project were to disappear tomorrow. To this end, the project must start to create an exit strategy and include a communication plan. The project will need to help develop models in collaboration with government, private sector, and civil society partners that will contribute more consistently to strengthening the Kenyan health system. Key recommendations:  Actively investigate and engage models of continuous improvement focused on how the project can provide increasingly higher-quality inputs to ensure the sustainable delivery of scaled services in the MOH.  Depending on the willingness of participants in the MOH and other critical stakeholders, begin to investigate ways to embrace critical aspects of a continuous quality improvement approach in a meaningful way in the final two years of the project.  Establish ‘exit strategies’ now for programmes, activities, and resource mobilisation in consultation with stakeholders, and institutionalise a structured analysis and reporting of progress toward those exit strategies as part of annual reviews and M&E.  Develop a communication plan that carefully articulates proposed exit strategies and clarifies expectations among stakeholders that will need to change when the project ends. The plan will need to include key target audiences, key messages, channels for disseminating information, proposed communicators, and a timeline for executing the plan, beginning in 2014 and continuing through the end of the project. REPORT Technical report - APHIAplus Western Mid-term Review report 34 The second half of APHIAplus will be stronger and more effective than the first half and lead to more sustainable results if the project team learns from the successes and weaknesses of the previous period. Many of the issues that should be reviewed and addressed are already recognised by APHIAplus staff and stakeholders. The challenge, then, is for APHIAplus management and staff to listen to internal and external feedback and have the systems and courage to try new things. This was demonstrated, for example, with the MTR activity. This exercise brought in new techniques that the project team had not previously used. The MTR team believes that this ability to learn, experiment, and document improvements for greater impact is a comparative advantage of APHIAplus in the sector and in the Kenyan national health context. APHIAplus needs to turn outward, using the lessons and skills acquired during the first two and a half years, exploring models to advance community and facility-based service delivery systems, and leading the learning and application process throughout the Kenyan health care system. REPORT Technical report - APHIAplus Western Mid-term Review report 35 5.0 Annexes Annex 1. Comparison of Results for Selected Indicators in Nyanza and Western Provinces Indicator Region Results 2011 2012 2013* Number of individuals who received counseling and testing for HIV and then received their test results (excluding TB) Nyanza Achievement 192,087 223,454 57,049 Target 139,828 146,819 38,540 % of Target Met 137% 152% 148% Western Achievement 549,539 329,418 48,287 Target 423,200 431,664 110,073 % of Target Met 130% 76% 44% Number of women provided with PMTCT services Nyanza Achievement 32,873 35,390 13,759 Target 40,000 30,000 2,000 % of Target Met 82% 118% 688% Western Achievement 133,997 113,783 18,947 Target 137,200 137,200 8,625 % of Target Met 98% 83% 220% Number of women provided with ARV prophylaxis for PMTCT Nyanza Achievement 3,950 3,087 908 Target 4,420 2,763 800 % of Target Met 89% 112% 114% Western Achievement 4,390 3,391 565 Target 4,665 4,665 1,050 % of Target Met 94% 73% 54% Number of infants tested for HIV at 5 weeks Nyanza Achievement 1,261 1,108 281 Target 2,652 1,658 432 % of Target Met 48% 67% 65% Western Achievement 1,464 1,631 280 Target 2,799 2,799 630 % of Target Met 52% 58% 44% Number of HIV-exposed infants provided with ARV prophylaxis Nyanza Achievement 3,494 3,115 986 Target 4,420 2,103 720 % of Target Met 79% 148% 137% Western Achievement 4,229 3,614 607 Target 4,665 4,665 893 % of Target Met 91% 77% 68% Number of individual patients newly enrolled into HIV care within reporting period Nyanza Achievement 11,342 13,613 2,224 Target 20,000 16,800 3,750 % of Target Met 57% 81% 59% Western Achievement 8,381 10,464 1,843 Target 14,000 14,000 3,000 REPORT Technical report - APHIAplus Western Mid-term Review report 36 % of Target Met 60% 75% 61% Number of individuals NEWLY initiating antiretroviral therapy during the reporting period Nyanza Achievement 9,180 6,835 790 Target 12,000 10,080 2,000 % of Target Met 77% 68% 40% Western Achievement 6,524 6,178 1,105 Target 8,400 8,400 1,800 % of Target Met 78% 74% 61% Number of individuals currently on antiretroviral therapy during the reporting period Nyanza Achievement (Monthly Average) 35,284 37,345 25,897 Target 40,000 48,000 46,000 % of Target Met 88% 78% 56% Western Achievement (Monthly Average) 22,549 27,183 31,224 Target 25,120 32,140 32,000 % of Target Met 90% 85% 98% Number of positive persons currently on cotrimoxazole prophylaxis Nyanza Achievement (Monthly Average) 66,605 63,384 40,272 Target 40,000 25,000 94,300 % of Target Met 167% 254% 43% Western Achievement (Monthly Average) 44,636 48,742 50,682 Target 50,600 61,800 60,000 % of Target Met 88% 79% 84% Percentage of CHWs in each CU who submitted a monthly report Nyanza Achievement (Monthly Average) 88.6% 94.2% 86.3% Target 100% 100% 100% Western Achievement (Monthly Average) 85.2% 98.5% 94.4% Target 100% 100% 100% Number of the targeted population reached with individual and/or small group level preventive interventions that are primarily focused on abstinence and/or being faithful, and are based on evidence and/or meet the minimum standards required Nyanza Achievement 726 69,553 8,767 Target 18,632 20,000 10,000 % of Target Met 4% 348% 88% Western Achievement 276 85,477 22,804 Target 20,000 10,000 % of Target Met 427% 228% Number of target population reached with preventative interventions that meet minimum standards Nyanza Achievement 2,052 8,807 76,838 Target 50,000 % of Target Met 154% Western Achievement 5,044 116,016 Target 50,000 % of Target Met 232% Number of MARP reached with individual and/or small group level interventions that are based on evidence and/or meet the minimum standards required Nyanza Achievement 1,176 3,211 7,440 Target 2,000 20,000 1,375 % of Target Met 59% 16% 541% Western Achievement 6,874 29,300 10,947 REPORT Technical report - APHIAplus Western Mid-term Review report 37 Target 2,000 15,000 1,250 % of Target Met 344% 195% 876% Number served in 3 or more core areas Nyanza Achievement 12,907 105,605 37,810 Target 100,000 100,000 25,000 % of Target Met 13% 106% 151% Western Achievement 9,670 104,181 34,150 Target 60,000 60,000 15,000 % of Target Met 16% 174% 228% Number of males circumcised as part of the minimum package of MC for HIV prevention services Nyanza Achievement 14,539 25,828 163 Target 7,000 18,000 1,658 % of Target Met 208% 143% 10% Western Achievement 4,700 8,291 41 Target 5,500 7,000 844 % of Target Met 85% 118% 5% * Targets adjusted to include only first quarter of 2013. Annex 2. Terms of Reference (ToR) for the Mid-term Review Phase I: Desk review  Review of relevant documentation, including available project studies, program reports, USAID strategy documents, current status of PMP indicators selected by Senior Management Team, , and other reports related to the project. Phase II: Site Visit  Structured interviews with project staff and stakeholders, including MOH and MOE  Field visits to project areas to interview field stakeholders including MOH officials, implementing partners, service providers, and field agents.  Visits to selected project supported clinical facilities to observe service quality, access, utilization, data accuracy, and completeness.  Structured discussions with local implementing partners and community members to assess the extent to which the project has achieved its goals in behavior change and communication.  Structured discussions with community stakeholders including local civil society organization staff and beneficiaries to assess OVC program delivery.  Sampling of the project areas to be visited Review Team Phases I & II: Lead consultant: PATH Global M&E Department, Seattle, Washington, USA, in consultation with APHIAPLUS Western Senior Management Team and project M&E team. REPORT Technical report - APHIAplus Western Mid-term Review report 38 Phase I: Phase II planning and desk review. The lead consultants will be assisted by the M&E team to finalize Phase II planning and to ensure secondary source of data and project information is provided in a timely and comprehensive fashion. Phase II: Through advisement of the project’s Senior Management Team, during Phase II, the lead consultants will lead a team made up of project technical staff from each of the partner organizations, county coordinators, one from Western and Nyanza provinces respectively, and M&E team members. Deliverables On the basis of the review, the consultants will prepare a report that presents the above objectives including recommendations on areas of improvement and way forward for maximizing potential to complete project deliverables and meet expected outcomes. REPORT Technical report - APHIAplus Western Mid-term Review report 39 Annex 3. Field Syntheses Notes These are enclosed separately. Annex 4. Top Level Findings Summary Sheet Preliminary Findings Presented to A+ Middle and Senior Level Management on 19-Jul-13 Planning Examples of Strengths Examples of Challenges Fac ility NKi: A+ agreed on AWP w facility WKa: Project supports quarterly meetings WKa: H/holds mapped for CHWs to plan visits NN: Facilities go through AWP formalities w/out understanding processes NN: A+ has not supported in AWP WKa: Not timely: last year’s work plan did not receive implementation funds WKa: Additional support required in infrastructure development CH MT WKa: Leds annual work plan process NKi: A+ almost solely supported AOP, AWP planning NN: A+ plan is not aligned w county work plan WKa: Follow thru is weak NKi: Planning not inclusive, doesn’t involve MOH DH MT WKa, NN, NKi: Development of work plan , action plan, process support WKa: NN: A+ support not aligned with AWP NN, WB: Support is not coordinated, A+ comes w own agenda WB: Communication not timely and often conflicting WB: Promises are not fulfilled CU NN: Integrated planning of activities such as dialogue days, action days, CHC/HFMC meetings NN: Have monthly performance review meetings, trainings, A+ supported CHC meetings NN: A+ staff have not attended any planning meetings at the CU LIP/ CSO WKa:Data collected is used in planning W: Accomplished planned activities, targets WKa: (OVC CSO) Project has supported Quarterly meetings with regional staff that enables them to plan in an integrated manner. Wka: A+ has provided a forum for harmonization of various ministries’ work with that of CBOs – created a leverage of CBOs on line ministries such as Ag. Wka: A+ has supported areas of CBO workplan that address gender equality issues. WKa: Both A+ and CAMP do their planning independently. This makes prioritization difficult. NKi:LIPs often not involved in planning, top/down approach Wka: urge harmonization with CU CHWs and CHVs such that resources to CHVs same as received by CHWs (uniforms, bags, communication support e.g. mobile phones, bicycles to move around) Wka: need support to mobilize government to harmonize community level activities. Kis E: better able to learn and scale up A+ interventions REPORT Technical report - APHIAplus Western Mid-term Review report 41 DE O/AEO WKa: Data collected is used in planning Financing Examples of Strengths Examples of Challenges Fac ility WKa, WB: Financial support for CHC/HFMT joint meeting and airtime for CHEWs NN: Strong funding accountability NN: Finances for lunches, transport, and facility renovation WB: Concerns of lack of flexibility with financing, especially on amounts reimbursements. WB: Resources inadequate WB, WB: Delay in reimbursements NN: Non-collection policy hurting facility NN: Mpesa payment delays WKa: A+ has been expected to support certain activities such as dialogues and action days but this has not taken place CH MT Strength Statement: CHMTs’ recognize APHIA Plus for their financial support to a broad range of planning, supervision, training, equipment, commodity provision and support activities. Challenge Statement: CHMTs’ have identified payment delays to program beneficiaries as a challenge and may require further inquiry into A+ Mpesa payment and approval processes. CH MT NN: Quick response when money is requested (within 2 weeks) WKa: Pays related costs for planning meetings, support supervision, conferences, etc NKi: Financial support in many aspects: rapid response to epidemics like polio, supporting trainings, equipment, etc NN, NKi: Mpesa not a good payment mode: demotivates, affects credibility NKi: Delay in response to financial needs, approval process is inconvenient, GoK officers feel marginalized in financial planning DH MT Strength Statement: Challenges Statement: DH MT NN, WB: A+ recognized for support on stipends, training, airtime, transport NN: Delay in disbursements (mpesa) and financial support. DHMTs need to front $ and don’t always get reimbursed NN: Funding comes after activity, encourages fraudulence REPORT Technical report - APHIAplus Western Mid-term Review report 42 Financing Examples of Strengths Examples of Challenges NN: Delayed funding CU Strength Statement: CUs recognize APHIA Plus for their stipend support of CHWs and their direct salary support of facility-basedpeer educators and counselors (referral, client loss and defaulter tracing) Challenge Statement: CUs have identified payment delayed or missed payments to program beneficiaries as a challenge. The amount and reason for the stipend may need further clarification directly communicated to CHWs, and CHCs. CU WKa, NN: Monthly stipend motivates CHWs, support for trainings WKa: A+ has employed peer educators and counselors who support client loss and defaulter tracing WKa, WB, NN: Stipend payments delayed most of the time WKa, NN: Some CHWs have never received any stipend since CU inception WKa, NN: Stipend not sufficient WKa, WB: No/minimal support for dialogue and action days LIP/ CSO WB: Stipends provided motivates CHVs to reach more clients NKi: finance activities occur at all levels (school fees and materials) W: Positive of financial support Wka: due to payment of fees by A+, cases of child labor, rape have declined. Kis E, Wka, WB, NKi, W: late payments for example school fees for OVCs and stipends for CHVs, not timely Eg 6 month delay in school fees payment NKi: Finances not adequate. WKa: high number of orphans that have not been enrolled and if cap placed by A+ lifted from 3,000 it will help address this challenge. Wka: Timely payment of fees should also be done. Facilitate disbursement of funds on a monthly basis needed. DE O/AEO WB: Lack funds to run program activities efficiently Orientation/Training/Mentorship/Guidelines Examples of Strengths Examples of Challenges Fac ility WB, WB: Positive comments NKi: Mentorship sessions WB: Elderly staff reluctant to undergo trainings and orientations REPORT Technical report - APHIAplus Western Mid-term Review report 43 Orientation/Training/Mentorship/Guidelines Examples of Strengths Examples of Challenges NN: Improving skills, enabling multi-tasking and integration NN: OJT, on-site mentorship better NN: Support in mentorship, CMEs, training of CUs WKa: Training for HCWs provided in ART, PMTCT orientation, child survival, malaria and stigma reduction NN: Some staff don’t get training opportunities, A+ requests certain people NN: Training venues unsuitable NN: Work plan is not suitable. WKa: More training required in areas such as CCC, YFS CH MT NN: A+ has provided support, but at DHMT level WKa: Supported on leadership, governance, lower level training of trainers NKi: A+ supports capacity building across the board. Distribution of guidelines has been exemplary. NKi: Ability to offer trainings per USAID regulation not understood, seems A+ is reluctant to do it DH MT WKa: Lots of activities NN,WB: Trainings are recognized NN: Training for integrated services, training for CHWs across multiple health areas WKa: No curriculum training, not certified or skills building NN: Not fulfilling promises (bags, shirts, etc), mentorship not enough NN: Not all key staff reached (CHEWs, Youth-friendly services, CHWs) CU WB, NN: Trainings in TB, HIV, FP, malaria appreciated and have enabled CHWs to offer integrated service/information WKa, WB, NN: Further training required in areas such as gender, reaching out to youth, men and OVCs, CBHIS LIP/ CSO WB: CHVs have undergone training NKi: A+ has capacity in this area W: Materials provided in time for training Wka: (CBO) community has been sensitized on work of CBO (Camp) and more responsive to activities. INTEGRATION: Wka: A+ has trained CHVs on paralegal issues and formed CUs which has led to the interlinking of different organizations which has led to the WB: Training materials are not enough NKi: Full coverage is a challenge Kis E: (OVC) No support on how to integrate service delivery in communities. Kis E (OVC): limited integration REPORT Technical report - APHIAplus Western Mid-term Review report 44 Orientation/Training/Mentorship/Guidelines Examples of Strengths Examples of Challenges dissemination of information. Wka: A+ does not have any discrimination in its provision of support and will support boys and girls alike. Wka: There is a tool that is used in referrals for services. Wka: Capacity building to CHVs/OVC caregivers, (Kis E) training of social workers Wka: training on food security resulting in starting kitchen gardens. Wka: through provision of NHIF registration to OVCs, more people seeking this service. WKA: Integration: other schools have seen results of WASH activities supported by project, and request the same Kis E (integration) – CBO started supporting only girl child education but A+ requested that they include boys. Kis E: A+ supports children clubs that address gender issues DE O/AEO WB: teachers undergone training WB: Training materials are not enough WB: Lack of frequent induction meetings and trainings with teachers NKi: Full coverage is a challenge, only 2/20 schools supported Supplies/Procurement/Equipment/Commodities Examples of Strengths Examples of Challenges Facility WB: Appreciation WB: KEMSA supply not reliable REPORT Technical report - APHIAplus Western Mid-term Review report 45 Supplies/Procurement/Equipment/Commodities Examples of Strengths Examples of Challenges WKa, NKi: A+ supplies equipment using “reach-in collaboration” NN: Commodities are “pull system” NN: Positive across many commodities WKi: A+ not able to meet all demands of facility NN: Lab supplies are not adequate and often late CHMT WKa: Support to furnish office, equipment, computers NKi: Procurement well supported (filing cabinets, lab equipment, commodities, etc) NN: In need of much support in infrastructure, office space, supplies, furniture, CD4 machines, etc WKa: Procurement is long and bureaucratic NKi: Limitations, there are need A+ can’t meet DHMT WB: Bikes, bags, supplies for cerv cancer screening, test kits NN: A+ provides buffer stocks NKi: Supported NN: Rapid test kits NKi: Supplies as expected WKa: Not able to support intfra supplies beyond pou equipment NN: Stocks for ART, RTKs should be decentralized NN: Shortages of lab reagents, anc profile reagents, URI strips, HB strips WB: Lack of transportation WB: Cryotherapy machine lacks gas CU WKa, NN: CHWs provided with reporting tools, t-shirts, bags, bicycles, key messages book and MUAC tapes WKa, NN: Bicycles inappropriate for hilly terrain WKa, NN: Need umbrellas, gum boots NN: Motorbikes required on standby for emergencies LIP/CS O WB: Provision of computers, filing cabinets, files, camera NKi: Resources good in all areas W: Provided with furniture, files, utensils, etc Wka: INTEGRATION: provision of sanitary pads and payment of fees “The camera is used to take pictures during distribution of materials to OVCs” WB: Lack of equipments such as bags, umbrellas, branded tshirts NKi:Full coverage is a challenge, can only supply a section of beneficiaries W: Demand for preventive commodities outstrip supply Wka: (CSO) Not timely – for example, if an REPORT Technical report - APHIAplus Western Mid-term Review report 46 Supplies/Procurement/Equipment/Commodities Examples of Strengths Examples of Challenges for youth, School uniforms and performance monitoring of youth in school ensure that they complete learning, support for youth clubs where youth are able to discuss WASH activities. Training on VSLA which provide loans for youth. WKa: provision of vegetable/fruits for planting to address food security. Support to erect a green house to grow vegetables – sales used to buy hh items such as soap, books ,et for OVC. WKa: Provision of VIP toilets and water tanks improved community level health and led to demand for more. Kis E: provision of mattresses and blankets order is made for sanitary towels it would really take a long time to be delivered. Wka: Delay in disbursement of funds for running of CBO activities and this therefore mean that they ran a whole quarter without any monies to pay CHVs and other facilitations. Wka: communication challenge – lack of funds to move around, lack mobile phones, bags, uniforms (CHVs) as compared to CU counterparts. Kis E: mattresses delivered ½ of what what was required. Poses distribution challenges. DEO/A EO WB: Once cases are identified, material support should be provided Supportive Supervision Examples of Strengths Examples of Challenges Suppor ting quotes (if any) Facility NKi: Happening on a monthly basis NN: Received from DHMT quarterly NN: Have received WB: Mentorship is not well structured or regularized CHMT NN: Received support but as DHMT, not CHMT NKi: Regularity seems to have REPORT Technical report - APHIAplus Western Mid-term Review report 47 Supportive Supervision Examples of Strengths Examples of Challenges Suppor ting quotes (if any) WKa: Well provided NKi: Positive in terms of resources and tools reduced DHMT WKa: A lot happening, doing very well NN: Financial support, hope for continued support WB: Supported well NN: Each facility is reached every quarter, supervision is integrated WKa: Follow thru poor CU WKa: Mentorship programme in place NN: No visits by A+ staff LIP/CS O W: A+ technical and M&E officers helpful in support WKa: (CSO) support supervision provided to help implement OVC activities. Kis E: imparted managerial skills. WB: They felt like they have been abandoned. Wka: increase support supervision by making visits to the community regularly DEO/A EO Information Systems/Data Examples of Strengths Examples of Challenges Facility WB: Appreciated, regular updates WKa, WB: Use data for decision-making in planning for services NKi: Addressed thru supportive supervision NN: Good data, used for decision-making. Coverage has improved from 50% to 60% now NN: Mentorship on data quality improvement and reporting WKa: A+ sponsored M&E training on data M&E and interpretation WB: Inadequate understanding of some indicators NKi: Need more support to improve data quality NN: The filling of register not complete, affecting monthly summaries NN: Lack of data personnel REPORT Technical report - APHIAplus Western Mid-term Review report 48 Information Systems/Data Examples of Strengths Examples of Challenges CHMT WKa: Timeliness has improved, WEPQAS reduced sample rejection to near 0 NKi: High level of support NN: Very poor quality data generated. Lack of computers, lack of DHRIO, lack of data quality assurance WKa: Not doing as much in terms of EMR, but there is EMR partner NKi: Timeliness, untrained data personnel DHMT NKi: Support good WKa: HMIS support Wka, NN, WB: Data review support, data review meeting = better, timely, complete data Potential findings NN: No DHRIO, lack of monthly review meeting, no support for audit WKa: Not all tools are available in all faciliites NN, WB:: Lack of trained data personnel, poor data which cannot be used WB: CHEWs not involved in data collection CU NN: Mentorship for CHWs on data capture and reporting WB: CBHIS tools were all available WKa: Data used for planning action/dialogue days NN: Some indicators are not captured in the reporting tools yet they are important e.g. nutritional data review NN: No training in data quality WKa: Additional support required in providing ARV registers and making CUs ICT compliant by providing computers to streamline reporting LIP/CS O WB:Activity report forms are considered to be good data collection tools NKi: Working albeit with challenges W: Utilizing data generated from activities W: Several levels of data quality checks WKa: (CSO) A+ provided reporting tools for monitoring and supervision, training on use. Availability process of review and signing forms at 3 levels guarantees better quality. Through provision of computers files have been well saved and also updated in a timely manner. Wka: data are used for referening of provision of knowledge on the progress of OVCs. Wka: funds for hiring of data clerks and WB: Lack of a tool that captures data on a monthly basis. Kis E: OVC validation study a flop because A+ did not involve CBOs in the planning and execution. Cannot rely on data. Kis E: A+ has trained only a fraction who cannot do much in terms of improving data collection and utilization. Reporting tools need to be changed or adjusted to be relevant to CBO needs – doesn’t include other items (e.g. sanitary pads) Kis E: need more cabinets for data storage; REPORT Technical report - APHIAplus Western Mid-term Review report 49 Information Systems/Data Examples of Strengths Examples of Challenges refreshers appreciated. Wka: file cabinets and cameras for data management appreciated. DEO/A EO WB: Data helps to plan and identify problems and their solutions WB: Lack of enough copies of activity report forms, getting data from different zones is difficult. WB: Suggests funds for facilitating timely delivery of reports NKi: Data generally of poor quality Kis E: data quality support in school programs inadequate. Working Relationship Examples of Strengths Examples of Challenges Facility WB: Cordial NKi, NN: Good, supportive NN: Good because A+ supports the CUs as well NN: Good, staff are accessible WB: No clear, deliberate effort by A+ to address gender/youth issues NN: A+ staff do not visit facility CHMT NN: Good relationship, quick response WKa: Cordial NKi: Cordial NKi: Major communication challenges DHMT Nki: Cordial, friendly. WKa: Cordial NN: Good relationship, work in harmony NKi, WKa: Non-consultation NN: Uncoordinated partnership and working relationship between M&E and County Community Strategy team NN: Request not responded to, absence of A+ officer on the ground Kis E: lack of proper communication with A+ office CU WKa, WB, NN: Good working r/ship. A+ transparent in support available WKa: Both HF staff and CUs rarely and have never seen (corresponsingly) A+ staff REPORT Technical report - APHIAplus Western Mid-term Review report 50 Working Relationship Examples of Strengths Examples of Challenges WB, WKa: Stipend delays of up to 5 months and disagreements on refunds have weakened r/ship LIP/CSO NKi: Finance and supportive structures are strength. The project is able to meet the needs of the partners W:Cordial WKa: Can be described as ‘tolerant’: Kis E: supported not always provided in expected timeframe – case of school fee delay good example. Kis E: do not have direct contact with A+ senior leadership. This does not auger well, particularly if they want to provide feedback to the higher level. Kis E: only interacts with SDH and finance teams. DEO/AE O WB, NKi: Good NKi: Desire to scale-up to more schools Annex 5. Persons Interviewed County Health Management Team District Health Managemen t Team Facility Staff Community Unit members District/Area Education Officers Community￾based Organization management (OVC) Community HIV Prevention Local Implementing Partners Kisumu Kisumu East Migosi HC Lower Kanyakwer B Kisumu Municipality Make Me Smile NOPE Ober Kamoth HC Osiri SAPTA Got Nyabondo dispensary Got Nyabondo Usoma dispensary Usoma Nyamira Nyamira Tinga HC Ikobe Nyamira North HEART Nyamaiya HC Bokiambori Gucha Gianchore HC Timi Manga Ogango HC Ogango Borabu Mwongori dispensary Manga Raitigo Nyandoche Ibere Dispensary Matutu Bungoma Bungoma North Ndalu HC Misanga Bungoma North Bungoma HBC Ace Africa WEAEP Bumula Bumula HC Muanda Bungoma central Chemwa Bridge dispensary Chemwa Cheptais Kang'nga dispensary Cheptais Kimilili Nasusi dispensary Nasusi Kakamega Matete Matete HC Kivaywa Lugari CAMP Kwisero Elwangal HC Munjiti Kakamega Kakamega East Shamakhubu HC Itenyi Lugari Marakusi Dispensary Marula Matungu Lunganyiro Dispensary Lunganyiro 1. CHMT: County director and team members. 2. DHMT: District director and team members. 3. Facilities: Officer in charge and records officer. 4. Community unit: Community health extension worker, member/s of community health committee, community health workers. 5. District and Area Education officers. 6. CSOs and LIPs: Director and program staff members PATH Kenya Administration 1. Country Director (PATH) 2. Finance Administration Director, PATH Kenya APHIAPLUS Administration/Management 3. Chief of Party APHIAPLUS Western 4. Deputy Director Western 5. Deputy Director Nyanza 6. Sr. Technical Advisor 7. Sr. Technical Advisor 8. Technical Advisor 9. Technical Advisor 10. Finance Administration Manager, APHIAplus 11. County Coordinator 12. County Coordinator 13. County Coordinator 14. County Coordinator 15. Team Leader: Service Delivery 16. Team Leader: M&E 17. Sr. Technical Advisor 18. Technical Advisor 19. PPP Team Leader 20. Team Leader: Social Determinant of Health 21. Team Leader: Community Prevention REPORT Technical report - APHIAplus Western Mid-term Review report 53 Annex 6. Evaluation Tools Tool Summary 1. Area Education Officer/District Education Officer Discussion Guide 2. County/District Health Management Team Discussion Guide 3. Community Unit Discussion Guide 4. Health Facility Discussion Guide 5. CSO/LIP Discussion Guide 6. Staff /Management Discussion Guide REPORT Technical report - APHIAplus Western Mid-term Review report 54 APHIAplus Mid-Term Review Discussion Guide: DEO/AEO Name of Organization or Group: County: Sub-County/District: Date of interview: Interviewer: Note taker: Observer: Target group: DEO AEO Introduction Good morning/afternoon. My name is ____________ and I work with APHIAplus. (All introduce themselves). As APHIAplus we are assessing our activities in _____county and in doing this we are talking to people in Organizations and Institutions that we collaborate with. The purpose of this is to find out how the Project activities are being implemented. This will assist us to make any necessary improvements/changes in the Project. Your contribution is going to be very helpful. There are no right or wrong answers and you are encouraged to participate and give your views. This discussion will take about one hour. Section 1: APHIAPLUS support to strengthen school-based activities (LSE, WASH, School Construction, Vitamin A, Deworming, School Fees etc.) 1.1 Thinking about your past year, what are you/your team most proud of? What would you each say are your top 2-3 accomplishments? 1.2 What kind of support does APHIAplus provide? Probe: In what ways does APHIAplus provide support to help you plan and execute school-based activities? Probe: What problems did you address with this support? Probe: Have you received any orientation, training, tools, or supportive supervision that have helped identify gaps, diagnose problems, and find solutions related to school based activities? Probe: Have you received any other materials from APHIAplus? 1.3 In your opinion, how useful is the APHIAplus support provided? Probe: What has been relevant or helpful? Provide one example. Explain. Probe: What needs has this support fulfilled? 1.4 Overall, is APHIAplus support provided in the expected timeframe? Probe: Can you give an example of APHIAplus support that was given in a timely manner? Probe: Can you give one or two suggestions on ways that APHIAplus can improve in this area? 1.5 Describe your current working relationship with APHIAplus. 1.6 How has APHIAplus supported you to address gender-related barriers that may prevent people, such as girls, from benefitting from school-based activities? Probe: What kinds of support were you given? Were you given any planning support? REPORT Technical report - APHIAplus Western Mid-term Review report 55 Probe: Ask about gender mainstreaming activities, gender equity policies, gender sensitization training, gender-based violence, etc. Probe: Choose the most important activity. Why is it the most important? Tell me what has happened. Probe: Have you noticed any changes since the activity was implemented? What happened? Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? 1.7 How has APHIAplus helped you to make your program activities be more responsive to the needs of students? Probe: What kind of support have you received from APHIAplus? Probe: Choosing one example, tell me whether you have seen a change in the way APHIAplus supported school-based initiatives are being delivered to students? 1.8 What strategies are you using to ensure service referrals are made, as needed? (E.g. HTC, VMMC, HIV, STI, legal aid, rape care, food security, etc.) Probe: What support has APHIAPHIAplus provided you to ensure service referrals are made, as needed? Probe: What has been relevant or helpful? Probe: What would you change in your current approach? Probe: What additional support do you need from APHIAplus to address these challenges? 1.9 What are you doing to ensure a functioning service referral network in your schools? Probe: To what extent have you been able to refer students? Probe: Has APHIAPLUS provided support to enable you to strengthen the existing service referral network?  Probe: What support has been especially relevant/helpful? Probe: What have you achieved in this area? Probe: What are the challenges? Probe: What specific support do you need from APHIAplus to address these challenges? 1.10 Has APHIAplus supported the Reaching Adolescents Via Health Education (RAVE) intervention in any of your schools? Probe: What have been your observations? Probe: Have you observed any implementation challenges? Probe: Have any solutions been provided to the identified challenges? Explain. Section 2: APHIAPLUS support to increase demand for services 2.1 In what ways are you increasing demand/service uptake in your schools? Probe: How has APHIAplus assisted you to increase uptake of health, social, legal and economic services to students? Probe: What target populations are not accessing services (e.g. health, social, legal, economic)? Why not? Provide an example. REPORT Technical report - APHIAplus Western Mid-term Review report 56 Probe: In what ways are you increasing demand/service uptake among vulnerable and most at￾risk populations: o OVCs? o Girls? Probe: In what ways is APHIAPHIAplus helping you to support these efforts with: ___? o OVCs? o Girls? Probe: What else can APHIAplus do? Section 3: APHIAPLUS support to ensure Integration 3.1 How are you integrating social, economic, and health activities into your routine school programming? Probe: Has APHIAPLUS provided any specific support? Probe: What support was given? What has been relevant or helpful? Probe: What challenges are you facing? What would you change? Probe: What specific support do you need from APHIAplus to address these challenges? Section 4: APHIAPLUS support to improve data quality and data utilization 4.1 Describe the quality of monthly data that cover APHIAplus supported activities (LSE, WASH, school construction, vitamin A, deworming, etc). Probe: Do you have all the tools you need? Probe: How are you using the data? Probe: Describe how data use has impacted your program. Probe: How satisfied are you with the current way in which data are collected and utilized? 4.2 Has APHIAPLUS provided any support aimed at improving data collection, management, and utilization? Probe: What support has been especially relevant or helpful? Probe: What achievements have you made in this area? Probe: What challenges have you encountered? Probe: What specific support do you need from APHIAplus to address these challenges? This concludes our discussion today. Is there anything else you would like to comment on or add regarding your partnership with the APHIAplus? Comments? Achievements? Recommendations? THANK YOUR RESPONDENT FOR THE TIME AND HELP PROVIDED AND PROCEED TO THE NEXT DATA COLLECTION SITE. REPORT Technical report - APHIAplus Western Mid-term Review report 57 APHIAplus Mid-Term Review Discussion Guide: CHMT/DHMT Name of CHMT/DHMT: County: Sub-County/District: Date of interview: Interviewer: Note taker: Observer: Target group: CHMT DHMT Introduction Good morning/afternoon. My name is ____________ and I work with APHIAplus. (All introduce themselves). As APHIAplus we are assessing our activities in _____county and in doing this we are talking to people in Organizations and Institutions that we collaborate with. The purpose of this is to find out how the Project activities are being implemented. This will assist us to make any necessary improvements/changes in the Project. Your contribution is going to be very helpful. There are no right or wrong answers and everyone is encouraged to participate and give their views even if that view is different from that of the other participants in the group. This discussion will take about one hour. Section 1: APHIAPLUS support to CHMT/DHMT to strengthen service delivery 1.1 Thinking about your past year, what are you and your team most proud of? What would you each say are your top 2-3 accomplishments? 1.2 In what ways has APHIAplus provided you support to help plan and execute your work to strengthen service delivery? Probe: Can you give an example? Probe: What was the support? What problems did you address with this support? Probe: Give an example of areas /support given to strengthen service delivery? Probe: What have you received in the way of specific orientation, training, tools, or supervision support on how to identify gaps, diagnose problems, and find solutions? Probe: What have you received in the way of other supplies, materials or equipment to strengthen service delivery? 1.3 In your opinion, how useful is the APHIAplus support aimed at strengthening the services you provide? Probe: What needs has this support fulfilled? Probe: What other kinds of support would you recommend from APHIAplus? Explain. 1.4. Overall, is APHIAplus support provided to you in a timely manner? Probe: Can you give an example of APHIAplus support that was given in a timely manner? Probe: Can you give one or two suggestions on ways that APHIAplus can improve on timely support? 1.5 Describe your current working relationship with APHIAplus. 1.6 How has APHIAplus supported you to address gender-related barriers that may prevent people, such as women and children, from receiving services you provide?  Probe: Which areas of support were you given? REPORT Technical report - APHIAplus Western Mid-term Review report 58  Probe: Ask about gender mainstreaming activities, gender equity workplace policies, gender sensitization training, gender-based and domestic violence, legal aid, etc.  Probe: Choose the most important activity. Why is it the most important? Tell me what has happened.  Probe: Have you noticed any changes since the activity was implemented? What happened? 1.7 How has APHIAplus helped you to make your services be more responsive to the needs of the youth?  Probe: What kind of support have you received?  Probe: Choosing one example, tell me whether you have seen a change in the way services are being delivered to youth? 1.8 Describe any support you have received from APHIAplus to roll out the Community Strategy?  Probe: Have you received any support from APHIAplus to better help you plan & manage Community Strategy activities?  Probe: How about support to better serve households in your catchment area? (provide at least one example)  Probe: What support has been especially relevant or helpful? Explain?  Probe: Any specific support you need from APHIAplus that you are not currently getting? 1.9 Describe any support you have received from APHIAplus that enables you to deliver other community prevention interventions more effectively.  Probe: Have you received any support from APHIAplus to better help you plan &manage other community prevention activities?  Probe: How about support to better serve vulnerable and at-risk populations in your catchment area? (provide at least one example)  Probe: What support has been especially relevant or helpful? Explain?  Probe: Any specific support you need from APHIAplus that you are not currently getting? Section 2: APHIAPLUS support to increase demand for services 2.1 In what ways are you increasing demand/service uptake in your county/district? Probe: What target populations are not accessing services? Why not? Provide an example. Probe: In what ways are you increasing demand/service uptake among: o OVC o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) Probe: In what ways is APHIAplus helping you to support these efforts with: ___? o OVC o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) REPORT Technical report - APHIAplus Western Mid-term Review report 59 Probe: What else can APHIAplus do? Section 3: APHIAPLUS support to ensure Integration 3.1 What are you doing to support service delivery integration? (e.g. TB/HIV, ARV, PMTCT, FP, MCHN, etc.) Probe: Support in:  Co-locating services?  Cross-site referrals?  Public private partnerships? Probe: How do you:  Plan and organize for service delivery integration?  Manage procurement for service delivery integration?  Organize supervision for service delivery integration?  Organize training for service delivery integration?  Support the community/facility integration? 3.2 How has APHIAplus provided support to help you manage integration of service delivery? Probe: What support was given? What has been relevant or helpful? Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? Support to reduce gender-related barriers 3.3 Are you implementing any activities in your county/district aimed to reduce gender-related barriers that may undermine access to health services?  Probe: Has APHIAPLUS provided any support to help you plan and manage any gender mainstreaming activities in your county/district? What support was given? What has been relevant or helpful?  Probe: Provide an example of a gender mainstreaming activity that APHIAplus has supported in your program.  Probe: What have been some achievements in this area?  Probe: What challenges have you faced? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? Youth-focused service provision 3.4 In what ways are you implementing youth-focused health programming?  Probe: Has APHIAPLUS provided any specific support (to you or your organization) to enable you to better implement quality youth-focused services?  Probe: What support was given? What has been relevant or helpful?  Probe: Provide an example of a youth-focused health programming activity that APHIAplus has supported in your program.  Probe: What have been some achievements in this area?  Probe: What challenges are you facing? What would you change?  Probe: What specific support do you need from APHIAplus to address these challenges? 3.5 In what ways has APHIAplus provided support to the DHMT to integrate community-based prevention efforts into the Community Strategy structure? REPORT Technical report - APHIAplus Western Mid-term Review report 60  Probe: Provide an example.  Probe: What have been some achievements in this area?  Probe: What challenges are you facing? What would you change?  Probe: What have been some achievements in this area? 3.6 What are you doing to ensure functioning service referral networks in your county/district? Probe: Has APHIAPLUS provided support to the management team to enable you strengthen the existing service referral networks? Briefly describe.  Probe: What support has been especially relevant/helpful?  Probe: What have you achieved in this area?  Probe: What are the challenges?  Probe: What specific support do you need from APHIAplus to address these challenges? Section 4: APHIAPLUS Support to improve data quality and data utilization (for the DHRIO) 4.1. How would you describe the quality of data that you are responsible for generating in your county/district on a monthly basis?  Probe: Describe what you do with the data collected? How is it utilized?  Probe: How satisfied are you with the current way in which data are collected, managed and utilized in your county/district?  Probe: What kind of support has APHIAPLUS provided aimed at improving data quality? How about data utilization?  Probe: What support has been especially relevant or helpful?  Probe: What achievements have you made in this area?  Probe: What challenges have you encountered? What specific support do you need from APHIAplus to address these challenges? This concludes our discussion today. Is there anything else you would like to comment on or add regarding your partnership with the APHIAplus? Comments? Achievements? Recommendations? THANK YOUR RESPONDENT FOR THE TIME AND HELP PROVIDED AND PROCEED TO THE NEXT DATA COLLECTION SITE. REPORT Technical report - APHIAplus Western Mid-term Review report 61 APHIAplus Mid-Term Review Discussion Guide: LIP/CSO Name of LIP/CSO: County: Sub-County/District: Date of interview: Interviewer: Note taker: Observer: Target group: LIP/CSO Introduction Good morning/afternoon. My name is ____________ and I work with APHIAplus. (All introduce themselves). As APHIAplus we are assessing our activities in _____county and in doing this we are talking to people in Organizations and Institutions that we collaborate with. The purpose of this is to find out how the Project activities are being implemented. This will assist us to make any necessary improvements/changes in the Project. Your contribution is going to be very helpful. There are no right or wrong answers and everyone is encouraged to participate and give their views even if that view is different from that of the other participants in the group. This discussion will take about one hour. Section 1: APHIAPLUS support to strengthen delivery of community-based services 1.3 Thinking about your past year, what are you/your team most proud of? What would you each say are your top 2-3 accomplishments? 1.4 In what ways does APHIAplus provide support to help you plan and execute your program delivery?  Probe: Give an example of support given.  Probe: What were the problems the support addressed?  Probe: Give an example of support given to improve the quality of service delivery.  Probe: Have you received any orientation, training, tools or supervision support on how to identify gaps, diagnose problems, and find solutions related to service delivery?  Probe: Have you received any other inputs, materials, equipment or other resources from APHIAplus? 1.3 In your opinion, how useful is the APHIAplus support aimed at strengthening the services/interventions you deliver?  Probe: What has been relevant or helpful? Provide one example. Explain.  Probe: What needs has this support fulfilled? 1.4 Overall, is APHIAplus support provided in the expected timeframe?  Probe: Can you give an example of APHIAplus support that was given in a timely manner?  Probe: Can you give one or two suggestions on ways that APHIAplus can improve in this area? 1.5 Describe your current working relationship with APHIAplus. 1.6 How has APHIAplus supported you to address gender-related barriers that may prevent people, such as women and children, from receiving services you provide or the interventions you deliver?  Probe: What kinds of support were you given? Were you given any planning support? REPORT Technical report - APHIAplus Western Mid-term Review report 62  Probe: Ask about control of domestic resources, decision-making, gender-based violence, sexual harassment, access to legal support, male involvement in reproductive health issues, etc.  Probe: Choose the most important activity. Why is it the most important? Tell me what has happened.  Probe: Have you noticed any changes since the activity was implemented? What happened?  Probe: What challenges are you facing? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? 1.7 How has APHIAplus helped you to make your program activities be more responsive to the needs of the youth? Probe: What kind of support have you received from APHIAplus? Probe: Choosing one example, tell me whether you have seen a change in the way services are delivered to youth? Probe for tactics used to engage youth by APHIAplus:  Poverty and education  Access to credit  Gender issues  Sexual and reproductive health education  Social interactions by youth  Financial planning 1.8 What strategies are you using to ensure service referral completion? (E.g. HTC, VMMC, TB/HIV, STI, legal aid, rape care, food security, etc.)  Probe: What support has APHIAPHIAplus provided you to ensure referral completion?  Probe: What has been relevant or helpful?  Probe: What would you change in your current approach?  Probe: What additional support do you need from APHIAplus to address these challenges? Provide an example. 1.9 As a group that makes referrals, what are you doing to ensure a functioning service referral network in your catchment areas?  Probe: To what extent have you been able to refer your target population?  Probe: Has APHIAPLUS provided support to enable you strengthen the existing service referral network?  Probe: What support has been especially relevant/helpful?  Probe: What have you achieved in this area?  Have you developed a service directory, referral forms, etc?  Probe: What are the challenges?  Probe: What specific support do you need from APHIAplus to address these challenges? 1.10 Have you been involved with any of the following APHIAplus supported interventions? HIV reminder system for HIV clients on antiretroviral treatment. REPORT Technical report - APHIAplus Western Mid-term Review report 63 Condolympics RAVE in Secondary Schools Community action sets  Probe: For each mentioned above what have been your observations?  Probe: Have you observed any implementation challenges?  Probe: Have any solutions been provided to the identified challenges? Explain. Question 1.11 is for OVC Community Based Organizations only 1.11 Describe any support that you have received from APHIAplus to better enable you to serve OVCs and their households more effectively?  Probe: What support has been relevant? What is not? Explain?  Probe: What bottlenecks or challenges are keeping you from reaching targets? Choose one challenge:  Where did it occur?  When did this most recently happen?  What did you do to resolve the challenge?  What support did you seek?  What kind of solution did you implement?  Did APHIAplus play any role is addressing this challenge? How?  Probe: Given the challenges mentioned above, what specific support do you need from APHIAplus? Question 1.12 is for LIPs targeting fisher folk, youth out of school, youth in higher learning institutions, MSM, sex workers and workplace populations 1.12Describe any support that you have received from APHIAplus that enables you to deliver prevention interventions more effectively?  Probe: What support has been relevant?  Probe: What is not?  Probe: How about support to better serve target populations in your catchment area?  Probe: What bottlenecks or challenges keep you from reaching targets?  Choose the most important challenge: o When did this most recently happen? o What did you do to resolve the challenge? o What support did you seek? o What kind of solution did you implement? o Did APHIAplus play any role in addressing this challenge? Explain?  Probe: Given the challenges mentioned above, what specific support do you need from APHIAplus? Section 2: APHIAPLUS support to increase demand for services 2.1 In what ways are you increasing demand/service uptake in your catchment areas?  Probe: How has APHIAplus assisted you to increase demand and service uptake? REPORT Technical report - APHIAplus Western Mid-term Review report 64  Probe: What target populations are not accessing services (e.g. health, social, legal, financial)? Why not? Provide an example.  Probe: In what ways are you increasing demand/service uptake among vulnerable and most at-risk populations: o OVC? o Women and girls? o Youth in and out of school aged 15-24? o Fishing communities? o Workplace populations? o Sex workers? o MSM?  Probe: In what ways is APHIAPHIAplus helping you to support these efforts with: ___? o OVC? o Women and girls? o Youth in and out of school aged 15-24? o Fishing communities? o Workplace populations? o Sex workers? o MSM?  Probe: What else can APHIAplus do? Section 3: APHIAPLUS support to deliver an integrated package of services at the community level 3.1 How are you promoting integrated service provision? (e.g Combination prevention, legal aid, micro financing, clinical service referrals, WASH, nutrition, food security, HIV prevention, family planning, malaria, child immunization, etc.)  Probe: Has APHIAPLUS provided support to help you implement activities across service delivery areas? What support was given? What has been relevant or helpful? Provide specific examples.  Probe: What challenges are you facing? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? 3.2 How has APHIAplus provided support to help you integrate service delivery?  Probe: What support was given? What has been relevant or helpful?  Probe: What challenges are you facing? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? 3.3 In what ways are you implementing youth-focused service provision?  Probe: Has APHIAPLUS provided any support to better implement quality youth-focused services?  Probe: What support was given? What has been relevant or helpful?  Probe: Provide an example of a youth-focused service that APHIAplus has supported in your program. REPORT Technical report - APHIAplus Western Mid-term Review report 65  Probe: What challenges are you facing in this area? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? Section 4: APHIAPLUS support to improve data quality and data utilization 4.1 Describe the quality of data that you are responsible for generating on a monthly basis?  Probe: Do you have all the tools you need?  Probe: How are you using the data?  Probe: How is it utilized? Describe how data use has impacted your program.  Probe: How satisfied are you with the current way in which data are collected and utilized? 4.2 Has APHIAPLUS provided any support aimed at improving data collection, management, and utilization?  Probe: What support has been especially relevant or helpful?  Probe: What achievements have you made in this area?  Probe: What challenges have you encountered?  Probe: What specific support do you need from APHIAplus to address these challenges? This concludes our discussion today. Is there anything else you would like to comment on or add regarding your partnership with the APHIAplus? Comments? Achievements? Recommendations? THANK YOUR RESPONDENT FOR THE TIME AND HELP PROVIDED AND PROCEED TO THE NEXT DATA COLLECTION SITE. REPORT Technical report - APHIAplus Western Mid-term Review report 66 APHIAplus Mid-Term Review Discussion Guide: Community Unit Name of Community Unit: County: Sub-County/District: Date of interview: Interviewer: Note taker: Observer: Target group: Community Unit Introduction Good morning/afternoon. My name is ____________ and I work with APHIAplus. (All introduce themselves). As APHIAplus we are assessing our activities in _____county and in doing this we are talking to people in Organizations and Institutions that we collaborate with. The purpose of this is to find out how the Project activities are being implemented. This will assist us to make any necessary improvements/changes in the Project. Your contribution is going to be very helpful. There are no right or wrong answers and everyone is encouraged to participate and give their views even if that view is different from that of the other participants in the group. This discussion will take about one hour. Section 1: APHIAPLUS support to strengthen Level 1 service provision (The Community Strategy) 1.5 Thinking about your past year, what are you/your team most proud of? What would you each say are your top 2-3 accomplishments? 1.6 In what ways does APHIAplus provide support to help you plan and execute your community strategy work? Probe: Give an example of support given to support the Community Unit. Probe: What were the problems the support addressed? Probe: Give an example of support given specifically to strengthen household level service provision. Probe: What have you received in the way of orientation, training, tools or supervision support on how to identify gaps, diagnose problems, and find solutions? Probe: What have you received in the way of other supplies, equipment or materials from APHIAplus? 1.3 In your opinion, how useful is the APHIAplus support aimed at strengthening the services you provide? Probe: What has been relevant or helpful? Provide one example. Explain. Probe: What needs has this support fulfilled? 1.5 Overall, is APHIAplus support provided in the expected timeframe? Probe: Can you give an example of APHIAplus support that was given in a timely manner? Probe: Can you give one or two suggestions on ways that APHIAplus can improve in this area? 1.5 Describe your current working relationship with APHIAplus. 1.6 How has APHIAplus supported you to address gender-related barriers that may prevent people, such as women and children, from receiving services you provide? REPORT Technical report - APHIAplus Western Mid-term Review report 67 Probe: What kinds of support were you given? Were you given any planning support? Probe: Ask about gender mainstreaming activities, gender equity workplace policies, gender sensitization training, sexual and domestic violence, legal aid, male involvement in reproductive health, etc. Probe: Choose the most important activity. Why is it the most important? Tell me what has happened. Probe: Have you noticed any changes since the activity was implemented? What happened? Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? 1.7 How has APHIAplus helped you to make household level service provision be more responsive to the needs of the youth? Probe: What kind of support have you received from APHIAplus? Probe: Choosing one example, tell me whether you have seen a change in the way services are delivered to youth? 1.9 What strategies are you using to ensure referral completion? (E.g. ANC, HTC, PMTCT, VMMC, C&T, TB/HIV ) Probe: What support has APHIAPHIAplus provided you to ensure referral completion? Defaulter tracing related to TB and C&T? Probe: What has been relevant or helpful? Probe: What would you change in your current approach? Probe: What additional support do you need from APHIAplus to address these challenges? Provide an example. 1.9 What are you doing to ensure a functioning service referral network in your Community Unit? Probe: Has APHIAPLUS provided support to enable you strengthen the existing service referral network? Briefly describe.  Probe: What support has been especially relevant/helpful? Probe for specific details. Probe: What have you achieved in this area? Probe: What are the challenges? Probe: What specific support do you need from APHIAplus to address these challenges? Please be specific. 1.10 What support have you received from APHIAplus to support the roll out of the Community Strategy? Probe: Have you received any support from APHIAplus to better help you administer Community Strategy activities? Probe: How about support to better serve households in your catchment area? Probe: What support has been especially relevant or helpful? Explain? Probe: Any specific administrative support you need from APHIAplus that you are not currently getting? Section 2: APHIAPLUS support to increase demand for services in communities 2.1 In what ways are you increasing demand/service uptake in your Community Unit? REPORT Technical report - APHIAplus Western Mid-term Review report 68 Probe: What target populations are not accessing services? Why not? Provide an example. Probe: In what ways are you increasing demand/service uptake among: o OVC o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) Probe: In what ways is APHIAPHIAplus helping you to support these efforts with: ___? o OVC o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) Probe: What else can APHIAplus do? Section 3: APHIAPLUS support to integrate services at the community level 3.1 How are you integrating Level 1 services in your Community Unit? (e.g. Clinical service referrals, WASH, nutrition, food security, HIV prevention, malaria, child immunization, ANC, etc.) Probe: Has APHIAPLUS provided support to help you implement activities across service delivery areas? What support was given? What has been relevant or helpful? Provide specific examples. Probe: Provide an example of how are you integrating. Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? Probe: Provide an example. 3.2 How has APHIAplus provided support to help you integrate service delivery? Probe: What support was given? What has been relevant or helpful? Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? 3.3 In what ways are you implementing youth-focused service provision? Probe: Has APHIAPLUS provided any support to your facility to enable you to better implement quality youth-focused services? Probe: What support was given? What has been relevant or helpful? Probe: Provide an example of a youth-focused service that APHIAplus has supported in your facility. Probe: What challenges are you facing in this area? What would you change? Probe: What support do you need from APHIAplus to address these challenges? 3.4 In what ways has APHIAplus provided support to integrate community prevention efforts into the Community Strategy structure? Probe: Provide an example Probe: What have been some achievements in this area? Probe: What are the challenges? Probe: What specific support do you need from APHIAplus to address these challenges? REPORT Technical report - APHIAplus Western Mid-term Review report 69 Section 4: APHIAPLUS support to improve the community-based health information system 4.1. How would you describe the quality of data your Community Unit generates on a monthly basis? Probe: Describe what you do with the data collected? How is it utilized by the Community Unit? Probe: How satisfied are you with the current way in which data are collected? Utilized by the Community Unit? 4.2 Has APHIAPLUS provided any support aimed at improving the community-based health information system? Probe: How about data quality, utilization? Probe: What support has been especially relevant or helpful? Probe: What achievements have you made in this area? Probe: What challenges have you encountered? What specific support do you need from APHIAplus to address these challenges? This concludes our discussion today. Is there anything else you would like to comment on or add regarding your partnership with the APHIAplus? Comments? Achievements? Recommendations? THANK YOUR RESPONDENT FOR THE TIME AND HELP PROVIDED AND PROCEED TO THE NEXT DATA COLLECTION SITE. REPORT Technical report - APHIAplus Western Mid-term Review report 70 APHIAplus Mid-Term Review Discussion Guide: Facilities Name of Facility: County: Sub-County/District: Date of interview: Interviewer: Note taker: Observer: Target group: Facility Introduction Good morning/afternoon. My name is ____________ and I work with APHIAplus. (All introduce themselves). As APHIAplus we are assessing our activities in _____county and in doing this we are talking to people in organizations and institutions that we collaborate with. The purpose of this is to find out how the project activities are being implemented. This will assist us to make any necessary improvements/changes in the project. Your contribution is going to be very helpful. There are no right or wrong answers and everyone is encouraged to participate and give their views even if that view is different from that of the other participants in the group. This discussion will take about one hour. Section 1: APHIAPLUS support to facilities to strengthen service delivery 1.7 Thinking about your past year, what are you and your team most proud of? What would you each say are your top 2-3 accomplishments? 1.8 In what ways does APHIAplus provide support to help you plan and execute your work? Probe: Can you give an example of support you received? Probe: What were the problems the support addressed? Probe: Can you give an example of support given specifically to strengthen service delivery? Probe: What have you received in the way of special orientation, training, tools, or supervision support on how to identify gaps, diagnose problems, and find solutions? Probe: What have you received in the way of other supplies, materials or equipment to strengthen service delivery? 1.3 In your opinion, how useful is the APHIAplus support aimed at strengthening the services you provide? Probe: What has been relevant or helpful? Provide one example. Explain. Probe: What needs has this support fulfilled? 1.6 Overall, is APHIAplus support provided in the expected timeframe? Probe: Can you give an example of APHIAplus support that was given in a timely manner? Probe: Can you give one or two suggestions on ways that APHIAplus can improve in this area? 1.5 Describe your current working relationship with APHIAplus. 1.6 How has APHIAplus supported you to address gender-related barriers that may prevent people, such as women and children, from receiving services you provide?  Probe: What kinds of support were you given? o Were you given any planning support? REPORT Technical report - APHIAplus Western Mid-term Review report 71  Probe: Ask about gender mainstreaming activities, gender equity workplace policies, gender sensitization training, sexual and domestic violence, legal aid, male involvement in reproductive health, etc.  Probe: Choose the most important activity. Why is it the most important? Tell me what has happened.  Probe: Have you noticed any changes since the activity was implemented? What happened?  Probe: What challenges are you facing? What would you change?  Probe: What support do you need from APHIAplus to address these challenges? 1.7 How has APHIAplus helped you to make your services be more responsive to the needs of the youth?  Probe: What kind of support have you received?  Probe: Choosing one example, tell me whether you have seen a change in the way services are being delivered to youth? Questions 1.8 and 1.9 are for C&T/PMTCT/ TB service delivery areas only 1.8 What strategies are you employing to prevent?  Care and Treatment client loss to follow up defaulters?  PMTCT client loss to follow up defaulters?  TB client loss to follow up defaulters? Probe: What would you change in your current approach? 1.9 In what ways has APHIAplus provided support to prevent loss to follow up? Probe: How? Provide at least one example. Probe: Have you seen any changes as a result of this support? Provide at least one example. Probe: What specific support do you need from APHIAplus to address these challenges? Probe for specific examples. 1.10 Have you been involved with any of the following APHIAplus supported intervention? HIV reminder system for HIV clients on antiretroviral treatment. TheraSim® model for coaching and mentoring. Uliza! Clinician’s HIV Hotline for real-time clinical consultations. Performance-based financing.  Probe: For each mentioned above what have been your observations?  Probe: Have you observed any implementation challenges?  Probe: Have any solutions been provided to the identified challenges? Explain. Section 2: APHIAPLUS support to increase demand for services 2.1 In what ways are you increasing demand/service uptake in your facility? Probe: What target populations are not accessing services? Why not? Provide an example. Probe: In what ways are you increasing demand/service uptake among: o OVC REPORT Technical report - APHIAplus Western Mid-term Review report 72 o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) Probe: In what ways is APHIAPHIAplus helping you to support these efforts with: ___? o OVC o Women and girls? o Youth aged 15-24? o Other most at risk populations (e.g. HIV positive orphans, widows) Probe: What else can APHIAplus do? Section 3: APHIAPLUS support to ensure Integration 3.1 What are you doing to integrate service delivery? (e.g. TB/HIV, C&T, PMTCT, FP, MCHN, etc.) Probe: What are you doing to:  Co-locating services?  Strengthen cross-site referral system?  Public private partnerships? Probe: How do you:  Plan and organize for service delivery integration?  Manage procurement for service delivery integration?  Organize supervision for service delivery integration?  Organize training for service delivery integration?  Support the community/facility integration? Probe: How do you ensure the quality of integrated services? 3.2 How has APHIAplus provided support to help you integrate service delivery? Probe: What support was given? What has been relevant or helpful? Probe: What challenges are you facing? What would you change? Probe: What support do you need from APHIAplus to address these challenges? 3.3 In what ways are you implementing youth-focused service provision? Probe: Has APHIAPLUS provided any support to your facility to enable you to better implement quality youth-focused services? Probe: What support was given? What has been relevant or helpful? Probe: Provide an example of a youth-focused service that APHIAplus has supported in your facility. Probe: What challenges are you facing in this area? What would you change? Probe: What support do you need from APHIAplus to address these challenges? Section 4: APHIAPLUS support to improve data quality and data utilization 4.1. How would you describe the quality of data that you generate at your facility on a monthly basis?  Probe: Describe what you do with the data collected? How is it utilized by facility staff?  Probe: How satisfied are you with the current way in which data are collected? Utilized by facility staff? REPORT Technical report - APHIAplus Western Mid-term Review report 73  Probe: Has APHIAPLUS provided any support aimed at improving data quality? How about data utilization?  Probe: What support has been especially relevant or helpful?  Probe: What achievements have you made in this area?  Probe: What challenges have you encountered?  Probe: Do you have all the resources you need to ensure timely, complete, and accurate monthly data reporting?  What specific support do you need from APHIAplus to address these challenges? This concludes our discussion today. Is there anything else you would like to comment on or add regarding your partnership with the APHIAplus? Comments? Achievements? Recommendations? THANK YOUR RESPONDENT FOR THE TIME AND HELP PROVIDED AND PROCEED TO THE NEXT DATA COLLECTION SITE. REPORT Technical report - APHIAplus Western Mid-term Review report 74 Management Discussion Guide Name Title 1. Please describe what you see as the critical technical support that you/your team provides to MoH or your technical counterparts? a. If there was no funding through APHIA how do you think service delivery would be different? Where would your technical counterparts go for support? Probe: Identify types of work that they do (capacity building, lab, cash support, procurement of commodity, supervision) and probe them. Ask about implementation challenges of their specific work areas under 2. 2. Please describe a (limited number e.g. top 3 or 5) internal challenges that you experience implementing your day to day APHIAPHIAplus support? 2a. Choosing one of the challenges: Can you give me an example of [a specific challenge]? Probe - When did this most recently happen? Probe - How frequently do you face this challenge? 2b. What did you do to resolve the challenge? Probe - Who did you go to? Rely on? Probe - What support did you seek? Probe – What kind of solutions did you implement? Probe - Do you think this will happen again? 3. To scale up the support in your area faster and reach more facilities what supports does your team need to do more of or differently? 3a. What can APHIAplus management do more of or do differently to support you? 3b. What skills or expertise do you need to shore up or build on your own team? 4. When you think about other implementers or models of implementation who (other organizations or agencies) do you feel is scaling up support well in your technical area? Who do you look to as a model? 5. Please describe what you see as the critical technical support that you provides to the Aphiaplus senior management team? Annex 7. Team Members PATH and APHIAplus Western staff REPORT Technical report - APHIAplus Western Mid-term Review report 75 Team member Designation Organisation Abel Alwanga TA HIV/AIDS care and treatment EGPAF/APHIAplus Alfred Ocholla Technical advisor/child survival/malaria PATH Kenya/APHIAplus Allison Bingham Sr. Research Scientist/Program Evaluation PATH Kenya/APHIAplus Clement Oindo M&E Advisor PATH Kenya/APHIAplus Edward Kariithi Service Delivery Advisor EGPAF/APHIAplus Eugenia Kingori M&E Officer PATH Kenya/APHIAplus Jeff Bernson Global M&E Director PATH Global Program Kirogo Mwangi Regional M&E Advisor PATH Global program KJ Zunigha Global M&E Team PATH Global program Masibo Wamalwa BCC advisor PATH Kenya/APHIAplus Oby Odhyambo Sr. Health Communications Specialist PATH Kenya Patrick Ochieng TA Health Systems Strengthening EGPAF/APHIAplus Peninah Murunga Program Associate PATH Kenya Rose Adera Community Advisor PATH Kenya/APHIAplus Tom Omurwa Sr. Strategic Information Specialist PATH Kenya Consultants Agnes Muloli Antonina Mutoro Douglas Waswa Dricillah Mwakideu George Ngollo Grace Magomere Jane Kweyu Ken Anjejo Lydia Mauko Mathews Ogutu Patrick Masitsa Secret Siku Stella Bii REPORT Technical report - APHIAplus Western Mid-term Review report 76 Stella Mapesa Yvonne Wangũi Machira (Sr. Research Consultant, Global Team) REPORT Technical report - APHIAplus Western Mid-term Review report 77 Annex 8. Thematic Framework This has been enclosed separately. REPORT Technical report - APHIAplus Western Mid-term Review report 78 Annex 9. Documents Consulted 1. USAID: USAID APHIAplus Results Framework – Year 2 Expanded Version. USAID; 2011. 2. USAID: Co-operative Agreement No. AID-623-A-11-00002. USAID | East Africa; 2010. 3. USAID: USAID/Kenya Five Year Implementation Framework for the Health Sector (2010 - 2015) USAID; 2010. 4. USAID/Kenya: RFA No. 623-10-000009 Technical Application, PATH. 2010. 5. PATH: APHIAplus Health Service Delivery Project Zone 1: Western and Nyanza Provinces 6. PATH: APHIAplus Zone 1 Kenya Project, Quarter 1 Report, PATH Kenya, 2011. 7. PATH: APHIAplus Zone 1 Kenya Project, Quarter 2 Report, PATH Kenya, 2011. 8. PATH: APHIAplus Zone 1 Kenya Project, Quarter 3 Report, PATH Kenya, 2011. 9. PATH: APHIAplus Zone 1 Kenya Project, Quarter 4 Report, PATH Kenya, 2011. 10. PATH: APHIAplus Zone 1 Kenya Project, Quarter 1 Report, PATH Kenya, 2012. 11. PATH: APHIAplus Zone 1 Kenya Project, Quarter 2 Report, PATH Kenya, 2012. 12. PATH: APHIAplus Zone 1 Kenya Project, Quarter 3 Report, PATH Kenya, 2012. 13. PATH: APHIAplus Zone 1 Kenya Project, Quarter 4 Report, PATH Kenya, 2012. 14. PATH: APHIAplus Zone 1 Kenya Project, Quarter 1 Report, PATH Kenya, 2013. 15. PATH: APHIAplus Zone 1 Kenya Project, Quarter 2 Report, PATH Kenya, 2013. 16. PATH: Quality of Care Assessment of Reproductive Health, PMTCT, and ART Services: Results of a Health Facility Survey, APHIAplus Zone 1 Kenya Project, PATH Kenya, 2012. 17. PATH: Results of the Community Strategy and Schools-based Needs Assessment, APHIAplus Zone 1 Kenya Project, PATH, Kenya, 2012. 18. PATH: Results of the Health Facility Baseline Assessment, APHIAplus Zone 1 Kenya Project, PATH, Kenya, 2012. 19. PATH: APHIAplus Zone 1 Kenya Project, Y2 Workplan, PATH Kenya, 2011. 20. PATH: APHIAplus Zone 1 Kenya Project, Y3 Workplan, PATH Kenya, 2012. 21. PEPFAR: The Next Generation Indicators Reference Guide, Version 1.2. The President’s Emergency Plan for AIDS Relief. 2013.