i END-OF-PROJECT EVALUATION POPULATION SERVICES INTERNATIONAL APHIAPLUS HEALTH COMMUNICATIONS AND MARKETING April 27, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by International Business & Technical Consultants Inc. i Award No: AID-615-TO-14-00010 Prepared for Trisha Savage United States Agency for International Development/Kenya C/O American Embassy United Nations Avenue, Gigiri P.O. Box 629, Village Market 00621 Nairobi, Kenya Prepared by International Business & Technical Consultants Inc. 8618 Westwood Center Drive Suite 400 Vienna, Virginia, 22182 USA ii ACKNOWLEDGEMENTS The evaluation team would like to thank Paul Mwai for his supervision, Cyndi Scarlett for her senior management and Rosemary Were for her excellent logistical support. Many thanks also go to Emma Mwamburi of the USAID/Kenya’s Office of Health, Population and Nutrition. Finally, the team would like to acknowledge the cooperation of all those interviewed, from the diverse array of partners to the Population Services Kenya (PSK) staff under the leadership of PSK Chief Executive Officer Anthony Okoth. Your time, insights and organizational support are greatly appreciated. iii USAID KENYA END-OF-PROJECT EVALUATION OF APHIAPLUS HEALTH COMMUNICATION AND MARKETING PROGRAM DISCLAIMER: The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government ii ACRONYMS AIDS Acquired immune deficiency syndrome ANC Antenatal care ART Anti-retroviral therapy ARV Anti-retroviral BCC Behavior change communication CA Cooperative Agreement CBO Community-based organization CDC Centers for Disease Control CEO Chief executive officer CPR Contraceptive prevalence rate CS Child survival DCAH Department of Child and Adolescent Health DFID Department for International Development (of the United Kingdom) DHP Department of Health Promotion ET Evaluation team FHI 360 Family Health International FP Family planning GOK Government of Kenya GSN Gold Star Network HCM Health communication and marketing HIV Human immuno-deficiency virus HPAC Health promotion advisory committee HPO Health promotion officer HPU Health Promotion Unit HTC HIV testing and counseling IBC Interpersonal-based communication IBTCI International Business and Technical Consultant Inc. IMCI Integrated management of childhood illnesses IPC Interpersonal communication IR Intermediate result ITN Insecticide-treated net IUD Intra-uterine device JICA Japan International Cooperation Agency KAP Knowledge, Attitudes and Practice KDHS Kenya Demographic and Health Survey KII Key informant interview K-MET Kisumu Medical and Education Trust KNBTS Kenya National Blood Transfusion Service LVCT Liverpool Voluntary Care and Treatment LLIN Long-lasting insecticide-treated nets LNGO Local non-governmental organization iii LOP Life of project MAP Measuring Access and Performance M&E Monitoring and evaluation MIS Management information system MNCH Maternal, Newborn, and Child Health MOH Ministry of Health MOU Memorandum of understanding NASCOP National AIDS and Sexually Transmitted Infection Control Program NGO Non- governmental organization NHIF National Hospital Insurance Fund OC Oral contraceptive ORS Oral rehydration salts OVC Orphan and Vulnerable Children PERForM Performance Framework for Social Marketing PEPFAR President’s Emergency Plan for AIDS Relief PIP Performance Improvement Plan PMTCT Prevention of mother-to-child transmission PPP Public-private partnership PSI Population Services International PSK Populations Services Kenya QIP Quality improvement plan RGD Round-table group discussion RH Reproductive health SBCC Social behavior change communication SCOPE Strengthening Community Partnerships and Empowerment SDP Service delivery partner SF Social franchise SFN Social franchise network SM Social marketing/Socially marketed SO Strategic objective SOW Scope of work SRH Sexual and reproductive health SWAP Safe Water and AIDS Project TA Technical assistance TB Tuberculosis TBD To be determined TMA Total market approach TRaC Tracking Results Continuously UNICEF United Nations Children’s Fund USAID United States Agency for International Development USAID/K USAID Kenya USG United States Government VAT Value added tax VMMC Voluntary Medical Male Circumcision WRA Women of reproductive age WHO World Health Organization iv GLOSSARY Following the CA for HCM and the HCM M&E plan (2012), we used the following working definitions to address outcomes with respect to health impact, equity and subsidy among priority poor and vulnerable populations. Cost recovery: Recoupment of the total cost associated with a product against its selling price. It is also used to reflect the level of subsidy associated with each product. Equity: Equity was to be addressed by ensuring that HCM activities reached target audiences, namely poor and vulnerable Kenyans. Equity also refers to the ability of all segments of the Kenyan population to access health services and health products, especially poor and vulnerable populations. Accordingly, we looked at equity in terms of HCM’s activities to enable the poor and vulnerable Kenyans to access to health services and products. Femiplan brands: PSI brands for manufactured RH condoms, pills and injectables. Health impact: According to the HCM CA, health impact was to be measured by changes in behaviors related to the four program elements targeted by USAID and the GOK: HIV/AIDS, family planning and reproductive health, malaria, and treatment of diarrhea in children under five. Following this direction, we define health impact as changes in health behavior related to these four health program elements. Market share: The proportion of a market controlled by the PSK HCM product. In this context, it is the proportion of SM brands under the activity compared to free distribution and other commercial brands. Market value: The basket volume of sales realized from a particular product, as determined by a multiple of product price and quantity of sales. Poor and vulnerable populations: In the context of HCM, poor are those segments of Kenya’s population that cannot afford health products and services. For that reason, they are most at risk of contracting HIV/AIDS, becoming infected with malaria and suffering from reproductive health problems from bearing children because of lack of protection to prevent pregnancies. Social franchise facility: A facility that is enrolled in a social franchise network to provide specified services. Social franchise network (SFN): Social franchising is an approach to organizing private healthcare providers into networks that deliver specified health services under a common brand, with a promise of quality assurance. Social marketing (SM): An approach that seeks to develop and integrate marketing concepts with other approaches to influence behaviors that benefit individuals and communities for the greater social good. Subsidy: HCM targets subsidies to those segments of the population who cannot pay in full for health services and health products by managing pricing strategies. We examined the use of subsidies for health products and cost recovery. Total market approach (TMA): A lens or process applicable to develop strategies that increase access to priority health products in a sustainable manner. This approach helps grow the market for health products by better targeting free or subsidized products; reducing inefficiencies and overlaps; and creating room for the private sector to increase its provision of health commodities. v Trust brands: PSI condom brands supported with leveraging from DFID. vi TABLE OF CONTENTS EXECUTIVE SUMMARY.......................................................................................................................................... 1 Evaluation Purpose and Evaluation Questions ........................................................................................... 1 Project Background .......................................................................................................................................... 1 Evaluation Methods and Limitations ............................................................................................................. 1 Evaluation Question 1...................................................................................................................................... 2 Evaluation Question 2...................................................................................................................................... 4 Evaluation Question 3...................................................................................................................................... 5 Evaluation Question 4...................................................................................................................................... 5 I. INTRODUCTION ............................................................................................................................................ 7 1.1 Background .................................................................................................................................................. 8 1.2 Methodology ............................................................................................................................................ 10 2. FINDINGS AND CONCLUSIONS............................................................................................................... 12 2.1 Evaluation Question 1: Findings, Evidence and Conclusions Associated with Health Impacts ........................................................................................................................................................................... 12 2.2 Evaluation Question 2: Findings, Evidence, and Conclusions on sustainable SM/SBCC models ........................................................................................................................................................................... 33 2.3 Evaluation Question 3: Lessons Learned ........................................................................................... 35 2.4 Evaluation Question 4: Key Recommendations ............................................................................... 36 ANNEXES...................................................................................................................................................................... 41 ANNEX 1: LISTS OF PARTNERS AND IMPLEMENTATION RESPONSIBILITIES ....................... 41 ANNEX 2: FINANCIAL ANALYSIS and FINANCIAL EXPENDITURES ......................................... 42 ANNEX 3: SCOPE OF WORK .................................................................................................................. 43 ANNEX 4: DATA COLLECTION SCHEDULE ..................................................................................... 68 ANNEX 5: CONSULTANTS’ CVS ........................................................................................................... 69 ANNEX 6: BEHAVIOR CHANGE FRAMEWORK FOR SOCIAL MARKETING .......................... 78 ANNEX 7: LIST OF DOCUMENTS REVIWED BY THE EVALUATION TEAM ........................... 79 ANNEX 8: RGD/KII ...................................................................................................................................... 84 ANNEX 9: RGD/KII GUIDES ..................................................................................................................... 87 ANNEX 10: QUESTION GUIDE FOR PROVIDER PERCEPTIONS .............................................. 117 ANNEX 11: DESIGN MATRIX FOR DATA ANALYSIS.................................................................... 124 ANNEX 12: CONTRACEPTIVE USE BY WOMEN ........................................................................... 130 ANNEX 13: NETS DISTRIBUTION BY SOURCE AND DONOR ................................................. 131 vii ANNEX 14: MAP OF SOCIAL FRANCHISE FACILITIES .................................................................. 132 ANNEX 15: Quanitities of socially marketed condoms sold ............................................................ 133 ANNEX 16: TOTAL MARKET VALUE OF SOCIALLY MARKETED CONDOMS .................... 134 ANNEX 17: QUANTITIES OF SOCIALLY MARKETED CONDOMS SOLD.............................. 135 ANNEX 18: MARKET SHARE OF SOCIALLY MARKETED CONDOMS .................................... 136 ANNEX 19: COST RECOVERY AND LEVEL OF SUBSIDY OF SOCIALLY MARKETED HCM PRODUCTS .................................................................................................................................................. 137 FIGURES AND TABLES FIGURES Figure 1: Services offered by SF networks ................................................................................................14 Figure 2: Type of Modern Method Used By % of WRA .....................................................................24 Figure 3: Use of Mosquito Nets by Household Members.....................................................................24 TABLES Table 1: Number of People Receiving Services from Social Franchises .............................................13 Table 2: Social Franchise Status in SafeCare Program for Year 3 .......................................................19 Table 3: Comparison of WRA Profiles represented in the 2012 and 2014 TRaC Surveys ...............................................................................................................23 International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This is the final performance evaluation (end-of-activity evaluation) report of the APHIAplus health communication and marketing (HCM) project undertaken on behalf of the United States Agency for International Development/Kenya, Office Health, Population and Nutrition (USAID/K/HPN) by an independent review team fielded by International Business & Technical Consultants Inc. (IBTCI). The evaluation took place from October 16, 2014 to December 15, 2014, covering the implementation period from March 2012 through September 2014. The purpose of the evaluation was to establish the extent to which the expected outcomes have been met and to provide lessons learned and recommendations for future activities in health communication and marketing. Evaluation questions include: 1. To what extent has the activity achieved its mandated outcomes in intermediate results (IR) one and two along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? 2. To what extent has the activity succeeded in creating sustainable social marketing and communication models for the health sector? What strategies and approaches facilitated the achievements/creation of such sustainable models, if any? 3. What are the key lessons learned, especially on the establishment of local sustainable social marketing models among private-sector partners, local NGOs and government departments–at the national and county levels? 4. What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them more effective in delivering sustainable services that reach targeted populations? PROJECT BACKGROUND APHIAplus HCM was awarded to Population Services International (PSI) under USAID Cooperative Agreement (CA) #AID-615-A-12-00002 on March 21, 2012 with the period of performance expected to end on March 20, 2015. PSI implemented the activity with nine partners. The activity’s goal is to improve the health outcomes and impact for Kenyans through sustainable, country-led programs and partnerships. The aim is to increase the use of quality health services, products and information and to strengthen leadership, management and governance in social marketing and social behavior change communication (SM/SBCC) to promote changes in health behavior. HCM focuses on improving health outcomes in the areas of malaria, HIV, family planning (FP) and reproductive health (RH) as well as child health (CH). EVALUATION METHODS AND LIMITATIONS The evaluation team used a mixed methods approach to provide evidence and answer the evaluation questions. Methods included:  Comparative analysis of data from the Tracking Results Continuously (TRaC) surveys;  A desk review of 178 documents;  Analysis of round-table group discussions (RGD) with eight different groups involving 60 stakeholders;  Thirty-two key informant interviews (KIIs) conducted with Ministry of Health (MOH) staff, private￾sector enterprises and development partners. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 2  Sixteen telephone interviews were also held with social franchise (SF) health providers and follow￾up interviews were conducted with Populations Services Kenya (PSK) staff. Limitations include limited field visits and the lack of availability of key MOH staff for interviews. EVALUATION QUESTION 1 To what extent has the activity achieved its mandated outcomes and impact in IR 1 and IR 2 along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? KEY FINDINGS AND CONCLUSIONS FOR IR 1–INCREASED USE OF QUALITY HEALTH SERVICES, PRODUCTS AND INFORMATION IR 1.1: INCREASE ACCESS TO AND DEMAND FOR HIGH-QUALITY HEALTH PRODUCTS AND SERVICES Findings: Evidence of increased access to and demand for high-quality health products and services is based on data from the HCM monitoring and evaluation (M&E) plan that show:  The number of SF increased from 213 in 2012 to a total of 473 in 2013 (122% increase).  SF facilities are located in 39 of the 47 counties in Kenya, representing an 82% reach.  Increased demand from year-to-year in the number of individuals receiving services from SF facilities related to the four targeted health elements; HIV/AIDS (e.g. HIV Testing and Counseling met 150% of its target), family planning and reproductive health (met 78% of the target), malaria (met 81% of its target), and treatment of diarrhea in children under five (51% of the target). Conclusion: Based on these findings, we conclude that HCM has succeeded in increasing access to and demand for high-quality health products and services. IR 1.2: IMPROVE ADOPTION AND MAINTENANCE OF BEHAVIORS Findings: Evidence from comparative analysis of data from TRaC surveys conducted by PSI and data presented from indicators in the HCM M&E plan demonstrate increases in the adoption of healthy behaviors related to the four areas of health concern targeted by the government of Kenya (GOK) and USAID. These include findings that show:  An overall increase in the use of condoms to prevent HIV by different population segments based on age, marital status and types of sexual behavior practices associated with these segments;  A reduction in the practice of concurrent sexual partners from 10.7% prior to the start of the activity to 8.9%;  An increase in the number of men who are circumcised to prevent HIV (with 66 men reported circumcised in 2012 to 1,166 reported in 2014 (based on data reported from SF facilities indicators));  An increase in the contraceptive prevalence rate among women of reproductive age in rural and urban areas from 40% to 77% and 40% to 73% respectively (p<0.01) over the life of the project;  Increases in the uptake of modern family planning methods among women of reproductive age (15 to 24 years) from 39.5% to 51% in urban areas (p<0.01) and from 38% to 44% in rural areas (p=0.04)  Increases in the use of malaria nets particularly among pregnant women (50% in 2010 to 58% in 2014) and children under five years of age (from 49.5% in 2010 to 60.1% in 2014); International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 3  The activity reached 81% of its target for malaria treatment;  An increase in the treatment of water to prevent diarrhea in children under five from 1,032,540,640 liters to 2,032,783,500 liters in 2013. Conclusion: Consistency in maintenance of behaviors is measured by data that show year-to-year increases in healthy behavior practices. While some findings do show variation in the consistency of use of condoms by specific population segments of men related to different sexual behavior, and in the use of modern family planning methods by different population segments of WRA, overall trends show increases in the adoption of healthy behaviors during the life of the project (LOP) related to each of the four health program elements. Based on these findings, we conclude that HCM has succeeded in promoting adoption of healthy behaviors and the maintenance of behaviors. IR 1.3: STRENGTHEN PUBLIC‐PRIVATE PARTNERSHIPS TO DELIVER HEALTH COMMUNICATION, MARKETING AND SERVICES Finding 1: HCM created public-private partnerships (PPPs) for establishing a telephone hotline to provide sexual and reproductive health counseling, and to create a health insurance scheme based on risk pooling to extend health insurance coverage to the poor. Finding 2: Data on the performance of the telephone hotline to provide sexual reproductive health counseling shows that the number of calls received from individuals grew from zero in 2012 to 442,925 calls to end of September 2014, surpassing the target of 175,000. Finding 3: Approximately 21% of the LOP target (100,000, cumulative) was met on numbers of individuals gaining access to health insurance through Linda Jamii. (HCM M&E plan) From a baseline of zero in 2012, 7,351 individuals accessed health insurance by the end of 2013. For the cumulative LOP a total 20,692 individuals were insured through Linda Jamii. Conclusion 1: Based on the findings on the PPP to establish and implement a tele-counseling program, HCM was successful in establishing capacity for the PPP to deliver health communication messaging. Conclusion 2: In its current form, Linda Jamii is unsustainable due to low uptake by the targeted population. KEY FINDINGS AND CONCLUSIONS FOR IR 2: STRENGTHENED LEADERSHIP, MANAGEMENT AND GOVERNANCE FOR SUSTAINED HEALTH BEHAVIORS IR 2.1: STRENGTHEN THE CAPACITY OF PUBLIC SECTOR INSTITUTIONS TO PROMOTE AND OVERSEE SOCIAL MARKETING AND SBCC INITIATIVES IN KENYA Findings: HCM introduced the Performance Improvement Plan (PIP) to MOH central to promote its capacity to oversee and monitor county-level alignment with the national health promotion strategy, and worked in collaboration with MOH to develop guidance and quality assurance standards for SBCC. Counties provide quarterly progress reports on PIP implementation and MOH central holds quarterly review meetings with county officials to identify and address gaps, strengthening capacity at both levels. Conclusion: Capacity has increased at the central and county levels to promote SBCC initiatives and for MOH to monitor and oversee these initiatives. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 4 IR 2.2: STRENGTHEN THE CAPACITY OF ONE OR MORE KENYAN ENTITIES TO IMPLEMENT SOCIAL MARKETING AND SBCC INITIATIVES Finding 1: PSI transitioned HCM operations to Population Services Kenya (PSK), and then played an oversight and mentoring role to PSK. PSK is a registered, independent entity whose CEO, management and staff are all Kenyan; 50% of the board of directors are also Kenyans. Finding 2: PSK awarded HCM grants to three local NGOs and provided them with training to increase their capacity to achieve objectives included in their grant contracts. The sub-grantees were also trained in the SM/SBCC approach and messaging delivery methods. Data from HCM quarterly reports and data from the HCM M&E plan describe the preparation of the sub-grantees to implement a pilot program on SM/SBCC methods for hard-to-reach and vulnerable populations at the sub-county level. The program was successful in applying interpersonal-based communication (IBC) methods to engage target populations in discussions of healthy behaviors to prevent malaria. Conclusions: Based on findings demonstrating the current role and activities of PSK, PSK has an empowered board and management comprising predominantly Kenyan members and the capacity to implement HCM activities. IR 2.3: INCREASE SYNCHRONIZED NATIONAL AND USG‐FUNDED SOCIAL MARKETING AND SBCC Findings: Memoranda of understanding (MOUs) were signed between HCM and five other APHIAplus implementers to synchronize SM and SBCC, and 41 joint planning meetings were held to implement MOUs. Conclusion: HCM developed the foundation for synchronized SBCC activities. Overall conclusions on the extent of success in meeting IR 1 and IR 2 mandated outcomes along the dimensions of health impact, equity and subsidies for priority poor and vulnerable populations: Based on a review of findings and conclusions related expected outcomes for IR 2, we concluded that IR 2 is not designed to directly affect outcomes along these lines. Our conclusions are based on findings and conclusions from IR 1. We have concluded, based on our review and assessment of expected outcomes for sub-IRs 1.1, 1.2 and 1.3, that APHIAplus HCM has succeeded in promoting:  Health impact as measured by increases in behavior change related to HIV, RH/FP, malaria and diarrhea;  Equity as measured by expanding access to quality healthcare and health products for all segments of the Kenyan population based on ability to pay, through the use of a TMA;  Provision of subsidies for health products for those Kenyans unable to pay full cost by targeting subsidies on the basis of need and managing cost recovery for subsidized products;  Attention to priority poor and vulnerable populations based on the design and delivery of evidence￾based campaigns and messaging to change health behaviors. EVALUATION QUESTION 2 To what extent has the activity succeeded in creating sustainable social marketing and communication models for the health sector? What strategies and approaches facilitated the achievements/creation of such sustainable models if any? International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 5 Findings: Based on a review of findings and conclusions from Evaluation Question 1, the evaluation team (ET) identified three sustainable SM and communication models. These include:  Successful strategies and approaches to promote MOH adoption and institutionalization of SM/SBCC model to promote behavior change, included widespread SBCC training for all levels of MOH; collaborative work to establish a national health promotion strategy, and on-the-job training and mentoring to county health promotion officers to conduct situation analysis for campaigns design;  Use of the “360-degree” behavior change communication approach to increase uptake of healthy behaviors–this approach has demonstrated effectiveness in promoting increases in behavior change;  The pilot model for extending messages to the sub-county level, which is sustained by a TMA in SBCC and use of the interpersonal communication (IPC) message delivery method. Conclusions: Based on the findings on these three models, we conclude that HCM has succeeded in creating excellent communication models for the health sector however, the ET questions whether or not these models are sustainable without external funding. EVALUATION QUESTION 3 What are the key lessons learned, especially on the establishment of local sustainable social marketing models among private-sector partners, local NGOs, and government departments at the national and county levels? Findings: Based on a review of findings and conclusions from Evaluation Questions 1 and 2, the ET concluded that the following provide lessons learned:  Campaigns and health messaging informed by contextual knowledge and factors driving behavior for specific populations and decentralized in implementation provide a successful model for reaching targeted population segments;  Evidence-based health promotion campaigns to increase behavior change in target populations are successful when multiple channels–the “360-degree approach” based on behavioral research–are used to place health messages;  The PPP created by HCM for sexual and reproductive health counseling through telephone hotlines was an effective approach for expanding SBCC messaging for healthy behavior change to individuals who may not otherwise be reached by health promotion campaigns or use services at public or private clinics;  Evidence-based campaigns, messaging and placement to promote positive health behavior designed for targeted population segments require applied research to prove any association between these interventions and behavior change they were designed to effect. EVALUATION QUESTION 4 What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them more effective in delivering sustainable services that reach targeted populations? This selection of key recommendations are based on the ET’s review and analysis of findings and conclusions made in response to Evaluation Questions 1-3: International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 6  Include an enabling environment component in future programs to assess the framework affecting the expansion of health insurance coverage and risk pooling, and advocate for changes;  Create stakeholder groups to develop a harmonized, acceptable accreditation framework for use in scaling-up the implementation of quality standards in SF facilities;  In order to accurately compute the required level of subsidy, reporting and forecasting should incorporate all costs related to the products including overheads, marketing and distribution costs;  Concentrate on expanding the franchised network presence to cover marginalized counties in Kenya;  Design indicators and establish targets to improve performance in equity, a key component of the sustainability model. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 7 I. INTRODUCTION This is the final performance evaluation (end-of-activity evaluation) report of the APHIAplus health communication and marketing (HCM) project undertaken on behalf of the United States Agency for International Development/Kenya, Office Health, Population and Nutrition (USAID/K/HPN an independent review team fielded by International Business & Technical Consultants Inc. (IBTCI). The evaluation took place from October 16, 2014 to December 15, 2014, covering the implementation period of March 2012 through September 2014. The purpose of the evaluation was to establish the extent to which the expected outcomes have been meet and to provide lessons learned and recommendations for future activities in health communication and marketing. Per the Scope of Work (SOW), (Annex 3), evaluation questions include: 1. To what extent has the activity achieved its mandated outcomes in IRs 1 and 2 along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? 2. To what extent has the activity succeeded in creating sustainable social marketing and communications models for the health sector? What strategies and approaches facilitated the achievements/creation of such sustainable models if any? 3. What are the key lessons learned, especially on the establishment of local sustainable social marketing models among private-sector partners, local NGOs, and government departments–at the national and county levels? 4. What are the key recommendations, especially as to how to make future social marketing and communication programs more effective at delivering sustainable services that reach targeted populations? The evaluation was carried out between October 16 and December 15, 2014 (Annex 4: Data Collection Schedule) by Iain McLellan, Rebecca Njue, Nicholas Dondi and Martin Riungu. All have certified that there was no conflict of interest in undertaking this evaluation. (Annex 5: Consultants’ CVs). In 2012, Population Services International (PSI), along with its partners Population Services Kenya (PSK), Health Promotion Unit (HPU), Family Health International (FHI 360), SafeCare, Changamka, Kisumu Medical and Education Trust (K-MET) and Liverpool Voluntary Care and Treatment (LVCT), was awarded USAID Cooperative Agreement (CA) #AID-615-A-12-00002, known as APHIAplus Health Communication and Marketing (HCM). Implementation of activities began on March 21, 2012 and is scheduled to end on March 20, 2015. (Annex I: Partners and Implementation Responsibilities) The award totalled USD36 million with 50% allocated to HIV/AIDS-related activities, 30% to malaria and child survival and 20% to reproductive health (RH) and family planning (FP) activities. To date, the project has spent approximately USD30 million, representing 82% utilization (Annex 2: Table 1 - Budget vs. actual expenditure). PSI designed the activity to transition from an implementation role to coordinating interventions, managing the delivery of subsidies and providing technical support to Kenyan private￾sector entities. A local non-governmental organization (LNGO), Population Services Kenya, was therefore born out of PSI and began official operations on January 1, 2014, transitioning PSI staff to PSK. As PSK evolved into a fully functioning Kenyan organization, PSI’s role increasingly shifted from direct International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 8 implementation to focus on technical advice and capacity building of PSK.1Activity documents state that HCM was designed to support the achievements of USAID Kenya’s (USAID/K) strategic objective 3 (SO3), “Reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated FP and health services”. SO3 focuses on improving the enabling environment for the provision of health services; increasing the use of proven, effective interventions; and decreasing the risk of transmission and mitigating the impact of HIV/AIDS. The strategy for SO3 also focuses on means to increase customer access to and use of FP, RH, and malaria, tuberculosis and child survival services. HCM’s goal was to improve health outcomes and impact for Kenyans through sustainable, country-led programs and partnerships. HCM was designed to achieve this goal through implementing activities to promote the use of quality health services and information and strengthening leadership, management, and governance for sustained health behaviors. According to the Cooperative Agreement, 2 the activity was designed to meet the following intermediate results (IR): IR 1: INCREASED USE OF QUALITY HEALTH SERVICES, PRODUCTS AND INFORMATION  IR 1.1: Increase access to and demand for high-quality health products and services  IR 1.2: Improve adoption and maintenance of healthy behaviors  IR 1.3: Strengthen public‐private partnerships to deliver health communication, marketing and services IR 2: STRENGTHENED LEADERSHIP, MANAGEMENT AND GOVERNANCE FOR SUSTAINED HEALTH BEHAVIORS  IR 2.1: Strengthen the capacity of public sector institutions to promote and oversee social marketing and social behavior change communication (SM/SBCC) initiatives in Kenya  IR 2.2: Strengthen the capacity of one or more Kenyan entities to implement social marketing and SBCC initiatives  IR 2.3: Increase synchronized national and United States Government (USG)‐funded social marketing and SBCC This report is intended for use by USAID/K/HPN, its partners and other stakeholders to inform the design of possible follow-on activities in the area of health communications and marketing. 1.1 BACKGROUND According to the most recent3 Kenya Demographic and Health Survey (KDHS), an estimated 6.3% adults aged 15-49 in Kenya were infected with HIV. About 130,000 new adult infections and 32,500 new infant infections occur in the country each year. 4 HIV is more prevalent in urban areas and along transport corridors, especially among most-at-risk populations. But increasing prevalence is also being seen in rural areas and among married couples.5 Among adults (15-49 years of age), HIV testing and counseling increased significantly from 14% in 20036 to 50% in 2008.7 Malaria accounts for 30% of 1. APHIAPlus Work Plan October 2013 – September 2014. 2 . All references in this report to IRs, sub-IRs, expected outcomes and activities related to each are taken from the PSI CA. 3 . A KDHS was conducted in 2013, however the data are still being analyzed and no report has been produced. 4. KDHS 2008 report. 5. National AIDS and Sexually Transmitted Infection Control Program (NASCOP) Kenya. Kenya AIDS Indicator Survey, 2012. Final report. 6 . KDHS 2003 report. 7 . KDHS 2008 report. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 9 outpatient attendance and 19% of admissions to health facilities in Kenya. 8 Diarrhea kills about 86 children in Kenya every day and every Kenyan child under the age of five experiences an average of three bouts of diarrhea every year, according to KDHS 2008-09. Figures from the KDHS 2008 also show that the prevalence of diarrhea is highest in children aged between 6 and 11 months, followed closely by children between the age of 12 and 23 months. The modern contraceptive prevalence rate (CPR) was 42%, with an unmet need of 25%. The Tracking Results Continuously (TRaC) study conducted by PSI in 2012 found a 26% unmet need for FP among young women between the ages of 15 and 24. Overall, 74% of all women of reproductive age (WRA) reported a desire to protect themselves from becoming pregnant (70% married compared to 81% unmarried).9 PSI, HCM’s lead implementing partner, used the Performance Framework for Social Marketing (PERForM) as its strategy to improve health behaviors among targeted poor, vulnerable and at-risk populations in Kenya. The PERForM theory of change to achieve improved health status is that opportunity, ability and motivation to change health behaviors will be increased by focusing on the implementation of activities within the sphere of the implementer’s influence (product, place, price, and promotion). See Annex 6: Behavior Change Framework for Social Marketing for further detail on this framework. To summarize the activities and interventions for those elements that are within the sphere of influence of HCM, PSI conducted surveys to determine the appropriate mix of health products for each population segment and worked with private-sector retailers and the Ministry of Health (MOH) to create supply chains ensuring routinized delivery of sufficient stocks of health products to private and public sector health providers. Placement of health products (in public and private health clinics, pharmacies, etc.) is based on research to determine where different segments of the population purchase health products such as condoms and family planning products. Pricing of health products is based on PSI’s use of the total market approach (TMA) to ensure that all of segments of the population have access to health products based on their ability to pay. Promotion to support health-seeking behavior and use of health services, products and information is based on the SM/SBCC approach to designing and implementing evidence-based health promotion campaigns. The activity applied three approaches to support an overarching sustainability strategy:  Maximizing health impact;  Minimizing financial vulnerability;  Strengthening local institutional capacity. First, HCM was designed to maximize health impact long into the future by promoting healthy behaviors to increase demand and use for health services and health products, and by expanding the scope and scale of healthcare provided. The activity focused on four health program elements targeted by USAID/K and the government of Kenya (GOK): HIV, reproductive health, malaria, and diarrhea in children under five. To minimize financial vulnerability, HCM focused on three key areas: active price management of SM products; the use of health financing initiatives, such as third-party purchasing of services; and risk pooling/insurance. 8. Knowledge, Attitudes and Practices (KAP) Survey conducted by Department of Child and Adolescent Health 2011 9. TRaC Survey on RH, 2012. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 10 To build local capacity, HCM strengthened institutions, clinics, pharmacies, distributors and retail shops to deliver quality health services. The activity also built the capacity of private-sector partners, non￾governmental organizations (NGOs) and community-based organizations (CBOs) by introducing priority medical services, training, improving private-sector support systems and assisting the government to rally the private sector in support of the GOK’s health strategies. 1.2 METHODOLOGY The evaluation used a mixed methods design, gathering data from various sources including:  Serial cross-sectional TRaC surveys (for RH, HIV and malaria), designed and conducted by PSI in 2011-2014 to assess the effectiveness of SM and communication interventions;  Retail audits measured through Measuring Access and Performance (MAP) surveys of 2012 and 2013 conducted nationally by PSI;  Round-table group discussions (RGDs);  Key informant interviews (participants were purposively selected);  Routine monitoring and evaluation (M&E) data from PSI and from social franchises;  Provider perception telephone surveys;  A desk review of 178 documents (Annex 7: List of Documents Reviewed). Data collection: To complement quantitative data from secondary sources, RGDs were conducted with eight key stakeholder groups totaling 52 participants. (Annex 8: List of RGD groups and KII respondents) Follow-up interviews from RGDs were conducted with 32 key informant (KIIs) and an additional 70 KIIs were conducted with activity stakeholders. (Annex 9: RGD/KII Questionnaires and Group Discussion Guides Summary). Data analysis: Health service delivery and uses of health services and products were analyzed using TRaC and MAP surveys; service statistics from social franchises in networks supported by HCM; RGDs and KIIs. Changes in health behaviors were analyzed by comparing data from TRaC surveys, descriptions of SM/SBCC strategies and campaigns that were aired, and evaluations conducted on the effect of social marketing programs based on TRaC data. Quantitative data from the TRaC and MAP surveys was analyzed using descriptive analysis. Sustainability was assessed on several measures of health market sustainability: data on demand for health services; scope and scale of franchised networks; marketing product sales; total market volume sales; pricing strategies; and institutional capacity for delivery of services and products. Equity was assessed using data drawn from:  MAP studies on health system coverage;  Individuals accessing a risk pool or financial products (PSI/PSK quarterly reports 2011-2014);  Subsidy targeting (PSI/PSK quarterly reports 2011-2104);  TRaC surveys, (2011-2014 conducted by PSI). Statistical analysis was conducted at 5% level of significance. (Annex 11: Data Analysis Design Matrix) Limitations to the evaluation: In some cases, indicators for secondary data were not consistently monitored during the life of the project (LOP) and therefore trends could not be established. For example, maternal health indicators on the use of treated water through WaterGuard was not monitored in 2014, but monitored in 2012. In such cases, a mix and match of data sources was provided by PSK. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 11 Exit interviews were planned with the beneficiary population. However, the approval to conduct these interviews was not obtained from the Ethics Review Board in a timely manner. It was therefore not possible to collect this data within the period of fieldwork conducted by the ET. To mitigate this, interviews were conducted with 22 randomly selected SF health providers. The RGDs were affected by limited participation, especially by senior government officials, as well as the high rate of staff transfers within the MOH. Many current staff were new to their positions and did not have the history needed to respond adequately to RGD questions. This shortcoming was overcome through follow-up KIIs. The financial reports reviewed (drawn from PSK’s quarterly reports and the management information system) were presented by budget categories rather than service delivery areas. Consequently, we could not validate budget utilization by service delivery area. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 12 2. FINDINGS AND CONCLUSIONS 2.1 EVALUATION QUESTION 1: FINDINGS, EVIDENCE AND CONCLUSIONS ASSOCIATED WITH HEALTH IMPACTS To what extent has the activity achieved its mandated outcomes and impact in IRs 1 and 2 along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? Organization of findings and conclusions for evaluation question 1: Question 1 asks to what extent mandated outcomes have been achieved for IR 1 and IR 2 along the parameters of health impact, equity, and subsidies for priority poor and vulnerable populations. To address this question, we present evidence-based findings and conclusions based on the findings for each expected outcome under each sub-IR associated with IR 1, followed by IR 2. Each sub-IR has two to four expected outcomes per PSI’s CA. Where appropriate to the sub-IR, we address issues of health impact, equity and subsidy among priority poor and vulnerable populations, and the extent to which HCM has achieved those outcomes along corresponding parameters. IR 1.1 INCREASED ACCESS TO AND DEMAND FOR HIGH-QUALITY PRODUCTS AND SERVICES The expected outcomes for IR 1.1 are: 1. Demonstrated improved sustainability of franchised network; 2. Improved provider perceptions toward health interventions after addressing barriers to behaviors through medical detailing programs; 3. Increased capacity of private sector to provide high-quality and high-impact health interventions. OUTCOME 1: IMPROVED SUSTAINABILITY OF FRANCHISED NETWORKS To develop the market for those who are better able to pay, the HCM team used social franchising and social market approaches, including the targeting of subsidies for services and SM products to attract new users into the market. These markets are comprised of private-sector clinics which have joined networks as social franchises. These franchises provide subsidized products and services for population segments that are somewhat better off than the poorest individuals, who rely on free products and services (Annex 13: Map of SF Facilities). The strategy HCM applied to attract these new users was to offer them an increased scale and scope of subsidized services and products through the development and expansion of social franchises via organized franchise networks.10 The activity measures increases in the sustainability of franchised networks through:  Increases in the number of individuals using health services and products through franchised networks (demand);  Increases in the scale and scope of health services they can provide (access);  Increases in cost recovery (financial viability of targeted subsidization approach). Increased Use of Services and Products from Social Franchises (Demand) 10. The source for this summary description is the APHIAplus M&E plan, p. 6. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 13 Finding - Increased demand: There has been a steady increase in the number of people choosing to go to social facilities in networks to receive services in the four health program elements targeted by USAID and the GOK. This is irrespective of the fact that the only service statistics which met or exceeded the LOP target were the number of people receiving HIV testing and counseling; and the number of adults and children with advanced HIV receiving anti-retroviral therapy (ART). This finding is based on data collected from all social franchises that have joined network franchises and reported in the HCM M&E plan. TABLE 1: NUMBER OF PEOPLE RECEIVING SERVICES FROM SOCIAL FRANCHISES SOURCE: HCM M&E PERFORMANCE MONITORING DATA Indicator Year 1 Year 2 Year 3 Cumulative11 LOP target % met Number of people receiving services for family planning 136,002 330,449 406,497 872,948 1,114,302 78% Number of people receiving HIV testing and counseling 47,078 157,522 267,471 472,071 314,601 150% Number of children under five receiving services for treatment of diarrhea 1,271 23,086 27,758 52,115 101,656 51% Number of people receiving services for treatment of malaria 0 29,968 104,909 134,877 165,758 81% Number of adults and children with HIV receiving a minimum of one service 2,079 3,170 3,715 3,715 5,037 74% Number of adults and children with advanced HIV receiving ART 1,734 2,530 3,419 3,419 3,240 106% Number of HIV-positive pregnant women who received ART to reduce risk of mother-to-child transmission 36 68 357 461 1,241 37% Scale and Scope of Services Provided by Social Facilities (Access) Finding 1 - Scale and coverage: The number of SF increased from 213 in 2012 to a total of 473 in 2013. SF facilities are located in 39 of the 47 counties in Kenya, representing an 82% of counties. The network distribution between rural and urban areas was, respectively, 43% and 57%.12 The evaluation team noted that SF distribution does not cover some of the marginalized counties in rural areas including Wajir, Mandera, Marsabit, Turkana and Samburu.13 Finding 2 - Scope and services: The number of facilities that offer integrated services has surpassed the HCM activity LOP target. By September 2014, 66% of facilities (313 out of a total of 473) had expanded the scope of services they can offer. SF facilities have integrated HIV care treatment, including 11. As of September 2014. 12. Social Franchise outlets M&E data for PSK, 2014. 13. PSI staff has noted that there is a very small private sector presence in those counties, which PSI is aiming to leverage to improve health outcomes. A strategy of increasing public sector providers in these counties may be necessary in the interim. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 14 ART, and a referral system was established for the integration of care and management of HIV between the SF and MOH. Out of 473 facilities, 313 (66%) are now offering integrated services (RH, HIV, diarrhea, malaria and pneumonia). In addition, 21 facilities are now providing male circumcision surgery as part of the minimum package for HIV prevention services. FIGURE 1: SERVICES OFFERED BY SF NETWORKS SOURCE: PSI M&E ROUTINE DAA To support the increase in the diversification of services provided by facilities, SF caregivers were provided with frequent in-service training and capacity building from PSK quality assurance officers and county government officials.14 Conclusions: There is an increased ability of SF networks to meet the demand for affordable health services and products. However, the scale and coverage of SM health services and products have not been fully achieved (for the period under evaluation) based on findings that show franchised networks have not yet reached Kenyans living in Wajir, Mandera, Marsabit, Turkana and Samburu counties. Equity objectives of the activity are also served by increases in the scale and scope of SF facilities, which in turn increase access to subsidized health services and products affordable to Kenyans who cannot pay full price. Total Market Approach of Subsidized Health Products 14. PSK training reports. Lab Pharmacy Dispense drugs Imaging services Maternity Antenatal care services Immunization Procedure room/area Accredited for INpatient by NHIF* Accredited for OUTpatient by NHIF* Has NHIF* both IN and OUT HIV testing HIV counseling and treatment * National Hospital Insurance Fund International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 15 HCM uses TMA to minimize financial vulnerability and increase the overall sustainability of the health market. Network franchises offer subsidized health products at SF facilities. Subsidization is very important means of increasing access to health services and products for those population segments in Kenya who can afford to pay a little, but cannot afford to pay full price. Finding 1 - Cost recovery of subsidized health products: Activities implemented included the introduction of new variants of products to improve market segmentation, targeting of subsidies to those who cannot afford to pay full costs for products, and continued efforts in social marketing for condoms, water treatment product and FP commodities.15 Of the 12 products currently offered by HCM through SF facilities, eight have graduated to full cost recovery.16 Finding 2 - Use and purchase of socially marketed condoms: The use of condoms grew from 57% in 2012 to 84% in 2014 in urban areas and from 62% to 79% in rural areas during the same period. Additionally, during the same period, purchases of socially marketed condoms, from private facilities, increased in both urban and rural areas from 25% to 34% and 19% to 23%, respectively. (Annex 15: Quantities of SM Condoms Sold). Finding 3 - Market Value and Cost recovery of SM condoms: The SM condoms total market value grew from USD1.04 million to USD1.6 million, representing a 52% increase from 2011 to 2014 mainly attributed to effective price management (Annex 16: Total Market Value of SM condoms). Despite the increase in the market value, there was a 15% decline in the number of units sold over the same period. According to PSK, this decline could be partly attributed to price changes effected to incorporate the value added tax (VAT) introduced in September 2013 (Annex 17: Quantities of SM Condoms Sold). Under TMA, efforts were made towards increasing the number of distribution channels, commercial brands and positioning of new Trust ribbed and vanilla condoms as premium products. The market share for SM condoms increased, albeit with irregular growth, moving from 15% in 2011 to 46% in 2012 and down to 34% in 2013 (Annex 18: Market Share of SM Condoms). According to management information system data provided by PSI, cost recovery of SM condoms supported by the program, for example Trust Base, moved from -13% in 2011 to 10% in 2014. (Annex 19: Cost recovery and level of subsidy for SM products). The cost recovery model used by PSI does not incorporate overheads, marketing and distribution costs. Finding 4 - Cost recovery of SM family planning products: In contrast to cost recovery for SM condoms, the level of subsidies for SM Femiplan injectables and Femiplan pills, as measured by the percentage of cost recovery is high. Cost recovery for injectables moved from -63% in 2011 to -65% in 2014, and cost recovery for Femiplan pills moved from -60% in 2011 to -61% in 2014. The HCM January-March 2014 Quarterly Report notes plans to adjust the pricing of Femiplan injectables and oral contraceptives to improve sustainability through increased cost recovery. HCM reported that they will begin focusing on activities to increase opportunities for private-sector brands to be introduced, and to improve cost-recoverable, socially marketed products for sustainable distribution. Conclusions–Cost recovery of SM health products: In both urban and rural areas, there was an overall increase in the utilization of contraceptives from 2012 to 2014 with growth shown in the total market value of SM RH/FP commodities. Based on data showing the percentage of cost recovery from SM injectables and pill from 2011 to 2014, the level of subsidization for these FP commodities is likely to remain high, but might improve in the long run following the implementation of HCM’s plan to adjust pricing. While overall sales of SM condoms decreased from 2011 to 2013, cost recovery for condoms improved and is moving towards sustainability. Progress in cost recovery of SM health products as demonstrated by findings 1-4 contributes to the financial sustainability of network franchises, allowing SF 15. HCM Quarterly Report, January-March 2014. 16. Based on PSK Cost recovery Model-PSI/PSK management information system cost recovery data. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 16 facilities in these networks to continue to offer both health services and products on a subsidized basis for Kenyans who cannot afford to pay full cost. Overall conclusion on improving sustainability of franchise networks: On the basis of increased use of services and products from franchise networks (demand), increasing scale and scope of services and products (access) and increases in cost recovery from socially franchised family planning products (financial viability of social franchising), we conclude that there are demonstrable increases in sustainability. OUTCOME 2: IMPROVED PROVIDER PERCEPTIONS TOWARD PRIORITY HEALTH INTERVENTIONS AFTER ADDRESSING THEIR BEHAVIORS THROUGH MEDICAL DETAILING PROGRAMS Finding 1 - Point of Entry into Health Care System for FP Products: RH TRaC data for 2012 shows public clinics and pharmacies are the most common point of entry into the healthcare system for FP products among poor WRA and young women between ages 15-24. However, data from interviews with public and private providers demonstrated a reluctance to provide adequate counseling and services to customers for modern family planning methods, specifically regarding intra-uterine devices and implants known as long-acting reversible contraceptives. Among the reasons cited 17 were misconceptions around client eligibility, time constraints and lack of technical skills among public and private (pharmacy) providers to offer modern family planning products and counsel customers on how to use them correctly. Further probing of the HCM team’s medical detailers revealed their perception that the behavior of pharmacy providers is based on their lack of knowledge about modern FP products, e.g. on their correct use and how to manage their side effects.18 Finding 2 - Institutionalization of Medical Detailed Products: To promote behavior change, HCM instituted medical detailing programs requiring multiple visits to the provider to improve perception and change behavior. For example, in the case of retail pharmacy providers, HCM provided pharmacists with one-on-one behavior change communication messages at their premises, focused on creating FP advocates to improve the quality of FP counseling through a program of training and mentorship. To enable providers to counsel customers appropriately and accurately, mentoring was provided on improving their ability to counsel clients on the range of modern family planning methods and their correct use, dispel myths and misconceptions, know potential side effects and how to manage them, and refer clients when and where appropriate. To follow up on the one-on-one mentoring and training, continuous medical educations sessions and small group meetings are held with medical detailers, which allows providers to raise questions and discuss concerns. Finding 3 - Number of Providers Reached Through Medical Detailing: Output level data from HCM’s M&E plan shows that against a 2014 target of reaching 5,500 providers (comprising clinics and pharmacies) through medical detailing, 98.5% (5,418 providers) had been reached. Finding 4 - Increase in Referrals Through Use of Vouchers: There are no specific data on changes in perception and behavior among providers that were reached by medical detailing programs. The HCM January-March quarterly report for 2014 notes that the project has continued to provide SBCC messaging to pharmacy providers to encourage them to counsel FP clients before dispensing contraceptives, with counseling aid to support them in this role. Data from the first quarterly report for 2013 and from the second quarterly report in 2014 shows an increase in referrals through use of vouchers from pharmacies to SF facilities in the Tunza Network as well as public clinics and other private facilities, with 25% to 26% of the client referrals for long-term methods and further counseling 17. HCM 4th Quarterly Report, 2013 18. HCM Quarterly Report, January-March 2014. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 17 and management of side effects that pharmacy providers could not handle. HCM planned to continue promoting the use of referral vouchers to ensure that poorer clients receive counseling. Conclusions: Based on these findings, using data from the HCM M&E plan, we conclude that HCM has addressed provider perceptions and behavior through medical detailing to increase their ability to counsel customers accurately and effectively in the use of long-lasting modern family planning methods, and to know when to refer customers to clinics based on the limitations of their own knowledge. We cannot make any conclusions about the effectiveness of medical detailing in the absence of data. Unless data on proxy measures such as sales of long-acting reversible contraceptives are systematically collected over time from pharmacies and public clinics that were engaged in medical detailing programs, it will not be possible to know how well medical detailing interventions have succeeded in improving provider perceptions and behavior. OUTCOME 3: INCREASED CAPACITY OF PRIVATE SECTOR TO PROVIDE HIGH-QUALITY CARE AND HIGH-IMPACT HEALTH INTERVENTIONS To examine the capacity of the private sector to provide high-quality care and high-impact health interventions, we examined data on SF facilities including:  The number of facilities that are able to provide integrated care;  The number of SF facility caregivers provided with training related to priority health program elements;  Enrollment of SF facilities in the SafeCare quality improvement program and the progress of enrolled facilities toward accreditation;  Bringing in private-sector retail distribution companies to ensure regular access to high-quality health products, particularly condoms, female condoms and a mix of modern contraceptives;  Data from the telephone surveys conducted with 16 SF facility managers in the Tunza network. To ensure equal representation by facility type and region–with an exception of the northern part of Kenya–a systematic sampling approach was used to generate the sample size of 22 health facilities recommended by USAID: 18 Tunza, two Huduma Poa and two Gold Star Network (GSN) franchises. The ET reached 16 social franchises (all Tunza from Central, Rift Valley, Nairobi, Eastern and Western). While attempts were made to reach the Huduma Poa and GSN social franchises, the team was not successful. Of the 16 respondents, 15 were facility managers/owners, while one respondent was an employee of the facility. Ability of SF Facilities to Provide Integrated Care Finding - Provision of Integrated Services: By the end of year 3, 66% of SF facilities (313 out of a total of 473) were offering integrated services (RH, HIV, diarrhea, malaria, and pneumonia),19 surpassing the HCM LOP target of 280 facilities. These facilities have also integrated HIV/AIDS care treatment including anti-retroviral treatment (ART); and a referral system was established for the integration of care and management of HIV between the SF and MOH.20 In addition, 21 facilities are now providing male circumcision surgery. SF Facilities Provided with Training Related to Priority Program elements 19. Social Franchise outlets M&E data for PSK, 2014. 20. Social Franchise outlets M&E data for PSK, 2014. KIIs with SF partners. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 18 The ET used this as an indirect measure of capacity to provide high-quality care and high-impact health interventions, which supports the ability to provide integrated services. Finding - Training for social franchise providers: To support increases in the capacity of SF facilities to provide high-quality and high-impact health interventions and their ability to provide integrated care and to support the overall increase in the diversification of services, SF caregivers were provided with frequent in-service training and capacity building skills from PSI quality assurance officers, medical doctors and MOH county government officials.21 Seven of 16 SF provider respondents from the telephone interviews said they were able to expand their services as a result of the training received by HCM. Training included adherence to national guidelines for HIV management, HIV prevention strategies, HIV counseling and testing, prevention of mother-to-child transmission, contraceptive technology updates, skills in implant and intra-uterine device insertions as well as infection prevention, TB prevention, integrated management of childhood illnesses, ART management and the role of nutrition in HIV management.22 Enrollment and Progress in the SafeCare Quality Improvement Program Background: The SafeCare quality improvement program is based on internationally recognized standards for quality improvement and provides a stepwise improvement path that describes the quality of each facility based on six steps. Step 0 is entry level. The end point, step 6, means that the healthcare provider (in this case, SF facilities) has excellent quality systems in place, a proven track record of continuous quality replacement and is in substantial compliance with the SafeCare quality standards. Facilities reaching step 6 are then qualified for accreditation.23 The quality standards are based on 13 service elements of SafeCare, which range from management and leadership, patient rights and access to care, and medication management to facility management and support services. 24 The MOH has recognized the SafeCare methodology as a basis for a national accreditation agency for Kenya. MOH officials from the Office of Standards and Regulations are working with SafeCare implementers to form a local certification body within the national regulatory framework. The process begins with on-site training of SF Facility health workers on the principles of quality management and quality improvement and an explanation of the SafeCare Standards. Quality Assurance Officers meet with facilities enrolled in the program to monitor and evaluate progress, and provide guidance to address their specific challenges. Finding 1 - Enrollment in SafeCare quality improvement program: There was an increase in the number of SF facilities enrolled in SafeCare from 73 in 2013 to 93 in 2014. By the end of 2014, 72% of the target (130) was achieved. Finding 2 - Quality improvement plans (QIPs): QIPs are developed by quality assurance officers for each SF facility enrolled in the program, based on an initial assessment of the quality of care the facility is currently able to provide. By the end of 2014, 76 of the 93 SF facilities enrolled in the program have QIPs. The target for the end of 2014 was 115, but this was based on a target of 130 SF facilities enrolled in the program. Facilities without QIPs by the end of 2014 had not yet provided all the data and information required by SafeCare to develop a tailored QIP. Finding 3 - Number of HCWs who Completed In-Service Training: HCM M&E data show that 654 healthcare workers successfully completed an in-service training program through a social franchise 21. PSK training reports. 22. Examples taken from HCM Quarterly Reports: September-December 2013, January-March 2013 and January-March 2014. 23. Background information on the SafeCare quality improvement program is from the HCM Quarterly Reports September-December 2013 and January-March 2014. 24. The source for the 13 SafeCare service elements are from the HCM Quarterly Report September-December 2013. We did not list each one. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 19 facility and the SafeCare program (over the three-year activity period), representing a 28% increase past the LOP target of 509. Finding 4 - Progress toward achieving accreditation: By the end of 2014, 85% (4) of the SF facilities from the Tunza Network reached minimum service quality standards, meeting the target of 85% set for the end of 2014. This represents step 3 in the SafeCare stepwise improvement plan. TABLE 2: SOCIAL FRANCHISE STATUS IN SAFECARE PROGRAM FOR YEAR 3 SOURCE: PSK MANAGEMENT INFORMATION SYSTEM AND M&E DATA Network GSN KMET Tunza Total % Number of SF facilities enrolled in SafeCare 31 25 37 93 Current SafeCare level: In progress toward step 1 10 2 3 15 16% Step 1 14 23 22 59 63% Step 2 5 0 10 15 16% Step 3 2 0 2 4 4% Private-sector Involvement in Retail Distribution of Quality Health Products Finding - Linkages with Private-Sector Retailers: HCM negotiated linkages with private-sector retailers to develop private-sector supply chains for health commodity supplies25 such as HIV test kits, anti-retrovirals (ARVs) and modern family planning products. Fourteen of the sixteen provider respondents contacted noted they are happy with the support provided. They highlighted that commodities are now more affordable to the consumer and they rarely run out of stock thanks to HCM’s support for private-sector retailers. Overall conclusion: Capacity of private sector to provide high-quality care and high-impact health services. The ET’s overall conclusion is that APHIAplus HCM has succeeded in increasing the capacity of the private sector, specifically the SF facilities in franchise networks, to provide high-quality care and high-impact health services. We base this overall conclusion on the findings from the dimensions we selected to measure capacity that demonstrate:  Increasing ability to provide integrated services in the four program elements targeted by the MOH and USAID/Kenya, supported by on-going training in high-impact health service provision;  Progress in collaboration with MOH in building a foundation to support accreditation that meets international standards on quality of care;  Ongoing increases in the quality of care that SF facilities are able to provide.  Finally, data from SF facility managers reached through telephone interviews provided an indirect measure of the capacity of private-sector distributors of health products, based on their increased confidence in receiving regular deliveries of stock. Based on these findings, we conclude there is a strong foundation to support further increases in the quality of care provided by private facilities in Kenya as measured by SafeCare’s stepwise improvement path. Additionally, we conclude that further increases in capacity to provide high-quality and high-impact health services can be expected over time given the strong buy-in, partnership and collaboration with MOH. 25. Note that HCM also developed public sector supply chains to deliver free health commodities such as condoms and mosquito nets. In this section of the evaluation report, we are examining private sector capacity. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 20 IR 1.2 INCREASED ADOPTION AND MAINTENANCE OF HEALTHY BEHAVIORS The expected outcomes for increased adoption and maintenance of healthy behavior are: 1. Improved, appropriate healthcare-seeking behavior; 2. Increased use of health products and services; 3. Designed, developed and implemented innovative SBCC campaigns. OUTCOME 1: IMPROVED, APPROPRIATE HEALTHCARE SEEKING BEHAVIOR The evidence we have for improved healthcare-seeking behavior is based on a comparative review of data from the HCM M&E plan on the numbers of people coming to social franchises to receive health services. We presented these data in Table 1: Number of People Receiving Services from Social Franchises (see p. 11) as evidence for increased demand for health services as part of our assessment of the sustainability of network franchises. The ET treated demand as one of the key components related to the sustainability of network franchises. The proxy measure we used to assess demand is the number of people coming to social franchises for health services. The data presented in Table 1 is an aggregate of service statistics regularly reported from social franchises in each of the franchise networks. The ET contends that this same data set can also be used as a proxy for measuring healthcare-seeking behavior. Finding - Increased Health-Seeking Behavior: Referring to the data presented in Table 1, the evidence shows that there is an increase in healthcare-seeking behavior based on the number of people receiving services from social franchises in each of the four areas targeted for improved health outcomes: HIV/AIDS (counseling, testing, prevention and treatment); FP/RH; treatment of malaria; and treatment of children under five with diarrhea. Conclusion: We conclude that there has been an increase in healthcare-seeking behavior during the LOP based on the evidence of increasing number of people coming to social franchises and receiving services. OUTCOME 2: INCREASED USE OF HEALTH PRODUCTS AND SERVICES The ET used data sets on the use of health products and services in the areas of HIV/AIDS, FP/RH, malaria and diarrhea as evidence for behavior change. This section provides findings with supporting evidence of behavior change for each targeted health area. Our presentation is organized by program elements targeted by the MOH and USAID/Kenya. HIV Prevention: Evidence of Behavior Change–Use of HIV Prevention Products: Condoms Findings on behavior change related to HIV prevention were derived from a comparison of data from HIV TRaC surveys 2011 and 2013: 1. Condom use among different cohorts based on the type of partners and age of respondents; 2. HIV testing and counseling-seeking behavior. Finding 1 - Consistent Condom Use: Consistent use of condoms among men between the ages of 15 and 24 increased significantly from 21% in 2011 to 44% in 2013 (p < 0.01). Among married respondents, consistent use of condoms increased from 9% in 2011 to 14.5% in 2013 (p < 0.01). International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 21 Finding 2 - Condom Use with Casual Partners: The percentage of young men between the ages of 15 and 24 who reported using condoms in the last 12 months with at least one casual partner decreased from 79% in 2011 to 63% in 2014 (p = 0.06). However, the decrease was not statistically significant. Finding 3 - Condom Use Non-Spousal Partners: Condom use at last sex with non-spousal partners decreased from 74% in 2011 to 69% in 2013 (p = 0.13) among men aged 15 to 49. Finding 4 - Condom Use Non-Spousal/Non-Cohabitating Partners The percentage of men between the ages of 15 and 24 who always use condoms with non-spousal, non-cohabitating partners increased from 58% in 2011 to 67% in 2014 (p = 0.25). Conclusion: There has been uneven progress in the use of condoms related to different sexual behaviors among men in different age groups during the period under review. While consistent use of condoms by young men and married men has increased significantly, as with young men between the ages of 15 and 24 with non-spousal, non-cohabitating partners. The findings show a sustained trend in use of condoms with non-spousal partners. There was a decline in use of condoms with casual partners and with non-spousal partners, however the observed decline was not statistically significant. There was an observed increase in condom use with non-spousal, non-cohabiting partners among young men (15- 24), however the increase was not statistically significant. Information provided on increases in condom price in 2013 from PSK may be a factor in these decreases, but without further data, we cannot say this conclusively. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 22 HIV–Findings on Other Healthy HIV Prevention Behaviors: Male Circumcision and the Practice of Concurrent Sexual Partners Finding 1 - Male circumcision to prevent HIV: There was an increase in the cumulative number of men circumcised as a method for preventing HIV from year 1 through year 3 based on data collected on the President’s Emergency Plan for AIDS Relief (PEPFAR) indicator “number of men circumcised as part of the minimum package for male circumcision for the prevention of HIV provided by a social franchise”. Some 66 men were circumcised in year 1, 449 men in year 2 and 651 in year 3 for a cumulative LOP total of 1,166 circumcised men, meeting 86% of the LOP target of 1,353. Finding 2 - Attitudes towards male circumcision: Based on a comparison of 2011 and 2013 TRaC data, changes in favorable attitudes toward male circumcision as a method to prevent HIV increased significantly from 36% to 81%. Conclusion: We conclude there has been a change in behavior toward the practice of male circumcision to prevent HIV, based on demonstrated increases in favorable attitudes toward this method and the cumulative increase in men who have been circumcised in SF facilities. Finding 3 - Addressing risky behavior, the practice of concurrent sexual partners: There was a decrease in the prevalence of concurrent sexual partnership from 10.7% in 2009 to 8.9% in 2013. (HIV TRaC surveys). Conclusion: With the reduction in the number of individuals with concurrent sexual partnerships we can conclude that behaviors are changing and the risk of contracting or spreading HIV has been reduced in the target population. Family Planning and Reproductive Health: Evidence of Behavior Change HCM’s strategies to increase the CPR, the use of modern methods and the use of long-lasting modern methods included:  Social franchising and evidence-based health promotion campaigns;  Medical detailing programs;  Increasing the availability of a greater mix of modern contraceptives, including condoms, in different locations. Our findings on changes in FP/RH behavior among all WRA aged 15 to 49 are based on a comparison between 2012 and 2014 RH TRaC survey data. We include a subset of those findings on sexually active young women between 15 and 24 because findings from the 2012 Report on RH TRaC survey data26 show a very high unmet demand for family planning products among them (76% in rural areas and 73% in urban areas), as well as a very high percentage of women in that same cohort in rural and urban areas, married and unmarried, who reported that they do not want to become pregnant but are not using contraceptives. Noteworthy is there are significant differences in the characteristics of WRA in the two surveys which indicates that either age confounds the association between CPR and marital status OR marital status confounds the association between CPR and age (see Table 3). The observed differences in the composition of the sample (rural vs. urban) in 2012 and 2014 were statistically significant. 26. Final 2012 Final Reproductive Health TRaC survey report. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 23 TABLE 3: COMPARISON OF WRA PROFILES REPRESENTED IN THE 2012 AND 2014 TRAC SURVEYS SOURCES: PSI TRAC SURVEYS, 2012 & 2014 WRA Characteristics 2012 2014 P Values Rural Urban Rural Urban Rural Urban WRA enrolled in survey 77% 23% 58% 42% -25% 83% 0.0001 15-24 year old WRA27 56% 100% 22% 33% -61% -67% 0.0001 Unmarried 15-24 year old 34% 44% 71% 73% 109% 66% 0.0001 Finding 1 - Contraceptive prevalence rate: There has been an increase in the CPR among WRA in both rural and urban areas. This finding is based on data from the RH TRaC surveys conducted in 2012 and 2014. The CPR in rural areas increased from 40% to 77% (p < 0.01). In urban areas, the CPR rose from 40% to 73% (p < 0.01). Finding 2 - CPR among young women: A comparison of RH TRaC data on the CPR rate among young women between 15 and 24 years shows there has been a significant increase between 2012 and 2014 in urban areas from 65% to 74% (p = 0.02) and in rural areas from 66% to 72% (p = 0.001). Finding 3 - Uptake of modern methods: Using the same data sources to examine the uptake of modern contraceptive methods among WRA (15-24), there was a significant increase between 2012 and 2014 in urban areas from 39.5% to 51% (p = 0.0001) and in rural areas from 38% to 44% (p = 0.04). 27 Young women is a subset of all WRA, therefore, in 2012 56% of all rural women were 15-24 years old and 100% of all urban women sampled were young women. In 2014, the proportion of all rural WRA who were young dropped down to 22%. Thus, the second row of this table (young women) is a subset of all WRA. Similarly, the third row of the table, “unmarried 15-24 year old” is a subset of the young women found in row 2. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 24 FIGURE 2: TYPE OF MODERN METHOD USED BY % OF WRA SOURCE: PSI M&E ROUTINE DATA Finding 5 - Unmet FP Need Among Unmarried Women: Comparing RH TRaC survey data between 2012 and 2014 shows there was a decrease in the percentage of unmarried WRA (15-24) who reported doing nothing to prevent pregnancy from 29% to 26% (p = 0.4) in urban areas and an increase 27.5% to 28.4% (p = 0.8) in rural areas. Finding 6 - Use of Modern FP methods among WRA: Comparing RH TRaC data between 2012 and 2014 shows there was a decrease in the proportion of married WRA (15-24 years) using modern contraceptive methods in urban areas from 59% to 48% (p = 0.01) and a decrease from 52% to 45% (p = 0.1) in rural areas. Among the unmarried, there was a decrease in the proportion of WRA (15-24) using modern contraceptive methods in urban areas from 59% to 48% (p = 0.01) and an increase from 19% to 42% (p = 0.0001) in rural areas. Conclusion: These results show an overall improvement in the use of modern contraceptive methods among target women, The overall CPR is up, the proportion of WRA starting to use modern contraception is up, and the percent of women with unmet need has decreased (although not at a significant level.) Logically, married young women are less likely to use modern contraception. Preventing Malaria Infection: Evidence of Behavior Change Finding 1 - Use of mosquito nets to prevent malaria: Comparing data on the percentage of household members who reported they slept under a mosquito net last night between 2010 based on GOK data and 2014 based on data from malaria TRaC survey shows an increase, particularly among the most vulnerable household members–children under five (from 49.5% to 60.1%) and pregnant women (from 50% to 58%). The data also shows an increase in the percentage of household members who reported sleeping under a long-lasting insecticide-treated net (LLIN) last night (from 29% to 34%). FIGURE 3: USE OF MOSQUITO NETS BY HOUSEHOLD MEMBERS SOURCES: GOK, 2010; TRAC SURVEY, 2014. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 25 Finding 2–Increasing net use, healthy behavior campaign: To promote behavior change on the use of mosquito nets, the mass media campaign “Mbu Nije, Sisi Ndani”, which literally translates to “We inside the nets, the mosquito outside the nets,” was aired nationally continuously from January through August 2012 and periodically thereafter. Additionally, LNGOs trained in SBCC/interpersonal communication (IPC) by HCM then trained CBO health promoters at the sub-county level in the use of IPC methods to reach local populations vulnerable to malaria. These health promoters then made household visits and conducted small-group discussions on the importance of continuous use of nets. The campaign was based on research findings that showed households in highly malarial areas were not consistently using nets throughout the year because of the common misconception that malaria is only contracted during the rainy season.28 Finding 3–Increasing net use, product supply: To support access to nets, HCM distributed free nets (LLINs and insecticide-treated nets–ITNs) through health clinics serving target populations in areas with the highest rates of malaria in Kenya. HCM outcome level data shows that a combination of 1,298,259 LLINs and ITNs were distributed to health centers in year 2. In year 3, an additional 1,034,265 were distributed for a LOP total of 2,332,524 or 86% of the LOP target of 2,699,195 (PSI M&E plan data). Distribution channels were developed by expanding the number of distributors, wholesalers, retailers and sales outlets including SFs, LNGOs and CBOs. The total number of SM bed nets distributed during the period of performance covered in the evaluation was 2.19 million.29 The number of free nets distributed through routine distributions was 6.3 million, while 10.5 million nets were mass distributed in 2012 (Annex 13: Nets Distribution by Source and Donor). To address the barrier of access to nets, the activity was supposed to begin piloting routine net distribution in October 2013. However, the start of this pilot was delayed until October 2014 and is expected to end in March 2015.30 Conclusion: The increased use of mosquito nets can be positively associated with the mass media campaign and with LNGO/CBO collaboration at the sub-county level, using IPC methods to provide tailored SBCC messaging among populations most vulnerable to malaria. Per finding 3, distribution channels have been increased, which was a key component of the HCM activity. Preventing and Treating Diarrhea for Children under Five: Evidence of Behavior Change 28. HCM Work Plan October 2013-2014 and PSK placing schedules and materials presented for review by the Evaluation Team. 29. PSI/PSK management information system sales and distribution data, 2011-2014. 30. HCM Quarterly Report p. 17, July-September 2013; feedback from USAID staff. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 26 HCM’s strategy to prevent diarrhea in children under five included promoting the use of water treatment methods as well as licensing and promoting WaterGuard for water treatment. By promoting health-seeking behavior, HCM also aimed to treat children under five with diarrhea by increasing their numbers brought into health facilities for treatment. Measures of behavior change in preventing diarrhea include: 1. The number of households using any water treatment methods; 2. The number of households using chemicals to treat water. The measure used for treating diarrhea is the percentage of children under five with diarrhea treated with oral rehydration salts (ORS). Finding 1 - Use of water treatment methods: According to PSK data, two types of water treatment methods were tracked: WaterGuard and PUR. There was an increase in use of treated water, but we only have indirect evidence for this based on a proxy indicator–the number of liters of water treated–rather than the number of households using water treatment methods. In year 1, the total numbers of liters treated was 1,032,540,640, of which 991,898,000 liters were treated with WaterGuard and 40,642,640 liters were treated with PUR. In year 2, the total number of liters treated increased to 2,032,783,500 of which 1,970,889,000 was treated with WaterGuard and 61,894,500 was treated with PUR. From January to March 2014, the total number of liters reported treated was 502,614,400 of which 428,256,000 was with WaterGuard and 74,358,400 liters were treated with PUR. Finding 2 - WaterGuard Brand Licensing: After a competitive bidding process, PSI signed a brand licensing agreement with Beta Healthcare to distribute WaterGuard in July 2014.31 “PSK will remain an active market facilitator in the safe water category through generic communication.”32 Finding 3 - Treatment-seeking behavior for diarrhea in children under five: Child survival TRaC data shows that 57% of children under five were treated with ORS in year 1, 67% in year 2, and 96% in year 3, surpassing the LOP target of 87%. Conclusion: HCM’s caregiver-focused campaigns to promote healthy behaviors related to children under five with diarrhea can be associated with increases in the number of children with diarrhea presenting at SF clinics for treatment. OUTCOME 3: STATE-OF-THE-ART TECHNIQUES AND INNOVATIVE SBCC CAMPAIGNS TO MOTIVATE HEALTHY BEHAVIOR HIV Prevention Campaigns Finding - Exposure to Media Campaign on Condom Use: Exposure to the condom SM television and radio campaign, with the addition of the Nakufeel33 campaign, was positively correlated with consistent condom use by young men with regular partners.34 The Nakufeel campaign was based on behavioral research on the barriers to consistent use of condoms by young men with their regular partners. HCM used the findings from this research to design a campaign that would increase the confidence of young people to discuss and negotiate condom use in regular relationships. Additionally, research evidence was used to deliver messaging based on IPC methods for discussion in small-group 31. PSK Quarterly Report, July-September 2014. 32. PSK Quarterly Report, July-September 2014. 33. The Nakufeel campaign was designed to increase the confidence of young people to discuss and negotiate condom use in regular n relationships. 34. HIV TRaC Survey: Evaluation of the Condom Social Marketing Program (Round 7) and the Concurrent Social Marketing Program, 2013. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 27 settings. Those who reported having been exposed to the campaign were 50%35 more likely to report using condoms every time with their regular partner. The Nakufeel campaign was also associated with increases in consistent condom use with a regular partner (45%) among those exposed to the campaign, compared to data on consistent use with a regular partner (36%) among those who were not exposed.36 SM/SBCC Campaigns to Prevent Malaria Infection through Increased Use of Mosquito Nets Finding–HCM-designed malaria prevention campaign to promote consistent use of mosquito nets: The campaign was based on behavioral research findings that showed households in highly malarial areas were not consistently using nets throughout the year because of the common misconception that malaria is only contracted during the rainy season.37 These campaigns were aired in targeted areas based on data showing high rates of malaria infection. A mass media campaign “Mbu Nije, Sisi Ndani”, which literally translates to “We inside the nets, the mosquito outside the nets,” was aired nationally continuously from January through August 2012 and periodically thereafter to reach audiences in high malarial areas. SM/SBCC Campaigns Promoting Use of Family Planning Products Finding – FP Messaging: Similar to the design and implementation of SM/SBCC campaigns to prevent HIV, FP/RH campaigns were designed for and delivered to different segments of the WRA population to promote increases in the use of contraceptives, in particular in the use of modern family planning methods, based on multi-year RH TRaC survey data and research on the barriers to uptake of modern and long-lasting modern family planning methods.38 Population segments included sexually active young women from ages 15 to 24, both married and unmarried, in rural and urban settings, and of differing socio-economic status, as well as women aged 25-49, segmented by the same characteristics. Particularly noteworthy is a study conducted for HCM that showed an increase in uptake of modern contraceptive methods among young women, which is positively correlated with exposure to FP messaging. Usage increased with exposure to multiple methods, compared to just one. 39 The “360-degree” SBCC approach used to reach target audiences was based on behavioral and socio-economic status research to select the most effective methods to reach young women. The data showed that the percentage of young WRA users of FP methods increased significantly from 18% of users who were not exposed to any messaging at all, to 38% after exposure to FP messages by radio. Statistics in the report showed a slight increase in this percentage, to 40%, was noted when exposure came from both radio and TV,40 followed by an increase to 49% when exposed to a combination of radio, TV, and IPC with non-users. Conclusions: PSK designed and implemented innovative SBCC media campaigns which were successful in reaching the targeted population. IR. 1.3 STRENGTHENING PPPs TO DELIVER HEALTH COMMUNICATION, MARKETING AND SERVICES 35 HIV TRaC Survey 2013 36 HIV TRaC Survey 2013 37. PSK presentation on placing schedules and materials promoting the use of bed nets for review by the Evaluation Team; discussions with PSK researchers on the mosquito net campaign; HCM Quarterly Reports, 2013-2014. 38. HCM Quarterly Reports, 2012-2014; PSI Kenya: Study on the Uptake, Drivers and Barriers of Modern Family Planning Methods, 2013. 39. PSI Kenya: Study on the Uptake, Drivers and Barriers of Modern Family Planning Methods, 2013. 40. The researchers (PSK and the Tupanga project) noted that the reason for the very slight increase in use of modern family planning methods with the addition of TV messages is because TV was very limited in the location where the research was conducted, and no commercials were placed on TVs except for the placement of messages at the bottom of TV screens during broadcasting of popular music shows. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 28 Outcomes expected for IR 1.3 per PSK’s M&E plan are: 1. Increased number of public-private partnerships formed and managed by APHIAplus HCM; 2. Increased number of accredited private providers that provide services to defined standards. OUTCOME 1: NUMBER OF PPPs FORMED AND MANAGED BY APHIAPLUS HCM HCM’s design used a total market approach that required establishing PPPs as one means of increasing sustainability. Finding – Number of PPPs formed: While there was not an indicator or set target for the number of PPPs to be established and managed by HCM, the activity did form and manage six PPPs. These were with K-MET, LVCT, Safaricom Ltd., Britam Insurance, Changamka Micro-Insurance Ltd. and the National Health Insurance Fund (NHIF). We present data from the HCM M&E plan on the performance of each PPP. Performance of PPPs Formed and Managed by HCM The LVCT communication partnerships–providing hotline services to franchise providers. HCM in partnership with LVCT developed the one2one hotline offering sexual and reproductive health (SRH)-related tele-counseling aimed at giving young people in Kenya correct, accurate, credible, non￾judgmental and sex-positive information. One2one utilizes social media (Facebook) to pass on credible information on sexuality and HIV via wall discussion, real chat (individual and group) and inbox messages. Within this partnership, LVCT is also required to refer the hotline clients for services delivered by the partnership members including: family planning, cervical cancer and TB screening and treatment, and other HIV services offered by Tunza clinics, K-MET, Gold Star Network, Changamka and APHIAplus partners. Similarly, the partners are expected to create awareness about the one2one hotline and refer clients for on-going counseling.41 Finding 1 – one2one Calls Received: The program achieved 253% of its target (442,925 calls against a target of 175,000)42 on the number of calls received from individuals enrolled in the one2one hotline program for SRH counseling. Finding 2 – Number of Individuals Reached: Data from the HCM M&E plan shows that between 2012 and 2014 of HCM, a total of 195,873 individuals were reached by a telephone hotline, representing 80% of the LOP target of 245,000 individuals. Conclusion: The introduction of hotline services for provision of SRH counseling was an innovative idea that succeeded in expanding access of counseling services to populations that may not have otherwise been reached. Health insurance partnerships. In order to address equity dimensions related to affordable and accessible healthcare, HCM established a partnership with local companies including Safaricom, Britam and Changamka as well as the NHIF and developed Linda Jamii,43 a health insurance scheme targeting low-income earners with an initial annual cost of KES6,000 (approximately USD70) per family. Finding 1 - Performance of health insurance PPPs in increasing access of targeted poor populations to health insurance: Approximately 21% of the LOP target (100,000, cumulative) was 41. PSK Quarterly Report, January-March 2014. 42. HCM performance management plan. 43. Linda Jamii translates to “protect the family”. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 29 met on numbers of individuals gaining access to health insurance through Linda Jamii (HCM M&E plan). From a baseline of zero in 2012, 7,351 individuals accessed health insurance by the end of 2013 and 12,555 by September 2014 for a cumulative LOP of 20,692. Finding 2 - Third Party Payment Options: The product development stage of Linda Jamii met with many challenges due to the complexity of the partnership.44 NHIF, whose role was to carry the medical risk, opted out of the partnership. The initial cost of KES 6,000 ended up being twice what was originally anticipated. The product design was based on targeted market volumes of one million units for risk pooling, and therefore had no entry restrictions.45 Additionally, while the target set by the underwriter (Britam) was one million units, the product sold just 15,000–representing an achievement of just 1.5% of the target market uptake. According to KII with Britam, the low uptake was plausibly due to a lack of experience and understanding of health insurance among Kenyans, barriers on provider information and coverage, and weak distribution channels including use of mobile phones as points of entry. The average policy made claims 4.5 times a year, with bills totaling KES19,000 and representing 158% of the cost. According to one Britam respondent, “For every KES100 received as a premium the insurance pays KES90 in claims. This does not include administration costs.” HCM continues to leverage other donor funding to facilitate family planning vouchers 46 and the M-Kadi47 programs as alternative third-party payment options, meaning payment for services by someone other than the beneficiary. In this case, Changamika Micro-Insurance Ltd facilitates payment for FP products on behalf of the clients. Finding 3 - Legal and Institutional Framework: There is a lack of progressive legal and institutional frameworks to support the development of new health insurance schemes where NHIF, the regulator, is also a provider.48 Conclusion: Expanding a low-cost third-party payment option targeting the poor was not successful. However, the lack of success could partly be due to behavioral issues on health-saving among Kenyans and lack of adequate information, as well as NHIF leaving the partnership, which impacted the annual cost to the consumer. In its current form, Linda Jamii is unsustainable due to low uptake by the targeted population. It attracts seemingly high-risk buyers as evidenced by the high claim rate referred to in finding 2. OUTCOME 2: INCREASED NUMBER OF ACCREDITED PRIVATE PROVIDERS THAT PROVIDE SERVICES TO DEFINED STANDARDS Finding - SafeCare Quality Improvement Program Accreditation: Data on progress of franchised SF facilities toward accreditation through enrollment in the SafeCare quality improvement program were presented under I.R. 1.1, Outcome 4: Increased capacity of private sector to provide high-quality and high-impact health interventions. Drawing from those findings, HCM M&E data show that by the time this evaluation was concluded, no SF facilities enrolled in the program had been accredited to provide defined services. However, four SF facilities from the Tunza network achieved step 3 toward accreditation, which indicates medium quality strength or, in other words, ability to provide services at a minimum standard of quality care. Conclusions: The conclusion we presented for IR 1.1, Outcome 4 is that steady increases of SF facilities toward accreditation can be expected to occur over time given the support and guidance provided through the SafeCare program. Future movement toward accreditation of facilities enrolled in 44. KII with Britam staff. 45. KII with Britam and RGD with consortium members. 46. Family planning vouchers were a DFID-funded program specifically for access to long term methods. 47. Developed by Changamka, M-Kadi sought to provide an electronic administration platform for voucher programs aimed at maternal healthcare. 48. KIIs with Britam, KHF and PSK. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 30 the program will be increasingly supported from the national level through advances in the development and institutionalization of a strong foundation being built by HCM in collaboration with the MOH to support sustainable, high-quality care in Kenya. IR 2. STRENGTHENED LEADERSHIP MANAGEMENT AND GOVERNANCE FOR SUSTAINED HEALTH BEHAVIORS HCM’s strategy to achieve IR 2 is to: 1. Strengthen the capacity of public sector institutions to promote and oversee social marketing and SBCC initiatives in Kenya (IR 2.1); 2. Strengthen the capacity of one or more Kenyan organizations to implement social marketing and SBCC initiatives (IR 2.2); 3. Increase synchronized national and USG-funded social marketing and SBCC. IR 2.1 STRENGTHEN THE CAPACITY OF PUBLIC SECTOR INSTITUTIONS TO PROMOTE AND OVERSEE SOCIAL MARKETING AND SBCC INITIATIVES IN KENYA According to the PSK performance monitoring plan, the expected outcomes from IR 2.1 are: 1. Increased MOH oversight and monitoring of SBCC and social marketing interventions; 2. Established social marketing and SBCC quality assurance standards and guidelines; 3. National inventory of social marketing and SBCC research, materials and tools managed by MOH. Outcome 1: Increase MOH oversight and monitoring of SBCC and social marketing interventions Finding - MOH Oversight: PSI introduced the Performance Improvement Plan (PIP), developed in collaboration with the WHO, CDC, JICA and UNICEF,49 to ensure adherence to standards, guidelines and policies for health promotion strategies, and to provide MOH with the means to conduct oversight and monitoring of SM/SBCC interventions. Health promotion advisory committees (HPACs) use this as a model to develop county-level PIPs to monitor their annual work plans for implementing their SM/SBCC activities, and submit progress reports based on their PIP for review by MOH at the national level. MOH officials meet with counties on a quarterly basis to review their findings together and address gaps.50 Conclusion: Based on the introduction of the PIP as well as RGD data on its use by HPAC members to develop county-level PIPs for monitoring and reporting progress on their annual work plans for MOH officials to review at the national level, we conclude that the project has provided MOH with a mechanism to oversee and monitor the implementation of SM/SBCC strategies to implement the Kenya National Health Promotion Strategy and ensure compliance. Outcome 2: Social Marketing and SBCC Quality Assurance Standards and Guidelines Established Finding 1 - Quality Assurance Standards: Quality assurance standards and guidelines were established during the first quarter of FY 2013. PSK worked with the MOH to develop National Health 49. World Health Organization, Centers for Disease Control, Japanese International Cooperation Agency and United Nations Children’s Fund. 50. RGDs with MOH officials at national and county levels, county level HPAC members. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 31 Communication Guidelines (2013-2017); Operational Guidelines based on SM and SBCC; standards for Health Promotion (2013-2018); and the Health Promotion Strategy for Kenya. Finding 2 - Dissemination and implementation of standards, guidelines and a national health promotion strategy for Kenya: Output-level data from the HCM M&E plan shows that 38 dissemination events were held to roll out the new standards and guidelines and the national strategy at national and county levels. HPACs were formed in 38 out of 40 targeted counties to develop concrete county-level strategies and work plans based on these standards and guidelines to implement the Health Promotion Strategy for Kenya. Of these 38 HPACs, 89% are operational.51 Conclusion: Based on the collaborative PSK/MOH development and finalization and roll out of SM/SBCC standards, guidelines and the Health Promotion Strategy for Kenya down to the county level, we conclude MOH has formalized the foundation for the institutionalization and use of evidence-based health promotion activities to address key health indicators in Kenya. Outcome 3: National Inventory of Social Marketing and SBCC Research, Materials and Tools Managed by MOH Finding - MOH/HPU Resource Center: The ET visited the national-level Health Promotion Unit of MOH and found that the resource room had not yet been fully established and equipment for the room was still being purchased. The ET was informed by PSK they expect MOH/HPU will complete the resource room and begin to inventory and maintain these items by the end of the HCM activity. Conclusion: Given the resource room was not established at the time of the evaluation, the ET draws no conclusions as to whether or not the MOH/HPU will take responsibility for the resource room upon the activity’s closure. IR 2.2: STRENGTHEN THE CAPACITY OF ONE OR MORE KENYAN ENTITIES TO IMPLEMENT SOCIAL MARKETING AND SBCC INITIATIVES Expected outcomes include: 1. Increased use of local Kenyan organizations and agencies in the essential functions of marketing, advertising and distribution of products and services, and percent of budget allocated accordingly; and 2. An increased percentage of the HCM sub-award budget allocated to participating Kenyan organizations. Outcome 1: Increased Use of Local Kenyan Organizations and Agencies Finding 1 - Transitioning PSI operations to PSK: PSK is registered as a Kenyan entity and now manages the sub-grants for PSI.52 The PSK CEO, management and staff are all Kenyan. The Board of Directors is half Kenyan, with the key positions of chair, deputy chair, chairs of strategy sub-committee and finance sub-committee all held by Kenyans. “PSK is more and more local with a broader diversity of backgrounds and values.”53 PSK is an independent Kenyan entity that makes its own decisions, but is benefiting from belonging to the PSI worldwide family. 51. PSK performance monitoring plan. 52. PSK contracts with sub-awardees. 53 PSK Board Member KII International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 32 Finding 2 - LNGO selection: HCM selected three local non-government organizations to implement SM/SBCC initiatives in 2012. The primary driver to select these LNGOs was that they are working in zones with a high incidence of malaria.54 The LNGOs are: Safe Water and AIDS Project (SWAP) and UZIMA in Western Kenya and, Strengthening Community Partnerships and Empowerment (SCOPE) on the coast. Sub-Award Budget Allocated to Participating Kenyan Organizations Finding - LNGO sub-award amounts: PSK awarded grants and allocated funding to each LNGO to carry out SM/SBCC activities. SWAP was allocated $285,714 and has expended $199,267 during the evaluation period; SCOPE was allocated $276,827 and has expended $267,962; and UZIMA was allocated $300,000 and has expended $229,432 55 (Annex 2, Table 2 on sub-award budget and expenditure analysis). Fostering Capacity Development of Participating LNGOs Finding 1 - Strengthened LNGO Capacity: The capacity of the three LNGOs has been developed through training, mentoring and participatory learning. A participatory improvement process was conducted by PSK with each LNGO to identify the organization’s strengths, weaknesses, challenges and needs. PIPs were developed for each LNGO to increase capacity in management, SBCC/IPC methods and social marketing. Management training covered USAID rules, regulations and reporting requirements. “We learned how to follow USAID rules,” a SWAP staff person said in a KII. “We developed internal financial control systems including use of M-Pesa payments instead of cash.” One LNGO attracted direct funding from USAID, and SCOPE was asked by Kwale County to provide leadership and training in health communication activities for the county. A pilot effort to extend healthcare messaging showed that LNGOs were able to train local CBO health promoters to use IPC methods to promote behavior changes in the use of bed nets. Conclusions: PSK has successfully transitioned from an international to a local entity with an empowered board and management. Based on findings on increasing capacity of the selected LNGOs and the evidence of their capacity to date, we conclude HCM has succeeded in developing the capacity of the LNGOs to implement SM/SBCC initiatives. IR 2.3: INCREASE SYNCHRONIZED NATIONAL AND USG-FUNDED MARKETING AND SBCC Expected outcomes from IR 2.3 include: 1. SBCC activities are regional and community levels aligned with national SBCC priorities; 2. Reduced duplication of SBCC materials. SBCC activities at regional and community levels aligned with national SBCC priorities Finding 1 - SBCC activities at the county level are aligned with national SBCC priorities: Evidence for this finding is that county-level SBCC strategic plans for healthy behavior are based on a county-level adaptation of the national-level SBCC strategy for Kenya. The development of SBCC activities is based on national guidelines and policies that were developed by the MOH in collaboration 54. Presentation by PSK to the ET; PSK Quarterly Report, p. 25, April-June 2013. 55. PSK management information system and financial reports. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 33 with PSK and disseminated to MOH offices and units at the national and county levels and to county health promotion officers (HPOs). Finding 2 - MOH Coordination and Leadership: PIPs and MOH PIP review meetings are the tools used to ensure alignment of county-level activities and national SBCC priorities. 56 Following the formation of HPACs and the development of county-level health promotion strategies, annual work plans are developed to implement county-level strategy. Quarterly reports based on PIPs to demonstrate performance against work plans are submitted to MOH at the national level. Quarterly and annual meetings are held to review PIPs and identify gaps as well as the means to address them. On this basis, MOH is able to ensure county SBCC activities are aligned with national priorities. Conclusion: The methods and process used for monitoring and oversight ensures alignment between county-level SBCC activities and national priorities. Reduced Duplication of SBCC Materials Finding 1 - Collaboration: Memoranda of understanding (MOUs) were signed between HCM and five of the six USAID APHIAplus partners57 for joint work planning and activities to reduce duplication of materials and approaches.58 To implement the MOUs, 41 joint planning meetings were held with HCM partners implementing SBCC activities against a target of 24.59 There were some minor differences in approaches, for example on how best to implement social mobilization, and reluctance by some APHIAplus SBCC experts to use HCM materials instead of their own. Conclusion: The MOUs and joint planning meetings were successful in laying down a foundation to reduce duplication among APHIAplus partners. 2.2 EVALUATION QUESTION 2: FINDINGS, EVIDENCE, AND CONCLUSIONS ON SUSTAINABLE SM/SBCC MODELS To what extent has the activity succeeded in creating sustainable social marketing and communications models for the health sector? What strategies and approaches facilitated the achievements/creation of such sustainable models if any? The APHIAplus HCM activity applies three approaches to support an overarching sustainability strategy: 1. Maximizing health impact; 2. Minimizing financial vulnerability; 3. Strengthening local institutional capacity. It was through the lens of these three approaches the ET assessed evaluation question 2. MAXIMIZING HEALTH IMPACT Finding - MOH at the national and county levels have adopted the SM/SBCC model and are using it to promote healthy behavior: Approaches to facilitate sustainability of this model included the immediate engagement of the MOH to adopt it and the provision of widespread training for 56. Data from RGD with national and county-level HPU officials. 57. These are USAID service delivery partners, not to be confused with APHIAplus HCM. 58. RGD with USG partners and KII with PSK. 59. HCM performance management plan. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 34 MOH officials at the national and provincial levels, which was then cascaded down to county levels by adopting a training-of-trainers approach. Buy-in and sustainability of this model was increased by engaging MOH in a collaborative process to develop a national health promotion strategy, quality assurance standards, policies and guidelines for implementing SM/SBCC coupled with the use of PIPs and quarterly and annual review meetings to ensure county alignment with national health promotion priorities and effective use of the model. The evidence for this finding is based on the data presented on findings for expected outcomes under IR 2.1 Strengthen the capacity of public sector institutions to promote and oversee social marketing and SBCC initiatives in Kenya; and on data from RGDs with officials from MOH at the national level, from Health Promotion Units, and with county-level health promotion officers whose remarks substantiate buy-in for the use of the SM/SBCC model to promote healthy behaviors. INTRODUCTION OF THE “360-DEGREE” SBCC APPROACH TO REACH TARGET AUDIENCES USING MULTIPLE CHANNELS FOR HEALTHY BEHAVIOR MESSAGES Finding - FP Messaging: The evidence-based findings presented under evaluation question 1 for IR 1.2 demonstrated there was an increase in uptake of modern contraceptive methods among young women, which is positively correlated with exposure to FP messaging based on behavioral research on the barriers to uptake of modern FP methods using multiple channels to deliver targeted messaging. Specifically, the data showed that usage increased with exposure to multiple methods, compared to just one.60 The “360 degree” BCC approach used to reach target audiences was based on behavioral and socio-economic status research to select the most effective methods to reach young women. Conclusions: Based on this finding, we conclude that use of the “360-degree” behavior change communication approach based on multiple channels to deliver messages using evidence-based campaign materials targeted to specific segments of the population is a proven model to promote changes in behavior. While it is too early to pin down factors that would lead to the sustainability of this model, we propose that it is likely to be used as part of the overall SM/SBCC approach. Given the familiarity of SM/SBCC because of the SM/SBCC guidelines and training provided to MOH at the national and county levels and to county-level health promoters, this model could be sustainable, given that it requires an extension of research to identify all media channels that specific target populations use. Based on the results of that research, placement of tailored messaging is implemented. However, county-level health promoters may need additional mentoring and on-the-job training to conduct this research. We hypothesize that there will be uptake of this approach by the public and private sectors and a demand for training in its use, based on the demonstrated effectiveness of using this cutting-edge approach when combined with data-driven design of health promotion campaigns. 60. PSI Kenya: Study on the Uptake, Drivers and Barriers of Modern Family Planning Methods, 2013. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 35 STRENGTHENING LOCAL INSTITUTIONAL CAPACITY THE PILOT MODEL FOR EXTENDING SM/SBCC ACTIVITIES THROUGH PROMOTION AND SUPPORT FOR LNGO/CBO COLLABORATION TO REACH POPULATIONS AT THE SUB-COUNTY LEVEL USING AN IPC APPROACH Finding 1 - Local Institutional Capacity Building: This model was designed to engage hard-to￾reach populations coupling SM/SBCC approaches for tailoring messaging with the use of IPC methods to promote behavior change. Evidence-based SM/SBCC messaging was designed using research results on factors of behavior related to inconsistent use of nets. The strategies used to make this model sustainable include the application of a training-of-trainers approach to provide three LNGOs with continuous training and mentoring in SM/SBCC and IPC methods. These LNGOs trained CBO health promoters in IPC methods to bring messaging on use of mosquito nets to people in their localities, which were then used to conduct effective visits with individuals and small group discussions to promote behavior change. Finding 2 - Links to SBCC and Malaria Prevention: There are no data to suggest that there is a link between the use of this model approach and changes of behavior in the use of mosquito nets. Conclusion: Evidence provided for evaluation question 1 on malaria prevention under IR 1.2 for the outcome on innovative campaigns designed and conducted to promote behavior change demonstrates that this is a successful model for reaching and engaging populations at the sub-county level, which could prove to be sustainable. ADDRESSING FINANCIAL VULNERABILITY THE TOTAL MARKET APPROACH FOR MINIMIZING FINANCIAL VULNERABILITY AND INCREASING SUSTAINABLE MARKET DEVELOPMENT Finding - Cost Recovery: Activities implemented included the introduction of new variants of products to improve segmentation of the market, targeting of subsidies for those least able to pay for products, and continued efforts in social marketing for condoms, water treatment product and FP commodities.61 Eight of the 12 products currently offered by HCM, eight have graduated to full cost recovery.62 These measures also support private sector development through product sourcing and distribution, and decreased donor dependence. HCM’s model of cost recovery excludes marketing, distribution and administrative costs. Conclusion: According to HCM’s model of cost recovery, financial vulnerability is being successfully addressed by virtue of the fact that eight of 12 products have graduated to full cost recovery. As PSK does not include all related costs when calculating cost recovery, the ET concludes that the model used does not accurately reflect an increasing sustainable market development. 2.3 EVALUATION QUESTION 3: LESSONS LEARNED What are the key lessons learned, especially on the establishment of local sustainable social marketing models among private-sector partners, local NGOs, and government departments–at the national and county levels? 61. HCM Quarterly Report, January-March 2014. 62. Based on PSK cost recovery model; PSI/PSK management information system cost recovery data. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 36 The team drew lessons learned from the evidence-based findings and conclusions made in response to the first and second evaluation questions. Campaigns and health messaging informed by contextual knowledge and factors driving behavior and decentralized in implementation constitute a successful model for reaching targeted population segments and increasing healthy behaviors. Including the “360 degree” BCC approach to determine the most effective combination of messaging channels is a successful way of gaining further increases in behavior change. Sustaining successful models among LNGO and private-sector partners and partners in government departments at national and county levels: Joint planning, preparation and adequate consultations are critical to developing sustainable private and public-private partnerships. However, this is a long process and it takes time for these partnerships to mature. In terms of the public sector, early involvement of national MOH officials in planning and establishing the SM/SBCC approach in Kenya at the beginning of HCM to promote their buy-in was extremely important for its institutionalization. Training and training-of-trainers approaches for national and county-level MOH officials and HPOs in SM/SBBC and in the design of healthy promotion campaigns supported the spread and institutionalization of SBCC-informed health promotion. The pilot activity to train LNGOs in SM/SBCC and use of IPC methods, and their subsequent training of local CBOs, is an effective model to increase messaging outreach for hard-to-reach populations at the sub-county level. The development of PPPs to provide counseling through hotlines is an effective way to expand SF/SBCC messaging to individuals who may not have been reached. Market share of SM products: Product price is a critical factor in growing the market share of SM products due to the target niche, consequently freeing government commodities and preventing the influx of competing brands from neighboring countries. Accrediting SF facilities to increase quality of care: The use of SafeCare as an HCM partner to increase the quality of care provided by franchised networks was not effective in promoting enrolment in the SafeCare accreditation program. 2.4 EVALUATION QUESTION 4: KEY RECOMMENDATIONS What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them more effective in delivering sustainable services that reach targeted populations? Recommendations are based on key conclusions from evaluation questions one and two and the lessons learned from these conclusions. RECOMMENDATIONS TO INCREASE THE SUSTAINABILITY OF SF NETWORK FACILITIES International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 37  To ensure the sustainability of health care facilities to provide high quality services and products, the future SM/SBCC activity should engage in collaborative efforts with the providers and county level health promoters and other county level health professionals from the private and NGO sectors too design county level sustainability plans to maintain and increase providers ability to offer high quality and affordable commodities, equipment and services and to ensure the continuity of demand creation activities and community mobilization initiatives to promote increases in health seeking behavior;  Continue promoting collaboration and linkages between public and private sector suppliers with Network Franchise managers to strengthen and increase the effectiveness of supply chains to ensure continuous, reliable delivery of high quality, affordable supplies and commodities to SF facilities on schedule. RECOMMENDATIONS FOR INCREASING USE OF HEALTHCARE SERVICES AND PRODUCTS  Continue promoting referrals from different sources frequented by specific population segments e.g. examples include referrals from public health clinics, from pharmacies included in the medical detailing program, and the PPP providing one2one tele-counselling to individuals using this hotline for counselling on SRH questions;  Consider expanding referrals from other sources by training LNGOs and LCBOs to know when to make referrals in response to questions when promoting healthy behaviors at the sub-county level. SUBSIDY MANAGEMENT  In order to accurately compute the required level of subsidy, reporting and forecasting should incorporate all costs related to the products including overheads, marketing and distribution costs. The current method has implications for sustainability measures based on cost recovery of subsidized products, specifically, that HCM reporting on gains in full cost recovery for certain subsidized products is actually less than the actual costs related to the products;  To support further increases in access to integrated services and a range of high quality services and products as a means of sustaining demand, there is need to support the improvement of facility infrastructures and the increases in the number of rooms, and in the expansion of room size that can accommodate modern equipment used for testing, treatment, and minor surgery. RECOMMENDATIONS TO ENSURE SUSTAINABILITY AND REACH OF MOH SBCC  Support closer national level MOH collaboration with the new county governments in order to eliminate missed opportunities for leveraging national priorities;  To ensure sustainability and to enable closer collaboration with new county governments, focus on resource mobilization activities and budget advocacy for health promotion activities at the county level;  Support county-level efforts to extend SM/SBCC activities to the sub-county level, and to increase the allocation of resources to the county level to increase the number of HPOs, the budget for customizing International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 38 and increasing reproduction of SBCC/SM materials, and coverage of transportation costs to obtain data for campaign design and to attend kick-off of health behavior promotion campaigns;  Promote MOH participation in funding activities with both cash and in-kind contributions in increasing amounts annually. RECOMMENDATIONS FOR HEALTH INSURANCE  To expand health insurance, future HCM programs should incorporate effective SBCC and demand creation activities for third-party payment options in Kenya;  Design innovative healthcare financing approaches to integrate health saving and risk pooling informed by adequate social behavioural research amongst the target population. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 39 RECOMMENDATIONS FOR INCREASING THE QUALITY OF CARE PROVIDED BY PRIVATE-SECTOR FACILITIES  Scale-up the requirements for SF facilities to remain in franchised networks through enrollment in the SafeCare Quality Improvement Program leading to accreditation. As a requisite for remaining in the SF network, require SFs to enroll in the SafeCare Quality Improvement Program and after enrollment, to show demonstrable evidence of implementing their Quality Improvement Plans and progress toward certification of achieving the minimum standards of quality care;  Support more structured and regular government ownership and involvement in support of SF facility quality improvements by providing MOH at the national and county levels with regular updates on progress, and inclusion of franchises during regular MOH supervisory visits to demonstrate and inform their understanding of what accreditation to provide quality care based on international and NHIF standards requires, and what it actually means for increasing health outcomes in Kenya. RECOMMENDATIONS FOR INCREASING POSITIVE HEALTH OUTCOMES FOR POOR AND VULNERABLE POPULATIONS  Consider conducting participatory research with LNGOs and CBOs to collect data on the specific factors that conferred success of the pilot project for net distribution in Samia sub-county and to determine whether it can be associated with increased use of mosquito nets by these vulnerable populations. The results of this research can be used to inform strategies to scale up this model.  Pilot the same model with priority poor and vulnerable populations for messaging to promote healthy behaviors related to reproductive health, child health, and prevention of HIV. RECOMMENDATIONS FOR FUTURE DIRECTIONS Include an enabling environment component in future programs to assess the current legal and regulatory framework affecting the expansion of health insurance coverage and risk pooling. Form a stakeholder group of key public- and private-sector actors who will work together to promote changes in the framework. Build on the pilot approach to increase reach among vulnerable and hard-to-reach populations at the sub-county level through fostering collaboration between LNGOs and CBOs. Conduct participatory research with LNGOs and CBOs on the specific factors associated with the success of this model to inform strategies to scale up this model. Support closer national-level MOH collaboration with the new county governments in order to eliminate missed opportunities for leveraging national priorities. To promote a learning environment, establish several platforms to document, share and discuss best practices in health promotion at both national and county levels. This will also enhance engagement in performance improvement and accountability. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 40 To test the theory of change for achieving health impacts, future programs should require use of counterfactual models and baseline data collection to evaluate the links between SM/SBCC activities and behavior change as well as changes in health status. To maintain a focus on equity issues and enhance achievement of equity objectives, create indicators on different dimensions of equity, establish evidence-based targets and collect data on a regular basis to redesign interventions as necessary, with focused attention on marginalized areas and areas with large concentrations of poverty. Design an equity TRaC survey to evaluate effectiveness activities designed to increase equity and to ensure these areas are persistently and aggressively targeted with SM/SBCC programming. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 41 ANNEXES ANNEX 1: LISTS OF PARTNERS AND IMPLEMENTATION RESPONSIBILITIES PARTNER ROLES AND RESPONSIBILITIES Partner Roles and responsibilities Population Services International (PSI) Prime Population Services, Kenya Implementation of the APHIAplus HCM Cooperative agreement through a PSI sub-grant Health Promotion Unit Provide leadership in implementation of SM and SBCC at MOH central and country levels 1-2 private sector entity providing leadership in SM and SBCC (TBD) This role was played by PSK played by PSK Private sector entities providing leadership in communications (TBD) These partners were not identified. Instead, the HCM uses different advertising agencies to develop different materials for the various campaigns. The agencies include Red House, which develops malaria materials; Transcend Media, youth HIV materials; Blue Print, HIV materials and Ogilvy, RH and Tunza materials. Kenyan no-profit agencies (TBD) Safe Water and AIDS Project (SWAP), Uzima Foundation and (3) Strengthening Community Partnership and Empowerment (SCOPE) selected through a competitive bidding and capacity built to provide IPC and SM and SBCC directly and through CBOs Family Health International 360 (FHI 360) Built the capacity of Gold Star Social Franchise during the first year of HCM, targeted funding to hire technical staff and acquire the necessary equipment, developing standards and procedures for the HIV continuum of care, supporting the network to develop management, including financial and reporting systems, training and mentorship, linking the network to other social franchises in APHIAPlus HCM. SafeCare/PharmAccess Responsible for developing health social franchise standards and working with individual clinics to raise service standards. The two collaborate to inspect and provide certification to service delivery entities as they attain higher standards of care. Changamka Microhealth Limited Develop health financing products, such as insurances and third￾party payment arrangements that can enable the poor to access quality health services at affordable rates. Kisumu Medical and Education Trust (K-MET) A social franchise network based in Kisumu and focusing on maternal, neonatal and child health and malaria. The network received funding from APHIAPlus to develop the Huduma Poa that provides integrated family planning, reproductive health, malaria, HIV and tuberculosis services. Liverpool VCT, Care and Treatment (LVTCT) Manages a hotline, the C-Word website and SMS RH communication platforms for young people. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 42 ANNEX 2: FINANCIAL ANALYSIS AND FINANCIAL EXPENDITURES63 Table 1: Budget vs. actual expenditure Budget line Budget Expenditure in USD Budget Utilization Labor 2,947,640 2,683,526 91% Fringe Benefits 1,184,981 1,228,806 104% Travel 739,748 955,288 129% Equipment 13,614 4,262 31% Other Direct Costs 9,596,435 10,646,295 111% Sub awards 20,238,657 12,794,288 63% Indirect Costs 1,278,847 1,204,388 94% Total 35,999,922 29,516,856 82% Table 2: Sub-awards budget vs. expenditure analysis64 Organization Total Award Amount in USD Total Expenditure to Date 30 Sept 2014 in USD Changamka 1,196,600 997,925 LVCT 618,384 501,660 Pharmaccess 1,091,250 757,398 SCOPE 276,827 267,962 SWAP 285,714 199,267 UZIMA 300,000 229,432 KMET 784,437 784,437 FHI 360 1,337,186 1,337,186 PS Kenya 13,631,229 7,604,206 63. Source: PSK MIS financial data for the period ending 30 September 2014. 64. Source: PSK MIS financial data for the period ending 30 September 2014. 65. This amount differed with the reported amount of 20,238,657 in the quarterly report by USD 717,028. PSI did provide any explanation for the variance. TOTAL 19,521,62865 12,679,476 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 43 ANNEX 3: SCOPE OF WORK Project details: Project Name: APHIAplus Health Communication and Marketing Program. Implementing Partner: Population Services International (PSI) Contract Number: AID-615-A-12-00002 Project AOR: Emma Mwamburi Life of the Project: March 21, 2012 – March 20, 2015 Total Funding: $ 35, 999,922.00 Period of Project to be Evaluated: March 21, 2012 – September 30, 2014 Type of Evaluation: Performance Evaluation (End of Project Evaluation) Completed Evaluation by: IBTCI Purpose & Background Information Purpose of Evaluation: The purpose of the evaluation is to conduct an end of activity evaluation that will establish the extent to which the expected outcomes have been met and which will also provide recommendations for future activities in the area of health communications and marketing. Introduction: USAID/Kenya’s Office of Population and Health’s (OPH) current strategic objective is to, “Reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated family planning and health services.” It directly supports the Government of Kenya’s (GOK) efforts towards reducing unintended and mistimed pregnancies, improving infant and child health, reducing HIV/AIDS transmission, and reducing the threat of infectious diseases. Background: APHIAplus HCM was developed to support the achievement of USAID/Kenya’s Strategic Objective 3 (SO3): Reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated FP and health services. SO3 focuses on improving the enabling environment for the provision of health services; increasing the use of proven, effective interventions to decrease the risk of transmission and mitigate the impact of HIV/AIDS; and increasing customer use of family planning (FP), reproductive health (RH), and child survival (CS) services. Kenya is also a recipient of significant funding through the President’s Emergency Plan for AIDS Relief (PEPFAR). As such, APHIAplus HCM contributes to the attainment of Kenya’s PEPFAR goals for averting new HIV infections, providing care and support to orphans and vulnerable children (OVC) and people living with HIV/AIDS (PLWA), and providing antiretroviral therapy (ART) to those in need. These United States Government (USG) objectives and goals support and strengthen the GOK’s health sector strategies and goals for the country. The goal of the current program is to improve the health outcomes and impact for Kenyans through sustainable, country-led programs and partnerships. This is achieved through increased use of quality health services, products and information, and strengthened leadership, management and governance for sustained health behaviors. The program focuses on improved International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 44 health outcomes in malaria, HIV, FP, and CS in Kenya. This is achieved through initiatives to improve the capability of the private health market to meet the needs of Kenyans through social marketing approaches. A subsidiary goal of this activity was to support local organizations in the overall implementation in line with USAID Forward objectives. The funding for the integrated APHIAplus HCM activity is $35, 999,922.00 and is allocated for HIV/AIDS-related programming (50 percent), with the remaining funds earmarked for malaria initiatives (30 percent) and CS activities, RH/FP initiatives (20 percent). B: ACTIVITY DESIGN: APHIAplus HCM is different than the social marketing programs of the past, and is expected to transition social marketing toward greater sustainability. Given the vast number of players in the private, public and NGO sector needed to reach Kenyans with critical health products, services and communications, the activity proposed a vibrant locally registered NGO must continue to play a coordinating role to ensure the many different components are working together effectively and efficiently, in line with, and supporting, government health priorities. As such, the program envisions a local entity – PS Kenya - coordinating interventions between private sector partners, the government, donors, and other partners. PS Kenya’s role is to focus on coordination, managing subsidy, and contracting out private sector partners to meet the health needs of Sara and her family. With no single, national level partner marketing partner identified, the activity proposed an approach wherein multiple partners were brought in at different levels. While many of those proposed partners, such as Changamka, K-MET, and grass roots NGOs have been budgeted, the great majority of partnerships that will ensure the success of this activity are not specifically named or budgeted as this was to be determined after initial assessment. Private sector Kenyan entities were proposed to take a leading role in the execution of aspects of product social marketing, leveraging distribution networks and private infrastructure to increase access. By the end of the activity, one or two Kenyan private sector /social marketing entities will be leading the marketing and distribution of a number of social marketed brands, and doing so with modest use of targeted subsidy. Social and behavior change communication (SBCC) initiatives will be led by local marketing and communication entities, informed by communication strategies and briefs led and coordinated by the GOKs Division of Health Promotion (DHP). An umbrella consortium of social franchises in Kenya will be overseen by the Kenya Health Federation (KHF), the leading medical services association supporting the private health sector in the country. The GOK will steward the social marketing initiatives by different partners. Finally a vibrant local NGO partner– PS Kenya - will provide the mainstay of implementation whether acting as a prime recipient or a subsidiary under an international prime. Sustainability Strategy: In the long-term (10-15 years), PSI/Kenya envisions that the number of social marketed products and services will be reduced as fewer people are vulnerable to unintended pregnancy, HIV and other infections, commercial actors are supplying those with high willingness to pay, and public services are serving those with low ability to pay. As that shift International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 45 happens, social marketing efforts and organizations will shift to concentrate on the public health priorities of the future, such as non-communicable disease and more complex service delivery while linking those serving the poor to health financing initiatives and products – much like is done in the west. While this is the long-term vision, in the near future, subsidies will continue to be required to meet the health needs of poor and vulnerable Kenyans. The key to increasing sustainability is better targeting of subsidies to those who need it most, becoming more efficient at delivering subsidized products and services, and diversifying the base from which subsidies are collected. The APHIAplus HCM activity applies three approaches to support an overarching sustainability strategy. 1. Maximizing health impact; 2. Minimizing financial vulnerability; and, 3. Strengthening local institutional capacity. 1. Maximizing Health Impact: The APHIAplus HCM activity is designed to maximize health impact long into the future by expanding the scope and scale of programs. While the partnership proposed that communicable diseases such as HIV and malaria and family planning form the core of the program, programs will aggressively expand interventions to address the untreated diseases that Sara and her family suffer from – such as tuberculosis, pneumonia, and diarrhea. Integrated programming – through providers, outlets, and communications – will ensure one-stop shops are meeting the comprehensive needs of Sara and her family. As the number of diseases addressed through cost effective interventions increases, so will the scale and equity of programs, through improved coordination with private and public sectors, increased partnerships with and linkages to NGOs, and improved program efficiency and effectiveness. The proposed increase in scale and scope of interventions comes as Kenya changes dramatically. A strengthened private sector is increasingly effective at distributing products, and government is making progress in playing a more proactive role as steward of the entire health system. That includes increased focus on total market approaches which leverage each sector to maximize health impact while improving the targeted delivery of scarce donor subsidy. Health impact is maximized by not just ‘doing more’ through social marketing – but by better coordinating work in the context of all sectors in Kenya. 2. Minimizing financial vulnerability: As the role of social marketing evolves, so will investments required to support programs. The APHIAplus HCM partnership’s strategy to minimizing financial vulnerability – and therefore improving sustainability – focuses on three key areas: active price management of social marketed products; the use of health financing initiatives, such as third party purchasing of services and risk pooling / insurance; and program efficiency. Deliberate pricing strategies will ensure affordable health products and services are meeting the needs of all Kenyans in a manner which concurrently maximizes cost recovery. Program income will play a more important role in supporting existing and new products as it becomes a strategically important funding source. Many commodities social marketed by PSI/Kenya are International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 46 already recovering the cost of the commodity and packaging, and there exists significant opportunity over the life of the activity for other products to achieve full commodity and packaging cost recovery. In some instances, such as WaterGuard, the need for subsidy is nearly eliminated, with the private sector recouping costs through sales. Many of those brands are not yet ‘profitable’ for full graduation to the private sector, as cost recovery does not yet include marketing and promotion costs required to support a branded product. But by being able to use program income to procure commodities, donor subsidies can focus on underlying barriers to behavior, rather than being used to support the full cost of a commodity. Sustainability will also be achieved by increasing access to third party payment of services – such as through vouchers and insurance – which will minimize the need for USAID to directly pay for the services private sector clinics are providing. A promising opportunity for sustainable health financing long after the activity ends is a pooled risk health financing product supported through partnerships with private sector partners and the NHIF. By pooling risk, we shift dependency from donors to the government and those with more disposable income. Rather than donors subsidizing the poor, risk pooling enables the healthy to subsidize the sick. A diversified funding base, building on existing partnerships with DFID, the Gates Foundation, and others will ensure that PSI/Kenya will manage subsidy in a way that continues to be targeted to meet the evolving needs of Sara in a cost efficient manner. Finally, value for money initiatives focusing on enhanced efficiencies and program effectiveness will drive programs toward achieving more programmatic impact with fewer resources. The APHIAplus HCM team built in key metrics to better manage, and monitor, value for money metrics. In summary, PSI/Kenya does not expect the need for subsidies for public health and family planning programs to go away. However, innovations to improve targeting of subsidy (through total market approaches; the deliberate management of product and service pricing; diversifying subsidy delivered through third party payment of products and services; and a focus on value for money) will collectively minimize financial vulnerability and maximize sustainable health impact. 3. Strengthened institutional capacity: The APHIAplus HCM program will build the capacity of private sector partners, NGOs and CBOs, and a PSI/Kenya to sustainably improve the health of Kenyans. Just as public sector systems are strengthened in the long run through support to human resources, the introduction and support of priority medical services, training, and improvements in support systems, similar investments are required in the private sector. Program interventions are needed to assist the government to organize the private sector in support of the GOK’s health strategies, and in the process maximize the full potential of the overall health system. The interventions supported in this activity collectively improve the capacity of the private sector to maximize its potential and meet the health needs of Kenya in line with government priorities. These strengthened institutions – clinics, pharmacies, distributors, and shops – will continue to deliver quality health impact long into the future. B.1: ACTIVITY RESULTS FRAMEWORK: International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 47 In 2010, USAID Kenya Office of Population and Health developed a five-year implementation framework with four key results areas. This activity was designed to contribute to the outcomes in result 1 and 3 as shown below: Strategic Objective: Improved health outcomes and impact through sustainable country-led programs and partnerships partnerships Results Framework Strategic Goal: Sustained improvement of health and well-being for all Kenyans Cross-Cutting Elements Whole Market Innovation Gender-Focus Youth-Focus Equity Result 1: Strengthened leadership, management and governance for sustained health programs Result 2: Health systems strengthened for sustainable delivery of quality services Result 3: Increased use of quality health services, products and information Result 4: Social determinants of health addressed to improve well￾being of targeted communities and populations B.2: GOALS and OBJECTIVES Program Goal: To improve the health outcomes and impact for Kenyans through sustainable, country led programs and partnerships. Program Results and expected outcomes IR 1: Increased use of quality health services, products and information IR 1.1: Increase access to and demand for high quality health products and services Key Interventions implemented: HIV Activities Increase Social Marketing of Condoms Increased Scale and Scope of social franchises Support MoH (NASCOP) in condom forecasting and quantification as well as free condom distribution through CBO’s Introduce ARV treatment through social franchise Family Planning and Reproductive Health Activities Social marketing of Femiplan OC’s, injections and condoms Increase uptake of a method mix of FP methods including short and long term methods through social franchises and pharmacies International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 48 Medical detailing to facilitate the adoption of balanced counseling for FP methods through private sector Implementation of communication to address barriers to uptake of contraceptives among married women and unmarried youth Addressing the overlapping risk behaviors of HIV and RH through promotion for dual contraceptive use (condom + other modern method) use Malaria Activities ANC/CWC net distribution leveraging other funding (Continuous LLIN distribution/ Mass net distribution/ Social Marketing and Routine distribution) Addressing barriers to net use through investment in IPC and mass media communication Provide technical support to DOMC PSI will continue supporting the Peace Corps Malaria project Child Survival Activities Identify and engage private sector in the brand licensing of WaterGuard Support supervision for the social franchise for providers to provide IMCI Continue to support private sector in the distribution of bundled ORS & ZINC into community based distribution channels Promotion of hand washing among school going children Capacity building of DCAH staff Expected outcomes for IR 1.1 Demonstrated improved sustainability of franchised network Support to Private Sector Social Marketing Entities Social Franchising – The APHIAplus HCM program proposed to harnesses the private sector through a social franchising approach, improving availability and access to high quality, affordable health products and services Building Capacity of Grass Roots Social Marketing NGOs Targeted Support to Public Sector Distribution Improve provider perceptions toward priority health interventions and address their barriers to desired behaviors through Medical Detailing Programs (MDP) Increase capacity of private sector to provide high quality and high impact health interventions IR 1.2: Improve adoption and maintenance of healthy behaviors Key Interventions implemented: Develop communication in: Men and women 25 – 49 yrs. (urban & rural) & youth 18 – 24 yrs. to Increase the number of couples tested and who know their status Increasing consistent condom use among youth by increasing correct and consistent condom use with all non-spousal partners, from 58.2% to 65%. Increasing consistent condom use among concurrent sexual partners by increasing the number of men and women who report consistent condom use with their side partners Developing a VMMC campaign around sexually active 25 – 35 year old uncircumcised men in Nyanza, Turkana, Teso and Nairobi Develop a campaign around promoting use of modern methods to married couples using correct knowledge of FP methods and dispel myths /misconceptions. Develop a campaign to promote use of modern contraceptives among unmarried youth by addressing Knowledge gaps, societal norms and their Self-efficacy to use modern contraceptives International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 49 Develop a SBCC campaign for diarrhea prevention to increased % of caregivers who know that diarrhea is a serious disease Develop a SBCC campaign for malaria to Increase % of caregivers who know that they are at risk throughout the year SBCC to promote regular blood donation SBCC to reduce demand for medical injections and reduce medical transmission of HIV SBCC to promote OVC awareness and services Hotline for FP counselling and referral – capacity building of counsellors to FP counselling, awareness creation for hotline, referral to services Expected outcomes for IR 1.2 Improved appropriate health care seeking behavior Increase use of health products and services The HCM program used state-of-the-art techniques and work with the DHP and other implementing partners at the national, county and sub county level to motivate healthier behaviors among target populations through the design and development of innovative SBCC campaigns. IR 1.3: Strengthen public‐private partnerships to deliver health communication, marketing and services Key Interventions implemented: Formation of a National Consortium of Franchising Organizations. Improving Quality, Scale and Scope of Networks Reducing out of pocket payments is critical to spur provision of health services in the private sector Provide hotline services to franchise providers. Expected outcomes for IR 1.3 Increased number of public-private partnerships formed and managed by APHIAplus HCM Increased number of private providers accredited to that provide services to defined standards of care IR 2: Strengthened leadership, management and governance for sustained health behaviors IR 2.1: Strengthen the capacity of public sector institutions to promote and oversee social marketing and SBCC initiatives in Kenya Key Interventions implemented: Building SBCC technical & leadership capacity of DHP officers to coordinate social marketing &SBCC strategies at national & county level Development, dissemination and operationalization of the health promotion guiding frameworks to enable them assume the leadership role of HCM initiatives Developing a roadmap for Health Promotion Unit as a Center of Excellence (CoE) in Health Communication & Marketing: Supporting HPU provide Technical Advice to other MOH divisions and departments Formation & Operationalization of National and County platform (Health Promotion Advisory Committees) to drive the Health Promotion Agenda: Using Bottom-Up Approaches to Advocacy Expected outcomes for IR 2.1 Increased MOH oversight and monitoring of SBCC and social marketing interventions. Social marketing and SBCC quality assurance standards and guidelines established National inventory of social marketing and SBCC research, materials and tools managed by MOH International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 50 IR 2.2: Strengthen the capacity of one or more Kenyan entities to implement social marketing and SBCC initiatives Key Interventions implemented: Solicitation of 3 local organizations to receive capacity building and funding on SBCC and SM Capacity building of the 3 LNGOs on SBCC and SM Sub awarding the 3 LNGOs to implement SBCC & SM Evolving PSI Operations in Kenya to PS Kenya Expected outcome for IR2.2: Increased use of local Kenyan organizations and agencies in the essential functions of marketing, advertising, and distribution of products and services, and percent of budget allocated accordingly Increased percentage of the APHIAplus HCM sub-award budget allocated to participating Kenyan organizations IR 2.3: Increase synchronized national and USG‐funded social marketing and SBCC Key Interventions implemented: Strengthening the Health Promotion Advisory Committees as a platform for adopting /adapting national communication for county/regional context Synchronizing Regional & County Level SBCC and Social Marketing Strategies Providing SBCC TA to APHIAplus Service Delivery & USG partners Expected outcomes for IR 2.3 SBCC activities at regional and community levels aligned to national SBCC priorities Reduced duplication of SBCC materials Program Partners Population Services International (PSI) Population Services Kenya (PSK) Department of Health Promotion (DHP) Private Sector Marketing Entity (TBD) Family Health International 360 (FHI 360) Goldstar SafeCare Changamuka Microhealth Limited Kisumu Medical and Education Trust (K‐MET) Liverpool VCT, Care and Treatment (LVCT) B.3: PROBLEM STATEMENT: The Government of Kenya (GOK) gives high priority to the improvement of the health status of Kenyans, and recognizes that good health is a prerequisite to improved socioeconomic development of the country. It has established a number of policies and strategies to address these health problems, and progress has been made in improving the health and well-being of Kenyans over the past five years as documented by the 2008/09 Kenya Demographic and Health Survey (KDHS). Significant challenges, however, remain. HIV/AIDS and Tuberculosis (TB): An estimated 6.2% of adults aged 15-49 in Kenya are infected with HIV. HIV prevalence is highest in the counties of former Nyanza province, Nairobi, and the counties of the former Coast Province. About 130,000 new adult infections and 32,500 new infant infections (via International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 51 vertical transmission) occur each year, but modes of transmission are markedly different in these three provinces. Even though HIV is typically more clustered in urban areas and along transport corridors, increasing prevalence in rural areas has been documented. New patterns of infection have also been documented highlighting discordant couples, casual sex, and Most At-Risk Populations (MARPs). Gains have been achieved over the last ten years. Consistent condom use has increased from 27% to 58% among youth, with similar increases in condom use at last sex. Malaria: More than 70 percent of Kenyans are at risk of malaria. This preventable disease is responsible for the loss of 170 million working days each year and 13 percent of all deaths among children under five (34,000 deaths). Malaria still accounts for 30% of outpatient attendances and 19% of admissions to the health facilities. In the last 10 years major gains have been made in the fight against Malaria. Malaria is no longer the leading killer of children under five, while data from a variety of surveys and operational research show declines in malaria parasite prevalence, malaria trends, and vector densities over the last ten years. Family Planning and Reproductive Health: The modern contraceptive prevalence rate (CPR) is 42%, with an unmet need of 25% (KDHS 2008). Recent studies conducted by PSI/Kenya reveal that unmet need in young women is high, at 53%. However, that unmet need is significantly higher for young unmarried women; 76% of sexually active unmarried women reported a desire to protect against an unplanned pregnancy, but did not use a modern method. The gap between demand for FP methods and use of modern methods – unmet need – is extremely high. Lack of consistent access to contraceptives and misinformation continues to be a barrier to use. Approximately 50% of WRA access reproductive health (RH) services through the private sector. Those access points are pharmacies, unregistered chemists, and private clinics. Providing subsidized Femiplan OCs and injectables through the private sector ensures consistent access to short term contraceptive methods, while the Tunza social franchising model ensures access to quality family planning services as well as subsidized long acting reversible methods. PSI approaches compliment the GoK priorities and investments in the public sector. Role of Private Sector: An assessment of the private health sector in Kenya conducted by the World Bank in 2010 summarized key challenges facing the private importation, distribution, and marketing sector as: “Too many suppliers, too many bad suppliers:” All levels in the supply chain, from importer to distributors to wholesale to retail, are highly fragmented in Kenya. Many of these roles are blurred: some manufacturers may import, distribute, and retail products, while other operators who identify as wholesalers fill only the most basic function of reselling to retailers in restricted areas. “The excessive competition between wholesalers means that few of them can achieve the economies of scale necessary to warrant investment in a national distribution system.” “Too many retailers, too many bad retailers:” Approximately 4,000 out of the 5,500 pharmacies in Kenya are unregistered drug sellers. The vast majority of those retail outlets are located in urban areas. As a consequence, private sector distribution is focused on urban areas. Unlicensed pharmacies, while serving the needs of their communities, too often do so with no training, support, or referral mechanisms in place. “Too many drugs, too many bad drugs:” With an abundance of counterfeits, inferior drugs, and poor inventory management, many consumers do not receive a quality drug supported by accurate information when they present at a pharmacy. Many drugs lack clear instructions, indications, and counter indications; even if a poor Kenyan has purchased a quality drug, misuse may occur through incorrect usage or dosage. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 52 The sector developing media and marketing campaigns is also fractured. While capacity is quite high in a number of traditional media agencies, they are typically focused on media production, development, and purchasing. These agencies will typically produce a campaign on contract, based on a detailed communication brief developed by clients. Development Hypothesis: If the PSI/HCM APHIAplus Activity, jointly with the Kenyan Ministry of Health, is able to harness a local, sustainable private sector through social marketing and communication approaches, then the country will achieve sustainable, an equitable and balanced out health impact, equity and through the targeted use of subsidy among the targeted poor and vulnerable population. B.4: PROGRAM STRATEGY Vision: The APHIAplus HCM program will enable the government to harness the skills and energy of all the players in the health market in Kenya to deliver sustainable health improvements for all Kenyans. In achieving its vision, the team will focus on sustainability: sustainable markets (addressed in IR 1), and sustainable institutions (IR 2). Sustainable Markets: APHIAplus HCM approaches sustainable market interventions using a total market approach (TMA) in order to reach all segments of society with high quality products and services according to their ability to pay. In a balanced market, the poorest access products and services through free distribution, those who are somewhat better off through subsidized products and services, and those with greater ability to pay through commercially available products and services. Social marketing grows the overall market by attracting new user groups into the market and by opening up new markets, particularly in rural areas. A TMA corrects market inequities and develops more sustainable solutions to health problems by providing wider and more effective choices. The APHIAplus HCM will ensure progress toward sustainable markets by focusing on and measuring: health impact, equity and subsidy. Managing any one of these factors in isolation can result in unintended consequences. For example, interventions that focus only on reducing subsidy by increasing the cost of social marketed products/services could result in products and services that are no longer affordable to the poor. Conversely, insufficient management of subsidy often results in markets that cannot be sustained due to their reliance on donor subsidy. To achieve health impact, the HCM will focus on changing behaviors to reduce the percentage of the population at risk, and meeting the product and service needs of those who are at risk. This requires a coordinated focus product and service provision and implementation of SBCC interventions under IR 1. The APHIAplus HCM will focus on equity to ensure that activities are reaching target audiences – poor and vulnerable people in Kenya. The activity measures equity by evaluating socio‐economic inequities in the practice of health behaviors and use of health products and services. Finally, the APHIAplus HCM will manage pricing strategies to ensure appropriate subsidy management. This approach targets subsidies to those who need them, providing the efficient use of donor resources while creating space in the market for viable commercial sector products. C: EVALUATION OBJECTIVES AND QUESTIONS The evaluation will focus on assessing the extent to which the APHIAplus HCM activity has met the program outcomes and will focus on both technical and administrative aspects of the activity. The evaluation approach should include a design and model which assesses implementation, capacity building, market reach, outcomes, and durability. The evaluation team will also provide recommendation for the type and scope of future assistance for health communications and marketing in Kenya. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 53 Evaluation Objectives: To assess the overall performance of the activity in terms of achievement of outcomes as outlined on the activity agreement and performance management plan and other sustainable solutions in social marketing and communication. To identify unique strategies and approaches that facilitated the achievement of the activity’s intended results and health outcomes; To assess the extent to which the creation of a “local” entity (PS Kenya) has been successful and any issues/concerns regarding its organizational and managerial capacity and potential for future success; To document key lessons learnt in the course of activities’ implementation; and To provide recommendations for the need/scope of any potential further activities in the area of Communication and Marketing. Evaluation Questions: To what extent has the activity achieved its mandated outcomes in IRs 1 and 3 along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? To what extent has the activity succeeded in creating sustainable social marketing and communications models for the health sector? What strategies and approaches facilitated the achievements/ creation of such sustainable models if any? What are the key lessons learnt, especially on the establishment of local sustainable social marketing models among the private sector partners, local NGOs, and government departments– at the national and county levels? What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them more effective in delivering sustainable services that reach targeted populations? The evaluation should provide empirical evidence to answer these questions; conclusions will need to be based on findings as recommendations for future mission action will be based on an assessment of the results of the evaluation exercise. Resources provided by USAID There is a comprehensive list of key documents that will form part of this evaluation. These documents will be made available to IBTCI at the appropriate time as determined by the contracting officer. IBTCI consultants are expected to review to inform the findings, conclusions and recommendations. D. EVALUATION METHODS, APPROACHES AND PROCEDURES A mixed method design that uses parallel combinations of methods is being proposed for this evaluation. The methods will be used separately and the findings are integrated after the data are analyzed. It is expected that all team members and/or individual team members at the discretion of the team leader might be involved in implementing multiple evaluation methods including the actual data collection. The timing of data analysis and triangulation of findings will as well be guided by the team leader as on whether they occur concurrently over the same period of time, or at different times. The proposed methods call for a balanced approach to using both quantitative and qualitative methods in answering each evaluation question. The evaluation team will use a combination of techniques including document reviews, key informant interviews, further data analysis, routine data analysis, and exit interviews. Desk Review: The evaluation team will review documents/reports related to the activity, including, but not limited to: technical proposal, work plan, performance monitoring plan, quarterly reports (technical and financial), annual report, key reports on various areas that the activity covered, guidelines/policy documents International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 54 developed with the support from the activity, survey reports conducted by the activity on health impact, equity and subsidy among other relevant documents. Each evaluation team member will focus in her/his areas of competence and will analyze to what extent the documents “respond” to what has been proposed in the proposal and work plan, reporting of results in quarterly and annual reports as appropriate, considering the quantity of inputs/resources available. Detailed list of data sources to be reviewed are included on this SOW. Key Informant Interview/Structured Interview Meetings: The evaluation team will conduct key informant interviews with health care workers from private sector partners, USAID Mission staff, and relevant Ministry of Health staff. Other key stakeholders and implementing partners as appropriate will also be interviewed. A comprehensive list of institutions that should be consulted through KIIs is included as an annex and IBTCI is expected to sample respondents this list. Exit Interviews From a sampled list of private sector social marketing and communication health facilities, the evaluation team will conduct exit interviews on randomly assigned clients visiting Tunza, K-MET and Gold Start Networks (GSN) health facilities and collect information on the equity and subsidy parameters by collecting information on the client’s economic quintile. Socially marketed product subsidies will be evaluated by collecting information on commodity selling price. The health commodity buying price will be collected at the activity level from the activity’s finance team. Further Data Analysis on TraCSurvey & Measuring Access & Performance (MAP) Datasets: Key data sets available to inform evaluation objectives include: TRaC Surveys: Over the activity’s implementation period, randomly sampled household TraCSurveys have been conducted, results analyzed and report disseminated. TRaC is a multi-round survey-based research approach that PSI utilizes to gather information from a representative sample of our target populations. The data collected from TRaC studies present understanding of trends of characteristics of the groups PSI works with such as socio-demographics, health behaviors, knowledge about health products, and much more. Multiple rounds of TRaC surveys were conducted for reproductive health, HIV/AIDS and condom use. Measuring Access and Performance (MAP) studies: Use mapping techniques and technology to measure coverage, quality of coverage, access, equity of access, and efficiency of social marketing products and service delivery systems. MAP studies meet a variety of objectives, including: Define and measure coverage and quality of coverage indicators based on publicly available geographic maps, the population census, and lot quality assurance sampling (LQAS); Define and measure coverage, quality of coverage, access, and equity of access indicators using existing geographic information system (GIS) population layers of geographical areas; Increase the efficiency of PSI field activities through the development of GIS applications to reduce the cost of decision-making, distribution, and field force management, and to increase the quality and utility of management information systems that monitor activities and inform stakeholders. TRaC surveys were to help the activity in monitoring the affordability of health products to ensure that it was not leaving out poor and vulnerable priority population; while MAP studies were to help in monitoring accessibility to and utilization of condoms and FP products among target populations. These two data collection methods were to help the activity monitor equity of access, use, and underlying behaviors in the population. Health impact is measured through coverage of most at risk population with International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 55 condoms, FP products and LLINs. Baseline assessments were conducted and should therefore provide the basis for determining the contribution the activity has made over its performance period. Critical assumptions: Sustainability strategies consist of three components: 1) maximizing health impact, 2) Minimizing financial vulnerability / dependence on subsidy, and 3) strengthened institutional capacity. The activity was to increase availability, affordability and utilization of health commodities and services (leading to improved health impact) while efficiently and effectively targeting subsidies to those most in need. Strengthened local entities are best positioned to be able to balance sustained health impact and targeted use of subsidy. Achieving higher subsidy increases availability, affordability and utilization of the health commodities, but doesn’t necessarily amount to achieving equity and by extension health impact among the priority population. Given that subsidy is a measure of sustainability, it is expected that the activity over time achieved less and less of subsidy in the market. Analysis on Sustainability Indices: Subsidy parameter was to help the activity measure the extent to which the target population was becoming less dependent over time. A demonstration of the market being less dependent on the subsidies through evidence is a strong measure of sustainability which the activity set out to achieve. An analysis of data on total market subsidy, total market value and number of commercial brands in the market is expected to show this. Routine Data Analysis: An analysis of routine data from the private sector health care franchise network which includes Tunza, K-MET and Gold Star Networks facilities will show trends over time on the uptake of services by the priority poor and vulnerable population. MIS, routine activity data, and qualitative studies should also inform evaluation goals. D.1: SAMPLING STRATEGY: Given the two unique units of analysis, that is health facilities and partners/institutions, two independent sampling approaches have been recommended for this SOW to ensure that all units of analysis are adequately represented in the overall sample for the evaluation. Health facilities: PSI/HCM APHIAplus activity currently works Tunza and Huduma Poa health facilities located across the country. For purposes of this evaluation only those facilities with client loads of 600+ have been included on the sampling frame, with a total of 228 health facilities (Tunza, Huduma Poa and GSN) on the frame. To ensure regional representation with an exception of Northern part of Kenya, a systematic sampling approach in every region by facility type (to ensure equal representation by facility type and region) is used to generate recommended sample size of 22 health facilities. IBTCI team is required to use the attached excel spreadsheet which provides specific details on health facilities included on the sampling frame for sampling 22 health facilities which represents 10% of facilities with 600+ clients. Regional Catchment Area Total Sampled Facilities Sample Size=22 Tunza Central 28 3 Tunza Nyanza 19 2 Tunza NEP 4 0 Tunza Coast 16 2 Tunza Rift Valley 24 2 Tunza Nairobi 39 4 Tunza Eastern 30 3 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 56 Tunza Western 23 2 Huduma Poa 23 2 GSN Nrb 6 1 GSN Coast 15 1 GSN Central 1 0 Total 228 22 Partners/Collaborating Institutions: PSI/HCM APHIAplus activity works with many different partners and collaborators at different levels of efforts. At the proposal development, it is suggested that IBTCI meets with the activity team to discuss and develop a clear understanding on the priority partners/collaborators by IRs, after which develop a realistic sample of partners/collaborators that should be consulted in this evaluation. See attached excel spreadsheet for a comprehensive list of partners/institutions /collaborators. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 57 D.2. Illustrative Data Collection Methods and Corresponding Analytical Approaches for Consideration by the Evaluation team Key Evaluation Question Type of Evidence Methods Source Sampling / Selection Data analysis 1. To what extent has the activity achieved its mandated performance targets and health outcomes along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? Comparative/ Analytic/Trend Desk reviews Data Mining KII Structured Interviews Sampled TUNZA health facilities Activity documents TraCSurvey dataset MAP dataset Routine program data Financial data Purposive/ Systematic Random Trend analysis on reported results against targets Comparative analysis on targets, outcomes Quantity of commodities sold, distributed Content analysis on various methodologies used to compute health impact by activity. 2. To what extent has the activity succeeded in creating sustainable social marketing and communications models for the health sector? What strategies and approaches facilitated the achievements/ creation of such sustainable models if any? Comparative/ Analytic Desk Reviews Key Informant Interviews Round table discussions with experts in social marketing field Activity documents Routine program data Survey reports (TraCSurvey, MAP datasets) Social marketing & communicatio n experts Purposive Random Content analysis of the strategic documents developed by the activity to guide implementation, Content analysis of the reports prepared by the progress on key result areas and outcomes Content analysis of interviews data to detect key themes related to implementation and management strategies used Content analysis of data collected, expert validation of the observed sustainable International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 58 social marketing models. 3. What are the key lessons learnt, especially on the establishment of local sustainable social marketing models among the private sector partners, local NGOs, and government departments? Comparative/ Analytic KII Desk Review Expert Consultations Forums Social Marketing & Communicatio n experts PSI staff Local Private sector staff Staff from relevant government department Content analysis on the raw data collected through KII, structured interviews, documents reviews, and expert consultations 4. What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them much more effective in delivering sustainable services that reach targeted population? Comparative/an alytic/triangulati on, expert validation Content analysis of the collected information Validation of findings, conclusions Team brainstorming meetings Key findings, conclusions and recommendati ons validation forum Synthesis and triangulation of information collected International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 59 E. TEAM COMPOSITION: It is anticipated that the evaluation will be carried out by a four person team (“Evaluation Team”) consisting of the following persons with specific expertise and experience: 1) Team Leader (TL): The TL will be an senior expatriate (Health/Population/Nutrition/HIV-AIDS Analyst) in social marketing and communication and will have overall responsibility for fulfilling the requirements of this SOW. S/he will have a master’s degree and significant experience with behavior change communication, social marketing, private sector and monitoring and evaluation is required. Ten years and above of extensive international experience related to health programs and at least seven years in evaluating donor funded activities is required. 2) Evaluation expert: The person will be a senior local (Monitoring and Evaluation or Research Specialist) expert with a master’s degree in Public Health, International Development, Social Science or a closely related field is required. S/he will have significant experience in participatory evaluation methodologies, design, and end of program evaluations with at between 6 – 8 years’ experience in conducting NGO/CBO/FBO level research in Sub-Sahara Africa. 3) 2 other team members: Both will be senior locals, one being a private sector/public finance analyst/insurance analyst, while the other being a social scientist/other technical advisor with a strong understanding of the Kenyan market context, social marketing theory, and behavior change communications as well as sustainability experience. Evaluation Management: IBTCI will provide overall direction to the Evaluation Team; avail all the key project documents, provide all the logistical support required to perform this evaluation. IBTCI/MTR team shall be responsible for arranging all roundtable discussions, Key Informant Interviews (KII) and booking meeting places. IBTCI is responsible for quality control and delivery of the required report as agreed to by USAID. IBTCI shall be responsible for arranging all domestic travel and hotel arrangements for the selected county health executives listed below. F. REPORTING REQUIREMENTS & DELIVERABLES: Reporting Requirements: IBTCI will be supervised by and the written report reviewed for acceptance by OPH Deputy Director or her designate and by APHIAPlus HCM’s Agreement Officer Representative. IBTCI will work collaboratively with USAID staff and implementing partners, though IBTCI is fully responsible for meeting the objectives in this Statement of Work. Briefings: The Evaluation team will provide regular in-country briefs to USAID/Kenya on progress and discuss problems and issues on a bi-weekly basis. Additional debriefings will be convened as required by either party. Proposal/Work plan: The evaluation team will provide a detailed proposal/work plan to USAID before commencing the evaluation. The proposal/work plan will outline how the evaluation will be undertaken and the methods to be used considering the proposed methods in this SOW. The work plan must be approved by USAID/Kenya before commencing field work. In-Country Presentation: The evaluation team will make an in-country PowerPoint presentation with handouts to USAID on the main findings at the end of the evaluation. A. Draft Report: Acceptance of the draft report by USAID/Kenya will be contingent upon the report adequately fulfilling the scope of work and addressing major important areas of inquiry International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 60 outlined in the scope. The draft report will follow the required format for the evaluation as listed below: i) Executive Summary ii) Table of Contents iii) Main body of the Report iv) Recommendations v) Conclusions vi) Annexes B. Final Evaluation Report. Upon final approval of the content by USAID/Kenya, IBTCI will have the report edited and formatted. The final report will be submitted both electronically and in hard copy. Four hard copies of the report will be provided to USAID/Kenya. In addition, all the raw data will be submitted to USAID on CD labeled “HCM APHIAplus Data” for future reference. Once USAID approves the final report, IBTCI will submit it and all the final evaluation-related information products to the Development Experience Clearinghouse (DEC) as provided for in the activity contract. IBTCI is responsible for ensuring that the final evaluation report includes all criteria listed in Appendix 1 of USAID’s Evaluation Policy. The final evaluation report shall have a maximum of 30 pages: 1. Executive Summary—concisely state the most salient findings and recommendations (2 pg); 2. Table of Contents (1 pg); 3. Introduction—purpose, audience, and synopsis of task (1 pg); 4. Background—brief overview of development problem, USAID project strategy and activities implemented to address the problem, and purpose of the evaluation (2-3 pg); 5. Methodology—describe evaluation methods, including constraints and gaps (2 pg); 6. Findings/Conclusions/Recommendations—for each evaluation question (15 - 17 pg); 7. Issues—provide a list of key technical and/or administrative, if any (1pg); 8. Future Directions (2 - 3pg); 9. Annexes —that document the evaluation methods, schedules, interview lists and tables should be succinct, pertinent and readable. These include references to bibliographical documentation, meetings, interviews and focus group discussions. Deliverables: IBTCI must ensure that all objectives in Section C, Evaluation Description and Objectives, are met using the evaluation methods, approaches, and procedures stated in Section D. Additionally, all the reporting requirements in this Section F must be delivered within the time frame of the contract. Finally, the Scope of Work must be carried out by team members who meet the key personnel requirements in Section E, Team Composition. G. PERIOD AND PLACE OF PERFORMANCE The period of performance for this evaluation is 10 weeks (2.5 months). The evaluation will begin by August 1, 2014 and end no later than October 15, 2014. The place of performance is Nairobi, Kenya. A six-day work week is authorized under this contract without premium pay. H. QUALITY EVALUATION REPORT: International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 61 IBTCI is expected to review USAID’s requirements and expectations on the draft and final reports as detailed on the “Checklist for Assessing Evaluation Reports”, see Annex I. It is important to note that USAID will subject the structure and content of the report to the parameters outlined on the checklist and will use this as a basis for accepting and/or rejecting the reports. I. THREATS TO VALIDITY: IBTCI is required to manage the Evaluation Team and guard against any possible threats to validity of findings, conclusions and recommendations drawn from the qualitative methods. Any conclusion drawn from the qualitative data sources must be supported by well-grounded body of evidence that is triangulated and confirmed. It is therefore expected that IBTCI will take the Evaluation Team through the parameters outlined on the USAID’s “Checklist for Reducing Threats to Validity for Qualitative Methods”. J. ESTIMATED COST: The proposed magnitude for this Scope of Work is $300,000. International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 62 EVALUATION REPORT CHECKLIST - V1.0 Title of Study Being Reviewed: _________________________________________ Main Implementer(s): Reviewer: Date of Review: GOOD PRACTICE ELEMENTS OF AN EVALUATION REPORTi Keyed to USAID’s 2011 Evaluation Policy EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments STRUCTURE OF THE REPORT 1. Does the evaluation report have a cover sheet attached indicating the type of evaluation conducted (e.g. performance evaluation or impact evaluation) and general design? 2. If a performance evaluation, does the evaluation report focus on descriptive and normative evaluation questions? 3. If the evaluation report uses the term “impact evaluation,” is it defined as measuring the change in a development outcome that is attributable to a defined intervention (i.e. impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual)? 4. Regardless of the type of evaluation, does the evaluation report reflect use of sound social science methods? 5. Does the report have a Table of Contents (TOC)? 6. Do Lists of Figures and Tables follow the TOC? 7. Does the report have a Glossary of Terms? 7.1 Are abbreviations limited to the essential? 8. Is the date of the report given? 9. Does the body of the report adhere to the 20 page guide? 10. Is the report well-organized (each topic is clearly delineated, subheadings used for easy reading)? 11. Does the report’s presentation highlight important information in ways that capture the reader’s attention? 12. Is the report well written (clear sentences, reasonable length paragraphs, no typos, acceptable for dissemination to potential users)? 13. Does the evaluation report focus on the essential issues concerning the key International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 63 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments questions, and eliminate the “nice to know”, but not essential information? 14. Does the evaluation report discuss any issues of conflict of interest, including the lack thereof? 15. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? EXECUTIVE SUMMARY 16. Does the evaluation report begin with a 3- to 5-page stand-alone summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation? 17. Does the Executive Summary concisely state the main points of the evaluation? 18. Does the Executive Summary follow the rule of only saying what the evaluation itself says and not introducing new material? INTRODUCTION 19. Does the report introduction adequately describe the project? 19.1. Does the introduction explain the problem/opportunity the project was trying to address? 19.2. Does the introduction show where the project was implemented (physical location) through a map? 19.3. Does the introduction explain when the project was implemented? 19.4. Are the “theory of change” or development hypotheses that underlie the project explained? (Does the report specify the project’s inputs, direct results (outputs), and higher level outcomes and impacts, so that the reader understands the logical structure of the project and what it was supposed to accomplish?) 19.5. Does the report identify assumptions underlying the project? 19.6. Does the report include sufficient local and global contextual information so that the external validity and relevance of the evaluation can be assessed? 19.7. Does the evaluation report identify and describe any critical competitors to the project that functioned at the same time and in the project’s environment? 19.8. Is USAID’s level of investment in the project stated? 19.9. Does the evaluation report describe the project components funded by implementing partners and the amount of funding? 20. Is the purpose of the evaluation clearly stated? 21. Is the amount of USAID funding for the evaluation indicated? 22. Are all other sources of funding for the evaluation indicated as well as the amounts? 23. Does the report identify the evaluation team members and any partners in the evaluation? 24. Is there a clear statement of how the evaluation will be used and who the intended users are? 25. Are the priority evaluation questions presented in the introduction? 26. Does the evaluation address all evaluation questions included in the Statement of Work (SOW)? 26.1. Are any modifications to the SOW, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline indicated in the report? 26.2. Is the SOW presented as an annex? International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 64 26.3. If so, does the annex include the rationale for any change with the International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 65 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments written sign-offs on the changes by the technical officer? SCOPE AND METHODOLOGY 27. Does the report provide a clear description of the evaluation’s design? 27.1. Is a design matrix or similar written tool presented in an annex that shows for each question/subquestion the measure(s) or indicator(s) used to address it, the source(s) of the information, the type of evaluation design, type of sampling if used, data collection instrument(s) used, and the data analysis plan? 28. Does the report state the period over which the evaluation was conducted? 29. Does the report state the project time span (reference period) covered by the evaluation? 30. Does the evaluation report indicate the nature and extent of consultation on the evaluation design with in-country partners and beneficiaries? 31. Does the evaluation report indicate the nature and extent of participation by national counterparts and evaluators in the design and conduct of the evaluation? 32. Does the report address each key question around which the evaluation was designed? 33. Is at least one of the evaluation questions directly related to gender analysis of outcomes and impacts? 33.1. Are data sex-disaggregated? 34. In answering the questions, does the report appropriately use comparisons made against baseline data? 35. If the evaluation is expected to influence resource allocation, does it include information on the cost structure and scalability of the intervention, as well as its effectiveness? 35.1. As appropriate, does the report include financial data that permits computation of unit costs and analysis of cost structure? 36. Is there a clear description of the evaluation’s data collection methods (summarized in the text with the full description presented in an36.1. annex)? Are all tools (questionnaires, checklists, discussion guides, and other data collection instruments) used in the evaluation provided in an annex? 36.2. Does the evaluation report include information, as appropriate, on the pilot testing of data collection instruments? 36.3. Does the evaluation report include information, as appropriate, on the training of data collectors? 37. Are all sources of information properly identified and listed in an annex? 38. Does the evaluation report contain a section describing the limitations associated with the evaluation methodology (e.g. selection bias, recall bias, unobservable differences between comparator groups, small samples, only went to villages near the road, implementer insisted on picking who the team met with, etc.)? 39. Does the evaluation report indicate the evaluation methodology took into account the time, budget, and other practical considerations for the evaluation such as minimizing disruption and data burden? 40. Does the report have sufficient information to determine if the evaluation team had the appropriate methodological and subject matter expertise to conduct the evaluation as designed? 41. If an impact evaluation was designed and conducted, does the evaluation report indicate that experimental methods were used to generate the strongest evidence? Or does the report indicate that alternative methods for assessing impact were utilized and present the reasons why random assignment strategies were not feasible? International Business & Technical Consultants Inc. APHIAPlus HCM, End-term Evaluation 66 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments 42. Does the evaluation report reflect the application and use to the maximum extent possible of social science methods and tools that reduce the need for evaluator-specific judgments? 43. Does the evaluation scope and methodology section address generalizability of the findings? ANALYSIS 44. Are percentages, ratios, cross-tabulations, rather than raw data presented, as appropriate? 45. When percentages are given, does the report always indicate the number of cases used to calculate the percentage? 45.1. Is use of percentages avoided when the number of cases is small (<10)? 46. Are whole numbers used or rounding-off numbers to 1 or 2 digits? 47. Are pictures used to good effect? 47.1. Relevant to the content 47.2. Called out in the text and placed near the call-out 48. Are charts and graphs used to present or summarize data, where relevant? 48.1. Are the graphics easy to read and simple enough to communicate the message without much text? 48.2. Are they consistently numbered and titled? 48.3. Are they clearly labeled (axis, legend, etc.)? 48.4. Is the source of the data identified? 48.5. Are they called out in the text and correctly placed near the call-out? 48.6. Are the scales honest (proportional and not misleading by virtue of being “blown-up”)? FINDINGS 49. Are FINDINGS specific, concise and supported by strong quantitative and qualitative evidence? 49.1. As appropriate, does the report indicate confirmatory evidence for FINDINGS from multiple sources, data collection methods, and analytic procedures? 50. Are adequate data provided to address the validity of the “theory of change” or development hypothesis underlying the project, i.e., cause and effect relationships? 51. Are alternative explanations of any observed results discussed, if found? 52. Are unplanned results the team discovered adequately described? 53. Are opinions, conclusions, and recommendations kept out of the description of FINDINGS? CONCLUSIONS 54. Is there a clear distinction between CONCLUSIONS and FINDINGS? 55. Is every CONCLUSION in the report supported by a specific or clearly defined set of FINDINGS? 56. Are the CONCLUSIONS credible, given the FINDINGS the report presents? 57. Can the reader tell what CONCLUSIONS the evaluation team reached on each evaluation question? RECOMMENDATIONS 58. Are RECOMMENDATIONS separated from CONCLUSIONS? (Are they highlighted, presented in a separate section or otherwise marked so that the reader sees them as being distinct?) 59. Are all RECOMMENDATIONS supported by a specific or clearly defined set of FINDINGS and CONCLUSIONS? (Clearly derived from what the evaluation team learned?) International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 67 EVALUATION REVIEW FACTOR 1 2 3 4 5 Reviewer Comments 60. Are the RECOMMENDATIONS practical and specific? 61. Are the RECOMMENDATIONS responsive to the purpose of the evaluation? 62. Are the RECOMMENDATIONS action-oriented? 63. Is it clear who is responsible for each action? 64. Are the RECOMMENDATIONS limited/grouped into a reasonable number? LESSONS LEARNED 65. Did this evaluation include lessons that would be useful for future projects or programs, on the same thematic or in the same country, etc.? 66. Are the LESSONS LEARNED highlighted and presented in a clear way? 67. Does the report indicate who the lessons are for? (e.g., project implementation team, future project, USAID and implementing partners, etc.) BOTTOM LINE 68. Does the evaluation report give the appearance of a thoughtful, evidence- based, and well organized effort to objectively evaluate what worked in the project, what did not and why? 69. As applicable, does the evaluation report include statements regarding any significant unresolved differences of opinion on the part of funders, implementers and/or members of the evaluation team? 70. Is the evaluation report structured in a way that will promote its utilization? 71. Does the evaluation report explicitly link the evaluation questions to specific future decisions to be made by USAID leadership, partner governments and/or other key stakeholders? 72. Does the evaluation report convey the sense that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? 73. Have all evaluation team members signed a statement attesting to a lack of conflict of interest, or describing and existing conflict of interest relative to the project being evaluated? 74. Was the Report Submitted to the Development Experience Clearing House (DEC)? 75. Has a dissemination plan been developed for this report? 76. Is the report widely shared to interested stakeholders? DEFINITIONS: Performance evaluation: focuses on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual. Impact evaluation: measures the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a ―treatment‖ or a ―control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 68 ANNEX 4: DATA COLLECTION SCHEDULE ACTIVITY DATES (2014) Literature review October 16-29 Briefing by PSK October 31 Presentation of methodology and work plan to USAID October 29 Round Table Group Discussions (RGDs) HCM Consortium partners Social franchise partners USG service delivery partners County Health Promotion Officers (HPOs) Sub-county Health Promotion Officers Community Based Organizations Ministry of Health Central GOK departments Private Sector enterprise partners November 3-6 Field visit and interviews LNGOs Suppliers Distributors Pharmacists Country Health Officials Country Health promotion officers (CHPOs) PSK fields managers and workers Franchise clinic owners and staff November 17-18 Key informant interviews (KII) Division of Child and Adolescent Health Division of Malaria Control Division of Reproduction Health National AIDS and STI Control Programme (NASCOP) Health Promotion Unit Community Health Unit PSK Board members PSK staff members Britam Beta HealthCare National Health Insurance Fund Private sector suppliers, distributors and retailers Department of International Development (DIFID) Kenya Health Federation USAID staff November 19-29 (alongside analysis) International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 69 ANNEX 5: CONSULTANTS’ CVS IAIN MCLELLAN Nationality: Canadian Affiliation: IBTCI Position Title: Team Leader Labor Category: Health/Population/Nutrition/HIV-AIDS Analyst Education: 1977 MA Political Science, University Aix-Marseille, Aix-en-Provence, France 1974 BA Communications/Political Science, Goddard College, Plainfield Vermont, USA Relevant Experience: Mr. McLellan has over 30 years of experience in communications focusing on Social and Behavior Change Communications related to the health sector. He has designed and implemented communication programming in multiple countries for a variety of donors. Since the late 1980’s Mr. McLellan has worked primarily as a BCC expert consultant conducting assessments and evaluations as both a team member and team leader. Selected Professional Experience: 1987-Present Consultant Care UK Evaluation Team Member Evaluated illegal migrant worker project in India, Bangladesh and Nepal focusing on support to migrant communities and HIV prevention. Social Impact Evaluation Team Leader End-term evaluation of USAID’s $21 million activity in India “Improving Health Behaviors” implemented by FHI360 and PSI. LTL Strategies Evaluation Team Leader Conducted evaluation of USAID’s AIDE project in Djibouti focusing on teacher training, reading and youth job skills creation. GHTech/USAID Evaluation Team Member Evaluated BCC interpersonal communication and social mobilization component of Maternal and Neonatal health project in Bangladesh. IBTCI Review Team Leader Reviewed ten years of USAID health sector programming in Mali and provided recommendations for future design of SBCC project. USAID/Rwanda Evaluation Team Leader Conducted mid-term assessment of Rwanda Livelihoods project. Conducted desk review, field research and led in the writing of the report. Johns Hopkins University International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 70 Updated NetWorks impregnated bed nets BCC promotion strategy for Senegal and trained print material and radio producers in BCC; analyzed existing behavior research on reproductive health and child survival in Haiti and developed IEC strategy; analyzed existing research on family planning in Burkina Faso and developed protocol for future research and IEC strategy. FHI360 Assessed the use of telephone hotlines, mobile texting and the internet to communicate with HIV most at risk populations in Jamaica. GHTech/USAID Evaluation Team Member Evaluated PSI/PACT evidence-based targeted HIV prevention project in Malawi covering MARPS outreach and condom social marketing. GHTech/USAID Evaluation Team Member Responsible for social marketing, mobilization and BCC components of USAID/Benin evaluation of two large integrated health projects. Research Triangle International Evaluation Team Member Evaluated Injecting Drug Users project in Telavi Georgia and developed BCC strategy for MARPs. Management Sciences for Health Aided Haiti South Ministry of Health in preparation for HIV social mobilization action plans and revised guide on BC planning. Academy of Educational Development Trainer Conducted training in SBCC of media-related NGOs in DRC and assisted in developing strategic plans. Georgia Ministry of Health Developed research plan and terms of reference for planned public health campaigns on issues surrounding influenza. GHTech Wrote an RFA and strategic plan for PEPFAR on preventing the sexual transmission of HIVE for USAID/Haiti. FHI360 Produced guide on developing BCC materials on treatment of opportunistic infections based on experience in Rwanda and Kenya; Developed formative assessments and a regional workplace BCC working in collaboration with the international labor organization in Ghana; conducted BCC strategic planning workshop and developed HIV workplace and stigma reduction strategies for Rwanda; Developed BCC strategy to reduce stigma, promote VCT and prevent MTCT of HIV in Ghana; Conducted assessments and developed BCC materials on nutrition for PLWHA in Rwanda and Zambia. PACT Evaluated PACT NGO and FBO sub-grantee’s capacity in BCC and conducted training in BCC planning, assisted in developing BCC materials for peer education; provided technical assistance to NGOs and FBOs targeting youth, OVCs and PLWHAs in Botswana, Lesotho and Namibia. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 71 Population Services International Reviewed interpersonal communication with high risk groups including field visits to India and Nigeria then designed and wrote a toolkit for field use; Trained PSI staff in BCC strategic planning for HIV and malaria prevention in Rwanda; developed formative research with marketing firm and designed work plan for ORS social marketing project in Morocco. DFID/OPTIONS Team Leader Conducted final review of mass media HIV project implemented by BBC World Trust Service in Nigeria; Conducted final review of HIV project using interpersonal communication mid and mass media to reach MARPs in Nigeria. OPTIONS/EU Developed policy papers for an advocacy campaign to encourage increased allocation of funding for HIV spending in Russia. World Health Organization Staff consultant on global AIDS programming in the Health Promotion Unit acing as condom social marketing focal point; conducted training and wrote and reviewed national plans; developed training materials and presented at workshop in Latvia and Albania. UNICEF Evaluated HIV stigma and conducted workshop with stakeholders to develop strategy to reduce stigmatization in health and education among PLWHA in Moldova; co-wrote paper on use of advocacy and social mobilization for HIV and AIDS prevention in Eastern and Central Europe; evaluated developing countries farm radio network for possible collaboration with UNICEF; evaluated social mobilization of EPI vaccination campaign in Senegal. Languages: English (native); French (native) MARTIN RIUNGU Nationality: Kenyan Affiliation: IBTCI Position Title: Team Member Labor Category: Senior Public Sector Finance Analyst Education: 2011, MBA Finance, University of Nairobi 2008, Certified Information Systems Auditor, Illinois USA 2006, B.Com Finance, University of Nairobi, Kenya 2005, Certified Public Accountant of Kenya (CPA (K), Nairobi Kenya Relevant Experience: Mr. Riungu is a finance expert with over10 years’ experience. He has undertaken evaluations and reviewed donor funded programs specifically in monitoring and evaluation, fund and grants management, advocacy and outreach, marketing and communications as well as knowledge management. Mr. Riungu’s experience cuts across various sectors including public finance, local government, health and public service delivery in different countries within Africa. He is adept at providing professional services to International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 72 promote accountability in financial/audit practices and procedures to improve efficiency with focus on donors and donor funded projects. Selected Professional Experience: 2003 – Present Consultant Danish Refugee Council Evaluation Team Member Finance expert for mid-term evaluation of Community Driven Recovery and Development (CDRD) to develop informed and accountable local governance through user empowerment (dialogue) program in Mogadishu and Beletweyne regions of Somalia. Government of Kenya Team member Public finance expert in the preparation of finance and accounting manual for the county governments of Kenya. DFID Evaluation Team Member Provided monitoring of fund management for DFID-funded Girl Education Challenge (GEC) program in sub Saharan Africa 2013 in Rwanda. MOH, Kenya and World Health Organization Assessment Team Leader Assessed service availability and mapping of the facilities in eight counties to scope the services and interventions being provided in Kenya. USAID/Kenya Evaluation Team Leader Reviewed and audited Kenya Medical Supplies Agency (KEMSA) support program on specific mission concerns regarding the management of the project and locally incurred costs for the period ending 9 May 2013. World Bank, South Sudan Multi-Donor Trust Fund (MDTF) Team Member Finance and procurement expert in the review of reconstruction of South Sudan programs at the national level and conducted regular evaluations on financial processes and procedures. Global Fund-Kenya, Zambia, Somalia and Eritrea Team Member Senior finance specialist assessing the readiness of the principal recipients for the implementation of the Global Fund and monitored subsequent funds and programs on behalf of global fund secretariat. Martin was the senior finance specialist. COMESA Evaluation Team Leader Evaluated all COMESA member countries on compliance with donor requirements, laws and regulations; procedures to verify grant incomes, expenditure and capacity building statistical agencies on financial management. USAID East Africa-GOK Assessment Team Leader Assessed procurement and financial management of over 20 organizations which included government ministries, State corporations and semi-autonomous government agencies. Embassy of Denmark Team Member Audited Joint Financial Arrangement for the African Union Liaison Offices for the period January 2009 to December 2010. Martin was a senior Auditor. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 73 Government of Rwanda/World Bank Team Member Computerization of the core functions of the Ministry of Economic Planning and Finance (MINECOFIN). Languages: English; Kiswahili International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 74 NICHOLAS DONDI Nationality: Kenyan Affiliation: IBTCI Position Title: Team Member, Behavior Change Communication specialist Labor Category: Senior Social Scientist Education: 2003, MA Communication, Almeda University, USA 1978, Post basic training in Management, Communication and Research Methods, Daystar University 1972, Diploma in Journalism and Communication, University of Nairobi Relevant Experience: Mr. Dondi is a consultant in social development and Behavior Change Communication. He has participated in numerous assignments including field assessments, analysis, planning, program implementation, monitoring and evaluation, proposal development and communication. Mr. Dondi has worked in more than 18 countries across Africa and the Pacific, supporting the work of international organizations, governments, NGOs and communities. Selected Professional Experience: 1990 - Present Consultant UNICEF Evaluation Team Member Evaluated Sierra Leone school based education program. WORLD VISION Team Member Behavior Change Communication consultant for Canada’s multi-country ENHANCE project. AFRICAN UNION Team Member Developed Avian Influenza Behavior Change Communication strategy for SPINAP project supporting 47 African countries. GOVERNMENT OF KENYA Team Leader Designed Butula constituency development strategic plan in Kenya covering aspects of development, health and entrepreneurship. PATH/UNIVERSITY RESEARCH CORPORATION Team Member Developed education and counseling materials for infant and young children feeding in Ethiopia. UNICEF ESARO Team Member Developed outbreak communicators’ toolkit for Eastern and Southern Africa. UNICEF Team Member Developed HIV/AIDs education and training materials for Fiji. ERITREA Team Leader Conducted a situation analysis and developed the water and sanitation Behavior Change Communication strategy. NAMIBIA EPI PROGRAM Communication Team Leader International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 75 Reviewed the international Namibia EPI Program. UNICEF/BASICS Team Member Developed IEC policy and behavior change communication strategy for Eritrea Evaluated community partnership for health program in Nigeria. UNICEF/WHO Team Member Developed integrated management of childhood illnesses for Uganda. PATH Communication Lead Member of international task force on hepatitis that introduced hepatitis B immunization in Kenya. ACADEMY FOR EDUCATIONAL DEVELOPMENT Team Member Developed the integrated prevention of mother to child transmission of HIV and community motivator course for Zambia. Languages: English; Kiswahili REBECCA NJUKI Nationality: Kenyan Affiliation: IBTCI Position Title: Team Member, Monitoring & Evaluation and Research expert Labor Category: Senior Social Scientist Education: 2015, PhD Sociology, Ghent University, Belgium (Dissertation submitted) 2013, MA Sociology, University of Nairobi, Kenya 2010, MSc and PGD in Epidemiology, London School of Hygiene and Tropical Medicine 2005, BA Anthropology, Moi University Relevant Experience: Ms. Njuki has over eight years’ experience as a social scientist with a focus in program management, research, monitoring & evaluation and advocacy. She has worked with multiple donor funded programs International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 76 in the East African region. Ms. Njuki has excellent skills and understanding in developing performance monitoring plans, M&E work plans, quantitative and qualitative data analysis, survey methodology, policy analysis, data management & quality assurance as well as documentation and dissemination of results. Selected Professional Experience: 2006-Present Consultant FAMILY CARE INTERNATIONAL Senior Technical Advisor/Country Manager Leading implementation of maternal health advocacy, monitoring and evaluation in Kenya for five (5) key projects; Mobilizing Advocates from Civil Society, Social Accountability, Respectful Maternity Care, and Price too high to bear for maternal mortality research project and Misoprostol for PPH. YOUTH EMPOWERMENT ALLIANCE Team Leader Evaluation consultant on a baseline study on uptake of Sexual and Reproductive Health (SRH) services among young people aged 10-24 and the underserved groups in twelve counties in Kenya. CARDNO EMERGING MARKETS/DFID Team Leader Research consultant on qualitative formative survey for the Private Sector Innovation Program for Health taking place across 14 counties in Kenya. CANADIAN CORPORATION/MICRONUTRIENT INITIATIVE Team Member Assessment of MNCH commodities supply chain and mapping the current national stock status and consumption trends for selected MNCH commodities in all public health facilities in Kenya. POPULATION COUNCIL Research Analyst End line Evaluation of the impact of voucher and accreditation approach on improving reproductive health behaviors and RH status in Kenya and Uganda. Researched social dimensions of gender and health to understand HIV/AIDS and Sexual Gender Based Violence (SGBV) in Kenya. Researched and analyzed maternal and child death audits and conduct implementation research aimed at designing, testing, and evaluating an approach to significantly reduce disrespectful and abusive (D&A) care of women during labor and delivery in facilities. AFRICAN POPULATION AND HEALTH RESEARCH CENTRE (APHRC) Coordinator NAIROBI URBAN DSS SYSTEM Managed research and longitudinal studies within the NUHDSS. KEMRI-WELLCOME TRUST KILIFI Research Officer Research and project management of a large field malaria epidemiological study and mortality surveillance studies. Languages: English, Kiswahili International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 77 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 78 ANNEX 6: BEHAVIOR CHANGE FRAMEWORK FOR SOCIAL MARKETING PSI’s Evidence-Based Approach to Behavior Change and Social Marketing. PSI’s approach to increasing condom use, uptake of CT, reducing concurrent partnerships, etc. is based on a behavior change framework called PERForM (a PERformance Framework for social Marketing). This approach, as set out in figure 1 below, demonstrates how PSI (1) delivers products, services and information (2) to enable (3) poor (4) and vulnerable populations (5) to change their behavior (6) and lead healthier, happier lives. In other words, PERForM connects the goal (improved health and quality of life) with the purpose (risk-reducing behaviors and use of products/services), outputs (e.g., the opportunity, ability and motivation to use a product/service), and activities (the social marketing intervention). Figure 1: Social Marketing Performance Framework (PERForM) For example, with condom social marketing, PSI influences the “Social Marketing Intervention” level (1) by providing quality condoms, at a price and place that is affordable to the poor and vulnerable (3 & 4) and with promotions that generate increased demand. This social marketing intervention is evidence￾based in that it relies on surveys and statistical analysis to design, implement, monitor and continuously improve programming. The social marketing intervention influences the target population’s opportunity, ability and/or motivation (2) to change their behavior (5) such that they increase consistent condom use and therefore reduce their risk of contracting or transmitting HIV and STIs, which leads to improved health and quality of life (6). International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 79 ANNEX 7: LIST OF DOCUMENTS REVIWED BY THE EVALUATION TEAM 1. Signed CA_AID-615-A-12-00002 APHIAPlus HCM PSI; 2. HCM Partner Institutions Stakeholders by IR; 3. HCM SOW External Evaluation Final; 4. Change Package for Quality Improvement in OVC Programmes in Kenya 5. Child Status Index Swahili OVC Booklet 6. Community Health Workers Basic Module Manual 7. Essential Malaria Action Guide for Kenyan Families 8. Health Promotion Strategy for Kenya 9. Health Promotion Unit_PIP Action Plan and Budget 10. Malaria Community Strategy 11. Malaria Prevention and Treatment-A Community Education Training Manuea 12. Minimum Service Standards for Quality Improvement of OVC Programmes in Kenya 13. National Guidelines for HTC in Kenya 2010 14. National Guidelines for Diagnosis, Treatment and Treatment of Malaria in Kenya 2013 15. National Health Community Guidelines 16. National HTC Strategy 2014-2018 17. National Infection Prevention and Control Guidelines for Health Care Services in Kenya 18. National Malaria Strategy 2009-2017 19. Operational Guidelines and Standards for Health Promotion 20. Participants Manual-Diagnosis, Management and Prevention of Malaria 21. Reproductive Health and Maternal Services Unit SBCC Capacity Assessment Report March 2013; 22. The Balanced Counselling Strategy Plus-Trainers’ Guide; 23. The Balanced Strategy Counselling Guide – Users’ Guide; 24. VMMC Strategy and Operational Plan 2014 – 2019 25. PS Kenya Education through listening Communication Facilitation Training Guide 26. PS Kenya IPC Handbook 27. PS Kenya SBCC Module; 28. Assessing Clustered Training and Implementation of Malaria IPC Interventions; 29. Evaluation of LLIN Distribution Campaign; 30. Kenya Malaria Indicator Survey, 2007; 31. Kenya Malaria Indicator Survey, 2010; 32. Kenya MIS 2010 Protocol Draft; 33. Malaria Qualitative Study Presentation; 34. Mapping of Target Outlets for the RDT Project in Mombasa and Msambweni Kenya 2013; 35. PSK RH Government Distribution Report, 2012 36. PSK RH Other Baseline and Endline Survey Unilever 2013 37. Trust Brand Equity Study, 2012; International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 80 38. Kenya Condom Social MARKETING Program 2001 – 2011 PSK Summary Report 39. HIV Insight into Packaging and Labeling for HIV Oral Self-Test Kits in Kenya 2013; 40. HIV Insight into potential users and messaging for HIV Oral self-test kits in Kenya 2013; 41. HIV Mpango wa Kando Campaign Pre-test 2012; 42. HIV Nakufeel Campaign pre-test 2012; 43. Draft 2013 HIV TRaC Survey Report; 44. HIV TRaC Concurrency 45. HIV Key Findings MARP Logo Pre-test, 2012; 46. Malaria Community Net Replacement Presentation 2013; 47. Malaria Pre-test 2013 48. Malaria Bednet use study 2013 49. Malaria Musimu Wote Communication Campaign, 2013 50. MNCH Diarrhoea Pre-test Presentation 2013 51. MNCH Nutrition Communication Campaign Report Final 2013; 52. RH Generic Family Planning Communication CAMPAIGN 2013 53. RH Client Exit survey 2012; 54. RH Report FoQus Providers 2012; 55. MAP 2012 Sales 56. MAP 2013 Sales 57. Study on the Uptake Barrier and Drivers of Modern FP Methods; 58. HIV Condom Marketing Plan 2012 59. HIC CSP Marketing Plan 2012; 60. HIV HTC Marketing Plan 2012 61. HIV Situation Analysis 2011 62. HIV VMMC Marketing Plan 2012; 63. MCH Malaria Marketing Plan 2011 64. MCH Child Survival Diarrhoea Marketing Plan 65. MCH Malaria Marketing Plan delta; 66. Branding and Marketing Plan; 67. Femiplan Trade Marketing Implementation Plan for 2014; 68. RH Marketing Plan summary for Femi plan 2014; 69. RH Marketing Plan summary for Older married 2014; 70. RH Marketing Plan summary for Youth 2014; 71. Environmental Mitigation and Monitoring Plan March 2012-April 2014; 72. HCM Quarterly Report July – Sept, 2012; 73. HCM Quarterly Report Oct – Dec 2012; 74. HCM Quarterly Report Jan – Mar, 2013; 75. HCM Quarterly Report Apr – Jun, 2013; 76. HCM Quarterly Report July – Sept, 2013; 77. HCM Quarterly Report Oct – Dec, 2013; 78. HCM Quarterly Report Jan – Mar, 2014; 79. Monitoring and Evaluation Plan FY 2012; 80. Work Plan April 2012 – Sept 2013; International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 81 81. Work Plan Oct 2013 – Sept 2014; 82. Blood Safety Brochure; 83. Blood Safety World Blood Donor Day Certificate; 84. Blood Safety Banner; 85. Blood Safety Blood Donor Notification Booklet; 86. Blood Safety Donor Notification Jijue Poster 87. Blood Safety Event Poster; 88. Blood Safety Flyer; 89. Blood Safety Hepatitis; 90. Blood Safety HIV Flier; 91. Blood Safety HIV Blood Safety KNBTS Logo 2009; 92. Blood Safety KNBTS Brochure; 93. Blood Safety KNBTS Certificate PK 2009; 94. Blood Safety Syphilis Brochure; 95. Blood Safety Solder Brochure; 96. Blood Safety Young lady Brochure; 97. Blood Safety Young Man Brochure; 98. Blood Safety Life saver flyer; 99. Blood Safety Muslim Man; 100. Blood Safety Muslim shortie; 101. C Word Booklet; 102. C Word Flipchart; 103. C Word Banners; 104. Femi plan Injection Brochure 105. Active Management of the Third Stage of Labour; 106. Administering Magnesium Sulphate Flow Chart 107. Emergency Contraception Brochure; 108. IUD Brochure; 109. Labour Ward Flow Chart; 110. Lactational Amenorrhea Brochure; 111. Levonorgesterl Intrauterine System Brochure; 112. Male Condom Brochure; 113. Mini Pill Brochure; 114. Ministry of Health Kenya Revised Partograph 2012; 115. Monthly Injection Brochure; 116. Procedure for use of Vacuum Extractor; 117. Progestin Only Injectables; 118. Standard Days Methods; 119. The Pill Brochure; 120. Tubal Ligation Brochure; 121. Two day Method Brochure; 122. Vasectomy Brochure; 123. Withdrawal Brochure; International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 82 124. HTC ABC of HIV 125. HTC Guideline for use of the client referral form; 126. Injection Safety Avoid needle stick 127. Injection Safety Occupational Brochure; 128. Injection Safety Occupational Patient Record; 129. Injection Safety Occupational Poster; 130. Injection Safety Oral Treatment; 131. Injection Safety Timing of care for occupational exposure to blood; 132. MCH Malaria Phase II Posters 0I; 133. MCH Malaria Phase II Posters 02; 134. MCH Malaria Phase II Posters 03; 135. MCH Malaria Phase II Posters 04; 136. MCH Malaria Phase II Posters 05; 137. MCH Malaria Phase II Posters 06; 138. MCH Malaria Phase II Posters 07; 139. MCH Malaria Phase II Posters 08; 140. MCH Malaria Phase II Posters 09; 141. MCH Net use Print Executions; 142. MCH Promotional Items; 143. MCH SGC Malaria Flip; 144. MCH Child Survival Diarrhea ORS/Zinc caregivers; 145. MCH Swahili AMFM Poster; 146. OVC HIV Booklet 147. OVC Nick and his HIV Story Booklet 148. OVC World Orphans Day Banner 149. Secure the Future – A Condom SBCC Guide 150. VMMC Message Business Leaders; 151. VMMC Message Faith Leaders; 152. VMMC Booklet Post Surgery 2010; 153. VMMC Booklet Women 154. VMMC Community Dialogue Cards; 155. VMMC Flier Centers 2010; 156. VMMC Low Cost Leaflet 157. VMMC Nyanza Community Guide; 158. VMMMC Urinal Poster; 159. VMMC Venue Poster 160. Caregiver Behavior change for child survival 2014; 161. Evidence of population level behavior change child survival; 162. Evidence review of gender integration in reproductive health 2014; 163. M-health and media for behavior change evidence 2014; 164. Systematic review of effectiveness of mass media interventions for child health; 165. Retail Audit write up; 166. HIV TRaC Survey Report 2013; International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 83 167. Taking a TMA in Kenya Presentation; 168. RH_TRaC_2012_Report; 169. HIVTRaC_2013_Report; 170. RH_WRA_TRAC_2011_Study_Design; 171. HIV_TRaC_2013_Study Design; 172. HCM Partners, Institutions, Stakeholders by IR; 173. KE_Questionnaires_RH_ClientExit_2013; 174. KE_Methodology_RH_Client_Exit_2013; 175. HCM_PMP_LOP_Targets_March-2014_Baseline_Indicated; 176. SUSTAINABILITY INDICES_PSI_16 July; 177. PSKE RM HIV Insights into Potential Users and Messaging for HIV Oral Self-Test Kits in Kenya_Rhoune_2013; 178. PO_Devolution_CountyMapping_wn_080413. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 84 ANNEX 8: RGD/KII MATRIX OF RGD/KII PARTICIPANTS No. Roundtable Group Discussion Target Group No. Invited No. Confirmed No. Attended % Confirmed No. Follow￾up KIIs 1 HCM Consortium Partners - SM 8 7 7 100% 2 2 HCM Consortium Partners - SF 4 4 3 75% 1 3 USG Service Delivery Partners 5 5 5 100% 0 4 County Health Promotion Officers 9 9 9 100% 0 5 CBOs 9 9 9 100% 0 6 Sub-County Health Promotion Officers 9 9 8 89% 0 7 MOH National Government Depts. 11 10 6 60% 8 8 Private Sector Enterprises 10 8 5 63% 7 Total 65 61 52 18 List of KII Respondents RESPONDENTS Name Designation Gender USAID Emma Mwamburi PSI AOR, USAID/Kenya Franchising F Daniel Gatheru Wacira Malaria Program Management Specialist M DR Sheila Macharia Senior Health Advisor, FP/RH F Jerusha Karuthiru Project Management Specialist, FP/RH F Consortium partners (SM, SBCC) Peter Mwarogo Country Director, FHI360, Nairobi M Lawrence Mbae, Executive Director, Gold Star and Secretary to ASFH M John Adongosi FHI360 and Gold Star Consultant M Eunice Achola Finance Manager K-MET, Kisumu F Celestine Gambo, Project Officer, KMET, Kisumu F Alice Eleneld Country Director Safe Water and Aids Project (SWAP), Kisumu F Chrispus Owaga Program Director Safe Water and Aids Project (SWAP), Kisumu M Edith Ali Program Manager Safe Water and Aids Project (SWAP), Kisumu F Juliet Ndolo Finance Manager, Safe Water and Aids Project (SWAP), Kisumu F Development & funding partners Ketema Aschenaki Bizuneh Chief, Health Result Area, UNICEF/Kenya, Nairobi M Christopher Ommi PMTCT and Pediatric HIV, UNICEF/Kenya, Nairobi M Eunice Ndung’u CSD Officer, Community Health, UNICEF/Kenya, Nairobi F Valentina Buj Malaria Programs Specialist, UNICEF/Kenya, Nairobi F Milka Choge Reproductive Health Advisor, DFID Kenya, Nairobi F MOPHS International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 85 Anne Ng’ang’a, Program Manager Program manager, DRH (formerly of NASCOP) F Isabella Mdwiga Health Promotion Officer, Health Promotion Unit, MOPHS F F S K Sharif Former Director, Public Health and Sanitation. Now Director of Community Health M Jonah Mwangi Jonah Mwangi, Program Manager Reproductive Health Commodities M Ruth Ngechu Community Health Information Systems, CHS Unit, MOPHS F Gladys Somoren Gladys Somoren, FP Program Officer, DRH F James Kibet Sang James Kibet Sang, Division of Malaria Control M Private sector Richard Sigey Manager, National Hospital Insurance Fund, Nairobi M Rajen Patel Rajen Patel, Director, Radbone Clark Kenya Ltd M Amit Thakker Director, Kenya Health Care Federation, Nairobi M Winfred Mwende Muhsya Haripharm Pharmaceuticals Ltd., Nairobi M Mayark Harma Beta Health Care M Alice Mwalimo Beta Health Care F James Irungu Micro Insurance Manager, Britam Kenya M Patrick Kihuria Assistant manager Micro Insurance Britam Kenya M Charles Mugondi General Manager Bancassurance & Micro insurance, Britam Kenya M County officials met during flied trips in Kisumu and Kiambu Dickens Onyango Acting County Director of Health, Kisumu M Jeremiah Ongwaya County Health Promotion Officer, Kisumu M Visgaal Shah Kentons Pharmacuticals Ltd., Kisumu M Dr D M Wanyee, County Health Director, Kiambu F Joseph Ng’ang’a County Health Promotion Officer, Kiambu M Social Franchise facility owners and workers met during field trips in Kisumu and Kiambu Kisumu and Kiambu Maurice Odhiambo Ojowi Medical Director, Ahero Medical Centre (Huduma Pao) M Jane Awuor Odicoh Administrator, Ahero Medical Centre (Huduma Pao) F George Kapito Clinical Officer, Geva Family Health Services (Huduma Pao), Kisumu M PSK Board members Salma Mazrui Chairperson, PSK, Nairobi F Nelson Gitonga PSK Board Member, Insight Health Advisors, Nairobi M Milton Lore Milton Lore, PSK Board Member, Land O’Lakes Inc., Nairobi M PSK staff in Nairobi Anthony Okoth Chief Executive Officer, PSK, Nairobi M Joyce Wanderi-Maina Chief Operating Officer, PSK, Nairobi F Lucy Maikweki Director, SRH F Wawira Nyagah Director, Partnerships and Outreach, Nairobi F Milka Choge Reproductive Health Advisor, DFID/Kenya F Alex Njeru Finance Director M Monica Waga Monica Waga, Finance Manager F Silvier Wamuhu Sales Director F Stephen Aloo Research Manager M Beatrice Syomiti Beatrice Syomiti, Monitoring and Evaluation Manager F PSK staff members met during filed trip in Kisumu and Kiambu Wahome Macharia PSK Regional Program Coordinator for Greater Nairobi Region that includes Kiambu, Kajiado and Nairobi M Jeniffer Wamaitha Regional Social Franchise SBCC Coordinator, PSK, Nyandarwa, F International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 86 Kiambu, Muranga Catherine Mwaura Regional Quality Assurance Officer , KSP Kiambu F Michael Karani Regional Trade Development Manager, PSK, Kiambu M George K.O Ayoma Social Franchise SBCC Coordinator, PSK, Nyanza M Kennedy Otiano Ayoo Social Franchise Quality Assurance Officer, PSK Kisumu M Evans Odhiambo Oduour Regional Program Manager, PSK, Kisumu M Ambrose Oudong Ochieng Trade Development Manager, PSK, Kisumu F Simon Mboyano Director, Health Service Delivery Department, PSK, Kisumu M Dorcas Odondo Supply Chain Director, PSK, Kisumu F Sylvia Wamuhu Sales and Distribution Director, PSK, Kisumu F International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 87 ANNEX 9: RGD/KII GUIDES 1 PARTNERS WORKING IN SOCIAL MARKETING (SM) AND SOCIAL BEHAVIOR CHANGE COMMUNICATIONS (SBCC) PSI/Kenya, LVCT Health, SWAP, SCOPE, UZIMA, Changamka Microhealth Ltd., and FHI 36 1. How is PS Kenya structured for implementation of APHIAPlus HCM? Please describe the structure from head office to the village? 2. How does the consortium work? Please describe the mechanisms and processes that hold together and operationalize the consortium. Social Franchises 3. How does the project work with social franchises?  To what extent has the project harnessed the private sector through social franchising to improve the availability and access to high quality, affordable health products and services?  To what extent has the project developed and expanded the scale and scope of social franchises? 4. What successes and challenges have social franchises experienced in (1) providing health product and services and in (2) implementing SM and SBCC activities and addressing barriers to health seeking behaviors? 5. To what extent has HCM been successful in creating a national consortium of franchising organizations?  What is the membership of the consortium?  What are the activities of the consortium and what have been its achievements to date?  What are the prospects for the consortium becoming sustainable and expanding its role? 6. What progress has been made in setting up a hotline for social franchise providers? Private sector 7. What progress has been made in creating and developing public-private partnerships to deliver SM, SBCC, SM and social franchising and provide quality health products and services? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 88  Through what mechanisms and structures does the partnership work?  How well is the partnership working?  What else needs to be done to strengthen the partnership further? 8. To what extend has HCM’s contributed to building SM and SBCC capacity in the private sector?  To what extent has APHIAPlus HCM empowered the private sector to play an expanded role in providing quality health products and services?  What support has the project given to the private sector to distribute the bundled ORS & Zinc into community based distribution channels? 9. How successful has the project been in identifying and developing the capacity of one to two private sector entities to play a greater role in distribution of key health products? NGOs and CBOs 10. How has the project involved NGOs and CBOs in its work?  To what extent has the project involved CBO’s in SM/SBCC and product distribution?  What mechanisms and structures are in place to facilitate engagement with NGOs and CBOs?  To what extent has the project succeeded in identifying and building the capacity of two to three LNGOs to lead SBCC activities?  What capacity has the project built in the selected NGOs? What capacity has it build in other NGOs and CBOs? What was the level of involvement of DHP in building the capacity of the NGOs and CBOs? 11. What sub-grant awards has HCM given to NGOs and CBOs to facilitate their work? 12. What have been the successes and challenges of this partnership? Department of Health Promotion 13. What progress has HCM made in building the capacity of the DHP and developing a center of excellence at the department?  To what extent has APHIAPlus built the capacity of DHP to provide leadership in SM and SBCC, provide TA and coordinate SM and SBCC activities of other MOH departments and the counties? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 89  To what extent has the Project developed innovative state-of-the-art techniques and work processes at DHP and other IPs at the national, county and sub-county level to motivate healthier behaviors among target populations?  Has the Project set up a national inventory of SM and SBCC research, materials and tools under the management of MOH? County level 14. Has the Project formed and operationalize the National and County Health Promotion Advisory Committees to drive the health promotion agenda and act as platforms for adopting/adapting national HCM plans for local implementation? 15. To what extent are the county and community level SBCC and SM initiatives, strategies and messages synchronized with national SBCC and SM priorities? Other MOH departments 16. To what extend has APHIAPlus HCM’s contributed to building SM and SBCC capacity in other MOH departments?  What is the nature and progress of building SM and SBCC capacity building of DCAH, DOMC, DRH and NASCOP? 17. To what extent has HCM supported NASCOP in condom forecasting, quantification and free condom distribution through CBOs? How successful has the effort been? Support to social and USG partners 18. Who are PS Kenya’s partners in APHIAPlus HCM?  What capacity building responsibilities does HCM have towards its partnership?  What capacity building support has the project given to its social development and USG partners? 19. What support does the project give to Peace Corps on its Malaria project?  What were the successes and challenges experienced in supporting the Peace Corps? 20. What success has the project had in reducing duplication of SBCC and SM communication materials? SM and SBCC approaches and the contribution they have made 21. How have the various approaches used in SM and SBCC contributed to the achievement of results? Please discuss the contribution of these and other approaches. International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 90  Media campaigns (how many were done during the project)  Road shows and community events (how many were done during the project)  IPC approaches?  CWord, social media and related websites  Medical detailing as a way of promoting adoption of balanced counselling for FP methods through the pharmacies and retailers  Hotline and associated referrals referral contributed to the overall achievements?  Various modes of distribution  Access to services by the voucher and insurance systems 22. What kinds of messages and materials have been effective in delivering messes to key audiences? Outcome and impact 23. To what extent has HCM SM/SBCC activities succeed in increasing uptake and utilization of health products and services through: (1) the public sector (2) private sector distribution (3) pharmacies and (4) social franchising?  Which sector has brought more clients into the health market? Why has it been more successful in doing so? 24. How has the project applied the total market approach to attain market sustainability in the context of equity of access, subsidy management and impact? With what success:  In the general population  Among women  Among young people 25. To what extend has APHIAplus developed a sustenance strategy for every product? Counselling and testing, condom sales and distribution, FP methods, mosquito nets, ORS, Zinc, etc.) Transitioning of PSI to PS Kenya 26. To what extent has PSI transitioned to a Kenyan entity?  What are the strengths and foreseeable challenges in the transformation?  How well is the new entity positioned provide leadership and build SM/SBCC capacity in the country? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 91 Water Guard 27. What progress has been made in graduating and brand licensing of WaterGuard to the private sector? Standards and certification 28. What progress has been made in establishing standards and accrediting private providers to provide services to defined standards of care?  To what extent has the project established SM and SBCC QA standards and guidelines?  How will certified entities benefit from certification? 29. To what extent has the project established SM and SBCC standards and guidelines? Funding 30. To what extent has the project leveraged additional finding?  What activities have been funded with the additional funding? Q1: ACHIEVEMENTS AND IMPACT: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? 31. To what extent has the HCM project succeeded in developing sustainable markets that reach all segments of society with quality health products and services according to the audience ability to pay? 32. What SM/SBCC model(s) have HCM succeeded in developing?  What are the discrete components of the model(s)?  What are the strengths and weaknesses of the model(s)?  How sustainable are the model(s)?  What are the challenges and threat to sustainability?  What can be done to increase the models’ sustainability? Q2: SUSTAINABLE To what degree is the social marketing and communication model that has been created by APHIAPlus HCM sustainable? What has facilitated project achievements and the creation of sustainable models? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 92 33. To what extent has HCM contributed to building sustainable institutions that can deliver effective SM and SBCC programs that market high quality health products, services and information? Q3 LESSONS LEARNED: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 34. What have been the bottlenecks, best practices and lessons learn from implementing APHIAPlus HCM? What have we learnt especially in the following areas:  Using the total market approach to develop sustainable markets with a focus on equity, subsidy and health impact  Developing sustainable institutions with the capacity to deliver quality health services and products  Reaching youth with health messages, products and services  Reaching women with health message, products and services  Reaching the poor, the vulnerable and rural populations with health messages, products and services  Building the capacity of the DHP to oversee and monitor SM and SBCC activities at the national and country levels and provide TA to other MOH departments  Evolving PSI to PS Kenya  Institutionalizing the national consortium of franchise organization  Providing TA to APHIAPlus social development partners, USG partners and SBCC partners  In supervising social franchises, social and commercial providers to offer quality products and services  Implementing SM, IPC and franchise activities to address barriers to the uptake of health products and services among target populations  Supervising and supporting social franchisee providers to offer quality products and services  Establishing standards, certification and licensing Q4 RECOMMENDATIONS What change do you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 93 2. SOCIAL FRANCHISING ROUNTABLE GROUP DISCUSSION GUIDE Partners in social franchising – M-NET, FHI 360/GoldStar Network, PSI/Kenya/Tunza, Huduma Poa and PharmAccess Goals and general operations of franchise networks 1. What entities form the APHIAPlus franchise partnership?  When and why were the various entities formed and why?  What do they specialize in? What do they do outside the HCM partnership?  What contribution does each entity make in the HCM partnership? 2. How do the franchise networks relate to and work with the agencies that helped form them (PSI, FHI 360, etc.)?  How do they work with HCM?  How do they work/collaborate with each other? Franchise coverage 3. How are the various franchises structured for service delivery from top to the village level? 4. What is the coverage of the various franchise networks? Give details of coverage by:  Services offered  Geographical area  Client characteristics Contribution to HCM work 5. To what extent has the uptake and utilization of quality products and services occurred through public sector, SM, pharmacies and social franchises? 6. To what extent have the individual franchise networks contributed to the uptake of quality health products and services under the HCM umbrella? Consortium of franchising organizations 7. Has the consortium of social franchising been formed?  What are its benefits?  What is the membership of the franchise? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 94  What are its operational structures?  What are its goals and objectives?  What activities has it carried so far?  What have been its successes and prospects for the future? Capacity building of franchises 8. To what extent has HCM built the capacity of the franchises to deliver quality health services and products and increase clients? Please discuss capacity building in broad terms, including:  Training  Supportive supervision  Equipment and tools  Funding 9. To what extent has association with HCM led to the extension of the scale and scope of the work of social franchises? NGOs and CBOs 10. What is the extent of using NGOs and CBOs to disseminate messages and distribute products? 11. What progress has been made in soliciting, building the capacity of 3 LNGOs and entering sub-grants with LNGOs to implement SM and SBCC initiatives? 12. To what extent has the Project built the capacity of LNGOS and Grass Roots CBOs in social marketing and SBCC? 13. How do franchises work with NGOs and CBOs? 14. How has the work of NGOs and CBOs contributed to the increase or decrease of client loads at clinics managed by franchises? Medical detailing 15. What has been the contribution of the medical detailing team? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 95 of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? Effect of APHIAPlus activities and support 16. What PSI/K support have you found to be helpful? 17. What PSK/K-provided tools have you found helpful in your work? 18. What activities conducted by HCM and in the community have helped increase uptake of health products and services at franchise clinics and in commercial outlets? 19. What communication materials and activities have been helpful and which ones have had challenges? Please explain. Successes and challenges 20. What successes and challenges have franchises experienced providing quality health services and products? MODEL Q2: To what degree is the social marketing and communication model that has been created by APHIAPlus HCM sustainable? What has facilitated project achievements and the creation of sustainable models? 21. What SM/SBCC model(s) have HCM succeeded in developing?  What are the discrete components of the model(s)?  What are the strengths and weaknesses of the model(s)? 22. How stainable are the models developed by HCM? How sustainable is the services currently offered by social franchises? What are the challenges and threat to sustainability? What can be done to increase the sustainability of both the models created and the services provided by the social franchises? LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 23. What have been the bottlenecks, best practices and lessons learn from participating in the implementation of HCM? Please list all the lessons learnt, including those learnt in the following areas:  Providing quality health products and services  Gaining institutional sustainability to deliver health impact, equity and subsidy International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 96  Working with HCM  Working with NGOs, CBOs and other local structures  Increasing scale and scope of provision of health services and products RECOMMENDATIONS Q4: What change so you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? 24. What recommendations can you make to help future SM, communication and health programs deliver even more effective services that reach all identified target groups? 3. PRIVATE SECTOR Distributors from Fast Moving Consumer Goods (FMCG) and Pharmaceuticals Distributors (PD) groups; Mahitaji Enterprises Limited, Jay Kay Enterprises, Sojpar Ltd., Jatomy Enterprises, Victoria Pharmaceuticals, Beta HealthCare, Transwide Pharmaceuticals and Lifecare Pharmaceuticals 1. What categories of private sector entities are participating in promoting the goals of the HCM project? Please list the category of entities and the roles they play. 2. What progress has been made in creating and developing public-private partnerships to deliver SBCC, SM and social franchises?  How do private sector entities work with APHIAPlus?  How do private sector entities work with GOK departments?  How is the private sector contribution to HCM coordinated and managed? Performance 3. How well have the different public-private partnerships used to deliver health products worked? Please discuss the strengths and challenges of all public-private sector approaches used including the following:  Manufacture  Manufacturers’ marketing support  Bulk distribution, including the “pull” system  Wholesalers  Pharmacies International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 97  Bicycle delivery  Shop and kiosk vending  Household Capacity building 4. What support has the private sector received from the HCM projects to build the SM capacity of the sector to deliver quality health products to the market from manufacture to the village? Please discuss support at all levels, including the following:  Manufacture  Manufacturers’ marketing support  Bulk distribution, including the “pull” system  Wholesalers  Pharmacies  Bicycle delivery  Shop and kiosk vending  Household PS Kenya tools and approaches 5. What tools and approaches used by PSK have you found to be useful to your department and your work as a person? Licensing Water Guard to the private sector 6. What is the progress in graduating and brand licensing of Water Guard to private sector? 7. What other health brands on promotion have the potential to transition to private sector licensing and marketing in the foreseeable future? Interaction with franchises, central and county government systems 8. How do private sector players interact with the following systems in the course of their work?  HCM central  HCM regional staff  Government health management systems at the national, provincial and county levels International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 98  Health facilities managed by the public sector  Heath facilities managed by the franchise networks  Government franchise employed community health workers 9. What have been the benefits and challenges of those interactions? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? MODEL Q2: To what degree is the social marketing and communication model that has been created by APHIAPlus HCM sustainable? What has facilitated project achievements and the creation of sustainable models? 10. What models have been created in public-private partnerships through the HCM project?  Please describe the discrete components of those models.  Please describe the public-private sector model created by the project. 11. How sustainable are the public-private partnership models created by HCM How?  What are the challenges and threats to the models and continued contribution of the private sector to public health good, including continued distribution of health products?  What needs to be done to address the challenges and threats and increase sustainability? LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 12. What have been the bottlenecks, best practices and lessons learn by private sector entities participating in the implementation of HCM? Please list all the lessons learnt, including those learnt in the following areas:  Distributing health products for lower market segments  Distributing health products with increased information and counselling  Gaining institutional sustainability to deliver health impact, equity and subsidy RECOMMENDATIONS International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 99 Q4: What change so you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? 4. GOK AGENCIES Ministry of Health-National government departments and agencies including: HPU, NASCOP, Division of Child and Adolescent Health (DCAH), Division of Reproductive Health (DRH), Division of Malaria Control (DOMC), Orphans and Vulnerable Children (OVC) Secretariat, Directorate of Preventive and Promotive Health Services (DPPHS), Community Health Services Unit (CHSU), National AIDS Control Council (NACC), National Blood Transfusion Service. Work relations with HCM 1. How do the various MOH department work with HCM?  How has HCM contributed to the work of the various departments? Capacity building 2. How has HCM contributed to building capacity in the various MOH departments?  What successes and challenges have been experienced in building capacity in the various MOH departments including the following:  NASCOP  Division of Child and Adolescent Health (DCAH)  Division of Reproductive Health (DRH)  Division of Malaria Control (DOMC)  Orphans and Vulnerable Children (OVC) Secretariat  Directorate of Preventive and Promotive Health Services (DPPHS)  Community Health Services Unit (CHSU)  National AIDS Control Council (NACC)  National Blood Transfusion Service (Capacity building at DHP will be discussed a little later below)  What approaches have been used to support NASCOP in condom forecasting and quantification and in free condom distribution through CBOs? How successful has this initiative been? Department of Health Promotion 3. What progress has the project made in building SM and SBCC technical and leadership capacity of the DHP and its officers to provide leadership and coordinate SM and SBCC strategies and activities at the national and county levels? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 100 4. To what extent has the Project developed innovative state-of-the-art techniques at DHP and other IPs at the national, county and sub-county level to motivate healthier behaviors among target populations? 5. To what extent has the HPU been transformed into a Centre of Excellence (COE) in HCM? 6. To what extent is the HPU providing technical advice to other MOH departments and coordinating their SM and SBCC activities? PS Kenya tools and approaches 7. What tools and approaches used by PS Kenya have you found to be useful to your department and your work as a person? National and county coordination 8. Did the Project form and operationalize the National and County Health Promotion Advisory Committees to drive the health promotion agenda using bottom-up approaches as a platform for adopting/adapting national HCM materials for local use? Explain 9. To what extent are the national, county and community level SBCC and SM strategies, messaging ad activities synchronized? 10. What have been the successes and failures in reducing the duplication of SBCC and SM communication materials? 11. Did the Project set up national inventory of SM and SBCC research, materials and tools under the management of MOH? NGOs and CBOs 12. What is the extent and progress of using CBOs for SM/SBCC in distribution of health products?  To what extent has the Project built the capacity of Grass Roots SM NGOs to carry out this work?  What is the progress in solicitation, capacity building and sub-award contracting with 3 LNGOs to implement SM and SBCC activities?  What has been the level of involvement of the DHP in this process? Franchise 13. How do the various MOH departments work with the social franchises? 14. What has been the main contribution of the franchises in delivery of health services and products? 15. How successful has HCM been in improving the quality, scale and scope of franchise networks? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 101 16. What progress has been made in the formation of National Consortium of Franchising organizations? Public-private private partnerships 17. What progress has been made in creating and developing public-private partnerships to deliver SBCC, SM and services through social franchises?  What structures are in place to promote the partnerships?  And what roles are the structures playing? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? Effect of major promotions 18. What have been the nature, effects and achievements of the campaigns HCM has implemented over the last two to three years? Please discuss the campaigns, including the following:  Hand washing in schools and in the community  Condom promotion  HIV testing  Discouraging concurrent sexual relations  Promotion of FP methods  Blood donation and blood safety  Injection use and safety  Malaria and use of mosquito nets  Voluntary male medical circumcision  Promotion of OVC standards of care  What were the challenges of the campaigns?  How could future campaigns be planned and managed better? Evolvement of PSI 19. What progress has been made in evolution of PSI to PS Kenya? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 102  What opportunities and challenges does the evolution present?  How could the challenges be addressed? Accreditation 20. What progress has been made in accrediting private providers to offer services to defined standards of care?  What benefits and challenges does accreditation come with?  How can the challenges be addressed? MODEL Q2: To what degree is the social marketing and communication model that has been created by APHIAPlus HCM sustainable? What has facilitated project achievements and the creation of sustainable models? 21. To what extent has HCM been successful in developing sustainable markets for health products and services that prioritize equity and impact through effective management of subsidies? 22. To what extend has HCM been successful in developing a sustenance strategy for the products it promotes? Counselling and testing, condom sales and distribution, FP methods, mosquito nets, ORS, Zinc, etc.) 23. What SM and SBCC models have been developing in the course of implementing HCM?  Please describe the components of the models created.  What are the challenges and threats to the sustainability of the models?  What can be done to address the challenges and threats and improve sustainability? LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 24. What have been the bottlenecks, best practices and lessons learnt by the public sector from APHIAPlus? Please list all the lessons learnt, including those learnt in:  Developing institutional capacity to attain market sustainability based on equity, subsidy and health impact using SM, SBCC and the total marketing approach  Building the capacity of DHP to oversee coordinate and monitor SM, SBCC strategies and initiatives at National and County levels?  The transitioning of PSI to a Kenyan entity International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 103  Offering health products and services through franchises  Offering health products and services through public-private partnerships RECOMMENDATIONS Q4: What change so you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? 5. COUNTY GOVERNMENT OFFICERS From Makueni, Muranga, Nyeri, Uasin Gishu, Meru, Siaya, Kisumu, Kajiado and Kiambu countie. Two sessions:  County Health Promotion Officers (CHPOs) and  Sub-County Health Promotion Officers (SCHPOs) Job position and relationships 1. What is the work of County Health Promotion Officers and Sub-county Health Promotion Officers?  How do they relate to the Department of Health promotion? Role played in HCM activity 2. What role have County and Sub-county Health Promotion Officers played in the implementation of HCM? 3. What support have County and Sub-county Health Promotion Officers received from APHIAPlus and its partners to help them make a stronger contribution?  What additional support would County and Sub-county Health Promotion Officers like to receive? Social franchises 4. One of the ways through which HCM uses to deliver health products and services to clients is social franchises.  What do you know about the social franchises?  Which franchises do you work with in your area?  How do you work with the franchises?  What are the benefits of delivering health services through social franchise facilities? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 104  What challenges come with delivering services through social franchises?  And what steps can be taken to address the challenges and improve performance? Private-public partnerships 5. HCM also works through the private sector.  What HCM-private sector partnership activities do you know of?  How do you work with HCM private sector partnerships in you county?  How is this component of HCM work helping quality health products to reach people of different social economic categories? Please describe.  What challenges do these activities experience?  What can be done to address the challenges and improve performance? NGOs and CBOs 6. HCM works with local NGOs and CBOs that carry out behavior change IPC activities in the community and distribute health product, such as condoms.  What do you now about this aspect of HCM work?  What CBOs and NGOs implement these activities in your county?  What role do County and Sub-county Health Promotion Officers play in these activities?  What challenges do these activities experience?  What can be done to address the challenges and improve performance? Department of Health Promotion 7. What support do County and Sub-county Health Promotion Officers get from the DHP? Please describe the detailed support received in these and other areas:  Work instructions, guidelines and guidance  Policies, strategies and wok guidelines  Training and mentoring  Supervision  Job aids  Communication materials  Other – specify International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 105 8. What are the strengths of DHP? 9. What are the main challenges of DHP? 10. How can the challenges be addressed? 11. Are you aware of the efforts that HCM has made to develop SM and SBCC capacity at DHP? Please describe What capacity would you like to see developed at DHP? What support would you like to see DHP give to County and Sub-county Health promotion officers and the county health structure? National and county coordination 12. How do counties coordinate health communication activities across MOH departments and with other sectors? 13. What mechanisms are in place to facilitate coordination? 14. What would be the ideal coordination mechanisms? 15. To what extend to counties and County and Sub-county Health promotion officers contribute to the development of national SM and SBCC policies and strategies? 16. And how do they go about customizing nationally developed policies, strategies and materials for local use? 17. To what extent are the national, county and community level SBCC and SM strategies, messaging and activities synchronized? 18. What have been the successes and failures in reducing the duplication of SBCC and SM communication materials? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? Desired county SM and SBCC capacity? 19. What SM and SBCC capacity would you like to see developed at the country level? MODEL Q2: To what degree is the social marketing and communication model that has been created by HCM sustainable? What has facilitated project achievements and the creation of sustainable models? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 106 20. HCM set out to develop sustainable markets for health products and services while prioritizing equity and impact through effective management of subsidies. To what extend has the project achieved this goal? 21. To what extend has HCM been successful in developing sustenance strategies for the products in promotes? (Counselling and testing, condom sales and distribution, FP methods, mosquito nets, ORS, Zinc, etc.) 22. What SM and SBCC models have been developed by HCM?  Please describe the components of the models created.  What are the challenges and threats to the sustainability of the models?  How can the challenges be addressed to improve sustainability? LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 23. What lessons have been learnt implementing HCM activities? Please list the lessons learned, including those learnt in:  Working with social franchises, private sector and NGOS/CBOs  Developing SM and SBCC capacity at DHP and in the counties RECOMMENDATIONS Q4: What change do you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? 6 USG PARTNERS ROUNTABLE GROUP DISCUSSION GUIDE APHIAPlus Kamili, APHIAPlus Coast/Nairobi, APHIAPlus Western/Nyanza, APHIAPlus Imarisha, Jhpiego and Management Sciences for Health (MSH) Roles or of USG partners in HCM activity 1. How many UGG service delivery entities are partners in HCM? Please list them.  Are the regional APHIAs represented here branches of PS Kenya or different organizations?  What role do the various USG service delivery partners play in HCM? Discuss the role of each, one by one.  The five APHIAs International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 107  Jhpiego  Management Sciences for Health (MSH)  Others listed Progress made in implementing various components Let us start by looking at the level of achievement on some of the goals the HCM set out to achieve. 2. To what extent has HCM harnessed the private sector, social franchises and opportunities in the community to promote access to high quality, affordable health products and services? 3. To what extent has the project improved the quality, scale and scope of franchise networks? Explain  What progress has been made in providing hotline services to social franchise providers?  What progress has been made to form the National Consortium of Franchising organizations? 4. What progress has HCM made in identifying, building capacity and awarding sub-grants to 3 LNGOs to implement SM and SBCC initiatives?  What was the level of DHP in the process?  To what extent have CBOs been used for SM/SBCC and distribution of health products? 5. What progress has been made in transitioning PSI to a fully Kenyan entity? 6. What progress has been made in creating and developing public-private partnerships to deliver SBCC, SM and social franchises?  What support has eh Project given to the private sector to distribute health products?  What has been the nature and extent of support to the private sector to continue distributing bundled ORS & Zinc into community based distribution channels? 7. What progress has been made in accrediting private providers to provide services to defined standards of care?  What benefits and challenges will come with the accreditation? 8. To what extent has the Project established SM and SBCC QA standards and guidelines? What have been the successes, challenges and lessons learnt? Capacity building International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 108 9. What roles have USG partners played in building capacity for SM and SBCC? Please discuss capacity built activities of each participating USG partner and the 5 APHIA chapters. 10. What capacity has HCM helped develop among USG and other social development partners? Please explain 11. How has HCM contributed to building capacity at DHP?  What progress has the project made in building SM and SBCC technical and leadership capacity of the DHP and its officers to provide leadership and coordinate SM and SBCC strategies and activities at the national and county levels?  To what extent has the Project developed innovative state-of-the-art techniques at DHP and other IPs at the national, county and sub-county levels to motivate healthier behaviors among target populations?  To what extent has the DHP been transformed into a Centre of Excellence (COE) in HCM?  To what extent is DHP providing technical advice to other MOH departments and coordinating their SM and SBCC activities? 12. How has HCM contributed to SM and SBCC capacity building in the various MOH departments, including:  NASCOP  Division of Child and Adolescent Health (DCAH)  Division of Reproductive Health (DRH)  Division of Malaria Control (DOMC)  Orphans and Vulnerable Children (OVC) Secretariat  Directorate of Preventive and Promotive Health Services (DPPHS)  Community Health Services Unit (CHSU)  National AIDS Control Council (NACC)  National Blood Transfusion Service 13. What concrete actions have been taken by HCM and its partners to build SM and SBCC capacity:  In social franchises  In the private sector  Among participating NGOs and CBOs International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 109  And in pharmacies and other outlets 14. What success has been realized in building capacity as the levels above?  What have been the challenges?  What further action is needed to address the challenges and strengthen SM and SBCC capacity? 15. To what extent has HCM strengthened supportive supervision, monitoring and reporting functions? Franchises, the private sector, NGOS/CBOs, pharmacies and other outlets? Please explain  What successes and challenges have been realized in this effort?  What else needs to be done to strengthen the two functions? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? Synchronization at the national, county and sub-county levels 16. To what extent are county and community level SM and SBCC strategies, messages and activities synchronized with national level priorities? 17. What progress has been made in reducing duplication of SBCC and SM communication materials?  What have been the successes and failures? 18. Has the Project formed and operationalized the National and County Health Promotion Advisory Committees to drive the health promotion agenda using bottom-up approaches as a platform for adopting/adapting national HCM materials for local use? Explain 19. What action is needed to ensure that national and county SM and SBCC priorities are synchronized, counties routinely input into national plans and have the capacity to customize and implement national plans in their areas? 20. Did the Project set up a national inventory of SM and SBCC research, materials and tools under the management of MOH? MODEL International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 110 Q2: To what degree is the social marketing and communication model that has been created by HCM sustainable? What has facilitated project achievements and the creation of sustainable models? 21. HCM set out to develop sustainable markets for health products and services, prioritizing equity and impact through effective management of subsidies. To what extend has the project achieved this goal? 22. To what extend has HCM succeeded in developing a sustenance strategy for the products it promotes? (Counselling and testing, condom sales and distribution, FP methods, mosquito nets, ORS, Zinc, etc.) 23. What SM and SBCC models have been developed by HCM?  Please describe the components of the models created. LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? 24. How sustainable are the models created?  What are the challenges and threats to the sustainability of the models?  How can the challenges be addressed to improve sustainability? 25. What have been the bottlenecks, best practices and lessons learnt from implementing the HCM project? Please list the lessons including the lessons learnt in:  Building institutional capacity to deliver sustainable health impact, equity and subsidy through SM and BCC interventions  Building MOH capacity to oversee and monitor SM, SBCC strategies and initiatives at the national and county levels?  Transitioning PSI to a local Kenyan entity  Providing TA to APHIAPlus SDPs and USG partners in SM and SBCC  Providing SM and SBCC support to enable franchises to address barriers to uptake and utilization of health products and services among target populations  Addressing barriers to uptake of health products and services through IPC and mass media communication  Supporting the Peace Corps Malaria Project?  Working to reduce duplication of SM and SBCC materials International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 111  Establishing SM and SBCC and QA standards and guidelines RECOMMENDATIONS Q4: What change so you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? 7 CBO ROUNTABLE GROUP DISCUSSION GUIDE Community-based organizations (CBOs) working in SM and SBCC. APHIAPlus HCM is collaborating with 18 CBOs. Another 30 CBOs collaborate with LNGOs Kinds of CBOs and staff/volunteers involved What CBO do you represent? What is the core work of the CBO? What areas does it cover? (Any kind of work, geographical coverage and target audiences) How did your CBO get recruited into working with HCM?  What contractual arrangement does the CBO have with HCM? 1. How is your CBO structured to carry out functions under the HCM project? Please describe indicating the staff categories that work in the community. 2. What activities does your CBO carry out in the HCM project? Please list the activities together with the staff/volunteer categories that carry them out.  Do the activities include IPC and condom distribution in the community? Please explain. 3. Where do you refer people who need the various health services that you promote?  Who runs the facilities you refer clients to? 4. What success has your CBO achieved doing HCM work? 5. What challenges has the CBO and staff/volunteers encountered doing this work? Capacity assessment 6. What training has your staff/volunteers received to equip them for the work they do for HCM?  Who offered the training? 7. In what areas of work are your staff/volunteers good? 8. In what areas do they have challenges and need help? LINKAGE, supervision and reporting International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 112 9. To whom does your CBO report in regard to HCM work?  What kind of information does the CBO report? 10. What other interactions does the CBO and its staff/volunteers have with the HCM office and officers? 11. What supervision does the CBO receive in regard to the activities it carries out on the HCM?  How has the CBO and staff benefited from the supervision? 12.Capacity building 13. To what extend has HCM been involved in building the capacity of your CBO to do a better job?  What training and exposure has it given your staff/volunteers? Please explain.  What more could HCM do to strengthen the capacity of your organization and your staff/ volunteers Collaboration with other APHIAPlus players 14. Who else is doing APHIAPlus work in your area?  What activities does the individual or organization do for APHIAPlus?  How do you work with this organization/individual? ACHIEVEMENTS Q1: What have the achievements been in terms of strengthening leadership, management and governance? To what degree is this sustainable? To what degree has this increased the quality of health services, products and information? What has been the impact on improved health and equity especially when reaching poor and vulnerable populations? Distribution of condoms, ORS and LLIN 15. In what ways do people get condoms in your area? Please list all the ways in which condoms may be accessed  What successes have been achieved in making condom available to those who need them?  What have been the challenges?  How can the challenged be addressed? 16. Who sells or distributes ORS in your area? Please explain International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 113 Do you know if the ORS is the type that comes with zinc? Please explain How available is the ORS in your area? Is it readily available or does it go out of stock sometimes? 17. How are mosquito nets sold and distributed in your area?  Who distributes the nets and from where?  Who gets the free nets?  Are nets also available for sale? Please explain  What successes and challenges does the net distribution program experience? 18. Who supports the distribution and marketing of condoms, ORS and mosquito nets in your area? Please describe the kind of support given. MODEL Q2: To what degree is the social marketing and communication model that has been created by APHIAPlus HCM sustainable? What has facilitated project achievements and the creation of sustainable models? PSI/K is implementing a SM and SBCC program that seeks to make available quality health products and services through multiple channels, including the community channels your CBO is contributing to. Some of the services are offer free at public facilities, some at subsidized rates and others at commercial prices. Individuals are free to choose kind of services they will use. 19. How much do you know about the whole program? Please list all the aspects of the program that you know. 20. What do you think about these approaches used by HCM as models for delivering quality health products and services according to the ability of the different target audiences to pay?  How effective are the approaches in delivering quality and affordable health products to all categories of people?  How sustainable are the approaches?  What are the challenges and threats to the sustainability of the models?  How can the challenges be addressed to improve sustainability at all levels? LESSONS LEARNED Q3: What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 114 21. What have been the bottlenecks, best practices and lessons learnt from the HCM project approaches? Please list the lessons including the lessons learnt in:  Creating SM and SBCC models that increase access to quality health products through various channels at rates that the various market segments can afford  Motivating people in the community to use health products and services  Promoting and distributing health products such as condoms  Working as a grassroots contributor in a big national program RECOMMENDATIONS Q4: What change so you recommend for social marketing and communication programs in the future to make them more effective in delivering sustainable services that reach targeted populations? B KII INTERVIEW GUIDE Modified to suit during interview ORGANIZATION/DEPARTMENT SPECIFIC INFORMATION 1. What is the mandate of your Department? 2. How has the Department worked with the HCM project?  Please describe the contacts, mechanisms and processes that have facilitated working with the HCM project 3. To what extent has PSI/K supported your department in its work? Please probe for support in:  Management  Communication and social marketing  Capacity building  Commodity forecasting, etc. 4. What additional support would your department like to receive from future SM and SBCC programs? EQUITY 5. To what extent has the HCM program been successful in increasing access to health products and services among the various target groups across the country? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 115 To what extent has the program increased access to health products among the following categories?  Women, especially in the lower socio-economic categories  Young people in and out of school MODEL DEVELOPMENT 6. What models has HCM developed in Kenya? What do you think of the appropriateness and benefits of the models? Please give your assessment of the models, including the following aspects of the model:  Total marketing Approach that brings together the public, private and the NOG sectors to promote health products and services  Promotion of health products and distribution through commercial channels  Social franchising  Decentralization to the counties, sub-counties and the community  Partnering with NGOs and CBOs to promote health behaviors in communities  Behavior change communication in communities Promotion of interpersonal  Building the capacity of government departments and local institutions to lead social marketing and SBCC activities in the country ACHIEVEMENTS 7. What have been the achievements of HCM Program? Please give all the achievements, including achievements in the following aspects:  Developing leadership of SM and SBCC in Kenya’s health sector  Developing the capacity of local institutions to manage SM and SBCC activities, including organizational, financial, M&E and HR functions  Development appropriate policies and governance standards, including appropriate policies, standards and practices 8. What factors have facilitated the program achievement? SUSTAINABILITY 9. To what extent are the achievements and models created by HCM sustainable? How sustainable are:  The models created International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 116  The institutions strengthened to manage SM and SBC activities  Governance structures, policies and standards  Funding the models over time  Possibility of GOK and local institutions taking over the management of the models with little or no external support LESSONS LEARNED 10. What are the key lessons learnt in the establishment of local sustainable models among (private sector partners, local NGOs, and government departments at the national and county levels? What lessons have been learnt in the following areas:  Bringing together the public, private and NGO/CBO sectors to work together in making quality health services, products and information available to Kenyans  Promoting health products and distribution through commercial channels  Social franchising  Decentralizing SM and SBCC activities to the counties, sub-counties and the community  Partnering with NGOs and CBOs to promote health behaviors in communities  Carrying out behavior change communication in communities  Building the capacity of government departments and local institutions to carry out effective SM and SBCC activities in the country  Reaching the various segments of the population with quality health products, services and information, especially (1) the poor, (2) women and (3) young people RECOMMENDATIONS 11. What change would you make to make future SM and SBCC programs more effective in delivering sustainable health services that reach targeted populations? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 117 ANNEX 10: QUESTION GUIDE FOR PROVIDER PERCEPTIONS Consent Name of enumerator Date of interview Start time End time Name of primary respondent Contact of the primary respondent Gender of primary respondent 1. Male 2. Female Position of primary respondent Name of facility Location of facility( county) Location of facility (District) Type of facility 1. Hospital 2. Health Center 3. Dispensary 4. Nursing Home 5. Medical Clinic 6. VCT Center (Stand-alone) Ownership of facility 1. Government of Kenya: Ministry of Health 2. Government of Kenya: Local Authority 3. Faith Based Organization: (specify) 4. Non-Governmental Organization (specify) 5. Private Enterprise (specify) Nature of clients How would you describe the clients that you serve in this facility? probe for gender, age, marital status, socio￾economic status Do you set performance targets for your clinic weekly, monthly or International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 118 annually? Please explain To what extent do you realise your targets? Please explain On average, how many clients do you serve 1. In a day? 2. In a week? 3. In a month? Do you consider this to be a big number? Please explain Management of the facility How is the facility managed? Probe for organizational structure and the departments How is the flow of clients managed in the facility? Probe for the departments that they need to pass through before being served What are the challenges faced in the management of client flow? How can the situation be improved? Services offered by the facility How do you charge for the listed services? Do the clients consider the charges for the services fair? For the services offered free of charge, ask why? Service offered Perceptions of cost by clients Free services( why) Does AphiaPlus HCM contribute in the 1. Yes 2. No International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 119 management of the facility? 3. Don’t know If yes, how is HCM plus involved in the management of the facility? Probe for type of support accorded Any other organization involved in the management of the facility? 1. Yes 2. No 3. Don’t know If yes, list the other organizations and give their roles? Organization Role What would you say are the impacts of the services offered by the facility? In your opinion, what are some of the factors that make clients to seek services from your facility? What are some of the factors that hinder clients from seeking services from this facility? What improvements would you recommend for service provision in this facility? AphiaPlus Support How does AphiaPlus provide support to the facility? Probe for the support accorded to the facility Have there been any expansions in the services provided by the facility over the past 2-3 1. Yes 2. No International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 120 years? If yes, please explain Did Aphia plus have any role in the expansion of these services? 1. Yes 2. No If yes, how? How satisfied are you with the support your facility receives from AphiaPlus? What improvements would you recommend? Community services Does the facility carry out any community activities? 1. Yes 2. No If yes, what activities? Who carries out the services/activities? How are the services carried out? What have been the successes of these services/activities? What have been the challenges in carrying out these services/ challenges? What improvements are needed to strengthen these services/ activities? Commodity management and reporting Do you keep any records for the facility? 1. Yes 2. No Please list all the records the facility keeps and their main purposes Record type Purpose International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 121 Do you do any form of reporting? 1. Yes 2. No Please give the following for the reports that are done in the facility Report type Content Frequency Recipient Model development and Sustainability APHIAplus set out to implement a SM and SBCC programme that can make available quality health products and services through multiple channels, including social franchise clinics such as this one of yours. How successful have the approaches used by APHIAplus been? Probe for the project components are aware of and mention how successful the components are How sustainable are the components? What challenges could threaten the sustainability of the approaches? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 122 How can these challenges be addressed to improve sustainability? Lessons Learnt What lessons have you learnt in providing health services in association with APHIAplus? Please give your answers under the following: Providing services at the clinic Motivating people to come for services Working in the community Receiving, distributing and accounting for commodities received Monitoring and reporting What recommendations can you make that can ensure that future programmes deliver even more effective services that reach all target groups? International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 123 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 124 ANNEX 11: DESIGN MATRIX FOR DATA ANALYSIS Matrix: Data Collection Methods and Corresponding Analytical Approaches Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis Evaluation Question One: To what extent has the activity achieved its mandated outcomes and impact in IRs 1 and 2 along the parameters of health impact, equity and subsidy among the priority poor and vulnerable populations? Health Impacts Comparative Desk Reviews Activity Reports on Performance Data Purposive Comparative analysis on Reported Results versus targets from performance indicators on campaigns aired for promoting behavior change related to targeted program elements Descriptive Desk Reviews Quarterly Reports Purposive Explanations of activity performance, descriptions of campaigns, Analytical Desk Reviews Review of data presentation Special Studies, Evaluations of SM campaigns on behaviors Purposive Content analysis of various methods to compute behavior change by activity Content analysis of evidence presented on linkages between SM campaigns and behaviors Analytical/Trend Data Mining TRaC survey data per health area Purposive Trend analysis of data on behavior and where data are associated with performance indicators, comparison with targets IR 1 – Expected Outcomes Comparative Desk Review Activity reports performance data on sustainability measures for franchised networks Purposive Comparative analysis on reported results vs. targets Descriptive Desk Reviews Quarterly Reports Purposive Explanations of activity performance Analytical/Trend Data Mining TRaC survey data Purposive Trend analysis of data on International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 125 Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis per health area behavior and where data are associated with performance indicators, comparison with targets Analytical Key informant interviews (KII) with individuals related to specific mandated outcomes Raw KII data Purposive Content analysis of raw KII data for opinions, viewpoints, explanations of phenomena related to mandated outcomes to supplement performance and TRaC data, and/or to clarify discussion and increase understanding of GRD data Analytical Group roundtable discussions (GRD) with participants related to specific mandated outcomes Raw GRD data Purposive Content analysis of raw GRD data for opinions, viewpoints, among participants, divergent points of view, etc., explanations among participants, explanations of phenomena related to mandated outcomes to supplement performance and TRaC survey data Analytical/Trend Review of data on volume, prices, sales of SF commodities Activity reports, IP documentation on volume, prices, sales over LOP Purposive Cost recovery analysis of SF commodities Analytical Review of findings on equity based on access to quality health care 2013 MAP report Purposive Review of data analysis and findings on equity based on coverage of and access to SF facilities in franchised networks Comparative Desk review on access to health insurance Activity reports on performance data Purposive Comparison of results vs targets on access to health insurance by targeted poor populations Analytical Review of data on effectiveness of expansion of health insurance and risk pooling PPP documentation on expansion of health insurance and risk pooling Purposive Analysis of data on performance of health insurance and risk pooling to reduce out-of-pocket expenses for payment of health care services for targeted International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 126 Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis KIIs with PPPs involved in the above Raw data from KIIs populations Content analysis of raw KII data to deepen understanding of approach and outcomes Comparative Desk review on individuals reached by telephone hotlines Activity reports on performance data Purposive Comparison of results vs targets on individuals reached by PPPs managing telephone hotlines for SRH counseling IR 2 – Expected Outcomes Comparative Desk review on training provided to MOH officials in SBCC activities Activity reports on performance data Purposive Comparison of results vs targets on number of MOH officials trained Descriptive Desk review Quarterly reports Purposive Review description of methods to strengthen capacity of public sector institutions to implement, promote, monitor and provide oversight of SBCC activities Analytical RGDs with MOH HPO officials at central and county levels and county HPOs KIIs with MOH officials and central and county levels Raw data from RGDs Raw data from KIIs Purposive Content analysis of raw RGD data on sustainability of SBCC models, description and effectiveness of capacity development, implementation issues at county level, etc. Content analysis of raw KII data to increase understanding of results from analysis of RGD data Analytical Desk review Documentation on QA standards, guidelines, national strategy for health promotion Purposive Content analysis of documents Observational Visit to HPU Resource Room MOH central Purposive Observations on readiness of resource room for inventory of International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 127 Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis SBCC materials Analytical KII with HPU officials, MOH central Raw data from KIIs Purposive Content analysis on explanation of progress in inventory and maintenance of SBCC materials Analytical KII with PKS staff on synchronization of SBCC approaches and reduction of duplication of SBCC materials KII with staff from other ALPHIA plus activities Raw data from KIIs Purposive Content analysis of raw data from KIIs with PSK staff and with staff from other ALPHIAplus activities to understand what has been done to date to synchronize and reduce duplication Comparative analysis Desk Review Activity reports on performance data on meetings to implement MOUs on SBCC synchronization Purposive Comparison of results versus targets on number of meetings Analysis KIIs with PKS staff on transition to become an independent entity KIIs with PSI staff on transferring functions to PKS Raw data from KIIs Purposive Content analysis of raw KII data on process of transfer, current activities Content analysis of process of transfer, opinions and evidence of PKS capacity Analysis Desk review PKS data on LNGO/LCBO model Purposive Content analysis of data from PKS records Analysis KIIs with PKS staff on pilot activity to extend messaging to vulnerable population at sub￾Raw data from KIIs Purposive Content analysis of raw KII data for a description of pilot approach, methods, viewpoints on effectiveness of pilot International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 128 Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis county level, selection of LNGOs, content and approach for training, TOT training in SBCC and IPC methods GRDs with LNGO staff on training and on collaboration with LCBOs Content analysis of raw GRD data on effectiveness of training, collaboration experience with LCBOs, viewpoints on effectiveness Comparative analysis Data Mining 2010 GOK Malaria Survey report 2014 data from Malaria TRaC survey Purposive Comparison of data on household use of mosquito nets between 2010 and 2014 Evaluation Question Two: To what extent has the activity succeeded in creating sustainable social marketing and communications models for the health sector? What strategies and approaches facilitated the achievements/creation of such sustainable models if any? Analytical Review of data on findings and conclusions of on SM/SBCC approaches Findings and conclusions from EQ #1, IR 1 Purposive Review of data used to develop findings and conclusions from EQ #1, to identify successful, sustainable SM/SBCC models, and approaches to foster sustainability of models, and linkages with expected outcomes Analytical RGDs with MOH officials at central and county level and CHPOs Raw data from RGDs Purposive Content analysis of raw data from RGDs in response to question on opinions of sustainability of SM/SBCC approach Evaluation Question Three: What are the key lessons learned, especially on the establishment of local sustainable social marketing models among the private sector partners, local NGOs, and government departments– at the national and county levels? Analytical Review of findings and conclusions Findings and conclusions from Purposive Review of data used to develop findings and conclusions EQ #1 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 129 Evaluation Questions Type of Evidence Methods Source Sampling / Selection Data Analysis from EQ #1 and EQ # 2 to identify possible lessons expected outcomes EQ# 1 and EQ 32 and EQ #2 to identify and develop lessons Evaluation Question 4: What are the key recommendations; especially on what changes could be made to future social marketing and communication programs to make them more effective in delivering sustainable services that reach targeted populations? Analytical Review of findings and conclusions from EQs 1, 2, and 3 Findings and conclusions from EQs 1, 2 and 3 Purposive Analysis of findings and conclusions from EQs 1, 2 and 3 to develop recommendations for future directions for new SM/SBCC activities International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 130 ANNEX 12: CONTRACEPTIVE USE BY WOMEN Figure 1: Contraceptive Use By Women 15-24 Years Of Age Who Do Not Want To Become Pregnant, 2012 Source: TRaC Survey 2012 Figure 2: Contraceptive Use by Women 15-24 Years Of Age Who Do Not Want To Become Pregnant, 2014 Source: TRaC Survey 2014 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 131 ANNEX 13: NETS DISTRIBUTION BY SOURCE AND DONOR Source: PSK management information system sales data on nets sales and distribution, KII with PSK staff Year/ source DFID PMI GF Total free nets SM DFID Overall totals FY 2011 1,218,262 55,882 608,370 1,882,514 655,386 2,537,900 FY 2012 1,266,089 1,161,894 - 2,427,983 801,994 3,229,977 FY 2013 1,390,988 136,365 1,527,353 718,543 2,245,896 FY 2014 (Jan￾Oct) 893,418 1,500,126 2,393,544 673,523 3,067,067 Free nets mass distribution Year/ Donor PMI GF World Vision World Bank Total FY 2012 2,667,560 5,200,000 330,000 2,300,000 10,467,560 International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 132 ANNEX 14: MAP OF SOCIAL FRANCHISE FACILITIES Source: PSK International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 133 ANNEX 15: QUANITITIES OF SOCIALLY MARKETED CONDOMS SOLD Source: PSK management information system sales data on nets sales and distribution, KII with PSK staff International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 134 ANNEX 16: TOTAL MARKET VALUE OF SOCIALLY MARKETED CONDOMS Source: PSK management information system sales data on nets sales and distribution International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 135 ANNEX 17: QUANTITIES OF SOCIALLY MARKETED CONDOMS SOLD Source: PSI Sales data International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 136 ANNEX 18: MARKET SHARE OF SOCIALLY MARKETED CONDOMS Source: PSI Sales data International Business & Technical Consultants Inc. APHIAPlus HCM, End-of-project Evaluation 137 ANNEX 19: COST RECOVERY AND LEVEL OF SUBSIDY OF SOCIALLY MARKETED HCM PRODUCTS Source: PSK management information system cost recovery data