1 Project Name and Number AMKENI CA # 623-A-00-00019-00 REPORT FOR USAID/KENYA FINAL EVALUATION OF THE AMKENI PROJECT Pinar Senlet Nimrod Bwibo Joseph Karanja Joyce Olenja Alice Mwangangi Fredrick Ombwori April 15, 2005 2 ACKNOWLEDGEMENTS The consultants would like to acknowledge the support they received from the staff of USAID/Kenya, the AMKENI Project, and the Ministry of Health in conducting this evaluation. The team expresses special appreciation to Sheila Macharia, USAID Reproductive Health Specialist, and Job Obwaka, AMKENI Project Director, who provided useful information and insights as the team progressed from gathering information to drawing conclusions and making recommendations. The time and assistance provided by staff of AMKENI’s international and local partners are greatly appreciated, including EngenderHealth, Family Health International, IntraHealth, Program for Appropriate Technologies, Aga Khan Health Services, Cooperative League of the USA, Family Planning Association of Kenya, and the UZIMA Foundation Finally, our special thanks go to those managers, service providers and community members in the Coast and Western Provinces, who graciously allowed us time to visit their facilities and conduct focus group discussions. 3 ACRONYMS AIDS Acquired Immune Deficiency Syndrome AKHS Aga Khan Health Services ANC Antenatal Clinic ARV Anti Retral Virus BCC Behavior Change and Communication BTL Bilateral Tubal Ligation CAI Community AID International CBD Community Based Distribution CLUSA Cooperative League of the USA COPHIA Community Based HIV/AIDS Prevention, Care and Support CPC Carolina Population Center DHC Dispensary Health Committees DHMT District Health Management Team DFPID Department for International Development DHR Division of Reproductive Health EMR Extended Management Review EOC Emergency Obstetric Care FHI Family Health International FPAK Family Planning Association of Kenya GOK Government of Kenya GTZ German Development Cooperation HCDC Health Center Development Committee HIV Human Immunodeficiency Virus IMCI Integrated Management of Childhood Illnesses IMPACT Implementing AIDS prevention and Care Project IP Infection Prevention KEPI Kenya Expanded Program on Immunization KDHS Kenya Demographic and Health Survey KMFF Kenya Music Festival Foundation M&E Monitoring and Evaluation KMTC Kenya Medical Training College MOH Ministry of Health MVA Manual Vacuum Aspiration NASCOP National AIDS Control Program NGO Non-Governmental Organization NSV No-scalpel Vasectomy OJT On-the-Job Training OVC Orphans and Vulnerable Children PAC Post-Abortion Care PATH Program for Appropriate Technologies in Health PEPFAR President’s Emergency Plan for AIDS Relief PHO Public Health Officer PHT Public Health Technician PMTCT Prevention of Mother to Child Transmission PIA Performance Improvement Approach 4 RFA Request for Application RH Reproductive Health RH-ICC RH Inter-agency Coordinating Committee SP Sulphamethoxazole-pyrimithamine STI Sexually Transmitted Infections T&S Training and Supervision UNFPA United Nations Population Fund USAID United Agency for International Development VCT Voluntary Counseling and Testing 5 TABLE OF CONTENTS Executive Summary 6 I. Introduction Purpose 14 Methodology 15 II. Background Trends in Reproductive Health, HIV/AIDS and Child Health in Kenya 16 USAID/Kenya’s RH/FP/CS/HIV/AIDS Portfolio 18 III. The AMKENI Project: Findings, Conclusions and Lessons Learned AMKENI’s Goals, Strategies and Major Interventions 21 Coverage of Geographical Areas and Pace of Project Implementation 26 Training and Supervision 29 Service Delivery 32 Behavior Change and Communication 41 Capacity Building and Sustainability 50 Project Management 52 Monitoring and Evaluation Plan 54 Funding Sources and Allocation of Funds 59 Relations and Coordination with USAID, MOH and Other Stakeholders 61 IV. Short-term Recommendations for AMKENI 63 V. Long-term Recommendations for USAID for Future Programming 66 ANNEXES A. Scope of Work B. Documents Reviewed C. List of Contacts D. List of Clinical Sites Visited E. List of Community Focus Group Discussions F. Data Collection Tools G. AMKENI Organogram 6 EXECUTIVE SUMMARY The purpose of this evaluation is to examine the activities of USAID/Kenya’s increased use of family planning, reproductive health, child survival, and HIV/AIDS services portfolio, implemented through a Cooperative Agreement. The project, AMKENI is being implemented by a cooperative partnership of four development agencies led by EngenderHealth. The evaluation aims to analyze: ß The progress of AMKENI towards the achievement of specific project objectives; ß The achievements in quality, demand for, and utilization of FP/RH/CH/HIV/AIDS services; ß The achievements in national training and supervision systems; ß The communities’ perspectives of the project activities; ß The appropriateness of the strategies towards achievement of the project activities; ß The pace of project implementation; ß The impact of shifting funding mechanisms on project implementation; ß The degree to which systems and processes are established that lead to sustained behavior change and increased demand for services; ß Overall challenges and lessons learned; and, ß Key areas of focus for future programming. The evaluation took place in early 2005 and was conducted by a core team of four independent consultants. The evaluation methodology included review of relevant documentation, structured interviews with key informants and stakeholders, field visits to Project provinces, visits to clinical facilities and focus group discussions with community members. Summary of Key Findings, Conclusions and Lessons Learned ß The strategic directions of the AMKENI have evolved significantly over the life of the project due to a number of external causes, including policy changes and availability of additional funding sources. A mid-term evaluation conducted in 2003 led the Project revise and consolidate its strategic focus. ß Currently AMKENI has two overarching objectives; 1) Increasing access to and quality of RH/FP/CS services including HIV/AIDS prevention services, and, 2) Promoting healthier behaviors among the population and increasing demand for services. These two objectives form the backbone of the project, aiming to link communities and service facilities. ß AMKENI implements a three-pronged strategy: 1) Improving the capacity of health facilities to provide FP/RH/CS services, including HIV/AIDS, 2) Working with communities to promote healthier behaviors and demand for services, and 3) Strengthening the MOH’s decentralized systems for training and supervising RH service providers. AMKENI aims to improve service delivery, training, community involvement and behavior change in several technical areas, implementing multiple approaches. 7 ß Linking the facilities and the communities is a powerful approach. Likewise, the three-pronged strategy is appropriate for AMKENI to achieve its objectives and goals. ß As a result of numerous interventions, Project implementation has been very comprehensive and complex. While the evaluation team understands the importance of each technical area and related interventions, the technical focus of the project has been extensive and lacks prioritization. ß Given the complexity of the project design, AMKENI has done remarkably well. There have been substantial improvements in all project elements. Nevertheless, the team concludes that if the project had more focus on family planning and selected reproductive health interventions, implementation would have been smoother, faster and the achievements would have been greater. ß AMKENI works in the Coast and Western Provinces, targeting 10 districts. The total population of the AMKENI-assisted districts encompasses majority of the province populations in both provinces. It also appears that on average, AMKENI was able to reach a critical mass of the populations living in project districts. ß Majority of the AMKENI-assisted facilities are public facilities, as opposed to privately owned or Mission facilities. The mix of public/private facilities is appropriate for achieving the goals of the project. Similarly, AMKENI’s focus has been on lower-level facilities such as maternity homes, dispensaries and health centers as opposed to hospitals. This approach is also valid and appropriate. ß Project components have progressed at different pace. In general, training and service delivery components have progressed at a higher pace compared with BCC. ß AMKENI formulated training and supervision (T&S) interventions with the purpose of improving the quality of services through a combination of direct training in specific skills and development of sustainable training and supervision systems at the national level. ß AMKENI completed establishment of decentralized T&S teams in all eight provinces, 10 target districts in Coast and Western Provinces, and 16 facilities. In addition, the project assisted in development of national RH T&S curricula and manuals. ß The Project promoted implementation of earlier policy reforms allowing nurses and clinical officers to provide implants and PAC services. ß AMKENI worked with the provincial and district health management teams to strengthen their service quality through the introduction of the Performance Improvement Approach (PIA). ß Overall, AMKENI has trained over 3,000 trainers and service provides at all levels, in various courses in accordance with service delivery needs. 8 ß AMKENI’s training and supervision interventions have been highly successful and have been valued by trainees and stakeholders. However, the team is concerned that the National PMTCT and VCT Training manuals developed by the MOH have not integrated family planning counseling. ß The team believes that AMKENI’s mandate to undertake direct training activities both at the national level and at the provincial and district levels was not justified and somehow led the Project to stretch itself too thin. ß AMKENI has not put in place a systematic follow-up procedure to support and evaluate the outcomes of its training efforts. ß Expanding existing services and establishment of new services was one of the key elements of the service delivery component of the Project. AMKENI enhanced the capacity of its supported facilities by training staff in specific skills and by providing essential equipment and supplies. ß In 2003, AMKENI expanded its HIV/AIDS interventions when funds from the PEPFAR track became available. The Project assisted several facilities to establish VCT and PMTCT services. The team has observed that in some PMTCT and VCT centers, basic family planning counseling is available. However there was no evidence of full integration of FP with VCT and PMTCT in the facilities visited. ß One of the innovative approaches of AMKENI has been to facilitate outreach activities for long-term and permanent methods, with the intention of organizing frequent outreach activities to health centers located at a distance from district hospitals. ß AMKENI interventions have contributed to increased number of family planning acceptors. Marked improvement is especially noted for long term and permanent methods. Increases in use of long-term and permanent methods indicate improvements in contraceptive method mix. Total number of deliveries from all AMKENI supported facilities have also raised significantly between 2001 and 2004. ß The team concludes that high input in training and supervision, equipment and renovations, and community mobilization has lead to high quality service and to increased utilization. There were however challenges that adversely affected family planning utilization. In some facilities visited family planning services were not fully integrated with other maternal and child health and HIV/AIDS services. ß The evaluation team has observed three system-wide issues that have been negatively affecting service availability, quality and utilization during field trips. These challenges are not necessarily within the mandate the AMKENI Project. However, the team believes that they should be mentioned since they are widespread and crosscutting challenges that will continue to have a negative impact on services in the future. These are contraceptive shortages, abolishment of MOH’s cost-sharing policy and staff shortages. 9 ß In the area of BCC interventions, AMKENI implemented four approaches; 1) Family￾to-family communication: 2) Direct outreach to men, women and youth: 3) Women’s agency, and 4) Magnification of behavior change. These interventions were implemented through partnerships with five local implementers. ß The BCC component has had a positive influence on the knowledge, attitude and behavior of individuals in project sites. This is further corroborated by the preliminary findings of the Household Survey and by the focus group discussions conducted by the evaluation team. ß The community response to the Project has been positive and this is evident in the level of engagement. In discussion groups, participants noted that they had benefited from the Project through education in FP/RH/CS and particularly on VCT and PMTCT, contributing to behavior change in the community. ß The Project’s uniqueness lies in its approach of improving services at the facility level and mobilizing communities to demand for quality services. The challenge then becomes maintaining the balance between the two components; so that once communities are fully mobilized and come for services, these would be in place. ß Multiple BCC approaches have been implemented in different sites and at varied levels, thus it is not possible to delineate the contribution of each of the approaches singly. The main models are Community/Women’s Agency Model (CLUSA) and Community Participatory Model (AKHS). ß Overall, BCC activities have contributed to increased demand and utilization of services. However, the implementation of several BCC strategies is labor intensive and costly. ß BCC activities/messages are increasingly inclined towards HIV/AIDS and VCT/ PMTCT and it seemed much less on FP/RH the core business of the Project. BCC activities have gone beyond this to include other facets of health, leaning much more towards general public health issues. ß The Project has been successful in capacity building both in personnel and physical inputs. Training and skills development are sustainable and replicable. There are certain elements, especially in BCC that are not sustainable beyond the project particularly where there are parallel systems of personnel. ß AMKENI partnership formed among four international agencies works well. The partners have mutually respectful relationship, developed around their respective technical expertise and their strengths. ß AMKENI has a highly qualified, committed and hard-working staff at all levels. The reporting and financial systems put in place are sophisticated, and satisfies USAID’s reporting needs. 10 ß The management style of the project is rather top-down and centralized. The Area Managers only have administrative oversight for the staff in the area offices. The technical staff reports directly to the respective technical advisors in Nairobi. ß There is some duplication in staffing. HIV/AIDS Advisor and the two HIV/AIDS Coordinators are separated from service delivery staff. This does not help better integration of HIV/AIDS with the other elements of the Project. ß AMKENI has spent substantial efforts and resources to develop its monitoring and evaluation plan. However, the plan is too comprehensive and complicated, thus it does not enable the Project to monitor progress on a daily and continuous basis. There is little evidence that the M&E is used as a management and performance improvement tool at the field level. ß To date, USAID has obligated AMKENI a total of $18,913,090. The bulk of funding is obligated from population funds, comprising 72% of the total budget, followed by AIDS funding at 12%. Largest line item in utilization of funds is the management category, which used 37% of the total budget, followed by BCC activities at 19%. Summary of Key Recommendations The recommendations are presented in two categories. First, there are short-term recommendations for AMKENI to consider over the remaining life of the Project. Secondly, the team provides long-term recommendations for USAID/Kenya for future programming purposes, specifically developed for the new five-year follow-on project. Since USAID/Kenya is planning for a new competitive procurement within a year, these long-term recommendations are procurement sensitive and are omitted in the report presented to AMKENI and its partners. . Short-term recommendations for AMKENI ß The basic thrust of the AMKENI, working with both the facilities and the communities to create linkages and to improve service quality while increasing demand for services is a powerful approach and should be maintained. ß The current three-pronged strategy implemented by AMKENI is appropriate and should be continued with some modifications outlined below. ß The Project should speed up the implementation of the decentralized T&S system in the remaining 81 facilities. ß AMKENI should undertake a rapid assessment and document the outcomes of its training activities by the end of the Project. ß By the end of the Project, AMKENI should ensure that all PMTCT and VCT providers in Project facilities have been trained in basic family planning counseling. 11 ß AMKENI should work with the newly awarded CAPACITY Project to coordinate pre-service training activities and should transfer technical assistance in pre-service training to the CAPACITY Project by the time of closure. ß USAID and AMKENI should consider assisting MOH with vehicles to transport supervisors by providing a vehicle to each of the AMKENI current provinces ß AMKENI should ensure that family planning services are fully integrated with other maternal and child health care services, PMTCT, and VCT in all target facilities by the end of the Project. ß Although there is evidence that family planning method mix has improved in project areas, additional efforts are needed to further improve method mix and to ensure continuity of family planning use and compliance. ß AMKENI should work closely with the MOH and the DELIVER Project to ensure that contraceptives are available at the facility level. ß AMKENI’s BCC component should place higher emphasis on FP/RH BCC messages and pay special attention on men and male youth as the locus of power. ß Given the role played by the PHOs PHTs, it is prudent that they are facilitated more by the Project in terms of transport. The team recommends that AMKENI provide bicycles for the PHOs and PHTs. ß AMKENI should analyze the Household Survey data and disaggregate it by province to facilitate comparison with the end of project targets. The analysis should also determine the most successful and cost effective BCC approach. ß The team recommends AMKENI to consolidate project interventions in current 10 districts within the Coast and Western Provinces over the next year. ß AMKENI should undertake an assessment within the current districts to analyze its exact coverage of the populations. This analysis can be done by mapping out of the catchment areas of the current AMKENI-assisted facilities. Based on this analysis, AMKENI should decide whether it is feasible to assist additional facilities within the current districts. ß If AMKENI decides to assist additional facilities, it should develop criteria for selecting those sites. The focus of the expansion should continue to be on lower level facilities, particularly the most needy dispensaries. ß AMKENI should delegate both administrative and technical oversight responsibilities to the Area Managers for field implementation. ß AMKENI should review its M&E Plan and retain those indicators only essential for tracking performance and progress of the project. 12 Long-term recommendations for USAID/Kenya ß The evaluation team recommends that the follow-on project should focus on FP, PAC, AC, IP, PMTCT and VCT. Maternal health activities should be minimized, with selected and targeted interventions. Child survival interventions should not be part of the follow-on project. ß USAID/Kenya should coordinate closely with the new DFID-funded Safe Motherhood Project while developing the follow-on procurement instrument. ß USAID/Kenya should continue collaborating with the donor community to ensure contraceptive security. ß USAID/Kenya may consider initiating policy level talks with the MOH to reverse the cost-sharing policy and/or explore alternative means of financing health care at lower level facilities ß USAID/Kenya may consider fostering synergies between the follow-on Project and the CAPACITY Project by directing emergency staffing assistance foreseen under the CAPACITY Project to understaffed dispensaries within the new provinces. ß The follow-on Project should maintain AMKENI’s approach to linking facilities and communities. The follow-on Project should also maintain AMKENI’s three-pronged strategy incorporating T&S, service delivery and BCC components. ß The follow-on Project should not be involved in pre-service training. Pre-service training interventions currently implemented by AMKENI should be transferred to the CAPACITY Project. ß All components of the follow-on Project should focus on integrating FP/RH and HIV/AIDS interventions at the field level. ß The follow-on Project should use lessons learned from AMKENI to simplify and streamline BCC activities by implementing the most effective strategy that reflects a hybrid of the most successful elements of the piloted BCC models. ß The team recommends that the BCC interventions of the follow-on Project should focus on FP/RH themes and on men and youth. · AMKENI should complete and phase-out its activities in Coast and Western Provinces by the end of the project in March 2006. The follow-on Project should select and move on to working in two new provinces. · The follow-on Project should focus on provincial, district and facility level activities and should not include national level interventions. Similarly, the next phase should focus on the lower level facilities, where high-level demand is obvious. 13 ß The follow-on Project should have a decentralized management structure and lean staffing. The Project should be required to consolidate technical staffing under teams in line with the project’s main technical components; training, service delivery and BCC. ß The follow-on Project should continue AMKENI’s effective coordination with the IMPACT Project and the current division of labor. However, the team does not recommend sharing of staff between the two projects. 14 I. INTRODUCTION PURPOSE The purpose of this evaluation is to examine the activities of USAID/Kenya’s increased use of family planning, reproductive health, child survival, and HIV/AIDS (FP/RH/CS/HIV/AIDS) services portfolio, implemented through a Cooperative Agreement with EngenderHealth. The cooperative agreement is a five-year initiative to improve FP/RH/CS/HIV/AIDS situation in Kenya with an emphasis on increasing utilization of facility based integrated services. The project, AMKENI is implemented by a partnership agreement among four development agencies led by EngenderHealth. AMKENI started in December 2000 and will run through September 2005, with a possibility of extending through March 2006. The evaluation aims to analyze the following: ß The progress of AMKENI towards the achievement of specific project objectives in line with USAID/Kenya’s Strategic Plan; ß The achievements in quality, demand for, and utilization of FP/RH/CS/HIV/AIDS services; ß The achievements in national training and supervision systems; ß The communities’ perspectives of the project activities; ß The appropriateness of the strategies towards achievement of the project activities; ß The pace of project implementation given the geographical focus and expansion of scope; ß The impact of shifting funding mechanisms on project implementation; ß The degree to which systems and processes are established that lead to sustained behavior change and increased demand for services; ß Overall challenges and lessons learned; and, ß Key areas of focus for future programming. In general, the assessment analyses the impact of the following topical issues on quality, utilization and demand for FP/RH/CS/HIV/AIDS services in project areas: ß Service delivery: The achievements in quality, access, utilization and integration of services. ß Training and supervision: The development of national policies and systems for training and supervision. ß Behavior change and communication (BCC): The impact on demand and utilization of services. ß Community ownership and capacity building: Change in knowledge and attitudes, establishment of systems and processes. ß Monitoring and evaluation: The efficiency of the monitoring and evaluation systems. ß Project management: The efficiency of administrative, programmatic and financial management. ß Relationships with the Ministry of Health (MOH) and other key partners. 15 The full Scope of Work for the evaluation is found in Annex A. METHODOLOGY The evaluation took place in February-March 2005 and was conducted by a core team of four independent consultants: Pinar Senlet, an international public health consultant; Prof. Nimrod Bwibo, a pediatrician and child health specialist; Prof. Joseph Karanja, a consultant obstetrician and gynecologist and senior trainer in maternal health; and; Prof Joyce Olenja, a medical anthropologist. The core team members had complementary skills and expertise, including experience in management, monitoring and evaluation, service delivery, training, and BCC with regard to FP/RH/CS/HIV/AIDS. The team was accompanied and supported by two additional members from the MOH throughout the evaluation process: Alice Mwangangi, a staff of Division of Reproductive Health; and Fredrick Ombwori, an economist from the MOH. The team closely collaborated with USAID/ Kenya, AMKENI and the MOH while conducting the evaluation. The evaluation methodology included the following: ß Review of relevant documentation, including project studies, program reports, USAID strategy documents, and other reports related to FP/RH/CS/HIV/AIDS in Kenya. The list of documents reviewed is attached in Annex B. ß Structured interviews with key informants and stakeholders, including USAID/Kenya staff, AMKENI partners, MOH representatives and other local implementing partners. The list of persons contacted is found in Annex C. ß Field visits to Project Provinces (Coast and Western Provinces) to interview field stakeholders including MOH officials, implementing partners, service providers, and field agents. ß Visits to clinical facilities to assess service quality, access and utilization. The team developed and used a simple facility checklist to collect data on access and quality. ß Structured focus group discussions with community members to assess the extent to which the project has achieved its goals in behavior change and communication. The list of sites visited, list of focus groups conducted and data collection tools are found in Annexes D, E and F. 16 II. BACKGROUND TRENDS IN REPRODUCTIVE HEALTH, HIV/AIDS AND CHILD HEALTH IN KENYA Following several decades of many achievements and improvements in health, Kenya faced many challenges in reproductive health, child health and HIV/AIDS over the 1990s. AIDS epidemic became a tragedy of devastating proportion during the last decade. According to estimates derived from the Sentinel Surveillance System, HIV/AIDS prevalence had peaked at around 10% for adults in the late 1990s. Since then, prevalence has been declining gradually. The prevalence is estimated to have declined to seven percent according to Kenya Demographic and Health Survey 2003 (KDHS 2003), meaning that there are currently 1.1 million adults infected with HIV. Nearly two thirds of those infected are women and HIV infection among adults in urban areas is almost twice as high in rural areas. Figure 1 HIV Prevalence among Adults 0 2 4 6 8 10 12 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Percent Source: Adjusted from the Sentinel Surveillance System. Since 1984, contraceptive use increased steadily in Kenya, as shown in Figure 2. The rate of increase slowed down in early 1990s and even more during the latter portion of the decade. The slowing down of the increase in contraceptive use, as documented by KDHS 2003 is in sharp contrast with previous trends: modern contraceptive use has increased from 32 percent to only 33 percent between 1998 and 2003. Since 1993, the method mix in Kenya has been shifting towards increased use of the injectables, while other methods use have been declining. The use of injectables has increased dramatically from 7 percent in 1993 to 12 percent in 1998 and 15 percent in 2003, making it the predominant method. Over the same timeframe use of pills has declined from 17 10 to 8 percent, IUDs from 4 to 3 percent, and female sterilization from 6 to 5 percent. This is a troublesome finding, since it may indicate problems such as inadequate access and availability of methods other than the injectables, and perhaps increasing biases against these methods. (See Figure 3) Figure 2 Trends in Modern Contraceptive Use among Married Women 0 5 10 15 20 25 30 35 1984 1989 1993 1998 2003 Percent Source: KDHS 2003 Figure 3 Trends in Contraceptive Method Mix 0 2 4 6 8 10 12 14 16 Pill IUD Injectables Condoms BTL Percentage 1993 1998 2003 Source: KDHS 2003 Since 1980s, the number of women of reproductive age has more than doubled, and the size of the reproductive age population will continue to grow markedly in the years ahead, due to population momentum. Despite the remarkable growth in family planning use since the 1980s, there remains a significant unmet need for family planning in Kenya: overall 20 18 percent of births are unwanted, and an additional 25 percent are mistimed, or wanted later. According to KDHS 2003, 25 percent of married women wish to limit or space their next birth, but still are not using contraceptives. In addition, the survey estimates that the number of contraceptive users will need to grow by three percent each year, just to maintain current contraceptive prevalence. Currently, the infant mortality rate is 77 and under-five mortality rate is 115 in Kenya, meaning that one in every nine children dies before his or her fifth birthday. The KDHS 2003 and other surveys indicate a worsening of child health mortality rates during the 1990, reversing decades of steady improvement: The surveys indicate that under five mortality rate has increased by 25 percent. The increase was primarily due to expanding HIV/AIDS epidemic, combined with malaria and other childhood diseases. Other child health indicators have also been worsening recently; for example, vaccination coverage (percent of children age 12-23 months who had received the full regimen of recommended vaccines) has declined significantly between 1998 and 2003, from 65 to 57 percent. The continued growth of the reproductive age population in Kenya, coupled with increasing HIV prevalence and child mortality increased the demand on the health care system in Kenya significantly throughout the 1990s. The additional burden and the negative impact for the health system became a major issue on the public sector health care financing since health care resources from the Government of Kenya (GOK) were not increasing. USAID KENYA’S FP/RH/CS/HIV/AIDS PORTFOLIO In response to the worsening FP/RH/CS/HIV/AIDS situation in Kenya over the last decade, USAID/Kenya developed a new health and population strategy through a process incorporating input from numerous sources in 2000. The Strategic Objective 3 was formulated as part of the USAID/Kenya Integrated Strategic Plan for 2001-2005. The Strategic Objective 3 is defined as “to reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated family planning and health services” The Results Framework for Strategic Objective 3 is shown in Figure 4. The framework illustrates the causal relationships between the high level IRs and the linkages with the lower level results. Key elements of family planning, reproductive health and child survival are integrated with HIV/AIDS activities. USAID/Kenya’s Strategic Plan provides detailed information on definitions of each intermediate result to be achieved and offers illustrative approaches to achieve each result. The plan also includes a comprehensive Performance Monitoring Plan on how the strategic objective and the expected results will be monitored and evaluated. During the Integrated Strategic Plan development process, USAID/Kenya developed a new emphasis, as well as a renewed commitment to family planning within the Strategic Objective 3. The new strategy incorporated numerous directions in the areas of FP/RH/CS/HIV activities: ß A recognition of HIV/AIDS as a multisectoral crisis; 19 ß Better integration of HIV/AIDS services; ß Integrated private sector programs in targeted geographic areas; ß Recognition of the need for behavior change and positive health care seeking behavior; ß Giving young people priority attention; and, ß Enhanced efforts to strengthen the policy environment and manage sector resources. Figure 4 USAID Results Framework for Strategic Objective 3 In accordance with the above strategic directions, USAID/Kenya issued a Request for Application (RFA) in July 2000 to enhance its efforts to improving the health of Kenyans through integrated family planning, reproductive health and child survival services, including HIV/AIDS. The program included two major components: 1) family planning, reproductive health and child survival NGO and private sector service delivery in selected geographic areas (with 75 percent of emphasis), and 2) Improvements in public sector FP and RH training and supervision systems (with 25 percent emphasis). In both components, family planning was described as the predominant focus. However, the RFA required that “HIV/AIDS, other reproductive health and child survival activities be included as feasible and appropriate”. While the RFA focused on FP, it was “designed to receive child survival and HIV/AIDS funding to take advantage of future funding opportunities and potential synergies and linkages.” The project was awarded to EngenderHealth (then AVSC International) and its partners in December 2000. The initial estimated budget for AMKENI was $15,997,129, which increased to $ 18,913,090 through 2004, including President’s Emergency Plan for AIDS Relief (PEPFAR) funds. IR3.1.1 Policies for FP/RH/CS services improved IR3.1.2 Efficiency in the management of health sector resources improved IR3.1.3 Quality of services in health facilities improved Intermediate Result 3.1 Improved enabling environment for the provision of health services IR3.2.1 Reduced key policy and other contextual constraints to preventing and mitigating the impact of HIV/AIDS IR3.2.2 Improved knowledge and practice of prevention behaviors IR3.2.3 Enhanced provision of HIV/AIDS/STI prevention, care, and support services Intermediate Result 3.2 Increased use of proven, effective inter￾ventions to decrease risk of transmission and mitigate the impact of HIV/AIDS IR3.3.1 Integrated FP/RH/CS services expanded IR3.3.2 Improved knowledge of and demand for FP/RH/CS services IR3.3.3 New and improved FP/RH/CS programs and interventions conducted and applied Intermediate Result 3.3 Increased customer use of FP/RH/CS services Strategic Objective 3 Reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated family planning and health services 20 USAID/Kenya is currently in the process of revising its Results Framework and extending it through 2008. While the Results Framework will undergo significant revisions in response to PEPFAR requirements, the IRs related to FP/RH are not expected to alter substantially. USAID/Kenya supports an array of other FP/RH/CS/HIV/AIDS projects under the Strategic Objectives 3. IMPACT, implemented by FHI is USAID’s flagship project for HIV/AIDS prevention and treatment. The POLICY Project, implemented by the Futures Group International works in the policy arena to address policy issues regarding FP/RH and HIV/AIDS. The DELIVER Project, implemented by JSI, is a large contraceptive logistics management project and has a mandate to distribute contraceptive supplies down to the district level. Recently USAID has awarded another large project to improve human resource capacity in health sector, the CAPACITY Project. In addition to USAID-funded projects, other international agencies support several FP/RH projects in Kenya. Among the most active donors are DFID, GTZ and the UNFPA. DFID supports a Safe Motherhood Project in the Western Province, where AMKENI is also working. DFID has issued a new bid with a substantial budget to expand the Safe Motherhood initiative, and the award is imminent. UNFPA supports a comprehensive FP/RH program implemented by the MOH, covering nine districts across the country. Two of these districts, Kwale and Kilifi, overlap with AMKENI targeted areas. GTZ had supported a large contraceptive community based distribution (CBD) project in the past, which has been scaled down considerably. GTZ is now considering increasing its support to CBD programs. 21 III. THE AMKENI PROJECT: FINDINGS, CONCLUSIONS AND THE LESSONS LEARNED The AMKENI project is a five-year (2001-2005) technical assistance effort aimed at improving the FP/RH/CS services, including HIV/AIDS status in Kenya. The project is being implemented by a cooperative partnership of four development agencies led by EngenderHealth. Other international implementing partners are Family Health International (FHI), IntraHealth International Inc., and Program for Appropriate Technology in Health (PATH). A fifth partner, Carolina Population Center (CPC) withdrew from the consortium two years ago. AMKENI aims to bring together the complementary strengths and expertise of these organizations. EngenderHealth has strengths in service delivery systems, practical quality improvement processes and management systems. FHI brings in expertise in integrating HIV/AIDS and STI prevention and integration with other FP/RH/CS services. IntraHealth has its strengths in developing performance improvement training and supervision systems in public and private sectors. IntraHealth also contributes to the partnership in monitoring and evaluation planning, a role it has taken on from CPC. PATH is the lead agency in developing behavior change communication and community mobilization interventions. AMKENI works with a variety of local implementing agencies. The Project’s principal partner is the MOH and it is being implemented with the MOH at all levels. Other implementing partners are Aga Khan Heath Services (AKHS), Cooperative League of the USA (CLUSA) Family Planning Association of Kenya (FPAK), Community AID International (CAI), and UZIMA Foundation (UZIMA). AMKENI’S GOALS, STRATEGIES AND MAJOR INTERVENTIONS The overall goal and strategic directions of the AMKENI have evolved significantly over the life of the project. The initial goal of the project was to increase use of integrated FP/RH services with limited focus on CS and HIV/AIDS activities, while placing an emphasis on the development of private and NGO sector services. The project conducted an assessment of the potential service delivery sites as an initial step. The findings of the assessment indicated a major change of focus in terms of public vs. private sector involvement in project implementation. The structures of the private sector facilities which were to play a major role in increasing access to and quality of FP/RH/CS/HIV/AIDS in the selected provinces were found to be weak. Well-established private hospitals were too expensive to increase access for the general population. Smaller private health care structures were scattered and lacked the required minimum quality of care standards to be involved in the activities of the Project. A significant policy change, which took place in the early days of AMKENI, further limited the Project’s ability to work with the NGO sector: The reinstatement of the so-called “Gag Rule” in 2000 restricted the project to work with the predominant NGOs in Kenya; Marie Stopes and FPAK. As a result, AMKENI had to revise its strategic focus to shift towards working mostly with the public sector. 22 Another development, which led AMKENI to revise its strategic focus, was the availability of PEPFAR funding. In 2003 USAID/Kenya allocated PEPFAR funding which enabled the project greater involvement and focus in HIV/AIDS activities. AMKENI emphasized VCT and PMTCT interventions, as they were mandated under PEPFAR. Finally, a mid-term evaluation, which is referred as an Extended Management Review (EMR), was conducted in April-May 2003. The EMR was a turning point for AMKENI to revise and consolidate its strategic focus. The EMR called for major changes in project design as well as significant shifts in focus areas. In summary, the EMR recommended the project to: ß Review the training program by determining training needs, the services/skills/sites that require training and the optimal training format; ß Strengthen collaboration with the MOH; ß Place greater focus on district priorities and needs; ß Concentrate on aspects of HIV/AIDS that are most closely related to RH, such as PMTCT and VCT; ß Focus and consolidate BCC activities; and, ß Improve the monitoring and evaluation system. The evaluation team highly appreciates AMKENI’s positive response to the recommendations of the EMR. The Project took immediate action on most of the recommendations. As will be discussed in other sections of this report, the Project was able to achieve greater impact over the last two years. A major change made in the project design was to form stronger relationships with the MOH at all levels (e.g., national, provincial, district and facility levels). The project also streamlined its operations by focusing on district level and graduating national level training interventions, as well as improving its monitoring and evaluation plan. The EMR led to the formulation and implementation of the current strategic directions and key interventions. Currently AMKENI has two overarching objectives: ß Increasing access to and quality of FP/RH/CS services including HIV/AIDS prevention services, and, ß Promoting healthier behaviors among the population and increasing demand for services. These two objectives form the backbone of the project, aiming to link communities and service facilities. The evaluation team believes that it is a very powerful approach and applauds the project’s efforts for working with both the facilities and the communities in order to improve services and increasing demand for services. In order to achieve the above objectives, AMKENI uses the following three strategies and related interventions: 1. Improving the capacity of health facilities to provide FP/RH/CS services, including HIV/AIDS related services through: 23 ß Training; ß Expanding range of services (e.g., upgrading facilities, provision of essential equipment and supplies); ß Strengthening supervision; and, ß Service outreach. 2. Working with communities to promote healthier FP/RH/CS/HIV/AIDS behaviors and demand for services by: ß Promoting preventive and health-seeking behavior; ß Fostering community agency; and, ß Creating a supportive environment for individual and community change. 3. Strengthening the MOH’s decentralized systems for training and supervising RH service providers through ß Facilitating the establishment of Decentralized RH training and Supervision System; ß Updating and strengthening the teaching skills and materials for public sector pre￾service and in-service trainers and supervisors; and, ß Facilitating the application of the Performance Improvement Approach (PIA). Within its overall strategic approach, AMKENI aims to improve service delivery and training in several interrelated technical areas. These are summarized in Table 1 Table 1 Interventions by Technical Elements of Service Delivery and Training Family planning and reproductive health o Short, long term and permanent contraception o Post abortion care (PAC) o Prevention of sexually transmitted infections (STI) o Infection prevention (IP) Maternal health o Antenatal care (ANC) o Safe delivery o Emergency obstetric care (EOC) o Newborn care o Prevention and treatment of malaria in pregnancy HIV/AIDS o Voluntary counseling and testing (VCT) o Prevention of mother to child transmission (PMTCT) Child health o Immunization o Growth monitoring o Integrated management of childhood illnesses (IMCI) o Vitamin A supplementation 24 In BCC, AMKENI implements four main approaches, with six local implementing partners. These approaches are: 1. Family-to-family communication; 2. Direct outreach to men, women and youth; 3. Women’s agency; and, 4. Magnification of behavior change. Table 2 summarizes the approaches used in BCC interventions in collaboration with the local implementing partners, in both the Coast and Western Provinces. A full description of these approaches is found under the section on BCC. Table 2 AMKENI BCC Implementing Partners and Approaches Implementing partner Coast Western Approach 1 Approach 2 Approach 3 Approach 4 MOH X X X X X X AKHS X X X X CAI X X X X CLUSA X X X X X FPAK X X X UZIMA X X X Conclusions and Lessons Learned The evaluation team believes that the above three-pronged strategy is appropriate for AMKENI to achieve its objectives and goals. The project was successful in revising and streamlining its strategic directions over the course of implementation and in incorporating lessons learned. The evaluation team, however, has some concerns regarding the broadness of some of the interventions: The two interventions outlined under Strategic Objective 2 “fostering community agency” and “creating a supportive environment for individual and community change” are defined too broadly. While the team recognizes the importance and potential impact of both interventions, these are too far-reaching and ambitious to be implemented under a RH project. Secondly, the team also believes that the inclusion of pre￾service training intervention under the Strategic Objective 3 is over ambitious. Pre-service training in FP/RH/CS/HIV/AIDS requires substantial inputs and will lead the project to spread its resources too thinly. As a result of numerous interventions summarized in Tables 1 and 2, project implementation has been very comprehensive and complex. While the evaluation team understands the importance of each technical area and related interventions, the technical focus of the project looks too extensive and lacks prioritization. The multiplicity of project interventions has stemmed from both design and implementation issues. The RFA issued by USAID was comprehensive and somewhat ambiguous; the RFA required focus on FP/RH but at the same time it allowed flexibility for implementation of 25 HIV/AIDS, child health, maternal health and malaria-related activities. Over the course of the project, USAID was able to allocate significant additional funding for HIV/AIDS, but CS funding remained very limited, about 2% of the overall project funding. Thus the project was able to undertake some child survival interventions, all were successful, but they were few and scattered. The team acknowledges that the term reproductive health encompasses a broad array of elements, which are somehow differently defined by individual organizations. RH needs in each country may also greatly vary. In order to avoid confusion, throughout this report, the term FP/RH is used in line with the way it is defined in the current RFA issued by USAID/Kenya in this report. While describing the FP/RH focus of the Project, RFA required an overall emphasis on family planning, and other RH priority areas were defined as PAC, STI prevention and treatment, HIV/AIDS prevention, and antenatal care. The team has found that although this focus is still a strong part of the training interventions; the overall emphasis of the project has shifted towards maternity services throughout the project. Improving maternity services have been absorbing a significant portion of the resources, especially in terms of renovations and equipment donations. Implementation of HIV/AIDS prevention activities has been the recent focus within the project, due to availability of substantial PEPFAR funding. With that said, the team does not necessarily disapprove expanding the concept of FP/RH. In particular, the team would like to note that the expansion of AMKENI’s mandate has taken place with the approval of USAID/Kenya. The team points out to the fact that the way the Project is currently being implemented is significantly different from the original design. Another underlying cause, which led AMKENI to diversify interventions and stretch itself far, is the project’s responsiveness to the needs of the local partners, especially the MOH, and the communities. The team interviewed numerous individuals throughout this evaluation, including MOH officers at all levels and community leaders and volunteers. FP/RH may not be perceived as a high priority need for stakeholders, especially the communities. Thus, AMKENI’s dilemma has been responding to and meeting the needs of the stakeholders and beneficiaries while keeping a focus on its mandate and core business. The evaluation team appreciates AMKENI’s high level of responsiveness to community needs and sees this as strength. At the same time, however, this has become a challenge for the Project by diverting the Project resources and efforts away from its core business. Given the complexity of the project design, AMKENI has done remarkably well in all of the above areas. There have been substantial advancements and improvements in all project elements and this report provides ample evidence regarding these findings. Nevertheless, it is the conclusion of the team that if the project had more focus on family planning and selected reproductive health interventions, implementation would have been smoother, faster and the achievements would have been greater in this area. As will be discussed in detail in further sections of this report, the team has found evidence of significant improvements in quality and utilization of FP/RH services. However, the team has also found ample evidence that there is a much greater unmet need in provision of FP/RH services and information, which the Project was not able to adequately address. 26 COVERAGE OF GEOGRAPHICAL AREAS AND PACE OF IMPLEMENTATION Geographic and population coverage AMKENI works in the Coast and Western Provinces targeting 10 districts. In Coast, the project is working in four of the total seven districts, and in Western AMKENI is active in six of the total eight districts. In these districts the Project targets 97 health facilities. Once the project districts were identified, the project undertook a facility assessment in each district to determine facility selection, in collaboration with the MOH. The facility assessments examined the availability of minimum quality standards at each site in order to be included in project activities. A short list of 30 to 40 facilities in each district was presented to the MOH and MOH identified 12 target facilities in each district. MOH’s selection criterion was to include those sites with low service utilization rates. Table 3 provides a breakdown of AMKENI supported facilities by province, district and facility type. Total populations of the provinces and districts are provided to understand population coverage of the project. Table 3 Breakdown of Facilities by Province, District and Facility Type Province/ Population* District/ Population* Facility type Total Public Private Mission Coast 2,927,273 Kilifi 640,593 9 2 1 12 Kwale 566,886 11 1 12 Mombasa 808,221 6 6 12 Malindi 338,077 11 1 12 Subtotal 2,353,777 37 10 1 48 Western 3,885,290 Bungoma 1,096486 8 2 2 2 B-Mumias 539,671 3 2 5 Kakamega. 749,861 3 1 1 5 Busia . 442,692 10 1 1 12 Lugari 268,320 1 1 2 Vihiga 595,180 12 1 13 Subtotal 3,692,210 37 8 4 49 Total 6,812,563 6,045,987 74 18 5 97 *Source: Population Projections for 2005; 1999 Kenya Population and Housing Census. As can be seen in the table above, the total population of the AMKENI-assisted districts encompasses the majority of the total province populations in both Coast and the Western Provinces. Overall, ten districts assisted by AMKENI make up close to 89 percent of the total province populations. This percentage is higher in the Western Province (95%) compared with the Coast (80%). 27 The team attempted to estimate the population coverage of AMKENI interventions within the project districts by assessing the catchment areas of the AMKENI assisted facilities. It appears that on average, AMKENI was able to reach a critical mass of the populations living in project districts: 50 percent of the populations living in AMKENI-assisted districts have been reached by AMKENI (See Figure 5). This ratio is significantly higher in Coast, (59%), compared with the Western Province (47%). It should be noted that these figures represent coverage of training and services delivery activities. Figure 5 AMKENI Population Coverage in Coast and Western Provinces 6.8 6.0 3.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 Total population of the provinces AMKENI districts' population AMKENI facilities' catchment area population million Facility Types and Levels AMKENI works with several levels and types of health facilities within both private and public sectors. Table 4 provides a breakdown of facilities by type and level. Table 4 Breakdown of AMKENI-assisted facilities by facility type and level Facility Level Facility Type Public Private Mission Total Dispensary/Clinic 24 5 1 30 Health Center 37 0 2 39 Hospital 13 2 2 17 Nursing/Maternity 0 11 0 11 Total 74 18 5 97 28 As seen in Table 4, AMKENI activities have been geared towards working with the public sector. 74 percent of the total AMKENI-assisted facilities are public facilities, as opposed to privately owned or Mission facilities. Similarly, AMKENI’s focus has been on lower-level facilities such as maternity homes, dispensaries and health centers as opposed to hospitals. Within the Kenyan health service delivery networks, demand for services and service utilization significantly increases at the lower level health facilities. The health centers, and particularly the dispensaries have considerably higher client loads per staff compared with hospitals. Pace of implementation The team has observed that project components have progressed at different pace. In general, training and service delivery components have progressed at a higher pace compared with BCC. Particularly in the West, BCC interventions have been highly intensive, requiring more resources and time. As will be discussed later in this report, the team estimates that the BCC activities have reached to about 20% of the target district populations. Conclusions and Lessons Learned The team concludes that AMKENI has already reached out to the majority of the Province populations. The districts that are left out have low population density and are hard to reach areas. The catchment areas of the existing AMKENI-assisted facilities also have high coverage but more data and further analysis is needed to determine the exact coverage of the Project. The team believes that the mix of public/private facilities is appropriate for achieving the goals of the project. A review of the initial site assessment and the team’s observations in the field confirmed that the project resources had to be shifted towards public sector in order to achieve higher impact. The team believes that AMKENI’s approach to direct more resources to working with the lower level facilities rather than the hospitals was valid and appropriate. However, the resources required for upgrading hospitals and training their staff were probably higher than the assistance required for health centers and dispensaries. Thus, the project might have invested more for the hospitals although the number of hospitals is fewer than the health centers and dispensaries. The team had detailed discussions with the AMKENI and MOH staff at the field regarding their views on future programming. AMKENI and MOH staff is keen on expanding the project activities province-wide in both Coast and Western Provinces to include other districts where AMKENI has not been active yet. In addition, many think that AMKENI should work in other facilities that have not benefited from the project over the last four years. The team does not believe such an over-reaching perspective is strategic or feasible. The 10 districts covered by AMKENI so far comprise over 80 percent of the total population of the two provinces. The remaining districts are small, with fewer inhabitants and most importantly, are hard to reach areas. The resources required to reach these districts will be too high and the approach will not be cost effective. 29 TRAINING AND SUPERVISION The AMKENI goal of a sustainable increase in the quality of, demand for and utilization of integrated FP/RH/CS services including HIV/AIDS at the community level required well trained and supervised staff. In this regard, AMKENI formulated Training and Supervision (T&S) interventions with the purpose to improve the quality of services in the public sector through a combination of direct training in specific skills and development of sustainable training and supervision systems at the national level. The T&S component implemented four main interventions: Assistance in development of sustainable training and supervision systems and development of national level training materials: Following change from the original mandate and also as a result of EMR, AMKENI worked with the MOH to consolidate existing training and supervision sub-systems into one Decentralized RH Training and Supervision System. The activities also included development of several national T&S curricula and manuals. Achievements included: ß Establishment of decentralized RH T&S System. At the time of this evaluation, AMKENI had completed training in clinical training skills and facilitative supervision of all eight provincial teams nationwide, all 10 target district teams in Coast and Western provinces, and 16 facility-based teams in Bungoma and Malindi districts. Overall, 200 supervisors have been trained nationally. A recent evaluation of the decentralized T&S system has found that the system is in place and operational. The evaluation also concluded that the system has potential for national replication. ß Development of national RH T&S curricula and manuals. These activities led to development of several national level training materials including: ÿ National Reproductive Health Curriculum including manuals for the trainers and trainees, which is currently awaiting pre-testing; ÿ National Facilitative Supervision Manuals for supervisors of the RH services; ÿ National Guidelines for Integration of FP in VCT is currently under development; ÿ In addition to the above, AMKENI participated in development of national materials for training in VCT and PMTCT. Assistance for revision of policies affecting availability of RH services: AMKENI assisted the MOH’s Division of Reproductive Health (DRH) and other stakeholders to review and revise policies affecting the availability of trained service providers. These interventions included promotion of implementation of earlier policy reforms allowing nurses and clinical officers to provided implants and PAC services, and advocacy for policy change to allow nurses and clinical officers to provide bilateral tubal ligation (BTL) and no-scalpel vasectomy (NSV). Achievements included the following: ß 417 nurses and clinical officers were trained in Norplant insertions and removals; 30 ß 101 nurses and clinical officers are trained in provision of PAC services; and, ß A concept paper advocating for policy change to allow nurses and clinical officers to provide BTL and NSV services was prepared and presented to the MOH, but was rejected. Improving performance of trainers and service providers: AMKENI worked with the provincial and district health management teams and other training institutions to strengthen their service quality through the introduction of the Performance Improvement Approach (PIA). The goal was to improve performance of trainers and service providers through training and supervision in FP/RH/CS and HIV/AIDS skills. The PIA training resulted in a total of 422 participants trained including MOH Heads of Departments, Kenya Medical Training College (KMTC) heads of departments and other staff, MOH provincial heads from Western and Coast, and many staff from MOH health facilities. The training of MOH staff in PIA helped adoption of the principles of PIA, ownership of the process, and identification of additional trainees and long-term support to trainees in district health facilities. The facility level training in PIA helped identify areas for improvement such as infection prevention, provision of wider range of contraceptive methods, improved morale, client congestion and better immunization rates. Training of trainers and services providers in FP/RH/CS and HIV/AIDS Up to now, AMKENI has trained a total of 3239 trainers and service provides at all levels, in various courses in accordance with service delivery needs, including clinical skills, contraceptive technology, facilitative supervision, VCT, PMTCT, on the job training methodology (OJT), and PAC. Table 5 presents a summary of skills training and number of personnel trained. In addition to the above, AMKENI supported pre-service training by upgrading technical skills pre-service tutors in medical colleges nationwide. A total of 250 participants from 18 colleges countrywide have been trained. Conclusions and Lessons Learned AMKENI’s training and supervision interventions have been highly successful have been valued by trainees and stakeholders interviewed by the evaluation team. At the time of the evaluation, the project had reached 85 percent of its direct training targets. The Project is not planning to expand its training activities until June 2005, except for additional training required for replaced or retired trainees. The Project is aiming to continue to assist the MOH to improve the functionality of the Decentralized RH T&S System. Overall, the Project has assisted in development of high quality training materials. The team observed that the materials have been widely distributed, and are being used to improve quality of services at the field. The team also observed that the decentralized teams are doing a commendable job of supportive supervision and on-the job training. An issue regarding 31 supervision is at times, inadequate commitment of the MOH supervisors and means of transport. MOH sometimes is not able to provide vehicles to its supervisors. Table 5 FP/RH/CS and HIV/AIDS Skills Training Skills Numbers Trained Nationwide Clinical Skills Training 241 Contraceptive Technology Update 244 Facilitative Supervision 304 FP Counseling 203 IMCI 91 Infection Prevention 229 IUD Insertion and Removal 171 Norplant Insertion and Removal 526 On-the Job-Training Methodology 60 Post-abortion Care 102 PIA Training and Orientation 422 PMTCT 171 STI Syndromic Management 199 STI/HIV/AIDS Counseling 53 VCT 223 Total 3,239 The team is concerned that the National PMTCT and VCT Training manuals developed by the MOH in 2003 have not integrated family planning counseling. AMKENI is currently helping in the development of materials for integration of family planning counseling in VCT. The team observed only a few VCT counselors who have been trained in FP and providing FP counseling to VCT clients. Yet, there are no efforts to incorporate FP counseling within PMTCT. The team believes that AMKENI’s mandate to undertake direct training activities both at the national level and at the provincial and district levels was not justified and somehow led the Project to stretch itself too thin. The Project was not able to follow-up on the outcomes of the national level training activities. For example, AMKENI trained 526 trainers and service providers in Norplant insertions and removals nationwide. The evaluation team was not able to assess the impact of national level training activities. Finally, the team has observed that AMKENI has not put in place a systematic follow-up procedure to support and evaluate the outcomes of its training efforts at the provincial and district levels. The Project probably relies on the MOH supervision system for the follow-up on the trainees; however, the team does not think that this is sufficient. One would expect that, as a project incorporating a large training component, AMKENI would have established an internal system to follow-up on trainees, not only to assess the results of training efforts but also to provide further on-the job support. 32 SERVICE DELIVERY The overall goal of AMKENI is to establish a sustainable increase in the quality of, demand for, and access to and utilization of integrated FP/RH/CS services, including HIV/AIDS. The service delivery component comprises of several technical elements, depicted in Table 1. To achieve its service delivery goal, AMKENI identified specific objectives: ß To expand the range of FP/RH/CS and HIV/AIDS services offered at target facilities and the coverage of these services through outreach; ß To institutionalize services at target facilities; ß To improve the service environment of target facilities through facility renovation and equipment supply; ß To enhance the quality of services available at target facilities; and, ß To promote client input and participation at target facilities. The team has analyzed AMKENI’s achievements in services delivery under two broad categories: ß Expansion of service availability and improvements in quality; and, ß Increased access to and utilization of services. Expansion of service availability and improvements in quality Expansion of services and quality improvement encompassed a wide range of interventions, including training, expanding existing services, establishment of new services, service outreach activities, facility renovations and provision of equipment and expendable supplies. Skills training to improve service availability and quality are discussed in detail in the previous section. Thus in this section we will focus on other interventions. Expanding existing services and establishment of new services: Family planning and selected reproductive health interventions was one of the key elements of the service delivery component of AMKENI. At baseline, most of the facilities could only offer short term methods such as pills, injection and condoms and even for these the numbers of clients were low. AMKENI strengthened the facilities to empower them to offer as many methods as possible with an emphasis on long term and permanent methods. The Project enhanced the capacity of its supported facilities by training in specific skills and by providing essential equipment and supplies. The Project trained doctor/nurse teams, and equipped theatres or minor theatres for procedures. Relevant equipment for family planning was also supplied to many AMKENI supported sites. (See Table 5 for numbers of service providers trained in FP/RH and Table 9 for equipment provided for FP/RH services). In Kenya, until recently, only doctors were allowed to provide the surgical element of PAC. To date, AMKENI has nationally trained 102 nurse/midwives and clinical officers to perform Manual Vacuum Aspiration (MVA) for uterine evacuation. The team was able to see instances where AMKENI has trained clinical offices or nurses, renovated and equipped procedure rooms or minor theatres making PAC services available and accessible nearer the communities. The training of nurses and clinical officers in PAC and implant insertion and 33 removal, for example, made these services available in health centers and dispensaries for the first time. As seen in Figure 6, there was marked increase especially in the number of facilities that could provide IUDs, implants, vasectomy, BTL, and PAC. Figure 6 AMKENI Supported Facilities by FP/RH Services Provided 0 20 40 60 80 100 120 PAC Maternity care Pills Injectables Norplant IUD Tubal ligations # of facilities Baseline (2001) 2004 Infection prevention practices were integrated in all service delivery interventions as well as in waste disposal. AMKENI trained staff in infection prevention and ensured availability of bleach for decontamination. The Project also provided sterilizers, autoclaves, and assisted in setting up incinerators, and placenta pits. AMKENI improved antenatal care by providing training for staff, equipment and supplies to health facilities, and laboratory tests. The equipment included weighing scales and examination couches. Health workers in health centers and dispensaries were trained in EOC to enable them to conduct appropriate antenatal care. During antenatal care pregnant women were given Sulphamethoxazole-pyrimithamine (SP) for malaria prophylaxis, tetanus toxoid immunization to prevent neonatal tetanus and Vitamin A to improve the immunity status of the newborn child. Many AMKENI facilities were empowered to conduct normal and complicated deliveries. This was achieved through provision of EOC training, providing delivery beds and delivery packs and renovation of maternity units. Some facilities were enabled to conduct complicated deliveries such as caesarean sections. The number of facilities capable of conducting deliveries was increased by 22%. The evaluation team visited several facilities that were assisted by AMKENI to establish or improve maternity care and confirmed that in fact training, supply of equipment, renovations and even assistance in building of maternity units were done. In 2003, AMKENI expanded its HIV/AIDS interventions in its project activities when funds from the PEPFAR track became available. In the original RFA, USAID had shown that the 34 expanding epidemic threat of HIV/AIDS in Kenya was escalating as shown by prevalence rates reported elsewhere in this report. AMKENI response by including HIV/AIDS in its activities appears justified even at that late period of inclusion and even though other USAID funded organizations are carrying out such activities. Figure 7 Maternity Services Capabilities of Facilities before and after AMKENI Interventions 0 10 20 30 40 50 60 70 80 90 100 Normal delivery Removal of retained placenta Blood transfusion Cesarean section # of faciities 2001 2004 AMKENI, in collaboration with the MOH, identified sites in the project districts where it was feasible to introduce additional VCT centers. Where a room was available AMKENI renovated it for VCT; where there was none, the Project assisted the community to establish one. VCT services are supported by National AIDS Control Program (NASCOP), which supplies test kits and supports training of staff. The VCT centers are stocked with the required laboratory and test kits. The team found that the VCT centers are utilized well by the communities; in many facilities, the demand had surpassed the available services. However, in a number of facilities the availability of HIV test kits ran short of supply, frustrating the clients. AMKENI has also established PMTCT services, which counsel antenatal clients, test for HIV, and distribute Niverapine as well. Table 6 shows selected activities undertaken by AMKENI to support HIV/AIDS activities. The team has observed that in some PMTCT and VCT centers, basic family planning counseling is available. However there was no evidence of full integration of FP with VCT and PMTCT in the facilities visited. So far, AMKENI has established 43 new VCT sites and 37 new PMTCT sites, in addition to the sites that were previously established in AMKENI-targeted facilities. Currently 73 of the 97 AMKENI supported sites are providing VCT services and 75 are providing PMTCT. Although AMKENI has not established all these sites, the Project continues to provide support to all facilities. 20 of VCT sites are included in operations research to evaluate the implementation of FP and VCT integration. The results will provide input for the 35 development of training modules to include in the VCT and PMTCT curricula. PMTCT and VCT services and clients are shown in Tables 7 and 8. In the early phases of AMKENI, child survival interventions were included among the main areas of project focus. As the high child hood mortality would influence utilization rates of FP services, it would have been expected AMKENI would strengthen child survival in order to uplift the acceptability of FP which formed major emphasis in its project. But AMKENI had four scattered areas of child survival strategies addressed i.e., safe delivery, immunization, support for growth monitoring and IMCI. Even then the funding for CS was very low at only 2% of the project funding. Table 6 Selected Activities related to HIV/AIDS Activity Number* Staff trained in IP 110 Staff trained in VCT 140 Staff trained in PMTCT 123 Health facilities received PMTCT whole site orientation 45 *Figures only include staff from the 97 AMKENI-supported facilities Table 7 VCT Services Number Health facilities offering VCT services 73 VCT services initiated with AMKENI support 37 Total VCT clients counseled and tested 46,959* Monthly average July-Sept 2004 4,672* *Figures include clients served at all VCT sites since 2002. Table 8 PMTCT Services Number Health facilities offering PMTCT services 75 PMTCT services initiated with AMKENI support 43 Total ANC clients counseled and tested 21,509* ANC clients testing positive 1,595* ANC clients received Niverapine 451* *Figures include clients served at all PMTCT sites since 2002. AMKENI has strengthened safe child birth by improving safe motherhood through improved maternity services – has renovated, even helped in building maternity units and equipped them with delivery beds, baby incubators resuscitation equipment for the newborn and the newborn weighing scales. The team found that other collaborators including MOH also strengthened this area of early childcare. In the strategy of immunization, AMKENI assisted 36 in advocacy and awareness through health messages in songs and drama. AMKENI also implemented a small project in Kilifi District with funding from Bill and Melinda Gates Foundation, using child-to-child communication to improve immunization coverage. In immunization the team found that the Kenya Expanded Program on Immunization (KEPI) of the MOH was very strong on the ground both in the Coast Province and Western Province. The team found that AMKENI supported growth monitoring by providing children weighing scales in many health facilities. Although AMKENI participated in conducting some training in IMCI, this was minimal. In various health facilities, particularly in Western province, the staff expressed their need for more IMCI training, stating that the four days’ lectures in IMCI they were given were inadequate. Out-reach activities: One of the innovative approaches of AMKENI has been to facilitate outreach activities for long-term and permanent methods, in collaboration with the MOH. The intention was to organize frequent outreach activities, especially for BTL, in health centers located at a distance from district hospitals in an effort to make services more accessible to the communities. AMKENI facilitated training for nurse/doctor teams from supported hospitals and helped to put in place a schedule for provision of BTLs in these facilities. The team has observed that outreach activities are now being conducted successfully in many health centers, and has contributed substantially to increasing access to BTL services. Additionally, AMKENI has supported integrated and comprehensive VCT outreach services. Facility renovations and provision of equipment: AMKENI’s provided substantial resources to improve the service environment at the target facilities through facility renovations, provision of essential equipment and expendable supplies. Minor renovations were conducted in selected health facilities to improve procedures in terms of space, confidentiality and quality, for example for implant and IUD insertions, and PAC and maternity services. There were also assistance to upgrade small district hospitals and health centers to help to provide more comprehensive services. Throughout the life of the Project 37 health facilities received funding for renovations. AMKENI provided comprehensive list of essential equipment to the health facilities. A complete list is found in Table 9. AMKENI also realized the need for safe disposal of all waste materials from the service delivery areas, and embarked on construction of incinerators. So far 20 such structures have been constructed within the Project districts. Infection prevention measures through the supply and use of chlorine were started in clinical areas. Roof water catchments or bore hole were effected in some health facilities; water tanks and necessary plumbing were provided by the project. Finally, AMKENI supplied necessary laboratory equipment and expendables to enable laboratory technicians perform biochemical analysis and hemoglobin estimation. Increased access to and utilization of services Overall, the team found that AMKENI interventions have contributed to increased number of family planning acceptors. Marked improvement is especially noted for long term and permanent methods. Increases in use of long-term and permanent methods indicate improvements in contraceptive method mix, which is heavily skewed toward injectable use in Kenya. Changes in family planning acceptance for both short term and long-term and permanent methods are found in Table 10. 37 Table 9 Equipment Provided by AMKENI to Supported Facilities by September 2004 # Description of item # Description of item 8 Absorber machine 4 Operating lamp 8 Anesthetic machine 39 Operating table 137 Angle poise lamp 21 Oxygen concentrator 44 Baby incubator 34 Oxygen flowmeter 86 Baby Resuscitator 256 Scales 7 Bench 103 Screen 102 BP machine 14 Signboard 19 Cabinet 124 Sterilizer-autoclave 3 Caesarean section set 118 Sterilizer-boiling 94 Chair 271 Sterilizer-drums 85 Cheatle jars 7 Sterilizer-steam 119 Cheatle forceps 79 Suction machine 160 Delivery set 4 Table 82 Delivery bed 97 Torches 224 Drip stand 4 Trocars 115 Endotracheal tube 341 Trolley 85 Examination couch 22 Vacuum extractor 24 Fridge 25 Vasectomy set 28 Instrument container 1 Video machine 1 Lantern 1 Water pump 60 Laryngoscope 1 Water tank 35 Minilap set Table 10 Trends in Family Planning Acceptance Method acceptors Baseline (2001) 2004 Percent change Short-term methods 127,740 173,606 +36 Long term & permanent methods 2,335 3,758 +61 Total Acceptors 130,075 177,364 +36 It should be noted that in Table 10, figures for short-term methods might be inflated due to the definition of family planning acceptance used by AMKENI. The Project reports family planning acceptance for short-term methods as the sum of new users and continuous users annually. Although the trend would be the same, the figures would have been lower for both years if the correct definition were used. 38 Total number of deliveries from all AMKENI supported facilities have also risen significantly between 2001 and 2004. The trend of reported deliveries as indicated in Figure 8 shows remarkable improvement. Figure 8 Annual Number of Deliveries From All AMKENI Supported Facilities 0 5,000 10,000 15,000 20,000 25,000 2001 2002 2003 2004 Normal Complicated There were however discrepancies between ANC attendance and deliveries in health facilities. This finding is reflected in KDHS 2003, which reports skilled personnel assist only 42 percent of deliveries. A few maternal deaths were reported in some health facilities. Some of them from outside the catchment areas of the facilities, may not have attended ANC. The deaths, therefore, do not necessarily reflect on the quality of service of those facilities. Rather than staff calculating maternal mortality ratios, the team advised them to conduct maternal death reviews to find avoidable factors for each case as lesson for future. By 2005, 65 facilities were offering PAC services, depicted in Table 11. It should be noted that only one half year of PAC data is available for 2002, so the reported value of 661 clients in 2002 should be multiplied by two to estimate an annual number. The total number of clients served could be estimated to have increased by 42 percent from 2002 to 2004. Table 11 Trends in PAC Services 2002* 2003 2004 # of PAC clients 661 1,641 1,880 # of facilities providing PAC services 43 62 65 Only one half year of PAC data is available for 2002. 39 Conclusions and Lessons Learned High input in training and supervision, equipment and renovations for family planning has lead to high quality service and to increased utilization. Training, supervision and family planning service delivery are generally well integrated. Frequent stock outs, shortage of staff in health centers and dispensaries and lack of integration in some facilities are adversely affecting utilization. The project has too many service delivery components, and risks losing focus from family planning and selected reproductive health services. There were, however, challenges that adversely affected family planning utilization. In some facilities visited family planning services were not fully integrated with other maternal and child health HIV/AIDS services. Facility client flow arrangements adversely affected the utilization. For example, in one case, family planning clients had to queue with a large number of sick children and adults to get served while another section of the facility has spacious rooms serving only a few PMTCT or VCT clients. In another facility, FP clients are first seen in the antenatal room next to the overall maternal and child health and HIV/AIDS services and then walk across the compound to receive an IUD or an implant. They then walk back to the clinic to complete the documentation. Although everything else was in place in terms of trained staff, equipment and supplies, the situations cited lead to client dissatisfaction and cause reduced utilization. The HIV/AIDS component of AMKENI is doing well and has implemented some HIV/AIDS activities in collaboration with NASCOP and COPHIA and IMPACT Projects. However they have not integrated fully with family planning services and other service delivery elements. In child survival, AMKENI’s assistance has been valuable, however, the Project has played little role as it’s funding was minimal. VCT services are very popular and AMKENI is beginning to consolidate and integrate the service with FP and other RH services. As noted previously, the team has not observed any efforts to incorporate family planning counseling into PMTCT. The evaluation team has observed three main issues that have been negatively affecting FP/RH/CS and HIV/AIDS service availability, quality and utilization during field trips. These challenges are not necessarily within the mandate the AMKENI Project. However, the team believes that they should be mentioned here since they are crosscutting challenges that will continue to have a negative impact on services in the future. ß Contraceptive and HIV test kit shortages: Frequent stock outs of contraceptive commodities are hampering utilization, skewing the method mix and adversely affecting clients’ choices. In a number of facilities the team visited the method mix was erratic and differed month-to-month depending on what methods were available. The team learnt that stock outs stem from two factors; first there has been frequent nationwide stock outs due to donor dependency and secondly, local stock outs occur due to logistical issues between the district MOH and facilities. The team was informed by the Head of Reproductive Health Division that there are no contraceptive shortages nationwide at the time of this evaluation. We were also informed that USAID-funded DELIVER Project distributes contraceptive commodities to the district level and has trained district MOH staff in logistics management. However, during field trips the team observed that only two of the facilities visited had all 40 contraceptives available. Facility level stock outs seem to stem from the fact that staff often does not prepare their returns on time and therefore cannot obtain supplies. Lack of transportation may also play a role in contraceptive stock outs. Similarly, the team was informed by NASCOP that currently they do have adequate stocks of HIV test kits, however, availability of test kits at the facility level was inconsistent, as frequent shortages occurred. ß Abolishment of MOH’s cost-sharing policy: Cost sharing was started in Kenya to enable facilities supplement the Government efforts in financing health care. With cost sharing funds, government health facilities were able to employ supplementary staff, both clinical and support, purchase drugs not provided by the MOH, purchase non-pharmaceutical supplies, hire transport for emergency referrals, and provide essential laboratory tests. In mid 2004, the Minister of Health abolished cost sharing in government health centers and dispensaries. The new policy sets a low fixed payment base for attendance at government health centers and dispensaries and eliminates community contributions and other charges. This is the single most important development that has seriously affected access and quality of services in AMKENI supported facilities. The policy has been erroneously equated to free services and the initial reaction to the directive was a sudden upsurge of client load, mainly minor curative services, that was overwhelming to the few staff at the facilities. The result was that preventive services e.g., FP/RH, VCT and PMTCT tended to be relegated to the back seat. Our observation in the areas we visited is that both staff and members of the communities are demoralized and their stamina to continue supporting development is eroded. The policy change is also having a serious detrimental effect on quality of services since the facilities are now lacking the funds to hire additional staff and purchase essential commodities and supplies. Our most important finding is that communities would like to have the policy reversed since they are willing to pay affordable fees for better quality services, as they were in the past. ß Staff shortages: Finally, shortages of staff adversely affect service availability, access and quality. Staff shortages are particularly an issue at the lower level facilities, e.g., health centers and dispensaries. The team has observed dispensaries serving more than 100 clients a day with only one or two nurses. Abolishment of the cost sharing policy has limited the facilities’ ability to hire additional staff, while MOH is not able to assign more service providers to these short-staffed sites. BEHAVIOR CHANGE AND COMMUNICATION The strategic objectives of the BCC component focus on helping individuals and communities to better understand and act upon their FP/RH/CS and HIV/AIDS needs. AMKENI uses a variety of partners and approaches to reach men, women and youth in communities served by target facilities. The Project also works with community groups and structures, helping to create an environment that supports positive change. In phase one of the project BCC activities were conducted in the field by several local implementing partners, including AKHS, CAI, CLUSA, FPAK, Kenya Music Festival Foundation (KMFF), UZIMA 41 Foundation and World Relief. However, in phase two the operating partners remain AKHS, FPAK, CLUSA, CAI and UZIMA. The project has a strong focus on linking communities and services using variety of complementary strategies of community mobilization and sensitization. BCC’s three key objectives continue to focus on increasing individual/community knowledge and `agency’ with regards to their health –in terms of both preventive and appropriate health seeking behavior. Specifically, the strategy seeks to: ß Promoting preventive and health-seeking behavior ß Fostering community agency ß Creating a supportive environment for individual and community change BCC Approaches A summary of the different approaches that AMKENI uses in its work with communities is presented below. ß Family-to Family Communication: The concept of family-to-family communication includes interactions between individual members of a family unit (e.g. parents and their children), as well as between separate families (e.g. two families that live in the same village). AMKENI works with a core group of “peer families” to explore relationships and health issues, and to discuss health enhancing family values and behaviors. As participating family members acquire skills, discuss real-life situations, re-define roles and seek health solutions, their insights are shared with other interested families and community members through family-to-family peer education, linkages with community structures and groups, larger “family bazaars”. ß Direct outreach to men, women and youth: In addition to working with family units, AMKENI continues to reach out to individual family members- men, women and youth- to raise the same issues of family relationships and health, as well as addressing the specific health needs of each target audience. Individuals have health concerns that may best and most quickly be addressed through focused activities. Therefore, AMKENI uses established networks/channels to reach discrete audiences including men at work places, youth in clubs and women in community groups. ß Women’s agency: Women’s agency is a term that describes the involvement of women as pro-active agents of change. A cornerstone of AMKENI’s approach, greater participation of women in health related decision making and utilization, helps create sustainable impact at the community level. The goal of fostering women’s agency involves working with the community as a whole to create a supportive environment for women’s agency. In addition to this, AMKENI conducts specific activities with women, both individually and through women’s organizations, which strengthen their skills/roles as advocates and decision-makers in health. ß Magnification of behavior change: New health norms and behaviors are emerging that should be shared beyond the family, village, workplace, church, school or group to let the larger community know that change has begun. Through its implementing 42 partners and networks, AMKENI works to identify desirable behavior change as it happens and magnify it quickly and effectively to a larger audience. Activities for magnification include providing health messages (via a range of discussion groups), linking to drama outreach activities, creating materials, and using community forums (e.g. family bazaars, barazas). Following the EMR, AMKENI revised its BCC workplan that incorporated three key areas; consolidating geographic and programmatic scope, strengthening MOH participation at all levels, and monitoring changes in knowledge, attitudes and behavior. This plan also included new activities addressing VCT and PMTCT. A significant change in the implementation of BCC was the involvement of Public Health Officers (PHOs) and Public Health Technicians (PHTs) in community activities. AMKENI also made efforts to consolidate its programmatic scope by focusing on partners and interventions that had performed well, thus engaging AKHS, CAI, CLUSA, FPAK and UZIMA. Although interventions such as Peer Family and Gender Based Violence continued in phase one districts they were not extended to phase two districts. During the fourth quarter, and with community networks active in all districts, the BCC strategy focused on supporting these networks and monitoring their activities. Implementing partners The engagement of implementing partners was based on the project’s broad mandate of addressing different audiences as well as broad areas of FP/RH/CS including HIV/AIDS. Table 2 in the previous section of this report summarizes the approaches used in BCC interventions in collaboration with the local implementing partners, in both the Coast and Western Provinces. ß CLUSA: BCC efforts under women’s agency target all segments of the community￾men, women youth and school children in Western Kenya. CLUSA in Western has a focus at the village/community level building up to the facility level. Although there is support at the facility level, capacity building in management was not done to the emphasis of AKHS in Coast province. ß Aga Khan Health Services (AKHS): The approach under AKHS aimed to increase community participation in managing and utilizing rural health facilities and to provide platforms for increased women’s agency in advocacy and decision making in reproductive health at the individual level. The main thrust is on capacity building at the facility level with supportive activities at the community level that contribute to demand creation and increased service utilization. This was only implemented in selected health facilities in Coast Province. ß Family Planning Association of Kenya (FPAK): FPAK’s approach is to increase young people’s ability to understand HIV/AIDS and related reproductive health issues. In addition, the purpose is to enhance their ability to take health action and to support community outreach activities by established youth groups using traditional media. FPAK works in Coast Province only. 43 ß Community Aid International (CAI): CAI set to reach men in the community via formal and informal sector workplaces or groups or both, with health information activities. CAI works in both provinces but more prominent in Coast Province where there is a larger array of established workplaces. ß UZIMA Foundation: Much like FPAK, UZIMA focuses on increasing young people’s ability to understand HIV/AIDS and related reproductive health issues. It also strives to enhance their ability to take health action and to support community outreach activities by established youth groups using traditional media. UZIMA works in Western Province only. BCC progress and achievements to date Table 12 BCC Inputs and progress through 2004 Activity Number # of field agents trained 1,300 # of men and women reached 1,000,000 # of villages worked with 400 # of women group members reached 16,000 # of men reached at worksites 50,000 # of youth groups reached 450 # of primary schools reached 49 # of peer families reached 148 # of PHO/PHTs trained 185 # of small grants to groups 19 Table 12 depicts the overall BCC inputs and progress through the end of 2004. The BCC component has continued to monitor its activities using tools that focus on changes in knowledge, attitude and behavior. From the evaluation team’s discussions with project personnel, the MOH and community members, it is evident that the project has had a positive influence on the knowledge, attitude and behavior of individuals in project sites. This is further corroborated by the preliminary findings of the Household Survey undertaken at the end of 2004. These are presented selectively in Figure 9 to demonstrate the effect of BCC activities in the community. Figure 9 compares the findings from AMKENI intervention areas with findings from control areas where AMKENI did not work. Selected indicators shown in this figure are: ß Indicator 1: Percentage of women knowing of key RH issues ß Indicator 2: Percentage of women using a FP method ß Indicator 3: Percentage of non-users intending to use a FP method ß Indicator 4: Percentage of women knowing about 3 STDs ß Indicator 5: Percentage of women knowing use of condoms for dual protection ß Indicator 6: Percentage of women who talked to husbands about condom use 44 Figure 9 Knowledge and Behavior in Intervention and Control Areas 0 10 20 30 40 50 60 70 80 90 Ind. 1 Ind. 2 Ind. 3 Ind. 4 Ind. 5 Ind. 6 percentage AMKENI + AMKENI - Depending on whether a community has been exposed to the full complement of BCC interventions and the community response in terms of participation (high or low), utilization data at the health facility from 2000 to 2004 has been analyzed and compared to baseline. The general observation is that those with full intervention record significant changes in the pattern of FP acceptors by quarter. This trend is corroborated by the group discussions which pointed to the fact that through BCC activities, service utilization at all levels has improved including for FP. The figure below illustrates the community /facility interaction in selected facilities. Figure 10 Service Utilization in Selected BCC Intervention Facilities 0 100 200 300 400 500 600 700 800 900 Jibana Kizingo Lenga Mgamboni percent increase in services Total utilization* FP utilization *Total number of clients served by the clinic. 45 Another strategy to expand access to services used by AMKENI was through outreach, which also offers opportunity for mobilization of other services. Outreach activities are largely integrated with other services. An outreach provided opportunity for registration of VCT, BTL and attendance of ANC. The field facilitators and animators under the BCC component used this as an opportunity to communicate relevant health messages. The evaluation team had an opportunity to witness one of the outreach activities held at Mihuu Dispensary in Bungoma District. The numbers seeking both curative and preventive services were overwhelming and a majority had to be referred. Although it is difficult to come up with actual number of people reached by BCC messages the catchment population of a given facility is estimated as the potential population reached. Estimates of populations reached by BCC messages are 1,000,000 people, or about 20% of the target population. During the project period 400 villages were earmarked and reached. However, through replication with minimum project input, the Project estimates that 500 additional villages have been reached with BCC messages; an indication that the messages are valued by the community beyond the project sites. Community perspectives, involvement and concerns The community response to the project has been positive and this is evident in the level of engagement. In the selected discussion groups participants noted that they had benefited from the Project through education in FP/RH/CS and particularly on VCT and PMTCT, contributing to behavior change in the community. ANC attendance has improved greatly and mothers are urged to go early enough to take advantage of PMTCT services. Though traditional birth attendants (TBAs) are still used widely, their role is slowly changing, playing a greater role in referral. In Kwale and Malindi Districts, for example, the team was informed that TBAs often accompany mothers to the health facility for delivery. At Gongoni Dispensary the community was satisfied with the health facility and suggested that it be upgraded to health center status. In Nangina Dispensary community members applauded the provision of services that incorporated privacy and confidentiality especially with regard to VCT. A key aspect of the project has been harnessing community resources and facilitating the linkage between the community and the health facility. Concurrently, the inputs in the adjacent health facilities have in tandem led to changes in the community perception of the health facilities. In Chwele, for example, it was stated that the communities now feel that they own the facility. The community is involved in decision-making and an example was given where through the influence of community representatives, the costs of male circumcision was adjusted from Kenya shillings 700 to 200. The Project’s uniqueness lies in its approach of improving services at the facility level and mobilizing communities to demand for quality services. The challenge then becomes maintaining the balance between the two components; so that once communities are fully mobilized and come for services, these would be in place. Community concerns were therefore centered on services that they expected but were not in place at the time they need to access them. These included availability of appropriate FP methods, VCT kits and ARVs. 46 The community and service delivery components are dependent on each other in the sense that the performance of one is a mirror reflection of the other. In the discussion groups it was evident that community mobilization and demand creation has taken place. However, the bigger challenge is at the health facility where constraints range from staff shortage to negative staff attitude in some facilities and shortage of supplies, affecting the quality of service. In all the sample communities visited, their overall perception of the services was that there had been tremendous improvement over the project period. It was only in one specific facility in Western Province that there were complains of poor quality services attributed to negative provider attitude. This is of concern given that the project has invested in a range of trainings that incorporated quality of care. Conclusions and Lessons Learned In the BCC component, the project employed a multiplicity of approaches and these have borne fruits. Because these approaches are employed in different sites and at varied levels, it is not possible to delineate the contribution of each of the approaches singly. The Project fully appreciates this and notes that one of the challenges of having such a complex approach is that it becomes difficult to isolate the impact that any one intervention or package of interventions is having on the uptake of RH services. However, two of the implementing partners stand out because of their larger scope of work, namely CLUSA and AKHS. In describing the intensity of BCC involvement either of these implementing partners had to be present as the core around which other partners rallied. The main models are Community/Women’s Agency Model (CLUSA) and Community Participatory Model (AKHS) implemented in the Western and Coast provinces respectively. Other implementing partners played a supportive role in bringing specific approaches based on their experience in the relevant fields. The youth drama groups were particularly resourceful in community mobilization and transfer of health messages. Below is a detailed description of the two main models. Diagrammatic illustrations of both models are shown in Figures 11 and 12. ß CLUSA Community/Women Agency Model: Using the participatory approach the process of community mobilization entails various steps around existing community structures, the village health committees if they already exist or new ones are formed. The composition reflects a gender balance, a man and a woman from each village. From the project side, the facilitators are trained in a number of skills, which they in turn pass on to the volunteer animators and PHOs and PHTs who work with communities as mobilizers. The facilitators take the community through problem identification, analysis and ranking. Animators and the Village Health Committees develop comprehensive action plans, which often go beyond reproductive health and family planning issues to incorporate poverty alleviation, water and sanitation. The sub-location health coordinating committees are formed by representatives from the Village Health Committees and constitute the bridge between the health facility and the community. ß Community Participatory Model (AKHS): This is a modification of the community participation approach used by AKHS in its primary health care projects over the years. A key aspect is the strengthening of management systems at the facility level, working its way back to the community level. Through the health facility they work with the PHOs/PHTs who in turn work with the Village Health Committees and 47 community health workers/community resource persons who are linked to the facility. At the community level they assist in the formation of village health committees and building capacity through training as well as defining roles, responsibilities, processes, planning and information/management systems. Management and financial skills are key ingredients in capacity building at the facility level. Figure 11 The CLUSA Model in the Western Province AMKENI Project Health Center / Dispensaries Health Center / Dispensary Health Committees Women's Agency Coordinator (CLUSA) Communities Animators Field Facilitators Lead Animators Sublocation Management Managemenet Committees Youth Drama Groups PHOs and PHTs Village Health Committes 48 Figure 12 The AKHS Model in Coast Province AMKENI Project Health Center / Dispensaries Health Center / Dispensary Health Committees BCC Coordinator Communities Community Health Workers PHOs PHTS Aga Khan Health Services Village Health Committees The team believes that AMKENI should have revisited the methodological approaches earlier during the course of the Project with the view of coming up with a hybrid model that has optimum effect and uniformly implementing this model in the project sites. This implies, taking the two main models (AKHS and CLUSA), teasing out what works best and combining into a hybrid BCC model. An interesting observation is the analysis undertaken by the project based on whether the intervention is “full” or “basic” and whether the attendant response or community engagement is “high’ or ‘low’. If high community response were the prerequisite for change then one would analyze all the models, isolating what produces this result as a key ingredient in the hybrid model. This means scaling down the approaches and keep those that still would achieve optimal results with less intensity and resources. There is a need for a uniform and comprehensive approach that can be documented for replication. At this stage, the Project is in a position to look at the activities against the inputs 49 and come up with estimation of the costs of each of the models. This is important given the fact that in all the reports for BCC, resource constraints were raised as an issue. The evaluation team envisages that an intervention that focuses on the community, encompassing selected interventions with the PHOs and PHTs playing a major role as facilitators would facilitate sustainability. The incorporation of PHOs and PHTs has greatly boosted the implementation of the BCC component. This cadre of health providers has a wider mandate and covers a larger area. PHOs/PHTs are a crucial part of a structural continuum that links communities to MOH. It is acknowledged that they are often transferred and this applies to all other cadre of service providers and therefore the issue of sustainability would have to be applied across the board to all facets of the project. PHT/PHO activities are still constrained by transport. The evaluation team feels that besides the emphasis on the development and training of animators to replace the project facilitators, the PHOs/PHTs deserve more support. This issue was also pointed out at the EMR 2003. Overall, BCC activities have contributed to increased demand and utilization of services. However, the implementation of several BCC strategies is labor intensive and costly. In discussions with the BCC coordinators, it was acknowledged that inasmuch as the approach has mobilized the community for services; it was also noted to be labor intensive as well as requiring a lot of resources. In terms of resource allocation, BCC component has the largest share of the budget compared to other components. Acknowledging the contribution of communities makes it even more resource and labor intensive. BCC activities/messages are increasingly inclined towards HIV/AIDS and VCT/ PMTCT and it seemed much less on FP/RH the core business of the Project. This focus at the community level has to be seen in the context of programmatic shifts with the entry of PEPFAR funds. This is the push effect of the Project but also is indicative of the priority needs of the community. On the ground, the team observed BCC activities have gone beyond this to include other facets of health. Granted that communities may apply the skills gained at their own will and extend to other community priority areas beyond FP/RH and health in general, but the Project needs to aggressively push for behaviors that address the core of their mandate. Currently, in this Project we see a lot more than FP/RH and in some instances it may lean much more towards general public health issues, such as the focus on water, sanitation, malaria and typhoid fever, which are real priority areas in most communities. The dilemma in a participatory approach is that the project may be community-needs driven and these may not always be in tandem with the original project strategic objectives. The question is how does one maintain the balance between addressing community identified needs and at the same time steer the community to focus on the core business of the Project. In sum, the team concludes that the main lessons learned from the BCC component of AMKENI are: ß A comprehensive approach in community mobilization, BCC interventions and concomitant improvement of services at the facility level make a difference by contributing to demand creation and increased service utilization. 50 ß The selection of facilities has to go hand in hand with catchment area analysis so that community mobilization for use of services is appropriately linked to availability of services within reach of the community to avoid frustration. ß Outreach services continue to provide services as a stopgap measure to communities that have no access to such services, averting unnecessary referral. ß The project strategic objective may not always be in tandem with community priority needs and in responding to this one may inadvertently focus less on the project mandate. ß A successful community mobilization intervention is likely to trigger replication of the activities in adjacent communities and beyond with minimum support from the project. This is a good measure of the impact of the BCC strategy and the project as a whole. CAPACITY BUILDING AND SUSTAINABILITY Capacity building is one of the major outcomes of the AMKENI project. This has been implemented at all levels; the national, provincial, district, facility and the community. The technical input for the Decentralized RH T&S Teams and the production of relevant training materials have ensured that nationally these teams will support future trainings and supervision long after the project. To enhance service delivery, the project has supported the DHMTs to strengthen their capacity in training, management, monitoring and supervision. In collaboration with the DHMTs the Project has worked to strengthen the role of the Health Center Development Committees (HCDCs), Dispensary Health Committees (DHCs) and Village Health Committees in order to enlist community involvement and promote service utilization at the facility level. In the discussions with these groups it was evident that one of their lasting benefits from the project is relevant skills that have enabled them to perform better towards ensuring quality care to community members. The decision to build capacity of existing community groups and structures in some measure ensures that these groups can sustain the activities and would use these skills to attract other partners. Financial management and record keeping among the HCDCs and communication and counseling skills among the animators were cited. One of the key observations of the team was the Project has enriched the skills of the local implementers not only technically but also in community mobilization. For example, in three community discussions in Western Kenya field health workers and women network members acclaimed that “We are AMKENI”, an indication of high level of ownership and that given the skills acquired they would be able to push the AMKENI agenda forward after the project. The incorporation of PHOs and PHTs in Project activities at the community level no doubt ensures that long after the project certain activities are fully entrenched given that this cadre 51 of health providers are part of the MOH structure that operates up to the village level. PHOs and PHTs are new entrants in the Project, brought in after the EMR. In discussions with the local implementing partners, it was evident that although they came in as experts in their own fields, they have over time benefited from the interaction with AMKENI. This interaction has enabled them address their own goals in other projects. The local implementers stated that they have been able to transfer the newly acquired skills to other regions where they work. They have also noted that this partnership has also enabled them to acquire visibility at the community level. Physical inputs and institutional buildings are sustainable and it will be the onus of the MOH to address the issue of maintenance as the project exits. The skills in training are not only sustainable but also have the potential of a multiplier effect. The strategy of the on-the-job training is certainly a move in the right direction in terms of creating critical mass of trained staff. The approach of strengthening the existing structures both at the community and facility levels has concretized the ethic of ownership and this is critical for sustainability. In Western Province the animators have formed a local organization to be able to provide service to the community long after the project. However, they would have to be backstopped by a cadre such as the PHOs/PHTs. This is why strengthening the latter is so important. Conclusions and Lessons learned In conclusion, the evaluation team feels that the project has done a good job in capacity building both in personnel and physical inputs. Training and skills development are sustainable and replicable. The behavior change created and subsequent replication may be an aspect that stays on depending on the continued availability and quality of services at the facility level. There are certain elements, especially in BCC that are not sustainable beyond the project particularly where there are parallel systems of personnel for example the field facilitators who are part of the project. Sustainability of the infrastructure is fragile, however, where there is strong community involvement and ample management at the facility level these should be sustained though not at the level of the project. The embrace of cost sharing by the community has been one way towards sustainability of services through community support. Thus the removal of cost sharing has thrown into disarray the ethic of community ownership that has taken a lot of effort to build. It is for this reason that the project needs to discuss and start working on an exit strategy where by the MOH begins to take over some of the activities. PROJECT MANAGEMENT The AMKENI partnership AMKENI has a formal partnership among four organizations with complementary expertise and is governed by a jointly developed Management Agreement. The Management Agreement sets the following roles and responsibilities in the partnership: ß The partners: The partners have full and equal participation in the Management Committee and in providing overall strategic direction and setting the decision- 52 making framework for the project. Each partner has a lead role in one or more of the critical competencies required by the project. Because the partners also have overlapping strengths, they have worked among themselves to clarify lead roles. EngenderHealth is the lead in FP/RH services, client counseling and post-abortion care, while FHI is responsible for HIV/AIDS and STI prevention. IntraHealth leads the activities in performance improvement and supervision, training, and monitoring and evaluation. PATH has the responsibility to lead behavior change communication, child health activities, and serving the youth. ß The Management Committee: The overall management of the project rests with a Management Committee, which originally consisted of one senior representative from each of the partner organizations, and the Project Director. Each partner has appointed a senior member who resides in Kenya to represent the organization and make commitments to the project. On further discussions, USAID/Kenya CTO, and the MOH Head of Division of Reproductive Health are now included as members of the committee. The Management Committee has the overall responsibility for project performance, including review and confirmation of annual resource allocation, workplan and performance review. The committee provides oversight and strategic directions to the project team. The committee meets on a quarterly basis, and when necessary, additional meetings are also conducted. ß Managing Partner: As the managing partner, EngenderHealth is the recipient under the Cooperative Agreement, and is the primary contact point with USAID, and for all invoicing and financial reporting to USAID. ß Project Director and Project Staff: According to the Management Agreement, the Management Committee delegates full responsibility to the Project Director for managing and overseeing AMKENI. He has the authority to make staffing and key decisions in consultation with the partners, and in accordance with the ground rules and principles stated in the Management Agreement. The partners have seconded their staff to AMKENI to enable the project to work as a unified team. The project staff has their primary identity to AMKENI and secondary identity to the organization that employs them. However, they maintain a link with their home institutions to have a two-way exchange of lessons learned and technical information between the project and the home institutions. Project offices and staffing AMKENI has a main office in Nairobi to manage the project’s operations. Office space is shared with EngenderHealth’s Regional Office in Nairobi, thus achieving efficiencies and cost savings in use of space and administrative and support functions. The Nairobi Office houses the Project Director and the senior staff. The project has two field offices, one in Mombassa, Coast Province, and one in Kakamega, Western Province. The field offices house the Area Managers, technical staff and support staff. These offices are the primary implementation stations for activities in the focus districts. AMKENI staffing plan is presented in Attachment G. In Nairobi Office, there is an Operations Manager and Monitoring and Evaluation Advisor, in addition to five senior 53 technical advisors, all reporting directly to the Project Director. The technical advisors include Training and Supervision, Policy and Systems, Service Delivery, HIV/AIDS, and the Behavior Change and Communication. Each senior advisor oversees the activities of the Coordinators placed in the field offices in their respective areas of expertise. There are four coordinators in each field office, coordinating the activities in training and supervision, service delivery, HIV/AIDS, and behavior change and communication. In the Western Province office, there is an additional Women’s Agency Coordinator, and eight Field Facilitators reporting to her. The HIV/AIDS Coordinators in both field offices are shared with the IMPACT Project, working 50% of their time for AMKENI, and the rest for IMPACT. The technical staff at area offices reports administratively to the Area Managers. In total, the Project employs close to 50 staff. All staff has job descriptions, clearly outlining their roles and responsibilities. Project Reporting AMKENI prepares annual workplans including proposed activities, which are first discussed with the Management Committee, and then presented to USAID. While preparing the workplan, the project actively consults with USAID, other partners, and the home offices of the AMKENI partners. The Project prepares and submits detailed reports to USAID on a quarterly basis. The quarterly reports include an overview of project implementation and discusses achievements for the report period, implementation challenges and issues, and planned activities for the next quarter. The reports also present monitoring and evaluation status, expenditure statements and pipeline analyses. Financial Management The Project provides quarterly financial reports to USAID. The Nairobi Office serves as the focus for all projects financial operations. Nairobi Office receives financial reports from the provinces and consolidates the information for reports to USAID. The partners send quarterly financial reports to the Nairobi Office, where they are reviewed and approved, and then the partner expenses are reimbursed. All accounting, and disbursement functions are done at the Nairobi Office. Subagreements with expenditures over $300,000 require audits performed by independent auditors. Independent auditors also perform an audit at the end of each fiscal year confirming adequacy of systems implemented. A final audit is planned for next year to review expenditures for the entire project to confirm the end-of the project status. Conclusions and Lessons Learned In general terms, the AMKENI partnership formed among four international agencies works well. The partners have mutually respectful relationship, developed around their respective technical expertise and their strengths. The partners bring in complementary expertise and strengths, enabling the project to benefit from synergies, and helps to avoid duplications. Identifying a lead agency for each of the technical components has clarified the partners’ key roles. All agencies have seconded staff to AMKENI, thus allowing the project to work as a unified team. 54 The team had ample observations on how the project staff works together, as well as with local implementing partners and the communities. Thus the team concludes that AMKENI has a highly qualified, committed and hard-working staff at all levels. The reporting and financial systems put in place are sophisticated, and satisfies USAID’s reporting needs. The team has concerns, however, about the management style of the project, which is rather top-down and centralized. This style is reflected in the flow of authority between the Nairobi and the Area Offices. The Area Managers only have administrative oversight for the staff in the area offices. The technical staff reports directly to the respective technical advisors in Nairobi. This structure does not allow decentralization of technical and administrative responsibilities and authority to the field offices, where the actual project implementation is taking place. Finally, the team has observed some duplication in staffing. AMKENI made a decision a few years ago to de-link the HIV/AIDS Advisor and the two HIV/AIDS Coordinators from service delivery staff. The team does not think that this was a justified decision. Placing HIV/AIDS advisor and coordinators along with the Service Delivery Advisor and Coordinators creates parallel systems within the project and does not help better integration of HIV/AIDS activities with the other elements of the Project. Sharing the time of HIV/AIDS Coordinators with the IMPACT Project is also not rational. IMPACT Project has a much broader mandate in HIV/AIDS, thus expecting the coordinators to split their time equally between the two projects does not seem realistic. MONITORING AND EVALUATION PLAN The Project had formulated and submitted an M&E Plan as part of its initial workplan in 2001. As project implementation got underway, the staff decided that the initial plan is not meeting the Project’s and USAID’s needs and a decision was made to revise the plan. Prior to revising the M&E Plan, the Project developed a results framework and building upon this developed a causal model. The revised M&E Plan included 195 indicators and their definitions as well as data sources and frequency of data collection for each indicator. The plan also had end-of the project target for each indicator. In 2003, the EMR recommended to streamline and repackage the M&E plan. Thus the Project revised its M&E plan for a second time, and the process resulted in this final M&E Plan, which the evaluation team reviewed. The final plan reduced the number of indicators to 114, while the results framework and the causal model were not modified. Figures 13 and 14 show the project results framework and the causal model. The indicators were selected with the intention to serve project monitoring and evaluation in terms of activities supported (process indicators), the products of these indicators (output indicators), the consequences of these products (outcome indicators) and how these consequences are reflected in improving health (impact indicators). Impact indicators such as total fertility rate and HIV prevalence are not included in the plan, although they pertain to AMKENI’s strategic objective. These impact indicators are subject to many other factors beyond those controlled by AMKENI. Majority of the indicators selected are process indicators, followed by output indicators. 55 The revised M&E indicators include 21 of the USAID indicators, contributing to five sub I.Rs of the USAID/Kenya results framework. These sub-IRs are: IR1.3.2: Training and supervision for RH improved (2 indicators) IR2.2.3: Effective community outreach programs carried out (1 indicator) IR2.3.3: Enhanced provision of HIV/AIDS prevention (10 indicators) IR3.1 Integrated FP/RH/CS services expanded (2 indicators) IR3.2: Improved knowledge and demand for FP/RH/CS services (3 indicators) IR3.2.1: Effective community outreach and interpersonal communications programs conducted (3 indicators) AMKENI uses a computerized database system containing data from the provinces, using the ACCESS software. This system, called AmoS, combines service utilization data, trainee data equipment data and community interventions. Before AMoS was created, data was entered on multiple excel worksheets, which led to data-entry errors. These problems are now minimized with one complete data set and standard reports and graphs are easily compiled. With improved management of information the Project has been able to produce newsletters, a brochure, a summary sheet and maps showing the location of the intervention facilities. Such presentations of AMKENI’s efforts and results have helped the Project share its experiences more broadly. The data entry and editing for AMoS is done in Nairobi, not at the Area Offices. The Area Offices collect data from the facilities and the local implementers and send them to the Nairobi office in paper form. This decision was made because of the high workload of the field offices, although it is recognized that this is not the ideal situation. Editing and correcting the data at the Nairobi Office is cumbersome and takes a lot of efforts and time. The data is computerized in a week and sent to the Area Offices for corrections, and also that the Area Offices can produce their own tables and reports. Final data is usually compiled in six to eight weeks and is incorporated into the Quarterly Management Report. Then the quarterly reports are distributed to USAID, the MOH, consortium partners and the Area Offices. The team observed that data collection from the facilities follows different procedures in the Coast and Western Provinces. In the Coast, there is a functional MOH reporting system developed through the assistance of the AKHS. AKHS has helped the MOH to develop a simple data collection tool, which is used throughout the province; however, some data needed by AMKENI are not included in these forms (e.g., PAC attendance). Thus AMKENI Area Office in Mombassa completes AMKENI forms and sends them to the District MOH office when the AKHS forms are sent. Data collection in the Western Province is more complicated since there is no such system in the province. The Area Office requires the facilities to report them every month, but when the reports are late, the staff needs to visit the individual facilities to collect data. Conclusions and Lessons Learned The team fully recognizes and appreciates the efforts that have gone into the monitoring and evaluation component of the project. In fact, the team has received much detailed and well- 56 organized information about each project facility, as well as the overall inputs and the outputs of the Project. The problem with AMKENI’s M&E system is that it is too comprehensive and complicated. The issue begins with the AMKENI Results Framework; which is exhaustive and complex. For example, the framework has 17 lower-level IRs, which leads to a large number of indicators. Although the number of indicators has been reduced from 194 to 114, this number is still excessive. It is not possible to monitor project progress on a daily basis using all the indicators, which is stated as the intention in the revised monitoring and evaluation plan. The team has observed that only a small portion of these indicators is used at the central level for overall project monitoring. However, we observed little evidence that the M&E is used as a management and performance improvement tool at the field level. The evaluation team attempted to comprehend why the M&E plan got so complicated and complex. One explanation is that the project started off “thinking too big” and did not receive adequate management and technical assistance to develop a functional M&E plan. Once the M&E had started so complex, the Project was not able to simplify its M&E plan. The team observed that the technical teams want to include all of their activities reflected in the M&E so that their contribution to the project achievements would be recognized, which is not a purpose of an M&E plan. The team also witnessed that the staff viewed the M&E plan as an USAID requirement. While the M&E plan is a requirement by USAID, many AMKENI staff fail to understand that the primary purpose of an M&E plan is to use it as a management tool to track and improve performance within the project. 57 Figure 13 AMKENI Results Framework IR-2. Healthierbehavior at individual, family and community levels IR-1. Increased utilization of quality, integrated FP/RH/CS services IR 1.2.2 Increased “agency” taking action to preserve health IR 1.1 Increased Quality and Accessibility of integrated FP/RH/CS service delivery IR 1.2 Increased demand for high quality FP/RH/CS services Strategic Objective: To improve the reproductive health of the people of Kenya IR 1.1.1 Improved clinical skills among public and private providers IR 1.1.7 Improved systems for training and supervision 1.1.6 Improved FP/RH/CS policies and standards IR 1.1.2 Improved resources management in service delivery facilities IR 1.1.5 Enhanced community input into facility governance IR 1.1.4 Continuous quality improvement activity at service facilities IR 1.1.3 Improved service delivery environment at health care facilities IR 1.2.1Increased knowledge of how to preserve health IR 1.2.1.1 Increased understanding of FP/RH/CS IR 1.2.1.2 Increased awareness of the quality and availability of FP/RH/CS services IR 1.2.1.3 Increased understanding of ways in which individuals and communities can influence the delivery of quality FP/RH/CS services IR 1.2.2.1 Increased ability to analyze FP/RH/CS problems and determine solutions IR 1.2.2.2 Increased ability to discuss and negotiate health behaviors IR 1.2.2.3 Enhanced community and provider support for health action by individuals/communities IR 1.2.2.4 Increased advocacy for FP/RH/CS by individuals and communities IR 1.2.2.5 Increased individual/community participation in influencing FP/RH/CS service delivery 58 Figure 14 AMKENI Causal Model TECHNICAL OUTPUT SUB IR s IR s SO SD,HIV,T&SS………… …...….………………… ……… …. SD,…………..………… …...….………………… …………… SD,HIV,………….…… ……………...….……… ……………………. SD, HIV,BCC… ……... SD,T&SS……………… ……………………….… ….....…... BCC………….…………………… ……... BCC,HIV,SD…………………….. ……. BCC,HIV,………………………… ………. T&SS,HIV,…………… …... T&SS,…………… …… …... BCC,SD……………………… ………….……………...…... BCC…………………………… ………………………...…... BCC…………………………… …………...……………...…... CLINICAL SKILLS MANAGEMENT ENVIRONMENT QUALITY IMPROVEMENT QI CLIENT INPUT POLICIES SYSTEMS ISSUES K/SERVIC ES K/AGENCY ANALITICAL ABILITY NEGOTIATION PROVIDER SUPPORT ADVOCACY GOVERNANCE IR1.1 QUALITY & ACCESS IR1.2 DEMAND IR 1.2.1 KNOWLEDGE IR1.2.2 AGENCY IR1 UTILIZATION IR2 BEHAVIOR SO HEALTH BCC…………………… ………………….…… ………...…... BCC,SD……………… …………………..…… 59 FUNDING SOURCES AND ALLOCATION OF FUNDS To date, USAID has obligated AMKENI a total of $18,913,090, using funding from several sources. The bulk of funding is obligated from population funds, comprising 72% of the total budget, followed by AIDS funding at 12%. Other funding sources make up a small portion of the total budget. Child survival funding is 2% of the total obligation and the project received one-time obligation from malaria funds, comprising 3% of the budget. PEPFAR funds became available in 2003, through which the project received $657,400. (See Table 13) Table 13 Breakdown of USAID Obligations by Source Funding Source 2001 2002 2003 2004 Total % Population 3,057,000 4,228,690 2,647,000 3,589,000 13,521,690 72 AIDS 22,000 1,848,000 283,000 2,153,000 12 Child survival 271,000 50,000 100,000 421,000 2 OVC 100,000 100,000 1 Other health 50,000 50,000 - Malaria 650,000 650,000 3 PMCT 960,000 400,000 1,360,000 7 PEPFAR 657,400 657,400 4 Total 3,400,000 6,826,690 3,940,000 4,746,400 18,913,090 100 Table 14 presents utilization of project funds by AMKENI. The financial system of the Project does not allow a breakdown of expenses by funding categories since funding sources are not known when workplans are made and sometimes even when expenses have occurred. It is not a requirement by USAID to track funds in this way and AMKENI does not do this tracking. Nevertheless, the evaluation team attempted to present the breakdown of utilization of funds in terms of program elements. Largest line item is the management category, which used 37% of the total budget, followed by BCC activities at 19%. M&E costs comprise 4% of the total budget. Service delivery and training expenses amount to 10% and 15% of the budget, respectively. Management line item includes the overhead, office set up and running expenses, purchase, running and maintenance of vehicles, salaries and benefits for the management and support staff. Service delivery expenses consist of costs of clinical skills training, supervision, training materials and salaries of technical staff. BCC line item covers the costs of subagreements with the local implementing partners. Table 14 Utilization of Funds by AMKENI Total % Management 6,947,899 37 M&E (CPC and IntraHealth) 814,250 4 Service Delivery 1,969,908 10 60 Equipment & Renovations 1,289,292 7 BCC (PATH) 3,614,291 19 Training (IntraHealth) 2,837,762 15 HIV/AIDS (FHI) 1,439,688 8 Total 18,913,090 100 The following table illustrates the utilization of project funds by project elements and activities. Table 15 Utilization of Funds by Project Elements and Activities Funding source Utilization by project elements and activities Population Family planning, particularly long-term and permanent methods, RH counseling, quality improvement, IP, facilitative supervision, equipment and renovations, PAC, development of training materials HIV/AIDS Establishment of VCT and PMTCT centers, training of service providers, training in IP, provision of water, setting up incinerators, equipment, and integration of FP to VCT Child survival Training and provision of equipment for newborn care, immunization, IMCI training OVC Youth activities to promote behavior change Malaria Prevention and treatment of malaria in pregnancy, promotion of ITN use PMCT Community activities, training, renovations and furniture PEPFAR Training fro PMTCT and VCT, prevention of opportunistic infections, development of training materials, renovations for clean water Under its agreement with USAID, AMKENI is responsible for providing approximately $800,000 in cost share funds. Over the course of its four years to date, several partners and stakeholders have helped share the cost of implementing the Project. The major partners have been the MOH and community members, who contributed in kind. The MOH provided facilitators for the training workshops at no cost to the project and its contribution is estimated about $152,952. The community volunteers (e.g., animators and youth group members) gave their time and effort at no cost to the Project; with AMKENI only meeting transport and refreshment costs incurred in conducting project activities. AMKENI estimates that the value of contributions of community volunteers amounts approximately to $379,792. 61 In addition to in-kind contributions, AMKENI raised funds from non-USAID sources to support its activities. (See Table 16) Table 16 Summary of Cost Share Obligations Contributions in kind Contributors and Activities USD MOH-Training facilitation 152,952 Community volunteer services 379,792 Contributions in cash from non-USAID sources Tides Foundation-NSV 100,000 SIDA-PAC 29,456 Marie Stopes-IP - 3,602 UNICEF-Cope for Child 25,168 CLUSA-Women Agency 20,000 Gates Foundation-Child-to-child 65,000 Hewlett Foundation-Youth pharmacy 65,069 Collins Bullett Foundation- Staff time 15,000 UNC-Office rental 6,662 Total 862,701 RELATIONS AND COORDINATION WITH USAID, MOH AND OTHER STAKEHOLDERS AMKENI has been able to establish excellent working relationships with the MOH since the EMR in 2003. A major concern raised by the EMR was the inadequate coordination with the MOH in project implementation. The Project succeeded to significantly improve its relation with the MOH over the last few years. The evaluation team was impressed to observe that MOH staff is closely involved in project implementation at all levels. The team also observed a high level of ownership of the project by the MOH, which is applaudable. The Head of Reproductive Health Division has recently became a member of the Project Management Committee. The team believes that this was another positive step taken by the Project to strengthen its collaboration with the MOH. AMKENI maintains a satisfactory relationship with USAID/Kenya. USAID/Kenya is content with the Project’s responsiveness and the team observed that the USAID/CTO is closely involved with project implementation. The CTO maintains daily contact with the Project, providing feedback and direction. The team was also impressed to observe a high level of ownership of the Project by communities throughout the field trips. AMKENI coordinates closely with the IMPACT Project, USAID’s flagship project for the prevention and treatment of HIV/AIDS. The areas rightly selected for AMKENI’s support are PMTCT and VCT, while IMPACT implements a much broader range of activities. 62 Population Council, with funding from DFID implements a Safe Motherhood Project in the Western Province. The mandates of the Safe Motherhood Project and AMKENI are almost similar, except that the Safe Motherhood Project does not have a community component and unlike AMKENI, the Project works province–wide, covering all districts. The two projects coordinate at central level through the RH Inter-agency Coordinating Committee (RH-ICC) where both AMKENI and Population Council are members. Their activities are also coordinated at the provincial and district levels and the team did not observe any duplications of effort. UNFPA supports the implementation of a comprehensive FP/RH program through the MOH in Kenya. The project works in 9 selected districts in the Coast, Eastern, Central and Rift Valley Provinces. The only districts AMKENI and UNFPA overlap are Kilifi and Kwale. The efforts of UNFPA and AMKENI are coordinated through the district stakeholders’ forums, and there were no duplications. 63 VI. SHORT-TERM RECOMMENDATIONS FOR AMKENI The recommendations outlined under this section are for AMKENI to consider over the remaining life of the Project. The team believes that these specific recommendations are realistic and appropriate for AMKENI to undertake over the next year to consolidate gains of the project and achieve a smooth close-up of the ongoing interventions. ß The basic thrust of the AMKENI, working with both the facilities and the communities to create linkages and to improve service quality while increasing demand for services is a powerful approach. This approach should be maintained over the last year of the project. ß The current three-pronged strategy implemented by AMKENI is appropriate and should be continued with some modifications. The project should streamline its interventions outlined under the Strategic Objective 2 over the next year by focusing on 1) promotion of healthier behaviors in FP/RH/HIV/AIDS, and; 2) increasing demand for FP/RH/HIV/AIDS services. ß So far, AMKENI has completed decentralized T&S system pilot interventions for 16 facility–based teams. The project should speed up and accomplish the implementation of the decentralized T&S system in the remaining 81 facilities. ß AMKENI should undertake a rapid assessment and document the outcomes of its training activities by the end of the Project. ß By the end of the project, AMKENI should ensure that all PMTCT and VCT providers in Project facilities have been trained in basic family planning counseling. ß AMKENI should work with the newly awarded CAPACITY Project to coordinate pre-service training activities. AMKENI should prepare to transfer technical assistance in pre-service training to the CAPACITY project by the time of closure. ß USAID and AMKENI should consider assisting MOH with vehicles to transport supervisors by providing a vehicle to each of the AMKENI current provinces ß AMKENI should ensure that family planning services are fully integrated with other maternal and child health care services, PMTCT, and VCT in all target facilities by the end of the Project. This includes a careful assessment of how services are organized in each facility and provision of technical assistance to the facility management and staff as needed. An analysis of patient flow in facilities might provide useful information to increase access to family planning services. ß Although there is evidence that family planning method mix has improved in project areas, additional efforts are needed to further improve method mix and to ensure continuity of family planning use and compliance. 64 ß AMKENI should work closely with the MOH, and the DELIVER Project to ensure that family planning commodities are available at the facility level. AMKENI should encourage facilities to report to the Districts on utilization of contraceptives so that facilities receive contraceptives before stock-outs occur. AMKENI should also work with the DELIVER Project to follow-up on District staff trained in contraceptive logistics by the DELIVER Project. ß AMKENI should continue working with AKHS to provide input and feedback to the report AKHS has been preparing on the impact of abolishment of cost-sharing policy on service availability, quality and utilization. ß AMKENI’s BCC component should pay special attention on men and male youth as the locus of power. This is particularly critical for Coast Province where although women use contraceptives, this is to a large extent done secretly. ß AMKENI should place higher emphasis on FP/RH BCC messages. ß Given the role played by the PHOs PHTs, it is prudent that they are facilitated much more by the Project in terms of transport. This is a small but valuable sacrifice to make alongside the heavy equipment supplied by the Project. The team recommends that AMKENI provide bicycles for the PHOs and PHTs. ß AMKENI should analyze the Household Survey data and disaggregate it by province to facilitate comparison with the end of project targets, some of which were province specific. The analysis should also determine the most successful and cost effective BCC approach. ß The team recommends AMKENI to consolidate project interventions in current 10 districts within the Coast and Western Provinces over the next year. It is not realistic for the Project to extend to other districts or provinces over a one-year timeline. ß AMKENI should undertake an assessment within the current districts to analyze its exact coverage of the populations. This analysis can be done by mapping out of the catchment areas of the current AMKENI-assisted facilities. Based on this analysis, AMKENI should decide whether it is feasible to assist additional facilities within the current districts. AMKENI should consider its remaining resources and the timeline when deciding to expand to additional facilities. ß If AMKENI decides to assist additional facilities, it should develop criteria for selecting those sites. The Project should consider 1) the catchment areas of the existing facilities and additional eligible facilities; 2) needs of each facility in terms of training, facility upgrades, equipment and linkages with the communities; and, 3) willingness of the facility management to improve quality and service utilization. The focus of the expansion should continue to be on lower level facilities, particularly the most needy dispensaries. 65 ß AMKENI should delegate both administrative and technical oversight responsibilities to the Area Managers for field implementation. In this transition, technical advisors should work with and provide technical support to the Area Managers. ß AMKENI should review its M&E Plan and retain those indicators only essential for tracking performance and progress of the Project. This task should be delegated to the M&E Advisor, with support from the Project Director. While revising the indicators, the Project should maintain those 21 indicators, which contribute to the M&E Plan of USAID/Kenya. Given the short period of time left for project implementation, the team does not recommend AMKENI to revise its Results Framework. ß Once a short list of indicators is developed, AMKENI should require the Area Offices to electronically enter the data into the database. The M&E Advisor should work with the Area Offices to train them on how to use the database. ß AMKENI should revise the definition it has been using for family planning acceptance and adopt the international definition. Data collected so far needs to be readjusted accordingly. 66 V. LONG-TERM RECOMMENDATIONS FOR USAID/KENYA FOR FUTURE PROGRAMMING The recommendations provided in this section are for USAID/Kenya for future programming purposes, specifically developed for the new five-year follow-on project. The evaluation team has based these recommendations on lessons learned from AMKENI, its achievements and challenges, as well as observations on the broader Kenyan health context. Since USAID/Kenya is planning for a new competitive procurement within a year, these recommendations should be considered procurement sensitive. ß The team recommends that the follow-on project should focus on selected reproductive health issues with special emphasis on family planning, VCT and PMTCT. Other focus areas may include PAC, ANC, and IP. Maternal health activities should be minimized, with selected and targeted interventions. ß The team recommends that child survival interventions should not be part of the follow-on project. The needs for improving child health are great and multifaceted, and should be the mandate of other large-scale projects. ß The team recommends that USAID/Kenya should coordinate closely with the new DFID-funded Safe Motherhood Project while developing the follow-on procurement instrument. At the time of this evaluation, the Safe Motherhood Project was about to be awarded. ß The team also recommends that USAID/Kenya should continue collaborating with the donor community to ensure contraceptive security, since contraceptive shortages are severely impeding service availability and quality. Coordination efforts among donors should focus not only on availability of contraceptives at the central level but also at the district and facility levels. ß USAID/Kenya may consider initiating policy level talks with the MOH to reverse the cost-sharing policy and/or explore alternative means of financing health care at lower level facilities. As discussed in detail throughout this report, the abolishment of cost￾sharing policy has had a significant negative impact on service availability, quality and utilization. AMKENI and AKHS would provide additional information on the impact of the change of MOH’s policy to assist USAID/Kenya’s advocacy. ß USAID/Kenya may consider fostering synergies between the follow-on Project and the CAPACITY Project by directing emergency staffing assistance foreseen under the CAPACITY Project to understaffed dispensaries within the new provinces. ß The follow-on Project should maintain AMKENI’s approach to linking facilities and communities. This has been a unique approach pursued by AMKENI and the team has documented ample evidence that the approach has been highly successful. ß The team also recommends that the follow-on Project should maintain AMKENI’s overall three-pronged strategy incorporating T&S, service delivery and BCC 67 components. The following specific recommendations provide directions to modify and refine AMKENI’s strategic approaches. ß The follow-on Project should not be involved in pre-service training. Pre-service training interventions currently implemented by AMKENI should be transferred to the CAPACITY Project, which has a crosscutting mandate in human resource capacity building. The transfer should be cautious and gradual, however, in order to avoid interruption of ongoing pre-service activities of AMKENI. ß All components of the follow-on Project should focus on integrating FP/RH and HIV/AIDS interventions at the field level. ß The follow-on Project should use lessons learned from AMKENI to simplify and streamline BCC activities by implementing the most effective strategy that reflects a hybrid of the most successful elements of the piloted BCC models. The best scenario is where there is a community participatory model as the umbrella that engages the whole community and different audiences. This would further be integrated with community (VHCs) and MOH (PHO/PHT) structures to ensure effectiveness and future sustainability. The revised model would then be implemented uniformly in the project sites. For ease of implementation, coordination and monitoring progress, it would be more feasible to work with 1-2 local implementing partners. ß The team recommends that the BCC interventions of the follow-on Project should focus on FP/RH themes and on men and youth. · The team recommends that AMKENI should complete and phase-out its activities in Coast and Western Provinces by the end of the Project in March 2006. The follow-on Project should select and move on to working in two new provinces. Subject to discussions with USAID, MOH and other stakeholders, the team recommends considering the Eastern and Nyanza Provinces for the next phase. ß The follow-on Project should focus on provincial, district and facility level activities and should not include national level interventions. Similarly, the next phase should focus on improving public sector services, specifically at the lower level facilities, where high-level demand is obvious. ß In line with the lessons learned from AMKENI, the follow-on Project should have a decentralized management structure. The overall authority and responsibility of overseeing project implementation should be delegated to the Project Director. Consequently, the Project Director should delegate the administrative and technical oversight of the field implementation to the Area Managers. The role of the senior technical staff should be providing technical assistance and support. ß One of the lessons learned from AMKENI is that the creation of separate HIV/AID staffing has not helped integration of HIV/AIDS and other services at the field level. Similarly, separation of senior staff overseeing policy and systems in training and staff responsible for training activities has had little value. The follow-on Project 68 should be required to consolidate technical staffing under teams in line with the project’s main technical components; training, service delivery and BCC. ß The follow-on Project should continue AMKENI’s effective coordination with the IMPACT Project and the current division of labor. With that said, the team does not recommend sharing of staff between the two projects. Sharing of field staff between AMKENI and IMPACT may have had merits, however, the team is concerned that fifty-fifty sharing of staff time between the two projects is not practicable. IMPACT has a much larger mandate in HIV/AIDS compared to AMKENI and it is not realistic to expect staff to allocate equal time and efforts to both projects. 69 ANNEXES A. SCOPE OF WORK External Evaluation of USAID/Kenya’s RH/FP/CS Portfolio 1. INTRODUCTION USAID/Kenya’s health sector strategic objective (SO3) is to reduce fertility and the risk of HIV/AIDS transmission through sustainable, integrated family planning and health services. It directly supports the Government of Kenya’s efforts towards reducing unintended and mistimed pregnancies, improving infant and child health, reducing HIV/AIDS transmission, and reducing the threat of infectious diseases. In 2000 the mission designed a reproductive health strategy in line with its Intermediate Result 3: Increased use of family planning, other reproductive health services, and child health services. Under this approach emphasis was placed on improved quality and demand for reproductive health services. Thereafter the Cooperative Agreement with EngenderHealth was developed to implement this strategy. BASIC PROJECT INFORMATION Project Name and Number AMKENI CA#623-A-00-00019-00 Initial Total Estimated Amount $ 15,997,129 Increased Total estimated cost $ 18,913,090 (expanded SOW to include PEPFAR) Obligations to date $ 14,166,690 2. BACKGROUND The EngenderHealth Cooperative Agreement is a five year initiative to promote improved reproductive health and child health in Kenya with an emphasis on increasing utilization of facility based integrated family planning/ reproductive health/ child survival/ HIV (FP/RH/CS/HIV) services. The project, called AMKENI is implemented by a partnership agreement of development organizations led by EngenderHealth. They include Family Health International (FHI), the Program for Appropriate Technology in Health (PATH), International Training for Health (INTRAH), and the Carolina Population Center (CPC). Local partners include the Aga Khan Health Services (AKHS), Community Aid International (CAI), and the Cooperative League of the U.S.A (CLUSA), Family Planning Association of Kenya (FPAK), the Kenya Music Festival Foundation (KMFF), the UZIMA Foundation and World Relief. AMKENI started in December 2000 and will run through September 2005. The project works with local NGOs, faith based organizations, private sector partners, provincial and district authorities and teams, and with various departments within the Ministry of Health. 70 AMKENI seeks to achieve a sustainable increase in the quality of, the demand for and the utilization of integrated FP/RH/CS/HIV/AIDS services at the community level. This goal is being achieved through two complimentary approaches: · working directly with the central level of the Ministry of Health, specifically the Division of Reproductive Health, to strengthen and reform its systems for managing FP/RH/CS training and supervision, and · working in ten target districts in Coast and Western Provinces, in partnership with the Ministry of Health and local partners, to improve the quality of FP/RH/CS/HIV/AIDS services in ninety-six target facilities, and to increase the community’s demands for these services. The project activities aimed at accomplishing this goal are grouped into four broad categories: service delivery, HIV/AIDS, training and supervision, and Behavior Change Communication (BCC) activities. 3. EXTERNAL EVALUATION SCOPE OF WORK i. Objectives The evaluation team will examine activities in USAID/Kenya’s increased use of family planning, integrated reproductive health, and child survival services portfolio to assess the following: · the progress of the AMKENI project towards the achievement of specific project objectives and USAID/Kenya’s Intermediate Result 3; · the impact on quality, demand and utilization of services; · the impact on national training and supervision systems; · the communities (in project areas) perspective of the impact of the project; · the pace of project implementation given the geographical focus and expansions of scope, and any general changes in area of emphasis and changes in specific activities necessary to successfully carry out project objectives; · the appropriateness of the strategies towards achievement of the project objectives; · the impact of shifting funding mechanisms i.e. PMCT and later PEPFAR; · the current and potential role of this portfolio of activities to developing sustainable community demand and utilization of services; · overall challenges and lessons learned; and · Key areas of future focus for the portfolio, the value, cost and potential for expansion into longer-term, more comprehensive activities and the actions necessary to realize this. · The degree to which the project has enabled communities and individuals identify health problems, seek solutions and move towards more health behavior including seeking services; · The degree to which systems and processes are established that lead to sustained behavior change and increased demand for services. 71 ii. Participants, critical skills, and audience for the evaluation Team Members: The review team will consist of the following persons with specific expertise and experience: · Team Leader: with senior program management experience particularly with regard to health sector reform, FP/RH/CS and HIV service delivery, financial systems and M&E. Should have excellent writing skills. · Two Consultants: experience in M&E, service delivery, and training with regard to FP/RH/HIV/AIDS/CS. · Two Consultants: experience in behavior change communication (BCC). · Ministry of Health Representatives: including Division of Reproductive Health, NASCOP, Coast and Western Province PMO and DHMT. · USAID and project staff Prior to commencing the evaluation the team will meet with USAID, MOH and AMKENI staff and management team to clarify the scope of the evaluation and to finalize the schedule of activities. · USAID/Kenya (OPH, controllers office, PDA) · MOH · AMKENI partners iii. Key Informants · Project representatives · MOH personnel · Other Cooperating Agencies · USAID staff · Other donors, as appropriate · Intended beneficiaries iv. Issues to be addressed by the team Topical Issues The evaluation will include, but not be limited to, an assessment of the impact of this project on quality, utilization and demand for FP/RH/HIV/CS services in Coast and Western Province Service Delivery – quality, access, utilization, integration Training and Supervision – Impact on National Policy and systems for training and supervision; training linked to service delivery; performance improvement approach BCC – impact on demand and utilization of services. 72 Change in knowledge and attitudes; community ownership and capacity building; establishment of systems and processes for increasing agency; steps community has taken to address their health problems HIV Services – quality, access, utilization, linkages, and community support Monitoring and Evaluation Systems – efficiency, and evolution during the life of the project Project Management – Administrative and financial management of the Nairobi, Western and Coast offices. Relationships with MOH and other vital partners Process Issues The team will assess issues related to the process of implementation of project activities Achievement of goals and objectives The evaluation team will assess and comment on the extent to which the project has: · met stated goals and objectives; · involved the MOH · addressed the integrated district workplans formulated by the DHMT · developed sustainable systems with the MOH and beneficiary communities · developed and implemented appropriate monitoring and evaluation tools to track progress towards stated goals and objectives; · developed capacity of partner organizations; · acted upon/implemented recommendations from the mid term evaluation; and · responded to changed environment for FP/RH/HIV/AIDS The evaluation team will also make recommendations concerning appropriate strategies for scale up of integrated RH/FP/HIV/CS activities. Reporting: While the primary objective is to assess programmatic effectiveness, the team will also assess the degree to which the project has complied with USAID reporting requirements. 4. Specific activities Activities to be conducted by the team will include, but not be limited to: · Initial conference with USAID/Kenya staff, AMKENI partners and MOH representatives during which the team will present a proposed schedule for undertaking the evaluation for input and finalization; 73 · Review of relevant documentation (including project studies, status reports, evaluations, USAID strategy documents, and other reports related to FP/RH/CS/HIV in Kenya). · Meetings with project partners and staff ; · Site visits and meetings with MOH officials and other stakeholders including project staff, BCC implementing partners, service providers, PMO, DHMT, field agents; · Structured interviews conducted with a representative sample of community members in Coast and Western Province to assess their satisfaction with the project in question and to secure recommendations for future program direction/project improvement; and · Dissemination meeting during which the team will present the first draft of the evaluation report, highlighting key findings and recommendations and receive clarification/input from stakeholders. 5. Report and recommendations for future program direction The team leader will submit a draft report to USAID prior to the post-evaluation debriefing. The final report will be submitted no later than 14 days from the date of the final debrief to USAID. The document will be organized as follows: I. Executive Summary II. Background III. Summary of key findings and conclusions IV. Recommendations for future directions V. Annexes a. Scope of Work b. Evaluation Team Members c. List of Interviewees d. Methodology e. Questionnaire tools The report will include the following information: · Review methodology · Project accomplishments to date: Brief description of progress on each component; notable successes/failures/challenges; attention to cross cutting program issues such as demand creation and increasing access to services; increase in quality, demand and utilization of services; Monitoring and Evaluation (M&E) activities. · Project organization and management: prime/sub contractor relationships; strengths/weaknesses; staff skill and mix; communication/relationship with partners, MOH; timeliness and quality of reporting. · Financial analysis: (Total Budget) obligations to date and cost of each program component · Issues and recommendations: problems/challenges their causes recommended actions; potential problems/challenges; suggested cooperative agreement amendments; future direction. · Upon acceptance by USAID the Team Leader will submit one report in four bound copies and an electronic copy on a 3.5” diskette in MS word. 74 6. References The management team will review all project documentation, including but not limited to the following: a) USAID/Kenya APHIA project document. b) USAID RFA prior to the AMKENI project. c) USAID PEPFAR strategy for Kenya d) Proposal by EngenderHealth led consortium. e) AMKENI cooperative agreement and subsequent modifications. f) AMKENI PEPFAR proposals (PMCT, VCT and SI and non ART HIV services) g) Sub-agreements between EngenderHealth and its core partners and local partners. h) All MOU’s between AMKENI and other CAs, other projects, target facilities. i) The project’s initial 18-month work plan; Year two 12-month work plan; Year three 12- month work plan; Workplan and budget for Year 3 and 4; M&E plan; Behavior Change and Communication (BCC) Strategy; PMCT proposal; 2005 PEPFAR proposal. j) AMKENI quarterly reports to USAID through December 2004. k) Mid term evaluation report l) Preliminary reports from Household survey; documentation of the decentralized supervision system; documentation of the AMKENI model linking “demand” with quality services and healthier behavior. These materials will be made available to the review team members two weeks prior to the start of the review in hard copies and electronically. 7. Proposed schedule It is anticipated that the evaluation can be completed in nine weeks. The evaluation will begin on January 24th 2005 and end no later than March 11th 2004 in terms of meetings, document reviews, interviews, fieldwork, draft report writing, and debriefings with USAID and dissemination to stakeholders. The Team Leader will provide a final report to USAID no later than Monday March 28th 2005. A six-day workweek will be authorized for this management review. The Team Leader will coordinate preparation of the final schedule of meetings and field trips. The Team leader will ensure that the schedule, as agreed with USAID and AMKENI is adhered to. The evaluation will take place over nine weeks beginning January 24th 2005, and will include: · A desk review of materials 6 days · Team briefing and orientation 1 day · Site visits (Coast and Western Province) and interviews 15 days · Prepare draft report 8 days · Dissemination to USAID, MOH and stakeholders 6 days · Completion of written report 18 days · Presentation to USAID in March 28th 2005 75 8. Logistics and Support All funding and logistical support for the evaluation team namely the Team Leader, two Service Delivery Consultants and two BCC Consultants, will be provided through POPTECH from the USAID/K Field Support. Activities that will be included in the budget include recruitment of the Team Leader and Consultants, their travel into and out of the country, payment of the Team Leader and four Consultants for a six-day work week, support for all expenses related to the management review and logistical support. This logistical support includes per diem, communication costs, supplies and/or fixed other direct costs (except local travel). POPTECH will meet the costs for stationery, computers for local consultants, and supplies. POPTECH will also meet the costs for publication and dissemination of the draft and final reports. The Mission will meet the expenses for the Ministry of Health Representatives, the costs for the local travel of the evaluation team, which includes air flights, and ground transportation. The mission will organize and meet the expenses related to the stakeholders meeting. The mission will provide logistical support in terms of office space and meeting facilities for the team. 9. Relationships The team leader will report directly to the Senior RH/FP advisor in Office of Population and Health USAID/Kenya and work closely with the CTO and other relevant USAID officers. 10. Completion of deliverables The contract will be considered successfully completed when the Team Leader submits to USAID/Kenya the final report document. 76 B. LIST OF CONTACTS USAID/Kenya Sheila Macharia, Reproductive Health Specialist, Office of Population and Health (OPH) Michael Strong, Senior Program Manager, OPH Janet Paz Castillo Chief, OPH Crispus Kamanga, Project Management Specialist (M&E) OPH Bedan Gichanga, Health Systems Specialist, OPH Joseph Ondigi, Senior Finance Specialist, Controllers Office Stephen Ragama, Program Development and Analysis MOH Nairobi James Nyikal, Director of Medical Services Ambrose Misore, Head of Department of Preventive and Promotive Health Services Josephine Kibaru, Head of Division of Reproductive Health Anne Wamae, Head of Division of Child Health Coast Province Anderson Kahindi, Provincial Medical Officer Ben Kinara, Provincial Health Administration Officer Alice Menza, Provincial Nursing Officer David Wanjalla, District Medical Officer, Mombasa Alice Kadilo, District Public Health Nurse, Mombasa Mwakangalu Dickson, Medical Superintendent, Kilifi DMOH Julius Jillo, District Health Education Officer, Kilifi Anisa Omar, District Medical Officer, Malindi Esther Mwema, Hospital Matron, Malindi DMOH Edward Mumbo, District Public Health Nurse, Kwale David Baya, District Health Education Officer, Kwale Western Province Janet Wasiche, Provincial Obstetric and Gynecologist Judith Sitti, Deputy Provincial Nursing Officer Zablon Kebeya, Provincial Public Health Officer Asconi Oca, Provincial AIDS and STDs Coordinator Dr. Shikanga, DMOH, Kakamega Monica Ongowo, DNO, Kakamega Carolyn Obiero, DPHN, Kakamega Dr. Wasike, DMOH, Bungoma Innocent Sifuna, PHO, Bungoma Ruth Odongo, DPHN, Bungoma Alice Selete, DPHN, Busia 77 Simon Danda, DHEO, Busia Mirasi Tom, DHNO, Busia J. Mukabi, DMOH, Busia AMKENI Nairobi Team Job Obwaka, Project Director Chris Oyoo, Service Delivery Advisor Joel Rakwar, HIV/AIDS-RH Advisor Linda Archer, Monitoring and Evaluation Advisor Anjala Kanasethasan, BCC Advisor Emma Njonjo, Learning and Performance Advisor Karanja wa Mbugua, Policy and Systems Advisor Mbugua Kang’ethe, Operations Manager, Finance and Administration Joyce Isiaho, Project Administrator Emah Macharia, Monitoring and Evaluation Assistant Coast Province Team Feddis Mumba, Area Manager George Avosa, Training and Supervision Coordinator Patience Ziroh , Service delivery Coordinator Frank Mwangemi, HIV/AIDS Coordinator (50% ) George Kaggwa, BCC Coordinator James Musyoki, Assistant BCC Coordinator Western Province Team Moses Lukhando, Area Manager Kennedy Odera, BCC Coordinator Eunice Okoth, Training and Supervision Coordinator Cornelius Kondo, Service Delivery Coordinator Allan Gohole , HIV/AIDS Coordinator Joyce Wafula, Women’s Agency Coordinator EngenderHealth David Adriance, Regional Director for East and South Africa Family Health International (FHI) Ndugga Maggwa, Regional Director IntraHealth International Mary Wieczynski Furnival, Regional Director 78 Alix Grubel, Kenya Country Director PATH Rikka Trangsrud, Country Director Michelle Folsom, Regional Director The POLICY Project Colette Aloo-Obunga, FP/RH Coordinator Population Council Charlotte Warren, Program Associate NASCOP Mary Wangai, Deputy Head, in charge of ARV Pauline Mwololo, Program Officer Aga Khan Health Services, Kenya Salim Sohani, Director of Community Health Department Swafiya Salim, Financial Management Specialist CLUSA Joyce Wafula, Women’s Agency Coordinator Family Planning Association of Kenya Linus Etyang, Program Director Juma Mwatsefu, Project Manager Community Aid International Joseph Kwaka, Program Coordinator UZIMA Foundation Malesi Kinaro, Executive Director 79 C. DOCUMENTS REVIEWED USAID/Kenya. Integrated Strategic Plan 2001-2005. United States Mission to Kenya. October 28, 2004. Five Year Strategy for the President’s Emergency Plan for AIDS Relief: Strong networks for a Sustained Response. USAID/REDSO/ESA. July 5, 2000. Request for Application for a Cooperative Agreement. AVSC International. September 18, 2000. Technical Application for a Cooperative Agreement for the Delivery of Family Planning, Reproductive Health and Child Survival Services in Kenya: RFA 623-00-A-020. USAID/REDSO/ESA. December 19, 2000. Award No. 623-A-00-01-00019-00, Delivery of Family Planning, Reproductive Health, and Child Survival Services in Kenya. USAID/REDSO/ESA. (eight documents dated between 2001 and 2004). Amendments to Award No. 623-A-00-01-00019-00. EngenderHealth. July 1, 2003. Award of Subagreement to U.S. Organization IntraHealth. EngenderHealth. January 1, 2001. Award of Subagreement to U.S. Organization Family Health International (FHI). EngenderHealth. December 19, 2000. Award of Subagreement to U.S. Organization Program for Appropriate Technologies (PATH). EngenderHealth. January 1, 2001. Award of Subagreement to U.S. Organization University of North Carolina. EngenderHealth. May 2001. AMKENI Initial Workplan and Budget (January 1, 2001-June 30, 2002). EngenderHealth. May 2001. AMKENI Year 2 Workplan and Budget (July 1, 2002-June 30, 2003). USAID/Kenya and MOH/Kenya. July 2004. Improving Reproductive Health Services in Kenya: AMKENI Project Summary. USAID/Kenya. July 2003. Expanded Management Review of AMKENI Project. AMKENI Project. January 29, 2004. Monitoring and Evaluation Plan (Revised January 2004). AMKENI Project. April 2000-October 2004. Quarterly Reports Q1-Q15. AMKENI Project. January 2005. Profiles. 80 AMKENI Project. No Date. The AMKENI Experience. Kenya Central Bureau of Statistics, Ministry of Health and ORC Macro. 1999. Kenya Demographic and Health Survey 1998. Kenya Central Bureau of Statistics, Ministry of Health and ORC Macro. 2004. Kenya Demographic and Health Survey 2003. Ministry of Health. 2003. Training Curriculum on the prevention of Mother to Child Transmission of HIV/AIDS, Kenya. Ministry of Health. 2003. Kenya National Manual for Training Counselors in Voluntary Counseling and Testing for HIV, Kenya. Ministry of Health. 1997. Family Planning and Reproductive Health Policy Guidelines and Standards for Service Providers. Ministry of Health, USAID/Kenya and the AMKENI Project. 2005. Coast Success Stories. Ministry of Health, USAID/Kenya and the AMKENI Project. Coast Update. (Several issues of the newsletter). Aga Khan Health Services, Mombassa. Policy Briefs, 1-4. Aga Khan Health Services, Mombassa. Training Curriculum for Community and Facility Committees. Mati, JKG, and Mwangi JN. Kenya, March 2005. Effectiveness of Decentralized Reproductive Health Training and Supervision System on Service Delivery in Kenya. 81 D. LIST OF CLINICAL SITES VISITED Coast Province Jocham Private Hospital, Mombasa Saint Luke’s Hospital, Kaloleni, Kilifi MOH Dispensary, Gongoni, Malindi MOH Dispensary, Diani, Kwale Western Province MOH Sub-District Hospital, Malava, Kakamega MOH Health Center, Chwele, Bungoma Lugugu Friends Mission Hospital, Bungoma MOH Dispensary, Nangina, Busia 82 E. LIST OF COMMUNITY FOCUS GROUP DISCUSSIONS Western Province Women network, Malava, Kakamega District Health Center Development Committee at Chwele Health Centre, Bungoma District Animators, Chwele, Bungoma District Community groups (Village health members and peer families) Mihuu, Bungoma District Men group, Nangina Dispensary, Busia District Youth group, Nangina, Busia District Coast Province Community group at JOCHAM private hospital, Mombasa District Community group (Peer families and youth) at Kaloleni Mission Hospital, Kilifi District. Community group(VHCs, PHT, TBAs) at Gongoni Dispensary, Malindi District Community group (TBAs,DHCs,Youth and PHO/PHTs) 83 F. DATA COLLECTION TOOLS FACILITY ASSESSMENT FORM (Section I) PROVINCE: DISTRICT: HEALTH FACILITY NAME: hospital/health center/dispensary GoK/NGO/MISSION/PRIVATE A. GENERAL IMPRESSION: 1. Cleanliness of the facility: clean/dirty 2. Status of the buildings: well maintained/dilapidated 3. Status of the compound: well kept/neglected 4. Water available: YES/NO; Source: PIPED/BOREHOLE/RAIN/SPRING/OTHER 5. ENERGY SOURCE: electricity/solar/generatr/paraffin/charcoal/firewood 6. IEC material displayed: yes/no 7. Relevant Policies, Guidelines, protocols, job aids available and accessible: YES/NO 8. Opening Hours: from------to--------DAILY/MON to FRI. Clients have to arrive by----- - (time) in order to be served. B. FAMILY PLANNING: How much of the following commodities are currently available? Amount PILLS Injectables IMPLANTS (norplant/jadelle) IUCDs C. ANC DRUGS AVAILABILITY: YES NO Folic acid Ferrous sulphate SP Quinine Tetanus toxoid Niverapine Vaccines Anticonvulsants 84 ANC EQUIPMENT AVAILABILITY Cards/registers Examination couch BP machine Weight scale/height scale (adult stethoscope) Stethoscope Fetoscope D. MATERNITY STATIONERY & EQUIPMENT AVAILABILITY: yes no Baby scale Adult weighing machine BP machine Fetoscope Stethoscope Delivery couch Oxygen source Ambu bag baby/adult Suction machine Baby resuscitaire Emergency resuscitation Delivery sets Incubators Examination lamp Ward & delivery registers, partograms MATERNITY (OTHERS) Expendables supplies: syringres, needles, gauze, gloves, ADEQUATE/INADEQUATE Incinerator facility ,, ,, Telephone service,, ,, E. CHILD SURVIVAL 1. Is cold chain well maintained? YES/NO 2. Is charting done? YES/NO ANTIGENS CURRENTLY AVAILABLE Number of Doses BCG POLIO DPT MEASLES 85 F. PROCEDURE ROOM 1. Infection prevention measures in place: YES/NO 2. Relevant equipment available and functional YES/NO F. VCT and PMTCT 1. Is space adequate? YES/NO 2. Is privacy maintained? YES/NO G. AVAILABILITY OF TRAINED STAFF TRAINING COURSE NO. OF STAFF FP Technology Update FP Counselling IUD insertion/removal Implant insertion/removal Tubal ligation VCT PMTCT PAC/MVA EOC 86 FACILITY ASSESSMENT FORM (Section II) PROVINCE: DISTRICT: GoK/NGO/PRIVATE/MISSION A. STAFFING STAFF ESTABLISHMENT NUMBER AVAILABLE Doctors (obgyn) Doctors (MOs) Clinical officers Nurses/midwives PHOs/PHTs Lab techs Support staff B. FAMILY PLANNING 2002 2003 2004 PILLS: number of clients served INJECTABLES: number of clients served IMPLANTS: number of insertions/removal BT/L: number of procedures Vasectomy: number of procedures C. ANTENATAL CLINIC 2002 2003 2004 Number of clients served Number of clients given 2 doses SP for IPT Number of ITNs given out Number of clients given Tetanus Toxoid D. MATERNITY The following services were performed/given at least once in the last three months: Yes No Parenteral (injection) antibiotics Parenteral (injection) oxytocics Parenteral (injection) sedatives/anticonvulsants How many of the following services/procedures were performed/provided in the last three years? 2002 2003 2004 Normal deliveries Complicated deliveries 87 Manual removal of placenta Blood transfusion Caesarean section Evacuation by MVA or D&C E. CHILD SURVIVAL 2002 2003 2004 Number of children who have completed immunization (BCG, polio, DPT, measles) Resuscitation of newborn 88 GUIDE QUESTIONS FOR THE BCC COMPONENT Introduction As key players and members of the community the team would like to receive from you views of the AMKENI project covering activities, achievements, challenges and what you would like to see in the future. A. Health Center Development Committees What are your views on the AMKENI project? Achievements so far? As a committee how has the project influenced you to serve the community better? (probe for skills/capacity building, management) What aspects of the project are you involved in? Are there any links you have forged with the community? What are these? Specifically what are the indications of community participation in this project? What are the constraints? What would you like to see? From MOH, AMKENI project and the community? B. Village Health Committees What are your views on the AMKENI project? Achievements so far? As a committee how has the project influenced you to serve the community better? (Probe for skills/capacity building, management) What aspects of the project are you involved in? What is the community response? Specifically what are the indications of community participation in this project? What are the factors that influence utilization of services in your community? What are the constraints? What would you like to see? From MOH, AMKENI project and the community? C. Women/Men/Youth What are your views on the AMKENI project? Achievements so far? In what ways has the project influenced your behaviour, attitude and practice towards your health and family? (Probe for decision-making and health seeking behaviour? Are there any notable changes in health seeking as a result of the project? What are the factors that influence utilization of services in your community? (Probe for access, provide attitude, service quality and waiting time) What are the constraints? What would you like to see? From MOH, AMKENI project and the community? 89 AMKENI Project Organogram Technical Responsibilities Area Manager - Coast Area Manager - Western Finance & Administration SD - Coast SD - Western Service Delivery Advisor BCC - Coast BCC- Western Field Facilitators Women's Agency BCC Advisor Policy&Sytems Advisor T&S - Coast T&S - Western Learning &Performance Advisor M & E Advisor HIV/AIDS - Coast HIV/AIDS - Western HIV/AIDS Advisor Project Director 90