USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM MAY 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Lucy Mize, Rebecca Miles, Mirgissa Kaba, Michael Dejene, and Kuleni Berhanu through the GH Tech Bridge Project. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-01-011) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00004. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM i CONTENTS ACRONYMS ....................................................................................................................................... iii EXECUTIVE SUMMARY ................................................................................................................... v Purpose and Objective of the Evaluation ............................................................................ v Evaluation Methodology .......................................................................................................... v Background of the Urban Health Extension Program ..................................................... vi Key Findings .............................................................................................................................. vi Conclusions ............................................................................................................................. viii Recommendations ................................................................................................................. viii Program Areas that Need Further Consideration ...........................................................ix I. BACKGROUND/INTRODUCTION .................................................................................. 1 Purpose of the Evaluation ....................................................................................................... 1 Rationale for an Urban Project .............................................................................................. 2 Ethiopian Government current support for urban health ............................................... 4 USAID Support for Urban Health ........................................................................................ 4 II. EVALUATION METHODOLOGY ...................................................................................... 5 III. FINDINGS ON JSI MANAGEMENT OF THE URBAN HEALTH EXTENSION PROGRAM ....................................................................................................... 7 achievements toward Goals and Targets of the project ................................................. 7 management issues ................................................................................................................... 8 Performance monitoring ......................................................................................................... 8 IV. FINDINGS ON STRENGTHS AND CHALLENGES OF THE URBAN HEALTH EXTENSION PROGRAM ...................................................................................11 Environmental Challenges for the UHEP ..........................................................................11 Systems Challenges FOR the UHEP ...................................................................................14 UHE-P Skill Development Challenges ................................................................................19 Gender Considerations within the UHEP .........................................................................19 Private-public Partnerships in the UHEP ...........................................................................20 V. FINDINGS ON OBJECTIVE 3: FUTURE USAID DESIGN INPUTS .........................23 VI. RECOMMENDATIONS ........................................................................................................25 Short-term Recommendations ............................................................................................25 Long-term Recommendations .............................................................................................27 ii USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Recommendations for Program Areas that Need Further Assessment and Consideration ..................................................................................................................29 Input on Recommendations from Reviewers and Stakeholders ..................................30 VII. CONCLUSIONS ....................................................................................................................31 The Role of USAID and Its Contractors ...........................................................................31 Urban Contexts ......................................................................................................................31 Support for an Operational Budget ....................................................................................31 Service Delivery ......................................................................................................................31 UHE-P Retention and Recruitment ....................................................................................32 The Evaluation of the Urban Health Extension Program ..............................................33 APPENDIXES: APPENDIX A. FINAL SCOPE OF WORK .................................................................................33 APPENDIX B. PERSONS CONTACTED ...................................................................................43 APPENDIX C: BIBLIOGRAPHY ....................................................................................................49 APPENDIX D: INTERVIEW TOOLS ...........................................................................................53 APPENDIX E: TABLE OF ACCOMPLISHMENTS TO DATE ................................................57 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM iii ACRONYMS AIDS Acquired immunodeficiency syndrome ANC Antenatal Care ART Antiretroviral treatment BCC Behavior change communication CBIA Community-Based Information for Action COC Certificate of Competence DHS Demographic and Health Survey FHC Family health card FP Family planning FMOH Federal Ministry of Health GH Tech Global Health Technical Assistance Project GiZ Deutsche Gesellschaft für Internationale Zusammenarbeit (German Agency for International Cooperation) GOE Government of Ethiopia HCT HIV counseling and testing HDA Health Development Army HIV Human immunodeficiency virus HMIS Health management information system HSDP Health Sector Development Plan IR Intermediate result JSI John Snow Incorporated M&E Monitoring and Evaluation MARP Most-at-risk-population MOH Ministry of Health PEPFAR President’s Emergency Plan for AIDS Relief PLHIV People living with HIV PMTCT Prevention of mother-to-child transmission PNC Prenatal care RHB Regional Health Bureau SNNPR Southern Nations, Nationalities and People’s Region TB Tuberculosis TFR Total fertility rate USAID United States Agency for International Development iv USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM UHE-P Urban Health Extension Professional UHEP Urban Health Extension Program UNICEF United Nations Children’s Fund VCT Voluntary counseling and testing USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM v EXECUTIVE SUMMARY PURPOSE AND OBJECTIVE OF THE EVALUATION The USAID-funded Ethiopia Urban Health Extension Program (UHEP) (Cooperative Agreement No. 663-A-00-09-00428-00) was awarded for three years to John Snow, Inc. (JSI) and is scheduled to end September 30, 2012.1 The purpose of the final evaluation is to assess project performance against documented objectives and targets, identify best practices, gaps, and challenges to the UHEP, and provide recommendations to the Mission and the Government of Ethiopia on future programming to more broadly impact the urban health care system. Specifically, the team was asked to respond to three objectives: 1. Assess the performance of the project in achieving its four expected results and corresponding objectives and targets and identify areas of success and challenges in the implementation. 2. Identify and document the outcomes, strengths, and challenges of the UHEP from the perspective of the Urban Health Extension professionals (UHE-Ps), Government of Ethiopia, and other key stakeholders. 3. Identify opportunities and make recommendations for future USAID/Ethiopia support for the UHEP for the next five years. EVALUATION METHODOLOGY USAID contracted a five-person team with extensive experience in urban health, evaluation, Ethiopia health systems, logistics, and USAID program design to carry out the final evaluation. Their experience, along with a wealth of information collected during field visits to seven out of 20 JSI sites2 and four non-JSI sites, allowed the team to develop findings, conclusions, and recommendations to USAID. The specific methods used for the evaluation include: Literature review, including project reports, technical material, and Government of Ethiopia documents, An in-brief with USAID, as well as a debriefing, to get feedback on findings to date, The development of tools for interviews , followed by interviews with key stakeholders, including group interviews with more than 50 UHE professionals, Limited visits to community households, Meetings with key partners United Nations Children’s Fund (UNICEF) and Deutsche Gesellschaft für Internationale Zusammenarbeit (German Agency for International Cooperation, GiZ), and Interviews with the JSI team at both the central and regional levels. 1 The project only began implementing activities in March 2010, thus, the evaluation is covering two years of implementation from March 2010 to March 2012. It has also been extended through March 2013. 2 The regions for the program and for JSI activities are Amhara, Harari, Oromia, Southern Nations, Nationalities and People’s Region (SNNPR), and Tigray. The city administrations are Dire Dawa and Addis Ababa city. vi USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM BACKGROUND OF THE URBAN HEALTH EXTENSION PROGRAM The Ethiopian government’s almost eight years of experience in implementing the rural health extension program was adapted for the urban setting nearly three years ago and is an explicit part of the current Health Sector Development Plan (HSDP) IV. Ethiopia’s Urban Health Extension Program is an innovative government plan to ensure health equity by creating demand for essential health services through the provision of health information at a household level and access to services through referrals to health facilities. The packages of interventions are in four primary areas: Hygiene and Environmental Sanitation, Family Health Care, Prevention and Control of Communicable and Non-Communicable Diseases, and Injury Prevention, Control, First Aid, and Referral. While many interventions are similar to the rural program, some key differences from the rural setting include the prevention and control of non-communicable disease, mental health, and violence and injury prevention, as these are expected to affect urban populations more significantly. Unlike the rural setting, the government chose to use clinical professionals with three years of nursing education in private colleges as the primary outreach workers and provided them with an additional three months of training to develop their public health skills. Ethiopia is currently experiencing one of the fastest urban growth rates in the world. There is a greater prevalence of HIV/AIDS in the urban setting, non-communicable diseases accounted for 51% of deaths in Addis Ababa, and there are significant issues in sanitation and waste management that affect urban health, given that only 14% of the urban population has access to an improved toilet facility. These data are clear justification for the urban program. KEY FINDINGS The evaluation team has the following key findings from their field work, conducted from March 26 to April 12: USAID support to the UHEP has been essential. USAID funding was from PEPFAR, which necessitated tracking most-at-risk-populations (MARPS), within the multifaceted UHEP program. USAID resources have contributed to the training, deployment, and supervision of the UHE-P and allowed the government to develop and refine its urban health approach. JSI has been successful as a conscientious partner to the government in its implementation of the UHEP. JSI has made evident progress toward meeting its four intermediate results, including reaching 73,027 MARPs, which is 91% of its target to date. Government counterparts stated, “JSI is the backbone of the program,” “best and first line of support,” and “JSI has had an open door for us technically.” There are wide disparities in levels of health knowledge, attitudes, and behaviors among urban populations in Ethiopia. The UHEP’s 15 standard packages of health promotion and prevention have not been sufficiently adapted to the different urban contexts and different health needs. The wholesale adaptation of the program from the rural context has not adequately factored in such urban variation. There is a lack of government operational budget for UHEP. Because there has not been a budget to pay for transportation, stationary, mobile phone calls, materials, and training, this remains a major constraint in achieving greater program effectiveness. There is also a lack of clarity as to whether the kebele administration or the health center is responsible for additional USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM vii budget support. To date, they have only provided budgetary support in limited settings and amounts. Regional Health Bureaus (RHBs) have allowed home-based service delivery to be added on an ad hoc basis to the household-centered health promotion and prevention education activities, community mobilization, and referral to health facilities. Distribution of birth control pills and home-based HIV counseling and testing (HCT) are examples of services provided. Delivering services improved access to households, won clients’ trust, and increased perceptions of professionalism. This service provision remains “informal” since the federal government has not changed the formal job description and tasks.3 The UHEP has experienced approximately a 10% attrition rate and 9% in the JSI sites. The basic professional preparation of the UHE-Ps as well as the quality of in-service training, in terms of a balance of theoretical and practical sessions and the initial number of trainees per class, have affected job satisfaction for the professionals. The uncertainties around their necessary core competencies, such as clinical care versus public health, the daily work objectives, and the expanding ratio of households to professionals, also contribute to high attrition rates. The UHE professionals have selectively implemented the 15 packages. UHE-Ps spend the majority of their time on environmental health and sanitation, home testing and counseling for HIV/AIDS, family planning (FP), antenatal care (ANC), some infectious disease management, and non-communicable disease monitoring. There are positive gains because of the program. The UHE-Ps have become key members of the health system in tracing defaulters for antiretroviral treatment (ART), tuberculosis (TB) treatment, and immunizations, successfully bringing people back into the health system and increasing demand for facility services such as obstetrical delivery. Selecting supervisors with a background in environmental sanitation was a limiting factor in their ability to support the work of the UHE-Ps in many technical areas. Supervisors were considered successful in their ability to resolve community acceptance issues. The short training that supervisors received also hampered their effectiveness, as did the demands on their time for other community activities unrelated to the UHEP. The strong cross-sectoral linkages the program needs are missing. Sectors such as Land Administration, Youth Affairs, Women and Children, Water and Sewage, and municipal administrations do not have a formal role in the implementation of the program, although they control resources that contribute to its functioning. There is no clear guide on commodities supply for services delivered by UHEP at the household level as the program has not been well integrated into facility-based primary health care services. The community outreach aspect of the program, the Model Families concept, is still evolving. There are different interpretations of what constitutes a model family, how much training they should receive, and what they need to accomplish in order to graduate. Due to 3 Provision of HTC service was decided by the RHBs, but the Federal Ministry of Health (FMOH) developed the training manual and asked RHBs to train the UHE-P on HTC as part of the three months’ training. However, providing HCT is not mentioned in the implementation manual. viii USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM constraints, such as language, availability of volunteers, and neighborhood characteristics, the dissemination of health promotion and prevention information by model families is not yet widespread. CONCLUSIONS Based on these findings, the following are the key conclusions: USAID support is an important contribution to the GOE’s urban health development plan and should continue with future programming. JSI has done what it was contracted to do in support of the UHEP and has contributed to the development of the program and successfully piloted approaches. Variations in Ethiopian urban contexts were not taken into account during adaptation from the rural program and, as such, there were missed opportunities to provide appropriate public health services to groups, such as those who are very poor, those who live in marginal urban areas, those who do not live in a household, and those who live in more affluent households. Improving urban health is not a function of the Ministry of Health (MOH) alone. Without support from the different government departments and from allies outside the government, it will be difficult to realize the desired improvements, particularly in environmental health, sanitation, and hygiene. The current implementation of the program has successfully reached TB and ART defaulters in selected cities but has missed opportunities for supporting mothers and newborns in the immediate postpartum, prevention of mother-to-child transmission (PMTCT), nutrition counseling, and more intensive support for HIV/AIDS. The UHE-Ps need better job descriptions, clear career ladders, and standardized approaches to community outreach and hands-on technical support to keep the current work force engaged. Regardless of their deployment station, the UHE-Ps will have to be part of the primary health care service at health centers. Setting defined targets for urban health is essential as is the further monitoring of the impact of the Model Family strategy. Examining the synergy between the Health Development Army (HDA), which is a national strategy, and the UHEP was not within the remit of the evaluation team but should be looked at in future programming. RECOMMENDATIONS The following list summarizes the evaluation team’s short- and long-term recommendations, which are fleshed out in greater detail in the main body of the report. Short-Term Recommendations Conduct operations research. Topics should include barriers to referral; analysis of which socioeconomic groups are benefiting most from the program; and how to reach the most vulnerable groups, including people without homes, in the urban setting. Capitalize on the multimedia channels available for message communication within the urban context to deliver health information to UHEP clients. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM ix Hold follow-up discussions with organizations and groups working in urban governance, including municipal offices, urban development agencies, social welfare bureaus, and local universities to identify possible areas of coordination and synergy with respect to urban health issues. Increase the capacity of UHE-Ps and their supervisors on communication skills, analysis, and application of data for planning and specific knowledge and skills to use in service delivery. Long-Term Recommendations Provide technical support to the RHBs to plan, budget, and institutionalize the costs for service delivery inputs under the UHEP so that regional budgets allocate more than just salaries to the program. Consider promoting modalities of reach other than the household to serve the marginalized and vulnerable urban populations and also the more educated and affluent residents as a way of ensuring health equity in the urban setting. Because the urban environment can influence health behaviors, a checklist should be developed that allows the professionals to classify the nature of the urban area in which they are working. For example, if they note the presence of many migrants and informal settlements, this will necessitate different outreach strategies from working in a more settled neighborhood. Make sure that future contractors have wide experience in both health and urban issues, along with possible municipal governance skills. The future program should support innovative behavior change communication (BCC) methods, such as radio spots, graphic materials, and social media to deliver messages in conjunction with the government program on materials development. PROGRAM AREAS THAT NEED FURTHER CONSIDERATION Because of the need to take into consideration the capacity of the health services to adequately respond and address increased demand, the program must coordinate with facilities before expanding its scope; i.e., mental health should not be prioritized until the government has scaled up the recruitment and training of psychiatric nurses so health facilities can receive referrals. The design team needs to look at how future programming can take into consideration the community impact of the HDA, which is a new national initiative that has areas of overlap with the UHEP Model Family approach. The level of service delivery needs to be standardized to ensure the appropriate mix of health education, preventive and promotive services, and curative services to support health, i.e., providing contraceptive implants for family planning at the household level. Once standardized services are agreed on, then training needs to be done to support the delivery of these household-level clinical services. Clarify the equity issues of the program and whether the program will revise its focus to reach the poorest of the poor and marginalized or provide services to any population subgroup in a catchment area. x USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 1 I. BACKGROUND/INTRODUCTION PURPOSE OF THE EVALUATION The USAID-funded Ethiopia Urban Health Extension Program (UHEP) (Cooperative Agreement No. 663-A-00-09-00428-00) was awarded for three years to John Snow, Inc. (JSI) and was scheduled to end September 30, 2012, but has been extended until March 2013. The purpose of the final evaluation is to assess project performance against documented objectives and targets, identify best practices, gaps, and challenges to the UHEP, and provide recommendations to the Mission and the Government of Ethiopia (GOE) on future programming to bring about wider impact on the health system in the context of urban health care. The evaluation was done from March 22 to May 30, and field work was from March 26 to April 14. Per the terms of reference, the objectives were as follows: Objective 1: Assess the performance of the project in achieving its four expected results and corresponding objectives and targets and identify areas of success and challenges in the implementation. To what extent has each of the goals and targets been met? What are the main reasons/factors for exceeding or not meeting expected results? Have there been any management issues or problems during the project and, if so, how have they been resolved? How well has the partner monitored and evaluated the outputs and outcomes of the program and the extent to which the results are achieved? How can the monitoring and evaluation (M&E) system be improved in the future for the UHEP? What are the key lessons learned from the current UHEP? To what extent has the partner communicated project successes and challenges and disseminated lessons learned to the GOE and other key stakeholders? Objective 2: Identify and document the outcomes, strengths, and challenges of the UHEP from the perspective of the UHE-Ps, GOE, and other key stakeholders. How do the UHE-Ps perceive their role as community health agents? What are the incentives and disincentives to staying in the job? What are their perceived strengths and challenges? What do they recommend for strengthening their role? At the health system level, what are the key issues that have impacted the UHEP? (HMIS? Supply chain? HRH? Health care financing? ) What are the priorities in urban community health needs, based on the experience of the UHEP to date? What do government officials (city, regional, and federal levels) and other stakeholders perceive as priorities for the UHEP over the next five or so years? Objective 3: Identify opportunities and make recommendations for future USAID/Ethiopia support for the UHEP for the next five years. The recommendations should address revisions in the current UHEP framework in order to: 2 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Build and promote a broader integrated urban health plan based on community determinants of health; Create opportunities for focused demonstration projects to comprehensively address priority community health needs (sanitation, mental illness, etc.); Create opportunities for focused demonstration projects to address key health system barriers though innovative approaches (e.g., addressing information management including continuous health surveillance data and reporting and community-facility referral networks, etc.); Create strategic partnerships with other international donor groups to advance broader integrated urban health planning; Address sustainability issues for the future of the UHEP and the appropriate amount and timeframe for USAID financial contribution to the program. RATIONALE FOR AN URBAN PROJECT Ethiopia’s Urban Context:4 Ethiopia is one of the least urbanized countries in the world with only about 17% of its population living in urban areas in 2010. However, it is currently experiencing one of the fastest rates of urban growth in the world. Ethiopia’s cities are growing faster than the country as a whole (2010-2015) at 3.8% vs. a national 2.5% growth. It is estimated that by 2050, 42% of Ethiopia’s population will be urban. Other aspects of the urban context that justify an explicit urban focus are: The size of Ethiopia’s cities varies widely. Its broad definition of urban settlements allows for a total of 925 cities, the majority of which are small with no more than 10,000 inhabitants. A quarter of the urban population lives in the capital, Addis Ababa,5 which is eight times larger than the second largest city of Dire Dawa Poverty and income inequality are rising sharply in Ethiopian urban areas. About 80% of the poor and non-poor urban populations live in slums characterized by substandard housing and a lack of basic sanitation, services, and infrastructure. 11% of the urban population still accesses drinking water from non-improved sources, and 54% of the urban population still accesses a non-improved toilet facility, with only 14% of households having access to an improved facility. 4 Data in this section are drawn from the World Statistics Pocketbook, page 7 of the Ethiopia Urban Profile, and the 2011 Demographic and Health Survey (DHS). 5 Addis is, of course, much larger and more diverse than any other city in Ethiopia and continues to grow more rapidly than most of the other cities. Primacy tends to concentrate economic and population growth in one city; it maintains itself because of the increasing benefits that economic actors obtain when locating near each other or near administrators and politicians. As a result of its primacy, Addis has a disproportionate share of the benefits of such growth but also of the problems that can accompany growth, including associated health problems. These include congestion, pollution, and streams of rural-to-urban migrants, many of whom end up unemployed or underemployed. Although urban primacy is on the decline in Ethiopia as several other cities grow more rapidly than Addis Ababa, the dominance of Addis is likely to remain for many years to come. Therefore, the nature of the problems the UHE-Ps face in Addis will be different from any of the other cities. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 3 15% of the urban population still does not have access to electricity, the same percentage as in 2005 but a growing absolute number. 31% of women in urban settings state that distance to a facility is the reason they do not access health services. Health in Urban Settings: In order to understand the urban health picture in Ethiopia, the evaluation team looked to secondary data. In one burial surveillance survey, 51% of deaths in Addis Ababa were attributed to non-communicable diseases and 6% to injury. The authors concluded, “Non-communicable diseases are the leading cause of death among adults in Addis Ababa, where the health system is still geared toward addressing communicable disease.” 6 In addition, because of the small size of the DHS urban sample, it is not possible to examine differences within urban areas, such as differences between slum-dwelling and non-slum populations. Nonetheless, there were some significant findings in the 2011 survey: The total fertility rate (TFR) for urban women is 2.6, which is half of the 5.5 TFR for rural women; 49.5% of urban women use some method of modern contraception, which is approximately double the rate for the nation. Neonatal mortality is roughly the same in the urban and rural settings, at 41 deaths per 1,000 live births. 32% of urban women do not get antenatal care (ANC) until their second trimester, and 50% still deliver at home7. In urban settings, the median duration of exclusive breastfeeding is 6 months, which is sharply lower than the 2.5 months in rural settings. Twice as many urban women had two or more sex partners in the last twelve months than rural women, although the overall percentage is still very low, at 0.4%. 10 times as many urban men, approximately 50%, report using a condom compared to rural men. 61% of urban women have been tested for HIV, and 56.3% of urban men have been tested; the overall prevalence in urban settings is 4.2%. In general, the following subset of characteristics of urban areas are more pronounced for the poor: (1) the reliance on a cash economy which increases the pressure on men and women to work for pay; (2) overcrowded living conditions (slums); (3) environmental hazards (stemming from density and hazardous location of settlements, as well as exposure to multiple pollutants); (4) social fragmentation (lack of community and inter-household mechanisms for health security); (5) crime and violence; (6) traffic accidents; and (7) natural disasters. 8 While this is a 6 Misganaw et al. 2012. 7 The 2011 Demographic and Health Survey data, which averages data within the urban category, possibly masks real disparities that exist between population subgroups within the heterogeneous urban settings. Thus, for data like these, it is not possible to tell if there are more births in a peri-urban area, a slum, or an established neighborhood. 8 Vlahov et al. 2011:803. 4 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM global characterization, many of these issues were raised in discussions with stakeholders and counterparts. ETHIOPIAN GOVERNMENT CURRENT SUPPORT FOR URBAN HEALTH The 2004-2009 HSDP III intensified sanitation programs in the urban sector. The finding that 62% of people living with HIV/AIDS (PLHIV) were in urban settings and that only 25% of the inhabitants of the capital city Addis Ababa had access to toilets, despite increasing government efforts, further justified the government’s urban focus. Under the HSDP IV (2009-2014), the government explicitly supported urban health with its scale-up of the UHEP. The UHEP was modeled on the rural extension program and is designed to focus on behavior change through the delivery of preventive and promotive health messages. There will be more than 5,400 UHE-Ps in Ethiopia, and JSI has trained 2,100 professionals, which is almost half of those currently serving. In announcing the program in 2009, the minister said they would be deployed in urban areas of all the states. They are currently deployed in 216 urban and peri-urban areas in all regions except Afar, Benishangul Gumua, and Somali regions. Further expansion will happen as the program solidifies. The goal of the UHEP is to improve access and equity in the distribution of basic health services and help mitigate urban issues such as HIV and poor sanitation. The government grouped services into four categories: Hygiene and Environmental Sanitation, Family Health Care, Prevention and Control of Communicable and Non-Communicable Diseases, and Injury Prevention, Control, First Aid, Referral and Linkages. There are 15 packages of interventions under these four categories. In 2011, the Minister of Health supported the development of an “army” to be change agents in the community and to push community-based services deeper. The HDA is designed to reach every household in order to create demand for health services and promote health. The HDA is to create a network of up to five households under the leadership of one that is recognized as an innovator/frontliner in practicing healthy behavior and that is referred to as a “model family.” The model family is expected to lead the group of households and gradually influence them with positive attitudes and skills toward healthy behaviors. USAID SUPPORT FOR URBAN HEALTH USAID has a diverse portfolio that supports urban health, including many programs in the HIV/AIDS sector. Their support for the UHEP uses funds from the President’s Emergency Plan for AIDS Relief (PEPFAR). Because of the funding source, USAID needs to document reaching the MARPS, while at the same time supporting the GOE’s overall comprehensive health program. USAID’s support to the program covers approximately 10% of the 216 current implementation sites and is in all the regions where the program is active. The current project is very much viewed by government counterparts as a pilot and a learning laboratory, and they are eager to refine the program based on lessons drawn from JSI-supported sites. Because the UHEP program remains such an essential pillar of the GOE’s health sector program, USAID intends to continue its support. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 5 II. EVALUATION METHODOLOGY USAID funded an independent final evaluation of the current project while also collecting input on how future program efforts could be targeted. Because of the nature of the information desired, the evaluation was qualitative and did not expend extensive resources on gathering quantitative data. From March 22 to May 26, 2012, USAID contracted a five-person team with extensive experience in urban health, evaluation, Ethiopia health systems, logistics, and USAID program design to do the final evaluation of the current project. Their experience, along with a wealth of information collected during field visits from March 26 to April 12 in seven out of 20 JSI sites9 and four non-JSI sites, allowed the team to develop findings, conclusions, and recommendations to USAID. The specifics of the evaluation methodology include: Literature review, including project reports, technical material, and GOE documents, An in-brief with USAID, as well as a debriefing, to get feedback on findings to date, Negotiation on site selection between JSI and USAID that took into account travel time, length of time activities had been implemented, and variability in execution to ensure that both strong and underdeveloped sites were visited, The development of tools for interviews , followed by interviews with key stakeholders, including group interviews with more than 50 UHE-Ps, Limited visits to community households, Meetings with key partners UNICEF and GiZ, and Interviews with the JSI team, at both the central and regional levels. JSI has activities in Amhara, Harari, Oromia, SNNPR, and Tigray regions. The city administrations where JSI works are Dire Dawa and Addis Ababa city. The assessment was carried out in Addis Ababa, Amhara, Dire Dawa, Harari, SNNPR, and Tigray regions, which covers 40% of JSI’s 20 operational sites as well as four non-JSI sites in Oromia, Amhara, and Tigray regions. RHB staff, city and wereda health office staff, health center heads, UHE-Ps, and their supervisors were the primary informants as they are the key stakeholders. These individuals were interviewed in either Amharic or English, using a predeveloped checklist. Representatives of donor stakeholders in urban health and governance (GiZ and UNICEF) were also interviewed. The team interviewed more than 50 UHE-Ps in groups of seven to 10, using a checklist developed to guide the interview. The professionals were selected from those who were working on the day the team arrived. The interviews lasted between two and three hours and were conducted in Amharic. In addition to interviews, the evaluation team conducted observations at community sites. Finally, the data were synthesized by the team during a three￾day discussion of field data in a moderated setting where themes and sub-themes emerged. The team then came to conclusions and developed recommendations oriented toward answering the 9 The regions for the program and for JSI activities are Amhara, Harari, Oromia, SNNPR, and Tigray. The city administrations are Dire Dawa and Addis Ababa city. 6 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM objectives noted in the terms of reference. USAID conducted a more extensive debriefing of government partners after the draft was completed, and their feedback was incorporated into the ultimate draft. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 7 III. FINDINGS ON JSI MANAGEMENT OF THE URBAN HEALTH EXTENSION PROGRAM ACHIEVEMENTS TOWARD GOALS AND TARGETS OF THE PROJECT JSI was awarded a sole source cooperative agreement to implement USAID’s support to the UHEP, in part because of their prior extensive input into the design and start-up of the program, especially support to training the first group of UHE-Ps in Addis Ababa and Amhara. JSI has received an extension to the original contract and will continue implementing activities until March 2013. Overall, JSI has been successful in its achievements to date considering that actual program implementation only began in March 2010, rather than October 2009 when it was awarded. Once the legal10 issues were resolved, there was no lag time in technical implementation. Intermediate result (IR) one is “Improved access and decreased barriers to public health services for vulnerable populations through engagement of households and communities.” JSI has reached 73,027 MARPs, 73% of whom are female, which is 91% of their target. Of those reached, 31,839 individuals received HCT, and 29% of those are men. Of those tested during the 1st quarter of FY 2012, 2.7% tested HIV positive. JSI exceeded by 2.5 times their target for the number of vulnerable individuals who received HCT services, including receipt of test results. So far, they have reached almost 26,000 people, which is 257% of their planned target. Overall, four of five indicators for this IR are on track or better; only the number of individuals referred to health and other social services is low at 31%. One possible explanation for this rests in the fact that UHE-Ps are now delivering services within the home, thereby negating the need for referral. Intermediate result two calls for “Increased demand for public health services through active engagement of vulnerable groups, households, and communities using behavior change communication (BCC) for health prevention, promotion, and risk reduction.” The primary indicators measure numbers of individual members of the community who have received orientation and guidance to be involved in health promotion activities. JSI has only reached 119 of the 2020 planned to date, a low 6%. However, this result reflects the changes in the program orientation and the development of the Model Family and the HDA. They have not yet crafted an indicator that reflects the extent to which these two efforts create active engagement with vulnerable groups and how they are increasing demand for public health services. Intermediate result three focuses on “Improved quality of UHEP service delivery through training and professional development of UHE professionals on public health, including HIV/AIDS services for vulnerable populations.” To date, JSI has trained more than 50% of the national UHE-P workforce—2,122 nurses and 152 supervisors. There are six indicators for this IR. In three of them, JSI is meeting 90% or better of the planned targets—the number of UHE-Ps and supervisors who attended refresher training on risk assessment, the number of supervisors and regional health bureaus trained in facilitative supervision, and the number of supportive follow￾up visits. In fact, the number of supportive follow-up visits, at 131%, has exceeded the target. 10 There was no lag time in the project implementation from a technical perspective, although from a legal perspective, JSI was unable to implement activities for six months as their company headquarters negotiated with USAID about funding that had been misallocated to its for-profit entity, instead of the intended not-for-profit entity. 8 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM The one indicator that is very low is the number of completed referrals by UHE-Ps, which is at 28% of the anticipated target. There are two reasons for this. The first is that services are being delivered at home and thus no longer need referral, and the second is that the indicator was changed to include the word “completed.” Prior to this, the program only measured the initiation of referral. However, after working on the referral system and developing a referral card and feedback mechanism, the program now measures “complete” referrals from initiation to feedback. Thus, there are many referrals being made but only some can be counted as complete. Intermediate result four is framed as “Support an enabling environment to implement a sustainable UHEP including support for development and implementation of community information systems.” There are four indicators. JSI has reached 100% on the distribution of UHEP kits and piloting Community-Based Information for Action (CBIA). The other two indicators measure the number of review meetings at cities/towns and the regional level and are at an aggregate of 58% of the target. In the first year, JSI only met 50% of the target for review meetings in cities, but this year they have already met 85% of their target. As the program gains stature and JSI personnel are more integrated into routine counterpart activities, the need for review meetings lessens. MANAGEMENT ISSUES There are no management issues in the implementation of this cooperative agreement. The Chief of Party has instituted a transparent management system, in which staff believe they are able to communicate easily and openly. They stated they were able to have dialogues with their managers and that decision-making is “nonhierarchical,” both positive findings. Field staff had two minor issues—access to transportation in order to supervise employees and access to conferences and workshops as part of staff development. They stated they were able to raise these issues with management, and although they were disappointed to learn that there is a limited budget available to fund staff participation in international meetings, they were philosophical about the outcome because they felt their concern had been heard and addressed. JSI-recruited coordinators at the regional level and supervisors at the operational level were recognized by government authorities for their unreserved support in rolling out the UHEP and their management strengths. The only obvious issue is that JSI does not have many women in this role. Future recruiting could focus on creating better gender balance. USAID also reported no issues with the management of deliverables; work plans and quarterly reports are delivered on time. JSI has been very flexible and accepted two other activities under this program—the fielding of 13 technical staff to the MOH and the management of the Accelerated Health Officers Training Program. This extension of their responsibilities did not impact in any way JSI’s ability to deliver results for its primary activity. PERFORMANCE MONITORING Monitoring JSI monitors its program through an extensive array of tools. They have supported the introduction of a reporting and recording format for the UHE-P to use in capturing data. They are also piloting and implementing the family folder in Addis Ababa. This is a tool that the GOE is particularly interested in disseminating because it is to play a critical role in identifying needs and providing health services to the community. JSI has regularly scheduled supervision visits, USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 9 and they conduct both regional and city review meetings to serve as problem-solving forums. This extensive monitoring has identified, among others, the following issues: The skills of the UHE-Ps in conducting HBHTC need to be improved, and their counseling and communication of messages needs support; Supervisors still need support to plan, implement, and monitor activities; The lack of HIV test kits within the health system is impacting the UHE-Ps’ ability to provide services to vulnerable populations; and Health center data are trending positive on facility-based delivery, assumed to be a consequence of the UHE-P increasing referrals. The head of the Dire Dawa health center said the center experienced a sharp uptake in services delivered after the program began. JSI has prioritized the UHEPs’ use of data to plan their work and conducted data quality control assessments in 14 towns. This assessment indicated that data had “improved in terms of completeness, consistency, accuracy, reliability, and validity.” Nonetheless, JSI understands that there remain issues with the use of data and how it is reported throughout the health care system. JSI struggles a little with the government emphasis on the overall health management information system (HMIS), while operational data captured at the local level is more useful for planning than some of the data required for government reporting. This difference accounts for occasional inconsistencies in the data they collect. However, JSI has depth in its M&E team and is committed to continued improvement in data collection. Activities planned for this year will continue to target improved capture and use of data. JSI has piloted a system for CBIA in two towns. CBIA is an approach that helps the UHE-Ps engage community members in their catchment areas to assist with the collection of community information, which is compiled, analyzed, and interpreted at the kebele level before action is taken. In Hawassa, the number of daily clients to the Ayder health center increased from 15-20 to 60-70 after CBIA was introduced. Health centers in Debremarkos also showed progressive increases. This program will continue to be rolled out over the next year. JSI also does a good job of disseminating the findings of their monitoring. Regional coordinators routinely share findings with their counterparts in the health sector. They also publish successes in a newsletter and provide USAID with information that can be shared within the larger development community. Challenges within the program are shared more discretely in routine meetings with their government counterparts. JSI is very appreciated by the government. “JSI is the backbone of the government Urban Health Extension Program” and “JSI serves as a learning site to scale up to other towns” were among the statements made during the evaluation. Evaluation There was not a baseline study done to collect data. Given the fairly small resources and the very tight implementation schedule, JSI decided that it was not a good use of resources to collect baseline data. However, in each catchment area, as the UHE-Ps were deployed, they did baseline assessments to determine the health priorities. They have not used any staff or funding resources to do operational studies but think that this should be part of a longer-funded program. 10 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM The results framework currently in use is a hybrid of the GOE’s goal, the PEPFAR program goals and indicators, and an evolving set of IRs as JSI learns what the UHEP can and cannot achieve. One very obvious discrepancy is that the GOE discusses urban food insecurity as something that this program should address, but there are no supporting linkages to activities within the existing results framework. JSI believes that any future program will need a more rigorous evaluation framework, hammered out between USAID and the government, with resources allocated for operational studies. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 11 IV. FINDINGS ON STRENGTHS AND CHALLENGES OF THE URBAN HEALTH EXTENSION PROGRAM ENVIRONMENTAL CHALLENGES FOR THE UHEP Urbanization Urban settings faced by UHE-Ps range from small towns (sometimes very small) to pockets of people within urban areas who are distant from population centers and who are still living rural lifestyles, including raising livestock. In Amhara, for example, some of the city administrators supervise both rural and urban health extension workers because of this duality. Other urban configurations include densely settled, rapidly growing peri-urban areas on the outskirts of cities, more established slums within larger cities, and affluent neighborhoods. In such diverse settings, the UHE-Ps felt inadequately prepared to meet the needs of the residents and to identify those most vulnerable. In particular, they felt that many of the more privileged residents were dismissive of what the UHEP had to offer; UHE-Ps felt the program better fit the needs of the poor. MOH staff at various levels had assumed (and many we spoke to still do) that all residents in urban areas have access to health facilities. Physical access, measured as distance from a health facility, is better in urban than in rural areas, but it still remains a barrier to health service utilization along with the availability and cost of transportation. The latest DHS bears this out, with 31% of respondents reporting that the distance to the facility is why they had trouble accessing services. Urban residents face other barriers as well: financial constraints outside of transportation costs, language barriers, waiting time at health facilities, and the demeanor of health facility staff. There is, therefore, demand for UHE-Ps to deliver services in homes when they visit. The construction, management, and use of latrines exemplify the complexity of working in the urban setting. The GOE considers the construction of latrines to be an individual’s responsibility, and having a private latrine is one of the criteria for becoming a model household. But consider these issues that were identified during the evaluation: UHE-Ps encountered many households who understood the importance of latrines but would not or could not meet the criterion for adequate space; in crowded areas, households often did not have enough space to put in a latrine. In households consisting of a number of renters, absentee landlords were impossible to reach and, therefore, did not take responsibility for latrine construction. In some regions such as Amhara and Tigray, rapidly growing peri-urban areas where basic public services and infrastructure such as schools and health centers did not exist, residents felt they were at risk of displacement from their homes at any time, and thus households were unwilling to invest in latrines. Many UHE-Ps felt that their credibility was undermined because solving problems related to environmental sanitation was well beyond their capacity. For example, in parts of Addis Ababa, households have private latrines but the number of municipal suction trucks that 12 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM come to collect and dump the contents of pit latrines or septic tanks that fill up after years of use is inadequate, and households often have to wait for the service. UHE-Ps feel that when they are not able to obtain better service from the municipality, “the community feels like they are wasting their breath;” “they indirectly blame the UHE-P” for the delay. The evaluation team heard that in some areas of Addis Ababa, UHE-Ps try to organize the community to keep septic tanks or latrines from overflowing and collect money to make whatever repairs are needed. Political Structures and the Ministry of Health The MOH has realized there is need for cross-sectoral support for urban health, particularly in regards to sanitation. Many cities have organized an inter-sectoral task force, whose members include the urban beautification agency, water and sewage authority, education, the Bureau of Women, Youth and Children, the RHB, and the office for land administration under the leadership of the deputy city mayor. These task forces appeared to vary in their composition and effectiveness across areas. Many of the UHE-Ps that the team interviewed stated that once the mayor or the deputy mayor publicly supported their efforts, access to households was significantly easier. The current placement of UHE-Ps illustrates the issue of how the municipal (governance) and technical (health) sectors interact. When professionals are relegated to kebele space, they feel disassociated from the health care system, while at the same time more engaged in the political life of kebele. When they are posted to a health center, they state they have a greater sense of being part of the health system but garner less support from the political structures. Community Building: Model Families As envisioned in HSDP IV, the Model Family and its extension, the HDA, are needed to achieve optimal improvements in maternal and child health outcomes, especially ANC, skilled attendant delivery, prenatal care (PNC), Expanded Program on Immunization , and FP. The role of the UHE-P is to technically support the HDA on a regular basis and meet every month to specifically discuss health-related issues. In addition, because of the Army, the UHE-Ps are free to expand their role of providing basic services at the household level because the Model Family and HDA are now charged with demand creation. Discussing the implementation of the HDA is beyond the remit of this evaluation, as it is a new national project that is just starting. However, the UHE-Ps and RHBs have indicated that it will be a significant factor in the success of the UHEP in the next few years and that its current implementation is still shifting as regions gain experience. In some instances, the UHE-Ps are being asked to be more intensively involved in the HDA activities, and they have some difficulty in stretching to accommodate this task. Despite the evident value enshrined in the philosophy of the Model Family, the UHE-Ps raised some interesting issues about the implementation: There were inconsistencies in determining who becomes a model family. In principle, the selection of a model family follows the analysis of baseline assessment that provides evidence on which households have outstanding health problems. Per the implementation guide, each family has to complete 96 hours of learning and implementation on all the packages to “graduate” as a Model Family. The practice in most of the sites, however, is that USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 13 a potential model family is selected using locally set criteria, irrespective of findings from the baseline assessment. Approval from the kebele administration is also necessary to be a model family, a criterion not included earlier. The training requirements for becoming a successful model family are changing. Families must now complete 15 days of learning and implementation on the packages; many of the model families are “graduated“ without having mastered the health practices in all 15 packages. The community outreach function continues to evolve. Originally, other extant community groups, such as Idirs,11 were intended to do what model families are now doing. Since this initiative is new—it is currently taking shape in Tigray and SNNPR—there is a lack of clear guidance to the Model Family on how to roll out their responsibilities. There are also few monitoring and evaluation tools. Model family members are meant to support their neighbors in the positive practices of the UHEP, but this can be difficult to sustain at times. For example, in Shasamene, one model family member was experiencing difficulty in getting her neighbors to adapt certain sanitation practices, such as the disposal of solid waste. Equity Issues: Reaching Vulnerable Populations The government program requires the UHE-Ps to work on four primary components, only one of which includes a package on HIV/AIDS. At the same time, because of USAID funding sources, the goal, objectives, and four expected results of the JSI project require collection of data on the MARPs. Using common PEPFAR language, this is defined as sex workers, domestic workers, waitresses, taxi drivers and their assistants, bus drivers and their assistants, young married girls between the ages of 15 and19, petty traders, migrant workers, out-of-school youth, PLHIV, spouses/partners of high risk groups, and others (uniformed force and disabled individuals). To creatively manage this dichotomy, JSI worked within the comprehensive framework but added special activities and services that focused on MARPS. Specific activities included the following: Supported the UHE-Ps by preparing maps of their catchment areas that identify the locations where some of the MARPS, such as sex workers, are found in large numbers. This helped the UHE-Ps reach the groups with information on HIV/AIDS and with services like voluntary counseling and testing (VCT), PMTCT, care and support at the household level, and referral to a health facility. Helped the UHE-Ps identify PLHIV and ensured that they are enrolled in pre-ART and ART care. Using model families to provide psychosocial support to the known PLHIV is carried out in the context of reaching MARPS. JSI required the UHE-Ps to directly report their activities on MARPS to the JSI regional coordinators, something they did not have to do for the other packages. Thus, they reported on the number of MARPS reached, the number of people tested for HIV who had received their results, the number of HIV positive people who received care and support, 11 Idirs are traditional Ethiopian burial societies that help members bury their dead. 14 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM and the number of people referred to different services as well as those with completed referrals. Overall, while this dual emphasis did not seem to affect the implementation of the program, there were some counterparts in the RHBs who were frustrated at the continued focus on MARPS. They stated that there were other donors and groups working in the HIV/AIDS sector who could reach MARPS but there was only USAID via JSI that was supporting the UHEP, and they did not want support to be narrowed to just MARPs. In terms of reaching vulnerable populations, many believe that the household modality of the program is not the best channel for reaching some of the other identified MARPs, such as street children and migrant workers. All of those interviewed stated that this program doesn’t reach those who live on the streets or who are itinerant for one reason or another. The UHE-Ps perceive that the target population for their services is the poor, while the government managers believe anyone living in a catchment area, whether rich or poor, is the target for services. According to the UHE-Ps, they feel most comfortable reaching out to the poor and, in fact, feel the poor can benefit the most from the program. UHE-Ps have been turned away from wealthier households and dismissed by the owners. They say that there are only maids and animals in rich households by day. In discussing this issue with UNICEF, they noted that this tension has been present since the design of the program, when factions argued for focusing on the poorest of the poor and others argued for greater community inclusion. There have been no studies done on the demographic characteristics of those who have benefited to date from the program, but the evaluation team believes, after discussions with the UHE-Ps, that it would indeed be the poorer urban groups who are embracing these services. SYSTEMS CHALLENGES FOR THE UHEP The Ethiopian government’s almost eight years of experience in implementing the rural health extension program was adapted for the urban setting nearly three years ago. The following section focuses on key findings on deployment, training, and support in the UHEP. Deployment: Recruitment and Training The UHE-Ps have an identity as clinical nurses because of their initial three-year training. They are drawn from the ranks of clinical nurse graduates of private colleges who did not have employment after graduation, unlike nurses trained in public colleges. The preservice training of three months was to foster their public health skills and prepare them to work at the household level on health prevention and promotion. The evaluation team found, however, that this training would neither make them public health practitioners nor fully competent to be UHE-Ps. Each region has slight variations on recruitment criteria, including age, professional background (clinical nurse), language, place of residence, employment status, and willingness to work in a community setting. RHBs indicated that under the current federal autonomy practices, it was alright for them to modify and adapt the national guidelines. One example is the Certificate of Competency (COC). According to the federal bureau, the UHE-Ps were not allowed to take the COC because it would encourage the UHE-Ps to seek employment in other clinical settings and create too large an attrition rate. However, having a COC was added to the criteria of recruitment in most of the regions except Tigray and Addis Ababa. In Tigray, experienced nurses who were older than the norm and possessed a COC were recruited, based on their interest to serve at the household level even though they might have had previous employment USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 15 within the system. This differed from the overall recruitment profile of the program. In the rest of the regions, UHE-Ps invariably indicated that getting an employment opportunity and putting their clinical skills (note: NOT their public health skills) into practice were major reasons to join the UHEP. Following recruitment, training was offered on the 15 packages of UHEP with a focus on demand creation for public health services. Theoretical classroom sessions took most of the time, while practical sessions focused more on role play, visits to health facilities, and development of questions for household interviews. The training generally lasted three months, but in Oromia and SNNPR, additional hands-on training of one month was provided on HCT, PMTCT, immunization, and nutrition, depending on the interest area of the donor partner. Following this formal training, which was government-organized and run with support from the USAID-funded JSI program, there were a few refresher trainings on MARPS, data management, and reporting. These were provided by JSI to both UHE-Ps and their supervisors. Some new recruits have experienced a different kind of training than those who were initially recruited to the program. Addis Ababa conducted a modular training, as opposed to a three￾month intensive course, for new recruits who were replacing those who had left the program. Some towns in Oromia and Tigray deployed UHE-Ps without the formal training because of the need to have staff in place. Thus, they are already in service but without complete understanding of the material they need to promote. The remaining regions are training new recruits according to the standards previously determined. The Federal Government mandated that one UHE-P be provided per 500 households and used this as a recruiting target, but they have fallen short. In some places, the ratio was 1 to 650 households, while in Tigray, some UHE-Ps cover up to 2,200 households. The UHE-Ps felt that many of their catchment areas were too big and that they needed more colleagues to distribute the burden of work. RHBs are aware of this issue and trying to recruit more professionals but have budget constraints and an inability to pay more salaries. When the professionals were assigned to their respective communities, deployment was determined by lottery method. In some cases, such as in Harar, reassignment was necessary because of community language barriers—there was the need to have Somali-speaking professionals to reach communities speaking only the Somali language. Prior to deployment, community-level sensitizations were carried out in Shashamene and Hawassa, but in Dire Dawa, Harari, and Addis Ababa, community sensitization came after the deployment of UHE-Ps. This delayed sensitization had a negative impact on their acceptance by the community; this was only resolved when the kebele leaders and city mayors expended political capital to ensure the community was more welcoming. Logistical Support and Motivation The catchment area of 500 households, or more in urban areas, seems to be considered small enough to facilitate easy access. The team found, however, that in all settings, the catchment areas were too big to be managed productively and this impacted the efficiency of supervisors as well as providers. During interviews, many UHE-Ps said they spend an average of 5 Ethiopian Birr, which are not reimbursed, to travel to and from the furthest points within their catchment area. Otherwise, they walk for 40 minutes between clients, on average. In fact, a number of the women said they had worn out their shoes in trying to keep up with all their households. JSI 16 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM recently bought and delivered bicycles and small motorcycles to mitigate this problem, but it is not enough, especially since few UHE-Ps received them. There is not yet uniform ownership established for the program; neither the kebele, the wereda health office, nor the health center takes the responsibility to provide basic supplies. In Sashamene, the town health bureau argued that, “Currently, we do not have specific budget for UHEP. For next fiscal year, however, the health center is advised to include stationary materials in their plan.” This issue further extends to paying for telephone calls. UHE-Ps stated that they use their telephones a great deal to call clients at the household and to set up appointments; they also use their phones to manage the development army and model family outreach. None of these calls are reimbursed under the current program budget. Commodities supply for community health services is a critical issue. Since the UHE-Ps are not an integral part of facility services, it is not clear to all where they should get their supplies. The same issue extends to nonclinical supplies that are necessary for them to do their job. The UHE￾Ps do not have a uniform. In some places, they were issued a white short coat to wear, and in others they just wear regular street clothes. Badges were not standard. In some catchment areas, they wore them, and in others they had not been issued. A few months prior to the assessment, many UHE-Ps in the JSI- supported sites were supplied with kits that contained a blood pressure apparatus, a fetoscope, condoms, paracetamol, gauze, alcohol, a thermometer, and contraceptive pills. However, this content varies from one site to another, depending on what additional items the health center is willing to provide. For example, in Dire Dawa the UHE-Ps were able to draw on health center stocks for contraceptive implants, which they then provided to clients in the home. In all the sites, it appeared that professionals do not know what to do when they run out of stock except report it to the health center, which may or may not have stock or budget to replenish their kits. In as much as transportation remains a concern, the UHE-Ps also complained about a lack of stationary materials as a common problem at all the sites visited. Neither the professionals nor supervisors know who is responsible to fulfill these needs. The evaluation shows that the allocation of office space for UHE-Ps was not yet a uniform practice. While in some regions (Dire Dawa), UHE-Ps operated from the health centers, in the majority of the sites visited, they sit within the kebele compound. Comments made by the UHE￾Ps indicate that many of them prefer to sit within the health facilities instead of the kebele as they associate sitting at the health center with their identity as a health professional. Current office space, irrespective of whether it is in the kebele or in the health facility, was generally found to be small and insufficient for record keeping and data analysis. This issue of office space also has political ramifications on the ownership of the program and the willingness to expend resources to make it work. Supervisor Support for the UHE-Ps Supervisor support to the UHE-Ps varies remarkably. The ability to supervise was impacted by the level of interaction with the wereda health office and the location of that office. In cases where the supervisor sat in the wereda health office or health center, supervisors were often demanded to engage in routine activities out of their job description, which affected the time they spent with UHE-Ps. Furthermore, the fact that the majority of supervisors are male sanitarians with specific competence in environmental health may have affected the level of technical support they could provide to the whole UHEP package. Some of the RHB staff acknowledged that drawing so heavily from the sanitarians skewed how they supported the USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 17 program and that perhaps it would be better served by supervisors with broader competencies. Already, more clinical nurses are being recruited to serve as supervisors. While limited in their ability to be a source of technical support, supervisors were cited as instrumental in solving problems between the UHE-Ps and households who fail to recognize them, in sharing the concerns of the UHE-Ps with higher bodies of the kebele and health center or the wereda health office, and in collecting and compiling the monthly report. Service Delivery: Practices and Packages According to the Federal Government, the UHE-P’s mandate is to focus on health promotion and prevention, but in all the regions visited, their services have extended to provision of basic health services both at the household level and through outreach. The services vary by region and consist of a mix of the following: FP, ANC, and counseling for HIV; in a few settings, UHE-Ps reported providing advanced FP services (implants), but most provided more basic FP (pills, condoms, and Depo-Provera), HCT, blood pressure monitoring, and first aid for emergency cases. RHB managers feel that UHE-Ps can provide basic services at the household level due to their competence as clinical nurses. Emphasis to date has been on health education and referrals but demand by clients has influenced the UHE-Ps, and they now provide simple curative and “biomedical” preventive interventions, such as implants. In some areas, urban residents were not willing to engage with the UHE-Ps just to get health information, as they believe themselves, in general, to already understand basic health messages. In conversations, the UHE-Ps often did not mention the work they do on non-communicable diseases, except for comments about measuring blood pressure. UHE-Ps also provide immunizations in some sites and counseling and testing services through outreach in the community, schools, and youth centers. Although the program mandated that all three venues were of equal importance, to date, the UHE-Ps have directed the majority of their services just to the household. There have been isolated cases of working with schools. One UHE-P detailed how a family who had been reluctant to embrace hand-washing became convinced by the messages their school age child brought home from the program implemented at the school. Interestingly, officials in two places (Dire Dawa and Harar) mentioned that the UHE-P had a role in working with prisons, to provide education on hygiene and environmental sanitation. One result from the program is that the UHE-Ps have been very successful in tracing clients who have defaulted on TB and ART regimens in some cities and linking them back with health facilities. They are also tracking immunization records. Although the methodology of this evaluation did not include gathering quantitative data, heads of health centers anecdotally credit the UHE-Ps for increasing the demand for facility delivery and the utilization of other primary health care services. One consistent issue that was raised is that either through their own interest or perhaps because of client demand, the UHE-Ps are not implementing the full set of 15 packages as originally envisioned. None of the UHE-Ps raised the topics of violence, food hygiene, 18 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM postpartum care, or leprosy prevention during the interviews.12 There were few mentions of nutrition or malaria. It appears that there is no clear guidance provided to UHE-Ps on which package of services they should provide at the household and outreach levels; this contributes to an inconsistency of services and lack of uniformity in the type of services UHE-Ps provide. It also represents a real missed opportunity to have an impact on PMTCT, neonatal health in the early postpartum period, and overall nutrition. Referral Networks At the beginning of the project, the referral networks managed by the UHE-Ps were not standardized and were unidirectional, with no feedback returned on the outcome of referral. According to some of the supervisors, health center staff did not value the contribution of the UHE-Ps, and, in fact, in some cases, duplicated services such as HCT because they did not trust the professionals to have done it correctly. The supervisors and the health professionals indicated that this has been substantially improved during the last two years, following a series of joint discussions between health officials at different levels, UHE-Ps, and their supervisors. In some instances, referral outcomes and follow-up are specifically discussed during meetings between the UHE-Ps and their supervisors. In addition, JSI’s creation of a common referral slip and procedures for feedback has greatly ameliorated the problem. Nonetheless, the following issues remain: There are still inconsistencies regarding referral feedback. In some cases, the UHE-Ps collect feedback from the boxes at the health center; in other cases, the client returns the slip, and yet in other sites, the supervisor is charged with managing the collection of the referral feedback. There is no system in place for tracking clients who do not complete the referral and seek services at a facility, although supervisors are engaged to put pressure on recalcitrant clients. The UHE-Ps’ linkage to referral sites is limited to public health facilities and does not adequately consider the role of private health facilities in urban settings nor the role of other social service agencies. Data Collection and Analysis One of the centerpieces of the program has been the development of, and support to, the family folder as a streamlined way to collect data. Despite some delays, it is now progressing. According to JSI, the family folder will play a critical role in identifying needs and providing health services to the community. Coupled with the CBIA work, which supports the UHE-Ps engagement of community members in their catchment area for assistance with the collection of community information, there is increased capacity to collect and use data. A data quality control assessment found that, for the most part, figures were consistent, complete, and accurate. 12 While the evaluation team used open-ended questions and probes, they were careful to determine what the UHE￾Ps spent their professional time working on. The information on what they worked on was remarkably consistent in the regions visited. While there is the possibility that in other regions, these topics are of greater concern and thus addressed, there is the equal possibility that the UHE-Ps don’t know how to broach the topic of violence or that leprosy, which has had a declining incidence in Ethiopia, does not appear among the top urban health concerns in the populations with which the UHE-Ps work. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 19 However, during the evaluation field work, the UHE-Ps indicated that they had trouble keeping up with reporting and recording requirements and considered them onerous. In its program brief, JSI stated, “Due to delays in implementing the community based information systems, including the Family Folder, which would enable the recording, reporting and tracking of information and progress for households, each city/town resorted to using varying recording and reporting formats.” Thus, while there have been improvements in data collection, it remains an area where the UHE-Ps and the government are in need of further technical assistance. UHE-P SKILL DEVELOPMENT CHALLENGES Communication Skills for Behavior Change UHE-Ps state they are ill equipped to do some of the delicate counseling that arises when dealing with a discordant couple, treating people on ART, or trying to convince someone to use FP or have an HIV test. In addition, there seems to be a limited understanding among the UHE￾Ps that there are phases in communication, such as initially developing rapport with a client prior to discussing more sensitive subjects. Training on communication skills to develop the UHE-P’s skills to respond to differing information needs within households has yet to take place. Nonverbal cues to communication, particularly in the matter of dress, are also a challenge. Many of the young women would like to have a readily identifiable uniform, even if it is just a white coat, because they think it would make their acceptance into the community a little easier. Use of Materials for Behavior Change According to the 2011 DHS data, 62% of urban households have access to a radio, 42% have access to a television, and 65% have access to a mobile phone. In addition, 56% of urban women have had either no education or only some primary school; of those, 30% of urban women cannot read at all. Yet the communication materials that were available during the field assessments were primarily written in nature. The only BCC tool JSI adapted from the rural setting is the family health card (FHC), a booklet with a range of small doable actions along with illustrations showing the actions. Overall, the evaluation team found a surprising paucity of innovative BCC materials. Training Needs for Clinical Skills Currently, the rural extension workers are getting trained to do significant clinical work, such as inserting implants for FP, despite the fact they are only tenth-grade graduates and not clinicians. At the same time, the UHE-Ps, who have graduated from nursing school, are not supposed to do this kind of clinical work. Despite this, many of the regions are allowing the UHE-Ps to perform clinical services at home, but they are not investing in their training to ensure quality and competence. GENDER CONSIDERATIONS WITHIN THE UHEP USAID has a vested interest in promoting gender equality and to that end published a new gender policy in March 2012 that espoused: Reducing gender disparities in access to, control over, and benefit from resources, wealth, opportunities, and economic, social, political, and cultural services; Reducing gender-based violence and mitigating its harmful effects on individuals and communities; and 20 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Increasing capability of women and girls to realize their rights, determine their life outcomes, and influence decision-making in households, communities, and societies. While in the design of the UHEP, there was little formal consideration given to gender issues, there have been benefits for women in particular. The federal and regional government’s women-only employment policy for recruiting nurses for the UHEP created an important new source of employment of women at the local level. Other benefits mentioned by interviewees include: Collaborating with the UHE-P as a model household confers increased status to women in households; for example, in Harar, a male head-of-household team member reported that he has an increased appreciation for his wife’s skills after watching her reach out with health messages to other community members. The implementation modality followed by the UHEP gives an opportunity for women at the community level to become actively involved and often to assume a leadership role in deciding on issues related to the household and environmental sanitation, their health, and the health of their children. Some of the women interviewed by the team also feel that the support that they are getting from UHE-Ps empowered them to engage their husbands in dialogue on issues related to HIV/AIDS, FP, and the health of their family. Anecdotal evidence suggests that women are benefiting from the program by an increased use of FP and having more of their babies at health facilities—both positive contributions to the long-term reproductive health of women. However, the evaluation team also noted that the use of an all-woman health force makes it less likely that men will be reached with messages on sensitive issues such as sexuality and HIV. For cultural reasons, UHE-Ps are also limited in their capacity to address men’s understanding of and concerns about their own sexual health. Since the UHE-Ps primarily visit households during the day and during work hours, they are more likely to miss men than women because the employment rate for men is substantially higher than for women in urban areas, 77% vs. 50%. Furthermore, there are only limited opportunities for UHE-Ps to involve men as allies in their prevention and promotion efforts in order to reduce men’s resistance to FP and condom use or HIV/AIDS testing. In Tigray (which is not necessarily representative of the entire nation), women often reach a collective agreement to be tested for HIV with other women members of the development army and then, collectively, they invite their husbands to join them. If they choose not to, the women still go ahead with being tested. In other areas, some UHE-Ps who had tested women without their husband’s permission reported being subjected to verbal abuse and threatened physically. The UHE-Ps are aware that working daylight hours limits their interactions with male householders and working women and have extended their working hours to include evenings. However, this has a downside too, as they are more vulnerable as single women at night and thus increase the risk to their own personal safety. PRIVATE-PUBLIC PARTNERSHIPS IN THE UHEP The evaluation did not call for addressing the public-private partnership despite USAID’s interest in fostering such partnerships to leverage resources and contribute to sustainability. The only finding that the team is able to note is that, currently, there is no explicit mandate USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 21 within the program to capitalize on private sector inputs. The UHE-Ps mostly refer clients to the government sector despite the fact there are private clinics that could support both HIV care and FP, as well as other services. USAID has private clinics supported through other program mechanisms, but there does not seem to be any point of intersection with the UHEP. The other richer aspects of partnership, such as working with businesses within the communities to address broader urban health issues, will need further evaluation and follow-up. There is literature outside of the UHE-P program that suggests workplace interventions would be appropriate in the current Ethiopian context. For example, diabetes (an emerging noncommunicable disease in Ethiopia) lends itself to workplace interventions. As urban settings in Ethiopia attract more industry, such as the leather manufacturers, the UHEP could explore possible linkages with workplace programs, perhaps getting referrals of clients who need continued diabetes education messages. 22 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 23 V. FINDINGS ON OBJECTIVE 3: FUTURE USAID DESIGN INPUTS The GOE is interested in receiving continuing assistance from USAID for its UHEP. It would like future support to be more pluralistic and to move away from the focus on MARPS that has been evident in this first phase. As Ethiopia continues to urbanize, investments in the urban sector will be increasingly important. Future design efforts will need to respond to challenges identified both through this evaluation and through the government’s own monitoring work. The training content for future UHE-Ps, as well as how they practice within specific urban contexts, will be germane to future design efforts. For example, the current program does not adequately exploit opportunities to support nutrition education nor does it use its access to households to promote neonatal health, both areas that may feature in the future USAID support. 24 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 25 VI. RECOMMENDATIONS This section provides recommendations that can be immediately implemented during the remainder of the JSI contract, recommendations that need a 2-5 year implementation period and a discussion of longer term issues before the Mission can support further programming. Among USAID’s seven criteria used to judge the efficacy of recommendations are whether they are practical, specific, and action oriented. These criteria were used when developing the following recommendations, as was the feedback from the debriefing discussion on what role the GOE would play in advancing the program. For the government, recommendations need to be couched in the positive and feasible within the constraints of the GOE’s resource environment. There is the assumption on the side of the evaluation team that USAID, both through its own technical staff and through its contractors, has a sufficiently close relationship with key decisionmakers in the MOH to that these recommendations will be discussed openly and as a beginning of government-to-government advocacy for changes in the UHEP. The implementation of many of these recommendations will fall to the government, but USAID has a role in raising the concerns, which is why they are included here. Equally, the recommendations on the design process are more pertinent to USAID than the government. Some of the short-term recommendations can hopefully be taken up by JSI as it manages the next year of the program. SHORT-TERM RECOMMENDATIONS Training Focus training efforts on supervisors to improve their technical knowledge and management skills. Emphasize training in counseling and communication skills and use the vehicle of the new home-based HCT curricula to develop these skills. While the evaluation team feels this is a priority, there should first be a formal training needs assessment to determine global program priorities in training and to identify what needs the partners could best fulfill. Increase the capacity of UHE-Ps and their supervisors on communication skills, analysis and application of data for planning, and specific knowledge and skills to use in service delivery. Urban Context Address boundary issues in dealing with the implementation of the UHEP. In areas where the urban is encroaching on the rural, the roles of the urban and rural extension workers need to be clarified. Town administration health offices, wereda health offices, and RHBs should take the lead on this discussion. Operations Research Conduct operations research on specific issues to inform programming and policy. Such research may help to engage Ethiopia’s universities with the program and to provide evidence for policy change. Topics to be considered are listed below, but the research agenda should be determined after a consultative process with key stakeholders: 26 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM A needs assessment to determine skill gaps of the current UHE-P. This is needed because there is concern that newer entrants into the program have received a less thorough training than the initial cohort and also because the public health precepts that guide the program might not be reflected in the basic knowledge of the practitioners. An analysis of disease burden, other health issues, and service utilization within urban settings to inform city-specific strategies. (This might be in the longer term because of the research protocol that needs to be processed.) Analyze which groups of MARPs were the most effectively reached under this program. Investigate which population subgroups were targeted during the initial phase of the program. This research would also allow an exploration of message as well as service differentiation, based on economic strata or other population characteristics. It would also contribute to the debate on whether the program’s modality is best for reaching the poorest of the poor or all socioeconomic groups. Logistics Provide appropriate transport means for the UHE-Ps who practice in extended catchment areas and work with the appropriate administrative unit (town administrative office, subcity and kebele administration, RHB, etc) to determine office space for UHE-Ps and furnishing needs. Develop an operational budget that covers transportation, communication, and stationary costs and discuss with counterparts how the government could fund the costs. Data Management Update the catchment area maps and network resource manuals, using greater community participation and including all the water resources, community latrines, other sanitation infrastructure, and pedestrian safety amenities. Continue to strengthen the CBIA work that has been started and take it to other sites. Continue to focus on the use of data as a program planning tool so the UHE-Ps continue to increase their ability to use evidence-based planning. Future Program Design Process When writing the contracting instrument, USAID should ensure the contractors have both urban and sanitation experience, along with broad health experience and possible municipal government management experience. Ensure that the gender perspective within the project is explicitly stated and identify expected results to be monitored through indicators and progress toward goals. In the current USAID portfolio, there has been work funded on improving urban administrative management and sanitation. The USAID Health Office should confer with other sector colleagues to determine if there are opportunities for future integration and coordination. Confirm the nature and scope of the Gates Foundation Urban Sanitation program to identify points of collaboration. Confer with the World Bank and the GiZ to determine if the new USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 27 phases of their municipal services projects could also be linked to the UHEP for optimal results. LONG-TERM RECOMMENDATIONS Urban Context Because the circumstances in which people live can influence their health behaviors, a checklist should be developed that allows the professionals to better understand the urban setting in which they work. For example, if they note the presence of many migrants and informal settlements, this will necessitate different outreach strategies than working in a more settled neighborhood. Consider promoting other modalities of reach besides the household to serve the marginalized and vulnerable urban populations and also the more educated and affluent residents as a way of ensuring health equity in the urban setting. Provide implementation guidelines for how to vary the approach and messages in different types of urban settings, such as areas where residents face the insecurity of knowing they could lose their home at any time, areas where households face severe land constraints, and areas with a significant itinerant population. In the future, there will be a need to ensure that the training of the UHE-Ps and the packages of services they offer are tailored to the location of cities, their size and residential density, and their socioeconomic and sociocultural characteristics. Urban health should be a focus area for more government departments, sectors such as infrastructure and housing, and other partners. Support for the Operations Budget Logistical support, such as telephones, transportation, and uniforms, needs to be provided to increase the professionalism of the UHE-P. If supported initially by USAID, there need to be discussions on how this will be budgeted by the government in later years. Provide technical support to the RHBs to plan, budget, and institutionalize the costs for service delivery inputs under the UHEP so regions allocate budgets for more than just salaries to the program (i.e., the cost of contraceptives, medicines, and other necessary inputs that UHE-Ps use via home service delivery). Ensure that the UHEP is an integral component of any regional and wereda annual plan and that agreement is reached on the types of services that will be provided by the UHEP so that budget is available to support them. As part of the One Budget approach under HSBP IV, the MOH should also receive assistance to identify other partners in support of the UHEP. As GiZ moves into a new phase of municipal governance, it will be important to look at how cities deliver services, and as the Gates Foundation moves in with an anticipated large urban sanitation program, it will be important to identify who is doing what. Service Delivery Maintain an active operations research agenda to provide evidence for policy change. 28 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Develop more animated materials for inclusion in the UHE-P’s teaching arsenal. These materials should take into account the multiplicity of channels available in urban settings, as well as the education limitations of clients. Use new technologies (smart phones) to facilitate the reporting and recording of data, as well as to communicate with clients. While the current mobile phone penetration rate is only 7.7%, this represents a doubling of the market in the last three years, and the government intends to reach a 35% penetration rate in the next five years, making cellular phones a viable tool for the health program. Support innovative BCC methods, such as radio spots, graphic materials, cell phones, magazines, billboards, church sermons, and social media to deliver messages in the urban context and in conjunction with the government program on materials development. Standardize the level of service delivery to ensure the appropriate mix of health education, preventive and promotive services, and curative services that support health, i.e., contraceptive implants for FP within the household level. Once standardized services are agreed on, then training needs to be done to support the delivery of these household-level clinical services. Emphasize intervention areas, such as PMTCT, TB tracking, postpartum care for newborns, and nutrition. Encourage UHE-Ps to promote linkages to other social services, NGOs, and CBOs, in particular for clients in need of treatment and support of HIV/AIDS. This can be done through enhanced mapping of resources and catchment meetings. Expand the referral linkages to include all the private sector service providers within specific catchment areas. Assist the UHE-Ps in offering a complete package of services around the use of home-based VCT, including how to deal with discordant couples and how to access community services for support. This will come after the standardization of the complete package of services. Training Invest further in recruitment of qualified supervisors to ensure adequate support for the UHEP. This entails reexamining the professional background and criteria for recruitment and tapping other existing groups, such as older clinical nurses, to serve in this function. Include more active learning opportunities in refresher trainings and plan them carefully so that they are focused on development of interpersonal skills and communication strategies. Other early trainings should focus on clinical skills needed to deliver preventive care in the home. Recruitment and Retention The GOE needs to review the ratio of households to UHE-Ps and further refine it, taking into account the HDA and model families as support networks. This could result in changing the established ratio and recruiting more health workers to work as UHE-Ps. Initiate incentives, such as certificates or acknowledgments of achievement, for the UHE-Ps to foster improved morale. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 29 Work with the GOE to pilot a trial program, pairing existing UHE-Ps with men to facilitate their outreach and support to men in the community. RECOMMENDATIONS FOR PROGRAM AREAS THAT NEED FURTHER ASSESSMENT AND CONSIDERATION Target Population Discuss and refine the target group for the program to determine whether the UHE-Ps should target the lower socioeconomic groups first and as a priority and whether the household approach is the best modality to reach the marginalized and vulnerable urban populations. Determine what would be the best modality to reach the more educated and affluent residents, if they continue to be one of the targets of the program. Service Expansion Coordinate with facilities before expanding the scope of UHEP. Consider the capacity of the health services sector to adequately respond and address increased demand before making changes; i.e., mental health should not be prioritized until the government has scaled up the recruitment and training of psychiatric nurses so health facilities can receive referrals. Discuss with the GOE how community health should be promoted and who should pay for it, especially considering the recommendation to employ more multimedia channels in delivery of messages. Model Family The nature and role of the Model Family aspects of the UHEP need further refinement in order to reflect accountability and the program’s catalytic role in achieving primary health care for all. This includes developing an evaluation framework to assess impact and standardizing the training and graduation criteria. Private-Public Partnership The UHE-Ps need to update resource networks to include all private providers, work with them to understand the referral mechanisms of the UHEP, and then track the use of private services to determine whether the UHEP contributes to increased access and use of private facilities. Explore the other richer aspects of partnership, such as working with businesses within the communities to address broader urban health issues. Possible options would be to see how wellness programs in industry can be supported by the UHEP, particularly in the better management of noncommunicable diseases, such as diabetes13 and cardiac problems. 13 According to the fourth edition of the Diabetes Atlas, twice the number of people in urban settings in Ethiopia have diabetes than in the rural sector. However, this is offset by the tragic fact that “The landscape of sub-Saharan Africa is dominated by the twin disasters of poverty and HIV infection. While HIV infection and consequent AIDS so dominate the health needs for sub-Saharan Africa, there is only a small proportion of the population reaching ages at which type 2 diabetes becomes a major health concern. In 2010 only 9.8% of the population will be 50 years of age or older, and this is expected to increase to only 11.5% by 2030. Thus, the effects of HIV and malnutrition combine to greatly reduce the size of groups most at risk for type 2 diabetes.” 30 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM INPUT ON RECOMMENDATIONS FROM REVIEWERS AND STAKEHOLDERS USAID followed an inclusive review process and distributed the draft evaluation report to stakeholders. After those consultative meetings, there were proposed recommendations that the evaluation team believes are germane. These include: Strengthen support for community-based postnatal care, PMTCT, TB, and nutrition services in selected sites and document experience. Hold follow-up discussions with organizations and groups working in urban governance, including municipal offices, urban development agencies, and social welfare bureaus, and local universities to identify possible areas of coordination and synergy with respect to urban health issues. Conduct a baseline study on health service utilization and coping strategies for health needs by income quintiles to better understand urban health dynamics and the impact of the UHEP. Promote public-private partnerships for urban health, including improving sanitation and waste management services and the use of market principles and techniques for community health services. Develop a vulnerability assessment tool for the urban centers to determine who is vulnerable, where they live, and how they cope with health challenges. Feed that information into the urban environment checklist. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 31 VII. CONCLUSIONS Based on these findings, the following are the team’s key conclusions: THE ROLE OF USAID AND ITS CONTRACTORS USAID support is an important contribution to the GOE’s urban health development plan and should continue with future programming. JSI has done what it was contracted to do in support of the UHEP and has contributed to the development of the program as well as successfully piloting and trialing approaches. There are more opportunities for USAID collaboration with other donors who are funding municipal service improvements, such as the World Bank and GiZ. URBAN CONTEXTS Insufficient attention is paid to the urban context, and there is still a tendency to view all urban areas as similar, whereas in reality, they are very different in size, density, complexity, population dynamics, services, and physical location. For example, the health bureau staff in Shashamene could not emphasize enough that their challenges were related to the fact that every day more than 30,000 people come through the city, due to its location on five roads. A program for Addis Ababa will be different from other urban programs because of the nature of Addis as a primary city. Variations in Ethiopian urban contexts were not taken into account during adaptation from the rural program, and as such there were missed opportunities to provide appropriate public health services to groups, such as those who are very poor, live in marginal urban areas, are not in a household, or are in more affluent households. Improving urban health is not a function of the MOH alone. Without support from the different government departments and from allies outside the government, it will be difficult to realize the desired improvements, particularly in environmental health, sanitation, and hygiene. SUPPORT FOR AN OPERATIONAL BUDGET At present, there are no budget allocations for the program other than salaries. While this program was initiated by the federal MOH, it was evident that financial resources for the routine operation of the UHE-Ps have not been fully planned. Budget allocations for equipment, transportation, and communication tools and access to supplies, such as FP products and vaccines, are not routinely available now. SERVICE DELIVERY Since the UHE-Ps fully intend to continue to give services, they are in need of training and clinical skills refresher courses according to identified needs. The current implementation of the program has successfully reached TB and ART defaulters in selected cities and has contributed to an uptake of delivery at health facilities, but there are missed opportunities, including support for mothers and newborns in the immediate postpartum, PMTCT, nutrition counseling, and more intensive support for HIV/AIDS. 32 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM The current program does not really use the multiplicity of media channels available in urban settings to broadcast its messages. UHE-P RETENTION AND RECRUITMENT The UHE-Ps needs better job descriptions, clear career ladders, standardized approaches to community outreach, and hands-on technical support to keep the current workforce engaged. Regardless of their deployment station, the UHE-Ps will have to be a part of the primary health care service at health centers. Supervisors do not have the training they need to function as envisioned and to provide supportive supervision that will retain the professionals. The sustainability of the program is of concern because of the extensive demands being placed on the UHE-Ps and the real risk of burn-out. The UHE-Ps indicated that the current catchment area size exceeds their coverage ability and the household to professional ratio is too large. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 33 APPENDIX A. FINAL SCOPE OF WORK THE EVALUATION OF THE URBAN HEALTH EXTENSION PROGRAM GH Tech Bridge Project Contract No. AID-OAA-C-12-00004 SCOPE OF WORK (Final: March 9, 2012) I. TITLE: USAID/ETHIOPIA: ETHIOPIA URBAN HEALTH EXTENSION PROGRAM (UHEP) FINAL EVALUATION Contract: GH Tech Bridge Project II. PERFORMANCE PERIOD: Work is to be carried out from mid March 2012 until mid May 2012, depending on the availability of the selected consultants. In-country fieldwork should begin by o/a March 26, 2012, and be completed by o/a April 13, 2012. III. PURPOSE OF THE EVALUATION: The USAID-funded Ethiopia Urban Health Extension Program (UHEP) (Cooperative Agreement No. 663-A-00-09-00428-00) was awarded for three years to John Snow, Inc. (JSI) and is scheduled to end September 30, 2012. The purpose of the final evaluation will be to assess project performance against documented objectives and targets, identify best practices, gaps and challenges to the UHEP, and provide recommendations to the Mission and the Government of Ethiopia (GOE) on future programming to bring about wider impact on the health system in the context of urban health care. IV. BACKGROUND: Country Context Ethiopia has a population of approximately 80 million and is the second most populous nation in sub-Saharan Africa. The Single Point HIV Prevalence Estimate places the adult prevalence rate at 2.4% for 2010, while the corresponding rate in urban populations is 7.7% (Ministry of Health, 2010). In Ethiopia, the highest concentrations of people living with HIV/AIDS are in the urban areas. As a result, most new infections emanate from cities and more populous towns (Ethiopian DHS, 2005). In addition the urban areas of Ethiopia are under-resourced to address the health needs of the approximately 12 million people residing there or the stream of people migrating to urban areas seeking health care and employment. Shortages of health care workers, overcrowded facilities, perceptions of quality of care, knowledge, and difficulties with transportation lead to poor access to services and result in poor uptake of facility-based HIV/AIDS prevention care and support services. Low accessibility of services in urban areas, both perceived and real, and the resultant lack of uptake or delayed uptake of services lead to increased rates of preventable disease. Six percent of women give birth in a health facility and less than 34% of expectant mothers have received 34 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM even a single antenatal care visit in a health facility (EDHS 2011). Surveillance and epidemiological evidence reveal that most-at-risk-populations (MARPs), including commercial sex workers, migrant laborers, and other marginalized populations, are the drivers of the HIV/AIDS epidemic in Ethiopia. Urban areas contain the largest populations and high concentrations of MARPs. Data from the 2005 EDHS show that among urban women who had sexual intercourse with a non-cohabiting partner in the past 12 months, about 40% reported using a condom. Among the subset of younger women and men (15–24 years of age), the reported condom use was 28 and 40 percent, respectively. Due to high levels of stigma and low levels of income, these populations have some of the lowest rates of health knowledge and access to health services. UHEP Background In 2009, in order to address the health services crisis and the HIV/AIDS epidemic in urban Ethiopia, the GOE introduced a new skilled and rapidly deployable cadre of health workers, the Urban Health Extension professional (UHE professional). Using the rural community-based Health Extension Program (HEP) as a model and building on the successes of this program, the Urban Health Extension Program (UHEP) was envisaged and designed to improve access and equity to basic health services in urban areas. Central to this approach is the expansion of physical health infrastructure and training and deploying UHE professionals. The foundation of UHEP is the premise that availing the right knowledge and skills to households will lead to adoption of positive behavior and, ultimately, improved health outcomes. To this end, UHE professionals select, train, and monitor model households to produce better health and improve health outcomes. Overall, it is anticipated that Ethiopia will require approximately 5,200 UHE professionals to cover all urban areas in the country. The primary difference between the urban and the community-based health extension program is that the urban health extension professionals are diploma nurses. Upon selection for the program, the nurses receive three months of UHEP specific training. The fundamentals of UHEP are the same as the rural program with training provided in the 16 health extension packages. In addition, UHE professionals are expected to cover mental health, non-communicable diseases, and accident and injury prevention and provide home-based HIV counseling and testing. Each UHE professional is expected to cover 500 households when fully operational. Rather than work from health posts, they work from health centers. However, space constraints at the health centers mean some work from the woreda health office. The goal of the Ethiopia Urban Health Extension Project (USAID/UHEP) is to support, at scale, the implementation and monitoring of the Government of Ethiopia’s Urban Health Extension Program (UHEP) and to improve access to and demand for health services. USAID/UHEP also contributes to the overall strategic goals of the US Government (USG) President's Emergency Plan for AIDS Relief (PEPFAR) program in Ethiopia and, specifically, its emphasis on vulnerability. In order to achieve the desired outcome, the program has set forth four objectives organized as the following Expected Results (ER): Expected Results (ER) of the UHEP ER 1: Improved access and decreased barriers to public health services for at-risk populations through engagement of households and communities. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 35 ER 2: Increased demand for public health services through active engagement of at-risk populations, households, and communities using BCC for health prevention, promotion, and risk reduction. ER 3: Improved quality of UHEP service delivery through training and professional development of UHE professionals on public health, including HIV services for at-risk populations. ER 4: Support and enabling environment to implement a sustainable UHEP including support for development and implementation of community information systems to support a multi￾sectored HIV/AIDS and Health program. Geographic Coverage USAID/UHEP supports Ethiopia’s UHEP implementation activities in five regions and two city administrations, covering 19 towns and cities: Tigray Region (Adigrat, Axum, Mekelle, and Shire) Amhara Region (Bahir Dar, Debre Markos, Dessie, and Gondar,) Oromia Region (Adama, Jimma, Nekemte, and Shashamane) SNNPR Region (Arba Minch, Hawassa, Hossana, and Wolayita Sodo) Harari Region (Harar Town) Dire Dawa Administration (Dire Dawa) Addis Ababa City Administration (Arada and Yeka sub-cities). V. OBJECTIVES AND EVALUATION QUESTIONS: The purpose of the final evaluation will be to assess project performance against documented objectives and targets, identify best practices, gaps, and challenges to the UHEP, and provide recommendations to the Mission and the Government of Ethiopia (GOE) on future programming to bring about wider impact on the health system in the context of urban health care. From the perspective of USAID, the support for UHEP was to focus largely on activities that are designed to reach most-at-risk-populations (MARPs) in the context of HIV/AIDS. The vision of the Government of Ethiopia, however, is to cover all major public health concerns in urban settings through the use of UHE professionals, including increasing demand and access for benchmark public health services (e.g., ANC, institutional delivery, immunization, TB care, nutritional support, and HIV services), improve household hygiene and community sanitation, improve health-seeking behavior and community ownership in the production of community￾level health outcomes. Thus, there is wide disparity between the GOE’s perspective and vision with respect the UHEP and what USAID/PEPFAR is supporting. One of the major objectives of this evaluation will be to determine the extent that the current support for UHEP has benefited the overall health system in the context of urban health care. 36 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Evaluation Objectives and Questions. The evaluators are required to address each of the three objectives and corresponding evaluation questions, which can be refined in collaboration with USAID/Ethiopia during the team planning calls and meeting prior to data collection. 1. Assess the performance of the project in achieving its four expected results and corresponding objectives and targets and identify areas of success and challenges in the implementation. a) To what extent has each of the goals and targets been met? What are the main reasons/factors for exceeding or not meeting expected results? b) Have there been any management issues or problems during the project and, if so, how have they been resolved? c) How well has the partner monitored and evaluated the outputs and outcomes of the program and the extent to which the results are achieved? How can the M&E system be improved in the future for the UHEP? d) What are the key lessons learned from the current UHEP? To what extent has the partner communicated project successes and challenges and disseminated lessons learned to the GOE and other key stakeholders? 2. Identify and document the outcomes, strengths and challenges of the UHEP from the perspective of the UHE professionals, GOE, and other key stakeholders. a) How do the UHE professionals perceive their role as a community health agent? What are the incentives and disincentives to staying in the job? What are their perceived strengths and challenges? What do they recommend for strengthening their role? b) What are the key health system issues that have impacted the UHEP? (HMIS? supply chain? HRH? Health care financing? ) c) What are the priority urban community health needs based on the experience of the UHEP to date? d) What do government officials (city, regional, and federal levels) and other stakeholders perceive as priorities for the UHEP over the next five or so years? 3. Identify opportunities and make recommendations for future USAID/Ethiopia support for the UHEP for the next five years. The recommendations should address revisions in the current UHEP framework in order to: a) Build and promote a broader integrated urban health plan based on community determinants of health. b) Create opportunities for focused demonstration projects to comprehensively address priority community health needs (Sanitation, mental illness, etc.) c) Create opportunities for focused demonstration projects to address key health system barriers though innovative approaches (e.g., addressing information management USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 37 including continuous health surveillance data and reporting, community-facility referral networks, etc.) d) Create strategic partnership with other international donor groups to advance broader integrated urban health planning. e) Address sustainability issues of the future of the UHEP and appropriate amount and timeframe for USAID financial contribution to the program. VI. METHODOLOGY: The final methodology will be developed by the team once the evaluation questions have been refined and in collaboration with USAID prior to any in-country evaluation work. The following are illustrative methods. Team Planning Meeting (TPM): The assignment work will commence with a two-day Team Planning Meeting (TPM). This meeting will allow the team to meet with the USAID/E staff to be briefed on the assignment. It will also allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will clarify team members’ roles and responsibilities; review and develop final survey questions; review and finalize the assignment timeline and share with USAID; develop data collection methods, instruments, tools, guidelines, and analysis; review and clarify any logistical and administrative procedures for the assignment; establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion; develop a preliminary draft outline of the team’s report; and assign responsibilities for the final report. Document and Literature Review: Including but not limited to the following: Ethiopia’s Global Health Initiative Strategy USAID UHEP program description USAID UHEP Cooperative agreement GOE Health Sector Development Program (HSDP IV) HAPCO SPM II (Strategic Plan for Intensifying Multi-sectoral HIV and AIDS Response in Ethiopia) Semiannual performance report (SARS) Interviews and site visits: Including but not limited to the following: Project staff at JSI Federal MOH and Regional Health Bureau staff (Tigray/Mekele, Amhara/Bahir Dar, SNNPR/Hawassa, and Oromia/Addis) City Health Office staff (Gondar, Sheshemane, Dire Dawa, Harare, and Arada and Yeka subcities of Addis) Interviews with selected Urban health professions in the 9 cities listed above 38 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Interviews with key informants at USAID mission Selected other donors and stakeholders (e.g., UNICEF, Gates Foundation) USAID/E will provide a contact list of key informants and a draft schedule of site visits to the consultants prior to arrival in-country. VII. TEAM COMPOSITION, SKILLS, LOE Team Composition: The four-person evaluation team should be comprised of one or two international (expatriate) consultants and two local Ethiopian consultants in addition to USAID/Ethiopia staff. One of the international consultants can be from USAID/Washington or another mission. 1. The Team Leader will be an international consultant with more than 10 years of experience in international health (Africa experience required) and experience leading at least two external performance evaluations. Strong writing, evaluation methods, and analytical skills required (a writing sample will be requested). Experience with USAID programs and PEPFAR desired. The Evaluation Team Leader will be responsible for team coordination and performance, and for ensuring the timeliness and quality of deliverables. The consultant will hold conference calls with the other team member and USAID/E representatives before and after the visit to Ethiopia in order to develop the evaluation methodology and take the lead in developing the evaluation report for USAID/E. The team lead is expected to present preliminary findings of the evaluation prior to USAID/Ethiopia and JSI staff as well as key stakeholders prior to departure of in-country work. 2. Two Urban Health Specialists are requested (one international and one local Ethiopian) with experience in urban development, preferably in the context of improving equity to basic health services. The consultants will have extensive experience in program evaluation and knowledge in conducting surveys, key informant interviews, focus groups, and demonstrable skill in assessing health systems is essential. Strong writing skills required. One of the international consultants can be from USAID/Washington or another mission with experience in similar programs. 3. A local logistics coordinator will assist in the evaluation preparations and implementation, including making hotel reservations for the team members; scheduling key informant interviews and focus group discussions; and organizing field visits and all associated in￾country travel. The logistics coordinator should be available to start planning for the evaluation prior to the external team’s in-country arrival. USAID/Ethiopia project AOTR and others may participate in selected meetings and site visits but will not jeopardize the objectivity of the external evaluation. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 39 Selection Criteria for Team Members (Maximum 100%) distributed as follows: Criteria Maximum Points Advanced degree in public health or related social science field 10 International health experience—preferably Ethiopia and/or Horn of Africa experience 15 Knowledge and skills in urban health issues and programs, especially the use of extension workers to improve access to and demand for health services for vulnerable populations and HIV/AIDS programs 25 Knowledge and skills in training and capacity building of health workers 25 Evaluation experience/Qualitative and quantitative methods and analysis skills 25 Fluency in English is required for all team members. A writing sample will be requested of finalists. Team lead must provide examples of past evaluation reports. The desired start date for the in-country work is late March 2012 and includes three to four weeks spent in-country. Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort and proposed timing for each task. Illustrative Activities and Estimated Level of Effort Activity Person(s) Responsible Total LOE Period of Performance (illustrative depending on start date) Mission sends background documents to team members. USAID/E Mid March Review of documents and initial drafting of evaluation protocol and instruments. Team planning call with USAID/Ethiopia. Logistics coordinator prepares for survey. All 5 Mid March Travel to country. International consultants 2 March 24–25 In-briefing with USAID, TPM to finalize protocol and survey tools and methodology for data collection; clarify team roles, meet with key stakeholders to finalize tools, organize logistics for field work. Eval team 2 March 26–27 Conduct Addis-based interviews and field work (including travel days). Mtgs with FMOH, RHB Oromia Group 1: Addis subcities, Bahir Dar, Godar, Mekele Dire Dawa/Harare, Hawassa, Shahamane Eval team 11 March 28–April 7 40 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Activity Person(s) Responsible Total LOE Period of Performance (illustrative depending on start date) Preliminary data analysis and synthesis; drafting report and presentation materials with additional follow-up meetings as needed in Addis. Debriefing of mission staff with draft findings and recommendations. Eval team 4 April 9–12 Team departs country. International consultants 2 April 13–14 Draft report writing. (Team Leader=5;/Urban Health Specialists=4) Eval team 5/4 April 16–20 Draft report submitted to Mission. Eval team April 23 Mission sends technical feedback/comments on draft to team leader. USAID/E April 30 Draft report revisions (Team Leader=5;/Urban Health Specialists=3). Eval team 5/3 April 30–May 4 Revised report submitted to Mission. Eval team May 7 Mission approves report. USAID/E May 14 Total LOE = Team Leader (1) 36 Total LOE = Urban Health-International (1) 33 Total LOE = Urban Health – Local (2) 29 Total LOE = Local Logistics Consultant (1) 20 Travel over weekends may be required during site visits. A six-day work week is approved while in-country. Note: Friday April 13 is Ethiopian Good Friday and all GOE and U.S. Government offices will be closed. Consultants need to complete work by Thurs April 12. Debriefing should take place on Wed April 11 to ensure availability of staff and stakeholders. VIII. DELIVERABLES Based on the above-stated purpose, objectives, and key tasks, the consultant team will submit the following deliverables: Evaluation framework including revised evaluation questions, detailed approach/methodology to be used including the documents to review, key informants to interview, sampling frame, survey protocols and instruments, and plans for analysis and dissemination of findings. The team leader will submit the evaluation framework to USAID/E upon completion of the in-country TPM. The evaluation framework must be finalized and approved prior to the initiation of the interviews and site visits. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 41 Debriefing to Mission and Presentation slides (in MS PowerPoint) used debriefing to HAPN staff on the preliminary findings and recommendations that addresses each of the three objectives and associated questions. The debriefing will take place at the mission prior to departure. Draft report in English no longer than thirty pages with an executive summary, introduction, methodology, findings, and recommendations that address each of the three objectives and subsequent questions with bibliography and annexes. The team leader will submit the first draft report to USAID/E and GH Tech Bridge within 10 days of departing the country. The Mission will provide consolidated, written comments to the evaluation team and GH Tech Bridge within 5 working days of receiving the draft report. Final report will address the Mission’s comments. The team leader will submit the final unedited report to USAID/E within 5 working days after the team receives consolidated comments from USAID/E. The contractor will provide the edited final DRAFT document approximately 5 working days after USAID/E provides final approval of the content. The report needs to follow the standardized report format and meet the quality requirements provided by the mission before final approval will be given. IX. USAID/ETHIOPIA CONTACT PERSONS Kristin Saarlas Evaluation Coordinator Health, AIDS, Population & Nutrition Office USAID/Ethiopia ksaarlas@usaid.gov Faris Hussein Private Health Sector Specialist Health, AIDS, Population & Nutrition Office USAID/Ethiopia fhussein@usaid.gov 42 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 43 APPENDIX B. PERSONS CONTACTED ETHIOPIA U.S. Agency for International Development Kris Saarlas, Monitoring and Evaluation Specialist Dr. Faris Hussein, Technical Advisor Federal Ministry of Health Dr. Negihist Tesfaye , Director, Urban Health Promotion and Prevention Directorate John Snow Incorporated (JSI) Samuel Yalew, Chief of Party Woldemariam Hirpa, Regional Program Coordinator, Shashemenie Berhanu Bekele, Program Officer, Shashemene Shemshedin Omer, Regional Program Coordinator, Harrar Matiyos Kebede, Regional Program Coordinator Hawassa Africa Mulugeta, Regional Program Coordinator, Dire Dawa Askual GebreHiwot, Regional Program Coordinator, Tigray Tewodros Yemane, Urban Health Extension Program Officer, Tigray Saba Amdework, HIV Advisor Fisseha Eshetu, UH System Advisor Worku Berhe, M&E Advisor Bekalu Tesfa, Regional Manager Zelalem Adugna, Deputy Chief of Party Eftu Ahmed, Technical Manager Allison Molenda, Operations Director Bisrat Kurabachew, Finance and Admin Officer Blen Dender, Operation Assistant Addis Ababa Dr. Getachew Teshome, Head of Health Promotion and Disease Prevention and Deputy Head of BoH Zertihun Emiru, Woreda Health Office, Yeka Sub-City Girma Sissay, UHEP Supervisor Askual Demissie, UHEP Supervisor Befikadu Bekele Tamene Achamo, Head, Bole SC, Woreda 4 Health Office Daniel Admassu, Supervisor, Bole SC, Woreda 4 Health Office 44 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Muluken Kassa, Supervisor, Bole SC, Woreda 4 Health Office Girma Berhanu, Coordinator, Bole SC, Woreda 4 Disease Prevention and Control Process Sr. Omega Garuma, UHE Professional Sr. Hirut Tesfakiros, UHE Professional Sr. Mehret Gizaw, UHE Professional Sr. Yordanos Wolde Yesus, UHE Professional Sr. Haimanot Debele, UHE Professional Sr. Fikirte Alemayehu, UHE Professional Seble Fegissa, Head, Arada SC, Woreda 3 Health Office Eyerusalem Kassa, Health Extension Supervisor, Arada SC, Woreda 3 Health Office Sr. Tigist Dubale, UHE Professional Sr. Medina Omer, UHE Professional Sr. Alem Belachew, UHE Professional Sr. Meseret Gudeta, UHE Professional Sr. Misrak Tadesse, UHE Professional Ashrafedin Yuya, UHEP Officer, AAHB Yemane Gebre Yohannes, Officer, Arada SC Dire Dawa Mustafa Mohammed, Health Promotion and Disease Prevention Core Process Owner, Regional Health Bureau Ibrahim Yassin, Health Extension Program Officer, Regional Health Bureau Sr. Nesib Kelil, UHE Professional, Sabian Sr. Mahider Million, UHE Professional, Sabian Sr. Hiwot Debebe, UHE Professional, Sabian Sr. Hipitu Tolera, UHE Professional, Sabian Sr. Mekdes Habte, UHE Professional, Sabian Sr. Mesay Adane, UHE Professional, Sabian Sr. Meskerem Degefa, UHE Professional, Sabian Dejene Kemeni, UHEP Supervisor Mekonnen Kassaye, Head of Health Center Harrar Ato Afendi Basha Sherif, Vice Head, Harar Regional Health Bureau Sr. Jewahir Mohammed, Manager of Urban Health Program, Health Promotion Bureau Saba H. Meskel, Associate Dean of Center of Health Studies, Haromaya University Orit Imran, Supervisor, Aboker Woreda Health Coordinator Kaddir Ahmed, Head of Aboker Health Center USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 45 Muhidin Redi, Department Head, Health Services Sr. Fikirte Teka, UHE Professional, Kebele 11 Sr. Yetinayet Tadesse, UHE Professional, Kebele 12 Sr. Tibeka Nigusse, UHE Professional, Kebele 11 Sr. Hilina Abebe, UHE Professional, Kebele 11 Sr. Yemisiratch Tadesse, UHE Professional, Kebele 13 Sr. Birenael Biruk, UHE Professional, Kebele 12 Sr. Beza Abate, UHE Professional, Kebele 12 Sr. Elisabeth Bejirond, UHE Professional, Kebele 12 Sr. Feven Mekonnen, UHE Professional, Kebele 13 Bahir Dar Zenebetch Chekol Gelaw, Supervisor, Bahir Dar Health Center Aschalew Tsegaye, Head, Bahir Dar Health Center Alemaw Workie, Health Extension Program Officer, Amhara Regional Health Bureau Melesew Chanyalew, Health Extension Program Officer, Amhara Regional Health Bureau Dereje Mesfin, Health Extension Program Officer, Bahir Dar Administration Health Office Kidist Melke, UHE Professional, Bahir Dar Tigist Shitahun, UHE Professional, Bahir Dar Fikir Debu, UHE Professional, Bahir Dar Tigist Degefaw, UHE Professional, Bahir Dar Lemlem Alkie, UHE Professional, Bahir Dar Tarike Yohannes, UHE Professional, Bahir Dar Ketsela Mengistu, Resident, Bahir Dar Wubayehu Esayas, Resident, Bahir Dar Mulu Tegegnhu, Resident, Bahir Dar Woreta Mogus Asmare, Head, Woreta town Health Office Beletu Tadesse, Health Extension Program Officer, Woreta town Health Office Eskedar Tadesse, UHE Professional, Woreta town Mekele Girmay GebreSelasse, Head, Mekele Health Office Sr. Leberhan GebreMeskel, Deputy Head, City Health Center Assefa Meruts, Head, Wukra Town Health Office Gebre Georgis Atnafu, Supervisor, Wukra Town Health Office Yalem Tsegay, Deputy Head, Tigray Regional Health Bureau 46 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM Mulugeta Cherinet, Health Extension Program Coordinator, Tigray Regional Health Bureau Tadesse Alemseged, Coordinator, Health Promotion and Disease Prevention Program, Mekele Health Department Woyni Bereshet, UHE Professional, Mekele Abrehet Gebre Michael, UHE Professional, Mekele Gunnet Mulu, UHE Professional, Mekele Mitslal Taeme, UHE Professional, Mekele Silas Zeru, UHE Professional, Mekele Rahel Hailay, UHE Professional, Mekele Askual Amanuel, UHE Professional, Mekele Tsegi Beyene, UHE Professional, Mekele Asegedetch Reda, Resident, Kebele 18, Mekele Aynalem Beyene, Resident, Kebele 18, Mekele Terhas Berhane, Resident, Kebele 18, Mekele Tsega GebreChirkos, Resident, Kebele 18, Mekele Mekebeb Abadi, Resident, Kebele 18, Mekele Letehawarya Tesfaye, UHE Professional, Wukro Shishay Surafel, UHE Professional, Wukro Hawassa Ato Kare Chawicha Debessa, Head of Regional Health Bureau Desto Dogiso Dagne, Head of City Health Department, Core Process Coordinator Tesema Jomba, UHEP Supervisor Yonas Medebu, UHEP Supervisor Berhanu Getachew, City Health Department, Disease Prevention and Promotion Officer Worishil Emiru, UHEP Supervisor Sr. Lydia Chaka, UHE Professional, Aderi Kebele Sr. Edilishet Assefa, UHE Professional, Andinet Kebele Sr. Tsion Eyob, UHE Professional, Aderi Kebele Sr. Genzeb Tumuro, UHE Professional, Harrar Kebele Sr. Tigist Eyasu, UHE Professional, Harrar Kebele Begashaw Dabana, Coordinator, Hawassa Disease Prevention and Health Promotion Core Process Shashemene Wudinesh Boru Katep, Process Owner of Health Office Fitsum Wasihun, Town Health Office, UHEP Focal Person Sister Genet Kassahun, former Process Owner of Health Office USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 47 Birhanu Cheneke Korma, UHEP Supervisor Zekditu Wakuma, EPI Focal Person at Health Center Riyad Umer, UHEP Supervisor Firomsa Mofuma, Health Center Staff Sirmeberat Gemechw, Head of Health Center Afework Assefa, Deputy Head, Woreda Health Office Tsehai Siraj, Coordinator, UHEP Sr. Genet Kassahun, Expert, Communicable Disease Control Sr. Elisabeth Lemma, UHE Professional, Halelu Kebele Sr. Bezawit Worku, UHE Professional, Abosto Kebele Sr. Urji Dulle, UHE Professional, Abosto Kebele Sr. Gisho Keniso, UHE Professional, Halelu Kebele Sr. Gelgele Qumbi, UHE Professional, Abosto Kebele Sr. Tigist Yigezu, UHE Professional, Halelu Kebele Sr. Ensete Jemaneh, UHE Professional, Abosto Kebele Bishoftu Afework Assefa, Deputy Head, Woreda Health Office Tsehai Siraj, Coordinator, UHEP 48 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 49 APPENDIX C. BIBLIOGRAPHY Abdella, Ahmed. “Maternal Mortality Trend in Ethiopia.” Ethiopian Journal of Health Development, 24, Special Issue I, 2010. Abosse, Zeine, Woldie, Mirkuzie and Ololo, Shimeles. “Factors Influencing Antenatal Care Service Utilization in Hadiya Zone.” Ethiopian Journal of Health Science, 20 : 2. (Addis Ababa, Ethiopia, July 2010. Adam, Lishan. A Survey of ICT Access and Usage in Ethiopia: Policy Implications., ICT Policy Brief No. 1, 2008.researchictafrica.net Central Statistics Agency and Measure DHS, ICF Marco. Ethiopian Demographic and Health Survey 2011: Preliminary Report. Addis Ababa, Ethiopia and Calverton, MD: Central Statistics Agency and Measure DHS, ICF Marco. (Undated). Federal Democratic Republic of Ethiopia Ministry of Health. Health Sector Development Program IV, 2010/11–2014/15. Addis Ababa, Ethiopia: Federal Democratic Republic of Ethiopia Ministry of Health, October 2010. Federal Ministry of Health–Ethiopia. National Comprehensive PMTCT/MNCH Training Package: Reference Manual. Addis Ababa, Ethiopia: Federal Ministry of Health–Ethiopia, December 2011. GH Tech. USAID/Ethiopia External Mid-Term Evaluation of the HIV/AIDS Care and Support Program (HCSP). Washington, DC: Global Health Technical Assistance Project, December 2009. Girma, S., G/Yohannes, A., Kitaw, Y., Ye-Ebiyo, Y., Seyoum, A., Desta, H. and Teklehaimanot, A. “Human Resource Development for Health in Ethiopia: Challenges of Achieving the Millennium Development Goals.” Ethiopian Journal of Health Development: 21 (3): 216- 231, 2007. Haramaya University. Harrar Bulletin of Health Sciences. Harar, Ethiopia: Haramaya University, July 2011. Jabeen, M., Yakoob, M., Imdad,A. and Bhutta,Z. Impact of Interventions to Prevent and Manage Preeclampsia and Eclampsia on Stillbirths. BMC Public Health, 2011 JSI. “Ethiopia Urban Health Extension Program.” USAID/UHEP Newsletter, September 2010. JSI. Contributing to Healthy Homes in Ethiopia. USAID. (Undated). JSI. Ethiopia Urban Health Extension Program and Federal Ministry of Health Supplemental Activity: Annual Workplan for Year III/ Fiscal Year 2012. Addis Ababa, Ethiopia: JSI, November 2011. JSI. Ethiopia Urban Health Extension Program: HIV Vulnerability and Risk Assessment Training, Facilitator’s Manual. Addis Ababa, Ethiopia: JSI, March 2012. JSI. PEPFAR Ethiopia In-Country Reporting Systems (IRS): USAID/Ethiopia Urban Health Extension Program Semi Annual Progress Report for FY 2011. Addis Ababa, Ethiopia: JSI. (Undated). 50 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM JSI. USAID Urban Health Extension Program: Cooperative Agreement No. 663-A-00-09-00428-00. Boston, MA: JSI, September 2010. JSI. USAID/Ethiopia Urban Health Extension Program Data Quality Control Checklist : For Use By USAID/UHEP M&E Team to Assess and Verify the Data Reported from the USAID/UHEP Field Team. Addis Ababa, Ethiopia: JSI. (Undated). JSI. USAID/Ethiopia Urban Health Extension Program Supportive Follow Up Visit Checklist: To be Used by USAID/UHEP Staff to Assess Program Performance (Based on Expectations), to Identify Major Limitations and to Support and Provide Targeted Technical Assistance Based on the Assessment Findings. Addis Ababa, Ethiopia: JSI. (Undated). JSI. PEPFAR Ethiopia In-Country Reporting Systems (IRS): USAID/Ethiopia Urban Health Extension Program Annual Progress Report for FY 2011. Addis Ababa, Ethiopia: JSI. (Undated). JSI. Program Brief: An Overview of the Current Health Status, Urbanization and Vulnerability in Ethiopia, the Evolution of the Health Extension Program, the USAID – Supported Urban Health Extension Program (USAID/UHEP) Key Activities, Challenges and Future Considerations. Addis Ababa, Ethiopia: JSI. (Undated). JSI. USAID/Ethiopia Urban Health Extension Program Data Quality Control Checklist : For Use by USAID/UHEP Staff and/or UHEP Staff to Assess and Verify the Data Reported by the UHE Professionals. Addis Ababa, Ethiopia: JSI. (Undated). Koblinsky, Marge, Tain, Frances and Tesfaye, Solomon. “Reducing Maternal Mortality and Increasing Use of Skilled Birth Attendance: Ethiopia and MDG 5.” Ethiopian Journal of Reproductive Health, 4: 1. Addis Ababa, Ethiopia: Ethiopian Society of Obstetricians and Gynecologists (ESOG), December 2010. Misganaw A, Mariam DH, Araya T. The Double Mortality Burden among Adults in Addis Ababa, Ethiopia 2006-2009, Preventing Chronic Disease 2012. 9:110142. Oromia Regional State, Shashemane City Administration Health Office. “Profile of Urban Health Extension Program in Shashemane Town.” Shashemane, Ethiopia: Shashemane City Adminsitration Health Office, August 2011. RAD-AID. Ethiopia Country Report. Chevy Chase, MD: RAD-AID.org. (Undated) Short, J. R., and Pinet‐Peralta, L. M. “Urban Primacy: Reopening the Debate.” Geography Compass, 3(3), 1245-1266, 2009. The Federal Democratic Republic of Ethiopia, MoFED. “Growth and Transformation Plan (2010/11-2014/15), Vol 1. Addis Ababa, Ethiopia, November 2010. “The Global Burden, 4th Diabetes Atlas.” Accessed from www.idf.org on June 6, 2012. “The ICT Landscape in Two African Population Powerhouses: Nigeria and Ethiopia.” Accessed from www.siliconafrica.com on June 6, 2012. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 51 The U.S Ethiopia GHI Team. Ethiopia Global Health Initiative Strategy. The United States Global Health Initiative. Addis Ababa, Ethiopia: The United States Global Health Initiative. (Final approved) UN HABITAT. Ethiopia Urban Profile. Nairobi, Kenya, 2008. United Nations Statistics Division. World Statistics Pocketbook. Accessed from http://data.un.org/CountryProfile.aspx?crName=Ethiopia on April 17, 2012. USAID Ethiopia. Success Story. Paving the Way for Better Health: Sensitizing Communities to New Approaches. Washington, DC: USAID. (Undated). USAID Ethiopia. Success Story: Hope Renewed, Helping Individuals Towards a Brighter Future. Washington , DC: USAID. (Undated). USAID Ethiopia. Success Story: Medical Supplies Increase Urban Health Extension Professionals’ Efficiency. Washington, DC: USAID. (Undated). USAID Ethiopia. Success Story: Sensitizing Communities Boosts Health Program. Washington, DC: USAID. (Undated). USAID Ethiopia. Success Story: Successfully Engaging the Most Vulnerable Populations. Washington, DC: USAID. (Undated). USAID Ethiopia. Success Story: Supporting HIV Prevention Through Education. Washington, DC: USAID. (Undated). USAID. Gender Equality and Female Empowerment Policy. Washington, DC: USAID, March 2012. Vlahov, D. et al., Roundtable on Urban Living Environment Research (RULER). Journal of Urban Health, 88(5) : 793-857, 2011 World Health Organization. The Algiers Declaration: Ministerial Conference on research for Health in the African Region. Narrowing the Knowledge Gap to Improve Africa’s Health. Brazzaville, Republic of Congo: WHO Regional Office for Africa, 2009 World Health Organization. “Our cities, our health, our future: acting on social determinants for health equity in urban settings.”. Final report to the WHO Commission on Social Determinants of Health from the Knowledge Network on Urban Settings (KNUS) 2007. WHO Kobe Centre, Kobe, Japan. Accessed from April 21, 2012 http://www.who.int/social_determinants/publications/urbanization/en/index.html on April 21, 2012. Yemane Ye-Ebiyo et.al. “Study on Health Extension Workers: Access to Information, Continuing Education and Reference Materials.” Ethiopian Journal of Health Development, 21: (3):240-245, 2007 52 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 53 APPENDIX D. INTERVIEW TOOLS Please note that these tools were used to frame the conversation, but as different points were raised in meetings and themes began to emerge, the team felt free to probe and pursue other points. QUESTIONS FOR THE REGIONAL HEALTH BUREAU AND THE FEDERAL LEVEL Interviews will begin with a very short introduction of the team and a statement of the purpose of the evaluation, which is to get insights from the Ministry of Health regarding future health systems directions, needs, and challenges. We will also ask them for a brief overview of how they see the current situation. 1. How do you envision the Urban Health Extension Program links with other development programs in the urban setting, given that Ethiopia is urbanizing very rapidly? 2. Have you experienced some challenges in establishing the linkages between the Urban Health Extension Professionals and the other elements of the health service delivery system? 3. Are you carrying out some initiatives in your implementation of this Urban Health Extension Program that are not found in other areas? To what extent is regional autonomy allowing you to make specific changes in the program based on your own regional needs? 4. What are the potential implications of the expanding urbanization and the emergence of peri-urban settings that may not necessarily be addressed by either the urban health extension program or the rural health extension program? Are there any plans to address the most vulnerable who don’t have fixed housing? 5. How does the Urban Health Extension Program work in terms of governance? Where do regional towns fit, such as the big towns in regions that currently fall under the Agrarian Directorate or Pastoralist Directorate? Do you think the current classification of only three cities under the Urban Health Directorate will need to change in the future to reflect the urban growth? 6. Can you please tell us how the Woman’s Development Army initiative will impact on how the UHEP does its job? 7. Can you tell us about the possible expansion of activities that will include some service delivery options for the UHEP, even though the current thrust is on prevention and promotion? Given their competency, can they play a bigger role in the PHC than first envisioned? 8. Can you tell us what the different program needs might be based on the different size of urban towns? For example, Addis Ababa is much larger than any other town, so the population needs might be really different. 9. Can you tell us a little bit about how you work with the Land Office on the sanitation program? 10. What do you suggest needs to be changed in the future for urban health programs? 54 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM QUESTIONS FOR PARTNERS 1. Could you tell us what you know about the UHEP? What are its main strengths? Can you suggest possible areas for improvement? 2. Can you please tell us if your organization has any activities that support the Urban Health Extension Program? If so, where are these activities taking place and what do they consist of? Do you have any projects in the urban sector? Can you please give us the details on those programs? Do you think there could be intersection between your urban projects and the UHEP funded by USAID? 3. Have you had any experience partnering with USAID in the UHEP? If not, do you envision any such partnership in the future? 4. Tell us about any links you may envision between the UHEP and other development programs in urban areas, given that Ethiopia is urbanizing very rapidly. (Prompt for discussion of development in peri-urban areas.). 5. Tell us about any challenges you have faced in your partnerships with the Ministry of Health or with your primary ministerial counterpart. 6. Can you tell us, from your perspective, what are the biggest health challenges for Ethiopia in the next five to 10 years, particularly in the urban setting? QUESTIONS FOR THE URBAN HEALTH EXTENSION PROFESSIONAL 1. Are you part of the JSI UHEP program? Do you receive training from UHEP or did you get training from UHEP? What other support has the program provided for you? Do you know the staff and have they provided any routine supervision for you? 2. Tell us about your professional training and how long you have been working. Have you worked in places or programs other than the UHEP? 3. How did you come to be a part of the program? (Prompt—Did you join the UHEP because of interest or because you were asked to become a part of the program? Did you have any other work possibilities? 4. Please describe a typical day for us. How do you set your daily work schedule? 5. Can you tell us about your typical client and the major services that you frequently provide? Can you tell us about any problems you have encountered getting access to households to do your work? 6. Tell us about your working relationships with the health systems, such as the health center, hospitals, and other professionals working in facilities? 7. Tell us about the supervision you receive. Is it enough? Do you have any issues with your supervisors? How do you think they best support you? 8. Tell us about where you sit when you need to do your registers and paperwork. 9. Can you tell us about referrals to the health centers and how that works for you? If you have experienced difficulty in a client completing the referral, can you tell us what you did for the follow-up? How is the feedback mechanism working for you? 10. Tell us about your job satisfaction. Where do you see yourself working in the future? Do you think there are more services you could provide in the home because of your training? Can you tell us what you are most proud of and what are the best practices you have been using? USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 55 11. From your perspective as a professional, tell us what you think are future health challenges in the urban setting and how these could be addressed? Please give us some suggestions for what you think would work to address health problems in the urban setting. QUESTIONS FOR JSI 1. You have had a success in the number of UHE-Ps that you have trained. Can you please tell us more about their retention? Have some of the UHE-Ps you trained left the program? If so, can you tell us some of their reasons for leaving? 2. You recently changed your staffing configuration and added a deputy COP position but have left the BCC position unfilled. Can you please give us more detail on this decision and how it supported programmatic management? 3. USAID has asked you to extend your scope twice, with training of the AHOP and also placing 13 advisors to support the MOH. Can you tell us the impact of this on how you were able to implement activities in support of the core objectives? Do you think these additional tasks have fit into your original mandate? Has that significantly impacted your routine activities? 4. What did the mapping of other resources yield in terms of how your program fits together with other donor and government initiatives? 5. Because USAID has not signed on to the International Health partnership and does not comingle funds, do you think the MOH views your efforts as integral to the fourth HSDP? 6. You had consulting support to help rework the definition of “reached.” Can you please tell us more about that process and how the change impacted your reporting of progress towards targets? If this is a PEPFAR reporting indicator, how did your change relate to that? 7. What do you think the role of the UHEP is in sharing information other than HIV/AIDS and a few key messages? Do you think that UHEP provides the right level of intervention to deliver messages for 16 + primary health care initiatives or will this reduce their effectiveness? 8. What are partners that you have worked with closely in the past two years? Do you think that the Church has a role to play in supporting the UHEP? Do you think there are new partners that might be supportive? 9. You have met many of the targets that were set out in your initial contract. There are a few, however, that are underperforming. Can you tell us what has led to that underperformance and what, if anything, you will be able to do in the remaining six months to move forward? If you have an extension until March 2013, what else will you do to meet those objectives? For those areas where you have greatly exceeded the objectives, can you provide information as to why? Was it initial design issues or did certain aspects really take off? 10. You stated that at the beginning of the contract, there was some issue with USAID that resulted in the project being delayed. Can you please provide the history of those events? Can you please tell us your perception of the support and management you received from USAID? 11. In your last report, you listed some of the biggest challenges in going forward. Can you please elaborate on those challenges, particularly in light of how the government is continuing to fully support this program? 12. If there are any lessons drawn from your experience, can you tell us those? 56 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 13. Can you elaborate on best practices that you have been implementing, such as the use of high school students? 14. Can you please explain in greater detail the issue of referral and linkages to the health center? What is the perception of how, and the extent to which, referrals happen, and how does UHEP connect to this? 15. What factors from the risk assessment plan do you use in your forward planning? Can you tell me how this has been used? Who participates in the planning? 16. How do you use the Family Card and the Register of Daily Activities? How, if at all, does this relate to the government’s health monitoring? Did you do the risk assessment after the baseline data was collected? Can you provide greater information on how they were used? 17. Can you please discuss the issue of the tension between being a PEPFAR-funded project, with a focus on MARPS, and the daily activities that seem to have a greater focus on primary health care? 18. Can you please tell us about the task force that you serve on and how it moves the UHEP program forward? 19. Are there any other key points that you would like to raise for discussion? USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 57 APPENDIX E. TABLE OF ACCOMPLISHMENTS14 TO DATE S.No Key indicators by expected result Deliverable targets: The figure here is drawn from progress reports for FY11 and FY1215 Percentage of accomplishments to date16 Expected result I: Improved access and decreased barriers to public health services for vulnerable populations through engagement of households and communities. 1.1 Number of MARPs/vulnerable individuals and groups reached with evidence￾based HIV prevention interventions FY 11 FY 11 FY 12 FY 12 Plan Actual 91% Plan to date Actual to date 48,445 44,177 18,162 16,228 1.2 Number of vulnerable individuals who received counseling and testing services for HIV and received test results FY 11 FY 11 FY 12 FY12 Plan Actual Plan to 257% date Actual to date 4,845 16,704 5,171 9,059 1.3 Number of eligible adults and children provided with a minimum of one care service by UHE-Ps FY 11 FY11 FY 12 FY 12 Plan Actual 165% Plan to date Actual to Date 342 3,594 2,641 1,317 1.4 Number of UHE-Ps and supervisors who developed MARPs￾focused workplan guided by baseline data FY 11 FY 11 FY 12 FY 12 Plan Actual 99.3% Plan to Date Actual to Date 426 541 1,026 901 1.5 Number of vulnerable individuals referred to health and other social services FY 11 FY 11 FY 12 FY 12 Plan Actual 31% Plan to Date Actual to Date 12,111 3,415 1,450 838 14 All data are drawn from published JSI reports, including annual reports and quarterly reports. 15 FY1 covers Oct 2010–Sept 2011, while FY2 covers Oct 2011–Sept 2012. 16 Plan covers the period of Oct 2010–Sept 2012, while accomplishment draws data for Oct 2010–Sept 2011 and one quarter of Oct 2011–Sept 2012, i.e., Oct 2011–Dec 2011. 58 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM S.No Key indicators by expected result Deliverable targets: The figure here is drawn from progress reports for FY11 and FY1215 Percentage of accomplishments to date16 Expected result II: Increase demand for public health services through active engagement of vulnerable groups, households, and communities using BCC for health prevention, promotion, and risk reduction.17 2.1 Number of individual members of community (iddir, mahber…) who received orientation and guidance to be involved in health promotion activities. FY 11 FY 11 FY 12 FY 12 6.0% Plan Actual Plan to Date Actual to date 560 119 1460 0 Expected Result III: Improve quality of UHEP service delivery through training and professional development of UHE professionals on public health, including HIV/AIDS services for vulnerable populations. 3.1 Number of UHE-Ps and their supervisors who successfully completed a pre￾service training. FY 11 FY 11 FY 12 FY 12 Plan Actual 72% Plan to Date Actual to Date 1,582 1,129 130 111 3.2 Number of UHE-Ps and their supervisors who successfully completed an in￾service training FY 11 FY 11 FY12 FY12 Plan Actual 70% Plan to Date Actual to Date 890 654 60 15 3.3 Number of UHE-Ps and their supervisors who attended a refresher training on risk assessment, FHC, and recording and reporting FY 11 FY 11 FY 12 FY 12 93% Plan Actual Plan to Date Actual to Date 1,104 1,030 0 0 3.4 Number of supervisors trained and RHBs18 trained on supportive supervision. FY 11 FY11 FY12 FY12 Plan Actual 92% Plan to date Actual to date 249 224 77 75 17 When they have achieved results in the first year that they consider sufficient, they no longer track these in the second year. Thus when health centers and media personnel were oriented to UHEP, even though it was not in the numbers originally envisioned, they no longer thought it pertinent for the second year of operations and stopped monitoring it formally. 18 Original indicator did not track regional health bureaus but was only focused on supervisors. USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM 59 S.No Key indicators by expected result Deliverable targets: The figure here is drawn from progress reports for FY11 and FY1215 Percentage of accomplishments to date16 3.5 Number of supportive follow-up visits conducted to UHE-Ps. FY 11 FY11 FY12 FY12 Plan Actual 131% Plan to date Actual to Date 1,014 1,565 500 425 3.6 Number of completed referrals by UHE-Ps. FY 11 FY 11 FY 12 FY 12 Plan Actual 28% Plan to Date Actual to Date 3,028 875 1,450 381 Expected Result IV: Support an enabling environment to implement a sustainable UHEP including support for development and implementation of community information system 4.1 Number of review meetings conducted at cities/towns to review progress of UHEP. FY 11 FY 11 FY 12 FY 12 Plan Actual 58% Plan to Date Actual to Date 72 36 20 17 4.2 Number of review meetings conducted at regional level to disseminate project findings and experiences. FY 11 FY 11 FY 12 FY12 Plan Actual 57% Plan to Date Actual to Date 7 4 0 0 4.3 Number of cities/towns where CBIA is piloted. FY 11 FY 11 FY 12 FY 12 Plan Actual 100% Plan to Date Actual to Date 2 2 0 0 4.4 Number of UHEP kits distributed. FY 11 FY 11 FY 12 FY 12 NA NA 100% Plan to date Actual to Date NA NA 897 897 60 USAID/ETHIOPIA: END OF PROJECT EVALUATION FOR THE URBAN HEALTH EXTENSION PROGRAM For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com