External End-of-Project Evaluation: The Ethiopia Family Planning and Reproductive Health Project Final Report August 2008 This publication was produced for review by the United States Agency for International Development/Ethiopia. It was prepared by The Mitchell Group under Contract No. 663-C-00-08-00409-00 End-of-Project Evaluation of the Family Planning Reproductive Health (RH/FP) and Extending Service Delivery for RH/FP (ESD) Projects This document was submitted by The Mitchell Group, Inc. 1816 11TH Street, NW Washington, DC 20001, USA Telephone: (202) 745-1919 Facsimile: (202) 234-1697 To The United States Agency for International Development/Ethiopia under USAID Contract 663-C-00-08-00409-00 This is an evaluation report carried out by team members: Harriett Destler, Team Leader, The Mitchell Group Ababu Beshane, Regional Health Bureau, Oromia Betsy Brown, The Mitchell Group Carmen Coles, Office of Population, Bureau for Global Health, USAID/W Mesganaw Fantahun, Reproductive Health Department, School of Public Health, Addis Ababa University Hailu Kassa, Ministry of Finance, Government of Ethiopia Alemu Kebede Engida, Head, Family Health Department, Regional Health Bureau, Amhara Roy Miller, Office of Sustainable Development, Bureau for Africa, USAID/W Nenko Sorso, Family Health Coordinator, Gedeo Zonal Health Bureau, S.N.N.P August 1, 2008 Disclaimer: The authors’ views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government i ACKNOWLEDGEMENTS The team would like to acknowledge the life saving work, spirit and creativity of all partners working on the family planning reproductive health project. Special thanks goes out to the Ministries of Health and Finance and the four regional ministries, and woreda level health officials who readily granted us full access to programs and important GOE health information. We highly valued our contacts with households and communities who warmly welcomed us and provided us with valuable insights adding considerable richness to this evaluation. ii ACRONYMS, ABBREVIATIONS, AND ETHIOPIAN TERMS ANC Ante Natal Care BCC Behavior Change Communication CBD Community-based Distribution CBR Crude Birth Rate CBRHA Community-based Reproductive Health Agent CHP Community Health Promoters CORHA Consortium of Reproductive Health Associations CPR Contraceptive Prevalence Rate CSM Contraceptive Social Marketing CYP Couple years of protection DOTS Demonstrated On-Site Treatment for Tuberculosis EDHS Ethiopian Demographic and Health Survey ESD Extending Service Delivery ESHE Essential Services for Health in Ethiopia Project FGA Family Guidance Association FGM Female Genital Mutilation (FGM) RH/FP Family Planning/Reproductive Health FP/MNCH Family Planning/Maternal, Newborn and Child Health Program GOE Government of Ethiopia HAPN Health, AIDS, Population and Nutrition Office, USAID HEW Health Extension Worker HEP Health Extension Program HC Health Center HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HP Health Post HSDP III Health Sector Development Plan III HTP Harmful Traditional Practices IEC Information, Education and Communication IPO Implementing Partner Organization IPPF International Planned Parenthood Federation IUD Intra-uterine Device JHU/CCP Johns Hopkins University/Center for Communication Programs Kabele Smallest Administrative Unit in the Ethopian Governmental Structure LAPM Long Acting and Permanent Methods MCH Maternal and child health MOE Ministry of Education MOH Ministry of Health NCTPE National Committee on Traditional Practices of Ethiopia iii PAC Post-abortion care PI/E Pathfinder International Ethiopia RHB Regional Health Bureau SNNP Southern Nations Nationalities and Peoples’ Region TFR Total Fertility Rate UNFPA United Nations Fund for Population Affairs USAID United States Agency for International Development USG United States Government VSC Voluntary Surgical Contraception WAC Woreda Advisory Committee Woreda District YFS Youth Friendly Services ACKNOWLEDGEMENTS .......................................................................................................... i ACRONYMS, ABBREVIATIONS, AND ETHIOPIAN TERMS............................................ ii EXECUTIVE SUMMARY .......................................................................................................... 1 INTRODUCTION......................................................................................................................... 7 Background................................................................................................................................. 7 The RH/FP Project...................................................................................................................... 8 EVALUATION OBJECTIVES.................................................................................................. 9 II. METHODOLOGY .................................................................................................................. 9 III. FINDINGS, CONCLUSIONS AND RECOMMENDATION.......................................... 12 Introduction............................................................................................................................... 12 Major RH/FP Project Accomplishments/Challenges................................................................ 12 Overall ...............................................................................................................................................................12 Strengthening the RH/FP referral system ..........................................................................................................14 Other services and strategies.............................................................................................................................15 Family Planning Use .........................................................................................................................................16 Community-based Approaches................................................................................................. 18 Overview ............................................................................................................................................................18 Implementing Partner Organizations.................................................................................................................18 Strengths and Weaknesses..................................................................................................................................19 Woreda Advisory Committees............................................................................................................................20 CBRHAs ................................................................................................................................... 22 CBRHA/HEW Added Value and Evolving Roles................................................................................................22 CBRHA Outreach to the Community .................................................................................................................24 CBRHA Attrition ................................................................................................................................................24 CBRHAs and Commitment to Voluntarism ........................................................................................................25 CBRHA and HEW Counseling Skills .................................................................................................................26 CBRHA Obstacles and Challenges ....................................................................................................................26 Other RH/FP Issues................................................................................................................... 27 Quality of Care...................................................................................................................................................27 Infection Control................................................................................................................................................28 Supportive Supervision and Reporting...............................................................................................................28 Contraceptive Supplies ......................................................................................................................................29 Method Mix and Permanent Methods ................................................................................................................29 Links to the Expanding Private Sector Contraceptive Social Marketing Program............................................30 Sustainability............................................................................................................................. 31 Financial Sustainability............................................................................................................. 31 Fee-for-service...................................................................................................................................................31 Community contribution.....................................................................................................................................32 Institutional Capacity................................................................................................................ 32 Training of the CBRHAs and the HEWs ............................................................................................................32 Capacity to manage the program.......................................................................................................................32 Enabling Environment .............................................................................................................. 33 The relationship between CBRHAs and HEWs..................................................................................................33 Attitudes.................................................................................................................................... 34 Ability to Pay ............................................................................................................................ 34 RECOMMENDATIONS............................................................................................................ 36 GUIDANCE TO USAID FOR STAFFING AND MANAGEMENT OVERSIGHT AND FUTURE CONTRACTOR WORK PLANS............................................................................ 39 Appendices A: Scope of Work B: Persons Contacted C: Interview Instruments D: Team Field Visits E: References F: Pathfinder International comments 1 EXECUTIVE SUMMARY After a competitive award process, USAID/Ethiopia awarded a contract to an independent firm to perform an evaluation of Pathfinder International/Ethiopia’s (PI/E) reproductive health/family planning (RH/FP) performance under two agreements: the RH/FP Project, October 2002- September 2008, and the Extended Service Delivery (ESD) project, October 2007-September 2008,to make recommendations on which RH/FP activities should be included in the work plan of the USAID follow-on Family Planning/Maternal Newborn and Child Health Program (FP/MNCH). The team consisted of four Americans and five Ethiopians. The team conducted 177 separate interviews which included 46 interviews with community￾based reproductive health agents (CBRHAs); 32 interviews with health extension workers (HEWs); 9 interviews with CBRHA Implementing Partner Organizations (IPOs) and Ministry of Health (MOH) supervisors (9); and 90 clients (Households). The team visited and observed service delivery at 17 health facilities in Amhara, Oromia, Southern Nations Nationalities and Peoples’ Region (SNNP), and Tigray and observed service delivery at these sites. Six other sites were visited but services were not in progress at that time of the day when the team reached the site. This methodology enabled the evaluation team to develop firm impressions of the nature of the program on the ground and learn, first-hand, the opinions of the program from federal level down to the clients receiving services. The selection of kebeles to be visited was done by USAID/E as part of a purposeful sample within the randomly selected woredas: the sample was not scientifically rigorous and does not lend itself to statistical analysis. Also, it is important to recognize that this evaluation is based on a “snap-shot” of the program taken at a single point of time. There was no baseline data and, therefore, no opportunity to collect similar data at the project’s end to measure impact. All of the materials reviewed and interviews conducted, suggested strongly that PI/E had succeeded in meeting the project goals and targets and contributed importantly to the ability of many Ethiopians to plan their families, immunize their children and improve the sanitation of their homes. PI/E’s work with implementing partner organizations, regional and district health programs, communities, health workers and households has been particularly notable in improving access to and use of RH/FP and related maternal and child health (MCH) services. Fundamental to this has been the use of community-based approaches that involve local leaders and community residents to promote RH/FP and, in the case of the Community-Based Reproductive Health Agents, deliver RH/FP and related services. The CBRHAs are multi￾functional volunteers1 . They routinely perform eight distinct tasks: mobilization of communities for the Health Extension Workers, family planning and MCH outreach, family planning education and counseling, distribution of short term contraceptives, support for immunization campaigns and referrals, identification of and public education on potential harmful traditional practices (HTP), epidemic control actions and help for pregnant women in securing ante-natal and postpartum care. The CBRHAs are known members of the community and have been particularly successful in gaining the trust and willingness of households to practice new 1 Because the work of the CBRHAs goes beyond the delivery of just FP/RH services, their title will be changed to Voluntary Community Health Workers (VCHWs) in the follow-on project, FP/MNCH. 2 behaviors. While CBRHAs reach on average 125 clients, some exceptional CBRHAs serve many more with one CBRHA reaching 1,256 households. CBRHAs work between 8 and 15 hours a week with monthly stipends for transportation and meeting support of between 40 and 75 Birr. At a maximum current cost of U.S. $1,000 a year2 including supervision, transport, training and materials, CBRHAs represent a “best buy” and an important Ethiopian resource. With the rollout of the Health Sector Development Plan III, the CBRHAs work closely with the health extension workers. Each supports the other in enabling rural households to access critical primary health care services. These community-based approaches have also tapped into the multi-faceted talents of a diverse set of local organizations and their networks, generating thousands of volunteer hours and significant additional financial resources for the program. PI/E implements its programs by providing sub-grants to 46 partner organizations (IPOs) including the Ministry of Health. Sixteen IPOs receive sub-grants for implementing RH/FP services at the community level. These implement the community-based RH/FP program through three different models. While each model is distinct in the way it is financed by PI/E, the implementation of all three models at the community level is similar. The three models include: Model 1: Local Organization, Model 2: MOH supported IPOs, and Model 3: MOH and Regional Health Bureau. The team found it difficult to discern a difference in the retention and performance of the CBRHAs under the different models. However, the team did find that each model had its strengths and weaknesses vis-à-vis supervision, training and materials, provision of commodities and involvement of civil society. PI/E has also renovated primary health care facilities, ensured a reliable supply of contraceptives and helped marshal community resources to combat harmful traditional practices (HTP) such as early marriage and female genital mutilation (FGM). Over the six-year life of the project, there have been major changes in GOE and donor support for RH/FP and primary health care more generally. The 2005 Ethiopian Demographic and Health Survey documented a dramatic increase in FP use. The available information suggests the number of FP users has continued to increase with more and more users shifting to longer term methods such as injectibles and implants. But the job is far from done with less than 20%3 or 35%4 of the women of reproductive age using contraception and a young5 and growing population which increases by two million a year. The annual increases in women of reproductive age mean that just maintaining current contraceptive prevalence rates (CPR) requires serving hundreds of thousands of new users each year. Not 2 This is expected to be considerably lower in the follow-on project by streamlining processes, phasing out IPO roles in supervision and reporting and some non-essentials like CBRHA uniforms and generating additional community in-kind and financial support. 3 The 2005 EDHS reported a CPR of 13.9% among married women 4 The 2006-7 MOH Annual Statistical Report on Health & Health Related Indicators reports a CPR of 35%. The MOH uses service delivery statistics to estimate the CPR and uses the estimated number of women 15 to 49 who are not pregnant as a denominator rather than the estimated total number of women 15-49 used in the DHS (which is a population based sample). This smaller denominator results in higher estimated CPRs. 5 44% of the population is under 15 years of age 3 surprisingly, there are outreach and service delivery issues involving roles and responsibilities, quality of care and the best use of public and private resources-human, financial and institutional. The report that follows examines these successes and challenges and makes recommendations for USAID and the FP/MNCH work plan on how these opportunities and challenges might be addressed. The team’s recommendations fall into five broad areas: • Build on USAID’s Comparative Advantage in Supporting RH/FP in Ethiopia, • Ensure USAID-assisted programs to maximize support to the MOH, • Retain the CBRHAs as adjuncts to the HEWs in rural areas and continue other activities that encourage civil society involvement but phase out the IPO CBRHA supervisory and reporting functions, • Strengthen FP/RH/MCH service delivery at health centers and health posts, • Continue to test alternatives and learn from best practice. Detailed recommendations and the findings that led to these recommendations are presented in Section III Findings, Conclusions and Recommendations and summarized in Section IV Major Recommendations. USAID has an important opportunity, now, as it launches the new integrated FP/MNCH project to join the GOE, other donors, public and private groups and citizens in making history in the extension and successful adoption of FP/RH and other primary health care interventions needed to protect and save lives. Ethiopia has a strong and progressive population policy with ambitious FP and related primary health care goals. There are many talented Ethiopians with strong health backgrounds who could help roll out the program. There is a very high level of unmet demand especially for long acting and permanent FP methods. The GOE is investing heavily in improving and extending PHC services, particularly to rural populations. There are unprecedented donor resources for FP and MCH. In addition to the support from the Swedish International Development Agency and the UN Agencies, other RH/FP donors include: a consortium of European donors making a $22 million grant to expand contraceptive social marketing; the Packard Foundation whose annual $5 million budget supports community-based FP/RH services, female education and operations research; and an anonymous donor providing approximately $120 million for expanded delivery of permanent FP methods and other RH services, over the next 5 years. Ethiopia will be one of the first 11 countries to receive funding from UNFPA’s new maternal health initiative. USAID, with its extensive experience working in RH/FP in Ethiopia and globally, has important contributions to make in this time of expanded resources and strong national commitment. Seizing this opportunity requires that USAID take a number of important steps now. The first is to pay urgent attention to the issue of transition between the current FP/RH and ESHE projects which end in September 2008 and the newly-awarded FP/MNCH project. It is imperative that there are support mechanisms for those who rely on these projects for their FP/RH and MCH care. The second is to participate actively in formal coordination with the GOE and other donors to insure that greatly augmented RH/FP/MCH resources are used effectively in a synergistic manner. Under Government of Ethiopia (GOE) leadership, USAID and the new project team should continue to be active participants in the coordination of all new initiatives 4 including those involving the “anonymous” donor, the UNFPA Maternal Health Thematic Fund and DKT/Ethiopia’s new expanded contraceptive social marketing project. USAID should be ready to provide any additional technical or other support requested by GOE to ensure effective coordination and sharing of information. USAID and the new project staff should also actively facilitate technical exchanges and coordination at the working level to take advantage of the strengths each party brings to the table. One example is to use the client base already developed in the RH/FP project to identify those who would want to be early acceptors of voluntary surgical contraception. USAID should also help the new contractor integrate FP/RH services fully into food aid and humanitarian assistance programs. One important example would be in SNNP where the team did not see RH/FP services being provided to those receiving supplemental food. USAID, through its new contractor, must ensure the FP/MNCH program fully supports the national Health Sector Development Plan. USAID may need to assign more technical oversight support in the launch of the new project to assure an effective merger and integration of approaches by the Essential Services for Health in Ethiopia Project (ESHE) and existing PI/E projects. The Ministry of Health should designate a focal person to serve on the project’s technical advisory committee. Among the areas that require attention and harmonization are supervisory and reporting systems, provider and client materials, training, the distribution of contraceptives and other supplies and outreach. The evaluation team recommends that IPO supervision and support of CBRHAs and duplicative reporting be phased out. All the CBRHAs should report to HEWs. This would strengthen the existing government system of supervision and support to HEWs and CBRHAs. The IPOs role should be changed to focus on training health workers rather than overall administration of services. Similarly, the current USAID phased approach to merging the USAID-supported contraceptive management and distribution system with the government pharmaceutical distribution system being rolled out so that there are no stock outs should be maintained. Owing to the important national population and health goals and the considerable demand from citizens for safe and effective ways to space and limit their children, it is critical that current successes and momentum be maintained. This requires an “all spigots open” approach. There must be no flagging of effort and what works must be retained while the non￾essential is eliminated. The CBRHAs represent an important national resource that should be retained in the near-term to help rural Ethiopians successfully adopt family planning and related MCH services. The work of CBRHAs in the larger towns needs further evaluation to determine if it should be continued. Other vital CBRHA functions include discouraging HTP such as early marriage and FGM. As adjuncts to HEWs, CBRHAs need to be encouraged to work in the eight critical areas where they now serve their communities as well as serve as model households but not asked to do more. The evaluation team observed that the interplay of the CBRHAs and the other volunteers works very well under the technical oversight of the HEWs and administrative oversight of woreda health officials. Ways need to be identified and tested for communities to provide CBRHAs modest compensation and support for the long hours they currently donate to the program. The team recommends that USAID through the new FP/MNCH project allocate sufficient resources to enable the MOH to 5 strengthen the HEW and woreda health team supervisory oversight of all volunteers and mainstream all recording and reporting within the MOH system. USAID and the MOH will seek to leverage other resources to support the GOE RH strategy. The program may qualify for USAID Title II Section 206 local currency funds to be used for volunteer incentives. Other important ways of involving citizens such as the Woreda Advisory Committees which are led by local government officials but include other members of the community such as religious leaders, women’s groups and agricultural cooperatives need to continue. Every effort must be made to strengthen RH/FP/MCH systems and services at all levels. A major priority is developing the systems, facilities and human resources required to offer Ethiopians more safe choices. Two important ways are broadening the method mix and addressing infection control issues. The contractor must work with other donors and groups to make VSC, the copper-T-380 IUD, the progesterone only pill, and other family planning and maternal health life saving products such as misoprostol available as soon as these products are improved. There is also a need for regular contraceptive technology updates for providers as well as strengthening of the FP/RH pre-service training in medical and nursing schools. The team recommends ending current non evidence-based mass media approaches. Any new information, education, communications and behavior change components need to be strengthened and tested to ensure that the messages are evidence-based and meet specific audience’s needs. Particular attention must be paid to the special information and service needs of adolescents and first time mothers. Ethiopia is a large and diverse country and no single model or message will meet the needs of its various populations. For example, the need for CBRHA support to HEWs may be far more important in some regions and settings than others. Ethiopia has succeeded in unprecedented gains in providing information to a dispersed and largely rural population and increasing contraceptive prevalence. As the program moves into this phase of greatly expanding program activity, it is very important to build upon past experience, mine the considerable data that has already been collected, build strong monitoring and evaluation (M&E) systems and test new approaches or changes with practical field-based research. In general, the PI/E program lacked baseline and end line surveys to report on impacts while the present PI/E data collection systems and reporting are cumbersome and labor intensive. Much of the good data is not used for planning and program management. The new project must develop better program data collection and analysis systems and pay greater attention to monitoring and evaluation. Three distinct actions on the part of the new project could go a long way to providing a remedy or the M&E deficiencies: a) Develop an affordable survey instrument to establish a baseline survey on key indicators in project districts and carry out this survey on a regular basis to measure change; b) Establish an accounting system that can track the flow of project funds and, if possible, any funds leveraged from other donors, to facilitate a cost analysis of the project components; c) As the GOE tests for new ideas for modifying the current program (such as fee-for-service plans or conversion to all-purpose volunteers), systems should be put in place to evaluate those. The report identifies a number of important areas for such applied research. These include, but are not limited to, testing: a) ways that communities can support CBRHAs, b) the value and Deleted: . 6 feasibility of fee-for-service programs, c) alternative approaches to using volunteers and d) the impact of different information and behavior change communication approaches and campaigns. Responses to the draft report from the USAID mission and from Pathfinder International were considered and responded to in the writing of the final report. The full text of the response from Pathfinder International is included in Annex F. 7 INTRODUCTION Background Ethiopia is the second most populous nation in sub-Saharan Africa. With a current population estimated at 77.1 million people, and an annual population growth rate of 2.5%, the population of Ethiopia is projected to increase to 108.7 million by 2025. People under the age 15 currently account for 43% of the total population.6 . Recognizing that such rapid population growth will limit Ethiopia’s ability to meet its development objectives, the Government of Ethiopia (GOE) has recently taken steps to expand family planning (FP) and reproductive health (RH) services. The Government of Ethiopia (GOE) has a national reproductive health strategy which includes ambitious contraceptive prevalence and fertility reduction goals. This forward thinking and unprecedented document makes the links between population growth, food security and economic development. Over the last five years, the Government of Ethiopia has strongly supported family planning as a key element of primary health care. They have liberalized prescribing practices for short-term contraceptive methods, registered a broad product line for socially marketed contraceptives and expanded access to safe, affordable abortions and long acting FP methods. Current RH/FP use is low and varies substantially by a woman’s place of residence, educational level and religion. On average, a woman in Ethiopia gives birth to 5.4 children during her lifetime. The high fertility rate contributes to Ethiopia’s high rates of maternal (673 deaths per 100,000 live births) and child mortality (123 per 1000 live births). Nationwide, only 6% of women receive delivery assistance from a health professional (EDHS, 2005). Early marriage is widely practiced and exposes young women to premature and prolonged childbearing leading to poor health outcomes. While the national modern contraceptive prevalence rate among married women remains very low at 14%, it more than doubled in the past five years (EDHS 2000 and 2005). In the four regions receiving direct USAID support for RH/FP community-based programs, the increase in the prevalence of modern contraceptive use among married women was dramatically higher (two- and three-fold in three regions) compared to Ethiopia’s other seven regions. These results underscore the need to make RH/FP services accessible to the population at the community￾level. The GOE annual report derived from service statistics and contraceptives supplied shows a 35% overall prevalence. To expand access to basic health services, the Ethiopian government has launched a program for the “Accelerated Expansion of Primary Health Care Coverage” with the Health Extension Program (HEP) as its centerpiece. This program calls for the training and deployment of more than 30,000 female health extension workers (HEWs) for more than 15,000 health posts and the construction and upgrading of 3,153 health centers by 2009. This new primary care cadre holds official GOE civil servant positions within the regional Ministries of Health (MOH). 6 Population Reference Bureau, 2007 World Population Data Sheet, August 2007. 8 To date, almost 27,000 of the planned 30,000 HEWs have been trained and deployed throughout the country. HEWs focus on 16 health packages broadly categorized into four areas: family health care; major communicable diseases prevention and control; hygiene and environmental health; and health education/communication. Service delivery at the community and household level includes: Antenatal Care, Immunization, Family Planning and Malaria Prevention and Control. The work of HEWs is currently enhanced through partnerships with other community health volunteers trained with support from USAID, such as Community Health Promoters (CHPs) under the Ethiopia Child Survival Project - Essential Services for Health in Ethiopia (ESHE), and Community-Based Reproductive Health Agents (CBRHAs) under the Pathfinder International/Ethiopia (PI/E) RH/FP Project. These different cadres of volunteers are part of the communities they serve and help communities adopt healthy behaviors. The RH/FP Project In 2002, USAID/Ethiopia funded a $34.2 million dollar six-year cooperative agreement from 09/2002-09/2008 to Pathfinder International/Ethiopia (PI/E) and its partners, the Johns Hopkins University/Center for Communication programs (JHU/CCP) and the National Committee on Traditional Practices of Ethiopia (NCTPE) to support the Ethiopia Family Planning and Reproductive Health (RH/FP) Project. This project serves as the Mission’s centerpiece of USAID/Ethiopia mission’s RH/FP strategy. The mission also has an umbrella child survival program ESHE, which focuses on child survival interventions and health systems strengthening. Over the past six years, USAID has been the lead donor in RH/FP and Child Survival. In the fall of 2008, this may no longer be the case in RH/FP if an U.S. private foundation makes the planned donation of $120 million over five years for the expansion of permanent methods and other reproductive health care services. . In addition to the :”anonymous donor”, other RH/FP donors include a consortium of European donors, making a $22 million grant to support the expansion of contraceptive social marketing, the Packard Foundation with an annual budget of $5 million for community-based RH/FP services and operations research and female education and participation, UNFPA, IPPF and SIDA. The RH/FP project focuses on providing selected RH/FP and MCH services in four regions representing more than 88% of the Ethiopian population: Amhara, Oromia, The Southern Nations, Nationalities and Peoples Regions (SNNP) and Tigray.7 The project has also provided some services in Addis Ababa8 . The backbone of the PI/E project is a broad network of Community-Based Reproductive Health Agents (CBRHAs) who provide RH/FP information (and non-clinical contraceptive methods) at the community level, and make referrals to health facilities. The PI/E RH/FP Project currently supports more than 10,000 CBRHAs in more than 300 woredas (districts). The project objective is to strengthen and increase the capacity of the public sector and local NGOs to provide quality, gender–sensitive reproductive and child health services. 7 Email from Girma Kassie, Monitoring and Evaluation Team Leader, Pathfinder International, Ethiopia to Harriett Destler, June 16, 2008. 8 The evaluation team was not asked to review the work in Addis.Ababa. 9 As the PI/E award ends in September 2008, USAID/Ethiopia has developed a new strategic approach to combine the Ethiopia Child Survival Project - Essential Services for Health in Ethiopia (ESHE), which also ends in September 2008 and the RH/FP Project. USAID has just awarded a new project, which will focus on both FP/RH and MCH including malaria. The new project will combine a community-based platform for the delivery of services with a health systems strengthening approach. The evaluation team observation of the need for both elements confirmed the strategic and technical merit of the new FP/MNCH project. The new project’s integration within the health extension program will work towards greater sustainability of family planning and reproductive health programs and strengthening primary care. EVALUATION OBJECTIVES To make recommendations for RH/FP activities to be included in work plan for the follow-on Family Planning/Maternal Newborn and Child Health (FP/MNCH) Program and to document important lessons learned from the RH/FP project, USAID commissioned an independent, external end-of-project evaluation. The team consisted of nine Ethiopian and U.S.–based experts. The following are objectives of the evaluation: • Make recommendations on how best to sustain and increase sound FP/HR services within the new FP/MNCH Project and assure greater contraceptive security. • Give priority to an in-depth assessment of community-based strategies for delivery of RH/FP services, with an emphasis on the role of the CBRHA within the RH/FP program, the contributions of the IPOs, and the links between health facilities and HEWs. An important consideration is how these community interventions relate to the GOE’s important and considerable investment in improved primary health through its third Health Sector Development Plan. • Assess progress to date in improving access to quality FP services and increased use of modern RH/FP methods, especially long-term and permanent methods including improved access to PAC services. The scope of work for the evaluation team is included in Annex A. II. METHODOLOGY This evaluation exercise was carried out by a team of external evaluators comprised of two United States-based consultants, a representative of USAID’s Bureau for Global Health in Washington D.C., a representative of USAID’s Bureau for Africa in Washington D.C., one professor from the University of Addis Ababa, three representatives of the Ministry of Health (MOH) selected from three of the four regions covered by the PI/E program and one representative of the Ministry of Finance and Economic Development. The evaluation was carried out in six phases: 10 1. Literature review, team planning meetings and interview with informed colleagues in Washington D.C. (carried out by the four Washington D.C.-based team members), 2. Continuing literature review and interviews with key stakeholders in Addis Ababa, 3. Team planning meetings with the full team and the design of questionnaires for data/information gathering during visits to the field, 4. Field visits by four teams to each of the four regions covered by the PI/E program, 5. Data analysis and development of findings, and 6. Report writing and final briefings with PI/E, USAID and the Ministry of Health. The team designed nine questionnaires to guide the field work: Households, CBRHAs, CBRHA Supervisors (usually HEWs), health providers and health centers or hospitals, IPOs, Woreda Advisory Committee (WAC) members, health officers at regional and woreda levels, RH/FP officers at regional and woreda levels, and private pharmacies. The questionnaires covered region-wide, woreda-level, provider and household level user and non-user questions (see Annex C). The team conducted 177 separate interviews with CBRHAs (46); HEWs (32); CBRHA IPO and MOH financed supervisors (9); and Households (90) in the four regions. 10 health facilities were visited and services in those sites were observed. USAID/Ethiopia selected a purposefully￾stratified sample of zones in four regions and randomly selected woredas within those zones to be visited. However, owing to time constraints and logistical realities, the kebeles visited in three of the four regions were selected to assure the presence of the necessary individuals and officials at remote sites. The field visits included direct field observation, structured interviews and formal prepared presentations in the four regions: Amhara, Oromia, Southern Nations Nationalities and Peoples’ Region (SNNP), and Tigray where the PI/E project has focused its activities (See Map of Ethiopia Regions on the next page) In addition to the formal interviews carried out over the ten-day field visit, the evaluation team visited examples of other program components: youth centers, marriage cancellation programs and regional obstetric fistula hospitals, university health centers and two public sector radio stations. (See annex C for the interview instruments.) This methodology enabled the evaluation team to develop firm impressions of the nature of the program on the ground and learn, first-hand, the opinions of the program from federal level down to the clients receiving services. However, given that the selection of kebeles to be visited within the randomly selected woredas was a purposefully selected sample and that the clients for services were in many cases asked to come to the health posts by PI/E to be interviewed by the evaluation team, the sample was not scientifically rigorous and does not lend itself to statistical analysis. And, except for the team that visited SNNP, the teams were not able to observe actual counseling sessions and spoke to very few women who were not already practicing birth spacing or limiting their family size. 11 Map of Ethiopia Regions 12 Also, it is important to recognize that this evaluation is based on a “snap-shot” of the program taken at a single point of time. There was no baseline data and, therefore, no opportunity to collect similar data at the project’s end to measure impact. As described elsewhere in this report, government service statistics do suggest substantial increases in contraceptive use and prevalence; however, the denominators used to estimate a prevalence rate from government use statistics do not include women who are pregnant making comparisons between such estimates and others such as those in the DHS problematical. There were other gaps in the data made available to the evaluation team that limited the team’s ability to do certain in-depth analyses. Most important was the lack of cost data for the project, especially data that would give an indicator of the project’s cost per couple years of protection. And, as the award for the new project was pending, the team was not given advanced information of the anticipated funding for family planning and reproductive health over the next few years. Therefore, recommendations for the new project were made without consideration to the likely funding for the project. III. FINDINGS, CONCLUSIONS AND RECOMMENDATION Introduction This section includes the evaluation team’s findings, conclusions and related recommendations in four broad areas: • PI/E RH/FP overall accomplishments, • PI/E community-based approaches, • Other RH/FP issues: Private sector, method mix, contraceptive supplies • Sustainability. In particular, the sections on community-based approaches and sustainability respond to a number of specific questions raised by USAID in the evaluation scope of work. Major RH/FP Project Accomplishments/Challenges Overall According to all accounts9 and the background papers and other project data, the team reviewed, the RH/FP project10 implemented by PI/E. PI/E has played a major role in introducing and extending RH/FP/MCH knowledge, referrals, services and use and addressing harmful traditional practices (HTP). PI/E’s community-based service delivery approach is dynamic. Over the years it has contributed to and built strategically upon actions by non-governmental and governmental 9 The Team interviewed more than 300 USAID and GOE officials, donor and other NGO leaders, service providers and clients. See Annex B Persons Contacted. 10 For the purposes of simplicity, PI/E’s work under the original five year RH/FP project and the first nine months of the program’s one year extension under the centrally-managed Pathfinder program, Extending Service Delivery (ESD), September 2007-8 are referred to as a single project, RH/FP. 13 organizations11, lessons learned and targets of opportunity. Pathfinder has forged alliances with a range of dynamic implementing partner organizations (IPOs) and regional governments. This has led to new Ethiopian family planning service delivery models, increased voluntary commitment and financial support12 beyond USAID funding. PI/E has also provided critically-needed policy and advocacy support for RH/FP at the federal, regional, woreda and community levels. Three regions, Oromia, Tigray and SNNP credit Pathfinder with enabling them to get RH/FP included in the regional health budget. In Oromia, after regional officials participated in a PI/E observational study tour to a third country, the regional government made its first RH/FP budget allocation, six million Birr in 2007. In SNNP, the regional head of population credits Pathfinder’s timely service delivery data, results and advocacy support with enabling them to double the family planning services budget from 500,000 Birr in 2006 to one million Birr in 2007. Some of the SNNP regional funds have been used to deal with shortfalls and purchase contraceptives through DKT/Ethiopia. Other important breakthroughs include getting the Awassa University and Awassa State Agricultural colleges to reverse prior policy and include RH/FP services and peer advocacy groups on their campuses. The Regional Health Bureaus effectively coordinate and serve as advocates for any future health sector programs. The regional health bureaus seem to operate somewhat independently and do not seem to have regular contact with the Federal authorities. Foe example, they were not well￾informed about the timing for the roll-out of the health extension program. Two well-attended student clinics at the University of Awassa’s campuses make FP and HIV/AIDS services available to the university’s 3,000 students. Both campuses of Mekelle University have active RH/FP peer educator groups. Recommendation: Regional Health Bureaus (RHBs) need to need to be fully involved in the start up and planning of the new FP/MNCH project. USAID should ensure RHB representation on the new project’s technical advisory group. According to the table from Pathfinder provided to the Evaluation Team by USAID, PI/E has succeeded in meeting almost all of the targets in its project work plans. Reduced funding in the sixth year ESD extension did result in some shortfalls in programming due to inflation and increased overheads. Pathfinder and USAID should be commended for the extensive follow through on the majority of mid-term evaluation recommendations. A major unfulfilled recommendation related to ensuring evidence-based use of mass media. The team found that while the radio was being in used in three regions to reinforce community and facility-based RH/FP messages, important and routine message development, testing and target audience and listenership studies were not regularly being carried out. This made it difficult to determine the impact of radio messages. 11 At the federal, regional, zonal, district and community level and including Ministry of Health (MOH), Ministry of Education (MOE) and others such as government radio stations. 12 Over the six year period, USAID support of $34 million has been augmented by $6.2 from other donors such as the Packard Foundation, Swedish International Development Agency and UNICEF and an estimated $1.5 million in kind and cash from the IPOs who must contribute at least 25%. 14 Recommendation: In the follow-on project, there should be consistent use of evidence-based message development and listener and audience studies to develop behavioral change communications to the public. Important as these accomplishments are in meeting project goals, benchmarks and targets, the real success of the project rests in how it has be able to contribute to changed RH/FP and related MCH attitudes, knowledge and practice in the four regions where project resources are concentrated. And, while these changes can not be “attributed” to a single organization or project, no one the team met challenged the concept that the PI/E, particularly through its support of CBRHAs, has been a major contributor to change in the regions and woredas where PI/E provided assistance. Ultimate credit or attribution should go to the FP user who walked one to three hours over a rugged terrain to a health facility to get a long term method or to the young girl who defied family or community to resist early marriage or female genital mutilation (FMG). Over the last six years, the RH/FP project has built a diverse, community- based platform for a nationwide expansion of both long and short-term FP methods. This has been done primarily through in-service training in four regions for 2,279 RH/FP clinical providers predominantly operating in MOH health facilities, equipping and renovating health posts and health centers, building community-level capacity through supportive supervision and delivering a steady supply of long and short-term contraceptives. Because of the GOE’s major investment in expanding services in health centers (HCs) and health posts (HPs) under Health Sector Development Plan III and PI/E’s support of training and contraceptive distribution, increasingly Ethiopians have access to modern methods of contraception as well as immunization services for pregnant women and children. The MOH deserves particular credit for making Depo-Provera available at health posts and PI/E for providing CBRHAs and other health workers with the information clients need to avail themselves of this service. Regional and other local health officials told the team that PI/E played a critical role in distributing and ensuring a supply of longer acting methods such as Depo-Provera and Norplant. The RH/FP program has successfully helped desensitize family planning, by making birth spacing and smaller family size a new and acceptable norm for families in various stages of life. Again and again, religious and other community leaders, health care providers and clients reported that, while in earlier times children were counted as blessings, now, the community and families understood the need to space and limit births. CBRHAs, who said that they had been earlier attacked and vilified for promoting family planning, now said that they were welcomed and sought after. Equally dramatic changes in attitudes about and practice of harmful traditional practices (HTPs) such as early marriage and female genital mutilation (FGM) G were reported. Everyone the team interviewed said that there was a decline in HTPs. In addition to the work with woreda advisory committees (WACs) and CBRHAs, PI/E has directly reached out and trained traditional FGM practitioners and made them allies in the promoting safe reproductive health practices. PI/E has facilitated civil society’s dialogue on issues related to female circumcision, early marriage, obstetric fistula, violence against women, and gender imbalance in schools, community gatherings and other public outreach activities. Strengthening the RH/FP referral system 15 The program has made important contributions towards breaking down barriers to family planning through its CBRHA outreach services and by strengthening the RH/FP referral system. Over 609,000 referrals for MCH services were reported in P/IE areas in the final year of the project (P/IE 2007 final report). In three of the four regions a standard referral slip system was in place. The referral slip also provides a means for CBRHAs to track client compliance with the recommendation and follow-up on those who default. In one woreda, however, neither the HEWs nor the CBRHAs had the referral forms on hand to send clients to higher level facilities and their clients were not given written feedback to pass along to the health post following treatment at a higher level facility. In those instances it was clear that the quality of patient follow-up was not as high as the regions where HEWs and CBRHAs got a clear sense of the action taken at the health center or the hospital and any follow-up they needed to take. Since referral is one of the main areas of emphasis for the health extension program, this needs to be tightened up. For example, in 2007 the number of RH/FP referrals had gone to well over one million. Other services and strategies Pathfinder and its IPOs have also served as a valuable source of referrals to the country’s highly￾specialized regional obstetric fistula hospitals. These hospitals, by virtue of their service, are reaching some of the most vulnerable populations of women between the ages of 15 and 55. A useful addition to the current fistula programs would be the routine provision of FP information and services to clients before they leave the hospital and return home. The RH/FP program has engaged men in the promotion and delivery of RH/FP services. Many members of the WACs and the student peer educator groups are men. These men reach out to largely male audiences including agricultural coop members, the police and religious groups. Male church leaders working with PI/E funded IPOs have been instrumental in some communities in introducing new FP methods including sterilization and LAM. The team was told of greater male involvement in family planning decision making, including child spacing, and advocacy against harmful traditional practices and saw in some instances husbands accompanying their wives for RH/FP services. Men serve as health facility and CBRHA supervisors and in some cases as CBRHAs. In the four regions, where PI/E has focused its efforts, the program provides services in the woredas where 57% of the regions’ population lives and where there are approximately ten million woman of reproductive age. 13Through a combination of service provider training: Health Officers, Health Extension Workers (HEWs), Community-based Reproductive Health Agents (CBRHAs) and Traditional Birth Attendants (TBAs) and the development and distribution of provider information materials, PI/E has contributed significantly to the FP and other health information readily available to Ethiopia’s largely illiterate rural populations. The CBRHAs we interviewed who had materials on hand were able to demonstrate their use of FP/MCH flip charts and cue cards. Client interviews suggested that they were well informed about the FP methods they had selected. This is consistent with the 2005 Ethiopia Demographic Health Survey which showed that a high percentage of Ethiopian women (86%) and men 13 Email from Girma Kassie, Monitoring and Evaluation Team Leader Pathfinder International, Ethiopia to Harriett Destler, June 16, 2008. 16 (90.7%) know of at least one modern method of family planning. Estimated Number of Women in PI/E Targeted Woredas Table 1 PI/E RH/FP Coverage14 Total population of country 77,000,000 % of women population in the reproductive age out of any total population 23% Region Total population in the region (all targeted and non-targeted Woredas) Total Population in the targeted Woredas Estimated Total Women population of reproductive age in the region (all targeted and non￾targeted Woredas) Estimated Total Women population of reproductive age in targeted Woredas Amhara 20,128,804 14,874,217 4,629,625 3,421,070 Oromia 27,841,941 15,640,845 6,403,646 3,597,394 SNNPR 15,321,210 11,329,498 3,523,878 2,605,785 Tigray 4,487,260 2,167,273 1,032,070 498,473 Total (Four Major regions) 67,779,215 44,011,833 15,589,219 10,122,722 % out of the total country population 88.0 57.2 20.2 13.1 Family Planning Use The 2005 EDHS reported that modern method contraceptive use among married women had more than doubled from 2000 to 2005 from 6.3% to 13.9 %. This is one of the most rapid increases in the modern method contraceptive prevalence rate15 (CPR) in the world. It is even more amazing when Ethiopia’s size, diversity, geography, infrastructure and literacy rates are considered. An analysis of the 2005 EDHS16 found that “after controlling for their background, women living in areas covered by the CBRHA program were three times more likely to use contraception than the average Ethiopian woman”. The EDHS reported that only 31% of the demand for family planning was met in 2005. 14 Ibid. Girma Kassie Email 15 The MOH uses the term contraceptive acceptance rate which is defined the same way as a contraceptive prevalence rate, i.e. the percentage of women of reproductive age using contraception. 16 Yenehun Tawye et al, “The Potential Impact of Community Based Distribution Programmes on Contraceptive Uptake in Resource-poor Settings: Evidence from Ethiopia, African Journal of Reproductive Health, Vol. 9 No 3, and December 2005. 17 Project records estimate that during the project’s initial five years it reached about 3.7 million new FP clients and generated 3.3 million CYPs.17 Federal government reports based on service statistics show CPR increases in the four regions where PI/E is active. According to these statistics, the CPR increased from 25.9 in 2003-4 to 42.3 in 2006-7 in Amhara; from 17.7 to 26.1 in Oromia; 28.8 to 34.6 in SNNP and from 47.8 to 54.9 in Tigray. In Ethiopia as a whole the increase was from 22.9 to 33.3. Another small donor survey found a CPR rate of 45% in part of SNNP.18 The 2010 EDHS will provide more definitive information on RH/FP/MCH changes19. All the evidence now suggests very positive trends in the contraceptive prevalence rate, even with the annual increases in the number of women of reproductive age. The present PI/E data collection systems and reporting are cumbersome and labor intensive. Much of the data, which have been collected, have not been analyzed or used for planning and program management. For example, the client data have not been analyzed or used. The parity information gathered, if analyzed, might have led to an earlier design and launch of permanent contraception services. There has also been little or no cost information gathered such as cost per CYP. Strengthened M& E Systems As was noted in the description of the methodology, quantitative data to measure project impact on contraceptive prevalence and use was lacking. The upcoming DHS will provide definitive data about the change nationally in contraceptive prevalence but teasing out the contribution of any one project to national changes may be difficult unless there is increased sampling and a sampling frame is developed which makes it possible to contrast project areas with other areas. In general, the PI/E program lacked baseline and end line surveys to report on impacts. The new project should not fall prey to the same deficiency in data gathering for monitoring and evaluation. Three distinct actions on the part of the new project could go a long way to providing a remedy. Recommendation: Develop a survey instrument to establish a baseline in project districts, an instrument that can be repeated at reasonable cost to establish trends in key indicators over time. Budget should be set aside to assure that the data collection and subsequent analysis will happen. Historically, when funds for service delivery become tight for any reason, the funds set aside for monitoring and evaluation are often the first to be tapped. This should be avoided whenever possible. Recommendation: Establish an accounting system that can track the flow of project funds and, if possible, any funds leveraged from other donors, to facilitate a cost analysis of the project components. 17 P.2, Five year Close-out Report, Ethiopia, October 2002- September 2007, RH/FP Project, PI/E, January 11, 2008. 18 Conversation with Dr. Sahie Sitia, June 2, 2008. This rate reportedly came from an ESHE survey from which the team was shown results but not given a copy since the survey results had not been finalized. 19 If there is over sampling in the 2010 DHS in the regions where USAID focused its resources, it could be possible to determine if and what differences there were between assisted and non-assisted areas. 18 Recommendation: As the GOE launches tests for new ideas for modifying the current program (such as fee-for-service plans or conversion to all-purpose volunteers), systems should be put in place to evaluate those new ideas to determine their efficiency as well as effectiveness. Community­based Approaches Overview This section addresses the questions and topics related to community-based approaches raised by USAID/Ethiopia. Included in this section are: Implementing Partner Organizations, Woreda Advisory Committees, The CBRHAs’ Added Value and Evolving Roles, CBRHA Outreach to the Community, CBRHA Obstacles and Challenges, CBRHA Attrition, CBRHA Commitment to Voluntarism, and CBRHA and HEW Counseling Skills. Implementing Partner Organizations20 PI/E implements its Family Planning and Reproductive Health program by providing sub-grants to 45 indigenous Implementing Partner Organizations (IPOs) and the MOH. Sixteen of these IPOs receive USAID-only funded sub-grants specifically for implementing family planning and reproductive health (RH/FP) services at the community level. In addition to providing sub￾grants, Pathfinder also provides technical assistance and institutional and capacity building support and supplies to the IPOs. These agreements are a major innovation in family planning as they harness the energy of a diverse set of organizations and their networks in support of the RH/FP program. For example, the Ethiopian Kalehiwt Church (EKC) has 6,000 parishes and 6 million parishioners have now been introduced to RH/FP services through its network. PI/E funds the 16 IPOs through three basic models. While each model is distinct in the way it is financed by Pathfinder, the implementation of all three models at the community level is similar. For example, CBRHAs in all three models receive a basic two-week training, offer pills, condoms, and referrals for injectables and LAPMs, submit monthly reports to a supervisor, and receive a regular supply of contraceptives from Pathfinder. Moreover, all of the IPOs, regardless of the model, help to strengthen referral in the woredas in which they work. Differences in each model were found by the team in the sample of CBRHAs interviewed in the number of refresher trainings the CBRHAs are offered, the manner in which they are supervised, and how they receive their commodities. Based on the methodology used, the team found it difficult to discern a difference in the retention and performance of the CBRHAs under each of these models. This suggests that there could be savings in streamlining support activities. Operations research is needed to determine the impact of each of these models on the performance of the CBRHAs. Model One: Implementing Partner Organization (IPO) Model - P/IE provides funding directly to the IPO. The IPO recruits, trains, supervises and monitors and evaluates the CBRHAs. Additionally, the IPO provides commodities directly to the CBRHA. This is the most common of the three models, with 14 of the 16 IPOs implementing the RH/FP community-based model using this approach. 20 The strengths and weaknesses of each IPO model were assessed using questionnaires 1, 2, 3,4, 6, 7. 8. and 9, Appendix c. 19 Model Two: MOH Supported IPOs - In this model, PI/E awards the sub-grant to the IPO and the IPO directs the funds through the existing government structure to recruit, train, supervise, monitor and evaluate the CBRHAs. Currently, there is only one IPO operating under this structure, REST in Tigray. Model Three: MOH and RHB Model - PI/E provides a sub-grant directly to the Zonal Health office. This model relies solely on the MOH system for implementation of the CBRHA program. The government health officials recruit, train, supervise and monitor and evaluate the performance of the CBRHAs. PI/E works with the MOH in Amhara and Oromia. Two IPOs, the Amhara Development Association (ADA) and the Oromia Development Association (ODA) are categorized under the first model. However, both ADA and ODA are government affiliated-IPOs, which means they receive support from the government for other programs. However, although they collaborate closely with the regional and local health officials to implement the CBRHA program, they recruit, train, and supervise the CBRHAs independently from the MOH structure. Strengths and Weaknesses Each of the three models has its strengths and weaknesses vis-a-vis supervision, training and materials, provision of commodities, and the involvement of civil society. For example, while the strength of the first model is that it is able to provide the most consistent support to the greatest number of CBRHAs nationwide, its weakness lies in the creation of a parallel system of supervision, reporting, training, and logistics. Model one does not strengthen the capacity of the government to support CBRHAs and other community volunteers as much as models two and three. While model two’s strength is that it builds the government’s capacity to supervise and train CBRHAS, as well as deliver commodities, its weakness is that the ultimate responsibility for ensuring the performance of the CBRHAs lies with the IPO. The strength of model three is that it is inherently more sustainable because it attempts to strengthen the existing MOH system with limited external involvement. However, as this model is currently being implemented, it provides the least support to the CBRHAs. Supervision: Model one provides the most consistent support for the CBRHAs. CBRHAs submit their reports to their IPO supervisor who is responsible for reviewing the reports and providing feedback. The IPO supervisor in turn submits this report to the Woreda Health Officials. While this model provides the CBRHA with close supervision and counseling, the weakness of this model is that it establishes a parallel reporting system. For example, under the new HEW program, all community volunteers are supposed to be supervised by the HEWs. Therefore, CBRHAs would have to report to both the IPO supervisor and the HEW. The strength of models two and three is that the CBRHA reports directly to the HEW who reviews the reports and submits them to the Woreda-level officials. While this avoids a parallel reporting system, and integrates the CBRHAs reports directly into the MOH system, it does place additional responsibilities on the HEW. Currently HEWs have had limited training in supervision. Recommendation: Harmonize the supervisory system so that all volunteers report directly to the HEWs. All IPOs should revise their reporting systems to ensure that the CBRHAs report directly to the HEWs. This will ensure one reporting system. IPOs should focus their efforts on providing 20 supervision training for HEWs and strengthening the links between the CBRHA, HEW and local health center. In all models Pathfinder should systematically support training on supervision for HEWs and Health Officers. A simple supervisory check list should be developed for use of all HEWs. Based on the small survey of 32 HEWs, there was no evidence that the HEWs are using the CBRHA data to make work related decisions in terms of the division of labor or areas of emphasis for their work. There did not seem to be a systematic analysis of CBRHA performance records by the HEWs. Training and Materials: CBRHAs supported under models one and two were more likely to receive annual refresher trainings and own and use cue cards and other teaching aids. Of the three models, the MOH model provided the least amount of training. For example, in one woreda where the MOH is operating, the CBRHAs had not received any refresher training since their initial selection as a CBRHA. Moreover, few CBRHAs had record books on which to record their clients’ information and several inquired about teaching aids. Despite the lack of refresher training, the CBRHAs supported under the MOH model were familiar with the key messages in FP, MCH, HTPs, and HIV and AIDS. Recommendation: Strengthen the MOH Model 3: The new project should consider providing concerted technical assistance, funding and other inputs for implementation to the MOH to ensure that the CBRHAs and HEWs receive adequate refresher training, educational materials, and record-keeping tools. Commodities: Under model one; Pathfinder delivers the commodities directly to the CBRHAs. The team found at least one case in which the HEW was obtaining contraceptives from the CBRHA because the Pathfinder-supported supply chain was more reliable than the government’s. Under models two and three, the CBRHA collects the commodities directly from the HEW. Additionally, in model three, the government is supplying the pills, while Pathfinder is supplying the Depo-Provera. While there were stock outs reported in previous years for the pill, over the past year there have been no stock outs in any of the areas visited. Since the government is supplying the pills for the CBRHAs through the HEW, this strengthens the linkage between the CBRHA and the government-run health system. Recommendation: The CBRHAs should obtain commodities from the HEW. Where possible, the CBRHAs should obtain their supplies of pills and condoms from the HEW. This will help strengthen the overall health system at the community-level. Woreda Advisory Committees The Woreda Advisory Committees organized by Pathfinder are outstanding examples of civil society organizations with strong community representation from both the public and NGO sectors, including well organized faith based groups from three faiths (Ethiopian Orthodox, Protestant and Moslem) and leading local non-governmental organizations. Through the WACs, PI/E has created an important mechanism for community involvement in RH/FP did not previously exist. Three important key functions of the WACs are: setting programmatic targets, coordinating inputs, and giving family planning the legitimacy it needs to operate effectively in the woreda. 21 In all four regions, the local woreda administration leads the committee, which is charged with: reviewing performance of the CBRHAs (including in some cases maintaining a file on individual CBRHA performance (SNNP); identifying solutions to problems surfaced by the CBHRAs; planning quarterly activities with teams of HEWs and CBRHAs; and sensitizing and mobilizing the community for key events such as Norplant insertions. All WACs observed benefited from the project planning and project management training provided by P/IE and P/IE IPOs. This training helped the committees to set a focused agenda and track RH/FP district goals. In two regions, the visited woredas have adopted the WAC structure as an official government coordination body. This idea is excellent because it ensures better integration and planning of health services and programs at the woreda level, better monitoring and evaluation of programs and better use of health information generated by both the HEWs and CBRHAs. A key lesson learned from the six year P/IE experience is that the composition of these committees needs to be representative of the community and include civil society groups. In Tigray, one WAC had 17 people who met regularly twice a month, four hours per month of volunteer time. This translates roughly 68 hours per month of voluntary time that these representatives devote to their community on RH/FP. In SNNP, the religious leaders represented on the 12-person WAC visited were incorporating RH/FP messages into their sermons and organized religious meetings. There was further evidence of this when a Sunday school teacher told the team about the child spacing and HTP prevention messages included in his teachings. The WACs supported by models one and two were the most functional. No WACs were observed in model three, as they have yet to be established within this context. However, there is a Woreda-level health committee led by the woreda Vice Administrator that is responsible for addressing all health-related issues. However, this committee did not meet regularly and all of its members did not about the PI/E program. The Vice Administrator was responsible for the oversight of the population’s health status. The committee met irregularly and some members had a relatively small amount of information about the RH/FP project. The committee operating in model three is more sustainable because it is part of the MOH structure, but it needs additional support to build its capacity and ensure it operates effectively and efficiently. It will be important that this additional support is part of the new project. Recommendation: Continue to support the establishment of government-led health committees which regularly devote a significant share of their time to RH/FP planning and coordination. Civil society needs to be represented on these committees; their role is important. Recommendation: The GOE should continue to harness the energy and vigor of civil society to sustain and expand upon the successes of the current RH/FP program. Recommendation: The new project should consider shifting IPOs to model two. Then most of the money would be used to support and strengthen the MOH system in these areas. For example, in Amhara and Oromia, two IPOs already work very closely with the MOH. With this shift to model two, the IPO should assume a new function accentuating training rather than the administration of services. While this may result in some temporary slowing of momentum, it will help to strengthen the link between the MOH, the HEWs and the CBRHAs and lead to greater sustainability. Additionally, the new project should identify other IPOs that could effectively make this 22 transition as well, so that over the next five year period, there are more IPOs working on model two. This may not be appropriate in every situation. If it is done too drastically, it may have negative effects on the program implementation. The ODA and ADA seem to be appropriate IPOs to make this shift. The new project should consider providing concerted technical assistance to the MOH to ensure that the CBRHAs and HEWs receive adequate refresher training, educational materials, and record-keeping tools. CBRHAs CBRHA/HEW Added Value and Evolving Roles A major objective of the RH/FP evaluation was to provide a completely neutral outside view on whether the CBRHAs add value to the health extension program, serve sufficient numbers of acceptors to justify their continuation and cost and could in the foreseeable future do more outside of the five packages of RH/FP and MCH packages they currently offer. The team conducted 177 separate interviews with CBRHAs (46); HEWs (32); CBRHA IPO and MOH financed supervisors (9); and Households (90) in the four regions. The evaluation team concluded that the CBRHA community outreach approach has consistently supported and strengthened the GOE health extension worker program in all four regions. The team found that the CBRHAs, whose estimated support cost is $1,000 per CBRHA a year, were very cost-effective. A P/IE M&E and Finance Team estimated this average support cost in June 2008; the $1,000 includes training, supervision, transport and materials21. All the 32 HEWs interviewed expressed strong support and a continuing need for the CBRHAs. The HEWs repeatedly stated that they would not be able to carry out all of their 17 functions were it not for the work of the CBRHAs who focus on eight key RH/FP and MCH referral functions. This finding of the mutual benefit of HEWs and CBRHAs echoes an earlier study on this subject fielded by CORHA in 2006 and presented to the MOH, and the donor community. Using structured questionnaires (see annex C, questionnaires 4 and 7), the evaluation team observed the relationships of CBRHAs and HEWs who had been working together for two to three years. Separate interviews with CBRHAs and HEWs enabled each to explain their experience. Time allocations for the organization of daily work duties were provided by the interviewees. In all four regions, it was reported that the HEWs were the primary source for basic curative care, advice and counseling on epidemic control, offering up to 15 minutes per patient on FP when requested The CBRHAs devoted on average 45 minutes per client for new clients on family planning methods, prevention of harmful traditional practices, defaulter detection, training and referrals for long acting, and in some very rare cases, permanent contraception at the nearest health post or health center. The CBRHAs noted that it was not unusual to also hold more than one session with a client before the woman agreed to a method. The CBRHAs performed the important task of ensuring that clients continue treatment, that children complete vaccination, mothers attend ante-natal and postpartum clinics and return for well-baby sessions. CBRHAs spent about 20 minutes per client counseling continuing users. 21 Pathfinder 6/2008 estimate prepared for the evaluation team by the PI/E Finance team. 23 There is strong evidence that there is a logical and systematic division of labor evolving between the HEWs and CBRHAs similar to the functional roles and responsibilities outlined in the 2006 CORHA paper. Less than a quarter of the 32 HEWs interviewed received formal supervisory training yet 59% of the HEWs interviewed said that they were directly and officially supervising CBRHAs and other health agents (there are up to 7 different types). Almost half, 46% of the 32 HEWs were in officially recognized partnering relationships with the CBRHAs and received some oversight by the woreda health officials. HEWs said that another key function performed by the CBRHAS was promoting method continuation and addressing issues related to side effects. All of the HEWs interviewed said the CBRHAs had helped increase vaccination coverage through community mobilization during immunization campaigns, increased the number of latrines in project communities and positively contributed to the dissemination of epidemic control information. CBRHAs in all four regions routinely perform eight tasks and already offer services that strengthen primary health care: 1) mobilizing the community for the HEWs; 2) distributing short term contraceptive methods, 3) counseling clients on family planning; 4) addressing small groups of potential users; 5) supporting immunization campaigns and referrals; 6) working with the community to eliminate harmful traditional practices; 7) supporting the woreda’s epidemic control actions, and; 8) routinely advising and referring and often accompanying clients for ante-natal and post partum care at health posts and health centers. A January 2008 study on early marriage practices in two P/IE regions found that the CBRHAs and HEWS were the single largest group referring and counseling clients about the prevention of early marriage. Almost all of the 46 CBRHAs interviewed, had mastered the messages derived from the eight intervention areas listed above and were filling a strong need in the kebeles. The number of clients served at any one time by the CBRHA ranged from a low of 50 to a high of 1256 in a densely populated zone. The median number of clients reached is 150 per CBRHA. The range of clients is consistent with the figure of an average client caseload of 125 clients served by female CBRHAs reported in a September 2007 report by Pathfinder on long term family planning service delivery. It was clear that the presence of an HEW did not obviate the need for the services provided by the CBRHA volunteer. The needs are so great and the distances between households require significantly more time and attention than an HEW could handle on her own. CBRHAs demonstrate a great deal of creativity in attracting new clients with some CBRHAs visiting workplaces and places where farmers congregate or food aid is distributed. No one at any level voiced any concern that there was a duplication of effort; although two out of the 32 HEWs interviewed stated that supervising multiple categories of volunteers was burdensome. In 2006, Pathfinder briefly introduced the idea of using the CBRHAs to perform other tasks such as on-site treatment of tuberculosis (DOTS). While in principle, the idea of adding a broader range of tasks beyond MCH and RH is attractive, the evaluation team did not believe the addition of infectious disease interventions to CBRHAs duties at this time would best support the priority attention the Ministry of Health places on reproductive health. Global experience has shown that well-coordinated but specialized volunteers work more effectively to deliver quality service. The GOE has set ambitious goals which require a dramatic increase in contraceptive use over the next two years to reduce fertility. The CBRHAs should strive over the next two years to reach more clients, broaden the method mix, increase referrals for both longer term and 24 permanent contraception and work to eliminate HTPs. Current client monitoring systems and reporting are cumbersome and require considerable copying for reports. Recommendation: Maintain the eight current CBRHA functions and competencies but work to expand outreach to the extent possible with the means at their communities’ disposal. To further improve and clarify HEW roles and functions the MOH should move to formalize their roles and responsibilities as outlined in the 2006 CORHA report. Recommendation USAID through the new FP/MNCH project, should allocate sufficient resources to enable the MOH to strengthen the HEW and woreda health team supervisory oversight of all volunteers and mainstream all recording and reporting within the MOH system. CBRHA Outreach to the Community CBRHAs operate differently depending on their clientele and geography of their neighborhoods. In no case is the project currently reaching all of the woredas in a region. Two very successful CBRHAs in different regions reached the majority of their clients through the workplace. One in Amhara, has a multi-purpose kiosk in a marketplace where she works and also dispenses FP information and contraceptives, another travels to cooperatives and cash-for-work sites so that she can reach a large number of people at one time. The team found that a key ingredient for success is flexibility in the way the CBRHA operates. There should not be an attempt to standardize the way the CBRHA interacts with the community, provided that the work is planned and carried out under the supervision of the HEW and woreda health team. Recommendation: The future FP/MNCH project should capitalize on opportunities for CBRHAs to reach large groups of potential clients. One underutilized opportunity is reaching clients at scheduled immunization sessions and other group health activities. USAID/Ethiopia raised several other technical questions about the CBRHA approach, including the challenges, attrition, commitment to voluntarism, and the quality of care and CBRHA skill level. CBRHA Attrition The CBRHAs have a low rate of attrition, a strong work ethic and are stable members of the community, with a commitment to their community’s well being. The majority of the 46 CBRHAs interviewed are working between 8 -15 hours per week as volunteers. One highly successful CBRHA has 1,256 clients and works 48 hours per week. Attrition among the CBRHAs in the woredas visited ranged from 1 to 5%. There was virtually no staff turnover related to poor performance or lack of interest. Only one CBRHA was released. The primary reason for CBRHA turnover was promotion within the MOH structure, either to become HEWs (6 in SNNP, 2 in Oromia) or to assume other public health administrative functions. Death, illness and family tragedy were the other reasons cited for a CBRHA’s departure. The team concluded on the basis of its sample that attrition was not a major issue and that the CBRHA selection criteria and support system were effective. Overall job satisfaction amongst the volunteers was high. There were anecdotal reports in several regions of HEW turnover. In several regions, HEWs had left their posts due to marriage and transfer of their spouses to other geographical locations. 25 CBRHAs and Commitment to Voluntarism The CBRHA program is a strong expression of voluntarism. The time devoted to the program and the social and public health risks of working on such a program far exceeds any incentive offered to the CBRHA. The team found that altruism and a desire to serve one’s community are the prime motivating factors for CBRHAs followed by community recognition for community service. All of the CBRHAs interviewed had other pressing household and job responsibilities. Two of the four regions (SNNP and Tigray) offer no incentives to the CBRHAs other than the monthly transport allowance, training and community and program recognition. In the ODA￾operated zone of Oromia and the ADA-operated zone of Amhara, various fees are charged by the CBRHAs for the contraceptives which are supposed to serve as an incentive. In practice, this was not a great incentive as clients would get those same contraceptives for free when in the town. Revolving drug funds with a percentage set aside to offset CBRHA costs have been successfully implemented. Other communities contribute in-kind incentives such as housing (Oromia) or free labor to cover household duties like getting water, harvesting their field in lieu of payment. The Packard Foundation has tested the relative benefits of offering fees for service versus offering the services free or charging a nominal fee which doesn’t recover costs. They allowed kebeles to choose which model they wanted to implement. After one year they found that there was no better uptake in contraceptives when the services were given for free. Their conclusion is that communities are willing to pay for quality services if they are reliable. The GOE seems to be striving to make contraceptives part of a basic primary health care package of services which it delivers for free. Since the issue of incentives for volunteers has precedent-setting financial implications for the GOE, each region should decide how to handle it. Since the country is diverse it is unlikely that one model or approach to volunteer incentives would work for the entire country. For some woredas, the population and public health results of the CBRHA programs are so great that they warrant budgetary support. In highly food-scarce regions, it may make no sense to transfer this cost to households at this time. Other complementary non-health resources such as the GOE’s women’s fund or the local currency accounts of the USAID Title II Section 206 food aid might be alternate sources of funding for CBRHAs. Three different models of incentives are currently operating: 1) all services for free, 2) fee for service and 3) some services are free and others require payment. The transport allowance merely defrays a fraction of CBRHA transport costs which have continued to rise even in rural areas. In several cases CBRHAs family members’ with a donkey and cart or in one case, an Isuzu truck, had donated their families’ transport to carry out their duties. If one considers that a day’s work in the rural areas is compensated at 15 Birr per day, two days of CBRHA labor per week could be valued at 120 Birr per month which far exceeds the 40 to 75 Birr that the project has allocated for a monthly transportation allowance. Recommendation: Over the course of the next year shift the burden of responsibility for covering CBRHA incentives to the woredas. Continue to test ways to cover the impact of incentives on the Ministry of Health’s priority FP and RH objectives. After one year hold a conference to share lessons learned on the issue of incentives to encourage the Ministry of Health’s priority population and reproductive health objectives. Recommendation: USAID should help the MOH leverage other resources to support the GOE RH strategy 26 CBRHA and HEW Counseling Skills22 An important feature of the HEW program is strong face-to-face counseling for households and clients. All of the CBRHAs interviewed were able to provide a strong, lucid explanation of how they presented FP and RH information to clients. In the few instances where direct CBRHA/client interactions were observed, the counseling was appropriate, responsive to the questions raised and offered accurate, up-to-date information on family planning. Counseling for non-users, however, was not easy for either the CBRHAs or HEWs; these clients required more information to correct misinformation, and more time to meet with husbands or partners. The non-users all shared the same belief that the perceived benefits of having children at a young age (below age 18) was healthier for the young woman. For prima-paras, another “non-user” traditional belief was that, “since God gave us two eyes we need to have two children right away”. Counseling needs to be improved for women under age 18 to clearly explain the risks associated with child birth for underage women and the direct link between underage women, pregnancy and delivery complications and maternal mortality. Adolescents represent a major under-served population. Better client materials, as well as access to appropriate counseling and services, for young people are especially critical at this time, an estimated 44% of Ethiopia’s population is under the age of 15. Counseling on the use of dual methods was also inconsistent and in some cases neither the HEW nor the CBRHA understood the concept of dual methods. This was particularly striking in that two of the regions sampled that have HIV education programs and where dual protection is so important.. Recommendation: Regular contraceptive technology updates including updates on dual protection, and messages for adolescents and prima-paras for HEWs and CBRHAs must be given priority and followed up through supportive supervision. CBRHAs in all but one region (SNNP) are giving out only one packet of pills per client visit. This practice, for continuing users should be reviewed in light of the successful distribution of a three month supply of pills in most CBD programs in other countries. Recommendation: Any restrictions on the number of pill packs dispensed should be reviewed for possible revision. CBRHA Obstacles and Challenges The majority of the 46 CBRHAs interviewed overcame harassment and in some cases physical abuse by their communities at the outset of their tenures as CBRHAs. It is a testimony to their courage and tenacity that the program has advanced and the stigma against family planning and the shroud of secrecy surrounding harmful traditional practices have diminished. Early marriage and HTP records are being recorded and reported in all but one region. During one field visit, the team heard from four girls whose circumcisions had been averted. Two had been referred to local law enforcement authorities for follow-up and had been amicably resolved. In the other family, the father decided not to circumcise his daughters. The team also met with 17 women whose marriages had been cancelled in Amhara. PI/E is to be commended for raising funding from other sources to enable these young women to remain in school. There is strong evidence 22 The strengths and weaknesses of CBRHA and HEW counseling skills were assessed primarily through questions 4, 7 and 8, Appendix c. 27 that traditional circumcisers in SNNP are leaving the profession following training by the IPO EKC on the harmful nature of the practices. Two circumcisers in Oromia, who were interviewed for this evaluation, noted that they had unknowingly performed circumcisions on young girls without understanding the harm it could do. These two women, as a result of the PI/E project, were now educating the public about the dangers of female genital mutilation. The team was told that there was far greater community awareness of the dangers of HTPs and reporting of HTPS to law enforcement. The teams saw two cases where FGM was averted and another 17 cases where early marriage had been cancelled. A number of clients and CBRHAs as well as some WAC members said that practices such as early marriage would not be tolerated because these were against the law. Other RH/FP Issues   In the course of the evaluation, RH/FP service issues surfaced related to the quality of care, method mix, infection control, quality of materials, supportive supervision and reporting. Also raised were issues such as the contraceptive logistics system and the links between community￾based public sector programs and private sector social marketing programs. Quality of Care All ten health facilities visited were offering new, highly sought after short and long term family planning services, STI diagnosis and treatment, ante and postpartum and post abortion care. There were big crowds of clients waiting to see providers in the Southern region. All but two of the facilities had welcoming waiting areas which displayed excellent informative materials displayed. The youth friendly services at Awassa University and at the city’s health center also had brochures, videos and flipcharts. In several regions outside of SNNP, the team saw no effective client materials. This was particularly true in the case of effective materials for young adults. In some regions, and at the University of Awassa, excellent use was made of the computers donated by Pathfinder for computerized maintenance of patient files. In Tigray and Amhara, little use of the computer was made due to irregular power sources. In SNNP the files at every facility were found to be well organized and easily accessible in the public sector sites. The IPO records and the CBRHA records were also up to date and contained extensive information on client histories, and follow-up. However, the majority of data collected in the CBRHA ledgers are not used to improve program quality and better target services. Currently there is more data generated on parity, client histories, marital status and client preference for no more children than can be analyzed by the field workers. This over reporting places a burden on CBRHAs and other health workers. Recommendation: Before the end the project in August 2008 Pathfinder/Boston should work with Ethiopian schools of public health to analyze key information as a basis for future new programming directions under the FP/MNCH project. This analysis should be done in close collaboration with the ESHE project which also has generated considerable, under-utilized information. This would benefit all health projects operating in Ethiopia. The new project should simplify project reporting at the community level, improve the analysis of data and develop systematic feedback to providers, supervisors and others charged with managing and reporting on the program. 28 Infection Control All health facilities visited lacked infection control procedures, particularly the ability for providers to wash their hands between clients. No sharpes boxes were seen in any SNNP sites and no gloves were provided. It is not clear if staff are following adequate procedures for the disposal of syringes for injectibles. In terms of occupational safety, none of the HEWs or CBRHAS were provided antibacterial solutions that can be used without water. CBRHAs routinely travel from house to house without washing their hands. When this was raised with the P/IE supervisors it was noted that antibacterial solutions were provided to CBRHAs at the outset of the project but had been discontinued. At least two health posts in Oromia had to stop doing deliveries due to lack of institution-based infection control procedures. The lack of running water or other alternatives to cleaning hallways, crowded waiting areas and examination rooms was an issue in two regions where multiple health facilities were visited. It was observed that communities are not involved in cleaning health posts or health centers. This is one way used to tackle the problem in other resource constrained settings. In marked contrast to these centers, the RH/FP clinics visited at the University of Awassa, the Awassa Health Center and the Awassa Zuria health center had high standards of cleanliness. Recommendation: Carefully review infection control procedures and use the lessons learned from the well run and clean facilities about successful approaches to dealing with this pressing issue. Supportive Supervision and Reporting The team found various models of supervision by the IPOs, emanating from the HEW or led by the head of the WAC. While the team observed many good examples of supervision by IPOs the overwhelming finding was that sustainability and capacity building would be better served by transitioning to a government-led technical and administrative supervisory structure. The process of transition may require up to a year to be seamless. It might be best accomplished by using the best IPO supervisors to provide in-service training to woreda level supervisors and HEWs. Existing project level transport for woreda supervisors should be turned over to the woreda. The broader issue of transportation should be given priority attention in the new USAID project. Since transportation is a critical element in the success of supervision, the new USAID project should assure adequate, sustainable transportation is built into the design. This does not have to be completely funded by USAID. Another finding is that, while HEWs have been doing a good job in some woredas of providing technical oversight to CBRHAs, the size of their workload is such that it is unreasonable to expect them to cover all monitoring and reporting duties. Woreda officials should take on the reporting, recording and other analytic functions in the MOH chain.. The HEW can simply compile and transfer data to the woreda. Conversely, great attention needs to being paid to the timely analysis of data and the provision of feedback to those delivering, managing and supervising services. In three of the four regions, PI/E’s supervisors were consistently using supervisory checklists. In one woreda in Oromia, however, the checklist was not used at all. Supervisors in that woreda complained that it was too long and could take up to a half a day if done well to cover one CBRHA. The length of the forms may be one of the reasons it poses a problem for supervisors. This checklist should be simplified in the future with a heavy emphasis on infection control measures which need to be routinely verified. The new USAID project should consider transitioning to a government supervisory structure and improving the supervisory checklist. 29 Contraceptive Supplies The team was struck by the lack of contraceptive logistics problems. The well maintained stock on hand in the health facilities, at health posts and among the CBRHAs is a sign of a well￾functioning system. PI/E is to be commended for establishing a user-friendly supply chain. The one product, implants, seemed to have been in short supply in Tigray and Oromia. The demand for this product was underestimated when it was introduced. Care should be taken to estimate client demand correctly when introducing other FP/MCH services. Given the extensive work currently required to strengthen and make fully functional the MOH pharmaceutical supply chain, the team believes that the current system of distribution should remain in place until the new MOH system is fully operational. From what the team has been able to discern this process may vary by region. Recommendation: A phased approach adding the contraceptives to the new government pharmaceutical distribution system should be taken to avoid any disruption of the contraceptive supply so necessary for clients. Method Mix and Permanent Methods Client, CBRHA and IPO interviews confirmed that since 2005 the method mix has shifted from one dominated by short term methods (pills and condoms) to longer acting methods. Since 2005, PI/E trained more that 300 health providers in long term family planning methods. (September 2007 Pathfinder assessment entitled “long term family planning service delivery”). According to female clients Depo-Provera, an injectible, is the most popular method “because it can be taken without anyone knowing”. At the community level this has meant a greater workload for the CBRHA and HEW and nurses who give the shots. CBRHAs remind women to go to the health post or health center to get their shot every three months. Depo-Provera now constitutes 53.5% of the contraceptive market (MOH 2007 annual report). In the project-financed regions, discontinuation rates of Depo-Provera are low. The most common reason given for discontinuing Depo-Provera was switching to a longer-term protection implant. Norplant (7 year) and Implanon (3 year) went from 3.8% of the contraceptive market 2006 (MOH 2006 Annual Health Indicators Report), to over 5% of the contraceptive market in 2007. Pathfinder is to be commended for introducing Norplant in 2006 and training hundreds of providers in insertion and removal and promoting its use through the CBRHAs, HEWs and health centers. Implant clients consistently voiced their deep appreciation for implants and cited them as “life changing”. One woman with seven children, of whom the last two are three year old twins, credited her Norplant with saving her last two children’s lives. She also said that because of Norplant, her family stopped having children. This allowed the family to concentrate on educating her older children and enabling them to advance through primary school. The increase in the public sector’s method mix has greatly improved since the mid-term evaluation, with the share of IUD, and Norplant users growing from three to five percent; IUD use rates still remain low. The unavailability of permanent surgical contraception for men and women is a very real gap. Client interviews, and a review of the CBRHA registers in two regions, revealed that there were between 10 to 20 long-term contraceptive users of high parity who might be good candidates for permanent contraception due to their age and their expressed desire to have no more children. The team in Amhara also heard about a male client who had sought out vasectomy services. The need for VSC services was confirmed in SNNP by the 30 regional Pathfinder representative who is trained in VSC. The widespread introduction of VSC services should be a goal of the follow-on project and should build on the community-based platform of clients PI/E created. The training, equipment and infection control requirements for a broad-based introduction should not be underestimated. It will likely require a full year to gear up before services can be offered. This should be an early goal of the new FP/MNCH project in collaboration with Engender Health and other partners working in family planning. Recommendation: Before the RH/FP project ends, Pathfinder and its IPOs should analyze their extensive client records to assess the possible demand for VSC services. Recommendation: Both long-term and permanent contraception should be integrated into the pre-service and in-service training modules for doctors and nurses. Links to the Expanding Private Sector Contraceptive Social Marketing Program DKT/Ethiopia, a registered local NGO, is the number one distributor and procurement agent for both public sector and private sector contraceptives with a 44.4% share of the overall contraceptive market based on GOE data (GOE annual public health report for 2006 and 2007). There is no evidence of any non-subsidized commercial contraceptives in the market at this time. DKT asserts that the reason is that 95% of current contraceptive consumers are lower middle “C” class customers not higher income “A” or “B” level that might be interested in higher end products. DKT’s contraceptive social marketing program was evident in all four regions. The nine pharmacies visited and sampled during the evaluation showed a slight up-tick in condom sales and no increase in pill sales. On average, pharmacies in the towns and cities had between 20-30 family planning clients per day. The “Sensation” condom product line was by far the most sought after contraceptive by men. It can be found at all levels of the 800 pharmacies and drug shops nationwide. DKT’s product line currently consists of 14 products, including a progesterone-only pill suitable for breastfeeding mothers, emergency contraception, and an ORS product (LEM). DKT has five products being registered now including Misoprostol to prevent hemorrhage and other implants. Several other products not currently available are in the registration process. In addition to the expanding product line, DKT intends to launch a new “youth marketers” program that will consist of a 100 person marketing team who will sell contraceptives in urban areas and towns. USAID should work to assure that there is a high degree of coordination between its future FP/MNCH project and that of the DKT, financed by European donors. USAID and DKT should map out their service delivery sites and target clients and assess the relative value added that a separate pubic sector CBRHA would play in the towns. Some of the young current CBRHAs may also be excellent members of the future DKT sales force. There is no discernible link between the Pathfinder RH/FP community level CBRHA project or the private sector providers’ project in Addis and Oromia and the well established DKT/Ethiopia contraceptive social marketing program. CBRHAs and HEWs rarely refer clients to pharmacists, the IPOs have not trained pharmacists and DKT has not been offered RH/FP materials for detailer or pharmacist training. In fact, private providers training by PI/E, who previously purchased socially marketed contraceptives from DKT/Ethiopia, switched to public sector ‘free’ commodities after being trained by PI/E. This is arguably a step back from developing sustainable private providers. DKT’s new three year $22 million grant with the UK, Irish and 31 Dutch governments, which includes subsidized contraceptives, plans to grow the contraceptive market through an expanded product line and general advertising campaign for family planning. Recommendation: The future USAID public sector FP/MNCH project should seek greater links with the CSM project from the outset. Joint mapping of regions should be done and a careful analysis of whether CBRHAs can join the DKT detailing team or serve as a referral for DKT’s growing retail pharmacy contraceptive product line. Products such as the progesterone only pill or the Copper –T 380 IUD, not currently available in the public sector for free, should be promoted and linked to the new USAID financed FP/MNCH project. Sustainability The evaluation team spoke to a variety of stakeholders in the public and private sectors about the issue of sustainability and institutional strengthening. This section examines several aspects of financial and managerial sustainability In the development community, the term “sustainability” has been overworked and abused. In this evaluation report, when the team speaks of “sustainability” the team means first the preservation and expansion of the health and/or population impacts of a given program or set of programs. The team does not believe that the objective of any organization should be to preserve any particular program or approach. Instead, the team believes that organizations should be prepared to adapt to changing conditions and situations in order to sustain the benefits derived from its own earlier efforts and the work of others. In addition, it is useful to distinguish two types of sustainability: system sustainability and demand sustainability. System sustainability has, in turn, three distinct components, financial sustainability, institutional capacity and enabling environment while demand sustainability can be viewed as change in attitudes and the ability to pay. Financial Sustainability The progress made in the years since the introduction of CBRHAs, and more recently the HEWs has been impressive. Ethiopia has not yet achieved a high enough contraceptive prevalence to slow its population growth. Accordingly, any plan to move toward financial sustainability (removal of dependence on donors) should be conceived so as not to slow the pace of increased contraceptive use. In the field, the evaluation team saw a high degree of voluntarism among CBRHAs and a growing recognition in the official community and in individual households of the importance of family planning for health. To capitalize on this growing recognition, measures should be taken to mobilize resources at household and community level to initiate a transition from total donor financial responsibility to local, regional and national responsibility. Two such measures were discussed during this evaluation: fee-for-service and community and regional contributions. Fee-for-service The fee-for-service model has been tested by the Packard Foundation [conversation with its Director, Dr. Sahlu Hailie] and has been shown to work well. Households in kebeles paying a small fee-for-service had equivalent gains in contraceptive acceptance as those receiving the service for free. However, the collective experience of the evaluation team suggests that the establishment of the systems to manage the funds collective through fee-for-service can be costly 32 and difficult to apply on a large scale. Moreover, the GOE may not be in favor of fee- for￾service approaches since FP/RH services are part of the GOE’s approved package of free services. USAID should get specific GOE consent to pilot fee for service systems in the new project if this is still a priority Recommendation: The team recommends that continued experimentation with fee-for- service be undertaken to determine the feasibility and value of fee-for-service. Community contribution The overwhelming support at all levels (regional, woreda, kebele and household) for the work of the CBRHAs and the HEWs in bringing family planning and other health services to the community suggests that communities might be willing and ready to make a contribution in cash or in-kind to enable CBRHAs to continue without the small incentives provided by the PI/E program. Recommendation: In order to validate the team observation that the level of commitment in communities has grown enough by now to generate resources and to determine the technical and organizational assistance communities may need in order to manage such a contribution. Institutional Capacity Generally, institutional capacity refers to the ability of the people delivering services at the periphery to do so correctly and the management skills of the human infrastructure in place to those front line workers in their efforts to perform their primary tasks. Training of the CBRHAs and the HEWs In the field, the evaluation team observed a high level of understanding of family planning and reproductive health issues among CBRHAs and HEWs and at woreda and regional levels. It can be presumed that the understanding among the CBRHAs can be attributed to the training provided by PI/E and its IPOs. The HEWs are trained by the government with additional training provided by PI/E and/or the IPOs. Additional training at all levels will be needed if the recommendation that permanent methods be introduced on a large scale is accepted. Funding to make permanent methods available is coming from other sources. Recommendation: The FP/MNCH project should be prepared to assure that, at the very least, the training of workers in geographical areas covered by the project is administered at a very high level. In the related area of maternal health, momentum is gathering globally to increase and improve skilled attendance at deliveries and the provision of emergency obstetric care. In January of 2008, UNFPA launched The Maternal Health Thematic Fund to address the major causes of maternal mortality. Ethiopia has been designated as one of the eleven first-wave countries to receive funding under this initiative. There is little doubt that additional training for both the CBRHAs and the HEWs will be supported under this initiative. The new project should become a partner in this initiative and should offer its experience in training in Ethiopia to help make it a success. Capacity to manage the program 33 The evaluation team observed a well-run reporting and supervision system in some of the 11 woredas visited. One issue regarding the ultimate ability of the Government of Ethiopia to assume greater responsibility for the program is the management and distribution of contraceptive commodities. It was noted that there were four contraceptive supply crises prior to 2006. One of the sites visited noted earlier supply problems. With the entry of PI/E into the picture, these problems were eliminated. Other USAID projects are working with the Government of Ethiopia to develop an overall logistics system to facilitate the distribution of all essential drugs, including vaccines, as well as family planning commodities. Recommendation: The evaluation team applauds the effort to have a single national pharmaceutical distribution system but feels that an independent PI/E logistics system be retained for contraceptives by the new USAID project until the Government of Ethiopia system is fully functional. Enabling Environment The success of the program reflects a very positive enabling environment for family planning and reproductive health which limits population growth and saves lives. Slowing population growth is essential if the government is to develop its human capacity and physical infrastructure to keep pace with and, hopefully get out in front of the expanding population and the reemerging food scarcity. This reoccurs seasonally due to local production problems, drought and, now, rising prices for staple foods. The relationship between CBRHAs and HEWs The relationship between CBRHAs and HEWs was discussed a great deal during the evaluation. The CBRHA program pre-dated the Health Extension Program (HEP) by many years. While in the field, the evaluation team looked carefully at the evolution of the relationship between the CBRHAs and the HEWs. In all cases, the CBRHAs and HEWs expressed the view that despite some shared responsibilities between the two cadres, the CBRHAs and the HEWs were mutually supportive and not redundant. The HEWs benefitted from the rapport already established between the CBRHAs and the community and certainly appreciated the outreach the CBRHAs could achieve working in a more limited geographic area with families known to the CBRHAs their entire life. Many CBRHAs sited the benefits accruing to their efforts to promote family planning by having the HEWs in the community, most notably, the improved community education provided by the HEWs and the facilitation of the provision of selected services (long acting methods in the case of family planning) by bringing them closer to the community. One can anticipate that the HEWs role will evolve over time as they take on additional curative and preventive services (for example, the treatment of respiratory infections with antibiotics and the provision of emergency obstetric care at deliveries). The ability of the HEWs to maintain personal contact with accepters of family planning, already limited by the numbers of people the HEWs are obliged to serve, is likely to diminish even more and the need for community volunteers will become ever greater, making it difficult to eliminate the CBRHA function in the near term. The issue of the relationship between the CBRHAs and other community volunteers was discussed widely in the field. Many communities visited by the evaluation team mentioned the presence in their communities of health promoters and malaria promoters. The CBRHAs were 34 generally active in the Model Family program. Many were the first in their communities to implement the 11 steps in their own homes and participated in malaria campaigns where impregnated bed nets were distributed. The question arose, “Are all of these specialized volunteers needed or should the Government seek to meld the diverse activities now undertaken by the cadre of volunteers into a single community volunteer to work with the HEWs on all aspects of preventive and curative health care delivery within their scope of work?” Based not only on the experience vested in this evaluation team but also on the views expressed by some HEWs and other officials in the field, the evaluation team cautions against taking such an approach. The specialized knowledge and, in particular counseling skills required to address family planning issues correctly call for a more specialized and talented volunteer than one whose responsibilities are numerous but not too deep. Recommendation: Should the Government of Ethiopia move toward the all-purpose volunteer, this evaluation team recommends strongly that the concept be fully and carefully tested in an operations research setting before any wholesale changes in the current panoply of volunteers are made universally. Attitudes Ultimately, the sustainability of any new program rests in the continued strong demand from the people in the community for the program. There was universal acknowledgement in the field that the attitudes (and practices) of the people toward family planning have continued to change over time under the PI/E program. A number of the CBRHAs with longer tenure noted that the resistance encountered from husbands and other elements of civil society years ago has melted away and, in select cases, transformed itself into ardent support. Much of this change was attributed by woreda level officials to the continued advocacy of the CBRHAs. During the field visits, the possibility of a cessation of support for the CBRHAs now afforded through the PI/E was raised with officials at Woreda level. The possibility of such a cessation of support was met with a high degree of anxiety. The discussions that followed did not lead to a clear course to take should the support be terminated. The team was, however, very concerned to find no evidence of a transition plan in place in June 2008 for a project ending in September 2008 to ensure that there would be no break in contraceptive supply or client services between the end of the PI/E RH/FP project and the start up of USAID’s new, yet to be awarded, Family Planning/Maternal, Newborn and Child Health Program. The message that came through loud and clear was that there should be no abrupt change in the program to give the powers that be the time to develop ways to maintain the program by some other means. Ability to Pay As noted earlier in this report, donor support for family planning and reproductive health in Ethiopia is growing. Nonetheless, in the long-term, the users of contraception in Ethiopia will have to assume some of the burden for family planning by paying for a portion of their contraceptive commodities. Based on conversations with DKT, the commercial market is already the source of 44% of the contraceptive commodities in the country, demonstrating the 35 willingness and ability of one segment of the population to pay for subsidized commodities. Given the influx of donor funds, the likelihood of a major shift to a program based on the purchase of commodities in the near term is small. Recommendation: The new FP/MNCH project should consider further experiments and/or operations research to determine how to structure a transition to a program in which contraceptive users contribute according to their ability to pay. 36 RECOMMENDATIONS The evaluation team has a set of recommendations that fall into five categories: 1) Build on USAID’s Comparative Advantage in Supporting RH/FP in Ethiopia, 2) Ensure USAID-assisted programs to maximize support to the MOH, 3) Retain the CBRHAs as adjuncts to the HEW and continue other activities that encourage civil society involvement, 4) Strengthen FP/RH/MCH service delivery at health centers and health posts, 5) Strengthen M&E Systems and continue to test alternatives and learn from best practice. 1. Build on USAID’s Comparative Advantage Recommendation: Regional Health Bureaus need to be fully involved in the start up and planning of the new FP/MNCH project. USAID should ensure Regional Health Bureau representation on the new project’s technical advisory group. Recommendation: Pay urgent attention to the issue of transition between the current FP/RH and ESHE projects which end in September 2008 and the newly- awarded FP/MNCH project. Recommendation: The future USAID public sector FP/MNCH project should seek greater links with the CSM project from the outset. Joint mapping of regions should be done and a careful analysis of whether CBRHAs can join the DKT detailing team or serve as a referral for DKT’s growing retail pharmacy contraceptive product line. Products such as the progesterone only pill or the Copper –T 380 IUD, not currently available in the public sector for free, should be promoted and linked to the new USAID financed FP/MNCH project. Recommendation: The evaluation team applauds the effort to have a single national pharmaceutical distribution system but feels that an independent PI/E logistics system be retained for contraceptives by the new USAID project until the Government of Ethiopia system is fully functional. 2. Ensure USAID-assisted programs continue to maximize support to the MOH Recommendation: Harmonize the supervisory system so that all CBRHAs report directly to the HEWs. Recommendation: Consistently use evidence-based message development and listener and audience studies to develop behavioral change communications to the public. Recommendation: Strengthen the MOH Model 3 with technical assistance, training and other support as needed.. Recommendation: The CBRHAs should obtain commodities from the HEW. Recommendation: Continue support the establishment of government-led health committees which regularly devote a significant share of their time to RH/FP planning and coordination. Civil society needs to be represented on these committees; its role is important. Recommendation: The GOE should continue to harness the energy and vigor of civil society to sustain and expand upon the successes of the current RH/FP program. 37 Recommendation: The new project should consider shifting IPOs to model two. In this way, most of the money would be used to support and strengthen the MOH system in these areas. Recommendation: The donor community, including USAID through the new FP/MNCH project, should allocate sufficient resources to enable the MOH to strengthen the HEW and woreda health team supervisory oversight of all volunteers and mainstream all recording and reporting within the MOH system. Recommendation: The future FP/MNCH project should capitalize on opportunities for CBRHAs to reach large groups of potential clients. One underutilized opportunity is reaching clients at scheduled immunization sessions and other group health activities. Recommendation: Over the course of the next year shift the burden of responsibility for covering CBRHA incentives to the woredas. Continue to test ways to focus the impact of incentives on the Ministry of Health’s priority FP and RH objectives. After one year hold a conference to share lessons learned on the issue of incentives to encourage the Ministry of Health’s priority population and reproductive health objectives. Recommendation: Carefully review infection control procedures and use the lessons learned from the well run and clean facilities about successful approaches to dealing with this pressing issue. Recommendation: A phased approach to adding the contraceptives to the new government pharmaceutical distribution system should be taken to avoid any disruption of the contraceptive supply so necessary for clients. Recommendation: Validate the team observation that the level of commitment in communities has grown enough by now to generate resources and to determine the technical and organizational assistance communities may need in order to manage such a contribution. Recommendation: The FP/MNCH project should be prepared to assure that, at the very least, the training of workers in geographical areas covered by the project is administered at a very high level. 3. Retain the CBRHAs as adjuncts to the HEW Recommendation: Maintain to the eight current CBRHA functions and competencies but work to expand outreach to the extent possible with the means at their communities’ disposal. To further improve and clarify HEW roles and functions the MOH should move to formalize their roles and responsibilities as outlined in the 2006 CORHA report. 4. Strengthen FP/RH/MCH service delivery at health centers and health posts Recommendation: Regular contraceptive technology updates including updates on dual protection, and messages for adolescents and prima-paras for HEWs and CBRHAs must be given priority and followed up through supportive supervision. Recommendation: Any restrictions on the number of pill packs dispensed should be reviewed for possible revision. 38 5. Strengthen M&E systems and continue to test alternatives and learn from best practice Recommendation: Before the end the project in August 2008 Pathfinder/Boston should work with Ethiopian schools of public health to analyze key information as a basis for future new programming directions under the FP/MNCH project. This analysis should be done in close collaboration with the ESHE project, which also has generated considerable, under-utilized information. This would benefit all health projects operating in Ethiopia. The new project should simplify project reporting at the community level, analyze data more rapidly and provide feedback to those delivering, supervising, managing and reporting on services and results.. Recommendation: Before the RH/FP project ends, Pathfinder and its IPOs should analyze their extensive client records to assess the possible demand for VSC services. Recommendation: Both long-term and permanent contraception should be integrated into the pre-service and in-service training modules for doctors and nurses. Recommendation: The team recommends that continued experimentation with fee-for- service be undertaken to determine the feasibility and value of fee-for-service. Recommendation: Should the Government of Ethiopia move toward the all-purpose volunteer, this evaluation team recommends strongly that the concept be fully and carefully tested in an operations research setting before any wholesale changes in the current panoply of volunteers are made universally. Recommendation: The new FP/MNCH project should consider further experiments and/or operations research to determine how to structure a transition to a program in which contraceptive users contribute according to their ability to pay . 39 GUIDANCE TO USAID FOR STAFFING AND MANAGEMENT OVERSIGHT AND FUTURE CONTRACTOR WORK PLANS Staffing/Management 1. Recognizing that the integration of the two existing USAID projects with the MOH’s new Health extension program is bringing together three completely different approaches, USAID needs to assign more staff hours to overseeing and guiding this transition. This will likely require an off-site retreat with the MOH to review the MCNH work plan and its implications for field technical staff, the IPOs, and other key service delivery systems such as transportation, delivery of contraceptives, supervision etc. 2. The MOH should identify a focal point/manager(s) or management team who will oversee and assure the successful merger of these programs into the health extension program. This team should include regional health bureau managers. Future Contractor Work plan 1. Be on the forefront of coordination with all new initiatives including the “anonymous” donor, the UNFPA Maternal Health Thematic Fund and DKT’s new expanded contraceptive social marketing project funded by a consortium of European donors. 2. Support the MOH to set up an MOH–led mechanism for coordination among donors. 3. The new project’s Technical Advisory Committee must be led by the Ministry of Health which is a critical stakeholder in this new project. 4. The new contractor should identify several FP/RH policy issues within the first three months that need to be in place by the end of the project, such as the introduction of misoprostol (which is currently being registered by DKT and studied by UC Berkeley in Tigray) and the use of progesterone only pill suitable for lactating women and the copper-T-380 IUD available from DKT/Ethiopia pharmacies. 5. Task Pathfinder and Addis University’s school of public health to develop a basic data set culled from both Pathfinder and ESHE databases. Data on cost per CYP, the costs of training and supervising different categories of volunteers and the costs of supervision should be done now using the existing Pathfinder and ESHE teams. 6. Task the new contractor in collaboration with the MOH to develop a streamlined reporting and monitoring system which is compatible with the MOH health and reporting system and to map out the key steps to achieve this objective. For example, the two existing USAID projects have two completely different health and project information flow charts. The best practices from these two systems needs to be identified and harmonized with the MOH reporting system. 40 7. Task the new contractor in collaboration with the MOH to develop an affordable survey to provide a baseline and regular updates on key indicators in project district to monitor change over time. Budget should be set aside to assure that the data collection and subsequent analysis will happen. Historically, when funds for service delivery become tight for any reason, the funds set aside for monitoring and evaluation are often the first to be tapped. This should be avoided whenever possible. 8. Task the new contractor to establish an accounting system that can track the flow of project funds and, if possible, any funds leveraged from other donors, to facilitate a cost analysis of the project components. As the GOE launches tests for new ideas for modifying the current program (such as fee-for-service plans or conversion to all￾purpose volunteers), task the contractor with the MOH to develop systems to evaluate those new ideas to determine their efficiency as well as effectiveness. 9. Task the new contractor with devising a streamlined supervisory checklist suitable for HEWs to administer for all categories of volunteers. This checklist should be based on the best practices gleaned from the ESHE, PI/E and other operational projects with strong community based volunteers. 10. Continue advocacy for regional financing for contraceptives and support for family planning programs. 11. Greatly expand adolescent services in both formal health delivery settings and informal settings. 12. Greatly expand voluntary surgical contraception for both men and women in all project supported regions. 13. Assure that pre-service training modules for adolescent and VSC services are an integral part of medical and nursing school curriculum. Be sure to link a practicum to these training modules. 14. The future work plan should include a conscious strategy to integrate FP/RH services in the food aid and humanitarian assistance programs. This means reaching out to new partners such as Mercy Core, ADRA and other relief agencies to be able to use food aid sites to offer FP/RH services and information along side growth monitoring and vitamin A distribution and routine immunization programs. 15. Given that the 44.9% share of the commercial contraceptive market occupied by the CSM project, the future work plan should finance pharmacist/drug shop seller training and closely coordinate with DKT’s Ethiopia’s “Young Marketers” program. 16. Tighten up IEC/BCC component of the overall project. Ensure that evidence-based message development and audience impact studies are carried out to ensure behavioral change. Task a subcontractor in the PI/E consortium to develop a strategy for mass media. This should be an early priority if this is an important component of the MCHN project. 17. Task a member of the PIU/E consortium, to develop a clear adolescent services strategy in conjunction with the MOE, MOH and Ministry of Youth which addresses both in- 41 school and out of school youth, high risk and low risk youth. The strategy should identify an initial target population for the first two years of the project, and suggest the most cost effective ways to reach this target groups. PI/E should realign its FP/RH youth programs accordingly. 18. Task the contractor with coming up with a key list of operations research topics for the first year of the project. Topics such as the community’s ability to pay financial and or in-kind incentives for “volunteer” workers should be a priority. 42 APPENDICES A: Scope of Work B: Persons Contacted C: Interview Instruments D: Team Field Visits E: References F: Pathfinder International Comments 43 Appendix A: Scope of Work 44 Draft Statement of Work (SOW) for End-of-Project Evaluation of the Family Planning Reproductive Health (FP/RH) and Extending Service Delivery for RH/FP (ESD) Projects (Draft # 6, May 12 2008) PROJECT IDENTIFICATION DATA 1. Project Title: Family Planning Reproductive Health (FP/RH) 2. Project Number: 663-A-00-02-00385-00 3. Project Dates: October 2002-September 2007 4. Project Funding: $28,369,486 5. Implementing Organization: Pathfinder International/Ethiopia 6. Cognizant Technical Officer (CTO): Anita Gibson, Dr. Kidist Lulu 1. Project Title: Extending Service Delivery for RH/FP (ESD) 2. Project Number: GPO-A-00-05-00027-00 3. Project Dates: October 2007 - September 2008 4. Project Funding: $1,054,437 5. Implementing Organization: Pathfinder International/Ethiopia 6. Cognizant Technical Officer (CTO): Anita Gibson, Dr. Kidist Lulu I. Identification of the Task The USAID/Ethiopia (USAID/E) Health, AIDS, Population and Nutrition (HAPN) Office requests technical assistance from the Mitchell Group (TMG) under the USAID/E Evaluation Contract number 663-C-00-08-00409-00 to design and implement an independent external end-of-project evaluation of the FP/RH Project and the extension of the project for one year, the ESD program. The overall project objective is to strengthen and increase the capacity of the public sector and local NGOs to expand quality, gender – sensitive reproductive and child health services and HIV/AIDS prevention by promoting an enabling community environment for informed decision making. The program is designed to provide high impact family planning and reproductive health and maternal health services within areas that account for more than 85 percent of Ethiopia’s population. This external end-of-project evaluation will first and foremost, make recommendations for FP/RH activities to be included in work plans for the follow-on Family Planning/Maternal Newborn and Child Health (FP/MNCH) Program and document important lessons learned concerning future FP/RH staffing and skill mix, future program sustainability and NGO/public sector partnership models. The USAID/E HAPN office requests that the in-country activities for this evaluation be completed by June 18, 2008 in order that the findings, conclusions and recommendations can be used to inform the work plan for the follow-on Family Planning/Maternal Newborn and Child Health (FP/MNCH) Program. 45 Selected Ethiopia FPRH and Health Indicators National Surveys: EDHS 2000 EDHS 2005 % Married Women Use Modern FP 6% 14% Total Fertility Rate 5.9 5.4 Total Desired Fertility Rate 4.9 4.0 Unmet need for FP 36% 34% % Married Women Want to stop or delay* 68% 77% Under-five mortality rate 166 123 II. Background Since 1993, the Ethiopian Government has initiated steps to improve the reproductive health (RH) status of women. Various policies have endorsed the role of RH and family planning (FP) services toward meeting the development objectives of the country and have called for the expansion of these critical services. In Ethiopia, the levels of maternal and infant mortality and morbidity are among the highest in the world (EDHS, 2005) with an estimated 21 % of deaths among women aged 15-49 related to pregnancy or pregnancy-related causes. Utilization of RH care services in Ethiopia is low and varies substantially by women’s place of residence, educational level and religion. Nationwide, only 6% of women receive delivery assistance from a health professional (EDHS, 2005). Early marriage is widely practiced, exposing young women to premature and prolonged childbearing and poor health outcomes. With an estimated 78 million people, Ethiopia is the second most populous nation in sub￾Saharan Africa with an annual population growth rate estimated at 2.5%, adding almost 2 million people every year (World Bank, June, 2007; PRB, 2005). The population is projected to increase to over 117 million by 2025. Many of the health problems of women and children are related to high fertility, with an average of 5.4 births per woman in 2005 (EDHS, 2005). In addition, almost half of the total population (44%) is under the age of 15; large numbers of individuals will be of reproductive age in the near future. A young population, combined with high fertility, limited access to FP, and low contraceptive usage not only predicts rapid population growth for at least another generation, but will exacerbate Ethiopia’s poor maternal health and place extra stress on the overall development process. Fertility in rural areas is almost twice as high as urban areas, and 85% of Ethiopia’s population lives in rural areas, primarily in the Highlands, which are characterized by subsistence rain-fed agriculture with depleted soil and little access to agricultural inputs. While the national modern contraceptive prevalence rate among married women remains very low at 14%, it more than doubled in the past five years (EDHS 2000 and 2005). In the four regions receiving direct USAID support, which contain about 85% of Ethiopia’s population, the increase in the prevalence of modern contraceptive use among married women was dramatically higher (two- and three-fold in three regions) compared to 46 Ethiopia’s other seven regions (increases of 53% or less). USAID supports government plans to strengthen district level health management capacity and promote rural outreach in FP service delivery through community based programs. While there have been major advances in access to basic community based health services, continued expansion is needed. Despite clear indications of need, access to quality RH and FP services is limited. Only 10% of women who are not using FP reported discussing FP with a field-worker or at a health facility in the last 12 months (EDHS 2005). In response to the human resources crisis in the health sector and the inaccessibility of health services to many Ethiopians, the Ethiopian government has launched a new program for the “Accelerated Expansion of Primary Health Care Coverage” with the health extension program (HEP) as its centerpiece. This program calls for the training and deployment of more than 30,000 female health extension workers, official GOE civil servants with salary and benefits, for more than 15,000 health posts and the construction and upgrading of 3153 health centers by 2009. These new, government-deployed HEWs (community health extension workers) work to varying degrees with the many thousands of non-civil service community family planning workers and other volunteer health workers who have been in place, and been provided supportive supervision, since 2005 and 2006. The main objective of the HEP is to improve access and equity to preventive essential health interventions provided at the kebele and household levels with a focus on sustained preventive health actions and increased health awareness. HEP also serves as a mechanism for shifting health care resources from a dominant urban focus to rural areas where the majority of Ethiopians live. The successful implementation of HEP is central to the achievement of the Millennium Development Goals (MDGs). To date, almost 25,000 of the planned 30,000 HEWs have been trained and deployed throughout the country. HEWs focus on sixteen packages broadly categorized into four areas: family health care; major communicable diseases prevention and control; hygiene and environmental health; and, health education/communication. Service delivery at the community and household level includes: Antenatal Care, Immunization, Family Planning and Malaria Prevention and Control. The work of HEWs is enhanced through partnerships with other community health volunteers trained with support from USAID, such as Community Health Promoters (CHPs) under the Ethiopia Child Survival Project - Essential Services for Health in Ethiopia (ESHE), and Community-Based Reproductive Health Agents (CBRHAs) under the Pathfinder International/Ethiopia (PI/E) FP/RH Project. These different cadres of volunteers are part of the communities they serve and have demonstrated success in supporting communities to adopt healthy behaviors. III. PI/E FP/RH Project Overview and Background FP/RH Project overview: The PI/E FP/RH Project is central to USAID/Ethiopia’s investment in Health, Population and Nutrition. Awarded to PI/E as a $28 million dollar 47 five-year cooperative agreement from 09/2002-09/2007 (and extended for one year through 09/2008), the project focuses on providing selected FP and selected RH services and in providing health care delivery in five regions representing more than 85% of the Nation’s population: Amhara, Oromiya, Tigray, The Southern Nations, Nationalities and Peoples Regions (SNNPR) and in Addis Ababa. The backbone of PI/E’s work is a broad network of Community-Based Reproductive Health Agents (CBRHAs) who provide FPFH information (and non-clinical contraceptive methods) at the community level, and make referrals to health facilities. The PI/E FP/RH Project currently supports more than 10,000 CBRHAs in more than 300 districts (woredas). FP/RH Background: In September 2002, USAID awarded a five-year cooperative agreement to PI/E and its partners, the Johns Hopkins University/Center for Communication programs (JHU/CCP) and the National Committee on Traditional Practices of Ethiopia (NCTPE) to support the Ethiopia Family Planning and Reproductive Health (FP/RH) Project (The text in this section adapted from FP/RH Expansion Amendment 11, September 2005). The Project focused on providing integrated family planning and selected reproductive health services, improving health care delivery, as well as the provision of HIV/AIDS Care and Support services. The Project supported the provision of integrated services with a primary focus on family planning, which are offered through facility-and community-based delivery systems. The major emphasis was on the delivery of FP/RH information and services at the household level and referral linkages to facility-level service provision. When Ethiopia was named a U.S. Presidential Focus Country for HIV/AIDS, a USAID management decision was made to delete HIV/AIDS care and support from the PI/E Cooperative Agreement, and leave the prevention component only. The amendment to remove HIV/AIDS care, and orphan support went into effect July 13, 2004 with all funds for HIV/AIDS expended by September 30, 2004. As of 2005, the ceiling of the cooperative agreement was $22.5 million. As explained below, it was subsequently increased to $28 million based on findings from a mid-term evaluation. The FP/RH Project has implemented project activities related to the following eight program outcomes as defined by USAID, as follows: 1. Improved health of families at rural level by providing information and referral services on Maternal and Child Health (MCH); 2. Increased Contraceptive Prevalence Rate (CPR) through the use of diversified alternative service delivery approaches; 3. Prevention and control of STI/HIV/AIDS including stigma and discrimination reduction; 4. Enhanced post abortion care (PAC) services through facility and community level support; 5. Improved quality of reproductive health services through training, material development and facility support; 6. Strengthened community, woreda, zonal and regional capacity to develop, manage and implement community based reproductive health services; 7. Improved community capacity to develop and manage community-based health services; and 48 8. Gender issues addressed through women's empowerment, male involvement and the prevention of harmful traditional practices. The Project enjoyed a very rapid start up and the results from first half of the project period (October 2002 - June 2005) were very encouraging (see the September 2005 FP/RH Expansion Amendment document for a summary of accomplishments during this time period). FP/RH Mid-term Evaluation: USAID/Ethiopia commissioned a mid-term evaluation in March 2005 to assess the Pathfinder Project's progress to date and to make recommendations for changes as needed in project direction. The evaluation team was comprised of 16 persons, including representatives from the Global Health Bureau of USAID/Washington, Ethiopian representatives from Ministry of Health (MOH), Ministry of Finance and Economic Development (MOFED), the National Office of Population, and the Regional Health Bureaus of the project's four focus regions, and independent consultants from the U.S. and Ethiopia. The evaluation methodology included briefings with USAID/Ethiopia project managers, Pathfinder staff and Ministry of Health leaders in Addis Ababa, and field observations in the four Project regions. The evaluation team concluded that the Pathfinder Project was succeeding in meeting its objectives. Problems and shortcomings existed, (See the outcome sections of the 2005 FP/RH Mid-Term Evaluation Report), but the Project was clearly having a positive impact on contraceptive use and on improved health-seeking behaviors in the communities where the Project was active. Additionally, the Project contributed to USAID/Ethiopia's strategy by building community resilience and capacity to respond to issues that affect community survival, including the Project's positive effects on the health of women, children and families. Moreover, the project was helping to lay down a solid platform for longer-term cooperation between USAID, the GOE and other donors, which was viewed as essential to significantly impact family planning and reproductive health outcomes in Ethiopia. While the Project could claim early and significant success, it still represented only a beginning to a long and costly effort to make basic FP/RH services available to Ethiopia's mostly rural population. The evaluation team therefore proposed a general set of recommendations for near-term changes to strengthen the strategic impact of the FP/RH Project. September 2005 Amendment to Expand FP/RH: In response to the evaluation recommendations, USAID/Ethiopia increased Project funding by $ 5,869,486, raising its award ceiling from $22.5 million to $28,369,486, and developed an amendment to the Project to address six areas for improvement: 1. An intensification of current project activities that includes more Community Based Reproductive Health Agents (CBRHAs), and the related activities for training, monitoring and supervision; 2. More training of health workers in long term and permanent methods; 49 3. More health workers trained in post abortion care; 4. Purchase of equipment for the training of Health Extension Workers in reproductive health and family planning and for community orientation and planning; 5. Annual documentation (success stories) of all activities undertaken as part of reproductive health; and 6. Minor upgrading of health facilities and procurement of medical equipment and supplies for these facilities. Pathfinder developed activities to respond to all the above six areas for improvement as recommended on the basis of the mid-term evaluation and revised the program accordingly (see the Sept 2005 Program Amendment for details). Pathfinder developed programs to intensify CBRHA Services through a) increased numbers of CBRHAs, b) strengthening the Health Extension Program (HEP) by supporting Technical Vocational Education (TVET) schools, c) introduction of Champion Community (Kokeb kebele) Initiative to link health and education programs at the community level, d) strengthening monitoring with the Integrated Supervision and Quality of Care (ISQOC) system and capacity building for monitoring for IPOs and e) strengthening Pathfinder International/Ethiopia’s Regional coordination offices (see the Sept 2005 Program Amendment for details). Extending Service Delivery Agreement (ESD): The 12-month ESD agreement to extend the original FP/RH program was instituted as a bridging activity for the FP/RH Program, which formally ended in September 2007, while USAID/Ethiopia explored longer-term alternative program design options for FPRH delivery in the context of maternal, newborn and child health services (MNCH). The intent of the ESD is to ensure continuity of high impact family planning, reproductive health and maternal heath services in the four regions, two woredas of Benshangul Gomuz and Addis Ababa and Harari Regions (See the ESD Program description, dated 12-2006). The ESD Program has six expected outcomes: 1) improved access to quality FP services and increased use of modern methods, 2) strong links between health facilities and HEWs, 3) improved access to PAC, 4) services, integration of FP into HIV/AIDS services, 5) improved access to youth clubs, and 6) improved access to fistula repair services. Funded at $1,054,437, the ESD supports a wide range of activities in ten areas: 1) FPRH services within the CBHR program with emphasis on long term and permanent methods; 2) The integration of FP/RH with HIV/AIDS programs; 3) Capacity building for health workers to provide quality PAC services; CBHRAs provide health education referral for PAC; 4) Support for Youth Clubs; 5) Fistula identification and referral for repair; 6) Prevention of Child Marriage and other Harmful Traditional Practices (HTM); 7) Capacity building for health service providers, WACs and local NGOs; 8) IEC activities; 9) Integration of FP/RH services in the workplace; and 10) continued involvement in Kokeb Kebele initiatives. FP/RH and ESD Strategic Framework: When the original PI/E FP/RH Cooperative Agreement began in 2002, the USAID Mission had as its Strategic Objective, “Improved Family Health.” Intermediate Result (IR) number 2 was, “Increased use of high impact 50 reproductive health interventions, including maternal nutrition in focus regions and target areas nationwide. There were three sub-IRs: • Increased access to modern family planning services, • Improved quality of family planning services, and • Increased access to key maternal health and nutrition services. In February 2004 after helping to address the famine of 2002-2003, the USAID Mission to Ethiopia developed a new strategy to manage the transition from an emergency response-dominated program to one which proactively builds capacity, strengthens economic development and to increase the capacity to manage through shocks. Under the new strategy, health and education were combined into one Strategic Objective, SO 14, Human Capacity and Social Resiliency Increased. Intermediate Result, 14.1 includes family planning, “Use of high impact health, family planning, and nutrition services, products and practices increased.” Under IR 14.1 there are four sub-IRs: • IR 14.1.1: Community support for high impact health interventions increased, • IR 14.1.2: Availability of key health services and products improved, • IR 14.1.3: Quality of key health services improved, and • IR 14.1.4: Health sector resources and systems improved. The FPRH and ESD monitoring and evaluation indicators address all four of these sub￾IRs. The 2007 Foreign Assistance Framework: In 2007, the SO14 was incorporated into a new Foreign Assistance Framework (F-Framework) for the USAID 2007 Operation Plan. The activities under the FP/RH and ESD projects now fit under the F-Framework Priority Objective: Investing in People, Program Area: Health, Program Elements: MCH, FP/RH, HIV/AIDS; Program Sub Elements: Service Delivery, Communication, and Policy Analysis and Systems Strengthening. Additional applicable program sub￾elements include: Birth Preparedness and Maternity Services and Treatment of Obstetric Complications and Disabilities. For the remainder of the implementation of the FPRH and ESD projects, however, the monitoring and evaluation indicators continued to be reported as before, arrayed under the four SO 14.1 sub-IRs listed above. FP/RH and ESD Monitoring Indicators: The FP/RH reported on a total of 69 indicators arrayed under the major headings of the four IRs: • IR14.1 Use of high impact health, FP and nutrition service, products and practice increased: (40 indicators) as follows, Family planning (3 indicators), MCH (2 indicators), IE/BCC activities (7 indicators), ASRH (7 indicators), Integrated HIV/AIDS/STI activities (4 indicators), PAC Services (2 indicators), Gender Issues (15 indicators), • IR14.2 Availability of Key Health Services and Products improved: (11 indicators), • IR14.3 Quality of Key Health Services Improved (8 indicators), • IR14.4 Health Sector Resources and Systems Improved (10 Indicators). 51 Based on the project agreement document, the one-year ESD project is expected to report on a total of 35 indicators under six headings: FP/RH (17 indicators); Adolescent Youth Reproductive Health (6 indicators), MCH (3 indicators), HIV/AIDS Service Integration (7 indicators), Harmful Traditional Practices (2 indicators) and System Strengthening (2 indicators). In practice, however, the ESD reports with the same indicators as FP/RH, arrayed by the four IRs shown above. See the workplan matrix for the 2007-2008 ESD Workplan for a detailed summary of all 69 indicators. For a recent summary of FP/RH Project results based on the above indicators, see the table below for five-year FR/RH Project close-out report results arrayed by the eight program outcomes. 5-Year FP/RH Target and Achievement for Selected Outcome Indicators Indicator Outcome Indicator 5-Year Target Achievement Source of Data (See RH/FP Closeout Report, January 2008) Outcome 1 Children Referred & Received Health Care 4,044, 418 6,644,466 IPO Reports Outcome 2 New FP clients 2,671,576 3,698,165 IPO Reports Outcome 3 Number condoms distributed NA 45,896,849 IPO Reports Outcome 4 Number PAC services provided 21,373 37,718 IPO Reports Outcome 5 Number community based workers trained 9,000 10,112 IPO Reports Outcome 6 Number of Kebeles agreeing to have CBRHAs work in their community 6,000 6,315 IPO Reports Outcome 7 Number of PACs/WACs established 247 200 IPO Reports Outcome 8 Community members & major target groups informed on HTPs 7,794,844 10,749,113 IPO Reports The new USAID/Ethiopia Family Planning/Maternal, Newborn and Child Health Program (FP/MNCH) Program: USAID currently supports two highly effective programs that run largely in parallel: the Ethiopia Child Survival Project - Essential Services for Health in Ethiopia (ESHE), which ends in September 2008 and has demonstrated significant results for child health services, and the FP/RH Project. In many instances, these two major programs work in the same regions, zones and woredas (See HAPN regional briefing documents for Amhara, Oromia, and SNNP for detailed summaries of ongoing ESHE and FP/FH activities). Recognizing an urgent need to increase attention for preventive services for newborns and in hopes of developing better coordination, USAID/Ethiopia has developed a new strategic approach to combine both programs into one. The new program will focus on health and more specifically, the elements of: family planning/reproductive health, maternal and child health including malaria. The new FP/MNCH Program is to be an integrated package of assistance for family planning, maternal, newborn and child health. Investments will directly support the GOE’s third Health Sector Development Program (HSDP III) and the HEP with a focus 52 on the delivery of key services and products through a continuum of quality care from the health center to the health post and community level. Community mobilization and empowerment to change health behaviors and the engagement of households and communities as partners in the health care system are central to the successful implementation of this new program. Geographic Scope and Program Coverage of FP/MNCH: Building on prior USAID investments in RH/FP and child health, the new FP/MNCH program will work primarily in Oromiya, Tigray, SNNP and Amhara regions. RH/FP services will continue to be offered in current USAID-supported woredas while new activities in maternal, newborn and child health will be integrated into existing RH/FP Project efforts. The new FP/MNCH program will reach existing woredas supported by FP/RH Project-Ethiopia (about 300) and existing woredas supported by ESHE (totaling 101). With the exception of Tigray, where ESHE does not provide services, most of the ESHE woredas are also covered by the FP/RH program. It is expected that geographic scope and integrated program coverage in these four major regions will expand over time. Emerging regions may be considered but coverage of the four major regions is the primary priority. The program will focus on rural and hard-to-reach populations and peri-urban areas (particularly where health centers are placed) with limited support in urban areas of the country, particularly in Addis Ababa. Diverse funding sources: The FP/RH Program’s USAID-funded activities are complemented by additional financial and material support from the David and Lucile Packard Foundation and the Swedish International Development Agency (SIDA). These non -USAID funds help to fill gaps for activities directly supported by USAID, such as women’s empowerment and for activities not supported by USAID in HIV/AIDS care and support services. The entire FP/RH program is implemented through 46 indigenous Implementing Partner Organizations (IPOs) including the MOH, through sub-granting arrangements with Pathfinder International’s Ethiopia country office. Seventeen of the 46 IPOs are currently funded by USAID through the ESD. These 17 IPOs cover more than 250 out the 300 woredas served by PI/E programs. The remaining 29 IPOs are funded by Packard and SIDA. Other Background Factors: The following issues should be taken into consideration in the course of the evaluation. • Favorable policy environment: In 2006, the Ministry of Health (MOH) developed a National Reproductive Health Strategy for the period 2006-2015 (MOH, 2006). This comprehensive document covers six priority areas, namely: the social and cultural determinants of women's RH; fertility and family planning; maternal and newborn health; HIVIAIDS; RH of young people; and reproductive organ cancers. The national RH strategy was complemented in 2007 by another MOH policy document that addresses the RH rights and needs of adolescents and youth (MOH 2007). In addition, the benefits of family planning programs for addressing development and poverty reduction concerns of the country have recently been underscored in the national “Plan for Accelerated and Sustainable Development to End Poverty” (PASDEP)-2005/06-2009/10. 53 • The problem of measuring impact: Due to the lack a baseline and end-line data for the FP/RH project, it is difficult to measure its impact. While the 2000 and 2005 EDHS data show important favorable changes on key indicators, such as CPR, it is not appropriate to attribute these changes to a particular project though if disaggregated data by woreda is available in prior or planned EDHS, it is possible to make some inferences about change in an assisted and non-assisted woredas. The mission might want to consider oversampling in key regions in the next EDHS with augmented question sets so that it will be possible to have both program and counterfactual data, e.g. what happened in woredas where there were program interventions and those where there was not.. In its interviews with key informants and field reviews, the evaluation team should seek information on and describe the relative direct contribution of the FP/RH project in terms of additional resources brought to bear on FP/RH services over the life of the project and the other possible factors such as female literacy rates, rising incomes and increased GOE spending that have led to improved FP/RH outcomes.. • High turnover of staff: High staff turnover is a recurring theme among all volunteer and salaried health personnel in Ethiopia, not just the FP/RH project. To the extent that information is available, the evaluation team should explore any linkages between recent turnover and the health extension program (HEP) e.g. has service as a CBRHA enabled women to improve their status and become paid health extension workers or conversely discourage volunteerism because others are being paid to carry out similar functions. IV. Purpose of the Assignment The purpose of this assignment is to conduct an external end-of-project evaluation to make recommendations for the FP/RH components to be included the work plan for the new FP/MNCH follow-on program. The evaluation will document key lessons learned related to staffing FP/RH programs, program sustainability and NGPO/public sector partnership models which can be applied in the new follow-on program and reinforce GOE sector health sector objectives. The evaluation will include USAID/Washington, USAID/Ethiopia and GOE staff and three key TMG team members: 1) a senior expatriate Team Leader (29 days), 2) senior or mid-level expatriate FPRH expert (29 days) and 3) a local senior FPRH expert (24 days). The evaluation will cover the FFRH/ESD program performance period of September 200523 to through April 2008. This evaluation will: • Make recommendations on how best to sustain and increase sound FPHR services within the new FP/MNCH Project and assure greater contraceptive security. 23 The start date of September 2005 corresponds to Modification Eleven, when the FP/RH Project description was modified to reflect six recommendations for improvement based on the 2005 mid-term evaluation, the budget was expanded by $5.8 million and the project completion date was extended to September 30, 2007. 54 • Give priority to an in-depth assessment of community based strategies for delivery of FPRH services, with an emphasis on the role of the CBRHA within the FP/RH and ESD programs , the contributions of the IPOs and the links between health facilities and HEWs • . Assess progress to date in improving access to quality FP services and increased use of modern FP/RH methods, especially long term and permanent methods and including improved access to PAC services. Caveats: Two important caveats need to be considered during this evaluation. • First, by documenting the realities of CBRHA activities in the field, this evaluation will inform the design of the FPRH components of new FP/MNCH strategy. The findings of the evaluation will support efforts to do this design work, but the evaluation team is not asked to design the FPRH components for the new FP/MNCH work plan or strategy. • Second, the evaluation should not subscribe to any underlying assumptions that the CBRHA cadre will or should remain in its current role. It should be neutral on the role of the CBRHA. Sam and Anita – We are puzzled by this caveat. Don’t you want the team to consider and make recommendations about whether USAID support of community workers under the new FP/MNCH Project seems important to complement the work of the HEWs?Harriett The evaluation will analyze three overarching questions while documenting challenges, lessons learned, successful interventions and significant products and tools and best practices from the FP/RH ESD for possible dissemination and replication that can inform the new follow-on program development : 1) Future FP/RH support and staffing as it relates to effective outreach and provision of essential FPRH services, e.g. both geographic and program coverage, service quality and desired client response The role of the CBRHA • What services have the CBRHAs been providing? Are there differences in their performance in the three different models or in different geographic regions? What services do GOE program managers, community leaders and clients see as the most important or lacking? What is the most appropriate role of the CBRHA in the new context of the expanded HSEP and the recruitment of 30,000 HEWs? Is there added value of CBRHAs in the presence of HEWs? If so, what functions are most important for the CBRHAs. • How is the CBRHA information management system linked to the national HMIS? 55 • What is the current relationship among/between CBRHAs, HEWs, and other community health workers including those supported under the ESHE and other donor supported projects? How can these relationships and roles be adjusted to meet the needs of the new FP/MNCH program? • What are the strengths and weaknesses of systems for CBHRA referrals and linkages to health centers, health posts, and other primary health care units? • Have the FP/RH Project and the ESD program adequately addressed CBRHA turnover issues? • How are PI/E FP/RH supervisory activities linked to GOE FPRH programs? • What are the key successful elements of the PI/E FP/RH supervisory system? How can they be integrated into the new FP/MNCH program? • Has the FP/RH Project and ESD program adequately addressed staff (not just CBRHAs) turnover issues? What approaches need to be used for the new FP/MNCH program to address this critical manpower issue? 2) IPO/Public Sector Partnership Models The Role of Implementing Partner Organizations (IPOs) • Compare the relative success of three different FP/RH Project partnership models: 1) working primarily with IPOs, 2) working primarily with the MOH and RHBs, and 3) working with MOH supported NGOs. • Assess best practices for program performance among the above three partnership models. 3) Sustainability Cost Recovery • Can sustainable alternative sources be found to cover recurrent costs, such as CBRHA transport expenses; • Fee- For-Service: In areas served by initiatives for cost recovery, does fee-for￾service impact uptake and how are funds used? Are there lessons learned from Packard and other groups that could be of use to the new FP/MNCH program? Program Maturity and Sustainability • What systems need to be in place to meet unmet need for family planning and related services? • Based on past FPRH experience and achievements, what are the most promising approaches to emphasize toward program maturity and sustainability? • How can these approaches be incorporated into the new FP/MNCH program? • Given the presence of FPRH TA in some woredas for more than 10 years, are there any woredas where the FPRH service delivery can be sustained with reduced support or without continued support? If such areas exist, how are they identified. 56 Capacity Building • Is the project’s approach to training and capacity building sustainable? Which components, if any, should be built into the new FP/MNCH program? • Assess FP/RH and ESD contributions to improving the FPRH policy environment and the implications for sustainable of FP/RH programs in Ethiopia Is the section below needed since these issues appear to be covered above?.) • Special emphasis is needed to assess the following four expected outcomes from the ESD project to determine what progress has been made, if the interventions have been adequate, and their implications for the new FP/MNCH program: i. Improved access to quality FP services and increased use of modern methods, especially long term and permanent methods (subset of original FP/RH Outcome Number 2), ii. Strong links between health facilities and HEWs (subset of original FP/RH Outcome Number 6) iii. Improved access to PAC services (subset of original FP/RH Outcome Number 4) iv. Integration of FP into HIV/AIDS services (subset of original FP/RH Outcome Number 3). • Special emphasis is needed to assess progress and adequacy of interventions for the prevention of early marriage, per Outcome 8, Gender issues addressed through women's empowerment, male involvement and the prevention of harmful traditional practices. This is information should be collected in a population-based survey such as the EHDS or KAP survey. Since it is unlikely that we could valid information through review of documents, key informant interviews and field visits, we would recommend deleting this from the scope of work. . • . • Have the interventions in FP and RH been adequate to improve access to quality services? • What can be done to ensure that the gains for “high impact FPRH services” within FP/RH/ESD are not lost within the new FP/MNCH Program (this will be addressed in the sustainability section and the staffing)? V. Evaluation Methods The evaluation will be carried out by a core team of three independent, external consultants along with team members from USAID/W and Ethiopia and the GOE over a three-week period using multiple methods, including key informant interviews, field observation, and a review of FP/RH ESD reports, tools, and materials. In addition to the two FPRH experts from USAID/Washington and USAID/Ethiopia staff, these 57 consultants will be joined by at least four representatives from the GOE MOH, MOFED, and the National Office of Population, at the National and regional level. The team will interview key informants in USAID, in the GOE at the federal, regional, zonal and district level, the donor community, implementing partners including both contractors and NGOs and service providers and clients, The team will draft protocols for collection of primary data when in the field to observe the program through direct observation, interviews with clients, health providers, Woreda Health Committee members, CBRH Agents, Regional Health Bureau Staff, etc. With the help of USAID/E and GOE colleagues, the evaluation team will select regions and zones purposefully to address key program questions, such as the comparison of PI/E’s different operational modalities (MOH, MOH affiliated NGOs, and IPOs/local NGOs). These and Woredas and woreda-level service delivery sites should be chosen in a way that reflects the three overarching questions and provides an opportunity to : 1) talk to GOE officials at all four levels, service providers and clients; 2) observe the current relationship among/between CBRHAs, HEWs, and other community health workers including those supported under the ESHE and other donor supported projects provides an opportunity 3) compare the relative success of three different FP/RH Project partnership models 4) assess the contribution of a representative sample of implementing partners (?just USAID –assisted or others? 5) observe differences in performance in different settings (Are there important regional or zonal differences that need to be reflected in the choice of site visits?)or programs of different maturity and ensuring that not just the best performing programs are visited 6) see different approaches to cost recovery including the Packard and other models the mission believes important and more generally contributing to long term sustainability. Up to four sub-teams will each travel to one or more of four regions: Amhara, Oromia, Tigray and SNNP in collaboration with PI/E FP/RH and USAID/E and GOE colleagues. Activities will include: • Review of relevant documents. • Process evaluation (interviews and document review). • Key informant interviews – meetings and conference calls with FP/HR Project staff and central and regional offices, FP/RH IPO staff, donor agencies, and other stakeholder agencies in the FPRH service delivery field. • Field visits to selected Regions, Zones and Woredas. Given the significant regional differences in program implementation, the team may split-up into up to four groups order to accomplish this task; it is anticipated that each team will be out in the field for seven days. • Skills assessment of selected CBRHAs. • Key informant interviews with Government of Ethiopia Ministry officials at all levels, National, Regional, Zonal, Woreda and kebele, as well as service providers and diverse FP/RH clients. • Evaluation of the type, amount and quality of clinical skills development experience provided to CBRHAs 58 • Skills competency assessment, expert observation using a standard checklist for selected skill areas for CBRHAs. • Data analysis of training numbers, retention rates, among CBRHAs trained by region and IPO. Specific evaluation techniques may include: surveys, interviews using structured and semi-structured questionnaires, focus group discussions (FGDs), and skills assessments for various skill sets. Diverse approaches will be used to assess CBRHAs, including their actual skills. Data collection instruments may include facility assessment forms, and competency checklists for CBRHA skills and skills simulation and case studies to assess CBRHA competence. Are there GOE or project performance standards for CBRHAs and will we have someone on the team from the GOE who can assess CBRHA performance and skills vis a vis the GOE standard? Questionnaires may be administered to assess CBRHA self-reported readiness to practice and their training experience in FP/RH Project programs. The core-team will develop an evaluation workplan that explains how the evaluation questions will be addressed with examples of draft data collection instruments. VI. Information Sources Consultants will be provided the following background documents in preparation of the assignment: • FP/RH and ESD Cooperative Agreements, including modifications • FP/RH and ESD Annual Reports • FP/RH and ESD Quarterly Reports • FP/RH and ESD M&E Tools • FP/RH Close-out Report • FP/RH Mid-term Evaluation Report • USAID/Ethiopia-Washington Assessment of FP/RH Services March 2006 • ESHE Evaluation January 2008 • 2000 and 2005 EDHS and any other more recent KAP or Reproductive Health Surveys • Recent PI/E FP/RH end of project assessment reports re CBRHAs, Training for Long-term and Permanent Methods, Quality of Care, Training Impact, etc. • Various FP/RH assessments (2004 KAP Survey, WAC, IUCD, CBRH assessments and reports) • Various FP/RH Guidelines and manuals (WAC Guidelines, WAC Took Kit, PAC Manual, Guidelines on Counseling, Service Supervision Checklists, M& E Forms) • FP/RH Data on Service Delivery Sites by Region, Zone and Woreda • Maps of FP/RH service delivery sites by Region, Zone and Woreda • USAID trip reports summarizing past field visits to FP/RH and ESD sites. 59 • GOE strategic documents related to Reproductive Health, Adolescent RH, PASDEP III, Child Survival, the Health Services Extension Program and Accelerated Health Officer Training Program • Pertinent World Bank and UNICEF documents • GOE Road Map for HIV/AIDS Prevention, Care and Treatment • Pathfinder map of project sites; GIS map of woredas by Region and Zone. VII. Tasks to be accomplished: Below is a list of the specific tasks to be accomplished by the consultant team, with an estimated level of effort for each task. See Attachment 1: Planning Calendar for the schedule. Key Activities Level of effort in days Team Reviews background documents, initial discussions with TMG Coordinator, develop draft evaluation methodology/ and field visit and interview schedule in consultation with CTO and Evaluation Coordinator. 3 Team planning meeting in TMG Washington. Review Evaluation methodology, conference calls to CA HQ, development of outline of report, Team develops Draft Work Plan. Draft work plan to be sent to HAPN, USAID/Ethiopia for review. 2 Travel Days for Team to Addis 1 In country in-brief consultation with CTO, Evaluation Coordinator and HAPN, Team discusses and submits Draft Workplan to USAID/E. Workplan approved based on HAPN comment. 1 MOH and other stakeholder meetings, key informant interviews. 2 Full Team (Core Team plus GOE expert participants) Team Planning Meeting (TPM) 2 Up to four sub-teams conduct field visits and interviews 8 Full team reconvenes to analyze and synthesize findings 2 Core team analysis, report writing and prepare for debriefings. 4 Conduct debriefings for USAID and FP/RH ESD stakeholders (separately)/ submit draft report to USAID/E in country 1 Travel Day for international consultants 1 USAID/E Review draft report Up to 10 Days Finalize Report – Response to USAID/E comments: Team leader 60 (2); Team members (1) 2- TMG edits Final Report Up to 3 Weeks Total LOE is estimated at 29 days for the Team Leader and up to 29 days for other Team Members and 24 for the local FP/RH expert, including two travel days each for the two expatriate team members. A six-day work week is authorized for work in Ethiopia. VIII. Team Composition and Participation USAID/E seeks three core team members: 1) a senior expatriate Team Leader (29 days), 2) an expatriate senior level FPRH expert (29 days), 3) a senior local FPRH expert with evaluation expertise (24). The three consultants will be joined by two USAIDW and ? USAID/E staff and up to five representatives from the GOE MOH, MOFED at the National and regional level. Local evaluation logistics will be provided by the local sub￾contractor hired by TMG. USAID mission will request that members of PI/E management be available for in-depth interviews, and if possible accompany the evaluation team on site visits. 1. The Team Leader will be an international consultant with extensive FP/RH program implementation and evaluation experience and will play a central role in guiding the evaluation. The consultant with the other USAID and TMG team members will brief USAID/E on arrival, debrief USAID/E and FP/RH ESD stakeholders including the GOE on evaluation findings, and produce a draft report to be left with USAID/E prior to departure, followed by a final report for USAID/E. The Team Leader will: • Play the key role in the technical approach for the assignment • Take particular responsibility for the assessment of the effectiveness of the management, technical assistance and in-country coordination aspects of the FP/RH ESD evaluation. • Discuss and finalize with USAID/Ethiopia the team Work Plan for the evaluation assignment. • Establish assignment roles, responsibilities, and tasks for each team member, including USAID and GOE participants. • Working with TMG’s logistics sub contractor and the assigned local logistics team member, ensure that the logistics arrangements in the field are complete. • Facilitate the in-country Team Planning Meeting (TPM). • Take the lead on preparing, coordinating team member input, submitting, revising and finalizing the assignment report. • Manage team coordination meetings in the field. • Coordinate the workflow and tasks; support team members to work effectively and according to schedule. • Work with the local logistics sub contractor to ensure that team field logistics are arranged (e.g., administrative/clerical support is engaged, ensuring that payment 61 is made for services, car/driver hire or other travel and transport is arranged, etc.). Team Leader qualifications: • An advanced degree (MD, BSN, PhD, MA, MS or MBA) from a reputable accredited institution in Medicine, Public Health and/or any related social sciences pertinent to work on FP/RH issues. • Minimum 10 years of progressively responsible experience with recognized organization(s) in the design, implementation and evaluation of FPRH programs with demonstrated technical expertise and skills in HPN. • Demonstrated strong analytical, managerial and writing skills. . • Exceptional leadership in coordinating, assigning the team with the appropriate responsibilities, communication, and interpersonal skills. • Ability to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. • Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. • Must be able to produce a succinct quality draft Final Report that gives direction to FPRH services within the new FP/MNCH program. 2. The Senior level FPRH Specialist will be an international consultant with extensive FPRH implementation and evaluation experience in Africa. The consultant will be responsible for writing some sections of the report. The consultant will assist the team leader in the development of any qualitative and quantitative instruments to be used during site visits as well as the analysis of any data collected. Consultant qualifications: • MD, BSN, Ph.D., MA, MS, MBA or BA from a reputable accredited institution in Medicine, Public Health and/or related social sciences pertinent to working with FPRH programs. • Minimum 10 years of progressively responsible experience with recognized organization(s) in the design, implementation and evaluation of FPRH programs with demonstrated technical expertise and skills in FP, MNCH in Sub-Saharan African countries. • Demonstration of strong analytical, managerial and writing skills. Able to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGO counterparts. • Must be fluent in English. • Proven ability to communicate clearly, concisely and effectively both orally and in writing. 3. The local Senior FPRH Specialist will be a qualified consultant with extensive FPRH implementation and evaluation experience in Ethiopia. The local consultant will be responsible for writing some sections of the report. The consultant will work with the 62 team to develop needed qualitative and quantitative instruments for the evaluation- these can include guides for informational interviews, data reviews etc, as well as assist in the analysis of data collected. Consultant qualifications: • An advanced degree (MD, BSN, PhD, MA, MS or MBA) from a reputable accredited institution in Medicine, Public Health and/or related social sciences pertinent to work in the field of FPRH. • Minimum 5-10 years of progressively responsible experience with recognized organization(s) in the design, implementation and evaluation of FPRH programs with demonstrated technical expertise and skills in HPN and HIV/AIDS. • Demonstrated strong analytical, managerial and writing skills. • Ability to interact effectively with a broad range of internal and external partners, including international organizations, host country government officials, and NGOs counterparts. • Must be fluent in English and have proven ability to communicate clearly, concisely and effectively both orally and in writing. Non-TMG Team Members: Two or more USAID staff and up to five GOE representatives will join the evaluation team during the team planning meetings, site visits, debriefings, and report preparation. USAID staff and GOE representatives are expected to participate in the evaluation under the guidance of TMG Team Leadership. USAID staff team members are encouraged to participate for the full duration of the evaluation as part of the Core Team. In recognition of the difficulty in take time away from their regular duties, GOE representatives are asked to participate for a shorter duration than Core Team. GOE representatives join the Core Team to make up the Full Team for: 1) up to two days of initial Full Team planning meetings in preparation for field visits, 2) participation in Full Team field visits, and 3) participation in up to two days of Full Team analysis and synthesis of site visit findings. The number of days for Full Team meetings before and after field visits should be considered flexible, based on the recommendation of the TMG Team Leadership. PI/E FP/RH Project and ESD Program staff may accompany the team on site visits as appropriate, but will not be present during interviews with stakeholders, or beneficiaries. Evaluation Logistics: Evaluation Logistics will be provided by the local sub-contractor hired by TMG with support staff who are fluent in Amharic and English, with a demonstrated: ability to be resourceful and to successfully execute complex logistical coordination; ability to multi-task, work well in stressful environments and perform tasks independently with minimal supervision; ability to work collaboratively with a range of professionals and others. The local sub-contractor will be responsible for logistics, coordination and administrative support, and ensuring all aspects of the evaluation are carried out seamlessly. Local sub-contractor staff will support the Team Leader and provide coordination and logistics and administrative support for the evaluation in facilitating meetings, providing translators and organizing and supporting site visits. As needed, the local sub-contractor will collect and disseminate background documentation to the evaluation team. 63 IX. Schedule and Logistics The in-country phase of the evaluation will be conducted over a period of approximately 23 days with a desired start date on or about 27 May 2007. See LOE summary above as well as Attachment 1: Planning Calendar for the proposed schedule. The local TMG Evaluation Logistics sub-contractor, in collaboration with the USAID/E Evaluation Coordinator and PI/E, will arrange all of the partner meetings and site visits in advance. Some meeting space may be provided at USAID/E, but the agency cannot provide access to fax and email. All associated travel and per diem costs for non-USAID staff will be covered by TMG under the technical directive with USAID/E. X. Period of Performance Work is to be carried out over a period of approximately seven to eight weeks, beginning on or about (o/a) May 20, 2008 [NB: Dates subject to change] and concluding o/a June 18, 2008 (not including approx four weeks time for USAID/E (up to ten days) comment and completion of final editing of the Draft Evaluation Report by TMG (3 weeks). XI. Financial Plan A budget plan agreement between the USAID/Ethiopia and TMG will be reached and USAID/E will approve the evaluation activity by TMG under the USAID/Ethiopia Evaluation Program. XII. Deliverables Prior to arrival: The Team, working with the TMG Coordinator, will develop a Draft Work Plan with evaluation methodology and field visit and interview schedule in consultation with the USAID/E CTO and USAID/E Evaluation Coordinator. The Draft Work Plan will define the roles and responsibilities of team members, the details of the evaluation methodology and a planned interview and fieldwork schedule. Three days after Team arrival: Team meeting and in-briefing with USAID/E. USAID/E HAPN technical staff to review and comment on evaluation methods. The Draft Work plan will be presented and discussed with Mission Staff for approval. After agreement, the Draft becomes a Final Work Plan. Prior to departure: Team makes presentation to USAID/E HAPN staff, a separate presentation to FP/RH Project and ESD Program stakeholders and GOE stakeholders. The Team leader will submits a draft report, in the format specified by the USAID/E Evaluation coordinator in consultation with TMG (See separate MS Word file for TMG Evaluation Report Guidelines), to USAID/E CTO - two hard copies and one electronic copy on CD ROM or flash drive before departure. 64 After departure: USAID/E has up to ten days (o/a June 28th) to review and send comment on draft report to Team Leader. Team leader with help from the other expatriate team member and USAID/W teams members to the extent possible will have two days to edit or otherwise revise the draft report to respond to USAID/E comments.. The team leader will submit this revised draft report to TMG for submission to USAID/E within five days of receiving comments from USAID/E. The report (not including attachments) will be no longer than 30 pages with an Executive Summary, Introduction, Methodology, Findings, Conclusions, and Recommendations in English in the format specified by the USAID/E Evaluation Coordinator in consultation with TMG in advance of the May 22 TPM in Washington. . Upon final approval of the content by USAID/E, TMG will edit and format the report within three weeks. The final report will be submitted electronically to USAID/E, the CTO and the Contract Officer. TMG will make the results of its evaluations public on the Development Experience Clearinghouse and on its project web site unless there is a compelling reason (such as procurement sensitivities) to keep the document internal. Therefore, TMG will request USAID/E confirmation that it will be acceptable to make this document publicly available. If there are certain restrictions regarding specific parts of the report that should be removed from a public version due to procurement-sensitive information, TMG will develop a second version suitable for public availability. 65 Attachment 1: Planning Calendar as of May 12 Family Planning/Reproductive Health Evaluation Team Monday Tuesday Wednesday Thursday Friday Saturday Sunday May 19 May 20 Document review ( 3 days) May 21 May 22 TPM in DC May 23 TPM in DC May 24 May 25 26 US Holiday Travel 27 Start work in Addis Stake holders meetings In – brief USAID 28 Ethiopian Holiday Document review – team meetings 29 Stakeholders meetings 30 Full Team planning discussions and document reviews 31 Full Team planning, discussions and document reviews June 1 Travel to field 2 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 3 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 4 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 5 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 6 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 7 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 8 Rest 9 Field work - up to 4 teams in up to 4 Regions: Amhara, Oromia, Tigray & SNNP 10 Field work/Travel Back to Addis 11 Full Team data analysis 12 Core team data analysis And writing 13 Core team data analysis and writing 14 Core team data analysis and writing June 15 Rest 16 Data analysis and writing 17 Data analysis and writing 18 Team writing Debrief USAID, stakeholders Submit Draft report Team travels late evening 19 Travel 20 LOE Team leader 29 Ex pat TM – 29 Local expert 24 21 Two weeks later in June -- incorporate response to comments into final report Conference call TMG 22 2 - team leader 2- expat team member 1-- local team member 66 Appendix B: Persons Contacted 67 UNITED STATES John Snow Mary Carnell Senior Child Health Advisor Pathfinder Cathy Solter Director of Technical Services Gywn Hainsworth Technical Advisor Mizanur Rahman Technical Advisor USAID/Washington Jim Shelton Global Health Bureau, Senior Science Advisor Scott Radloff Director, Office of Population and Reproductive health Sue Anthony DCHA/FFP/EP, Senior Advisor Mary Anne Abeyte-Behnke GH/PRH/SDI, Senior FP/RH and HIV/AIDS Integration advisor Linda Lou Kelley Acting Ethiopia Desk Officer Ishrat Husain AFRO/SD/HT, Senior Family Planning Advisor Sharmila Raj GH/PRH/CSL, Technical Advisor Commodities Security and Logistics Mary Ellen Stanton GH/HIDN/MCH, Senior Reproductive Health Advisor Alexandra Todd GH/PRH/SDI, Repositioning Family Planning Advisor Consultant Margaret Neuse Former Director of the Office of Population and Reproductive Health ETHIOPIA/Addis Adaba24 CORHA Dr. Edit Kebede Acting Executive Director Tigest Alemu Former Executive Director DKT Andrew Piller Director Haymanot Assefa Nadew Technical Manager EngenderHealth Dr. Gelila Kidane, Director, EngenderHealth, Ethiopia Essential Services for Health in Ethiopia (ESHE) Dr. Peter Eerens Project Director Dr. Tesfaye Bulto Deputy Director for Child Survival 24 The evaluation team estimates that an additional 50 individuals were interviewed, but their names were not recorded, especially at the household level. 68 Frank White Deputy Director for Finance and Administration Ministry of Health .Dr. Tedros Adhanon Minister of Health Dr. Neghist Tesfay Head of the Family Health Department Ministry of Finance and Economic Development Dr. Genet Mengistu Head, Population Department Mr. Woinstet Nigatu Team Leader of Reproductive Health Pathfinder International/Ethiopia Tilahun Giday Country Director Girma Seifu Finance Administrator and Logistics Team Leader Girma Kassie, Monitoring and Evaluation Program Team Leader Mehari Belachew, Monitoring and Evaluation Officer Abousemed Musa Monitoring and Evaluation and Data Program Officer Lelise Dembi Monitoring and Evaluation Program Officer Metiku Moldegengies, Operations Manager Packard Foundation Sahlu Hailie Director Population Council Mr. Gebeyehu Mekonnen Chief of Party Dr. Tekleab Mekib Senior Consultant USAID/Ethiopia Meri Sinnit Chief HAPN AnIta Gibson Health Team Leader Sam Clark Program Activity Evaluation Coordinator Deneke Kassahun Strategic Information Advisor Misrak Nadew Public Health Specialist Xerses Siahwa IDI Intern Yared Alara FP/RH Expert AMHARA REGION, Awabel Woreda Awabel WAC Desmale Dagne Woreda Health Officer Shiferaw Ayele Administrator Sirenat Assefa Deputy Woreda Health Officer Yitayish Abeje Director, Woman’s Affairs Teshome Tekle IPO Supervisor (Ethiopian AID) Awabel Woreda HEWs and CBRHAs Muluken Assaye HEW, Yegeder Amele Work CBRHA, Amber Enatesh Gashe CBRHA, Amber 69 Tirusew Mekonen CBRHA, Yegeder AMHARA, Dangila Woreda Dangila, WAC Tadele Mekonen Woreda Health Officer Simachew Amare Tesseme Administrator Ardnet Dagnew MoFED Tsehaynesh Yirga Secretary Simachew Kassie Youth and Sport Hulumagerch Tazeb Woman’s Affairs Dangila Woreda HEWs, CBRHAs Yeshalem Yaregal HEW, Zeguda Mignotie Kelelaw HEW, Gumdri Mulu Degarege HEW, Gumdri Fantanesh Gerea HEW, Batcha Kerebish Kinfrech HEW, Batcha Demekech Yimeny HEW, Zeguda Muluken Assaye HEW, Muluken Assaye Mantsegbah Chekol CBRHA, Batcha Megele Hailu CBRHA, Zeguda Silenat Tedese CBRHA, Zeguda Kassin Ayele CBRHA, Zeguda Terefe Cheyle CBRHA, Gumdri Addis Kassa CBRHA, Gumdri Getahun Mekonen CBRHA, Gumdri Emita Mekonen CBRHA, Gumdri AMHARA, Farte Woreda Wondale Tassew Deputy Woreda Administrator Gobezie Axalew Woreda Health Office Head Jegnaw Belary CBRHA Supervisor for the Woreda Youth and Supervisor AMHARA, Farta Woreda, HEWs and CBRHAs Bishat Aragie HEW, Kanat Menbere Demisse HEW, Kanat HEW, Woma Magera HEW ,Woma Magera Werke Mengesha CBRHA, Kanat Tangut Wale CBRHA, Kanat Yezab Mengist CBRHA, Woma Magera Lakech Tilahun CBRHA, Woma Magera OROMIA REGION 70 MOH, Oromia Regional Health Bureau Kaasaa Haayliuu Health Bureau Head Abera Seifu Head of Family Health ODA Regional Office Mulegeta Hawas RH/FP SAP Regional Program Manager Tessena Firdissa RH Regional Program Officer Meselech Mengistu IEC Regional Coordinato Medium Level Clinic in Addis Ababa Dr. Dembie Clinician Pathfinder, Oromia Regional Office Adaba Tasissa Regional Coordinator Mitrou Bekele Accountant EECMY Asefa Woiyessa Vice President Gudefa G/Mariam Family Planning Expert Temeesgen Feyissa CBRHA supervisor Rahel Youseph Secretary Cashier Talome Derssi Driver Zonal Office Finance and Economic Development Berhanu Hirpo Head FSDPPO Teshome Gemeda Head MOH/East Wollega Zone Aduana Mamo Zonal Coordinator Oromia Pharmacy Head Pharmacist Zemenan Pharmacy Head Pharmacist OROMIA/Wayu Tukka, WAC Fetene Amensisa WAC Youth and Sport chair Nuresa Regasa WAC Social Affairs Chair Mothma Tesfaye Health Officer Tariku Dengiya Woreda, Health Office Head Tamiru Erkosa Woreda ,TB and Leprosy Expert Emirou Jalesa Woreda, MCH Expert Wayu TukkaWoreda, HEWs, CBRHAs and Households Luche Tesema HEW, Gute Badeye Alemtsgetsehay Kebede HEW, Gute Badeye 71 Nesattu Asfaw HEW, Migna Kura Berhane Shiferaw HEW, Warababo Migna Sisay Regasa HEW, Warababo Migna Adamu Oliku CBRHA, Babu Demani Wodago CBRHA, Gaba Jimata Birhane Mekonen Household, Gaba Jimata Ejigayehu Regasa Household, Gaba Jimata Meseret Lelbessa Household, Gaba Jimata Mestawet Sori Household, Gaba Jimata Zamzam Mohamed Household, Gaba Jimata Atmenesh Obsa HEW, Migna Kura Adise Oliko Household, Warababo Migna Ayane Touloum Household, Warababo Migna Fatouma Zadu Household, Warababo Migna OROMIA/ Horo Woreda ODA Regional Staff Tezerash Aklilu Zonal Program Coordinator Adugna Wakjira Zonal RH/FP Program Officer Desalegn Oljirra, Woreda Program Supervisor WAC, Horo Woreda Health Kenate Wirty Zonal Health Department Deputy Wakigari Zewedi Head, Woreda Health Office Leema Desu Head of Family Planning, Woreda Health Office Dessalegn Olijera WAC Woreda Supervisor Emebet Marissa Head of Women's Affairs Dereje Jabessa Adolecent Affairs Fekadu Tsega Representative Education Office Birribaa Baami Woreda Administrator Bekele Baayisa Member Zenabu Asres Former Circumcisors Aschelew Tilahous Former Circumcisors Horo Woreda, Nurses, HEWs, CBRHAs, and Households Agasa Oleba Clinical Nurse, Health Center, Sakala Kebele Bikile Workalema Clinical Nurse, Health Center,Sakala Kebele Asmare Neegeasa HEW, Dedipe Kistana Chaltu Adugena HEW, Dedipe Kistana Demeku Baba CBRHA, Dedipe Kistana Annduaem Mouligeta Household, Dedipe Kistana Daditou Keorkeneh Household, Dedipe Kistana Yadesha Gutama Household, Dedipe Kistana 72 Melkitu Werlomej Household, Babo Elfinash Regasa Household, Babo Agitu Luma Household, Babo OROMIA ,Gadeb Asasa Woreda Gadeb Asasa, Woreda Adminsitration Medhin Mahlo Woreda Health Office Head Ararso Boka Capacity Building Vice Administrator, Woreda Hussein Shanko Finance and economic Development Head Medhin Mahlo Woreda Health Office Head Musen Mersho Woreda Health Office Gadeb Asasa Woreda Nurses/HEWs/CBRHAs/Hous eholds Jhatu Byou Junior Nurse, Kaka Wolkite Lemlem Adbare HEW, Bucho Bedira Hasan HEW, Bucho Kamila Amana HEW, Kaka Wolkite Mohamed Amano CBRHA, Kaka Wolkite Rukiya Kawo CBRHA, Bucho Medina Jarso CBRHA, Gadeb Asasa Haja Geneho CBRHA, Gadeb Asasa Belay Alemu CBRHA, Gadeb, Asasa Emebet Hordofa CBRHA, Gadeb, Asasa Aman Buli CBRHA, Kaka Wolkite Abonesh Delamo Household, Bucho Agozenech Glmedhin Household, Bucho Sentayehu Kabelo Household, Bucho Aliah Kassin Household, Bucho SNNP/Awassa Woreda Pathfinder, Regional Office SNNP Ketsela Regional Director Elsa Mekele Kai Deputy Program Coordinator University of Awassa Mr. Atemu Gonfa Head Student Services Mr. Aleyyehu Deputy Director, Student Services Mrs. Tisgist Nurse, Main Campus Clinic Mr. Melkame Nurse, Agriculture Campus clinic Awassa Health Center Mrs. Asnaku Acting Head EKHC 73 Mrs. Tewabech Tesfaleon Head Nurse Supervisor Debub Radio FM Mrs Producer MOH Dr. Sahie Sitia Head Regional FP/MCH Programs Mr. Tafesech Mamo Nurse, FP Awassa Sidama Pharmacy Pharmacist Pharmacist Awassa’s Largest Pharmacy Manager Manager Awassa's Largest Pharmacy Ministry of Finance and Educational Development Mr. Mengesha Meshena Head Population Affairs SNNP/DoreBafena Woreda Mr.Endrias Yisahak Head FP/MCH Mrs. Woinshet Wotngo CBRHA, Dora Bafena Mrs. Fantage Detamo Household, Dora Bafena SNNP/Borecha Health Center Mrs. Yenenesch Nurse, Borecha Health Center FP Staff Mrs. Elfnesh Naushe Nurse, Borecha Counseling Team Mr. Tseaye Eyamo CBRHA, Borecha Mrs. Mesetet Manja Household, Borecha Genet Abe Household, Borecha Abeba Abe Household, Borecha SNNPR/Dilla Zone Mr. Tamirat Debebe Dilla Zone Population Officer Dilla Medan Acts Mr. Gezahagn Supervisor Mr. Akilu Nurse,Medan Acts Supervisor Dilla’s Largest Pharmacy Pharmacist Dilla’s Largest Pharmacy SNNP, Dilla Woreda/ CBRHA/ Household/HTP Mrs. Meselech Bereded CBRHA Dilla Town Mrs. Tadelech Basa Household Dilla Town Mrs. Asnakech Gedebo Household Dilla Town Ms. Hamelam HTP Household Mr. Anmaw Alemu HTP Household and Community Teacher Ethiopia/SNNP/Wonogo Woreda 74 Mr. Wubishet Mekuria Wongo Woreda Head Health Depatrment Alemu Akele WAC, Chairperson Mr. Girum Bekele WAC, Member, Ethiopian Orthodox Priest WAC, Member, Minister Protestant Church Mrs. Bihukan Satisfied Post Fistula Repair/ FP Household Mr. Bihuken Husband Satisfied Fistula/FP Household Meseret Assefa HEW, Wonogo Ms. Tisit Fulasa HEW, Wonogo Ms. Genet Beyne HEW, Wonogo Ms. Meseret Alemayehu HEW, Wonogo Mrs. Zewiditu Tiaso CBRHA, Wonogo Ms. Zena Tamiru CBRHA, Wonogo Mrs. Tadelech Basa CBRHA, Wonogo Mrs. Asnakech Gedebo CBRHA, Wonogo CBRHA, Wonogo Ms. Roman Tsegaye CBRHA, Wonogo TIGRAY MOH, Regional Health Bureau Dr. Gebreab Barnabas Head Pathfinder International Yeman Regional Director Relief Society of Tigray Teklewoini Assefa Awala Equar Health Department Head Kellali Tsegay RH/FP Coordinator Yaynshet Gebreyohannes CBHC Division Grrmay Belay Woreda Project Coordinator Girmay Zerabuk Worda health Coordinator Mekele Health Center Nuru Fetiwi Youth Friendly Service provider Mekele University Adi Haki Campus # 10 Tanet Mulugeta 2nd year management Birhanu Kindya 3rd year law Kiya Tsegaye 2nd year law Yewangesh Gebretsadic 2nd year computer science Gebremedhihn Takele 2nd year public and development management Eyasu Hadigo Social Worker Belaynesh Nega Nurse Staff Voice of Tigray (VORT) 75 Yirga Haile Silasie Coordinator Mums for Mums Ashenafi Asmelash Yifter Woldemichael Kiflemariam Eshetu Pharmacy Tehnician Birhanu Berhe Expert Durggist TIGRAY, Hawzen WAC, Hawzen Ahferom Woldegebreal Vice Administration Head Alemayehu Bayra-Biruk Woreda Health Head Habtom Melese Education Office Head Tewolde Girmay Farmers Association Berhe Hailu Extension Package Expert Yemane Gebregizabher Youth Association Shek Mohammed Mamud Muslim Religious Leaders Kassa Asgedom MCH expert Birhane Gebregiorgis Agriculture office head Ametemichael Gebru Women's association head Hawzen Health Center Eyerusalem Berehe Midwife Fisha Aregaw Clinical Nurse Gebremedihin Desta Supervisor Hawzen CBRHA Mulubrhan Hailesilase CBRHA TIGRAY Adrigat Woreda Adrigat WAC Angesom G/Mariam Deputy Woreda Administrator Akeza Woldu Agriculture Office Amira Yenus Town Health Office Berhane Tsegay Town Administration Representative Gerges Abadi Women's Association Representative Kahsay Merutse Tigray Youth Association Shek Mohammed Rejja Muslim Affairs Representative Cherkos Gebre Yohannes Town Education Office Araya Kidane Mariam G/afeshum Health Office Tesfameskel Asfeha Peasants Association Kahsay Ghiday MCH expert Melakemehret Teklay GebreAbe Rural Orthodoc Dioceses Beri Berhane Women's association Almaz Biargeleghn Women's affairs office 76 H/Selassie Abay Town Ortjhodox Dioceses Berhane Atsbaha Rural Education Office PAC tracker TIGRAY/ Agrigat,CBRHAs Bisrat Abay CBRHA Firwoiny Hagos Adigrat Town Kabele 01, CBRHA Genet Gebretensae Adigrat town Kabele 01, CBRHA Haimanot Wodlekiros Adrigat Town Kabele, CBRHA Tirhas Gebre Yohannes Adigrat Town Kabele 06, CBRHA TIGRAY/Buket TBAs, Households Almnesh Hailu TBA/CBRHA Kidusan Tesfamariam TBA Mihret Gebrehiowt TBA Letemariam Nere'a TBA Bisrat Bilhe TBA Tiberh Gebremariam TBA Askual Gebremariam Household Lemlem Girmay Household Mebrihit Gebremneskel Household Hewan Gebreyohannes Household Berhe Gebregziabher Household Libanos Teka Household Haregewoin Tesfay Household TIGRAY/Betewariat CBRHAs/Households Abrehet Gebremedihin CBRHA Kiros Gebresilasie CBRHA Temint Gebregiorgis Household Mihret Gebrekidan Household Alem Gebregziabar Household TIGRAY/Megab HEWs,CBRHAs Medhin Asmelash CBRHA Nigist Berhe CBRHA Abeba Kelew CBRHA Kahsay Kahsaye CBRHA Tsega Tekly CBRHA Harnet Adane HEW Medihn Gebremedihn HEW Kalayu Messele HEW Tsega Tekly HEW TIGRAY/Hatset, HEWs, 77 CBRHAs,Households Mitslal Fikadu CBRHA Letbrhan Hagos HEW Hagos Gebrekidan CBRHA Birhan Gebreegiher CBRHA Mulubrihan Gebrehiywot CBRHA Danait CBRHA Samuel CBRHA TIGRAY/Sibla Siat, HEWs Almaz Gebrekidan HEW Tirhas Gebregzihar HEW TIGRAY/Dagum CBRHAs,Households Tsega Hagos HEW Birhan Gebrtatios HEW 78 Appendix C: Interview Instruments 79 Questionnaire #1 Respondents: Regional Health Bureau Head and Woreda Health Office Head Topics:  Policy, fee for service, budget, sustainability, and NGOs Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _______________________________________________ Name of the interviewer: _____________________________________________ 1. Has the work of Pathfinder affected the type of FP/MCH services offered? If yes, how? Probes:  CBRHAs, contraceptives supplies, training, renovation of health posts, other? 2. Have attitudes and practices in family planning changed?  If so, how? (please circle all that apply) a. More people use contraceptives b. Smaller families   c. More knowledge of FP d. More services are available e. Obstetric fistula f. No change g. other 3. Does your region’s budget include family planning ?  Yes/No 4. If so, has this contraceptive budget gone up or down? Yes/No 5. Are families paying for family planning services  delivered to their home  by CBRHAs?   Yes/No 6. Do you have fee for service models for family planning services in your region?  Yes/No If so, what type? 7. Do you work with IPOs on family planning/reproductive health? Yes/No If so, which ones? 8. What do IPOs do in your region? (circle all that apply) a. Provide contraceptives   80 b. Conduct training c. Manage other resources d. supervision e. other 81 Questionnaire #2 Respondents: Regional Family Health Department Head and Wereda Family Health Coordinator Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _______________________________________________ Name of the interviewer: _____________________________________________ 9. Are you aware of Pathfinder’s work in family planning and reproductive health?  Yes/No 10. How long has the Pathfinder project worked in your region/wereda? 11. Have attitudes and practices in family planning changed?  If so, how? (Circle all that apply) h. More people use contraceptives i. Smaller families   j. More knowledge of FP k. More services are available l. Obstetric fistula m. No change n. Other 12. Do you think Pathfinder has contributed to preventing early marriage?  Yes/No a. If so, how? 13. Do you think Pathfinder has contributed to preventing harmful traditional health practices related to women?  Yes/No  (circle all that apply) a. Female genital mutilation   i. Better enforcement of laws ii. More information iii. Trained clinicians   iv. More outspoken men and women b. Early Marriage 82 i. Better enforcement of laws ii. More information iii. Trained clinicians   iv. More outspoken men and women 14. Are there any new FP/RH guidelines related to preventing harmful traditional health practices which Pathfinder contributed to?  Yes/No 15. Do you maintain records on early marriage?  Yes/No 16. Do you maintain records on female genital mutilation? Yes/No 17. Have any traditional female circumcisers been educated by Pathfinder? Yes/No   If yes , what new activities do these traditional female circumcisers do ? 18. Have health workers in your region/wereda been trained by Pathfinder? Yes/No If so, what type of training? (circle all that is applicable) a. Long term methods b. Logistics   c. Counseling   d. Community participation e. Youth programs f. Private provision of FP/RH services g. Harmful and traditional health practices 19. Has this training influenced services in your region/wereda ?  Yes/No If so, how? 20. Do you regularly receive Pathfinder data on client, and new accepters?  Yes/No a. If so, how do you include this in your monthly records? 83 21. What do you think are the most important contributions of the CBRHAs?    22. Are there other valuable contributions of Pathfinder activities? Yes/No a. If so, what 23. Have attitudes and practices in family planning changed?  If so, how? (please circle all that apply) o. More people use contraceptives p. Smaller families   q. More knowledge of FP r. More services are available s. Obstetric fistula t. No change u. other 24. Do you think the Pathfinder program should continue in some form? What activities should continue? 84 Questionnaire #3 Respondents: Members of the Woreda Advisory Committees (Focus Group) Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _______________________________________________ Please attach a list of all the respondents___________________________________ Name of the interviewer: _____________________________________________ 1.   Please tell me about yourself and why did you choose to participate in this committee? 2. What does this committee do ? 3. How often do you meet? 4. Now can we talk about CBRHA, what are the services do  you like which the CBRHAs provide? (circle all that apply) a. Family planning i. Bring/sell contraceptives ii. Provide advice/information/education iii. Increase financial commitment for fp iv. Post abortion care v. Set up appointments in health facilities for our community vi. Come to our house   vii. Clinical referrals viii. Follow up on complications ix. Other b. Maternal and child health i. Nutrition ii. Referrals for ANC/Post‐partum care iii. Provide advice/information/education iv. Referrals for child health, including immunization v. Set up appointments in health facilities for our community vi. Come to our house   vii. Clinical referrals viii. Follow up on complications ix. Other c. Harmful and traditional practices   85 i. Provide advice/information/education ii. Clinical referrals iii. Follow up on complications iv. Other d. Reproductive health/ HIV/AIDS/ STIs i. Provide advice/information/education ii. Referrals   iii. Set up appointments in health facilities for our community iv. Come to our house   v. Other 5. Do you think Community‐based workers can provide other services?  If yes, which ones?    6. Do you think community‐based workers, should serve as a model families? 7. Would additional responsibilities have a negative impact on the services that community‐based workers provide? 8. Have attitudes and practices in family planning change?  If so, how? (please circle all that apply) v. More people use contraceptives w. Smaller families   x. More knowledge of FP y. More services are available z. Obstetric fistula aa. No change   bb. Other 86 Questionnaire #4 Respondents: HEWs and Pathfinder CBRHA supervisors Note: At least four HEWs should be interviewed from four different kebeles, if possible. Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _______________________________________________ Date you began work________________________________________ Name of town or Kebele (PA) if rural _______________ Name of the interviewer: _____________________________________________ 1. Do you supervise any CBRHAs ? Yes/No   2. How many CBRHAs do you supervise?  _____ 3. Do you use a supervisory check list for CBRHAs? (ask to see the supervisory check list) a. Does the registration book record the visits of the client? Yes/NO b. Does the registration book record the type and number of contraceptives dispensed? Yes/No c. If so, what type?  __________ ,  __________,  _________ 4. Have you had any supervisory training? Yes/No 5. How often do you supervise/meet with CBRHAs during the month/quarter? ________ 6. For HEWs: Do you plan your work with CBRHAs?  Yes/No 7. What activities do the CBRHAs  perform?   How does this differ from your work as an HEW? 87 8. How do you follow up on CBRHAs performance? (circle those that apply) a. Site visit   b. Routine meetings c. Joint home visit to clients d. Other 9. How many CBRHAs under your supervision are outstanding?  Please rate them on a scale of 1‐5 (1 = low performance, 5=outstanding performance) 10. How many of the CBRHAs would you give a score of 5? 11. What factors make outstanding CBRHAs?   12. Could they become model households? 13. What happens if you find that a CBRHA is not performing well? 14. What feedback do you provide? 15. How many of the CBRHAs under your supervision dropped out during the last year?   ______ 16. Why did these CBRHAs dropped out?  Please cite the reasons   17. For CBRHAs who work for more than two years what are the reasons they continue this work? 18. Do you reward or recognize outstanding CBRHAs performance? Yes/No a. If so, how? 88 19. For HEWs:  Do CBRHAs contribute to your work?  Yes/No a. If so, how? b. If not why? 20. Are there activities of the CBRHAs program that should continue?  Yes/No a. If so, what? 21. Do you work with any other health volunteers? Yes/No,   22. In the last three months have you had clients referred by CBRHAs?  Yes/No a. How many _______ b. For what?  (circle if applicable, as many as possible) i. Contraceptives ii. Immunization iii. Sexually transmitted diseases iv. Ante‐ care/delivery v. Other 23. Is there anything else you would like to tell us about the CBRHA program? 89 Questionnaire #5 Commercial Pharmacy Visits (visit at least one regional and one district level pharmacy) Respondents:  Pharmacists and Pharmacy technicians Level:_____________________________________________________ Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _______________________________________________ Level of Pharmacy ________________________________________________________ Name of the interviewer: _____________________________________________ 1. Do you sell contraceptives?   Yes /No 2. Where do you get your contraceptives for this pharmacy? 3. What types of contraceptive did you sell over the last month?    Brand/type     Unit price a. _____________   ________ b. _____________   ________ c. _____________   ________ d. _____________   ________ e. _____________   ________ f. _____________   ________ 4. Over the last year what is the most popular type and brand for men and women? Men         women 90 5. Are you selling more condoms this year than last year (Yes/No) 6. Are you selling more pills/IUDs/injectibles  and other methods this year than last year?   Yes/No 7. Did you receive any training from the Pathfinder FP/RH project? Yes/No 8. Did you get any questions about contraceptive methods or side effects? Yes/No 9. Do you work directly with CBRHA or HEWs?  Yes/No 91 Questionnaire #6 (hospitals/health centers) Respondents: 1  Provider  per hospital/health center Region: ____________________ Zone: ________________ Wereda: __________________ Date of interview: _________ Name and title of respondent: ______________________________________________ Name of town or Kebele (PA) if rural _______________ Name of the interviewer: _____________________________________________ 1. Do you offer family planning services?   Yes/No 2. Is the family planning services consultation room private? Yes/No 3. Is confidentiality maintained?  Yes/No 4. Is the room equipped with booklets and other Family Planning information?  Yes/No 5. Are posters, visual methods displayed in the waiting hall?  Yes/No 6. Is there a place to store contraceptives?  Yes/No 7. Does the provider take a client history? Yes/No 8. Observe whether the provider offers information on this method.  Yes/No 9. Observe whether the provider talks to the client in a friendly manner with clear, plain language.  Yes/No 10. Are the patient registers up to date?  Yes/No 11. Do you receive referrals from CBRHs or HEWs?  Yes/No a. If so, how many referrals have you received in the past month.  _____ 12. Is dual protection (two methods promoted)?  Yes/No 13. Are the health providers trained in FP/RH counseling?  Yes/No a. If so, how many ____ 14. Was there specific hands‐ on practicum given in addition to lectures? Yes/No 92 a. If so, how many cases did the clinicians train on before applying this skills in your center ______ 15. Do you have adequate FP equipment?  Yes/No a. Norplant Insertion and removal b. IUD kits c. Speculum   d. Sterilizer   16. Do you have adequate FP  supplies?  Yes/No a. contraception b. injectables c. disposables d. Sterilizer   e. Gloves    17. What Pathfinder materials  do you currently use? a. Cue cards   Yes/No     number on hand _____ b. Posters  Yes/No        number on hand _____ c. Supervisory check list  Yes/No      number on hand _____ d. Other job aids or materials   Yes/No number on hand _____ 18. What types of contraceptive do you have in stock now?    19. In the past six‐month have you had any stock‐out for any type of contraceptive?  Yes/No a. If so, which ones? 20. What do you think are the most important contributions of the CBRHAs? 21. Do you think the Pathfinder program should continue in some form?  Yes/No   a. If so, what elements would you advise the government to continue?   93 Questionnaire #7 ( CBRHAs) Respondents: CBRHAS Note: 1) Review monthly report for number of clients, number and type of services and method mix; and 2) Record the information on the attached form. Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _____________________________________________ Name of town or Kebele (PA) if rural _______________ Name of the interviewer: _____________________________________________ 1. Sex   Female   Male 2. How old are you?  _______  age in years 3. What motivated you to be a CBRH agent? 4. When did you begin working as a CBRHA? (date)  ________ 5. Have you received training? Y/N    6. When was your last training? (Date) 7. Did you have any refresher trainings ever since you started working as a CBRH agent?   Yes/No a. If Yes, how many? 8. How many clients do you serve?   ____________ 9. How often do you see your clients, on average? a. Weekly b. Once in two weeks c. Once in a month 94 d. Once in two months e. Once in three months 10. To whom do you report?  ______________________ 11. How often does your supervisor see you?    12. Are you working with HEWs? Yes/No a. If so, what are you doing? 13. How has the addition of the HEW affected your service to the community?   14. Check message recall/counseling skills during household visit. a. Family Planning i. Did she record the name, age, marital status and number of previous deliveries? Yes/No ii. Did she present all family planning methods? Yes/No iii. Did she use visual aids?  Yes/No iv. Did she ask your client which method they preferred?  Yes/No v. Did she ask the client if they have other questions?  Yes/No vi. Did she give the client contraceptive?  Yes/No vii. If so, how many were supplied? b. MCH i. Did she record the name, age, marital status and number of previous deliveries? Yes/No   ii. Did she present messages on nutrition, immunizations, ANC and Post‐ partum care? Yes/No iii. Did she use visual aids?  Yes/No iv. Did she ask the client if they have other questions?  Yes/No v. Did she provide a referral for maternal and child health care services? c. Harmful and traditional health practices? I. Did she record the name, age, marital status and number of previous deliveries? Yes/No II. Did she provide advice, information, education  on harmful traditional health practices? Yes/No III. Did she provide advice, information, education? Yes/No IV. Did she use visual aids?  Yes/No V. Did she ask the client if they have other questions?  Yes/No 95 VI. Did she provide a referral to follow‐up on complications? Yes/No d. Reproductive Health, HIV and AIDS, and STIs? i. Did she record the name, age, marital status and number of previous deliveries? Yes/No ii. Did she provide advice, information, education, on Reproductive Health, HIV and AIDS, and STIs   iii. Did she use visual aids?  Yes/No iv. Did she ask the client if they have other questions?  Yes/No v. Did she provide a referral  for follow‐up? Yes/No 15. If she didn’t have the method on hand did she tell the client where to get it? Yes/No 16. Does she provide other supplies (malaria bed nets, treatment, DOTS, maternal; and child health, nutrition, etc)?   Yes/No 17. Does she sell the contraceptive? Yes/No a. If so, for how much? 18. Does she refer the clients to health facilities?  Yes/No If so, for what? 19. Have you given any educational talks on FP in the last six months? a. To whom and where 20. What challenges and constraints have you faced in your job? 21. What do you think have been your successes? Questionnaire #8 (households) Respondents: Households (at least 2 clients and 1 non­user) per Kebele Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: ____________________________________ 96 Name of town or Kebele (PA) if rural _______________ Name of the interviewer: _____________________________________________ 1. Has the CBRHA ever visited this house during  March, April, May?  Yes/No 2. Has the HEW ever visited this house during March, April, May?  Yes/No 3. What services did the CBRHA provide? (circle all that is applicable) a. Brings/sells contraceptives   b. Gives me or my family information and advice on family planning and reproductive health, and maternal and child health c. Schedules  appointments  for me at health posts or health centers d. Comes to my house with materials   e. Other 4. What services does the HEW provide? (circle all that is applicable) a. Brings/sells contraceptives   b. Gives me or my family information and advice on family planning, reproductive health, and maternal and child health   c. Schedules  appointments  for me at health posts or health centers d. Comes to my house with materials   e. Other 5. What services, if any, would you like to improve? 6. What services do you like best that the CBRHA provides? 7. What services do you like least that the CBRHA provides? 8. Do you use family planning methods?  Yes/No a. If so, which method? 97 9. What do you like most about the current method you are using? 10. If the client is pregnant, ask her if she has ever used a family planning method? 11. If you don’t use family planning methods why not? 12. Where do you go to obtain family planning methods/contraceptives? 98 Questionnaire #9 (IPOs) Name of IPO: Respondents:  IPO Directors Region: ____________________ Zone: ________________ Wereda: ___________________ Date of interview: _________ Name and title of respondent: _________________________________________________ Name of the interviewer: _____________________________________________ 24. What activities does this IPO do conduct with respect to RH/FP, Harmful and traditional practices, MCH, other? 25. What percent of the population do you currently cover?   26. What is your current budget? 27. What types of support do you provide the CBRHA? (circle all that apply) a. Provide contraceptives   b. Conduct training c. Manage other resources d. Supervision e. Outreach/advocacy f. other 28. What types of support do you provide the HEW? (circle all that apply) a. Provide contraceptives   b. Conduct training c. Manage other resources d. supervision e. other 99 29. What type of support does Pathfinder provide your IPO? 30. What are some of the suggestions you have to improve the implementation of the Pathfinder CBRHA program?   31. How many of the CBRHAs under your supervision dropped out during the last year?    32. Why did these CBRHAs drop  out?  Please cite the reasons. 33. Do you work with any other health volunteers? Yes/No,   34. Have attitudes and practices in family planning changed where you work?  If so, how? (please circle all that apply) cc. More people use contraceptives dd. Smaller families   ee. More knowledge of FP ff. More services are available gg. Obstetric fistula hh. No change ii. other 35. What role do you think IPOs should play in supporting the Government’s new Health Extension program?   36. What other roles do you think IPOs should play in FP/RH  and MCH? 37. Do you think the Pathfinder program should continue in some form? If so, what activities should continue? 100 Appendix D: Team Field Visit s 101 Site Selected for FP/RH Project Evaluation And Field work Schedule May 27, 2008 Note: The evaluation Team will visit at least 4 Kebles in each study Woreda Region Zone Implementing Partner Selected woreda Date(s) Focus areas for the review (Subjected to change based on the real situation on the ground) Regional Level June 2 - RHB - Population Office - Fistula Hospital/Centre - YFS Amhara (Team 1) E/Gojjam EAID (NGO) Awabel (reserve: Machakel) June 3: Morning -Travel to D/Markos, Afternoon field work June 4: Full day on field work - WAC - WorHO - CBRHA and Supervisors - HEW - Referral Health Facilities - Beneficiaries/Clients - On the way at Tilli town =Renovation - YFS/Youth Center at Debremarkos town - Other RH activities and beneficiaries 102 Awi MOH /work directly with government Dangila (reserve: Fagita Lekoma) June 5: Morning- travel to Dangela, Afternoon field work June 6: Full day in Dangla and COB-travel to B/Dar - WAC -WorHO - CBRHA and Supervisors - HEW - Referral Health Facilities - Beneficiaries - Other RH activities and beneficiaries South Gonder ADA (Gov affiliated NGO ) Farta (reserve: Fogera) June 7: Full day field work June 8: Sunday June 9: Full day - WAC -WorHO - CBRHA and Supervisors - HEW - Referral Health Facilities - Beneficiaries - Other RH activities and beneficiaries 103 Regional Level June 2 - RHB - Population office - Regional Government/Vice President East Wollega EECMY/ CY (NGO) Wayu Tukka (reserve: Digga) June 3: Travel to Nekemt and Discussion with IPOs June 4: Full day June 5: full day - WAC - WorHO - CBRHA and Supervisors - Referral Health facilities - Beneficiaries/Clients - Woreda health office - Renovation - HEWs & CBRHAs relation ship Oromia (Western) (Team 2) ODA (Gov affiliated NGO) Horo June 6: Travel and field work June 7: full day - WAC - WorHO - CBRHAs - Referral Health facilities - Beneficiaries/Clients - Woreda health office - HEWs & CBRHAs relation ship 104 W/Arsi - Oromia MOH Gedeb Assasa (Reserve : TBD) June 8: Travel to Addis June 9-10: full day - WorHO - Woreda Administration - CBRHAs - Beneficiary/Clients - Renovation - Referral Health facilities - HEWs/CBRHAs relationship SNNPR (Team 3) Regional Level June 2 - RHB - Population office - YFS (Hawassa University) - Awassa Lake side Youth Associations - IPO/EKC - Awass Zuria Woreda Health Office - Awass FM Radio 105 Sidama EKC (NGO) Awassa Zuria (reserve : Boricha / Aletawondo TBD based on coverage) June 3 -4: full day - WAC - CBRHA and supervisors - HEWs - Health Centre/ Trained staff on LAFP - Beneficiaries/Clients Gedeo EKC (NGO) Wonago (reserve; Dilla Zuria) June 5-7: Full day - WAC - CBRHA and supervisors - HEWs - Referral Health Centre - Beneficiaries/Clients 106 Regional Level June 2 - RHB - Population Office - YFS (Mekele University and HC) - IPO/REST, VORT M/Tigray REST Gentafeshum June 3-4 - WAC - WorHO - CBRHA and Supervisors - Referral Health facilities - YFS - Beneficiaries/Clients - Renovation - HEWs & CBRHAs relation ship Tigray (Team 4) M/Tigray REST Hawzien June 5-7 - WAC - WorHO - CBRHA and Supervisors - Referral Health facilities - Beneficiaries/Clients - HEWs & CBRHAs relation 1 Appendix E: References 2 Asnake, Mengistu MD, MPH. Expanding Long Term Family Planning Services through Service- based Training Approach. Pathfinder International/Ethiopia: pp. 1-21. Belay Dehab and Dejene Amare. Reasons for the Low Use of IUCDs in Ethiopia. (December2003) Pathfinder International/Ethiopia: pp. 1-90. Birhan Research & Development Consultancy. Knowledge, Attitudes, and Practices in Family Planning: Results of a September 2004 Survey in Amhara, Oromia, SNNPR and Tigray Regions of Ethiopia. (Jan. 2005) Pathfinder International/Ethiopia: pp. 1-60. Birhan Research Development Consultancy. Report on the Causes and Consequences of Early Marriage in Amhara Region. (July 2006) Addis Adaba: pp 1-58. Bowers, Gerard, Assefa Hailmariam, Willa Pressman, Joan Robertson, Elizabeth Warnick, Judith Robb-McCord, Kidest Lulu, Wolde Mariam Hirpa, Genet Mengistu, Assefa Abebe. Solomon Abebe, Sahle Sita, Tesfalem Hagos, and Abebu Beshane. Mid￾term Evaluation of the Family Planning and Reproductive Health Project of Ethiopia. (April, 2005) USAID/Ethiopia: pp. 1-90 Central Statistical Authority, Ethiopia Demographic and Health Survey 2000. (May 2001) Addis Ababa: pp. 1-283. Central Statistical Authority, Ethiopia Demographic and Health Survey 2005. (May 2001) Addis Ababa: pp. 1-283. Ethiopian Ministry of Health and the CORHA. Roles and Responsibilities of Stakeholders for Effective Implementation of Community Based Reproductive Health Care in Ethiopia. (2006) Addis Adaba: pp 1-31. Damtew, Ambaw, Asnake, Mengistu and Wilder, Jennifer. Delivery-Based Training for Long-Acting FamilyPlanning Methods: Pathfinder International/Ethiopia. (December 2007) Pathfinder International/Addis Ababa: pp. 1-7. 3 Family Health Department, Ministry of Health. Assessment of Community Based Reproductive Health Services in Ethiopia, 2002 – 2003. (April 2004) Addis Ababa: pp. 1- 44. Family Health Department, Ministry of Health and The Consortium of Reproductive Health Associations (CORHA). Curriculum for the Training Community Based Reproductive Health Agents. (May 2001) Addis Ababa. Federal Ministry of Health, Family Health Department. National Strategy for Child Survival in Ethiopia. (July 2005)Addis Adaba: pp 1-70. Federal Ministry of Health. Implementation Guideline of Health Extension Package. (December, 2004) Addis Ababa: pp. 1-6. Flavell, Carol Payne, Aregawi Akilu, Bassamo Deka, Maria Francisco, Hailu Kassa, Rachel Kearl, Brenda Moreno, Yogesh Rajkotia, Refisa, Aberra and Asmalu Shiferaw. Essential Services for Health in Ethiopia: Mid-Term Evaluation. (January, 2007) USAID/Ethiopa: pp.1-84. Gage, Anastasia. The Coverage and Effectiveness of Child Marriage Interventions in Amahara Region, Ethiopia: Findings from the 2007 Early Marriage Evaluation Study (Draft). (May 2007). USAID/Ethiopia: pp. 1-110. Giday, Tilahun, Asnake, Mengistu, and Wilder, Jennifer. Ethiopia’s Health Extension Program: Pathfinder International’s Support 2003-2007. (February, 2008) Pathfinder International./Boston: pp. 1-7. Haile, Sahlu.. Population, Development and Environment in Ethiopia. ECSP Report, Issue 10. (2004). Kuab Business Consultants (PLC). Pathfinder International-Ethiopia: Methods to Gauging Cost-Effectiveness of Project Proposals from Sub-grantees. A Brief Assessment of Fund Allocation Practices and Proposed Manual for Appraising and Evaluating Financial Aspects of FP/RH Project (Volume II). (September 2004) Addis Adaba: pp 1- 33. Mayling, Simpson-Hebert, Special Study of Woreda Advisory Committees. (August 2004) Pathfinder International/Ethiopia. Addis Ababa: pp. 1-17. 4 Melkamu,Yilma.. Assessment of Quality of Reproductive Health Care Applying the Integrated Supervision for Quality of Care Tools. Pathfinder International, Ethiopia. Minister of Health. National Health Reproductive Health Strategy 2006-2015. (March 2006) Addis Ababa: pp.1-45. Ministry of Finance and Economic Development. Ethiopia: Building on Progress A Plan for Accelerated and Sustained Development to End Poverty (PASDEP) (2005/06-2009/10) Volume I. Addis Ababa. (September 2006) Addis Adaba: pp. 1-17. Moore, Zhuzhi, Govindasamy, Pav, DaVanzo, Julie, Bizuneh, Genene, and Themme, Albert. Trends in Youth Reproductive Health in Ethiopia, 2000 and 2005. (2008) Calverton, Maryland: Macro International Inc. Pathfinder International. Annual Report Year V(Oct. 1, 2006 – Sept. 30, 2007) of Pathfinder International RH/FP Project – Ethiopia. Cooperative Agreement No. 663-A￾00-02-00385-00. (October 2007)Addis Ababa : pp. 1-38. Pathfinder International. Capacity Buidling Strategy. (October, 2005)Addia Ababa. pp.1- 11. Pathfinder International/Ethiopia. Creating Partnerships to Prevent Early Marriage in the Amhara Region. (July, 2006) Boston: pp. 1-7. Pathfinder International/Ethiopia. Supervision Manual of Community Basedl Reproductive Health Agents. (June 1998) Addis Ababa: pp 1-45. Pathfinder International/Ethiopia. Comparison Study of Two Districts: One with CBRH Agents and the Other Without Oromia Region, Ethiopía 2003. (2003) Addis Adaba: pp￾88. Pathfinder International/Ethiopia Extending Service Delivery Work Plan (Oct 1, 2007 – Sept. 30, 2008) of Ethiopia Family Planning/Reproductive Health Project. Cooperative Agreement No: GPO-A-00-05-00027-00. (2007)Addis Adaba: pp. 1-11. Pathfinder International/Ethiopia Year II Work Plan (Oct 1, 2003 – Sept. 30, 2004) of Ethiopia Family (2003) Addis Adaba: pp. 1-4. 5 Pathfinder International/Ethiopia Year III Work Plan (Oct 1, 2004 – Sept. 30, 2005) of Ethiopia Family Planning/Reproductive Health Project. Cooperative Agreement No: 663-A-00-02-00385-00. (2004) Addis Adaba: pp. 1-8. Pathfinder International/Ethiopia Year V Work Plan (Oct 1, 2006 – Sept. 30, 2007) of Ethiopia Family Planning/Reproductive Health Project. Cooperative Agreement No: 663-A-00-02-00385-00. (2006) Addis Adaba: pp. 1-11. Pathfinder International/Ethiopia. Service Delivery-Based Training for Long-Acting Family Planning Methods: Client/Provider. (Jan 2008) Addis Ababa:pp1-32. Pathfinder International/Ethiopia. (December 2003): Addis Ababa pp.1-27. Pathfinder International/Ethiopia. Annual Report Year II (Oct. 1, 2003 – Sept. 30, 2004) of Pathfinder International RH/FP Project – Ethiopia. Cooperative Agreement No. 663- A-00-02-00385-00. (October, 2004) Addis Ababa : pp. 1-38. Pathfinder International/Ethiopia. Contraceptive Logistics System at Pathfinder International-Ethiopia. (Fall 2007) Addis Adaba: pp 1-5. Pathfinder International/Ethiopia. Extending Service Delivery Quarterly Report(October – December, 2007). Cooperative Agreement Number: GPO-A-00-05-00027-00. (January 2008) Addis Adaba: pp. 1-14. Pathfinder International/Ethiopia. Five-year Close-out Report (Oct. 1, 2002 – Sept. 30, 2007) of Pathfinder International RH/FP Project – Ethiopia. Cooperative Agreement No. 663-A-00-02-00385-00. (January 2008) Addis Ababa : pp. 1-40. Pathfinder International/Ethiopia. Program Amendment on Ethiopia Family Planning and Reproductive Health Project - May 2004 to September 2007. Cooperative Agreement . No. 663-A-00-02-00385-00. (May 2004) Addis Adaba: pp. 1-11. Pathfinder International/Ethiopia. Supervision Manual of Community Based Reproductive Health Agents. (February 2001) Addis Adaba: pp 1-54 Pathfinder International/Ethiopia. Table: CYP Generated by Method Mix through out the Five Years Project Period. (Fall 2007) Addis Adaba: p.1. 6 Pathfinder International/Ethiopia. Year I Work Plan (Oct 1, 2002 – Sept. 30, 2003) of Ethiopia Family Planning/Reproductive Health Project. Cooperative Agreement No: 663-A-00-02-00385-00. (2006) Addis Adaba: pp. 1-39. Pathfinder International/Ethiopia. Year IV Work Plan (Oct 1, 2005 – Sept. 30, 2006) of Ethiopia Family Planning/Reproductive Health Project. Cooperative Agreement No: 663-A-00-02-00385-00. (2005) Addis Adaba: :pp. 1-13. Pathfinder International/Ethiopia: Assessment of PAC in Selected Public Health Facilities. (March-April 2003): Addis Ababa: pp. 1-17. Population Reference Bureau, 2007 World Population Data Sheet, August 2007. Putnam, Eliot, Cherinet, Haregewoin, Tibebu Senait, et al. Program Assessment on Expansion and Improvement of Family Planning and Reproductive Health Services in Ethiopia: Oct. 1998 – Sept. 2002. (December 2002) Pathfinder International. Addis Addis: pp. 1-34. Rahman, Mizanur, Tilahun Giday, Asnake, Mengistu and Wilcox ,Jennifer. Enhanced Access to Reproductive Health and Family Planning: Pathfinder International in Ethiopia 2002-2007. (2007)Pathfinder International/Addis Adaba: pp.1-13. Tawye, Yenehun, Fekadujotie, Jotie, Shigu, Tesfaye, Ngom, Pierre, and Maggwa, Ndugga. The Potential Impact of Community-BasedDistribution Programmes on Contraceptive Uptake in Resource-poor Settings: Evidence from Ethiopia. African Journal of Reproductive Health. Vol. 9 No3, December 2005. pp. 15-26. The Federal Ministry of Health. Implementation Guidelines of Health Extension Program All roads lead to health extension. (February 2007 )Addis Ababa: pp. 1-36. The Federal Ministry of Heath. Health Extension Program in Ethiopia. (April 2007) Addis Ababa: pp1-32. The Federal Ministry of Heath. The 2006-7 MOH Annual Statistical Report on Health and Health Related Indicators. (April 2007) Addis Ababa: pp1-32. 7 USAID/Ethiopia. Ethiopia Family Planning and Reproductive Health Project Program Amendment, September 2005 to September 2007.Cooperative Agreement No. 663-A-00- 02-00385-00, (2005). USAID/Ethiopia. Program Description Extending Service Delivery Agreement. GPO-A￾00-05-00027-00.(2006). USAID/Ethiopia. Program Description of FP/MNCH In Ethiopia: Cooperative Agreement No. 663-A-08-02-006 RFA. (2008): Addis Ababa: pp.1-28. Wilder, Jennifer. Community-Based Reproductive Health Care: Creating Demand in Ethiopia. (August 2005)Pathfinder International/Boston: pp. 1-6. Wilder, Jennifer. Ethiopia: Local Leadership Boosts Health Care Delivery. Woreda Advisory Committees Open Doors to Change. (August, 2005) Pathfinder International/ Boston: pp. 1-4. Wilder, Jennifer. Ethiopia’s Health Extension Program: Pathfinder International’s Support 2003-2007 (February, 2008) Pathfinder International/Boston: pp. 1-7. World Bank Web Site (www.worldbank.com). Accessed June 17, 2008 8 Appendix: F Pathfinder Comments on EOP Evaluation 9 Pathfinder International’s comments on the EOP Evaluation of the RH/FP program conducted by independent Evaluators contracted by the USAID Mission in 2008: General comments in relation to the recommendation of the Evaluation Team: The team has made recommendations on three broad areas: panel surveys to evaluate project effects, routine project data and their analysis, and operations research. They recommend that the project undertake a series of effective and affordable surveys that will be able to measure changes that occur in the project areas in terms of the FP/MNCH indicators (the new USAID funded Program). PI’s remarks: There was no baseline study conducted by PIE for the RH/FP project assuming that data from DHS 2000 can be taken as baseline for the outcome and impact level indicators. The current project also plans to use the DHS 2005 surveys and ESHE’s EOP assessment for reference. In non-ESHE areas (i.e. the Tigrai region) PIE plans to conduct EOP survey as well to be used as baseline for the FP/MNCH program. In addition Pathfinder is in the process of designing surveys that will be done in samples covering both project and non-project areas. The evaluation team recommended for development of an improved database of the routine project data that should be easy to collect, can be used by program personnel for routine monitoring and program improvement purposes, and can be analyzed as and when necessary. PI’s remarks: As to using data for planning, PIE has been using some of the data for program planning purposes in the implementation of the RH/FP program. However, Pathfinder will develop a more usable database as suggested to inform program planning and implementation. The team has recommended for taking help from the School of Public Health for analysis of already collected data, including those collected by ESHE. Such analyses will help new programming directions. PI’s remarks: Pathfinder will consider this recommendation concerning the FP/MNCH project. The team has identified some areas of operations research (OR) that need to be conducted. PI’s remarks: 10 Pathfinder has identified a number of OR’s to be conducted in the process of implementing the FP/MNCH program. Some of the recommended ORs are already considered by Pathfinder. The team has indicated that the sample of Kebeles and clients interviewed was not scientifically rigorous as it didn’t follow the proper sampling procedure. PI’s remarks: It should be clear that the selection of Kebeles and clients for interview was done by PIE at the request of the consultants as they didn’t have much time to randomly select and proceed. However the remarks are well taken and future such selections will be done following proper evaluations procedures. Some specific comments on the report: Page 1 foot note: ‘V’ stands for voluntary, not village Page 2 footnote # 4: DHS also uses number of non pregnant women 15-49 as denominator but the difference lies in that MOH take service statistics while DHS takes samples from the population (it is population survey). Page 17, 4th Paragraph, first line: DHS was used as reference to see the increase in contraceptive prevalence rate Page 18 ‘IPO’ section 2nd paragraph and page 20 under ‘Training Materials’: , it is stated that refresher training for CBRHAs is not planned in the government model of program management: PI’s remarks: According to the plan, the number of refresher training for CBRHAs in all the three models is the same. In this case the CBRHAs visited by the consultants might have not received refresher training because the MOH staff might have been given other urgent tasks and thus did not have time to implement the planned and budgeted refresher training activities. In all management models annual CBRHA refresher training is planned and budgeted for. Other: We propose to use ‘long acting FP methods’ instead of ‘long term FP methods’ throughout the document.