Essential Services for Maternal and Child Survival in Ethiopia: Mobilizing the Traditional and Public Health Sectors and Informing Programming for Pastoralist Populations Cooperative Agreement No. FAO-A-00-97-00054-00 September 30, 1997 - September 30, 2006 Ethiopia CS-17 Mid-Term Evaluation Report Submitted to USAID/GH/HIDN/NUT/CSHGP October 31, 2003 CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 ii Acknowledgements Special thanks go to the following individuals and organizations: ÿ Solomon Tesema, Save the Children’s new Health Sector Manager for Liben District whose patience and support during eight days of intense fieldwork made the MTE effort not only possible, but a success. ÿ The other members of the Midterm Evaluation TE Team who contributed their time, insights and long days to this effort, while also maintaining their regular responsibilities. The members of the Team were: Sr. Senait Bekele Zonal Health Department Acting Head Mr. Adamu Beyene Monitoring and Evaluation Coordinator, SC/Liben Mr. Niftalem Kumera Disaster Prevention and Preparedness Dept. Head Mr. Mohamed Manu HIV/AIDS Unit Head, SC/Liben Mr. Nina Negash Monitoring & Evaluation Officer, CARE/Ethiopia Mr. Worku Tefera Training Coordinator, SC/Liben Mr. Solomon Tesema Health Sector Manager, SC/Liben Dr. Taye Tolera Medical Director, Negelle Hospital ÿ CARE/Ethiopia for graciously contributing the time and energies of Mr. Nina Negash, who provided invaluable support and perspective to this effort. ÿ Mr. Alemayehu Boka, the Liben Impact Area Manager, who provided an excellent overview of SC’s work in the District and the entire SC/Liben staff who helped to make sure that everything that the evaluation went smoothly. ÿ Dr. Winnie Mwebesa, FP/ RH Advisor, SC/Washington, DC, for sharing the results of her thoughtful fieldwork in Liben. ÿ Dr. Eric Starbuck, (Child Survival Specialist, SC/Westport), Dr. Tedbabe Degefie, (Health Advisor, SC/Ethiopia Field Office) and Dr. Abebe Gebremariam, (SC/EFO) who openly shared their perspectives and vision for the project. ÿ The many community members, volunteers, health workers and staff who responded to our many questions with openness and a view towards what can be achieved in the future. Garth Osborn, MTE Team Leader This report was written by Garth Osborn with comments from other members of the Evaluation team and Save the Children staff. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 iii Table of Contents Acknowledgements………………………………………………………………. ii List of Acronyms and Terms……………………………………………………. v A. Summary………..……….…………………………………………………… 10 B. Assessment of Progress Made…………………….………………………… 12 1. Technical Approach……….………………………………………………… 12 a. Project Overview…………………..……………………………………… 12 b. Progress by Intervention Area…………………………………..………… 13 i. STI/HIV/AIDS Prevention (30%)……………………………………...... 13 ii. Maternal and Newborn Care (20%)……………………………………... 17 iii. Immunization (15%)……………………………………………………. 20 iv. Pneumonia Case Management (15%)…………………………………… 23 v. Control of Diarrheal Disease (10%)……………………………………... 24 vi. Control of Malaria (10%)……………………………………………….. 25 c. New Tools, Approaches and Operations Research………………………… 26 2. Cross-Cutting Approaches…………………………………………………… 26 a. Mobilization of Community Leaders and Traditional Practitioners……….. 26 b. Community Case Management of Childhood Illness……………………… 28 c. Capacity Building Approach……………………………………………….. 29 i. Strengthening Save the Children…………………………………………. 29 ii. Strengthening the Ministry of Health……………………………………. 31 iii. Strengthening Health Facilities and Workers…………………………… 32 iv. Training…………………………………………………………………. 33 d. Sustainability Strategy……………………………………………………... 34 3. Program Management……………………………………………………….. 36 a. Planning……………………………………………………………………. 36 b. Staff Training……………………………………………………………… 37 c. Supervision of Program Staff……………………………………………… 38 d. Human Resources and Staff Management………………………………… 39 e. Financial Management…………………………………………………….. 40 f. Logistics……………………………………………………………………. 40 CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 iv g. Information Management………………………………………………….. 41 h. Technical and Administrative Support…………………………………….. 42 4. Other Issues Identified by the Team………………………………………….. 44 5. Conclusions and Recommendations………………………………………….. 44 6. Results Highlight……………………………………………………………... 44 C. Action Plan (Please See Annex G) D. Annexes Annex A. Baseline Information from the DIP Annex B. List of MTE Team Members Annex C. Assessment Methodology Annex D. Persons Interviewed and Contacted Annex E. Questionnaires Annex F. CS-17 Progress at the MTE per the Revised M&E Matrix Annex G. Action Plan Annex H. Visit to Impact Area-Stanley Foster, MD, MPH Annex I. Trip Report-Winifride Mwebsa, MD CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 v Acronyms and Terms Abba olla ACNM ANC AIDS APH ARI BC BCC BEOC BHT BSS CAC CB-ARIM CBA CBD CBO CBRHA CCM CDC CDD CHW C-IMCI Cheresas Cheretas CMCI CMW COOPI CS CS-17 CSTS Father of the encampment American College of Nurse Midwives Ante-Natal Care Acquired Immune Deficiency Syndrome Ante-Partum Hemorrhage Acute Respiratory Infection Behavior Change Behavior Change Communication Basic Emergency Obstetric Care Bridge-to-Health Team Behavioral Surveillance Survey Community Action Committee Community-Based Acute Respiratory Illness/Malaria Community Birth Attendant Community-Based Distribution Community-Based Organization Community-Based reproductive Health Agent Community Case Management US Centers for Disease Control and Prevention Control of Diarrheal Disease Community Health Worker (BHT and HAC members, TBAs, and CMWs) Community-Integrated Management of Childhood Illness Wise Man/Male Traditional Healer Wise Woman/Traditional Birth Attendant Community Management of Childhood Illness Case Management Worker (CHW trained to do case management) Cooperazione Internazionale (Italian NGO) Child Survival The current child survival project in Liben District, Essential Services for Maternal and Child Survival in Ethiopia: Mobilizing the Traditional and Public Health Sectors and Informing Programming for Pastoralist Populations, funded as a cost extension of the CS￾13 grant, mainly through the 17th cycle of the PVO CS Grants Program, is referred to as “CS-17” throughout this document to distinguish it from the previous “CS-13” grant. Child Survival Technical Support Project (contractor to AID/DCHA/PVC) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 vi DAP DHAC Dhedas DHMT DHIMT DHO DHS DIP DKT DPT EFO EFO/IT EmOC EPI ERC FE FGD FHI FO FOD Forra FMOH FP FY GAVI GOE GTZ HA HAC HB-LSS HF HFMC HH/C-IMCI HIS Development Assistance Program (current FFP-funded Liben Title II program) District HIV/AIDS Council Groups of ollas that share grazing areas District Health Management Team District Health Information Management Team District Health Office/Officer Demographic and Health Survey Detailed Implementation Plan Ethiopian affiliate of PSI (Population Services International) Diptheria/Pertussis/Tetanus Vaccine Ethiopia Field (Country) Office of Save the Children/US Ethiopia Field Office/Information Technology Emergency Obstetric Care Expanded Program for Immunization Ethiopian Red Cross Final Evaluation Focus Group Discussion Family Health International Field Office Field Office Director The mobile part of a family and herd, which is composed of young men and older boys who travel long distances with the strong male cattle in search of pasture and water. Federal Ministry of Health Family Planning Fiscal Year Global Alliance on Vaccines and Immunization Government of Ethiopia The German Agency for Technical Cooperation Health Assistant Health Action Committee Home-Based Life-Saving Skills Health Facility Health Facility Management Center Household/Community Integrated Management of Childhood Illness Health Information System CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 vii HIV HIS HQ IEC IMCI IR ISA IUD Jalla/Jalletu Kafeno Kebele KPC L&D LQAS LSS Madda M&E MCH MCM MMR MN MNC MOH MTE MVA NGO NMR Obba OCA OH Olla ORS OVC PA Human Immune Deficiency Virus Health Information System Headquarters Information, Education, Communication Integrated Management of Childhood Illness Intermediate Result Institutional Strengths Assessment Intra-Uterine Device Extra-marital sexual partners, both for men and women Refers to the sexual practices of the rainy season after plenty of milk has been consumed. Community consisting of several villages, but smaller than a PA Knowledge Practice and Coverage Lot Quality Assurance Sampling Labor and Delivery Life-Saving Skills (maternal and newborn) Traditional community group comprised of several ollas sharing the same water point. Each madda corresponds roughly to one of Liben’s PAs Monitoring and Evaluation Maternal and Child Health Malaria Case Management Maternal Mortality Ratio/Rate Maternal and Newborn Maternal and Newborn Care Ministry of Health Midterm Evaluation Manual Vacuum Aspiration Non-Governmental Organization Newborn Mortality Rate Water points for cattle Organizational Capacity Assessment Office of Health (of Save the Children/US) Borana term for extended family encampment Oral Rehydration Solution Orphan and Vulnerable Children Peasant Association (“kebele”), an administrative division CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 viii PCM PH PHF PHN PID PLI PLG PLWHA PM PNC PPH PVO RDF RH RHA RHB SAT SC SC/EFO SC/HQ SCM SOS SOW SPA SSV STI TA TBA TOT TT TT2 TTBA UNAIDS UNICEF Pneumonia Case Management Peripheral Health Peripheral Health Facility Population, Health, and Nutrition Pelvic Inflammatory Disease Pastoralist Livelihood Initiative Program Learning Group of Save the Children People Living With HIV/AIDS Program Manager Post-Natal Care Post-Partum Hemorrhage Private Voluntary Organization Revolving Drug Fund Reproductive Health Reproductive Health Agent / Regional Health Advisor Regional Health Bureau (of the Ethiopian MOH) Service Area Team Save the Children Federation (USA) Save the Children/Ethiopia Field (Country) Office Save the Children/Headquarters Office Standard Case Management Sustainable Outreach Service Scope of Work Senior Program Assistant (SC staff in Liben District) Supportive Supervision Sexually Transmitted Infection / Southern Tier Initiative (USAID/Ethiopia) Technical Assistance Traditional Birth Attendant Training of Trainers Tetanus Toxoid Tetanus Toxoid, 2nd dose Trained Traditional Birth Attendant United Nations Program on HIV/AIDS United Nations Children’s Fund CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 ix USAID VCT Waaqeefatta Warra WHO Woreda United States Agency for International Development Voluntary Counseling and Testing Traditional followers of one God The stationary part of the family which consists of women and children who take care of the milking cows, weak or sick cows and calves at the olla World Health Organization District CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 10 A. Summary Save the Children/US has been implementing a child survival (CS) project in Liben District of Borana Zone in the Oromia Region of southern Ethiopia since October 1997 (CS-13 through September 2001, followed by CS-17 through September 30, 2006). The estimated total population of Liben District was 138,310 in 2002, resulting in a total direct beneficiary population of 64,000 infants, children under the age of five and women of reproductive age. Liben District covers 9,900 sq. kilometers. Semi-arid climatic conditions are harsh with low, unreliable, and unevenly distributed rainfall (500-700 mm per year), and very limited amounts of surface water. Approximately 65% of the population in Borana Zone can be categorized as poor or destitute, and lack the resources to meet their annual food requirements. In addition, this area has historically been one of the most underserved areas of Ethiopia in terms of health infrastructure and services. The overall program goals for SC’s Liben program, Essential Services for Maternal and Child Survival in Ethiopia: Mobilizing the Traditional and Public Health Sectors and Informing Programming for Pastoralist Populations, are the following: ÿ A sustained reduction in under-five and maternal mortality in Liben District, and ÿ CS-17 approaches inform policy or programming for pastoralist areas of Ethiopia in C-IMCI or reproductive health. Plans are to achieve these goals through: ÿ Improved district capacity to effectively support community health services and activities. ÿ Improved community capacity to effectively address priority health needs of mothers and children under five. ÿ Increased use of key health services and improved MCH practices at household level. ÿ Adoption of CS-17 approaches by the MOH or by other organizations in Ethiopia. The main accomplishments of the program to date are: ÿ The establishment of a two-tiered system of Bridge to Health Teams, Traditional Birth Attendants, and Health Action Committees that is establishing and strengthening the connections between the community and its health care system. ÿ The training of traditional birth attendants in Home-Based Lifesaving Skills for the first time in Ethiopia. ÿ The training of all health workers in the woreda on IMCI case management, which will effectively support the implementation of Community Case Management. ÿ A strategic partnership formed with the Liben District HIV/AIDS Council that is administering World Bank funds that are directly supporting CS-17 objectives. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 11 The project has faced significant challenges in its first two years: ÿ The senior health position with this project has changed hands four times since the beginning of CS-13 and once since the start of CS-17, which has resulted in several delays in project activities. In June 2003 a well qualified public health professional took this position. ÿ GOE approval on CCM was late in coming, which meant that its startup was delayed one year to the fall of 2003. ÿ The new GOE policy of ‘decentralization’ has led to a fundamental change in the structure, personnel and capacity of SC’s local partner – the District Health Office – which will likely have serious ramifications for the sustainability of project activities and services beyond CS￾17. A summary overview of the most important recommendations follows: 1. The current and projected capacity building needs of the new DHO should be reevaluated within the context of the remaining CS-17 project and the long-term needs related to its sustainability plans. This could require significant restructuring of the project, especially if a majority of these needs cannot be resolved through locally available sources. 2. The EFO needs to commit to increasing its overall involvement with this project, including making regular and more frequent site visits to Liben. 3. The completion of delayed project activities needs to be considered high priority, including the provision of cold chain equipment to health facilities, the hiring of additional field staff, and the startup of the Community Case Management of Childhood Illness. 4. Incentives from the communities for the CHWs need to be identified and institutionalized. 5. Resources need to be identified and actively solicited that can support Negelle Hospital in providing voluntary counseling and testing for HIV/AIDS and comprehensive emergency obstetric care. In summation, the current staff is probably the strongest this project has had since its beginning in 1997. The challenges it faces and the amount of work it needs to complete between now and October 2006 are significant. However, with the necessary support, its success is very possible. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 12 B. Assessment of the Progress Made 1. Technical Approach a. Project Overview Save the Children/US has been implementing a child survival (CS) project in Liben District of Borana Zone in the Oromia Region of southern Ethiopia since October 1997 (CS-13 through September 2001, followed by CS-17 through September 30, 2006). The District center and the site of both the District Health and SC/Liben Offices are located in the town of Negelle, which is a twelve-hour drive south of Addis Ababa. Liben District covers 9,900 sq. kilometers. Semi-arid climatic conditions are harsh with low, unreliable, and unevenly distributed rainfall (500-700 mm per year), and very limited amounts of surface water. The long rainy season occurs between March and May, with a short rainy season occurring between September and November. Early warning system data collected by SC and its partners indicate that approximately 65% of the population in Borana Zone can be categorized as poor or destitute, and lack the resources to meet their annual food requirements. The DHO estimate for the total population of Liben District was 138,310 in 2002. Based on DHO estimates the approximate population of infants was 6,000 (4.43% of the total District population), children was 26,000 (18.68%), and women between the ages of 15 and 49 years was 32,000 (22.9%) – resulting in a total direct beneficiary population of 64,000. The crude birth rate was 46.4 annual live births per 1,000 total population and there were 6,400 live births in Liben District in 2001. This area has historically been one of the most underserved in Ethiopia in terms of health infrastructure and services. MOH health facilities in Liben District include the 113-bed Zonal Negelle Hospital, and a total of nine functioning clinics (health stations/posts) outside of Negelle. Limited geographic access to health facilities and services is a primary constraint for most of the District’s population. The DHO estimates, based on the geographic distribution of people and health providers, that only an estimated 40% of the total population of Liben District lives within a ten-kilometer (6.2 mile) radius of an MOH or private health provider. This lack of access to MOH services is a likely reason for the popularity of traditional healers throughout the District. Male traditional healers in Liben, cheresas, or “wise-men,” include herbalists, bone setters, religious practitioners, and spiritual healers. These individuals are respected in the community as credible sources of information about health and healing. Moreover, these men act as “gatekeepers” for care-seeking outside of the olla or kebele. Cherites, or “wise women” traditionally provide birth assistance in the district. In most communities, one can find a cherite in every olla or group of ollas. Some cherites also practice other healing arts, such as herbalism, massage, bone setting, or female circumcision (e.g., infibulation). Their advice is often sought for children’s health problems, especially diarrhea and fever. Most are women in their mid-forties or older; all are respected by their communities. Although some cherites charge for attending a delivery, most accept payment in-kind or whatever is offered. The average number of births assisted on a monthly basis varies greatly from community to community and TBA to TBA. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 13 The goal of SC’s project, Essential Services for Maternal and Child Survival in Ethiopia: Mobilizing the Traditional and Public Health Sectors and Informing Programming for Pastoralist Populations, is two-fold: ß A sustained reduction in under-five and maternal mortality in Liben District, and; ß CS-17 approaches inform policy or programming for pastoralist areas of Ethiopia in C-IMCI or reproductive health. To achieve these goals, Save the Children/US and the partnering Liben District Health Office selected the following six interventions: ß Maternal and newborn care (at 20% of planned intervention-specific effort); ß Pneumonia case management (15%); ß Control of malaria (10%); ß Control of diarrheal disease (10%); ß Immunization (15%); and ß HIV/AIDS intervention (30%). Completing these CS interventions, the SC/Liben office is using DAP funding to address nutrition, breastfeeding and family planning. The CS-17 interventions are being implemented through the following strategies: ß Joint DHO/SC design, implementation, and evaluation of approaches to maternal and child health in Liben that inform development of strategies to address the needs of pastoralist populations in other districts of Borana Zone and Ethiopia. ß Mobilization of community leaders and traditional practitioners through Bridge-to-Health Teams (BHTs) and Health Action Committees (HACs), to support selected MCH services, and to conduct focused education to improve key emphasis behaviors at the household level. ß Introduction and evaluation of community-based case management of childhood illness, to improve access to and use of these services in Liben District, and to inform the nascent development of C-IMCI in Ethiopia. ß Building capacity of SC, the DHO, and the District HIV/AIDS Council, to provide leadership, coordination, and technical advice for integration of effective HIV prevention, care and support, and mitigation efforts into ongoing community and government activities in Liben District. Recommendation: 1. The project needs to strengthen its ability to document its successes and advocate for their adaptation to other project sites in Ethiopia and other Child Survival Projects throughout the world. b. Progress by Intervention i. STI/HIV/AIDS Prevention Activities proposed in the DIP: The DIP included the following activities for strengthening the prevention of STI/HIV/AIDS: CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 14 ß Build the capacity for HIV programming in Liben District, of SC, the District HIV/AIDS Council, and the DHO, through training, experience exchange visits, and joint implementation of HIV activities. ß Focus on strategies to change the determinants of the HIV risk behaviors among women, youth, and CSWs, identified during the CS-17 planning process, and advocate at the community and national levels to ban specific high-risk sexual behavior prevailing in Liben District (including jalla, jalletu, and kafeno). HIV/AIDS/STI behavior change materials will be collected, adapted, copied, and/or produced; and health facility staff, HACs, BHTs, TBAs, CBDs, and SC education facilitators trained and supported in conducting behavior change activities with target groups. ß Provide technical support to selected CBOs currently active in HIV-related activities in Liben District, such as training in home-based care for the Save Yourself Anti-AIDS Club (which are presently working in AIDS care and support in Liben). ß Seek to facilitate (with technical and financial support from other sources): · the establishment of VCT in the district; · the supply of condoms to health facilities and CBDs, STI drugs for facilities, and gloves for TBAs; and, · training of MOH staff and private providers in STI syndromic management. Progress toward benchmarks or intermediate objectives: The revised M&E table in the First Annual Report included the following six indicators specific to HIV/AIDS prevention: Indicator 2: District HIV/AIDS Council meets regularly, plans, and monitors HIV/AIDS activities in Liben. Progress: The District HIV/AIDS Council met four times between June 20, 2002 and June 2, 2003. While this was not the regular monthly meetings as initially planned in the DIP, the Council is collecting and reviewing reports and appears to be using this information to plan and monitor HIV/AIDS activities in the District. Indicator 10: HIV prevention efforts are effectively integrated into ongoing community and government activities through CS-17. Progress: A major focus of the CS-17 project’s HIVAIDS intervention is its participation on the District HIV/AIDS Council (DHAC). The DHAC is administering World Bank funding for Liben District through a three-phased competitive grant program that funds Peasant Associations, community based organizations (CBOs) and churches. A total of 42 grants were dispersed in Phase I (Emergency Funds) that are being used to purchase and distribute condoms and to support youth clubs, CBOs and churches in designing health promotion and HIV/AIDS prevention campaigns. The announcement for Phase II (Project Funds for Prevention, Care and Support) has just been issued. Phase III will then focus on Capacity Building so that the impact achieved in Phases I and II can be sustained. The DHAC establishes broad-based guidelines that define the types of programs to be supported, issues invitations for proposals, reviews proposals, and monitors progress and compliance of the donor recipients. In addition to its direct participation on the DHAC, SC has provided three days of capacity building training to the CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 15 potential applicants in program planning, proposal/report writing and financial management of the funds as well as helped to design the reporting forms for the donor recipients. The project has also incorporated HIV/AIDS messages into all of its trainings, refresher trainings and program activities that target the health facility staffs, the community health workers, and the communities. Indicator 11: Number of HIV-related training courses, workshops, and experience sharing visits in which SC/Liben staff have participated during CS-17. Progress: The HIV/AIDS Unit Head has attended two conferences on HIV-related topics and is therefore three fifths of the way towards reaching the final target. These trainings covered Behavior Change and Communication (April 2003), and Participatory Rural Appraisal (May 2003). Even though there were no current plans to attend other training opportunities at this time, the HIV/AIDS Unit Head noted that further training in documenting project activities, report writing and desktop publishing would be useful. In addition, training and support for the production of effective health education strategies using mass media would be timely, considering the increased availability and popularity of radio and TV in the project area. Indicator 13: Percent of respondents reporting condom use last time they had sex with non￾regular partners. Indicator 14: Percent of respondents reporting condom use every time they had sex with any non-regular partner over the past 12 months. Indicator 31: Percent of respondents who identify consistent condom use, mutual monogamy, and abstaining from sex as methods of reducing risk of HIV. Indicator 32: Percent of respondents who identify two or more signs/symptoms of STIs. Progress: Measurement of these four indicators requires data from the second FHI survey, which has not been implemented yet and therefore, fell outside the scope of the MTE. However, it is apparent that the World Bank funding noted above has increased the available supply of free condoms throughout the District. This is in addition to the condoms that continue to be available through the markets for about two cents each. The current demand for and use of these condoms, however, is not known. The project is using multiple strategies for increasing the demand for condoms and the general knowledge on STI/HIV/AIDS, including IEC targeting mothers through the project trained CHWs; peer education of commercial sex workers and hotel owners; training of health facility on a general orientation to STI/HIV/AIDS, syndromic treatment of STI, and counseling; working with 14-18 member school AIDS groups that provide mass-education to their peers; and broad-based health education through the World Bank-funded small grant recipients. In addition, the project has effectively incorporated HIV/AIDS IEC messages into its other project activities, including outreaches, static site clinics, and supervisory visits. Effectiveness of the interventions: The MTE Team found the following: Discussions with community members and leaders found that the CS-17 project and the DHAC have worked very well within the Geda (the social structure of Oromo society), its Abagadas (traditional leaders) and local Christian Churches. These groups have brought credibility to the CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 16 project’s work on the sensitive issues surrounding STI/HIV/AIDS, making it easier for communities to understand and accept the health education messages. The Mosques and the Moslem community, however, have been less involved in addressing HIV/AIDS, the reasons for which need to be further explored. FGDs with mothers found interesting HIV/AIDS-related distinctions between those living in rural areas (Genalle and Balambal) and those living in Liben’s largest town – Negelle. Mothers from the rural areas generally agreed that HIV/AIDS was not a serious problem in their communities because they “hadn’t seen anyone with AIDS” and “didn’t go with anyone except their husband.” In contrast, mothers from Negelle were emphatic that HIV/AIDS is a major problem in their community and that “everyone knows someone who has died from the disease.” There were also differences in how mothers learn about health. In the more rural areas it was from other community members and project-trained CHWs. While this was also true in Negelle, mothers there also noted that they learn through TV and radio which appear to be increasingly available and growing in popularity. The social stigma surrounding any sexually transmitted disease is significant, with the FHI/UNICEF study in Borena Zone finding that accepting attitudes of individuals infected with HIV/AIDS were found in fewer than 4% of respondents aged 15-49 years. Currently nothing is being done in the project area specifically for people living with AIDS or orphans and vulnerable children (OVCs). The stigma also might be contributing to an issue raised in FGDs that individuals with STI or AIDS related symptoms commonly bypass medical care at health facilities and go directly to rural drug shops to purchase medicines for treatment, not having been properly diagnosed or received counseling. Training of drug vendors in the importance of medical care and referral for these symptoms, as well as on the project’s other health interventions such as proper dosing for antimalarials, has shown to be an effective strategy in other projects and can be done in such a way that it increases the vendors’ customer base and therefore the incentive to participate. The project’s focus on increasing the demand for condoms and for care seeking in response to STI and HIV/AIDS related symptoms needs to be done in tandem with an increase in the “supply” of available services and supplies. Otherwise, the result can be increased frustration with the healthcare system and wider use of unsafe and ineffective traditional treatments. For the time being, the supply of condoms appears to be more than adequate due to the World Bank funding and health facility staff from throughout the District have received training in syndromic treatment of STIs. However, VCT services are currently limited to Negelle Hospital, which has only one person on staff who has received a three-day training on counseling. Changes in the technical approaches outlined in the DIP and rationale: None. Special outcomes, unexpected successes or constraints: This CS-17 Project’s participation with the District HIV/AIDS Council in its administration of World Bank funding throughout Liben District promises to be an excellent opportunity for reaching the project’s HIV/AIDS/STI goals and identifying and testing new responses to the growing threat posed by these diseases. At their best, these small grant programs can foster creative, locally-driven solutions to important challenges through a broad-based ‘competition of ideas.’ It is hoped that the Council, with SC’s CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 17 support, will not only keep its eyes open to these new ideas, but will foster their development through its training activities and then test and document their implementation through the grant￾making and reporting process so the successful strategies can be scaled up. Recommendations: 2. Build the capacity of the District in VCT. More staff in Negelle Hospital need to be trained in counseling and referral mechanisms need to be established at the health facility level that will involve orientation and training of MOH staff and the establishment of a monitoring system to track referrals from the outlying health facilities to the hospital. (The EFO is waiting on a response to a proposal submitted to the CDC for upgrading VCT services at the hospital in Negelle. If funded, it will cover the costs of training five health workers in counseling and lab testing as well as renovating the lab and counseling room.) 3. The project should explore opportunities to exploit all types of media (TV and radio) available in the project area for communicating IEC messages on HIV/AIDS. This again, could be promoted through the World Bank funds by providing grants to stations to produce ads or developing that expertise within SC. ii. Maternal and Newborn Care Activities proposed in the DIP: For Maternal and Newborn Care CS-17 proposed that it will: Strengthen maternal and community practices by: education of women, men, other family members, and community “gate-keepers” through trained TBAs, other BHT and HAC members, and health facility/outreach staff. Health facility staff and TBAs will work with pregnant women and their families to develop realistic birth plans, including plans for using trained birth attendants and contingency plans for obtaining emergency transport, and provide education on pregnancy-related danger signs, including signs during the postpartum period, nutritional requirements during pregnancy, newborn care and immediate and exclusive breastfeeding, and using antenatal outreach and health facility services. Improve access to MNC services through monthly MOH ANC outreach clinics to selected sites in the service areas of each health facility; 300 TBAs trained in HB-LSS; staff at each facility trained in LSS; and posting of three female Community Midwives at health facilities in the district following their CS-17-supported training at Negelle Junior Nursing School. The project will also seek to establish partnerships and referral relationships between trained TBAs and facility-based skilled birth attendants (such as Community Midwives). Finally, SC will seek to facilitate training for one or two Negelle Hospital physicians in EmOC, including cesarean sections, within Ethiopia. Improve the quality of MNC through a continuing partnership with ACNM for training of TBAs in HB-LSS for maternal and newborn care, and training health facility staff (nurses, Community Midwives, and MOH Health Assistants) in Life Saving Skills and as HB-LSS trainers/ supervisors. SC and Negelle Hospital trainers who have been trained in Life Saving Skills by the ACNM will train nurses, midwives, and health assistants at Negelle Hospital in LSS and as HB￾LSS trainers of TBAs. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 18 Progress toward benchmarks or intermediate objectives: The revised M&E table in the First Annual Report included the following five indicators specific to improving maternal and newborn care: Indicator 15: Percent of births attended by a trained TBA or health professional. Progress: This indicator will be measured using the KPC at the final evaluation. However, there has been significant progress towards its achievement. One hundred sixty-two TBAs have completed the basic TBA and the HB-LSS training courses since the beginning of CS-17, bringing the total number of trained TBAs, including the 100 trained during CS-13, to 262. In FGDs mothers said that pregnant women will usually turn to their closest adult relatives (spouse, mother, siblings, in-laws, etc.) for assistance once they realize the delivery is imminent. They will then contact the TTBA. Male relatives will find a female to look after his wife until the TTBA arrives. Mothers also reported in FGDs that in addition to attending deliveries, TBAs are also helping with initiating breastfeeding; providing health education on STIs, reproductive health and nutrition; visiting mothers during and shortly following delivery where they educate them on the associated danger signs; and accompanying referrals to the health facilities for complicated deliveries. Mothers also reported that some of the main reasons they elect to give birth at home are because TBAs by tradition do not charge for deliveries and are willing to make house visits, which allows the mother the privacy that many desire common to the tradition of ‘hiding’ pregnancies. (The hospital charges 25-50 Birr for deliveries depending on complications, however, health facilities only charge for the medicines.) Indicator 24: Feasibility and results of implementing CB-ARI/Malaria case management, MN/LSS, and/or BHTs, through CS-17, presented at conference(s), in publication, through media, and/or site visit. Progress: Sister Degefech Haileyesus, the SC/Liben MCH Nurse and HBLSS Coordinator, gave a formal presentation entitled “Lessons from Ethiopia: Maternal Care in Low Resource Settings,” at the April 15, 2002 Vienna, Austria Conference, “Low Tech, High Effect: Care for Women and Infants in Disasters” sponsored by Johnson & Johnson. Her presentation focused on the project’s experience in designing and training traditional birth attendants in the home-based life saving skills (HB-LSS) curriculum in conjunction with the American College of Nurse Midwives, the first such attempt in Ethiopia. In addition, SC/Ethiopia has received a request from CARE/Ethiopia for Sister Degefech to assist them in implementing HB-LSS in their CS program in west Harage, with the support of a NGO Networks for Health program that supports cross visits between PVO and NGO programs. Indicator 26: Percent of rural PAs with TBAs training in HB-LSS. Progress: This objective has been met as each of Liben’s 36 rural PAs has at least one TBA who has completed the project’s HB-LSS training. In addition, the project trained all of the Negelle Hospital midwives in LSS in October 2002. Plans are underway to complete the training of the MOH staff in LSS by the end of the year. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 19 Indicator 28: Percent of mothers/newborns with complications for which TBAs reported completing all HB-LSS steps correctly. Progress: The number of complicated deliveries, required for the denominator in this indicator, is not available. Instead, the reports of 14 complicated deliveries were reviewed. It was found that 36% (5/14) of TBAs had completed all the HB-LSS steps correctly. One possible reason for the lack of an accurate total number of complicated deliveries could be due to the difficulty many TBAs are having in completing and submitting the reporting forms. The project designed these forms recognizing that a majority of the TBAs, especially the older ones, are illiterate. Therefore, a series of drawings are used to convey the information to be collected and the TBAs have been trained in how to ‘read’ these drawings and complete the form. The challenges appear to be in the length of the reporting form which is six pages and remains difficult to use even with the training and use of the pictures. Indicator 30: Percent of mothers who report knowledge of at least two maternal danger signs during the postpartum period. Progress: This objective will be measured at the FE using the KPC survey. The main strategy the project has used to increase mothers’ knowledge on maternal danger signs is health education provided by the TBAs, HACs and BHT members. Each of these groups has received an orientation to HB-LSS, which also covers postpartum emergencies, such as bleeding which is the most common cause of maternal mortality. Effectiveness of the intervention: The main challenges voiced by TBAs were the lack of safe delivery kits and gloves – especially following the training they received on HIV/AIDS prevention. Their sense was that families would be willing to cover the costs of these supplies, if they have the money, however, there are no current plans for providing them. They expressed concern about the challenges associated with complicated deliveries due to the lack of EmOC in the district, the difficulties of transportation, and the lack of communication equipment. Finally, several noted that with the high number of home deliveries many pregnant women are still being attended by untrained individuals rather than project trained TBAs and that many of the project trained TBAs are getting too old, therefore, there is a need to recruit and train younger TBAs. There is some evidence that communities are beginning to address the problems posed by obstetric complications through the development of emergency transportation plans and pooling funds to cover these costs when needed. However, the difficulty in Liben is that none of its health facilities, including Negelle Hospital, is able to respond to obstetric emergencies. It is seven hours over 300 kms of bad road to go from Negelle Hospital to the nearest health facility sufficiently equipped to provide c-sections. GOE policy dictates that only obstetricians can be trained to do c-sections, even though few of the hospitals throughout Ethiopia, including Negelle Hospital, have obstetricians on staff. With an easing of this policy and appropriate training, general practitioners could do this basic procedure. In addition, Negelle Hospital is not equipped to do blood transfusions and it does not have access to a 24/7 electrical power source. The EFO has pursued funding to cover the costs of the training and the necessary infrastructure improvements for Negelle Hospital, but without success to date. Further, Negelle Hospital is not CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 20 reviewing complicated deliveries as a regular policy to ensure that all the procedures are understood and being followed by the TBAs and the facility staff. The project’s plan is to provide ANC weekly through the health facilities and monthly through each of the 14 outreach sites in response to the low rates found in the baseline KPC. The quality of ANC was found to be problematic, based on a review done one week prior to the full MTE by Dr. Winnie Mwebesa, FP/ RH Advisor from SC/Washington. Iron and TT were not being consistently given to pregnant women, presumptive treatment of malaria for pregnant women was not being done uniformly, and stockouts of ANC cards was common at health facilities. (See Annex I for Dr. Mwebesa’s Trip Report.) Changes in the technical approaches outlined in the DIP and rationale: None Special outcomes, unexpected successes or constraints: None Recommendations: 4. The EFO, along with other NGOs, multilaterals and government agencies involved in maternal health, should advocate for a change in the GOE policy that restricts the use of c-sections and other EmOC surgical procedures to obstetricians, so that general practitioners can be permitted to do these procedures upon completing an appropriate training program. The EFO and SC/HQ should also explore and pursue funding opportunities that could cover the costs of training a surgical team from Negelle (one to two general practitioners, one scrub nurse and one anesthesiology nurse) and assist them in developing the capacity to do blood transfusions and provide electricity 24 hours/day, 7 days/week. 5. The Project should assist Negelle Hospital in establishing a regular system for reviewing complicated deliveries within the District, interviewing the involved TBA, health workers and women to assess whether all the necessary steps were followed and identify areas needing improvement. 6. The project needs to develop sustainable mechanisms for ensuring a consistent supply of delivery gloves for TBAs. iii. Immunization Activities proposed in the DIP: To increase immunization coverage for women and children, the CS-17 DIP committed the project to strategies that will: Increase Supply: In health facilities where static immunization service is being provided, efforts will be made to gradually increase EPI service provision to a daily basis to reduce missed opportunities. Two new health facilities will be provided with cold chain equipment. Outreach activities will be jointly planned with the DHO and the concerned communities/HACs. Some of staff who were trained in EPI through the DAP have moved out of the district, so EPI training will be given to SC and MOH staff using MOH/UNICEF training modules. The topics include cold chain management, injection safety, and sterile technique. SC and the DHO will plan and CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 21 conduct joint quarterly supervision visits and give feed back to health facility staff on immunization performance. Increase Demand: To increase demand for immunization services in all areas where EPI is available, more attention will be given to health messages addressing immunization. Community workers will be provided with pictorial teaching aids and trained in their use. Community members will be mobilized for immunization through influential local and religious leaders. CHWs will identify drop-outs in their respective Ollas, Ketenas, and PAs, and counsel family members to use immunization services. HACs will organize a Village Health Day every month to facilitate immunization outreach sessions. Promote Sustainability: To promote sustainable improvements in immunization services and coverage in the least accessible parts of the district, CS-17 will adopt selected elements of WHO’s EPI strategy for Sustainable Outreach Service (SOS). SOS aims to deliver tailor-made immunization services on the basis of periodic contact with people who have limited or no access to health services due to their geographical remoteness. In Liben, there will be three rounds of immunization sessions per year using all antigens, allowing children to be fully immunized after one year of SOS services. ANC and FP service delivery will be integrated with EPI outreach activities. This strategy will first be piloted for one-year period in the PAs where there are no EPI services. Then after assessing whether the approach is successful in Liben District or not, it will be scaled up to cover more PAs if found successful. Number of PAs to be Covered by Immunization Strategy and Year Strategy Year 1 Year 2 Year 3 Year 4 Year 5 Static 8 10 10 10+ * 10+ * Outreach 28 22 20 18 16 SOS 4 6 8 10 12 * In the 4th and 5th year, the MOH may construct additional facilities. Progress toward benchmarks or intermediate objectives: Indicator 16: Percent of all mothers of children <2 receiving TT2+ before last child’s birth (by card) – per the KPC. Indicator 17: Percent of pregnant women receiving TT2+ – per DHO records. Progress: TT2+ coverage in the District increased from 26% to 35% based on data from Negelle Hospital. While the end of the project target is 55%, this increase represents a significant improvement in TT2+ coverage. However, considering the problems associated with ANC presented previously in the MNC section, it is apparent that more can be done to improve the rates further through supervision and refresher training. Indicator 18: Percent of all 12-23 month olds who received measles immunization (by card only) – per KPC. Indicator 19: Percent of infants who received measles immunization – per DHO records. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 22 Progress: While the first indicator (#18) will not be measured until the FE KPC survey has been implemented, Negelle Hospital data found that the second indicator (#19) has already been fully achieved, having increased from a baseline rate of 45% to 72% at the midterm evaluation. Indicator 20: Percent of all 12-23 month olds fully immunized (by card) – per KPC. Indicator 21: Percent of infants fully immunized – per DHO records. Progress: Complete coverage has also surpassed the project’s final target of 60%, by increasing coverage from a baseline of 34% to a midterm rate of 62% per Negelle Hospital data for the District. Effectiveness of the intervention: While the current picture of immunization coverage shows positive progress, significant challenges remain between now and the end of CS-17. At the community level, mothers expressed the need for more opportunities to immunize their children, which they said could be addressed with adequate supplies of vaccines in the health facilities and by providing more outreach sessions. Liben District has suffered from both nationwide and regional stockouts of DPT and measles vaccines since the beginning of the project. Within the health facilities, it was also found that the supply of vaccines has been limited due to the lack of refrigerators at two of the district’s nine health facilities. The project was supposed to provide these two refrigerators and the accompanying cold chain equipment by October 2002, however, this was not done due to insufficient funds in the budget. In addition, not all of the health facilities have EPI cards, which the project was also supposed to provide by October 2002. Further delays will undermine the project’s ability to increase coverage. The project has committed to purchasing and distributing enough cards to last to the conclusion of CS-17 by the end of August 2003. Although mandated to monitor and supervise the District’s cold chain, the DHO is not equipped to do this on its own due to its lack of skilled staff and vehicles. The acting EPI Coordinator at Negelle Hospital has provided some technical guidance to the DHO. The project is supporting the MOH’s immunization efforts by providing a one-week training on immunizations to health facility staff, assistance in soliciting 80 donated syringes from GAVI, and the provision of fuel and per diem for supervision and vaccine transport from Addis Ababa to the District. The project has used the SOS strategy, to integrate its ANC and FP outreach services with immunizations, focused on three hard to reach PAs in 2002 and five in 2003, so it is behind by two PAs at the time of the MTE when compared to the table above. Also at the community level, some negative attitudes about childhood vaccinations remain that need to be addressed through BCC strategies. When asked why some mothers are not getting their children immunized, FGD respondents reported that many are concerned about the side effects particularly diarrhea, fever, and irritability. It was also mentioned that childhood immunization card retention is weak because parents often discard the card once the child has completed his/her childhood immunizations. This is compounded by the previously mentioned stockouts of childhood immunization cards at some of the health facilities. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 23 The primary challenge to achieving sustainability of the immunization activities relates to the ongoing costs that Save the Children is currently covering, including the fuel to transport staff to/from outreaches and vaccines from Addis Ababa to the field; the kerosene to power the refrigerators; and the per diem for MOH staff during outreach sessions. Unless plans are made to cover these costs by the end of the project, the sustainability of the immunization intervention is in doubt. Changes in the technical approaches outlined in the DIP and rationale: None. Special outcomes, unexpected successes or constraints: None. Recommendations: 7. Providing the promised cold chain equipment and EPI cards as soon as possible needs to be a priority as further delays will directly limit the intervention’s impact. 8. The project needs to work closely with the HACs, BHTs and TBAs through the SATs to further assess the nature and underlying causes for any community misconceptions about the safety and efficacy of childhood vaccines and develop responsive IEC strategies. iv. Pneumonia Case Management Activities proposed in the DIP: Prompt Care Seeking: Recognition of pneumonia signs and prompt care seeking will continue to be promoted through education of community members by BHTs and HACs. Education will focus on recognition of pneumonia in older infants and children, on recognition of signs in young infants (through messages specifically for pregnant women and mothers of newborns), and on prompt care seeking. CS-17 Approach to Improving Access to Case Management: CS-17 will continue to support activities introduced through CS-13, but hopes to focus on substantially improving access to pneumonia case management services, initially on a pilot basis in approximately five PAs with poor access to health facilities, by training approximately two literate CHWs per PA in case management. Quality of Case Management: CS-17 will use materials developed by CARE with support from the U.S. Centers for Disease Control and Prevention (CDC) for training CHWs in Siaya, Kenya in case management of pneumonia and malaria, which are based on the WHO materials for training CHWs in ARI case management. CHW training activities will include practice at a health facility in assessment of ill children and counseling of mothers, and viewing and discussion of the WHO ARI case management video. CHWs will be required to successfully pass an assessment of their case management skills before being allowed to treat children in their communities. UNICEF beeping timers or watches with second hands will be provided to all health workers trained in SCM. CHWs will be trained to treat children with signs of non-severe pneumonia with oral Cotrimoxazole. All severe pneumonia, very severe disease, and cases of Cotrimoxazole treatment failure, will be referred to health facilities. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 24 Progress toward benchmarks or intermediate objectives: Indicator 4: Percent of CMWs trained in pneumonia case management with no stock-out of Cotrimoxazole in the previous month. Indicator 5: Percent of CMWs trained in malaria or pneumonia case management through CS￾17 from whom reports were received in past quarter. Indicator 12: Total rate of treatment for pneumonia in <5s by CMWs in all PAs with CMWs trained in PCM (number of treatments per <5 per year). Indicator 27: Percent of children under five assessed for pneumonia for which CMW reported completing all PCM steps correctly. Progress: Because of GOE policy restricting the use of community health workers to distribute antibiotics, antimalarials and other controlled pharmaceuticals, the training of CHWs in case management of childhood illnesses has had to be delayed. This is discussed in greater detail below in Section B.2.b. Community Case Management of Childhood Illness. Indicator 25: Percent of rural PAs which have an MOH facility or CMW(s) trained through CS￾17 in ARI or malaria case management. Progress: Each of the eight Health Facilities serving the 36 rural PAs in Liben District has at least one staff person who attended the project training on ARI and malaria case management of the sick child that was provided April 2003. Indicator 29: Percent of mothers reporting either fast breathing or difficult breathing as a sign of child illness needing treatment. Progress: This indicator will be assessed at the FE using the KPC. Effectiveness of the interventions: Because of the delays in starting Community Case Management and the lack of a KPC survey at the MTE, the effectiveness of this intervention cannot be measured or assessed at this time. Changes in the technical approaches outlined in the DIP and rationale: None. Special outcomes, unexpected successes or constraints: The primary constraint with regard to Community Case Management has been the delay in getting GOE approval to proceed with the intervention. The EFO has been actively advocating for the startup of this with the GOE and the National IMCI Task Force in Addis Ababa. Recommendations: See Section B.2.b. Community Case Management of Childhood Illness. v. Control of Diarrheal Disease Activities proposed in the DIP: The CS-17 CDD intervention will focus on continuing CS-13 efforts of educating caretakers about prevention and home care for diarrhea through BHT and HAC members, and community-based case management. CHWs involved in malaria and CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 25 pneumonia case management will assess ill children for the type of diarrhea (acute watery, bloody, and/or persistent) and severity of dehydration, treatment or refer based on the assessment, and counsel caretakers on home care and prevention. A focus of training and supervisory activities will be on improving the skills of CHWs to effectively counsel caretakers about: the early use of fluids available in the home (including milk, yogurt, cereal-based gruels, soups, and water); continued breastfeeding; frequent feeding of small amounts of food; catch-up feeding following recovery; recognition of and prompt care seeking at health facilities for dehydration, dysentery, and persistent diarrhea; and prevention, including hand washing and safe disposal of human feces. CS-17 will attempt to provide CHWs trained in case management with ORS packets for free distribution for children with signs of (non-severe) dehydration or profuse watery diarrhea. Progress toward benchmarks or intermediate objectives: Indicator 22: Percent of children <2 with diarrhea in the past 2 weeks receiving more fluids than usual and same or more food than usual during illness. Indicator 33: Percent of mothers with children <24 months who report washing own hands with soap or ash before food prep., before feeding children, after defecation, and after attending child who defecated. Progress: These two indicators will be assessed at the FE using the KPC. Effectiveness of the interventions: Because of the delays in starting Community Case Management and the lack of a KPC survey at the MTE, the effectiveness of this intervention cannot be measured or assessed at this time. Changes in the technical approaches outlined in the DIP and rationale: None Special outcomes, unexpected successes or constraints: See Section B.2.b. Community Case Management of Childhood Illness. Recommendations: See Section B.2.b. Community Case Management of Childhood Illness. vi. Control of Malaria Activities proposed in the DIP: CS-17 hopes to introduce community-based case management of childhood malaria, along with community-based case management of pneumonia, initially on a pilot basis in five PAs, training approximately two literate CHWs per PA, and using the same training materials adapted from those of CARE/Kenya, as will be used for the pneumonia intervention (please see the description of the pneumonia intervention above). As with pneumonia, CHWs will be trained to treat children with signs of non-severe malaria, and refer all children with signs of severe illness and cases of treatment failure. Monitoring and supervision of CHWs in malaria case management will be integrated with monitoring and supervision of CHWs with regard to ARI case management, and scale-up malaria case management activities through training additional CHWs in Liben District based on a thorough review of pilot malaria and ARI case management activities. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 26 Progress toward benchmarks or intermediate objectives: Indicator 5: Percent of CMWs trained in malaria or pneumonia case management through CS￾17 from whom reports were received in past quarter. Progress: Cannot be assessed until after the start of the CCM strategy. Indicator 25: Percent of rural PAs which have an MOH facility or CMW(s) trained through CS￾17 in ARI or malaria case management. Progress: Each of the eight Health Facilities serving the 36 rural PAs in Liben District has at least one staff person who attended the project training on ARI and malaria case management of the sick child that was provided in April 2003. Effectiveness of the interventions: Because of the delays in starting Community Case Management and the lack of a KPC survey at the MTE, the effectiveness of this intervention cannot be measured or assessed at this time. Changes in the technical approaches outlined in the DIP and rationale: None. Special outcomes, unexpected successes or constraints: See Section B.2.b. Community Case Management of Childhood Illness. Recommendations: See Section B.2.b. Community Case Management of Childhood Illness. c. New Tools, Approaches and Operations Research CS-17’s training and support of TBAs in Home-Based Life Saving Skills is the first such effort attempted in Ethiopia. Therefore, it is being closely monitored by SC, ACNM and other PVOs for lessons learned and the potential for replication in similar locations and scaling up. As mentioned previously, CARE/Ethiopia has already expressed interest in piloting the same intervention in their project with the assistance of SC/Liben staff. SC/Liben participated with Family Health International and UNICEF in the first nationwide HIV Behavioral Surveillance Survey. The Survey found disturbing results specific to male and female pastoralists between the age of 15 and 49 in Borena Zone, which includes the project area. Knowledge of HIV prevention methods was below 25%; those with accepting attitudes towards people infected by HIV/AIDS was below four percent; and condom use of less than five percent. 2. Cross Cutting Approaches a. Mobilization of Community Leaders and Traditional Practitioners Activities proposed in the DIP: Together the CS-13 and CS-17 projects have formed 147 Bridget to Health Teams that are responsible for mobilizing the community, specifically, communicating the project’s IEC messages, making and following up on referrals to health facilities, notifying the community about health outreach activities (e.g., ANC, EPI and RH outreach clinics), completing monthly reports on activities and vital events, and distributing family planning CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 27 supplies when they are available. Each BHT is made up of a wise woman (chaireti) who is often a TBA, a wise man (chiressa), and a younger ‘intern’ who is being groomed into a leadership role. The majority of the BHT members were selected by community consensus, based on the level of respect and esteem each is held. Every month the BHTs meet with their Health Action Committee (HAC) where they submit their written reports, receive supervisory feedback and plan future activities. The HACs are primarily responsible for supervising and supporting all the BHTs and TBAs in their area through these monthly meetings and collecting, reviewing and reporting on the monthly report forms completed by the BHTs and TBAs. In addition, they also support and participate in the same community activities as the BHTs. Each HAC has an average of 8-12 members who have been selected or elected by the community they represent. Both the HAC and BHT members have received training on each of the project’s health interventions and orientation on the project strategies. Progress toward benchmarks or intermediate objectives: Indicator 3a/b: Percent of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year. Progress: The meeting records or minutes necessary to assess this indicator were not available from each of the PAs or health facilities within Liben District at the time of the MTE. This indicator is going to be difficult, if not impossible, to quantify at the Final Evaluation as well. It will require reviewing lists of meeting attendees taken from MOH meeting minutes and comparing them with the list of HAC members for each PA. Anecdotally, though, it appears from the FGDs with HAC members and the SATs that the relationships between the HACs and the health facilities have improved significantly over the life of the project and that this strategy has helped to strengthen the connection between the community and the health care system. Each demonstrated in the FGDs an understanding of the roles and responsibilities of the other. Indicator 6: Percent of BHTs which in the last six months have conducted one or more community education activities for each CS-17 intervention and turned in four or more monthly reports to HACs. Indicator 7: Percent of HACs which in the last six months have reviewed BHT, TBA, or CMW reports, and have sent reports to a health facility. Progress: These two indicators could not be measured using the current HIS forms. During visits to health facilities to review BHT and HAC reports, it was apparent that the monthly report forms from the BHTs and TBAs were not being completed correctly from site to site. Some were being filled out by the individual BHT or TBA correctly, while other BHT and TBA reports were being prepared by the HACs from memory, consolidating all the results onto a single form. This is discussed in greater detail in section B.3.g. Information Management. Effectiveness of the BHTs and HACs: The establishment of BHTs and HACs has proven to be a very popular strategy in the eyes of the community and the MOH both. Mothers and health facility workers know who their BHT and HAC members are, even though they are not always CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 28 clear on the different roles of the two groups, and have also voiced support for recruiting and training more CHWs. The use of traditional leaders in these roles has also proven to be a good strategy in that they bring credibility to their work with the project and the IEC messages they communicate. Traditional treatments and herbal medicines continue to be popular for a variety of ailments. The project has focused behavior change strategies on limiting dangerous traditional practices, such as female circumcision, tooth extraction for treating diarrhea in small children, scarring/burning, uvulectomies to treat sore throat/cough, abduction practices, extra marital relations, and use of herbal/traditional medicines. The use of these practices can be harmful and, at a minimum, can delay effective medical treatment (e.g., giving donkey milk to treat malaria symptoms before seeking medical care). As traditional leaders, the BHT members are the best equipped to address these issues in the community. There was some initial confusion in the communities regarding the roles of the members of the BHTs and the HACs. There was a broadly held assumption that they were being paid by the government or the project, so many in the community were not as supportive at first. Now even though many understand that they are volunteers, there is a growing awareness that there needs to be some type of incentive or reward to support the continued work of the CHWs. There is also a growing concern that the health education the BHT and HAC members are providing in the community is creating a demand for health care services that the MOH is not adequately equipped to respond to, especially the supply of drugs, supplies (safe birthing kits and gloves) and health care services (including regularly and consistently available immunization, ANC clinics and comprehensive EmOC). CS-17 included HB-LSS and CCM components to at least partially address this demand, however, the delay of the startup of CCM and several other project activities has resulted in some degree of frustration in the community and the health facilities. Recommendation: 9. The project needs to identify and institutionalize rewards the communities are willing and able to provide as incentives to the CHWs to ensure their continued volunteering with the project and the ability to recruit new CHWs as others move on. b. Community Case Management of Childhood Illness Activities proposed in the DIP: To respond to the need for case management of the most common childhood illnesses in Liben District, an area where a majority of the population has woefully poor access to facility-based health care services, the CS-17 DIP proposed to pilot and scale up a Community Case Management component. During the initial pilot phase two to three literate community volunteers were to be recruited from each of five of the most secluded PAs and trained as Case Management Workers. They were to be trained in each of the project’s health interventions and also case management of pneumonia, malaria and diarrhea, including the provision of antibiotics and antimalarials. They would work under the supervision of the Services Area Teams at the health facilities. Based on the results of the pilot phase, CCM will then be scaled up to cover the remaining isolated PAs within Liben District. The DIP mentions the development of a Revolving Drug Fund (RDF) in year two of the project as a possible mechanism for ensuring sustainable access to high quality, low cost drugs at the community level in conjunction with the CCM strategy. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 29 Progress toward benchmarks or intermediate objectives: The SC/Liben Training Coordinator, the Acting DHO from Liben and the head of the MCH Team of the National MOH visited CARE’s Child Survival Project in Siaya District, Kenya in February 2003 to view first hand CARE’s Community Management of Childhood Illness (CMCI) component with a view towards using the lessons learned and best practices to inform the design and implementation of CS-17. The team reviewed the training plans/curricula, the revolving drug fund, the reporting forms/system, and the overall strategy. After this experience-sharing visit, the health facility staff in Liben were oriented in the CCM interventions in early 2003. However, the CMWs have yet to be recruited and trained, which was scheduled in the DIP to start in January 2003. This delay was due to the conflict between the CCM strategy and the MOH policy that restricts community health workers from prescribing and dispensing or selling antibiotics and antimalarials. During the MTE the EFO was informed by the national MOH that the project would be granted permission to proceed with the CMW training in September/October 2003. This is only a ten month delay, yet it will be very difficult for the project to complete this training before the end of 2003, without the close support of the EFO/Health Advisor who has been reassigned for the remainder of 2003 to work on the emergency response to the famine occurring north of Liben. No work has been done to date on establishing the RDF. Further, experience in other projects has shown that it can take three+ years to pilot and scale up a RDF at the health facility level, allowing sufficient time to identify a consistent source of supply (wholesale), raise the funds required to purchase the initial startup stock of drugs, establish the financial systems (bookkeeping, inventory, controls, reporting, etc.), establish/train an oversight committee, and train local pharmacists. Doing the RDF at the community level will require even more time and the specific expertise required for either option does not currently exist within Liben. Recommendations: 10. The direct and concentrated involvement of the EFO technical backstopping staff will be required for the adaptation of the CCM training materials and related IEC strategies as well as the overall planning and implementation of CCM. 11. Technical and administrative support is going to be required throughout the planning and early implementation of the RDF from skilled professionals who have relevant practical experience and solid theoretical grounding in RDF design and administration. c. Capacity Building Approach i. Strengthening Save the Children The CS-17 DIP lists four indicators for improving the capacity of Save the Children: Indicator 8: EFO Behavior Change Specialist hired by October 2002) and retained. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 30 Progress: This position was temporarily filled by transferring 15% of the EFO HIV/AIDS Specialist’s time to BCC efforts, which spanned all of SC’s programs in Ethiopia. It was noted by the project staff in Liben that this person did one site visit to Negelle in May 2003 to collect the project’s training curricula and has yet to return or respond with comments or feedback. The staff in Liben, especially the CS-17 Training Coordinator, were adamant in voicing their need for technical assistance and training in the design of appropriate and effective IEC and training materials. This position was filled by Ms. Ashley Aakesson in August 2003. She has had past experience in Ethiopia, including providing an orientation workshop on BCC for the Liben CS staff in March 2003 on overall BCC strategy and process. Indicator 9: BC strategy for all CS-17 interventions designed and implementation started (by March 2003). Progress: The new BCC Specialist began conducting formative research in Liben in August 2003 and doing capacity building with the staff on how to use data to inform an overall strategy and to design effective key messages and IEC materials. By January 2004 the overall strategy will be in place and training SPAs, MOH staff and CHWs will begin on the new strategy and messages. Indicator 10: HIV prevention efforts effectively integrated into ongoing community and government activities through CS-17. Indicator 11: Number of HIV-related training courses, workshops, and experience sharing visits in which SC/Liben staff have participated during CS-17. Progress: See the HIV/AIDS Intervention section above for a detailed report on progress related to indicators 10 and 11. The CS-17 DIP cited two assessment tools used by Save the Children to measure its capacity: ÿ The Institutional Strength Assessment (ISA) developed and implemented by the Child Survival Technical Support (CSTS) unit at MACRO to assessed SC’s Office of Health at its headquarters in Westport in early 2002. ÿ The Organizational Capacity Assessment (OCA) implemented by PACT assessing the capacity of the EFO in Addis Ababa after the DIP was completed. The scope of both of these assessments went far beyond this CS-17 project, where the ISA looked at the Office of Health’s backstopping capacity with regard to its support of all eight of SC’s Child Survival projects. The OCA assessed all of Save the Children’s activities in Ethiopia and the support the EFO receives from headquarters. For the OCA, only three individuals directly involved in CS-17 were interviewed – Dr. Dr. Taye Tolera (the Negelle Hospital Medical Director/former Chair of the DHMT), Tsegaye Sonto (the SC Health Sector Manager at that time in Liben), and Dr. Tedbabe Degifie (the Health Specialist from the SC/EFO newly responsible for backstopping CS-17). None of SC’s Unit Heads, Coordinators, SPAs or project￾trained CHWs working at the community level were interviewed as part of the assessment. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 31 There are indications that the project has increased Save the Children’s capacity. The support of training and educational opportunities for the senior health staff (Reference Section B.3.b. Staff Training) has increased their technical skills in areas directly related to the project objectives and activities. The development of the HB-LSS training in conjunction with ACNM has increased the organization’s ability to improve the health of mothers and their children in areas such as Liben that lack ready access to comprehensive obstetric care. Significant capacity building needs remain to be addressed, however, particularly in the form of technical and administrative assistance to the staff in Liben as discussed later in Section B.3.h. Technical and Administrative Support. ii. Strengthening the Ministry of Health SC’s primary local partner is the District Health Office, which at the time of the CS-17 DIP Workshop, was headed by the senior staff from Negelle Hospital who also chaired and made up much of the membership of the District Health Management Team (DHMT). The overarching project strategy in the DIP was to build the capacity of these institutions so they would be well prepared by the conclusion of CS-17 to provide leadership and direction over the continuing program activities and improved services established during CS-17. The GOE’s subsequent adoption of a policy of ‘decentralization’ has moved the authority and responsibility for planning, monitoring and budgeting of all district-wide health services from Negelle Hospital to the DHO. Now the DHO oversees the district’s eight health facilities (including five clinics and three health posts, but not the hospital) and reports directly to the District Administrative Council. Decentralization has had the immediate effect of transferring the District’s health leadership from the hospital staff to a newly formed District Health Office. The DHO, however, is significantly understaffed in terms of the number of staff positions (only one out of five mandated positions have been filled to date) and in technical and administrative capacity. (The DHO is now headed by a health officer rather than a physician as previously and the Unit Heads from Negelle Hospital with responsibility over EPI and MCH are no longer directly involved.) The Negelle Hospital staff have, to their credit, continued to support and coordinate with the project and the new DHO when possible, however, without a policy mandate, their continued direct involvement with the project can no longer be assumed. The CS-17 DIP included the following three indicators for measuring the process of building the capacity of the local partners: Indicator 1: District Health Management Team (DHMT) has met three or more times in last year and has used data to plan activities. Progress: Largely due to decentralization and the situation described above, the DHMT has not formally met since October 2002. It did meet at least five times in the preceding six months (between April and October of 2002) primarily to orient the new DHO and discuss program activities. There was no evidence in the minutes of using data to plan activities, probably since these meetings were directly following the DIP Workshop and the HIS was not prepared at that time to produce project data for use by the DHMT. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 32 Indicator 2: District HIV/AIDS Council meets regularly, plans, and monitors HIV/AIDS activities in Liben. Progress: Refer to Section B.1.b.i. STI/HIV/AIDS Prevention. Indicator 3: Percent of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year. Progress: Refer to Section B.2.a. Mobilization of Community Leaders and Traditional Practitioners. SC has actively sought the participation of the DHO, the Zonal Health Department, and the Negelle Hospital staff in all aspects of the project’s design, implementation and evaluation. However, because of the size and mandate of the principal partner, the DHO is overstretched and unable to contribute to the project as when it was led by the Hospital staff. Therefore, the project is at a critical juncture with regard to capacity building of local partners and ultimately to the sustainability of its efforts. This discussion is continued later in Section B.2.d. Sustainability Strategy. Recommendation: 12. The project needs to reassess the current and projected capacity building needs of the new DHO within the context of the remaining CS-17 project and the long-term needs related to its sustainability plans. This could require significant restructuring of the project, especially if a majority of these needs cannot be resolved through locally available sources. iii. Strengthening Health Facilities and Workers The main project strategy for building the capacity of the health facilities and their staffs is the Service Area Team (SAT). One SAT has been formed at each of six health facilities in Liben District. Each SAT is made up of one SC Senior Program Assistant (SPA) and the lead staff at the Health Facility. (Three more SATs will be established once the remaining three SPAs are hired by SC.) The SATs are responsible for providing technical and supervisory support to the HACs, BHTs and TBAs; documenting quarterly meetings with the HACs; providing training; and assisting the HACs in coordinating community health activities. The SPAs also channel technical support and training content into the health facilities from the SC/Unit Heads, including health education materials on HIV/AIDS, etc.. One area that would benefit from improvement is the documenting of meetings through minutes. This is especially important for CS-17 because several of the project indicators are measured based on these records. FGDs with SAT and HAC members found that the use of the SAT strategy has been effective and has strengthened the connections between the health facilities and the communities. To help ensure that the growing demand for services does not outstrip the currently available supply, the project needs to proceed quickly in providing the promised cold chain equipment (two refrigerators and EPI cards) and developing the revolving drug funds, so that when mothers bring CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 33 their children to the health facilities they can be reasonably assured that it will have the necessary vaccines and medicines will be available. Recommendations: 13. Training on recording meeting minutes is required for the SATs and through them, the HACs. See Recommendation 22 to hire remaining three SPAs in Section B.3.c. Supervision of Program Staff. See Recommendation 7 for supplying the remaining cold chain equipment and supplies in Section B.1.b.iii. Immunization. iv. Training The CS-17 training strategy has incorporated the following features: ÿ Joint planning with the MOH in all aspects of developing the training strategy and also involving the Ministry of Education, the community and religious leaders in the selection of topics and trainees, curriculum design, and logistics in an effort to build local capacity in planning and ensure compliance with MOH policies. ÿ The use of pre-existing training curricula, plans and materials from state-of-the-art international and Ethiopia-specific sources that are tested, adapted, and modified by the project to fit the local situation, including the training of low or semi-literate community volunteers. ÿ The use of pre- and post-tests to assess immediate comprehension of the training content and project-designed supervisory checklists used during supervisory visits to assess long-term retention. ÿ The development of Master Trainers in the MOH for LSS using TOT strategies to increase MOH training skills and help to ensure the sustainability of the program. ÿ The use of regular refresher trainings to build long-term retention and solidify positive behavior change. Overall, the training plan presented in the CS-17 DIP is on schedule, except for the CCM training of Case Management Workers due to delays in receiving GOE approval. A brief review of the results of pre/post tests found positive improvement following the trainings, however, long-term retention was not reviewed during the MTE. BHT members said that the project training and refresher trainings were an important benefit and reward for their participation in the program. They requested more training but did not identify new topics that should be covered. SC/Liben staff noted the need for technical assistance and support in the design and adaptation of training curricula, manuals and materials as well as in monitoring and evaluating the quality and value of their training programs. These are important and timely needs for CS-17, especially as it prepares for the CCM activities and should be a priority for the EFO to address. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 34 One issue of concern has been the payment of per diems to individuals for attending project trainings. The SC/SPAs noted that once the volunteers are paid for their attendance at project trainings they have begun to express frustrations over not receiving per diem for attending the monthly HAC meetings. While there are logical reasons for paying volunteers to attend trainings (i.e., reimbursement for time diverted away from other income producing activities and/or to be consistent with what other nearby PVOs are providing their trainees) and the distinction between payment for trainings and meetings can be readily explained, the practice of paying trainees often leads to unrealistic expectations. It also tends to draw in training participants who are more interested in the per diem than in volunteering to provide health care services in their community. Recommendations: 14. Increase the capacity of the CS-17 staff in the design, implementation, monitoring and evaluation of training materials and programs through the provision of TA, support and resources from the EFO that can be made available to the staff in Negelle. 15. The extent of the frustrations expressed about per diems needs to be assessed further and, if necessary, senior project staff (i.e., the Health Sector Manager and/or the Impact Area Manager) need to meet with the CHWs to discuss this issue and come to consensus on solutions if need be. d. Sustainability Strategy As stated in the DIP, CS-17 continues the CS-13 approaches to sustainability “by nurturing (1) district, (2) community, and (3) household-level capacities to continue delivery and use of essential promotive, preventive, and case management services, for maternal and child health, with minimal external input.” To achieve this, the DIP sets out the following four strategies for sustainability and uptake. (1) Sustained support for community health committee, workers and activities. Progress: As planned, the success of this strategy rests to a significant extent on the capacity of the DHO. Due to the GOE policy of decentralization, the ability of the DHO to support these efforts can no longer be assumed in the near future or at least until the DHO’s staffing is significantly increased. The project’s use of Master Trainers for the HB-LSS training does hold promising potential since the capacity to train new TBAs and orient BHTs and HACs rests within the health facilities. In addition, the communities themselves can be a source of support for these efforts through their Community Action Committees (CACs) established as part of the DAP and the CS-17 HACs. They can help to pool community resources for incentives for the workers and to cover some of the costs of activities. Mobilizing the communities to undertake this responsibility is labor intensive, requiring significant time. However, a solid groundwork has already been laid by the project in its relations with the communities and the health facilities. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 35 (2) Sustained delivery of key maternal and child survival services at the community level. Progress: Together the CS-13 and CS-17 projects have had solid success in recruiting, training and supporting CHWs who are providing health education, health promotion activities, perinatal care, assistance at deliveries using HB-LSS, and referrals to health facilities. Services provided through the health facilities have also been strengthened through the training of health workers and the work of SC’s SPAs. With the successful training of CMWs, these services will expand to include case management of malaria, diarrheal disease and pneumonia. (3) Sustained increase in the use of key health services and improved MCH practices Progress: Increasing household knowledge of the project’s health education messages has been a major focus of the CHW training and support to date. Although change in household knowledge has not been measured as part of this MTE, the progress on CHW training and support demonstrated so far suggests that all the factors are in place for positive change. The CCM approach should further strengthen not only household level knowledge, but practices as well. Finally, the possibility of implementing a KPC survey using LQAS was considered for this MTE to assess household level progress on knowledge and practice indicators. It was considered too early in the project to be able to gauge measurable changes from the baseline. With the pending start up of CCM and an additional year of community level activities completed, the project should consider implementing an abbreviated KPC using LQAS to assess progress, identify areas in need of improvement and make course corrections, in the summer of 2004. (4) Innovative CS-17 approaches sustained through uptake of these approaches by the MOH or by other organizations in other parts of Ethiopia. Progress: Save the Children, with the support of ACNM, is implementing the first HB-LSS training of traditional birth attendants in Ethiopia. CARE/Ethiopia has requested the assistance of SC’s MCH Unit Head to assist it in designing and implementing the training as part of their CS project. As stated in the CS-17 DIP, “SC does not believe that it is realistic to hope for a health system in Liben capable in the near future of reaching most of the population with quality essential MCH services without substantial external inputs.” This was written prior to the transition in leadership of the DHO and DHMT, and is therefore even more accurate today. No further discussions have been held on the Project’s exit strategy subsequent to the changes in the DHO or the SC/Health Sector Manager position. A reassessment of what can continue beyond the conclusion of CS-17 needs to be done that answers the questions: What outside resources can be accessed; which project activities can realistically be continued; and what needs to be in place for this to happen? Members in each of the FGDs voiced commitment to continue their work and support of project activities after the conclusion of CS-17. However, there was also general consensus that without outside assistance in the form of on-going support, refresher training, and the recruitment of new replacement CHWs, this might not happen. Interest was expressed in developing income generation projects and pooling community funds to support the continuation of project activities, however, there was not a clear vision from the respondents on how this could be done. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 36 Recommendations: 16. The project needs to identify and access all currently and potentially available sources of support for project activities to continue beyond CS-17. 18. The project should consider implementing an abbreviated KPC using LQAS in the summer of 2004 to assess progress on household knowledge/practices and identify any areas in need of improvement. 3. Program Management a. Planning CS-17 has directly benefited from the active participation of SC and partner staff throughout. In February 2002, SC/Liben hosted a twelve-day DIP Planning Workshop attended by all ten members of its senior health staff, the Child Survival Specialist from SC/Westport responsible for backstopping this project, the DHO/Negelle Hospital Medical Director, the Head of the Zonal Health Department, SC Health Advisor from the EFO in Addis Ababa and a HIV/AIDS Consultant from SC. Representatives from the institutions officed in Liben have continued to participate in the planning and implementation of subsequent program activities, including committee meetings, project trainings, site visits, joint supervisor visits, and this MTE. Unfortunately, there have been several delays implementing the activities listed in the DIP work plan, largely as a result of three factors: the repeated turnover in the Health Sector Manager position, GOE delays in approving the project’s CCM pilot, and the change in the membership of the DHMT resulting from the GOE decentralization, as noted in the Technical and Cross-Cutting Approaches sections above. While the first two issues appear to have been resolved, the changes in the DHMT will continue to have an impact on the project especially with regard to achieving its sustainability goals. All the senior SC health staff, the Medical Director of Negelle Hospital and the DHO have copies of the Project Work Plan and Monitoring & Evaluation from the project DIP. However, there has not been an orientation of the new Health Sector Manager on these important project documents, which is discussed in detail later. Much of the Project’s HIS is still being set up and senior CS-17 staff have not been meeting consistently, so the use of project data is limited more to supervision and feedback rather than program planning at this stage. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 37 b. Staff Training Save the Children has identified and provided training opportunities for the majority of its senior Liben/CS staff on program-related topics, so that each one has attended at least one training workshop or activity since the beginning of CS-17. Staff Attending Training Topic Duration/Year Location Health Sector Manager Formative Research 6 days/2003 Awassa, Ethiopia Community Mobilization 3 days/2003 Participatory Rural Appraisal 12 days/2003 HIV/AIDS Unit Head Behavior Change and Communication 12 days/2003 Nazareth, Ethiopia Program Design and M&E 7 days/2003 Addis Ababa, Ethiopia Participatory Rural Appraisal 12 days/2002 Nazareth, Ethiopia MNC Unit Head “Low Tech, High Effect: Care for Women and Infants in Disasters” 3 days/2002 Vienna, Austria Training Coordinator Experience Sharing Visit to CARE/Kenya CS Project 6 days/2003 Siaya District, Kenya M&E Manager Participatory M&E 5 days/2003 Addis Ababa, Ethiopia M&E Tools 7 days/2002 Johannesburg, South Africa The senior CS/Liben staff voiced a need for further training and capacity building on the development of training materials/curricula, the development of IEC materials, and staff supervision. In addition the new Health Sector Manager expressed a need for training and support in NGO and financial management. The six Senior Program Assistants (SPAs) also expressed strong interest in receiving further training and participating in learning experiences that would help them in their current work with the project and their future careers. Suggested topics of interest included more in depth training on IMCI, LSS and TOT methodologies. In addition, they noted that because each SPA works in a completely different location, they have never had the opportunity to visit each others’ project sites within the District. This would be an excellent opportunity for each SPA to view the work of the others first hand while working on the job – sharing experiences, lessons learned and best practices. The requests for further training are timely and reasonable considering what needs to happen over the remaining three years. It is evident that the training received to date has related directly to the needs of the project. And while staff returning from trainings usually report back to their peers either in writing and/or verbally about their experience, the project has no formal mechanism for monitoring trainee performance in the newly acquired skill areas other than through the regular, annual performance review. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 38 Recommendation: 18. Implement a training needs assessment and plan for the staff with the goal of improving their work with the CS-17 project and also enhancing their employability upon its completion. This plan should include the SPAs as well as the senior health staff. c. Supervision of Program Staff Because of repeated changes in the person holding the Health Sector Manager position, the senior Child Survival staff have not been meeting regularly, but rather on an as-needed basis to address specific issue or problems as they arise. It is critical at this stage of the project, especially with its new leadership and its reliance on integrated strategies that the senior staff meet on a regular basis. Interviews with the senior staff found a fairly broad range in how frequently these meetings should occur, ranging from weekly to quarterly. While some raised concerns that these meetings would distract from other activities or would not be well attended due to other commitments, they need to be a priority. It is important that regular meetings with the SPAs and their supervisor, the Training Coordinator also be restarted. Each of the CS health staff, including the Health Sector Manager, the Coordinators, the Unit Heads and the SPAs, submit monthly, quarterly and annual reports to their supervisors that cover work done to date as compared with plans and plans for the following time period. A review of the reports found that they continue to be done on a timely basis and are used during performance reviews and staff meetings. In addition, each of Save the Children’s CS-17 program staff have had a performance review within the past twelve months, however, there appears to be some confusion among senior staff on whether these reviews are supposed to be done annually or every six months. Even though the new Health Sector Manager has only been in his position for less than two months at the time of the MTE and is new to SC, he has had some past experience with this Project, having participated on the CS-13 Final Evaluation Team and his association with many SC employees. So far he has demonstrated a sound understanding of the project and developed good working relationships with the CS-17 Project staff and the partners. Because his previous professional experience was almost entirely within the MOH, he has commented that he would benefit from further training and support in NGO and financial management. His orientation included review of the primary project documents and a one-day of meeting with the EFO staff. At the time of the MTE he had not been out to the field yet. It is important that he make formal visits to the project communities as soon as possible, which will help to demonstrate to the project volunteers, the SATs and the community members that the project is there to serve them. It can also help to tie the SPAs more closely in with the project and demonstrate management’s support for their work. In observing and talking with the CS staff it was readily apparent that this is a hard working team, frequently dedicating weekends and evenings to the project. Two related issues regarding workload need to be raised. Responsibility for both the Maternal Newborn Care and Community Case Management, which covers CDD, ARI and malaria rests entirely on the shoulders of the MCH Unit Head. Together these interventions are weighted at 55% of the total planned CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 39 intervention-specific effort. Sr. Degefech H. Yesus who has held this position for four years, did not raise this issue. However, it is clear that once CCM is granted permission to start by the GOE that its implementation will require her full attention, taking away from the work that remains to be accomplished on LSS and BH-LSS. The other area of concern relates to the SPAs. There are currently six SPAs working out of six of the District’s nine health facilities, so, one third of the health facilities and the populations they serve are not receiving direct project services. Three additional SPAs were supposed to be hired by July 2002, however, this has not happened yet due to the difficulties in finding qualified candidates from those locations and the transition in the Health Sector Manager position. With only three years remaining of the project and the prospects for initiating CCM strong it is important that these positions are filled as soon as possible so services can reach the entire project area. Recommendations: 19. Regular staff meetings, involving the Health Sector Manager, the Unit Heads and the M&E and Training Coordinators, need to be restarted as soon as possible. It is suggested that initially they occur on a weekly basis at the same time and place. They can then be scaled back as seems prudent. 20. The workload for the MCH Unit Head needs to be reviewed prior to the startup of the CCM intervention and adjustments made as needed, whether in the form of additional staff or adjustment in responsibilities with other staff. 21. The three remaining SPAs need to be hired, oriented and placed in the field as soon as possible. d. Human Resources and Staff Management The Health Sector Manager is responsible for both this Child Survival Project and the Food for Peace-funded Title II Development Assistance Program in Liben. Specific to Child Survival, the Health Sector Manager supervises the Training Coordinator and three Unit Heads who are responsible for Immunizations, HIV/AIDS and MCH. The Training Coordinator oversees six SPAs each of whom works directly with one health facility to form an SAT. Technical support and oversight is provided by the Health Adviser who is officed at the EFO in Addis Ababa and a Child Survival Specialist who is officed at SC’s headquarters in Westport, Connecticut. In addition, the M&E Manager in Liben is directly involved in the health program through the development and management of the HIS. Job descriptions for these positions are available in the SC/Liben offices and with SC’s HQ in Westport. It is not clear whether the DHO has copies of these documents. A single copy of the organizational personnel policies and procedures is available for review by SC staff in the Health Sector Manager’s office. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 40 The morale and cohesion of the staff appears to be positive and strong, especially among peers within each level of the organizational. The few frustrations that were voiced seem to appear when one ‘tier’ has to interface with another, for instance the program staff in Negelle’s request for additional support from the EFO in Addis or the SPAs seeking more training opportunities from their supervisors officed in Negelle. These issues do not appear to be having a negative impact on the project overall and fall well within the range of normal relationships within an organization. Turnovers in staffing have only occurred with the Health Sector Manager position, which has changed at least once since the beginning of CS-17 and more frequently during CS-13. As the most important single position within the project, the transition from one Manager to another can mean a lost year to a project, requiring at least that long for the new person to be acclimated and become functional in decision making, supervision and delegating, These transitions have resulted in delays in implementing program activities, delays in making important decisions (especially those highlighted in the Response to Final Evaluation Recommendations), frustrations for the staff, and the diverting of staff time and energy away from project activities to orienting and training in replacements. Unfortunately, exit interviews were not done when these transitions occurred so valuable opportunities to make changes, which could further support staff retention were not taken. Although several important changes were made in the hiring of the current Health Sector Manager, such as an increased salary based on characterizing Negelle as a hardship post and allowing greater autonomy in decision making. Retention of the other CS staff has been very positive, with all having started with the organization well before the start of CS-17. The SPAs have been with the project between 2 ½ and 4 ½ years. Each of the Negelle-based staff have been with the project for at least two years. The project currently has no plans to assist its employees to find other employment opportunities after the conclusion of CS-17, especially since the project still has over three years remaining. The experience they have had working with an international NGO such as Save the Children enhances their future employment potential greatly. It is assumed that most will move on to other projects within Save the Children/Ethiopia, return to jobs within the MOH or seek employment with other international NGOs. e. Financial Management The SC/Liben Impact Area Manager, SC/Liben Administration Manager and the Health Sector Manager together have primary responsibility for developing and monitoring the CS-17 budget at the district level. In addition, the SC/Unit Heads and the SC/Training Coordinator provide input into the budget development process and some receive financial reports that cover the entire CS-17 project without providing detail on each intervention or unit. The partners have been involved in the preparation and monitoring of project budgets in the past, however, not since the change in the DHO structure. f. Logistics Liben District’s population is diffusely spread out across a wide area. This along with the lack of paved roads, consistent electricity and communication systems throughout most of the District CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 41 makes a community-based health promotion program a logistical challenge. The lack of transportation is the primary problem for the project. The District Health Office is often unable to receive supplies, medicines and vaccines because of the lack of a vehicle, so the project has had to drive them from Addis Ababa or the Regional Health Bureau to the District, which will be difficult for the government to sustain in the future. Only two of the six (and soon to be nine) SPAs have a motorcycle, which greatly restricts their access to the community, a critical factor especially with the proposed start of the CCM activities and the establishment and support of the Revolving Drug Fund pharmacies. Recommendation: 22. With the delays experienced to date and the amount of work that needs to be accomplished in the most rural, underdeveloped PAs in Liben District by the end of CS-17, SC needs to provide one motorcycle for each SPA. g. Information Management A single indicator was included in the CS-17 DIP related directly to the HIS: Indicator 7: Percent of HACs which in the last 6 months have reviewed BHT, TBA, or CMW reports, and have sent reports to health facility. Progress: This indicator could not be assessed due to the inconsistency in reporting from the CHWs up through the HIS as described below. A health information system (HIS) was designed in the CS-17 DIP that encompasses data collection at the community level, reporting up through the system to the District Health Management Team where it is to inform management decisions, and the provision of feedback at every level. At this stage the HIS is still in development with several issues remaining to be worked out before it is a fully functioning system. The CHWs (including the BHTs, TBAa and in the future, the CMWs) record their activities and vital statistics they collect on forms designed by the project. The HACs are charged with collecting and reviewing these forms every month and providing feedback to the CHWs. Then the HACs submit the completed forms and a tally to the SATs working at the health facilities, who use the data to complete their monthly reports for submission to the Liben HIS Team and provide feedback to the HACs. Discussions with HAC members and project staff found several issues that need to be addressed regarding data collection at the community level. The reporting forms used by the BHTs and TBAs were developed in CS-13 and adapted for use in CS-17, albeit inconsistently. Because of the low literacy rates within the CHWs, these forms have incorporated drawings in place of terminology, which has resulted in excessively long forms (six pages for the TBA forms). Staff report that they are costly to reproduce and tend to be difficult for the CHWs to read, understand and complete, especially the older ones who do not have as much experience with record keeping and reporting. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 42 During site visits to health facilities to review the reporting forms, the MTE Team found that many of the CHWs were not completing the reporting forms themselves, but coming to the HAC meetings and reporting their data from memory while the head of the HAC would complete a single form for the entire HAC. While using recall is adequate for purposes of providing direct feedback to the CHWs at the HAC level, it is probably not sufficient for accurately monitoring community level project activities and vital events. The project needs to reassess its information needs starting at the community level, focusing initially on the information the CHWs will use to improve their work. The decision on which information to be collected should be based on how it will be used by the project. If this cannot be determined for a particular piece of data, then that piece might not need to be collected. This discussion needs to involve representatives from the CHWs to ensure they support the process and recognize the benefits. Otherwise, the quality of the data collected and its usefulness will suffer. This process has already begun during the field visit made by Stanley Foster in August 2003. (Please see Annex H.) SC developed a computerized software program for use by the DHMT, when it was being led by the Hospital staff. Five members of the Hospital staff were trained on the software and have been using it, however, for the hospital population and not the entire district as was originally planned. With the subsequent changes that have occurred in the DHO and the DHMT, the capacity of the DHO to effectively use a computerized system needs to be reassessed, especially since the M&E Coordinator has noted the difficulty in using the software. Recommendations: 23. The project needs to reassess its information needs starting at the community level, focusing initially on the information the CHWs will use to improve their work and involving them in the process. 24. The capacity of the DHO to effectively use a computerized HIS needs to be reevaluated and any necessary adjustments to the plan/strategy be made. h. Technical and Administrative Support The project has received technical and administrative support from both outside and within Save the Children. Lynn Sybrey of the American College of Nurse Midwives (ACNM) has made three trips (the most recent in early 2003) to Negelle since CS-13 to adapt ACNM’s training curriculum on HB-LSS for this project. ACNM’s support has been very useful and greatly appreciated by the Project, especially by the MCH Unit Head and the TBA trainees. The project has also received technical support from CARE’s CS project in Siaya District, Kenya, which hosted project and partner staff during an experience sharing visit to learn about their work on community management of childhood illness. The Liben/CS project receives technical support from both SC’s headquarters in Westport, Connecticut and from the EFO in Addis Ababa. Eric Starbuck is responsible for backstopping this project from SC/HQ. He has made site visits at key points during the life of the project, CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 43 including the CS-13 Final Evaluation, preparation of the CS-17 proposal and the CS-17 DIP Workshop. He responds to technical questions from the field, forwards relevant materials, and helped orchestrate the Experience Sharing visit to the CARE/Kenya CS project. David Oot and Mary Beth Powers, both from SC’s Office of Health in Westport, led a training on program design and evaluation, and assisted in the development of SC’s country-wide health plan, which included CS-17. The training was attended by the EFO Health Advisor and the SC/Liben MCH Unit Head. SC Staff from both the EFO and Liben were consistent in their praise of the backstopping received from the SC/HQ in terms of its quality, appropriateness and timeliness. The majority of the backstopping support for this project falls on the EFO, specifically Dr. Tedbabe Degefie, Health Advisor. Dr. Tedbabe has held this responsibility since the DIP Workshop in early 2002 and has been to the project site four times for general backstopping visits averaging one week each and three weeks for the DIP preparation. She also meets with the Liben CS staff when they come to Addis Ababa, communicates with the Health Sector Manager by phone and fax, and reviews the monthly, quarterly and annual reports from the project. In late June 2003, Dr. Tedbabe was reassigned to work on the emergency situation that has arisen in Ethiopia and it is not clear who will be undertaking her CS-17 backstopping responsibilities. This is a major concern to the Liben CS staff as they adjust to a new Health Sector Manager and will soon initiate the CCM component. The CS staff in Liben have expressed frustration in the amount of support they have received from the EFO over the life of CS-13 and CS-17, even though they fully recognize that the EFO backstopping staff have to balance their work on CS-17 with several other important commitments and responsibilities. The response to one of the CS-13 Final Evaluation recommendations stated that “the EFO is currently in the process of substantially increasing its capacity to provide technical and administrative support to CS-17, with four senior Addis Ababa￾based staff, including a BCC Specialist, each devoting from 10% to 25% effort to CS-17 support.” Interviews with all the senior project staff in Liben questioned whether this was in fact the case, as it had been several months since anyone from the backstopping staff at the EFO had come to the field, the Health Sector Manager had only received the SOW and donor guidelines for the MTE three days before the field work was scheduled to begin, and no one from the EFO had participated in the MTE due to the need for the EFO staff to respond to the emergency famine that was occurring in Ethiopia at the same time. Further, the Liben/CS staff were consistent in noting the need for more support from the EFO, especially in the upcoming development of training plans, curricula and IEC/BCC materials and strategies; the design and implementation of the CCM component; and the orientation of the new Health Sector Manager. Recommendation: 25. The EFO needs to commit to making regular site visits at least once every quarter and more frequently during key points in the life of the project, such as during the design of program activities, the testing of training curricula, the startup of new project initiatives and all major evaluations and assessments. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 44 4. Other Issues Identified by the Team This Midterm Evaluation for CS-17 was scheduled and implemented a year earlier than usual so there would be 39 rather than 27 months to make any necessary adjustments. Therefore, the MTE Team reviewed those indicators in the Revised Monitoring and Evaluation Matrix to assess progress to date. Not all indicators could be assessed due to the methods required (i.e., KPC survey) or the fact that specific activities had not been initiated at the time of the MTE (i.e., start of Community Case Management). While each of the objectives remains imminently achievable, significant challenges remain – including those that can be impacted by the project and those that cannot. (Please see Annex F, CS-17 Progress at the MTE Per the Revised M&E Matrix Table.) 5. Conclusions and Recommendations NOTE: The conclusions and recommendations have been incorporated into the Action Plan Table, which can be found in Annex G. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 45 Results Highlight Mobilizing Traditional Health Practitioners for Behavior Change The Experience of Save the Children’s CS-17 Project in Ethiopia September 2003 Liben is one of Ethiopia’s most diffusely populated, underserved districts, with an estimated population of 180,000, spread out across 9,900 square kilometers, and served by only nine understaffed, under-equipped health facilities. With limited access to Ministry of Health services it is understandable why traditional treatments provided by untrained practitioners have continued to be a popular recourse, even though many of these practices can be harmful, or at a minimum, can delay effective medical treatment. These practices include female circumcision, tooth extraction to treat diarrhea in small children, scarring, burning, uvulectomies to treat sore throats in children, provision of donkey milk to treat malaria symptoms, and the widespread use of traditional herbal medicines for a myriad of ailments. In response to this challenge, Save the Children’s child survival project in Liben District, recruited, trained, and supported 147 Bridge to Health Teams (BHTs) that directly involve 440 volunteer community health workers in health activities within their communities. Unlike some other more traditional volunteer health worker programs, this project has focused its recruitment on traditional healers, chosen and elected by the people they serve. Each BHT is made up of a wise woman (Chireti) who is frequently a traditional birth attendant (TBA) or midwife, a wise man (Chiresa), and a younger ‘intern’ who is being groomed into a leadership role. As highly regarded traditional leaders, these BHT members have the credibility necessary to begin the process of changing long engrained unsafe behaviors and practices. Each BHT is responsible for mobilizing its community – communicating the project’s health messages, making and following up on referrals to health facilities, notifying the community about health outreach activities (e.g., antenatal, immunization, and reproductive health outreach clinics), organizing community health activities, completing monthly reports on activities and vital events, and distributing family planning supplies when they are available. Every month the BHTs meet with their Health Action Committee (HAC) where they submit written reports, receive supervisory feedback, and plan future activities. The HACs are primarily responsible for supervising and supporting all the BHTs and TBAs in their area through these monthly meetings, and collecting, reviewing, and reporting on the monthly report forms completed by the BHTs and TBAs. The establishment of BHTs has proven to be a very popular strategy in the eyes of the communities and the district’s government health staff, as consistently voiced in focus group discussions throughout the CS-17 midterm evaluation. Mothers and health staff were well aware of who their BHT members were and voiced their support for the continued recruitment and training of replacements as the older members begin to retire. Health staff also noted that the relationship between the health facilities and the communities has improved overall as a result of the BHTs and their work in the communities. Finally, mothers and health staff both adamantly stated their commitment to supporting the continuation of their BHTs after the conclusion of CS￾17 funding. C. Action Plan Please see Annex G for the Action Plan. ANNEXES A. Baseline Information from the DIP B. List of MTE Team Members C. Assessment Methodology D. Persons Interviewed and Contacted E. Questionnaires F. CS-17 Progress at the MTE per the Revised M&E Matrix G. Action Plan H. Visit to Impact Area-Stanley Foster, MD, MPH I. Trip Report-Winifride Mwebsa, MD CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 47 ANNEX A MIDTERM EVALUATION BASELINE INFORMATION FROM THE DIP 1. From the Executive Summary (From pages 9 and 12 of the DIP) Estimated Program Effort and USAID funding by intervention: ß Maternal and newborn care (at 20% of planned intervention-specific effort); ß Pneumonia case management (15%); ß Control of malaria (10%); and ß Control of diarrheal disease (10%), all previously implemented through CS-13; ß Immunization (15%), funded until recently through the DAP; and introduce an ß HIV/AIDS intervention (30%). Program Site Population: Children and Women: Type Number Infants (0-11 months): 6,100 0-59 month old children: 25,800 Women 15-49: 31,700 Estimated Number of Births: 6,400 2. Program Goals and Objectives NOTE: The M&E table that appeared in the DIP was revised in the First Annual Report as requested in the DIP review comments. The revised version is presented here. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 48 CS-17 Revised M&E Matrix According to the DIP Review CS-17 Capacity Building Results, Indicators/Sources, Measurement Methods, Data Collectors, Baseline Values, and End of Project Targets by Intervention Result / IR # Indicator indicator source Method Who Basel. Target Interv. 1 District Health Management Team has met 3 or more times in last year and has used data to plan activities.(1) Minutes MTE and FE DHMT and Eval. teams No By MTE and FE All 2 District HIV/AIDS Council meets regularly, plans, and monitors HIV/AIDS activities in Liben. Final evaluatio n Final eval. team No Yes HIV R-1: Improved Liben District capacity to effectively support community health services and activities. 3a % of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year. (1) Health facility records SPAs and HAs NA 80% All 3b % of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year.(1) Health facility records SPAs and HAs NA 80% All 4 % of CMWs trained in pneumonia case management with no stock-out of cotrimoxazole in the previous month. CMW survey Eval. teams NA 80% ARI 5 % of CMWs trained in malaria or pneumonia case management through CS-17 from whom reports were received in past quarter. Health facility records SPAs and HAs NA 80% ARI Mal. 6 % of BHTs which in the last 6 months have conducted 1 or more community education activity for each CS-17 intervention and turned in 4 or more monthly reports to HACs.(1) BHT forms at health facilities BHTs and HACs NA 80% All R-2: Improved community capacity in Liben to effectively address priority health needs of mothers and children under 5. 7 % of HACs which in the last 6 months have reviewed BHT, TBA, or CMW reports, and have sent reports to health facility.(1) BHT forms at health facilities HACs NA 80% All IR-5: Increased 8 EFO Behavior Change Specialist hired and retained. MTE and FE Eval. teams No Oct. 2002 All CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 49 9 BC strategy for all CS-17 interventions designed and implementation started. MTE and FE Eval. teams No March 2003 All 10 HIV prevention efforts effectively integrated into ongoing community and government activities through CS-17. MTE and FE Eval. teams No Yes HIV Increased SC Addis and Liben capacity in behavior change and integrated HIV programng. 11 Number of HIV-related training courses, workshops, and experience sharing visits in which SC/Liben staff have participated during CS-17. CS-17 records Eval. teams 0 5 HIV Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. CS-17 Results and Indicators Related to Use of Health Services and Health Practices Measurement Methods, Data Collectors, Baseline Values, and End of Project Targets Result / IR # Indicator indicator source Method Who Basel. Target Interv. 12 Total rate of treatment for pneumonia in <5s by CMWs in all PAs with CMWs trained in PCM (number of treatments per <5 per year). CMW Records CMWs NA 0.2 1 ARI 13 % of respondents reporting condom use last time they had sex with non-regular partner.2 FHI survey FHI surveyors 4% 30% HIV R-3: Increased use of key health services and improved MCH practices at household level in Liben District. 14 % of respondents reporting condom use every time they had sex with any non￾regular partner over past 12 months.3 FHI survey FHI surveyors Data from FHI soon 20% HIV 1 The actual incidence of WHO algorithm positive pneumonia is very difficult to measure accurately, and is likely to vary between sites. The Global Burden of Disease and Injury Series (Murray CJL, Lopez AD. Volume II, Global Health Statistics, Harvard University Press, 1996, Table 105) estimates an average incidence of "lower respiratory infection" of 0.45 episodes per infant/child under five per year in developing countries. The actual incidence of algorithm positive pneumonia in children in Ethiopia is unknown. However, SC believes that an effective community-based case management program in rural Liben District, a very high mortality setting, should achieve rates of treatment of at least 0.2. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 50 15 % of births attended by trained TBA or health professional.1,2 KPC KPC surveyors 36% 50% MNC 16 % of all mothers of children <2 receiving TT2+ before last child’s birth (card).2,3 KPC KPC surveyors 21% 50% EPI 17 % of pregnant women receiving TT2+. DHO HF/DHO staff 26% 55% EPI 18 % of all 12-23 month olds who received measles immunization (by card only).2,3 KPC KPC surveyors 32% 60% EPI 19 % of infants who received measles immunization. DHO HF/DHO staff 43% 70% EPI 20 25% of all 12-23 month olds fully immunized (by card).2,(3) KPC KPC surveyors 19% 40% EPI 21 25% of infants fully immunized. DHO HF/DHO 34% 60% EPI 22 % of children <2 with diarrhea in the past 2 weeks receiving more fluids than usual and same or more food than usual during illness.(3) KPC KPC surveyors 8% 50% CDD (#33: New objective added in First Annual Report, Sep. 2002, in response to DIP review.) 33 % of mothers with children <24 months who report washing own hands with soap or ash before food prep., before feeding children, after defecation, and after attending child who defecated. 3 KPC KPC surveyors NA 25% CDD Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. 2 “Number of male/female respondents who used a condom the last time they had sex with a non￾regular (i.e. non-spousal, non-cohabiting and non-commercial) partner, over number of male/female respondents who have had sex with at least one non-regular partner in the past 12 months.” 3 Number of male/female respondents who used a condom every time they had sex with any non￾regular (i.e. on-spousal, non-cohabiting and non-commercial) partner over the past 12 months, over number of male/female respondents who have had sex with at least one non-spousal, non-cohabiting and non-commercial partner in the past 12 months.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 51 CS-17 Results and Indicators Related to Uptake, Availability, Quality, and Knowledge Measurement Methods, Data Collectors, Baseline Values, and End of Project Targets by Intervention Result / IR # Indicator indicator source Method Who Basel. Target Interv. R-4: Uptake/ Sustainability: Adoption of CS-17 approach by MOH or by other organization. 2 3 MOH or other PVO/NGO in other district of Ethiopia has written plans for implementation of CS-17 approach to C-IMCI, MN/LSS, or BHTs. Reports of MOH or other orgs. SC EFO staff No Yes All IR-4: Dissemination of feasibility and results of implementing innovative CS￾17 approaches. 2 4 Feasibility and results of implementing CB￾ARI/Mal. case management, MN/LSS, and/or BHTs, through CS￾17, presented at conference(s), in publication, through media, and/or site visit. CS-17 reports and final eval. Final eval. team NA Yes ARI MNC IR-1: Increased availability of select MCH services in Liben. 2 5 % of rural PAs which have an MOH facility or CMW(s) trained through CS-17 in ARI or malaria case management. CS-17 Records CS-17 staff 19% 100% ARI Mal. 2 6 % of rural PAs with TBAs trained in HB-LSS.(1) “ CS-17 staff 36% 100% MNC IR-2: Documented quality of select community MCH services in Liben District. 2 7 % of children under five assessed for pneumonia for which CMW reported completing all PCM steps correctly. CMW report to superv. SPAs HAs NA 80% ARI 2 8 % of mothers/newborns with complications for which TBAs reported completing all HB-LSS steps correctly. HB-LSS tracking form SPAs HAs NA 50% MNC CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 52 IR-3: Increased maternal knowledge in Liben District of selected MCH issues. 2 9 % of mothers reporting either fast breathing or difficult breathing as a sign of child illness needing treatment. KPC KPC surve yors 21% 65% ARI 30 % of mothers who report knowledge of at least 2 maternal danger signs4 during the postpartum period. (1),3 KPC KPC survey ors NA 50% MNC 31 % of respondents who identify consistent condom use, mutually monogamy, and abstaining from sex, as methods of reducing risk of HIV.5 FHI survey FHI survey ors 10% 50% HIV 32 % of respondents who identify 2 or more signs/symptoms of STIs. FHI survey FHI survey ors Data from FHI soon 25% incr. HIV Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. 3. Program Location (From pages 17-23 of the DIP) Liben District, Borana Zone The CS-17 site covers all of Liben District, the largest of the twelve districts of Borana Zone of Region 4, in southern Ethiopia. (Not to be confused with Liben Zone of Somali Region, just to the east. Please see maps in Annex 7.) The district center (and location of the DHO and SC’s office) is approximately ten to twelve hours driving time from Addis Ababa. Borana Zone has an estimated current population of 1.7 million people,1 residing in an area of 91,200 square kilometers. Liben District covers 9,900 sq. kilometers. Semi-arid climatic 4 At least 2 of the following 3 signs: fever, excessive bleeding, smelly vaginal discharge (KPC 2000+ postpartum care module). 5 “Number of male/female respondents able to identify consistent condom use, mutually monogamy between HIV negative partners, and abstaining from sex as methods of reducing the risk of contracting HIV, in response to prompted questions over total number of male/female respondents surveyed.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 53 conditions are harsh with low, unreliable, and unevenly distributed rainfall (500-700 mm per year), and very limited amounts of surface water. The long rainy season occurs between March and May, with a short rainy season occurring between September and November. Early warning system data collected by SC and its partners indicate that approximately 65% of the population in Borana Zone can be categorized as poor or destitute, and lack the resources to meet their annual food requirements. The DHO estimate for the total population of Liben District in 2002 is 138,310. The DHO also estimates that infants comprise 4.43% of the total population of the district, children under five 18.68%, women between the ages of 15 and 49 years 22.9%, pregnant women 5%, and that the crude birth rate is 46.4 annual live births per 1,000 total population. This would mean that there are approximately 6,000 (6,127) infants, 26,000 (25,836) children under five, 32,000 (31,673) women of childbearing age, and 6,400 (6,418) annual live births in Liben District. The DHO’s estimate of the total population is very consistent with SC’s Liben population spreadsheet estimate for 2002 (of 137,939, based on the 1994 census, "de jure" population projections, and 4.11% urban and 2.23% rural annual growth, as used in 1994 census projections).1 There is considerable controversy among organizations working in Liben District about which population estimates are most likely to be the most accurate.1 For CS-17, SC plans to use the same estimates as those of the DHO, SC’s principle partner in the project.1 There are two towns in Liben District. Negelle, with a population of 33,120, is the administrative center of both Liben District and Borana Zone. The smaller town of Harakelo has a population of 1,448. The total rural population of the district is 103,370 (75% of the district). The rural population is divided into 38 Pastoralist Associations (PAs, or kebeles in Amharic), that form the administrative divisions of the district. The people of Liben District are mainly Afaan Oromo speaking ethnic Borana, Arsi, and Guji, with small pockets of Somali-speaking people, some of them returnees. The majority of Borana and Arsi communities are settled traditionally in extended family encampments called ollas, consisting of 20-50 families. Guji communities tend to be more dispersed; while Somali communities vary from small semi-nomadic encampments to densely populated peri￾urban pockets of returnee families. Each olla selects an abba olla (“father of the encampment”) from among the male heads of household as its leader. Ollas are organized into traditional units called dheda, groups of ollas that share grazing areas; and several dhedas comprise a madda, literally “people who share the same water point.” Maddas correspond, roughly, to the 38 PAs or kebeles mentioned above, and are the connection between the administrative units designated by the GOE, the Pastoralist Associations, and traditional pastoralist family and community units. The inhabitants of Liben District and the rest of the Borana Plateau rely primarily on a pastoralist economy, though people are increasingly, where possible, cultivating small plots of maize, wheat and beans. Liben pastoralists and agro-pastoralists are essentially cattle keepers with goats and sheep and a few camels. The staple food of the pastoralist society is milk (mostly cow, but also camel and goat). During the rainy season, diets can consist of up to approximately 80% milk and butter, and 20% grains, vegetables, and meat, with milk consumption increasing with milk production. Milk production is dependent on livestock health, which in turn depends on water, pasture, and availability of veterinary services. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 54 When milk production decreases, more livestock are sold to purchase grain, and the proportion of cereals in the diet increases. Pastoralism should not be confused with nomadism. Ollas, or encampments, are occupied by at least part of the family year-around, except in times of great food or water shortage. Many families have maintained the same encampments as a base for generations. The family and herd are divided into mobile and stationary parts. The mobile part, or forra, is composed of young men and older boys who travel long distances with the strong male cattle in search of pasture and water. When water and pasture availability permits, they return to the olla. The stationary part of the family, or warra, consists of women and children who take care of the milking cows, weak or sick cows and calves at the olla. The warra may move short distances for forage and water but return to the olla each night. The community elders, who are political leaders and managers of resources, also remain close to the olla. The Boran and Guji ethnic groups are organized into generation sets, which succeed every eight years in assuming political power. This system influences all their social, cultural, and economic institutions. It is also through Gada law making assemblies that attempts are made to maintain an ecologically balanced relationship with the environment. A law-making assembly – the Gumi Gayo – is held every eight years, the last being in 1996. The traditional systems of the Borana are changing rapidly. Today 51% of households can be considered poor; these households control about 10% of the cattle herd. Around 18% may be considered wealthy and control about 65% of the cattle herd. The human population is increasing at a net rate of 2.5% per year, with a 50% increase in population possible within 14 years.1 Limited land availability is restricting cattle numbers, and large numbers of cattle die during droughts. Liben is prone to both natural and manmade emergencies. Cyclical drought and armed conflict in localized pockets pose regular threats to the pastoralist and agro-pastoralist populations. Increasing human and livestock populations and the fragile ecology of the area makes the target population that much more vulnerable to environmental shocks as coping mechanisms become ineffective against both natural and man-made calamities. Water scarcity for both human and animal consumption is one of the greatest problems facing pastoralist and agro-pastoralist communities. Throughout Liben, nearly all rural households use unprotected water sources for human consumption, generally sharing open ponds, shallow wells, and river water sources with their animals. The problem is exacerbated in the dry season when many water sources dry up and people have to travel long distances to obtain water. It is estimated by GTZ1 that some 40% of cattle were lost during the last drought (1998- 2001) in Borana Zone. This amounts to a loss of some 600,000 to 700,000 cattle, the market value of which is in the order of 840 million Birr (US$ 100 million). During the same period 43,000 metric tones of ‘free’ food worth US$ 10.75 million was imported into the area. The net effect of these livestock losses is rapid and serious loss of food security and livelihood assets at times when they are most needed, and a perceived dependence on external support. Mothers surveyed for the 2001 KPC stated their religion as Muslim (59%), waaqeefattaa or traditional followers of one God (13%), or Christian (25%). Christians are comprised of Orthodox, Roman Catholics, and Protestants. Aside from religion, rural cultural traditions, CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 55 socio-economic arrangements, and customary laws reinforce patrilineality and the subordinate position of women. Marriage is universal; and polygyny widely practiced, with men taking up to four (and sometimes more) wives. Age of marriage for girls is usually at 15 to 17 years, but can be as young as 12 to 14 in some communities in the district. Age of first marriage for boys is about 18 to 20. The average household size in Liben District is 6.8, as determined by the DAP baseline survey. A majority (62%) of mothers surveyed in the KPC conducted in August 2001 reported working away from home.1 Childcare options included: taking the child with her (42%) or leaving the child in the care of older children (24%), relatives (15%), neighbors (8%), or with a husband or partner (9%). Health Status and Health Services in Liben District The most valid information on the nutrition status of children in Liben District is from SC’s assessments of approximately 500 6-59 month-old children at 18 sentinel sites during the month of August in each of three recent years: 1997, 1999, and 2000. In Liben District, August follows the lean period of comparative food scarcity.1 This data from Liben District is compared in the table below to that for 0-59 month-old children in Oromiya Region and in Ethiopia as a whole, from Ethiopia’s Demographic and Health Survey, conducted from February through May, 2000. Childhood Nutrition Status in Liben District, Oromiya Region, and Ethiopia, 1997 - 2000 Nutrition Status Indicator (Z score cut-off value) Liben, SC Aug. ‘97 6-59 mos. Liben, SC Aug. ‘99 6-59 mos. Liben, SC Aug. ‘00 6-59 mos. Oromiya, DHS Feb.-May ‘00 0-59 mos. Ethiopia, DHS Feb.-May ‘00 0-59 mos. Underweight (< -2Z W/A) 41% 28% 34% 42% 47% Wasted (< -2Z W/H) 7% 4% 11% 10% 10% Stunted (< -2Z H/A) 43% NA 42% 47% 52% Ethiopia is classified by WHO and UNICEF as a country with clinical vitamin A deficiency,1 with a national xerophthalmia survey in the country’s four agro-ecological zones finding the highest rates of Bitot’s spots in pastoralist areas.1,1 Iodine deficiency disorders are not believed to be prevalent in Borana Zone.1 Other valid estimates of measures of health status are not available for either Liben District or Borana Zone. However, socio-economic conditions in Liben, the presence of falciparum malaria, and information on nutrition status of 6-59 month-old children (above), suggests that the levels and causes of under-five mortality in the district are likely similar to those in Ethiopia as a whole. In Liben District, low coverage by sparse maternal health services, rudimentary family planning services, unhealthful traditional practices, geographic and cultural barriers to care, and poverty contribute to poor maternal health. SC and the DHO believe that the distribution of causes of maternal death in Liben District likely reflects the distribution of causes in developing countries in general: Hemorrhage, unsafe abortion, hypertension, obstructed labor, sepsis, and indirect causes, including anemia and malaria.1 Concrete information on HIV/AIDS prevalence is sorely lacking in all of the Southern Tier of the country, including Liben District. However, during recent months, the blood of 48% CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 56 of apparently healthy donors at Negelle Hospital in the district center has been screened HIV￾positive.1 While this data may not be reflective of HIV prevalence in the general population and prevalence in other groups of Liben District is not known, prevalence is likely to be high, given that there is an active commercial sex industry and a military base located in the district, and a mining industry in the adjoining district. This area has historically been one of the most underserved areas of Ethiopia in terms of health (and other) infrastructure and services. MOH health facilities in Liben District include the 113 bed Zonal Hospital in Negelle, and a total of nine functioning clinics (health stations/posts) outside of Negelle. Hospital staff include four general practitioners, one of whom serves as both Hospital Director and District Health Officer; and 14 Senior and Junior Nurses. The nine facilities outside of Negelle are each staffed by one Health Assistant, while four of these facilities also have a Community Midwife. Outpatient MCH services, including case management of childhood ARI, diarrhea, and malaria, and antenatal, delivery, and family planning services, which are provided from each of these facilities on a daily basis. Childhood and maternal immunization services are provided on at least a monthly basis in all but four of district’s Peasant Associations through these facilities and through outreach services, but immunization coverage remains low. (Please see KPC results in Section I.E). A needs assessment carried out in a number of facilities in Liben, in preparation for the LSS training showed critical gaps in staff knowledge and practice in maternal and newborn care, including prevention and management of danger signs during gestation, perinatal, and postnatal periods. (Please see further discussion of MNC services in Section III.F.) There are no active STI/HIV prevention activities in Liben District, except for several AIDS clubs. There used to be an STI clinic staffed by a medical doctor at the district hospital, but this was closed due to shortage of physicians. In addition to managing STI patients, the STI clinic ran a free certification service for sex workers, who were examined and given certificates pronouncing them free of STIs other than HIV. Sex workers found to be infected were treated free of charge, and only certified after completing the treatment. The District Health Office ran this program in collaboration with the District Women’s Affairs department, bar owners, and the municipal administration. Currently, STI services have been integrated into the general outpatient department services and clients have to pay for the service. The sex worker certification has been discontinued. STI services are available at the hospital and clinics. Diagnosis is based on the STI syndromic management approach, but not all providers have been trained in this approach. There are no voluntary counseling and testing (VCT) or other HIV/AIDS counseling services in the district. There are 13 registered rural drug vendors in Liben Woreda. RDVs are private dispensers of medications, mostly located in towns and/or near MOH health facilities. They are often trained Health Assistants who have moved into the private sector, though registered RDVs sometimes staff their shops with other, less trained, staff. Reports from communities and MOH staff indicate that there are not many unregistered RDVs or informal drug sellers, so most rural communities do not have access to a RDV. MOH guidelines require that RDVs refer all children with suspected pneumonia and malaria to an MOH facility, although it is recognized that RDVs sometimes give primary health care and medications. KPC data suggest that few children with diarrhea, pneumonia, or malaria are taken to a RDV. Routine supervision of RDVs is the responsibility of the DHO, though supervisory visits have been irregular. Based on their experience in the district, the DHO believes that RDVs in Liben are CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 57 not playing a major role in providing key maternal and child health services and doubts the benefits of training RDVs.1 Limited geographic access to health facilities and services is a fundamental constraint in Liben District. The DHO estimates, based on the geographic distribution of people and health providers, that only approximately 40% of the total population of Liben District lives within a ten kilometer (6.2 mile) radius of an MOH or private health provider. The July 1997 Health Status Information study conducted in parts of Liben and neighboring Arero District by the Italian NGO COOPI found that 72% of people in these two areas walk more than three hours to reach health services and that only 32% live within ten kilometers of a health facility. Male traditional healers in Liben, cheresas, or “wise-men,” include herbalists, bone setters, religious practitioners, and spiritual healers. These individuals are respected in the community as credible sources of information about health and healing. Moreover, these men act as “gatekeepers” for care-seeking outside of the olla or kebele. Cherites, or “wise women” traditionally provide birth assistance in the district. In most communities, one can find a cherite in every olla or group of ollas. Some cherites also practice other healing arts, such as herbalism, massage, bone setting, or female circumcision (e.g., infibulation). Their advice is often sought for children’s health problems, especially diarrhea and fever. Most are women in their mid-forties or older; all are respected by their communities. Although some cherites charge for attending a delivery, most accept payment in-kind and whatever is offered. The average number of births assisted on a monthly basis varies greatly from community to community. While PVOs and NGOs have been encouraged in the past few years by the Government of Ethiopia to work in this area, there are only two international NGOs operating in the district: Cooperazione Internazionale (COOPI), an Italian NGO; and Save the Children (USA). The German Agency for Technical Cooperation (GTZ) is also now operational in the Zone and is engaged in agro-pastoral and health activities. The only national NGO having a visible presence in the district is the Ethiopian Red Cross (ERC). Funded by the German and Japanese Red Cross societies, the ERC is making an ambulance available to Negelle Hospital for emergency, long distance transport for critical cases requiring more sophisticated medical treatment at a referral hospital located 270 kilometers away. SC has been implementing two complementary programs throughout Liben District since 1997, the CS-13 project through September 2001, followed by CS-17; and a USAID/DCHA/FFP-funded DAP. Maternal and newborn care, malaria, CDD, and ARI interventions were supported through CS-13, while the DAP supported nutrition, immunization until the start of CS-17, and family planning activities until the start of support from NGO Networks for Health in 2001. The DAP aims to “make a sustainable improvement in availability of, access to, and utilization of food for approximately 17,500 pastoralist households in Liben District.”1 Availability of and access to food are addressed through DAP activities to improve livestock management, while utilization is addressed through the DAP’s “human health” interventions, implemented through, and helping to build the capacity of, the same SC, MOH, and community-level structures supported through CS￾13 and CS-17, including the Health Action Committees (HACs), and Bridge-to-Health Teams (BHTs). Through the DAP, HAC and BHT members have been trained and supported to educate fellow community members in nutrition and immunization; MOH CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 58 immunization activities have been supported, including EPI+ campaigns, routine outreach activities, and the cold chain. DAP-supported activities have included food supplementation for children, based on the results of weight-for-height screening, and for women during the last trimester of pregnancy and first six months postpartum. These food supplementation activities are integrated with nutrition education/food demonstration, EPI, and other preventive MCH outreach activities conducted by SC and MOH staff. The DAP has contributed to the CS-13 and CS-17 goals of improving access to MCH services through constructing and equipping two new health posts in Liben District, both now fully functional MOH facilities. 4. Program Design (From pages 28-9 of the DIP) SC and the MOH Liben District Health Office will continue implementation through CS-17 of all four CS-13 interventions: ARI (at 15% of planned intervention-specific CS-17 effort), Malaria (10%), CDD (10%), and Maternal and Newborn Care (20%); and continue important support to the DHO in EPI (15%), previously funded through the DAP. CS-17 will devote 30% of intervention effort to introducing an HIV/AIDS intervention, in order to build SC and DHO capacity in Liben to begin addressing the district’s HIV epidemic. These CS-17 interventions will be implemented through the following major strategies: · Joint DHO/SC design, implementation, and evaluation of approaches to maternal and child health in Liben that inform development of strategies to address the needs of pastoralist populations in other districts of Borana Zone and Ethiopia. · Introduction and evaluation of community-based case management of childhood illness, to improve access to and use of these services in Liben District, and to inform the nascent development of Community-IMCI in Ethiopia. · Building capacity of SC, the DHO, and the District HIV/AIDS Council, to provide leadership, coordination, and technical advice for integration of effective HIV prevention, care and support, and mitigation efforts into ongoing community and government activities in Liben District. · Continued mobilization of community leaders and traditional practitioners through Bridge-to-Health Teams and Health Action Committees, to support selected MCH services, and to conduct focused education to improve key emphasis behaviors at the household level. CS-17 builds on the central strategy of CS-13, community mobilization and health education through “Bridge-to-Health Teams” (BHTs). BHTs were introduced and discussed with local leaders by SC’s community mobilizer in two to five visits to each of the “Peasant Associations” (PAs) in Liben District between April 1998 and March 1999. Each of 150 communities elected a three-member BHT composed of a chereti (wise woman/TBA), a cheresa (wise man/male traditional healer), and a young traditional apprentice. At least two thirds of the 450 BHT members are influential, respected traditional healers or birth attendants, and most traditional healers in the district are BHT members. SC and District Health Office (DHO) partners trained BHTs to provide health education for home treatment of watery diarrhea; recognition and care-seeking for pneumonia, malaria, and pregnancy￾related danger signs; and to promote use of antenatal care, family planning, and immunization services at MOH health facilities and at joint SC/MOH outreach sites. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 59 Together with the staff from the nearest health facility, Health Action Committees (HACs) in every PA each support four to six BHTs, and review and respond to health information from BHTs and TBAs. Service Area Teams, composed of MOH health facility staff and SC Senior Program Assistants temporarily posted to facilities, train and support BHTs and HACs. 5. Partnerships (From pages 30-1 of the DIP) Relationship with Other Health-Related Activities and Roles of Major CS-17 Partners CS-17 is closely integrated with MOH health activities in Liben District, building on the close relationship developed through CS-13, which concluded with the full-time participation of the District Health Officer in the CS-13 final evaluation. The DHO has played an active role in CS-17 design and planning workshops over the last 1.5 years, and joint SC/DHO planning and management of MCH activities through a DHMT will be a focus of district￾level collaboration through CS-17. SC is continuing its key role in supporting MOH immunization activities throughout the district. BHTs and HACs are continuing to promote the effective use of MOH MCH services among members of their communities. SC’s Senior Program Assistants (SPAs) work out of MOH health facilities with MOH health staff, and work with MOH Health Assistants to prepare a joint health facility/ CS-17 monthly report using the standard MOH format with additional space added for reporting of community￾level activities. Save the Children will: play a leading role in managing CS-17, an important role on the District Health Management Team, support the District HIV/AIDS Council; lead formative HIV research, other baseline and final assessments, and development/revision of training and BC curricula and materials; with MOH partners, train and support (on at least a quarterly basis) health facility staff, and through Service Area Teams train and support HACs, BHTs, and TBAs; sponsor the training of three Community Midwives at Negelle Nursing School; monitor HIS performance and findings, health worker performance, and availability of essential supplies/equipment with MOH partners, and; conduct immunization/ANC/ DAP outreach sessions together with MOH facility staff. The District Health Office will: co-manage CS-17 with SC; chair the DHMT; participate in baseline and final assessments, and development/ revision of training and BC curricula and materials; with SC, train and support (on at least a quarterly basis) health facility staff, and through Service Area Teams, train and support HACs (including quarterly meetings with HACs), BHTs, and TBAs; provide facility-based MCH services; with SC, monitor HIS performance and findings, and health worker performance; ensure availability of essential supplies/equipment at facilities and for outreach activities, and; conduct immunization/ANC/ DAP outreach sessions together with SC. (Please see jointly developed agreement between SC and the DHO in DIP Annex 3.) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 60 6. Health Information System Information Management (From page 66 of the DIP) Information is exchanged between Office of Health staff based in Westport and Washington, D.C. and Ethiopia Field Office staff mainly using e-mail. The EFO presently has a single e￾mail address, requiring EFO administrative staff to print out and pass along hard copies of messages to the EFO Health Advisor and others staff. The EFO is on the Health PLG listserv. According to the EFO, the Health Advisor should have an individual e-mail account by April 2002. She will be on the Health PLG listserv as soon as she has an individual e￾mail account. The EFO has access to the internet, but individual technical staff members currently do not have access to the internet at their own work stations. This feature will be added for senior staff, including the Health Advisor, in 2002. The EFO Health Advisor communicates with CS-17 staff during her visits to Negelle and during visits by CS-17 staff to Addis, as well as by phone and fax. Communications between the EFO and SC’s office in Negelle remains an important challenge, with frequent breakdown of communication. But SC hopes that this will change with the current renovation of the telecommunication lines, and introduction of e-mail in the near future, which will allow Negelle to dial into the EFO server. All CS-17 staff are fluent in the English language. SC Field Offices exchange information on health programs through meetings in person at the annual meeting of the Health PLG and using the Health PLG listserv. Program Monitoring (From pages 74-6 of the DIP) The CS-17 approach to monitoring and evaluation is participatory in nature, involving local partners. MOH Health Assistants, SC SPAs, and HAC members are members of the Facility Health Management Committee which will meet quarterly to discuss targets and goals for health service indicators (i.e. immunization coverage, ANC services utilization); share progress and experiences, and provide an opportunity for technical support. Except for the community level data collection tally sheets, other reporting formats are those used by the MOH, with community-level information added to the MOH facility-level reporting format. Reports come through the MOH system from health facilities to the DHO, not through a separate or parallel channel. Other than routine monitoring tools, CS-17 will have a mid￾term and final evaluation, and conduct a KPC survey towards the end of the project. At the community level, TBAs and BHTs keep records using tally sheets (Please see Annex 7). HACs collect and review the tally sheets for their PAs. The data collected includes counts of important activities performed and events or cases like numbers of maternal deaths, measles cases and under five deaths (to be added to the BT tally), and treatment for pneumonia and malaria (by CMWs). HACs will meet with TBAs and BHTs monthly to discuss implications of the reports, identify and discuss problems and successes. Then the reports are given to the health facility. At the facility level, data from registers and tally sheets will be combined onto a health facility monthly reporting form. Service Area Team members meet with HACs monthly to review HAC report based on targets set for each PA (e.g. immunization coverage); process data manually for the service area, identify problems and take action together with the HACs, and the send report to the DHO. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 61 TBA and BHT tally sheets were developed by CS-13 and will be used for monitoring at the community level. SC has supported the development of computer software for data entry and analysis at the district level through the CS-13 grant, which will be tested for six months beginning in April 2002. However, until this new system is determined to be useful, emphasis will be on manual data collation and analysis. At the end of each month the HIS team will present its findings to the District Health Management Team. CS-17 will use data from the following sources to track performance: Forms from HACs, BHTs, trained TBAs, Case Management Workers, supervisory checklists, MOH facility￾based records and registers, and midterm and final evaluations. CS-17 will use the same population estimates as the Liben District Health Office. The 1994 census is used to determine population denominators using estimates of annual population growth rates of 4.11% for urban and 2.23% for rural areas. Women between 15 and 49 years of age are estimated to be 22.9% of the total population, infants 4.4%, and children under five 18.7%. The annual number of live births is estimated from the total population and the crude birth rate of 46.4. Data from both the community and the facility is reported on a monthly basis, while evaluations will be done about two years into the program and towards the end of the five￾year program. Trained TBAs, Case Management Workers, and BHT members are the front line data collectors at the community level, while MOH Health Assistants and SC Senior Program Assistants collect and collate data at the health facility level. Each trained TBA covers an average of about 73 households, each BHT member covers an average of about 49 households, while each CMW will cover about 284 rural households.1 They collect data at the time of service delivery, and thus don’t use most of their time for data collection. They also collect data on important events which they hear about, such as cases of measles and maternal deaths, but are not expected to conduct household visits to obtain information. Both MOH Health Assistants and SC SPAs working in the facilities collect data at the facility level, and aggregate community and facility data for reporting to the DHO. Service Area Teams (MOH Health Assistants and SC SPAs) assure quality of data by meeting with the HACs from the area to review and aggregate data and use it for action. At health the facility level, for the purpose of maintaining uniformity of reporting instruments throughout Borana Zone, the MOH monthly report form is used unaltered, while a page summarizing the community level data is used in Liben District by the facilities reporting to the DHO. Service Area Teams, composed of MOH and SC staff at each facility, will provide supervisory support to HACs on a monthly basis, reviewing data collection done by the HACs and actions taken by the community. The Service Area Teams will integrate data from the HACs with data from the health facilities and outreach sites. They will calculate key indicators, identify problems, and take action at their level, and compile a report that incorporates problems identified, actions taken, and request for assistance required from the DHO. The Service Area Team will have quarterly meetings with the Health Facility Management Committee (HFMC), consisting of one or two representatives from all the HACs to review performance and plan for the following quarter. Discussions during these meetings may include illustrations of comparative coverage indicators for the different HACs, using the Ethiopian flag (with green, yellow, and red colors, as in Bolivia where SC CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 62 has found this approach useful). If coverage for a PA falls in the red margin it would illustrate danger, and the team would facilitate investigation of problems and solutions, and provide support to the HAC to bring improvement. Coverage in yellow would be an alert to less than good performance, while green would represent good coverage to be recognized by awards or incentives. At the DHO level, the HIS team, consisting of the Liben DHO Statistics Clerk, MCH Coordinator, and Sanitarian, and SC’s M&E Coordinator, will be responsible for data entry, initial analysis, and identifying problems. SC has supported the development of computer software for data entry and analysis at the district level through the current CS-13 grant, which will be tested for six months beginning in April 2002. SC’s staff and DHO staff will be trained on the use of the data base. However, until this new system is determined to be useful, emphasis will be on manual data collation and analysis. At the end of each month the HIS team will present its findings to the District Health Management Team. The DHMT will carry out monthly and quarterly program reviews using key health service, morbidity, and management data, identify priority problems and develop plans, give feedback to the health facilities including recommendations for action, and send the report to the Borana Zonal Health Department. Data is computerized at SC Negelle and Addis levels. After testing new software, this will be done at the DHO level as well. Quarterly supervisory visits, with observations of sick child management, will provide technical support to monitor and improve health workers performance. The visits will be done using checklist to assess knowledge, skills, practices, and the health facilities supplies, drugs, and equipment. Health workers at the facility levels will also conduct quarterly supervisory visits to community-based providers (e.g., TBAs, CHWs) to assist them in carrying out the counseling of caretakers in early detection of signs and symptoms. CS-17 will use MOH guidelines and training manuals for health facility staff, CARE’s training manual and algorithms will be used to train community health workers on community case management. These tools will be used to improve health workers performance. Supervisory checklists will be developed for supervision of community health workers to be used to promote quality of service. Several approaches will be used to remind health workers of tasks for management of sick children and of key messages. SC, together with the DHO, will develop memory-jogging cards describing case management tasks and key messages to be used by Case Management Workers and BHT members. Supervisory checklists for use by HAs and SPAs when they observe CHWs doing case management or health education will also be used to monitor and improve CHW performance. The EFO will give support to strengthen the M&E skills of SC and MOH staff responsible for data management. SC has supported the development of computer software for data entry and analysis at the district level through the CS-13 grant, which will be tested. Several staff members will be trained to use the database for data management so that data analysis will not depend on one individual. At the DHO level, the HIS team, consisting of the Liben DHO Statistics Clerk, MCH Coordinator, and Sanitarian, and SC’s M&E Coordinator, will be responsible for data entry, initial analysis, and identifying problems. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 63 The CS-13 HIS has enabled BHTs and TBAs to collect vital data at the community level. While the CS-13 non-literate forms introduced at the community level have been useful for monitoring BHT, TBA, and HAC activities and strengthening HF-community links, revisions had to be made to promote community ownership of the system, and community analysis and utilization for concrete action. The new approach at the community level is designed to make the community-level component more than a mere extension of the facility reporting system. It builds on the current community structure and the non-literate reporting instruments to increase the level of community involvement in data analysis and utilization. The system allows HACs or BHTs to detect, investigate, and respond to important events, such as a case of measles, an increase in cases of diarrhea, or a maternal death. This approach is designed to empower communities to take action, allow communities to be heard at higher levels, and allow them to participate in initiatives of the Service Area Team or the DHO. The methods used in both data collection and analysis are simple and do not need a lot skill and are thus sustainable. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 64 ANNEX B MIDTERM EVALUATION TEAM MEMBERS Sr. Senait Bebele Zonal Health Department Acting Head Mr. Adamu Beyene Monitoring and Evaluation Coordinator, SC/Liben Mr. Niftalem Kumera Disaster Prevention and Preparedness Dept. Head Mr. Mohamed Manu HIV/AIDS Unit Head, SC/Liben Mr. Nina Negash Monitoring and Evaluation Officer, CARE/Ethiopia Mr. Garth Osborn External Consultant and Team Leader Mr. Worku Tefera Training Coordinator, SC/Liben Mr. Solomon Tesema Health Sector Manager, SC/Liben Dr. Taye Tolera Medical Director, Negelle Hospital CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 65 ANNEX C MIDTERM EVALUATION ASSESSMENT METHODOLOGY The Midterm Evaluation took place in Liben District July 14-23, 2003, in Addis Ababa on July 24-25 and finally in the United States on August 5. It should be noted that this MTE was implemented one year earlier than usual – less than two years into a five-year project – at the suggestion of the reviewers of the CS-17 proposal. The reason for this was the recognition that waiting until year three to do the MTE might not allow sufficient time to make necessary course corrections. It is the view of the MTE Team that this suggestion was a good one and that while there is general consensus that the project can meet its final targets, there were sufficient issues raised in this MTE that warrant timely and intense attention so success can be achieved. Following is the summary schedule of the MTE: DAY DATE ACTIVITY Friday July 11 Team Leader Arrives in Addis Ababa Sunday July 13 Leave for Negelle; overnight in transit in Awassa Monday July 14 3 PM: Arrive in Negelle PM: Introductory Meeting with SC Project Staff Tuesday July 15 8:30 AM: Planning Meeting with SC Project Staff 4 PM: Orientation of MTE Team (Agenda: Introductions, CS-17 Overview, MTE Purpose, Data Collection Methods, MTE Member Roles, MTE Report, and Review Schedule and Logistics) Wednesday July 16 Develop Assessment Tools/Questionnaires Thursday July 17 9:00: Meet with District Admin Council 10:30: Meet with Zonal Health Committee 2:00: Meet with DHMT Friday July 18 9:00 Meet with District HIV/AIDS Council 2 PM: Train MTE Team in FGDs. Finalize and print questionnaires. Saturday July 19 Sunday July 20 Monday July 21 AM and early PM: FGDs, interviews and document/HIS reviews PM: Report findings to MTE Team Interviews with SC and MOH staff on Sunday and Monday. Tuesday July 22 Develop consensus on main findings and recommendations, prepare Draft Action Plan, and prepare for debriefing in Addis. Wednesday July 23 Leave for Addis; overnight in transit in Awassa Thursday July 24 AM: Arrive in Addis 2:00: Interview Dr. Tedbabe Degefie at the EFO Friday July 25 Debrief Jeanne Kopsell and Dr. Abebe Gebremariam Depart Addis Thursday Aug 7 Interview Eric Starbuck by phone, SC/Westport Sunday Aug 10 First Draft of MTE Report due to SC Sunday Aug 24 Comments on First Draft due back from SC Monday Sept 1 Final Draft due to SC CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 66 The MTE used four methods of data collection: 1. Meetings were held between the MTE Team and various government committees that are involved with the project, including the Zonal Health Committee, the District Administrative Council, the District HIV/AIDS Council and the District Health Management Team. A questionnaire was developed for each meeting, however, each Team member was invited to ask his/her own questions as well. (See Annex E for MTE Questionnaires.) 2. Interviews were held between the Team Leader and the program staff members of both SC/Liben and Negelle Hospital. Two sets of questions were asked of each SC staff member – general questions about administration/human resources and specific questions tied to that person's technical responsibilities. These interviews were held individually and the results that relate to human resources and employment are reported on in a composite fashion so that employees would feel comfortable in responding to sensitive questions. (See Annex D for a list of the individuals interviewed.) 3. Several project related documents, reports, reporting formats, manuals and systems were reviewed by members of the MTE Team. 4. A set of three Focus Group Discussions (FGDs) was held with each of the following: ÿ Mothers of children under the age of five ÿ Members of Bridge to Health Teams ÿ Traditional Birth Attendants trained by the project ÿ Members of Health Action Committees ÿ Members of Service Area Teams To do the FGDs, the MTE Team received a brief training on FGDs by the Team Leader and then was split into two small groups, each with a moderator and note takers. The groups then reviewed, edited and translated the FGD questionnaires. A ‘purposeful’ sample was used in selecting which communities to target, selected based on urban-rural mix, geographic spread and ethnic background. Community Urban/Rural Location Ethnicity Negelle Urban Central Mixed Genalle Rural Northeast Arsi Balambal Rural West Guji At the end of every day of FGDs the Team met to read their notes, which recorded and then summarized by the Team Leader. At the conclusion of the data collection process in Liben, the entire Team met for a full day to identify the major findings, come to consensus on the main recommendations and produce an initial draft action plan for next steps. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 67 ANNEX D MIDTERM EVALUATION PERSONS INTERVIEWED AND CONTACTED Dr. Winnie Mwebesa, FP/RH Advisor, SC/HQ Dr. Abebe Gebremariam, SC/EFO Jeanne Kopsell, SNL Field Program Specialist, SC/HQ Dr. Eric Starbuck, Child Survival Specialist, SC/HQ Dr. Tedbabe Degefie, Health Advisor, SC/EFO Alemayehu Boka, the SC/Liben Impact Area Manager Solomon Tesema, SC/Liben Health Sector Manager and Member of the DHMT Worku Tefera, SC/Liben Training Coordinator and Member of the DHMT Sister Degefch H. Yesus, MCH Unit Head Mohammed Mamu, HIV/AIDS Unit Head and Member of the District HIV/AIDS Council Gebre Tola, EPI Unit Head Adamo Beyene, HIS/M&E Coordinator, Member, District Health Management Team Senior Program Assistants, SC/Liben District Chuluka Dullo Keneni Mekonnen Adunya G. Selassie Mestawot Negash Kote Ibrahim Zenebe G. Tsadik Selas Siminion, Secretary, District HIV/AIDS Council Lemma Legesse, District Health Officer and Member of the DHMT Dr. Taye Tolera, Medical Director/Negelle Hospital and Member of the DHMT Sr. Lelise Tadesse, MCH Nurse/ Negelle Hospital and Member of the DHMT Guji Zonal Health Office St. Senaiet Bekele, Acting Head, Guji Zonal Health Office Fekede Bayisa, Head, Malaria and Other Vector-Borne Disease Control Antenalew Asmare, Pharmacist, Guji Zonal Health Office In addition, a total of 32 mothers of children under the age of five, 27 HAC members, 24 BHT members and 12 TTBAs were interviewed in Focus Group Discussions. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 68 ANNEX E QUESTIONNAIRES QUESTIONNAIRE DISTRICT ADMINISTRATIVE COUNCIL SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for your willingness to participate today. Now for the first question….” 1. When was the District Administration Council formed? 2. Briefly, what are the overall responsibilities of the Council related to health? 3. How have you been involved with the Save the Children Liben Impact Area Project? 4. What are the future health-related goals of the Council to be achieved in collaboration with this Project? 5. What is the relationship between the Council and this Project? 6. What have been the most important lessons learned from this Project? 7. In what ways can this Project be strengthened? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 69 QUESTIONNAIRE ZONAL HEALTH OFFICE SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for your willingness to participate today. Now for the first question….” 1. When was the Zonal Health Office formed? 2. Briefly, what are the overall responsibilities of the Zonal Health Office? 3. What is the relationship between the Zonal Health Office and the Save the Children/Liben Impact Area Project 4. What are the future goals of the Zonal Health Office that you expect to achieve in collaboration with this Project? 5. What kinds of support has the Zonal Health Office received from this Project? 6. What have been the most important lessons learned in this Project? 7. In what ways can this Project be strengthened? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 70 QUESTIONNAIRE DISTRICT HEALTH MANAGEMENT TEAM SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for your willingness to participate today. Now for the first question….” 1. When was the DHMT formed? 2. How many members does it have? 3. How was membership determined and are there other positions that should be added to the DHMT? 4. How frequently does the DHMT meet? 5. How many times has the DHMT met in the past year? (Objective 1) 6. Are minutes kept for every meeting? If yes, who is responsible for recording them? Are minutes from the previous meeting reviewed at the beginning of the subsequent meeting? Does everyone get a copy? 7. Does the DHMT have a mission or goal statement? If yes, what does it say? 8. What are the responsibilities of the DHMT and its members from the various organizations? Is this written down? If yes, how often is it referred to? 9. What has the DHMT achieved since the start of CS-17? 10. Has the DHMT received any training from the CS-17 project? If yes, please describe. (DIP Workplan Page 81 point 4.5) 11. What are the primary challenges the project faces in its next two years? 12. What does the DHMT plan to achieve in the next two years and is this written down? CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 71 13. Is there a need for the DHMT to continue after Save the Children’s involvement is phased out in Negelle? If yes, what will its role be? What needs to be done to ensure its continuation? 14. Has each member of the DHMT been given a copy of the CS-17 Project Objectives? 15. From what sources does the DHMT receive data? How has this data been used in the past year to plan and make decisions? Please provide examples. (Objective 1) 16. What have been the most important lessons learned from the CS-17 Project? 17. What have been the most important achievements of the CS-17 Project? 18. In what ways can the CS-17 Project be strengthened? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 72 QUESTIONNAIRE DISTRICT HIV/AIDS COUNCIL SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for your willingness to participate today. Now for the first question….” 1. When was the District HIV/AIDS Council formed? 2. How many members does it have? 3. How was membership determined? 4. How frequently does the Council meet? 5. How many times did the Council meet in the past year? (Objective 2) 6. What are the Council’s responsibilities related to planning and monitoring HIV/AIDS activities in Liben? How are these responsibilities carried out? Is this written down? If yes, how often is it referred to? (Objective 2) 7. What has the Council achieved since the start of CS-17? 8. What kinds of support and feedback has the council received from the CS-17 project? 9. Has the Council received any training from the CS-17 project? If yes, please describe. (DIP Workplan Page 81 point 4.1.1) 10. How have HIV-prevention activities been integrated into ongoing community and government activities? (Objective 10) 11. What are the primary challenges the project faces with regard to HIV/AIDS in the next two years? 12. What does the Council plan to achieve in the next two years and is this written down? 13. Is there a need for the Council to continue after Save the Children’s involvement is phased out in Negelle? If yes, what will its role be? What needs to be done to ensure its continuation? CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 73 14. From what sources does the Council receive data? How has this data been used in the past year to make decisions and plan activities? Please provide examples. (Objective 2) 15. What have been the most important lessons learned from the CS-17 Project related to HIV/AIDS prevention? 16. What have been the most important achievements of the CS-17 Project related to HIV/AIDS prevention? 17. In what ways can the CS-17 Project be strengthened related to HIV/AIDS prevention? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 74 QUESTIONNAIRE MOTHERS OF CHILDREN UNDER THE AGE OF FIVE SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Number of Group Members INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for participating today. Now for the first question….” From whom do you learn about health? What does your Bridge to Health Team do in your community? What do TTBAs do in your community? What does your Health Action Committee do in your village? What are some things families can do to make childbirth and the time shortly after birth safer for mothers? (Indicators 15, 16 and 30) Where do most women give birth and why? Who do most women seek help from when they are going to have a baby and why? (Indicator 15) What are some reasons mothers might not get their children immunized? (Indicators 18 and 20) What should mothers give to a child with diarrhea? Why? (Indicator 22) What signs of illness in a small child would make you seek help and where would you go first? (Indicator 29) Where do you find medicines? Is HIV/AIDS a major problem in your community and if so, why? In what ways can we make this maternal and child health care project stronger? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 75 QUESTIONNAIRE BRIDGE TO HEALTH TEAMS (BHT) AND TRADITIONAL BIRTH ATTENDANTS (TBA) SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” First I will be asking some questions of the entire Bridge to Health Teams. Once that is finished, I would like to meet with those of you who are Traditional Birth Attendants to ask some additional questions specific to your work.” “Thank you for participating today. Now for the first question….” 1. When was your Bridge to Health Team formed? 2. What is the criteria for membership? 3. What are your responsibilities as BHT members? 4. What has your BHT achieved in the past two years? 5. Has your BHT received training in the past two years? If yes, how useful has it been in your work? Are there other training topics that should be included? 6. What other kinds of support has your BHT received from: a. Save the Children/SPAs b. Health Facility Staff c. Your Health Action Committee d. Your community 7. What kind of feedback has your BHT received from: a. SPAs b. Health Facility Staff c. Your Health Action Committee CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 76 8. How has your community responded to your services? 9. What have been the most important lessons learned from your experience working on a BHT? 10. In what ways can this Project be strengthened so that your work can continue in the future? CLOSE: “That is the last of the questions that we have for the BHTs today. Is there anything that you would like to add?” “Now I would like to have a little time to ask some questions of those of you who are TBAs, so the others are free to leave. Thank you for your help.” THE FOLLOWING QUESTIONS ARE ONLY TO BE ASKED OF THE TBAs. What are the main challenges you face in assisting women with deliveries? We have heard that fewer young women are becoming TBAs. Do you think this is true and if so, why is it happening? How did you become a TBA? How will someone replace you? Do you think families would help to pay for gloves and other delivery supplies? Has your relationship with the health workers at your local health facility changed since the beginning of this project? Please describe. What can be done to help women get to the hospital quickly when obstetric emergencies arise? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 77 QUESTIONNAIRE HEALTH ACTION COMMITTEES (HAC) SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for participating today. Now for the first question….” 1. When was your Health Action Committee formed? 2. How many members does it have? 3. How was membership determined? 4. How frequently does your HAC meet? 5. Are minutes kept for every meeting? If yes, who is responsible for recording them? Are minutes from the previous meeting reviewed at the beginning of the subsequent meeting? Does everyone get a copy? 6. What are your responsibilities as HAC members? 7. What has your HAC achieved in the past two years? 8. Has your HAC received any training from the Project in the past two years? If yes, please describe. (Make sure that this training was for the HAC and not training for individuals.) 9. Please describe any challenges your HAC has had in getting reports from the Bridge to Health Team and TBAs. Are they on time? Are they accurate? How do you check the accuracy? (Objective 7) 10. Please describe any feedback your HAC has received from SPAs or Health Facility staff in the last year. 11. How many of you have attended three or more meetings with Health Facility staff in the previous year? (Objective 3a/b) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 78 12. What have been the most important lessons learned from this CS-17 Project so far? 13. What does the Committee want to achieve in the next one year and is this written down? 14. Is there a need for your HAC to continue after Save the Children’s involvement is phased out in Negelle? If yes, what will its role be? What is your HAC doing to ensure its own continuation? 15. In what ways can this Project be strengthened? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 79 QUESTIONNAIRE SERVICE AREA TEAMS (SATs) SC/ETHIOPIA CS-17 MIDTERM EVALUATION JULY 2003 Before starting to ask the questions, the Note Taker needs to record the following: Name of the MTE Team Moderator Name of Note Taker Name of the other Evaluation Team Members Attending Start and end times Date Location Name of Group Number of Group Members Record the names, titles and affiliations of the respondents. INTRODUCTION: “Save the Children is carrying out an evaluation of its Child Survival Project to find out what has been successful and what can be improved. We will be asking you some questions and we want you to feel free to offer your opinions and thoughts based on your experiences and those of your family. If you do not understand a question, please feel free to say so. The information that you provide to us today will be used to strengthen our project.” “Thank you for participating today. Now for the first question….” 1. When was your Service Area Team formed? 2. How many members does it have? 3. How frequently does your SAT meet? 4. Are minutes kept for every meeting? If yes, who is responsible for recording them? Are minutes from the previous meeting reviewed at the beginning of the subsequent meeting? Does everyone get a copy? 5. What are your responsibilities as SAT members? 6. What has your SAT achieved since the start of CS-17 in 2001? 7. Has your SAT received training from the CS-17 project? If yes, how useful has it been in your work? Are there other training topics that should be included? 8. Please describe any challenges your SAT has had in getting Bridge to Health Team and TBA reports from the HACs. Are they on time? Are they accurate? How do you check the accuracy? (Objective 7) 9. How has the relationship between the health facility, the CHWs and the community changed during the project? 10. What type of feedback has your SAT received from the following in the past year: a. Save the Children b. Ministry of Health in the last year. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 80 11. Have you been having quarterly meetings with the Health Facility Management Committee to review performance and plan for the following quarter? DIP p. 75 12. What have been the most important lessons learned from this CS-17 Project so far? 13. What can be done by the Project over the next two years that will help to make sure that the positive changes that have happened will continue? What is your SAT doing to make sure this happens? 14. In what ways can this CS-17 Project be strengthened? CLOSE: “That is the last of the questions that we have for you today. Is there anything that you would like to add?” “Again, I want to thank you for your time and your advice.” CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 81 GENERAL QUESTIONS FOR ALL SC/LIBEN STAFF Provide an overview of the objectives and primary strategies of your project intervention(s). What have been the main accomplishments of these intervention(s)? What have been the lessons learned? What are the primary challenges you face in meeting your objectives by the end of the project and how can they be most readily addressed? PLANNING: (Guidelines page 11) Do you have a copy of the Program Objectives and the M&E Plan, or some other program plan that provides you with direction? Please show me. STAFF TRAINING: (Guidelines p. 11) What types of training have you and your staff attended since the beginning the project? What were the results of this training? Are there staff training needs that are not being met by the project and if so what are they? STAFF SUPERVISION: (Guidelines p. 12) Who do you supervise and how frequently do you have regular contact with them? Is this sufficient? Who is your supervisor and how frequently do you meet with him/her? Is this sufficient? What is the frequency of staff reviews for yourself and for any staff you supervise, and how are they done? Describe the frequency and a typical agenda for staff meetings. HR AND STAFF MANAGEMENT: (Guidelines p. 12) How long have you been in your position? Do you have copies of your job description and personnel policies? Please show me. What has been the impact from the changes in staffing that have occurred since the beginning of the project? CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 82 FINANCIAL MANAGEMENT: (Guidelines p. 12) Were you involved in developing the Project budget? If so, please explain. Do you have a budget and receive financial reports specific to your part of the project? Please show. LOGISTICS: (Guidelines p. 13) What logistical challenges do you face in carrying out your job responsibilities and meeting your objectives? What are the future logistical challenges? INFORMATION MANAGEMENT: (Guidelines p. 13) Describe any data reports you receive from the HIS and what is their frequency/regularity? Please show us samples. How have you used the information from these reports? TECHNICAL AND ADMIN SUPPORT: (Guidelines p. 13) What kinds of external TA have you received and how useful was it? What are your future external TA needs (i.e., needs that can not be addressed internally within the project.)? CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 83 ANNEX F CS-17 Progress at the MTE Per the Revised M&E Matrix CS-17 Results, Indicators/Sources, Progress at MTE and End of Project Targets by Intervention Result / IR # Indicator indicator source Progress at MTE EOP Target Interv. 1 District Health Management Team has met 3 or more times in last year and has used data to plan activities.(1) Review of 5 sets of minutes for mtgs. held April 4 – Oct 29, ’02 found: ÿ Meeting agendas focused on setting up DHMT and HIS Teams. ÿ No evidence of using data to plan program activities at that stage. ÿ No meetings held from October 2002 to the present. By MTE and FE All 2 District HIV/AIDS Council meets regularly, plans, and monitors HIV/AIDS activities in Liben. Review of 4 sets of minutes for mtgs held June 20, 2002 to June 2, 2003 shows that: ÿ The Council is meeting, but not on a monthly basis as initially planned. ÿ The Council is collecting and reviewing reports and is using this information to plan and monitor HIV/AIDS activities in Liben. Yes HIV R-1: Improved Liben District capacity to effectively support community health services and activities. 3a % of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year.(1) 80% All 3b % of PAs from which three or more HAC members have participated in three or more meetings with MOH staff over the previous year.(1) This indicator is going to be difficult if not impossible to quantify – requiring consistent and complete record keeping at every MOH facility (meeting minutes) and review of these minutes comparing meeting attendees with tallies of all HAC members. 80% All 4 % of CMWs trained in pneumonia case management with no stock￾out of cotrimoxazole in the previous month. 80% ARI 5 % of CMWs trained in malaria or pneumonia case management through CS-17 from whom reports were received in past quarter. Community Case Management has not been initiated yet. 80% ARI Mal. R-2: Improved community capacity in Liben to effectively address priority health needs of mothers and children under 5. 6 % of BHTs which in the last 6 months have conducted 1 or more community education activity for each CS-17 intervention and turned in 4 or more monthly reports to HACs.(1) Indicators #6 and 7 cannot be measured using the HIS as it is now structured. The BHTs are reporting to the HACs by memory and not by tally sheets as noted in the DIP (p. 72-6). 80% All CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 84 Result / IR # Indicator indicator source Progress at MTE EOP Target Interv. 7 % of HACs which in the last 6 months have reviewed BHT, TBA, or CMW reports, and have sent reports to health facility.(1) Reasons for this stated change were 1) the costs of reproducing the forms for 440 BHTs and 300 TBAs every month, and 2) the low literacy. 80% All 8 EFO Behavior Change Specialist hired and retained. This position has been temporarily filled by the HIV/AIDS Advisor/EFO. The permanent placement was made in August 2003. Oct. 2002 All 9 BC strategy for all CS-17 interventions designed and implementation started. This has not been completed yet. March 2003 All 10 HIV prevention efforts effectively integrated into ongoing community and government activities through CS-17. Solid progress has been demonstrated through the project’s involvement on the District HIV/AIDS Council and its support of capacity building of PAs, CBOs and community groups. Yes HIV IR-5: Increased SC Addis and Liben capacity in behavior change and integrated HIV programming . 11 Number of HIV-related training courses, workshops, and experience sharing visits in which SC/Liben staff have participated during CS-17. BCC Conference, 4/03, attended by 2 SC/Liben staff, incl HIV/AIDS Unit Head PRA Conference, 5/03, attended by HIV/AIDS Unit Head 5 HIV Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. CS-17 Results and Indicators Related to Use of Health Services and Health Practices Measurement Methods, Data Collectors, Baseline Values, and End of Project Targets Result / IR # Indicator indicator source Midterm Progress Target Interv. 12 Total rate of treatment for pneumonia in <5s by CMWs in all PAs with CMWs trained in PCM (number of treatments per <5 per year). Community Case Management has not been initiated yet. 0.2 ARI 13 % of respondents reporting condom use last time they had sex with non-regular partner. 30% HIV 14 % of respondents reporting condom use every time they had sex with any non-regular partner over past 12 months. These indicators require FHI assessment midterm results, which have not been collected yet. 20% HIV 15 % of births attended by trained TBA or health professional.1,2 50% MNC 16 % of all mothers of children <2 receiving TT2+ before last child’s birth (card).2,3 These indicators require KPC results, therefore, they can not be assessed during the MTE. 50% EPI R-3: Increased use of key health services and improved MCH practices at household level in Liben District. 17 % of pregnant women receiving TT2+. Coverage increased from 26% to 35% based on DHO data. 55% EPI CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 85 Result / IR # Indicator indicator source Midterm Progress Target Interv. 18 % of all 12-23 month olds who received measles immunization (by card only).2,3 This indicator requires KPC results, therefore, it cannot be assessed during the MTE. 60% EPI 19 % of infants who received measles immunization. Increased from 43% to 72% based on DHO data. 70% EPI 20 % of all 12-23 month olds fully immunized (by card).2,(3) This indicator requires KPC results, therefore, it cannot be assessed during the MTE. 40% EPI 21 % of infants fully immunized. Increased from 34% to 62% based on DHO data. 60% EPI 22 % of children <2 with diarrhea in the past 2 weeks receiving more fluids than usual and same or more food than usual during illness.(3) 50% CDD (#33: New objective added in First Annual Report, Sep. 2002, in response to DIP review.) 33 % of mothers with children <24 months who report washing own hands with soap or ash before food prep., before feeding children, after defecation, and after attending child who defecated. 3 These two indicators require KPC results, therefore, they can not be assessed during the MTE. 25% CDD Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. CS-17 Results and Indicators Related to Uptake, Availability, Quality, and Knowledge Measurement Methods, Data Collectors, Baseline Values, and End of Project Targets by Intervention Result / IR # Indicator indicator source Midterm Progress Target Interv. R-4: Uptake/ Sustainability: Adoption of CS￾17 approach by MOH or by other organization. 23 MOH or other PVO/NGO in other district of Ethiopia has written plans for implementation of CS-17 approach to C-IMCI, MN/LSS, or BHTs. CARE/Ethiopia has expressed interest in providing HBLSS training to the TBAs in their project area in West Harage and requested that SC’s MCH Unit Head assist in that training. Yes All IR-4: Dissemination of feasibility and results of implementing innovative CS￾17 approaches. 24 Feasibility and results of implementing CB-ARI/Mal. case management, MN/LSS, and/or BHTs, through CS-17, presented at conference(s), in publication, through media, and/or site visit. Presentation, “Lessons from Ethiopia: Maternal Care in Low Resource Settings,” given by Sister Degefech Haileyesus, SC/Liben MCH Nurse and HBLSS Coordinator at the April 15, 2002 Vienna, Austria Conference, “Low Tech, High Effect: Care for Women and Infants in Disasters” sponsored by Johnson & Johnson. Yes ARI MNC IR-1: Increased availability of select MCH services in Liben. 25 % of rural PAs which have an MOH facility or CMW(s) trained through CS-17 in ARI or malaria case management. Each of the 8 rural PAs that has a health facility has at least one staff person trained in ARI/malaria case management. 100% ARI Mal. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 86 Result / IR # Indicator indicator source Midterm Progress Target Interv. 26 % of rural PAs with TBAs trained in HB-LSS.(1) Each of the 36 rural PAs has at least one TBA trained in HB-LSS. 100% MNC 27 % of children under five assessed for pneumonia for which CMW reported completing all PCM steps correctly. Community Case Management has not been initiated yet due to delays resulting from GOE policy. 80% ARI IR-2: Documented quality of select community MCH services in Liben District. 28 % of mothers/newborns with complications for which TBAs reported completing all HB-LSS steps correctly. The exact number of complicated deliveries required for the denominator is not available. As a surrogate measure the records of 14 complicated deliveries were reviewed. It was found that 36% (5/14) of TTBAs had completed all the HB￾LSS steps correctly. 50% MNC 29 % of mothers reporting either fast breathing or difficult breathing as a sign of child illness needing treatment. 65% ARI 30 % of mothers who report knowledge of at least 2 maternal danger signs during the postpartum period. (1),3 These indicators require KPC results, Therefore, they can not be assessed during the MTE. 50% MNC 31 % of respondents who identify consistent condom use, mutually monogamy, and abstaining from sex, as methods of reducing risk of HIV. 50% HIV IR-3: Increased maternal knowledge in Liben District of selected MCH issues. 32 % of respondents who identify 2 or more signs/symptoms of STIs. These indicators require FHI assessment midterm results, which have not been collected yet. 25% incr. HIV Indicator sources: 1: CS-13; 2: Current DAP; 3: KPC 2000+ CATCH / KPC 2000+; ( ) indicator modified. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 87 ANNEX G SUMMARY FINDINGS, CONCLUSIONS, RECOMMENDATIONS AND ACTION PLAN SUMMARY FINDINGS and CONCLUSIONS RECOMMENDATIONS INITIAL ACTIONS PERSON(S) RESPONSIBLE PROJECT OVERVIEW 1. The SC/Liben CS-17 Project incorporates several promising and innovative strategies: ÿ HB-LSS ÿ District HIV/AIDS Council’s Community Capacity Building initiatives. ÿ Integration of DAP and CS program activities through joint EPI, ANC and FP outreach. ÿ The potential for CCM. 1. Document project successes and advocate for their adaptation to other project sites in Ethiopia and other Child Survival Projects throughout the world. 1. Collect information on program successes. 1. Health Sector Manager, with the support of the M&E Coordinator, the EFO and SC/HQ. HIV/AIDS 2. VCT capacity in Liben District is limited to the sparse services available in Negelle Hospital. 2. Build the capacity of the District in VCT. More staff in Negelle Hospital need to be trained in counseling and referral mechanisms need to be established at the health facility level that will involve orientation and training of MOH staff and the establishment of a monitoring system to track referrals from the outlying health facilities to the hospital. 2.a. Assess the sustainability of the supply of reagents. 2.b. Identify training opportunities for hospital staff in counseling. 2.c. Strategy for increasing demand for VCT services. 2.d. Assess infrastructure needs 2. Medical Director/Negelle Hospital, the Oromia RHB and the Health Sector Manager. 3. Mothers living in the towns report that they are learning about health through TV and radio, which are becoming increasingly available. 3. The project should explore opportunities to exploit all types of media (TV and radio) available in the project area for communicating IEC messages on HIV/AIDS. 3. Explore the potential for developing and placing health education ads on TV and radio that would appear in Liben District. 3. SC/HIV/AIDS Unit Head with the support of the CS-17 Training Coordinator and the EFO. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 88 SUMMARY FINDINGS and CONCLUSIONS RECOMMENDATIONS INITIAL ACTIONS PERSON(S) RESPONSIBLE MATERNAL/NEWBORN CARE 4. The District is unable to respond to obstetric emergencies and therefore, women are put at risk of maternal death. 4. The EFO, along with other NGOs, multilaterals and government agencies involved in maternal health, should advocate for a change in the GOE policy that restricts the use of c￾sections and other EOC surgical procedures to obstetricians, so that general practitioners can be permitted to do these procedures upon completing an appropriate training program. 4.a. Advocate for change in GOE policy restricting care for comprehensive obstetric emergencies to obstetricians. 4.b. Solicit funding to cover costs of training a team from Negelle Hospital (1-2 general practitioners, one scrub nurse and one anesthesiology nurse) and address related infrastructure needs. 4.c. Pursue development of emergency transportation plans at the community level once EOC services are available at Negelle Hospital. 4.a. EFO 4.b. EFO and SC/HQ 4.c. SC/Liben MCH Unit Head and the SPAs. 5. There is no regular mechanism for assessing and reviewing complicated deliveries and maternal mortality in Liben District, the information from which could be used to improve practices and the quality of care. 5. The Project should assist Negelle Hospital in establishing a regular system for reviewing complicated deliveries within the District, interviewing the involved TBA, health workers and women to assess whether all the necessary steps were followed and identify areas needing improvement. 5. Establish protocols and regular meeting times to review cases. 5. Medical Director/Negelle Hospital and the SC/Liben MCH Unit Head 6. The TBAs report lack of access to gloves for deliveries, which is a growing concern with the increasing presence and awareness of HIV/AIDS. 6. The project needs to develop sustainable mechanisms for ensuring a consistent supply of delivery gloves for TBAs. 6. Identify internal Save the Children expertise on RDFs and assess feasibility within Liben context and explore other supply sources. 6. MCH Unit Head and Health Sector Manager CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 89 IMMUNIZATIONS 7. Two health facilities lack functioning refrigerators and some lack EPI cards, which is limiting EPI coverage. 7. Providing the promised cold chain equipment and EPI cards as soon as possible needs to be a priority as further delays will directly limit the intervention’s impact. 7.a. Supply refrigerators as required in the DIP. 9.b. Reassess need for EPI cards as needed. 7. EPI Unit Head 8. Some mothers expressed concerns about having their children immunized due to perceived side￾effects. 8. The project needs to work closely with the HACs, BHTs and TBAs through the SATs to further assess the nature and underlying causes for any community misconceptions about the safety and efficacy of childhood vaccines, and develop responsive IEC strategies. 8. Implement FGDs with mothers to identify extent of the belief and if necessary develop IEC messages to counteract false perceptions. 8. EPI Unit Head, the SPAs and the Training Coordinator. MOBILIZATION OF COMMUNITY LEADERS AND TRADITIONAL PRACTITIONERS 9. The MTE Team found that the project has achieved solid community involvement through its participation as CHWs and in project activities. These workers and activities need to continue beyond the life of CS-17. 9. The project needs to identify and institutionalize rewards the communities are willing and able to provide as incentives to the CHWs to ensure their continued volunteering with the project and the ability to recruit new CHWs as others move on. 9. Bring the Project staff and partners together to design a system of incentives for community volunteers, which is sustainable by the community itself. 9. Health Sector Manager, Area Manager, senior Liben staff and program partners. COMMUNITY CASE MANAGEMENT OF CHILDHOOD ILLNESS 10. SC/Liben will need technical support in designing and implementing the CCM strategy. 10. The direct and concentrated involvement of the EFO technical backstopping staff will be required for the adaptation of the CCM training materials and related IEC strategies as well as the overall planning and implementation of CCM. 10. Hold a training meeting to develop an action plan for the design and implementation of CCM 10. EFO backstopping staff, the Health Sector Manager, the MCH Unit Head, the Training Coordinator and representatives from the MOH. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 90 11. A system that can provide a consistent supply of antibiotics and antimalarials for the CMWs needs to be developed. Options include the development of a revolving drug fund and/or improved supply systems through the MOH. 11. Technical and administrative support is going to be required throughout the planning and early implementation of the RDF or other mechanisms for improving drug supply within MOH facilities. 11. a. Identify and access individual(s) with both practical experience and expertise in the design and development of Revolving Drug Funds. 11. The Health Sector Manager, the EFO and SC/HQ. CAPACITY BUILDING OF THE MOH 12. The GOE policy of ‘decentralization’ has limited the technical and administrative capacity of the DHO and the DHMT. 12. The project needs to reassess the current and projected capacity building needs of the new DHO within the context of the remaining CS-17 project and the long-term needs related to its sustainability plans. This could require significant restructuring of the project, especially if a majority of these needs cannot be resolved through locally available sources. 12.a. The potential for growth within the DHO needs to be assessed, which will involve review of any financial and technical resources that it can access through the Zonal, Regional and National Governments and other sources. 12.b. All means for accessing technical and financial resources from outside the DHO need to be assessed and if feasible, pursued, including the possibility of contracting with Negelle Hospital for technical support using DHO budget funds. 12. The Health Sector Manager, the EFO and the members of the DHMT and DHO. STRENGTHENING HEALTH FACILITIES AND WORKERS 13. There is an absence of well￾written meeting minutes, necessary to track progress on some of the project indicators. 13. Training on recording meeting minutes is required for the SATs and through them, the HACs. 13.a. Design and implement training. 13.b. Followup supervision and review of minutes. 13. Training Coordinator and the SPAs. 14. CS-17 staff report not having received technical support in the design, implementation, monitoring and evaluation of training plans and curricula. This is especially timely with the start of CCM approaching. 14. Increase the capacity of the CS-17 staff in the design, implementation, monitoring and evaluation of training materials and programs through the provision of TA, support and resources from the EFO that can be made available to the staff in Negelle. 14.a. Complete a more in-depth assessment of the capacity building needs. 14.b. Collect and share materials on training with the SC/Liben staff. 14. EFO and Training Coordinator CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 91 15 In FGDs with the SPAs, they relayed frustrations expressed by the CHWs that they do not receive per diems for the HAC meetings they attend, which is different from their experience during the project trainings when they did receive per diem for attending. 15. The extent of the frustrations expressed about per diems needs to be assessed further and, if necessary, senior project staff (i.e., the Health Sector Manager and/or the Impact Area Manager) need to meet with the CHWs to discuss this issue and come to consensus on solutions if need be. 15. FGDs with CHWs to assess the extent of the frustration and to determine next steps. 15. FGDs can be done by the SPAs and then based on results, involve the Health Sector Manager and/or the Impact Area Manager in meetings with the CHWs. SUSTAINABILITY STRATEGY 16. It is apparent from the DIP and verified in this MTE that the DHO will not be able to adequately support this program beyond the life of CS￾17 without continued outside support. 16. The project needs to identify and access all currently and potentially available sources of support for project activities to continue beyond CS-17. 16. See the Initial Actions proposed for the MOH Strengthening Recommendation above. 16. Health Sector Manager, the EFO and SC/HQ. 17. “Sustained increase in the use of key health services and improved MCH practices at the community level” are crucial to the sustainability of project services. However, due to the lack of a population-based survey during this MTE, the project will not know its progress on the KPC￾measured indicators until the FE, when it will be too late to make adjustments. 17. The project should consider implementing an abbreviated KPC using LQAS in the summer of 2004 to assess progress on household knowledge/practices and identify any areas in need of improvement. 17.a. Assess the budgetary implications of doing the survey. 17.b. Identify and access expertise to design and implement the survey. 17. Health Sector Manager, the EFO, SC/HQ and the local partners. STAFF TRAINING 18. CS-17/Liben were consistent in expressing the need for further training that would improve their work on the project and their own professional standing. 18. Implement a training needs assessment and plan for the staff with the goal of improving their work with the CS-17 project and also enhancing their employability upon its completion. This plan should include the SPAs as well as the senior health staff. 18. Develop a detailed list by individual staff person on training topics. 18. Training Coordinator and the other Liben CS staff and partners. SUPERVISION OF PROGRAM STAFF The Liben CS-17’s greatest assets are N/A N/A N/A CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 92 its staff and the relationships it has developed with the program partners and communities. 19. The senior Liben CS-17 staff have not been meeting regularly, primarily due to the frequent changes in the Health Sector Manager position since the beginning of CS￾13. This has probably contributed to the number of delays the project has experienced. 19. Regular staff meetings, involving the Health Sector Manager, the Unit Heads and the M&E and Training Coordinators, need to be restarted as soon as possible. It is suggested that initially they occur on a weekly basis at the same time and place. They can then be scaled back as seems prudent. 19. Reestablish regular meetings of the senior health and M&E staff. 19. Health Sector Manager, with the Health Unit Heads and the Training and M&E Coordinators. 20. Responsibility for both MNC and the CCM interventions (CDD, ARI and malaria) rests entirely with the MCH Unit Head. Together these interventions are 55% of the total planned intervention-specific effort, which might be too much for one person. 20. The workload for the MCH Unit Head needs to be reviewed and probably adjusted prior to the startup of the CCM intervention. 20.a. Review the MCH Unit Head Position Description 20.b. Consider the addition of staff, sharing responsibilities with other current staff, or some other arrangement. 20. The Impact Area Manager and Health Sector Manager with the MCH Unit Head. 21. One third of the project health facilities in Liben and the communities they serve have yet to be served by SPAs due to the delay in hiring. 21. The three remaining SPAs need to be hired, oriented and placed in the field as soon as possible. 21. Complete the hiring process as soon as possible. 21. Health Sector Manager and Training Coordinator. LOGISTICS 22. Only two of the six (and soon to be nine) SPAs have a motorcycle, which greatly restricts their access to the community, a critical factor especially with the proposed start of the CCM activities and the establishment and support of the Revolving Drug Fund pharmacies. 22. With the delays experienced to date and the amount of work that needs to be accomplished in the most rural, underdeveloped PAs in Liben District by the end of CS-17, SC needs to provide one motorcycle for each SPA. 22. Negotiate borrowing DAP motorbikes (originally purchased for EPI) or seek authorization to purchase them. 22. The EFO, Impact Area Manager, Liben Impact Area Administration and the Health Sector Manager. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 93 INFORMATION MANAGEMENT 23. Community level data is not being collected, recorded and reported on consistently or accurately, due to CHW and HAC confusion about the forms and the process. 23. The project needs to reassess its information needs starting at the community level, focusing initially on the information the CHWs will use to improve their work and involving them in the process. 23.a. Practical TA is needed on the design and development of community-based health information systems. 23.b. FGDs need to be held with the CHWs to identify their information needs and secure their support for collecting the data. 23. M&E Coordinator, the Training Coordinator, the SPAs and a community-based HIS expert. 24. The project has experienced several delays and difficulties in developing and adapting the software database for the DHO – the primary challenge being its complexity and the question of whether the DHO can effectively use a computerized system. 24. The capacity of the DHO to effectively use a computerized HIS needs to be reevaluated and any adjustments to the plan/strategy be made. 24. The M&E Coordinator, Health Sector Manager and members of the DHMT need to review the strategy for developing this software, determine whether it remains viable, and identify next steps. 24. M&E Coordinator, the EFO/IT, the Health Sector Manager and the members of the DHMT. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 94 TECHNICAL AND ADMINISTRATIVE SUPPORT 25. The Liben CS staff identified the following areas requiring technical assistance and suggested strategies: TOPICS: ÿ NGO and project planning, management and supervision ÿ Budgeting and financial management ÿ Production of IEC materials and documenting program successes. ÿ Development of training materials. ÿ M&E STRATEGIES: ÿ Regularly scheduled site visits by the EFO. ÿ Updating the established health and training library. ÿ Exposure visits to other project sites and between SPA sites. ÿ Attending relevant training workshops and activities. 25. The EFO needs to commit to making regular site visits at least once every quarter and more frequently during key points in the life of the project, such as during the design of program activities, the testing of training curricula, the startup of new project initiatives and all major evaluations and assessments. 25.a. Establish a schedule and agenda items for EFO site visits. 25.b Plan exposure visits for SPAs. 25.c. Identify appropriate training opportunities. 25.a. Health Sector Manager, Liben CS staff and EFO 25.b. Training Coordinator and SPAs 25.c. Training Coordinator, EFO and SC/HQ. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 95 ANNEX H VISIT TO SC IMPACT AREA – LIBEN DISTRICT (NEGELE) 14/8/03 – 19/8/03 STANLEY O. FOSTER MD, MPH VISITING PROFESSOR OF INTERNATIONAL HEALTH EMORY UNIVERSITY’S ROLLINS SCHOOL OF PUBLIC HEALTH ATLANTA GA 30322; email:sfoster@sph.emory.edu BACKGROUND · In 1993, I was privileged to review the Child Survival 13 DIP for USAID. I found the DIP to be the best of the many Child Survival Projects that I have seen. · The DIP was identified as a great opportunity for MPH student learning; permission to use the data for teaching was requested from SC Westport and received. · In the spring 2003, 66 students studied your CS project. The group was divided into 6 groups. Each group was further divided into 4 sections of 4 each with responsibility for: child preventive, child curative, maternal and neonatal, or HIV AIDS. Each group analyzes the data from the CS 13 KPC or the recent DHS for Oromiya. The students made two presentations: 1. identifying the major problems, and 2. proposing intervention strategies. The presentations are oral with PowerPoint backup. One such PowerPoint is posted on the wall behind Solomon’s desk. · Over the years, I have read with admiration the documentation of progress in the CS 13 midterm and final evaluations, and the CS 17 DIP. · My current trip as a visitor to your project has four purposes: o To thank you for allowing us to watch from a distance, to congratulate you on the excellence of your project, and to thank you for the privilege of using your data o To observe first hand the organizational structure and activities of CS 17. o To gain a better understanding of Health Action Committees and Bridge to Health Teams. o To listen to your achievements and your concerns. METHODS: Review recent documents and reports Visit two Health Stations, Ganale and Jidola Visit the District Medical Officer, the EPI unit, the Pediatric Ward, and the Statistics unit. Listen to selected staff as to their achievements and concerns Hold a pre-departure dialogue with available CS 17 staff CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 96 FINDINGS · The greatest surprise and in many ways the most exciting was learning of the multisectorial strategy of the SC Impact Area. The evolution of relief to Human Health (Child Survival), Natural Resources (Animal Health, Range Land Management, and Water Development), Education and Assets/Income Diversification is extraordinary and merits documentation and sharing. · Dialogue with HACs and BHTs affirmed the read-about ownership of the project by the communities · Discussions with the HAC, BHTs (including TBAs and herbalists) members documented significant capacity building in terms of knowledge of disease transmission, prevention and treatment. In a response to a question as to what they were most proud of, responses included family planning, prenatal care and practices during pregnancy, delivery, care of the newborn, immediate and exclusive breast feeding, supplementation at 6 months, immunization and treatment of illness, and HIV prevention. · HAC and BHT members also identified barriers to effectiveness (all of which had been identified to me by project staff) including: o Lack of ability to get sick patients, especially obstetric emergencies, to the hospital. Communications and transport were identified as the major obstacles. o Lack of essential drugs (“undermining our advocacy on the need for early and quality care”); the current systems provides a fixed amount of drug to each health facility; fees collected for the drugs are returned to Ministry of Finance; no system exists for the replenishment of drugs (discussions on revolving funds are currently in process in the MOH). o Lack of gloves for TBAs. · An additional complicating factor is the absence of a surgeon or gynecologist at the Negele Hospital to address obstetric emergencies. Last year, only 8 C￾sections were carried out among 375 deliveries, just over 2% (versus an expected norm of 10-15%. Severe cases (ruptured uterus, placenta previa, and abnormal lies are referred to Awasa 300km and 6 hour drive to the north. Critical cases are unlikely to survive such a referral. A VISITOR’S RESPONSE TO CONCERNS RAISED BY PROJECT STAFF HIV/AIDS · The rates of HIV positivity reported from Negele Hospital are alarming, Table 1 Table 1: HIV Positivity at Negelle Hospital Ethiopian Year Positive Negative Total and % Positive 93 97 103 200 (48.5) 94 88 125 213 (41.3) 95 182 254 436 (41.7) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 97 · These results are difficult to interpret as the hospital aggregates results from 4 groups: 1. volunteer blood donors, 2. commercial blood donors, 3. VCT participants, and 4. suspect patients. In discussions with DHO, future results will be reported by risk groups. · Although the tests being used at the hospital are highly reliable, independent accreditation of the HIV lab is absolutely essential. If the results are confirmed, this is a public health emergency and will challenge the current understanding that HIV in Ethiopia is predominantly an urban disease. · Until proved otherwise, the reported seropositivity mandates that the issue of gloves for TBAs be addressed immediately. Capacity Building · From the limited sample of Health Workers, HACs, and BHTs in two Kabelles, CS 13 and 17 have been very effective in building and strengthening capacity (CONGRATULATIONS-AMONG THE BEST I HAVE EVER SEEN). · Maintaining this capacity requires more than training, it requires three things: continuing education, supportive supervision, and quality assurance. · This can best be achieved, my monthly supportive supervision visits to each health facility to carry out the following tasks: o Identify and commend them on strengths and achievements o To listen to their concerns and work with them to address them o To collect data on key program elements using a short maximum two page check list (modify check list developed by WHO EPI) such as drug supply. o Hold a continuing education session on a specific topic (EPI, Maternal Neonatal, HIV, Family Planning, Nutrition, Hygiene, etc). Staff would be cross trained to fill out check list; individual staff members would have responsibility for 1 (maximum) 2 Kabellis per month. Two day visits are preferable and would allow them to probe in depth their area of expertise. Case Management The Health Assistants at the two Health Stations visited (one government and one SC) described treatment of diarrhea and pneumonia correctly. (Simplified hands-on observation, preferably at Negele hospital, is needed to verify clinical competency). While not currently allowed by the MOH, community treatment especially for malaria is under discussion. Should SC embark on community treatment, it is suggested that lessons learned in the CARE Siaya project in Kenya be reviewed. Community treatment can not be started without ensuring a regular supply of affordable quality drugs. Again the Siaya experience with community operated revolving drug funds merits review. Need for community treatment may vary by Kabelle. In Ganale, most of the population had access within 2 hours walk. In contrast, Jidola catchment area reportedly extends to 6-8 hours walk. Clearly where access is possible, health assistant treatment is preferable. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 98 Monitoring · The CS 17 DIP has a realistic monitoring plan. · Discussions with the M&E officer identified the current tracking of CS 17 on 23-26 forms. This is far in excess of that required by the CS 17 DIP. · In response to this request, Attachment 1 has been prepared for SC Addis. Should after reading the midterm review, these comments appear useful; SC Addis would be the appropriate channel to provide them to SC Negele. FUTURE CONSIDERATIONS Diarrhea is a major cause of morbidity and mortality in children under five. Diarrhea is also a major contributor to weight loss, malnutrition, and chronic child ill health. Roughly, one quarter of OPD visits are for diarrhea, Figure 1. Traditionally, the diarrheal strategy has focused on treatment. Recently, introduction of home chlorination of water has proved very effective in reducing the incidence of diarrhea. Table 2 below demonstrates the synergistic impact of multiple diarrhea prevention strategies. More information on Safe Water can be found on the CDC website (http://www.cdc.gov). Search for Safe Water and you will find the Safe Water Manual on line. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 99 ATTACHMENT 1 - MONITORING BACKGROUND · In response to a request from M&E Liben Impact Area to assess the current and proposed information system for CS 17, I have reviewed both the current 27 forms and the MOH HIS format adapted to Liben. · These are far in excess of the DIP monitoring plan and indicators · My findings are prepared for SC Addis for their information and action as appropriate. CS 17 MONITORING FORMS · Listed in Table 1 is my assessment of the M&E forms with suggestions for SC consideration. *** Useful to program management # Subject ***. 20 Family Planning Yes Accountability for Commodity Use In this format or GOE HIS Better to divide category into two (new acceptors, continuing) Suggest remove remarks here and all forms – major burden on data entry 21 Counseling Sessions No Real question is whether KAP has been changed – the final KAP survey is the best way to measure this 22 Deliveries ? In 2002, only about half of deliveries were reported. Without denominator, difficult to interpret. Would defer to ACNM, e.g. Lynn Sibley on this 23 Visit to HF No Better captured in reportable diseases, EPI, and MCH data Form as propose not useful 24 Promotional Activity No A lot of activities are being carried out by volunteer HAC and BHTs. A lot of paper work is of limited value especially as some are not literate. Monthly meetings ( see supportive supervision below will be adequate to determine) 25 Morb and Mort Yes Mod An analysis of data reported (see graph above) show that 10 conditions account for most childhood illnesses. The HIS form shows 51 diseases, a disaster. WHO recommends 19, many of which can not be diagnosed without a laboratory. A selection from the 19 which can be diagnosed clinically is suggested. 26 BF No Like above # counseling sessions poor predictor of behavior change, rely on KPC to document changes 27 Family Ration Distribution ? NOT MY FIELD 28 Growth monitoring results No Children are weighed to help the mother (congratulate if growing, counseling if weight falling). Current strategy is to focus on the most vulnerable (0-2) and to focus not on percentiles but on slope (gaining, flat, declining) with appropriate counseling. Most important would to include Hts and Wts in final KPC to compare to 2001 Nutritional Survey. 29 TT Women Yes Used at HF and District Level, plotted 30 EPI <1s Yes Ditto 31 Equip. Trans. No Should be monitored at HQ by transport system HF equipment better monitored by monthly visit (See SSV below), this form of monitoring increases probability of appropriate action 32 Births No As most births occur at home, administrative estimate adequate KPC data will provide valid data on delivery by attendant type 33 Mat NN Deaths No Very low sensitivity, unless ongoing monitoring of the population 34 Referrals No Without surgeon at hospital to do C-sections, value of referral limited 35 Community. With HAC, BHT, Alarm Fund No Better captured on supportavisory SSV check list (see below) 36 Availability Essential Drugs and Transport Fund Yes Absolutely essential as it is key area limiting program effectiveness. Better to assess on monthly SSV visit and analyze from check list rather than routine reporting (HIS format also a possibility – only if actions are taken – no evidence of that at moment) 37 Drug and Transport Funds No Presence of fund unimportant, effective function more important, include on supervisory check list CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 100 38 DHT meetings Yes Best captured by monthly meeting with DMO as to the process actions taken and follow up – minutes of this meeting best are best indicator here 39 MOH HAC BHT Meet. Better capture SSV visit, did the meetings occur, what happened, actions taken 40 CMW Training Supplies Yes Provided the issues of CMW using drugs and drug availability are solved, monitoring of introduction important. 41 Health Ed Activities No Like counseling and promotion, written reports divert attention to paper work. Real key here is whether or not KAP are changing (better measured in KPC) Have the BHT and HAC report on their activities orally during SSV visit 42 Pneumonia Treated ? Very difficult to monitor, requires regular review of log of patients treated, quality assurance of case management, and adequacy of drug supply. 43 PAs with trained TBA Yes Semi or Annual audit by training unit (trainees by Kabelle and Pas) meeting with HACs, important to look at both numerator (trained, HBSS steps completely) and denominator (total TBAs in PA, 44 Schools with HIV clubs Yes List of all schools by kabelli, clubs, members, active 45 School HIV Activities ? Need to determine how these data are to be used, does the use of the data justify work in collection and collation 46 HF HIV Activities ? Clients requires VCT availability ? ready for this yet 47 Training Yes Easy to collect; poor predictor of performance, assessing and upgrading performance are key to capacity building CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 101 ANNEX I Ethiopia Trip Report July 2-21, 2003 Winifride Mwebesa I. Purpose of the trip · Review the Liben District Safe Motherhood Model using the Household to Hospital Continuum of Care gap analysis matrix to determine if key components needed to reduce maternal and neonatal morbidity and mortality are included. · Determine current practices by health workers/ trained TBAs at hospital, peripheral facilities and community levels regarding the management of pregnancy and delivery - related complications. · Identify gaps and barriers to effective practice. · Identify actions needed to improve and strengthen the system in place. · Provide feedback on findings to CS-17 MTE. II. Background information. Save the Children has supported 2 Child Survival projects throughout Liben District since 1997, CS-13 followed by CS-17 with activities that focus on maternal and newborn care. A third project Development Assistance Program focuses on nutrition, family planning, breastfeeding support as well as food security and livestock management. Liben District in southern Ethiopia has a total population estimated in 2002 at 138,310. 75 % of the population is rural and lives in scattered and temporary pastoral settlements grouped into 42 Peasant Associations or PAs. The economic mainstay is livestock but frequent droughts have led to a depletion of herd numbers. Women between the ages of 15 and 49 make up 22.9% and pregnant women account for 5% of the total population. The crude birth rate is 46.4 annual live births per 1000 total population or approximately 6400 annual live births. The MMR for 1994 -1999 is estimated at 871/100,000 live births (Ethiopia DHS 2000) but could be as high as 1800 deaths per 100000 live births (WHO/UNICEF/UNFPA model estimate for 1995). The NMR is estimated at 49 per 1000 live births and the IMR at 83/1000 live births. MOH health facilities in the District include Negelle Borana Hospital, 6 clinics and 3 health posts. Clinics and health posts are staffed by 2-3 health workers and provide primary health care services. According to a KPC survey conducted in 2001, birth assistance by a skilled attendant was only 11%, with 29 % of deliveries attended by untrained TBAs and 35 % of deliveries by a family member, friend or the woman herself. Maternal knowledge of danger signs that would prompt seeking for care was very low. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 102 CS-17 currently in its second year (October 2001 – September 2005) aims to reduce maternal and under-five deaths by increasing community demand, support and use of key maternal health service and by improving provider capacity to address the health needs of mothers and children under 5. To achieve these objectives, the following activities are being implemented: · Community mobilization by involving the community in increasing awareness about safe motherhood and generating a supportive environment. · Increase access to life-saving measures at community level and within the home by training TBAs in Home Based Life Saving Skills (HBLSS). · Improve provider capacity by training health facility staff in Life Saving Skills (LSS) The American College of Nurse-Midwives provided assistance in training. III. Methodology SC has developed a systems approach to Safe Motherhood that includes the household, the peripheral health facilities and the hospital. At each level of care, key interventions have been effective in improving maternal and neonatal outcomes.(Annex f: The Household to Hospital Continuum of Care). SC assesses the level of care available considering the basic interventions that can be performed in the home (where most women in this setting deliver) and building on progressively as skills and supplies gradually improve, to the quality of Emergency Obstetric Care available at the highest referral level in the district. By identifying and addressing where possible gaps in the system, SC hopes to link the 3 tiers and create a household to hospital continuum of care. Data was gathered from different sources including: · Project documents · SC workers at field office and at health facilities · MOH workers at health facilities · Community volunteer health workers: - Bridge to Health Teams (BHTs) - Health Action Committees (HACs) - Traditional Birth Attendants (TBAs) In addition to reviewing secondary data at the field office and health facilities, the evaluation involved collection of primary data. Data was collected through: · Interviews with health care providers to evaluate the knowledge and skills in RH care service delivery during pregnancy, delivery and the early postpartum period with a focus on the management of obstetric complications. Inspection of health facilities using checklists to assess the availability, adequacy and state of medical equipment, drugs and supplies available to provide health care services. Focus group discussions with community members including local TTBAs, HAC (Health Action Committee) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 103 members and BHT (Bridge-to-Health Team) members to assess the knowledge imparted to the community through community education sessions and level of the skills achieved by TBA training. Inquiries were made about the existence of financial schemes to fund emergency obstetric care and the quality of the referral and transport system linking the community to health facilities. Unfortunately due to time constraints, · Only 4 clinics and Negelle Borana Hospital were visited, as the health posts were too distant. One might assume that quality of services at health posts might be very different from what was observed in clinics. · It was as not possible to directly assess the type of care provided to patients, which could have been done by observing TBAs carrying out HBLSS, or health facility staff managing obstetric complications either in clinics or at the hospital. · No exit interviews were conducted to obtain patient perspective on quality of care. Furthermore, CS-17 is currently in its 2nd year and TBA training has not been completed. Feedback on data collected was shared with CS-17 MTE team leader prior to leaving Negelle. The evaluation addressed the following questions: · How effective has community mobilization been in creating awareness about pregnancy related issues? · How much support has been generated at community level for pregnant women and infants? · Has community education on danger signs led to increased referrals/ decreased delays in referrals of women in need of Emergency Obstetric Care? · How has TBA training improved the management of cases at household level and is there any documentation to that effect? · What is the current quality of care provided to pregnant women at each level: peripheral health facilities (clinics, health posts) and at Negelle Borana Hospital in terms of Essential Obstetric Care and Emergency Obstetric Care? · What input is needed to strengthen and improve the system? IV. Findings A. Community mobilization for MCH services. SC community intervention focuses on reducing the delay in seeking care by mobilizing the community to increase awareness of obstetric complications. Selected members of the community are involved in the process including: · HACs (Health Action Committees) and BHTs (Bridge-to-Health Teams from CS-13 program), which have been established in each PA (Pastoral Association). Each HAC is composed of 10-12 community members, mostly male (BHTs, elders, influential leaders, kebele chairperson) and staff from the nearest health facility who serve as intermediaries between the community and the health centers. They assist their communities in identifying health problems and planning and implementing solutions. HAC provide leadership by supporting the 4-6 BHTs and all the TTBAs in the kebele. HAC collate CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 104 data from TBAs, BHTs and the community and report to health facility staff. They review health information with Senior Program Assistants at health facilities. · BHTs who are traditionally accepted community based providers and include a chereti (wise woman often a TBA), a wiseman (traditional healer) and a young traditional apprenctice. BHT members were former local traditional healers and TBAs, bonesetters and herbalists. There are 450 BHTs, organized into 150 three-member BHTs. · TBAs, mature women, known by the community to provide assistance to women at the time of delivery. 308 TBAs have been identified and are included in the training program. CBRHAs, Community Based Reproductive Health Agents though not involved in educating mothers and their families about the danger signs of pregnancy also offer invaluable services. They provide information and counseling on FP, HIV and STDs/STIs. They distribute condoms and contraceptive pills and refer couples requiring other methods (injectables, IUDs) to health facilities. CBRHAs report to local health facilities. HACs, BHTs, TBAs and CBRHAs have been trained to provide health education and basic health services and promote the use of formal health services. · Training of HACs and BHTs includes topics to be promoted during community education sessions. The total training period covers 10 and 12 days respectively and includes a 3-day orientation on Home Based Life Saving Skills. A refresher course of 1 week took place in 2002. 485 HAC members and 440 BHT members have received training. · Training of CBRHAs follows the MOH curriculum and includes information on RH, Anatomy, counseling methods for FP, IEC/BCC, MCH, referral and follow up, management of community based RH services and HIV/AIDS education. A total of 114 CBRHAs have been trained. · Training of TBAs in HBLSS (see 1.2) Community education sessions: Committee members and TBAs offer health education at community gatherings (religious, market days, clinic days). Locally adapted IEC materials have been produced and include flipcharts, posters and even t-shirts decorated with health messages. Folk performances are a popular form of entertainment and health education messages have been incorporated into drama performances. Content of community education: · ANC and pregnancy related danger signs · Nutrition · Exclusive Breastfeeding and complimentary feeding · Family Planning · HIV/AIDS education · STIs/STDs education · EPI · Early recognition and response to Fever/Malaria CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 105 · Early recognition and response to dehydration · Early recognition and response to pneumonia · Sanitation BHTs and TTBAs have been very active in creating community awareness about the recognition of danger signs during pregnancy. FGDs with community members (HACs, BHTs, TTBAs) revealed that the community has responded well and is actively promoting better care during pregnancy, childbirth and the neonatal period. The knowledge of danger signs has enabled them to understand the importance of getting proper care for obstetric complications in lieu of traditional medicine. · Members interviewed were able to cite at least 3 danger signs during pregnancy and after delivery. · Community members interviewed state that the education received from SC has empowered them and enabled them to access better care for their mothers and children. It has also provided an opportunity to emphasize the need for FP and prevention of HIV/AIDS and STIs. · Advocacy for safe motherhood is going beyond PA boundaries; communities feel the need to reach out and involve neighboring PAs that are not exposed to SC interventions (Genale). · Partnerships have been established between communities and associated health facility personnel. B. Training of TBAs in HBLSS TBAs receive training on: · Prevention of anemia and tetanus · Clean, safe home deliveries and immediate newborn care · Home monitoring during the postpartum period · Recognition, initial management and appropriate referral of selected complications such as maternal hemorrhage, infection, prolonged or obstructed labor and infant resuscitation. Sixteen health facility workers have been trained as trainers of trainers (TOT) in HBLSS: 11 health assistants, 4 midwives and 1 junior nurse. All health facility staff are natives from the region and are accepted by the community. Training lasts 2-3 days and involves interactive meetings where 1-2 trainers and a group of 8-10 TBAs use written materials and picture cards to discuss MCH care. A review of traditional practices and past experience is followed by teaching on the proper management of specific conditions and enables the group to come to an agreement on how to handle problems. Training involves case studies, role-plays and clinical simulation to enhance learning opportunities. Training is still ongoing and to date, 308 TBAs have been trained on 6 modules. These include the “Introduction to HBLSS”, ‘Women and Baby Problems”, “Referrals”,“Too Much Bleeding”, “Sickness with Pain and Fever” and Baby falls Sick”. Pre and post-tests are conducted during each training session as well as 1 year after training. (Sr Degefech to communicate updated results). CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 106 After acquiring Home Based Life Saving Skills, trained TBAs are encouraged to practice and also train pregnant women and their families in HBLSS. This includes teaching pregnant women and their families to conduct safe home deliveries and to identify the danger signs that need prompt referral. In addition, TBAs promote exclusive breast-feeding and family planning and provide assistance to health facility staff during outreach ANC sessions. Highlights from discussions with TBAs: · Most TBAs interviewed reported attending to an average of 3 deliveries per month. · TBAs were able to mention 2-3 danger signs during pregnancy and 2 signs during the postpartum period. · Knowledge levels about the management of postpartum hemorrhage were also high. “Too Much Bleeding” is the most favored topic. TBAs did not hesitate to simulate how they would act if a woman they were assisting developed PPH. · TBAs have changed their practices, which prior to training included discarding colostrum and asking the mother not to breastfeed for the first 2 to 3 days. · TBAs spend 6-24 hours with the mothers after delivery. When possible, they try to make return visits to make sure both mother and baby are fine. Postnatal examination involved examining the mother to check for excessive bleeding and infection, and the newborn to ensure that breast-feeding was successful. · TBAs are encouraged to go beyond their own PAs to help women in neighboring PAs where women and their families are not aware of the importance of recognizing danger signs and seeking prompt care for obstetric emergencies.(member from Genale) · TBAs are concerned about the lack of gloves to protect them from HIV. Though most women are illiterate, activity records are kept in the form of monthly reports compiled with the assistance of HAC members. Each TBA keeps a picture record of her activities and for each mother she reports on whether: · The mother was educated on HBLSS, referred for ANC or danger signs. · The mother was referred during the first 42 days after delivery for hemorrhage, pain and fever, birth delay or swelling and fits. · The baby was referred during the first 28 days for trouble breathing at birth, small size or sickness. · The pregnancy was complicated by an abortion, stillbirth, maternal or neonatal death. The TBA also reports on the number of children under five that she referred for difficulty breathing, malaria or severe diarrhea. Any problems on cases reported the previous months are included in the report. Areas identified as needing attention included: · Lack of gloves and delivery kits. · Lack of transport. Distances to be covered by TBAs to reach patients are often enormous and transport is a problem especially at night. · Lack of direct supervision of home-based deliveries to ensure that TTBAs perform HBLSS as trained. Have TBAs actually used the skills when faced with complications? Are they referring women with problems as required? There has been a decrease in the CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 107 number of TBA referrals this year compared to the first one (43 versus 319). Is this due to better management of cases by TBAs or is the fall-off due to a loss in knowledge? · TBA reporting relies on recall since most of the women are illiterate. This could lead to incomplete and erroneous reports. · Lack of follow-up of TBA referrals. The lack of human resources has made it difficult to follow systematically on all the cases referred by TBAs. C. Training of health facility staff in Life Saving Skills. Five health workers from Negelle Borana Hospital including 2 Midwife nurses and 3 midwives have been trained in LSS. A 2-week training took place in Ambo Hospital to ensure sufficient cases and provided the midwives with an expanded number of skills for preventing and managing obstetric emergencies and complications. The training curriculum includes 10 modules that cover Antenatal and postnatal care, use of partograph to monitor labor, infection prevention, assessment and management of anemia, pre-eclampsia/ eclampsia, hemorrhage and sepsis and, resuscitation of the newborn. The Medical Director of the hospital agreed to exclude trained midwives from rotating to other areas of the hospital. Trained midwives also act as HBLSS trainers. Midwives interviewed noted an improvement in the management of obstetric patients and particularly in infection prevention. Physicians have not received training in Emergency Obstetric Care and lack the skills necessary to perform obstetric surgery. Limited exposure to obstetric surgery and policy restrictions preventing physicians not trained in EmOC from performing surgery without supervision results in a reticence in handling obstetric complications. Cases tend to be referred the next level, Yirgalem Provincial Hospital, too far to reach for a patient with a real need for emergency care. Health workers at peripheral facilities have been trained as TOT in HBLSS. Providers mentioned feeling inadequate when faced with obstetric complications due to the lack of practice and insufficient client caseload. In-depth evaluation revealed gaps in use of tools such as the partograph to monitor labor. Providers also complained of a lack of on-the-job training. D. Quality of health services Peripheral health facilities (PHF) There are 6 clinics and 3 health posts, all public facilities. One health post is not covered because of security issues. Due to time constraints and distances to be covered, only 4 clinics were visited. Both clinics and health post offer the same services and are staffed by 1-2 health assistants and 1 frontline worker who have been trained for 18 and 12 months respectively. Health assistants have received basic midwifery skills but have not been trained to manage complications of pregnancy and delivery. There is a plan to eradicate training of health assistants who will be replaced and upgraded to nurses. Frontline workers are generally junior nurses who are trained for a period of 12 months. They include junior midwives, junior clinical nurses and junior Public Health nurses. Junior midwives receive introductory training in ANC, labor CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 108 management and postnatal care and could be compared to trained TBAs. Junior PH nurses provide immunization, environmental health and care at outreach sites. The range of services to be offered at peripheral health facilities is very broad and includes both preventive (ANC, immunization and FP) and curative services. Outreach sessions providing the same preventive services are conducted on a monthly basis. (Outreach sessions provide an opportunity to bridge the distance between HF and the community. Sessions are scheduled in advance). Normal routine prenatal visits are offered once a week and include clinical obstetric history, determination of gestational and delivery dates and clinical examination. No laboratory tests are performed to test for syphilis, anemia or HIV and there are no dipsticks to test urine. Patients receive iron and folate tablets, tetanus immunization and nutritional advice. Patients also receive advice on birth preparedness and are given a return date. Hardly any deliveries take place in peripheral health facilities. Statistics show that despite community education, deliveries in health facilities have not increased (Table 2). Heightened community awareness alone might not be sufficient to bring about change in birth practices. A woman in need of emergency care, even if she and her family are able to recognize a danger sign and decide to act on it, still faces a daunting challenge reaching health facilities. None of the facilities reported seeing patients during the immediate postnatal period. During FGDs, community members mentioned that women should not leave the house during the first 40 days. Given distances to be covered, it might not realistic to expect PHF staff to conduct home visits. TBAs encourage patients to attend postnatal family planning and to bring their babies for immunization at health facilities or outreach sessions. Negelle Borana Hospital Negelle Borana Hospital is located in Negelle and is the only hospital in Liben District. It serves a total population of about 1 million (according to the Medical Director). There is a total of 113 beds of which 11 are allocated to obstetric and gynecology. Negelle Borana Hospital serves as the referral center for the 9 clinics and health posts. The next referral level is Yirgalem Provincial Hospital located 275 km away from Negelle- approximately 7 hours by car. In-patient services include normal deliveries, emergency care for women with complicated pregnancies and emergency care for gynecological conditions. Routine and high-risk antenatal care and Family Planning services are provided at outpatient clinics. All inpatients are admitted in the Labor and Delivery ward. Staff includes 3 senior midwives, 2 junior midwives and 1 junior clinical nurse and 5 general practitioners. Senior midwives have received 3 years training that provides midwifery skills and includes the management of complications of pregnancy and deliveries (use of forceps and vacuum extraction), evacuation and curettage and the use of MVA. Antenatal clinics are offered every morning of the week while family with support from physicians when required. Routine prenatal care involves the same services offered at peripheral health facilities with additional capacity to provide urine testing for protein, hemoglobin CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 109 measurement and screening and treatment of syphilis. Voluntary HIV counseling and testing is offered in a separate unit by 1 VCT trained counselor though hardly any pregnant women opt for VCT. Patients do not receive presumptive treatment for malaria even though the setting is endemic for malaria. All in-patients are admitted to the obstetric ward. 1 room is available for deliveries and contains 2 delivery beds. PAC activities such as MVA are conducted in a separate room. Midwives manage cases under medical supervision and support. The partograph is used to monitor labor and to make decisions on obstructed labor but sustainable supply of paper is an issue. SC MCH coordinator is currently providing staff with copies. Equipment is very basic and there are frequent stock outs of supplies such as sterile gloves and drugs. Drugs in the ward are kept in a cupboard in the staff room but are easily accessible. There is no oxygen available in labor room. There is no emergency set of instruments and IV fluids have to be borrowed from the emergency room in case of an emergency. There are 2 forceps and 1 vacuum extractor. The electric vacuum extractor has not been functional for the last 8 months. There is no resuscitation equipment for the newborn. There is a separate operating theatre where all surgical cases are performed. One health assistant has been trained to provide anesthesia. After normal deliveries, patients spend 6- 12 hours before discharge. No follow-up visit is scheduled during the first week. Women are informed about danger signs, and given dates for a six-week post-partum visit for family planning purposes. Table 1: Statistics from Negelle Borana Hospital June 2002 – June 2003 Number % of total deliveries in NH % of expected deliveries in Liben* UN Process Indicators For EmOC Total admissions gynecology (includes abortions) 172 Total admissions obstetrics 364 Total deliveries 375 ? 5.9 % 15 % Normal deliveries 321 88.2 % 5 % Assisted deliveries 32 + 10 11.5 % 0.66% Cesarean sections 10 2.7 % 0.16 % 5- 15 % Total referrals 13 3.6 % IUD 3 Maternal deaths 3 0.8 % < 1 % *Total expected deliveries in Liben District is approximately 6400 = 46.4 /1000 (crude birth rate) * total population. Negelle Borana Hospital is the only facility in the district capable of providing adequate assistance during delivery, however limited. 375 deliveries or less than 2 deliveries per day were reported during the period June 2002 – June 2003. This represents only 5.9 % of total estimated deliveries. Ideally, at least 15 % of total deliveries should take place in EmOC facilities assuming that approximately 15 % of all women will experience a complication during childbirth. Additionally, only 10 cesarean sections were performed or 0.16 % of estimated total CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 110 deliveries. This is below the 5 % minimum level indicating that women in need of emergency care may not be getting it. Drugs, equipment and supplies. · Peripheral health facilities are supposed to serve as the first “referral” points for women referred by TBAs at the community level. In the continuum of health services, these facilities should have the capacity to provide Basic Emergency Obstetric Care (BEOC), which involves the administration of IV fluids for rehydtration, IV or IM antibiotics, anticonvulsants or oxytocics, manual removal of placenta and assisted deliveries (forceps or vacuum extraction). However these facilities have not been designed and are not currently equipped to provide care during delivery let alone manage women presenting with obstetric complications. All 4 facilities visited lacked the basic equipment, drugs and supplies required to assist mothers or neonates during and after normal delivery. None of the peripheral health facilities had a separate room for women in labor. Health workers are therefore forced to interrupt all other activities while they attend to a patient in labor. · Negelle Borana Hospital is the district or referral hospital and should be able to provide comprehensive EmOC. The difference between basic EmOC and comprehensive EmOC lies in the capacity to provide blood transfusions and perform surgery such as a cesarean section. However, NBH also lacks equipment and supplies. In particular during the visit the following observation was made: non functional vacuum extractor, no oxygen, no suction machine, insufficient supplies of gloves, frequent stock out of drugs including antibiotics, sedatives and injectable contraceptives. · Blood transfusion services are not readily available at Negelle Borana Hospital. Type￾and-cross match and blood transfusions are only carried out for patients accompanied by a relative willing and able to donate blood. Blood is tested for HIV. · There is no electricity or running water at peripheral health facilities. Refrigerators containing vaccines run on kerosene. At Negelle Borana Hospital as in the rest of the town, electricity is cut off between 1 a.m. and 7 a.m. The hospital has received a generator but its use has been delayed by the absence of a technician. · Decentralization has increased problems in drug supplies. Peripheral health facilities are currently under the recently established District Health Bureau, while Negelle Borana Hospital is under the direction of the Regional Health Bureau. Separate budgets and different supply systems are used for procurement and peripheral health facilities can no longer obtain drugs and supplies directly from Negelle Borana Hospital which leads to more frequent stock out of most basic drugs. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 111 Cost recovery mechanism · Services at clinics and health posts are free to pregnant women and children under 5, except for medication. (Health care was totally free during the drought and led to an increase in utilization of clinic services including ANC attendance. Health workers have the impression that the slight decrease in ANC is due to the fact that patients shy from having to pay for medication). · Services at Negelle Borana Hospital are offered on a fee for service basis. Women are advised to prepare funds for emergency care but given the overall low-income status of the population; cost could be a deterrent to seeking care. There is a recovery drug fund at Negelle Borana Hospital that is used to purchase drugs that the MOH is not able to provide. The fund is supported by contributions from SC and hospital staff. The community does not contribute to the fund. · Emergency loan funds. Most communities have not established emergency funds for obstetric emergencies. Some PAs interviewed mentioned having established community saving schemes specifically for emergency care. Transportation and referral systems · Lack of transportation remains a critical issue. Pastoral associations are widely dispersed and the distances to be covered to reach healthcare facilities are often enormous. Transport to health facilities remains a challenge for all the communities visited. Most patients need to walk several days before reaching clinics. Roads are in poor condition especially during the rainy season. Patients who reach health facilities might still need to be referred to Negelle Borana Hospital due to the shortage in medical supplies, equipment and infrastructure. Further more, these patients still have to provide their own means of transport to cover the 20 to 96 kms separating them from Negelle incurring further delay. · There are no means of communication between facilities. Less skilled health workers posted at clinics and health posts are incapable of communicating with more skilled staff at Negelle Borana Hospital who could provide advice in some cases. There is no strategy to speed up or facilitate the referral of complicated cases. Patients are merely dispatched to or from NH with a referral letter. No feedback is received from Yirgalem Provincial Hospital and no feedback is sent to peripheral facilities. · There is 1 ambulance at Negelle Borana Hospital available to transport patients referred from Negelle Borana Hospital to Yirgalem Provincial Hospital. The ambulance does not service peripheral health facilities. Record keeping and reporting · At community level Most complications occur in the home and reporting from TBAs and BHTs provides information on events occurring in the community, enabling health workers to conduct more thorough investigations. Monthly reports are compiled by HAC members and include information from: - TBAs activities and referrals - BHTs on community education sessions (number and topics) - CBRHAs on the number of people counseled on FP, RTIs and HIV/AIDS as well as the number of new and repeat users of pills and condoms. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 112 · At peripheral health facilities A registry book is used to record all obstetric cases including ANC attendants, deliveries and complications. There is no separate book for deliveries. Health facility staff sends monthly activity reports to SC Monitoring and Evaluation Coordinator. RH-related information includes: - ANC attendance (1-4 visits), number of high-risk patients identified, ANC referrals for danger signs and number of pregnant women given iron tablets. - Number of deliveries: normal and abnormal, twins and mothers referred during labor. - Number of maternal deaths. - Condition of neonate: number of births, LBW, SB and neonatal deaths. - Number of spontaneous and induced abortions and related problems. - Postnatal care: by first (within 1st 10 days) and repeat visits. - Number of women vaccinated for TT (pregnant and non pregnant) · In Negelle hospital. Monthly report includes number of patients recorded for: - ANC: new and repeat - Delivery: normal and assisted. Indicates whether patient was a TBA referral. - Maternal morbidity due to PID, APH, anemia, malaria, sepsis, abortion, eclampsia and pre-eclampsia. - Maternal deaths due to APH, PPH, obstructed labor, abortion, sepsis and eclampsia. However abnormal deliveries are not reported by type ie: instrumental and cesarean sections. This information is available on a separate report sent to the DHO. In the same report, abortion related deaths were not considered maternal deaths but gynecological complications. Statistics received from health facilities are entered by staff at SC M&E unit. These numbers are incomplete as they do not include deliveries and complications that were not reported ie; some TBAs, clinics and health posts do not submit data are required, some women deliver alone or with untrained TBAs. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 113 Table 2: Summary of maternal health activities Oct 01- Sept 02 Oct 02- Mar 03 Indicator Number % Number % Total deliveries · TBAs · Health facilities · Negelle Hospital 3476 3118 27 331 89.7 0.8 9.5 1445 1267 20 158 87.7 1.4 10.9 Normal deliveries 3024 87.0 1363 94.3 Assisted deliveries · Instrumental · Caeserian sections 111 3.2 (33.5% deliveries in NH) 27 1.9 Referrals for danger signs by · TBAs · Health facilities 319 1 43 9 Maternal deaths · TBAs · Health facilities · Negelle Hospital 19 0 2 2 0 1 ANC attendance Total 10870 3875 First visit 3964 36.5 1707 44.1 Repeat visits 6906 63.5 2168 55.9 TBA referrals 1109 10.2 % of total visits 693 17.9 % of total visits TT immunizations 2865 26.4 % of total visits 1171 30.3 % of total visits PNC attendance None recorded None recorded Management and supervision According to staff in peripheral health facilities, supervision is deficient both in quantity (approximately twice a year rather than quarterly as scheduled) and quality: more attention is paid to clinic performances and results rather than the level of skills to achieve these results. Chronic problems in supplies and logistics lead to low staff moral and disincentive. All providers indicated a need for better feedback and more supportive supervision. Staff at Negelle Borana Hospital complained about the lack of guidelines to assist them in managing obstetric complications. Furthermore, there is no standard protocol for auditing maternal deaths. The senior midwife interviewed mentioned regular meetings to discuss obstetric cases and complications (did not see reports). Case reviews could help staff identify problems in the system that need to be addressed and set standards against which the quality of emergency obstetric care in the Hospital can be improved. SC has designed a health service supervision checklist to help in assessing the activities and needs of health facilities. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 114 V. Discussion How effective has community mobilization been in creating awareness about pregnancy related issues? How much support has been generated at community level for pregnant women and infants? CBWs seemed knowledgeable about danger signs during pregnancy and childbirth. However discussions were limited to the community members who had received HBLSS training and a survey is required to evaluate transfer of knowledge from BHTs and TBAs to pregnant women and their families. In addition, community education on danger signs will not improve pregnancy outcomes unless women in need of care are able to reach health facilities in a timely manner and get appropriate care. Communities have not implemented birth preparedness and complication readiness which requires that every pregnant woman has an emergency plan that would include identifying the place of birth, emergency funds, emergency transport and a blood donor. Communities are now aware of when to seek care but have not yet organized themselves on how to access care. Each PA should attempt to establish an emergency loan fund and lessons learned from successful PAs could be used to help other PAs set up their own emergency loan funds. (During a FGD, members mentioned the lack of stretchers to transport patients to health facilities. An attempt to get them to manufacture local stretchers beforehand has been met with fierce opposition. One TBA had several homemade stretchers destroyed. Cultural barriers still exist and need to be addressed.). Ideally, each PA should have a transportation plan. HAC members should mobilize local resources to create sustainable emergency transportation systems. One community described how they had forced someone driving by to carry a woman with prolonged labor to the hospital. Has TBA training improved the management of cases at household level and is there any documentation to that effect? In other terms, are HBLSS applied in the household when necessary? Are women and infants in need of emergency care being referred if measures to manage the complication in the household are insufficient? In the first year of CS-17 (October 2001- September 2002), TBAs reported 1109 referrals for ANC and 319 referrals for danger signs. This year (October 2002 – March 2003), referrals for ANC have remained steady at 693 but only 43 women were referred with danger signs. Records show that most reported deliveries are being carried out with the assistance of trained TBAs (TTBAs). In the first year of project cycle, 3118 deliveries (89.7%) were reported by TBAs, compared to 358 deliveries at health facilities (including Negelle Borana Hospital). This year, 1267 or 87.7% of reported deliveries are attributed to TTBAs. In addition, there has been a decrease in the number of maternal deaths reported by TTBAs: 2 the last 6 months compared to 19 the first year of CS-17. Several reasons could be responsible for the observed trend including improved management of complications in the home. During FGD, community members mentioned that community education and TBA training had led to a notable decrease in maternal deaths. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 115 Training TBAs in HBLSS has been a key component of the program. Post training assessment includes a posttest immediately after training and after a period of 1 year to assess knowledge retention. (Results to be communicated by Sr Degefech). It is important to assess whether training has resulted in better TBA practice. Unfortunately, baseline information on TBA practice is not available. It is equally important to verify that TTBAs are performing procedures correctly since most of the reported deliveries are currently assisted by trained TTBAs. TTBAs provide activity monthly reports. Unfortunately these reports do not include details on the case management, useful in documenting whether trained TBAs are using HBLSS the skills. Due to limited human resources, there has been no systematic follow-up of TBA activities. TBA reports should be reviewed and feedback should be given to TBAs. There is no system to verify if women referred by TTBAs actually made it to the next level and what reasons might have prevented them from getting there. What is the quality of care provided to pregnant women at each level: peripheral health facilities (clinics, health posts) and at Negelle Borana Hospital in terms of Essential Obstetric Care and Emergency Obstetric Care? Given the distance to Negelle Borana Hospital, both Essential Obstetric services and basic EmOC should be available at peripheral health facilities. However, none of the peripheral facilities visited currently has the capacity to provide adequate essential obstetric services. Theoritically, health facilities offer 24-hour coverage but in practice hardly any deliveries take place there. Women do not perceive any value added by traveling to a clinic or health post to deliver as none of them is designed to provide adequate assistance during labor and delivery, let alone to handle obstetric complications. Current efforts to educate communities about when and where to seek care and mobilizing communities to improve access to health facilities will not be successful unless services are available at the nearest point of contact. Health workers posted at peripheral health facilities have received basic training and are least likely to have the skills to handle obstetric emergencies. They are however the first point of referral for the majority of women needing emergency care. They need to be trained in LSS skills with ongoing training to maintain the level of competence. Peripheral health facilities need to be upgraded and a regular supply of drugs and equipment maintained to provide proper obstetric care. They also need to be backed by a functioning referral system should it be necessary to refer a patient. At present, peripheral health facilities function as triage centers where patients with complications are registered and sent on to the next level. Service delivery guidelines state that health workers should detect and refer complications immediately to the next level (Negelle Hospital) after minimal stabilization. However, no referral plans have been designed to enable women to access Negelle Borana Hospital. Negelle Borana Hospital is the only facility in the district with the capacity to provide basic EmOC. The quality of care leaves room for improvement as essential drugs for the management of obstetric complications are not readily available and staff has to deal with recurrent power CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 116 shortages. Moreover, mothers needing a cesarean section should not have to travel 7 hours to Yirgalem Provincial Hospital. Comprehensive EmOC should be available at Negelle Borana Hospital. Currently, only 1 physician is able to perform cesarean sections and blood transfusion services are extremely limited. Provision of quality care is also hampered by the fact that only midwives have received LSS training, physicians still need to be trained in emergency obstetric care. Furthermore, the only practitioner able to perform cesarean sections is not supported by policy, as GPs are restricted from performing obstetric surgery unless a specialist is able to supervise them or they have received training in EmOC. A proposal to fund training for a couple of physicians has been submitted. On a long-term basis, staff needs to be supported in its activities. This could be achieved by assigning a specialist to provide supportive supervision and guidance on a regular basis. An audit of cases; fatal and near misses should be conducted with the help of an expert so that protocols can be adapted and new guidelines established to improve clinic practice. VI. Recommendations Community mobilization about safe motherhood has generated a sense of ownership and commitment to saving women’s lives in Liben District. Community resources should be tapped to set up emergency plans for enabling women with obstetric complications to reach health facilities. Per se TBA training will not bring a sustainable reduction in maternal mortality unless backed by an effective referral system which includes health facilities equipped and staffed to provide life saving services. Suggestions to improve the referral system include: · Advocate for stronger commitment from MOH to prioritize SM strategies that will enable reduction of maternal morbidity and mortality. · Continue support and supervision of TBAs. TBAs attend most of the reported deliveries and should be given the necessary tools and support such as clean gloves and delivery kits. Ensure better supervision and follow-up of TBA activities to determine whether TBAs are perfoming HBLSS steps correctly and in order to evaluate whether TBA training has led to better case management at household level. · Encourage the transfer of knowledge and skills among TTBAs. ( Sr Degefech has noted that most experienced TTBAs are old and are not very good at teaching their younger more knowledgeable but less experienced counterparts who are hardly called for assistance by the community. She recommends funding the training of a TBA supporter for each PA to facilitate the work of TBAs at community level.) · Identify and implement local solutions for emergency transportation of women from their homes to health facilities. Assist communities in establishing emergency loan funds and developing transport systems. · Equip health workers at peripheral health facilities with the skills (midwifery skills acquired by competency based training and reinforced by ongoing training), equipment and supplies to provide essential obstetric care. · Establish referral links between facilities and Negelle Borana Hospital. · Establish an operational transport system linking facilities to Negelle Borana Hospital and to the Provincial Hospital. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 117 · Ensure Negelle Hospital is able to provide EmOC 24 hours/day, 7 days/week, not only basic EmOC but also comprehensive EmOC. A technician should be trained to run the generator to prevent power shortages. A doctor capable of performing obstetric surgery should be available at all times. Efforts should be made to provide blood transfusion services accessible to all patients needing blood. Sufficient supplies of medical equipment and consumables should be maintained. · Train physicians in Emergency Obstetric Care. To ensure sustainability, this should take place during medical training and therefore suggestions should be made for a revision of curriculum to adapt to current need of more skilled GPs in underserved areas. Medical doctors should be given more exposure to obstetric training during the internship period. · To improve the quality of services provided, clear guidelines should be made available and their implementation should be subject to regular audit. Areas needing particular attention include: ÿ TT immunization ÿ Presumptive malaria treatment ÿ Screening and treatment for Syphilis ÿ VCT for HIV ÿ Management of obstetric complications ÿ Postnatal follow-up · Most maternal deaths occur in the postpartum period: half of all maternal deaths occur in the first 24 hours and 70 % within the first week. New approaches should be piloted to ensure that women get a postnatal visit within the first 3 day after delivery. · An effort to standardize all reports has been made which will facilitate data analysis. Data should be utilized in planning program activities and feedback should be provided to health workers who should be encouraged to make the necessary changes. · Ensure more effective systems of supervision, monitoring, follow-up and evaluation to enable better evaluation of project impact. · A quality assurance team could be established in the hospital with key staff from all wards that could work together on improving standard of care by addressing crosscutting problems such as the patient registration, records, infection prevention, drug supplies, transport (how the ambulance can be used more efficiently), etc… · Evaluation should also include qualitative data from the community on the quality of care including exit interviews for clinic clients, questionnaires for women on discharge from maternity wards and focus group discussions with community members. · Maternal deaths should be investigated as soon as they occur to ensure the complete and accurate record of information while details are still fresh in the memories of all involved. · Provide extra support to the MCH coordinator. · For women who have to travel very long distances and where transport is not available, a maternity waiting room or home could be the solution. The maternity home or “tukul” established in Central Ethiopia near Attat Hospital has had good results and could be used as a model for replication in Liben District. CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 118 Annexes a) List of instruments used to collect data/information · HHCC Gap Analysis tool · Supervision Checklist for Health Center (The Design and Evaluation of Maternal Mortality Programs – Appendix A) · Supervision Checklist for Hospital (The Design and Evaluation of Maternal Mortality Programs – Appendix A) · Facility Functioning Assessment Form (The Design and Evaluation of Maternal Mortality Programs – Appendix A) · Interview Questions for the TBA, Lay Midwife or Community MCH worker (Postpartum and Newborn Care: A Self-study Manual – PRIME 1999) b) Testimonials “Before SC started helping us, our mothers were dying and we thought it was a curse and that nothing could be done. Now we have learnt about danger signs and we know when to look for help” – HAC member from Genale. “We have started going across the river to help the people who are not getting help from SC. We teach them about danger signs and our TBAs go there to help” - community member from Genale. c) List of persons interviewed: facilities – health workers. 1. Negelle Hospital · Dr Taye Tolera, Medical Director, Negelle · Sr Lelise Tadesse (Senior Nurse Midwife- MOH) · Sr Mezeret (Senior Nurse Midwife- MOH) 2. Harakelo Clinic · Sr Adunya (SPA- SC) · Sr Tigist (MOH frontline worker) 3. Genale Clinic · Sr Mestawot (SPA-SC) · Sr Buche (MOH) 4. Jidallo Clinic · Sr Kenene (SPA- SC) · Sr Bekele (MOH) 5. Mugayo Clinic · Sr Zenabe (SPA-SC) · Sr Sewalem (MOH) CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 119 d) The following people were instrumental in providing information and data about the program in Liben District: Mr. Alemayehu Boka Liben Impact Area Manager Mr. Adamu Beyene Monitoring and Evaluation Coordinator, SC/Liben Sr Degefech H/Yesus, MCH Coordinator, SC/Liben Sr Melkenesh Ketema FP Unit Head, SC/Liben Mr. Mohamed Mamu HIV/AIDS Unit Head, SC/Liben Mr. Worku Tefera Training Coordinator, SC/Liben Mr. Solomon Teseme Health Sector Manager, SC/Liben Dr. Taye Tolero Medical Director, Negelle Hospital e) Overview of health facilities visited in July 2003. Health Facility Tot. beds MCH beds Dist. to NH Maternal health services Number of staff and categ. No of staff trained in LSS Signal functions Negelle Hospital 113 11 ANC Normal and assisted deliveries (C/S - vacuum and forceps) MVA PAC FP VCT 5 midwives 1 junior nurse 5 MDs 5 midwives 6 BEOC and occasionally cesarean sections and blood transfusions. Not available 24/7 (Electrical power cuts) Harakelo Clinic None None 35km ANC (Deliveries) (PNC) FP 2 Health Assistants 1 frontline worker None Parenteral antibiotics (if available) Genale Clinic None None 54km ANC (Deliveries) (PNC) FP 2 HA 1 community midwife None Parenteral antibiotics (if available) Jidolla Clinic None None 64km ANC (Deliveries) (PNC) FP 2 HA None Parenteral antibiotics (if available) Mugayo Clinic None None 31km ANC (Deliveries) (PNC) FP 1 HA 1 junior nurse None Parenteral antibiotics (if available) Due to time constraints, the following facilities were not visted: Bulbul Clinic (60 km), Melka Guba Clinic (98 km), Algei Health Post (90km), Hadhessa Health Post, Miessa Health Post (30 km- security issues). CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 120 f) Training of facility and community health workers. LSS trained health workers: Name Sex Qualification HBLSS TOT S/r Lelise Tadesse F Midwife nurse Y S/r Meseret Eshete F Midwife nurse Y Umer awuku M Midwife Y Zehara Hussen F Midwife Y Emebet Tadesse F Midwife Y Total trained 5 Health facility workers trained as TOT for HBLSS: Qualification Number of trainees Total M F Health Assistants 8 3 11 Mid-wives 2 2 4 Junior Nurses 1 - 1 Total 11 5 16 Total Trained CHWs under each health facility catchment areas: N Health facility HAC BHT TBA CBRHA Total 1 Negelle Hospital 142 127 100 30 399 2 Genale Clinic 36 34 22 7 99 3 Harakallo Clinic 141 138 88 37 404 4 Jidola Clinic 47 40 26 14 127 5 Mugayo Clinic 24 21 16 7 68 6 Bulbul Clinic 24 22 13 6 65 7 Melka guba Clinic 12 6 7 1 26 8 Miessa health post 12 12 7 2 33 9 Alge health post 23 14 15 6 58 10 Hadhessa health post 24 26 14 4 68 Total trained 485 440 308 114 1347 g) The Household to Hospital Continuum of Care (DRAFT based on available literature) Community Level Pregnancy-ANC · Community education on safe motherhood for women, their families and decision￾makers · Nutritional support including iron and folate supplementation, balanced diet. · Support and encourages rest, personal hygiene · Encourages ANC at health facility or with trained TBA · Encourages maternal immunization with TT · Recognition of danger signs and seeking appropriate care CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 121 · Supports birth planning and preparedness (emergency funds for transport and cost of care, transportation, identifies blood donor) · Supports and encourages seeking health care for treatment of malaria, syphilis and HIV where appropriate · Supports breastfeeding with appropriate counseling Labor and Delivery (L&D) · Trained TBAs or other community birth attendant (CBA) · Community education on safe motherhood and danger signs for women, their families and decision-makers · Safe management of labor and delivery by TBA or CBA · Observe hygienic practices (Hand washing by TBAs or CBA, Clean delivery, clean delivery surface, clean cutting instrument for the cord, clean hands) · Recognition of danger signs, recognition of onset of complication during L&D · Prompt action and referral · First aid treatment where appropriate (largely for cases of hemorrhage) including hydration with oral fluids, initiate early suckling, encourage passing of urine, external bi￾manual compression where appropriate. · Proper use of oral or injectable oxytocics. Postpartum visit by TBA or CBA within 6-24 hours and 3 days · Immediate and exclusive breastfeeding · Checking for fever and foul smell, (indication of possible onset of infection), refer for treatment where appropriate. · Attention to the health Periphery level health facilities Pregnancy- ANC: · Nutritional support, including Iron and folate supplementation, (vitamin A and iodine in areas with deficiencies), · Check weight, test urine for protein, check gestational age and fetal heart beat, · Detect and manage pregnancy complications, · Maternal immunization with tetanus toxoid (TT) · Health information on self-care · Presumptive treatment of malaria where appropriate · Syphilis screening and treatment where appropriate · Voluntary counseling and testing for HIV where appropriate · Encourages birth planning or preparedness (items needed for birth, identifies blood donor, emergency funds and transport and cost of care) · Supports breastfeeding with appropriate counseling CS-17, Ethiopia Mid-Term Evaluation, Save the Children, October 2003 122 Labor and Delivery (services could be limited by staff skills and prevailing in-country policies) · Skilled attendants at labor and delivery · Clean delivery · Safe management of labor, with oxytocin, hydration or IV fluids where appropriate · Recognition of danger signs · Services for basic emergency obstetric care (BEOC): Basic emergency obstetric care include s: 1. Administer parenteral antibiotics 2. Administer parenteral oxytocics 3. Administer parenteral anticonvulsants/sedatives for eclampsia/pre-eclampsia 4. Perform manual removal of placenta 5. Perform removal of retained products e.g. Manual vacuum aspiration 6. Perform assisted vaginal deliveries Postpartum care at six weeks · Family Planning counseling · Breastfeeding support to maintain exclusive breastfeeding District Hospital Pregnancy- ANC and postpartum care at 6 weeks: Provides the same services as peripheral health facilities. Labor and Delivery (L&D) In addition to services available at peripheral health facilities, has additional capacity to provide CEOC, which includes: Comprehensive emergency obstetric care (CEOC) includes: 1. Administer parenteral antibiotics 2. Administer parenteral oxytocics 3. Administer parenteral anticonvulsants/sedatives for eclampsia/pre-eclampsia 4. Perform manual removal of placenta 5. Perform removal of retained products e.g. Manual vacuum aspiration 6. Perform assisted vaginal deliveries 7. Perform surgery i.e. cesarean section 8. Perform blood transfusion All must have: · Skilled staff to manage obstetric complications, · Available essential equipment, revolving drug fund and supplies · Functioning Blood bank, blood screening and transfusion services · Available surgery facilities: operating theatre, anesthesiologist (doctor/ nurse/paramedical staff) anesthetic equipment