WOriLD VISION ETHIOPIA CS XI1 Final Evaluation Report S henkolia Child Survival Project Grant #FAO-A-00-95-00024-00 World Vision, Xnc. World Vision CS XI1 Final Evaluation Report Shenkolla Child Survival Project Grant #FAO-A-00-95-00024-00 Beginning Date: October 1, 1995 Ending Date: November 30,2001 Final Evaluation: August-November 2001 Submitted to: Child Survival Grant Program USAIDIBHWPVCICS Ronald Reagan Building, 7t" Floor 1300 Pennsylvania Avenue, NW Washington, DC 20523 PVO Headquarters Contact: Thomas Hall, MPH World Vision, Inc. 220 I street, NE, Suite 270 Washington, DC 20002 Ph: (202) 547-3743 Fax: (202) 543-4834 Email: tomhall@worldvision,org Table of Contents A .................................................................................... . Summary 1 B ......................................................................... . Results and Impact 3 B . 1 Results Summary Table ........................................................ 3 B.2. Results: Technical Approach ................................................... 5 B.2.a. Overview of the Project ............................................. 5 B.2.b. Progress by Intervention.. ........................................... 5 ARI .................................................................. 7 EPX ...................................................................... 7 Maternal and Newborn Care ........................................ 9 Family Planning ....................................................... 10 Capacity Building ...................................................... 10 Lessons Learned ........................................................ 11 B.2.c. New Tools and Approaches ............................................ 12 B . 2 . d. Outreach Strategies ...................................................... 13 B.3. Results: Cross-cutting Approaches ............................................. 13 B . 3 .a. Community Mobilization ............................................. 13 B.2.b. Behavior Changes ...................................................... 14 . . B.3.c. Capacity Buildmg ....................................................... 15 B.3.d. Sustainability Strategies .............................................. 21 Lessons Learned ........................................................ 22 ................................................................... C . Program Management 23 C.1. Planning ........................................................................... 23 C.2. Staff Training. ..................................................................... 23 . . C.3. Supennsloiz ....................................................................... 23 C.4. Human Resources and Staff Management ..................................... 24 C.5. Financial Management ............................................................ 25 C.6. Logistics .............................................................................. 25 C.7. ......................................................... Information Management 25 C.8. Teclmical and Administrative S~~pport ........................................... 26 D . Other Issues Identified by the Team ...................................................... 27 E . Conclusions, Lessons Learned and Recommendations ................................ 28 F . Highlights ................................................................................... 33 TabIes Table 1: Indicators Table 2: IHF A ARI Indicators Table 3: IWA EPI Indicators Table 4: Supplies for CHWs Table 5: Major Activities Carried Out by CHWs Table 6: CSP Material and Medical Support Table 7: Training, Experience Sharing and IEC Activities Table 8: CSP's Staff Turnover Table 9; Availability of Records and Forms Table 10. Technical and Administrative Support Attachments Attachment A: Shenkolla Evaluation Team Attachment B: Evaluation Methodology Attachment C: List of Persons Interviewed List of Acronyms ADP ANC ARI BASICS CBD CBRHA CHA CHW CS CSP DIP EECMY EPlI FGM FP HC HF HIS HIVIAIDS HMIS IACT IEC IHFA IMCI KPC MC MCH MOH NGO N?D PA PHC RDF SCSP SWR TB A TT US AID WCBA w WVE Area Development Program Antenatal Care Acute Respiratory Infection Basic Support for Institutionalizing Child Survival Community Based Distributors Community Based Reproductive Health Agent Community Health Agent Comlnunity Health Worker Child Survival Child Survival Project Detailed Implementation Plan Ethiopian Evangelical Church Mekane Yesus Expanded Program on Immunizations Female Genital Mutilation Family Planning Health Center Health Facility Health Information System Human Immuno-deficiency VindAcquired Immune Deficiency Syndrome Health Management Information System Integrated Approach to Childhood Illness Information, Education and Communication Integrated Health Facility Assessment Integrated Management of Childhood Illness Knowledge, Practice and Coverage Maternal Care Maternal and Child Care Ministry of Health Non-governmental Organization National Immunization Day Peasant Association Primary Health Care Revolving Drug Fund Shenkolla Child Survival Project's Southern Nations and Nationalities Peoples Region Traditional Birth Attendant Tetanus Toxoid United States Agency for International Development Women of Child Bearing Age World Vision World Vision Ethiopia A. Summary This report presents the findings of the Shenkolla Child Survival Project's (SCSP) Final Evaluation conducted in the project area between August - November, 2001. It included: Integrated Health Facility Assessment (MFA); Knowledge, Practice and Coverage (WC) Survey, and a qualitative assessment of the results achieved from the SCPS's activities and programs carried out during the past three and a half years. The SCSP was a collaborative effort between Ministry of Health Ethiopia (MOH) and World Vision (WV) to provide sustainable, high quality health services to a beneficiary population of 117,470, including almost 50,000 children under5 years and women of reproduction age, The project was initiated in the Shenkolla Area Development Program (ADP) January 1,1998 and was completed November 30,2001. The Shenkolla ADP is located in the Soro Woreda (District), one of the four woredas in the Hadiya Zone in southern Ethiopia and is approximately 260 km. south of Addis Ababa, The SCSP covered 28 peasant associations (PAS). The SCSP's major activities focused on: acute respiratory infection (ARI), immunization, maternal and newborn care, family planning, revolving drug fund and capacity building. Its two inter-related goals were to: 1) accelerate the reduction of under-five and maternal mortality and morbidity and decrease the total fertility rate, and 2) build the capacity of partners - including the government, comnities and WVIEthiopia and other non￾governmental organizations (NGOs) - to provide quality, sustainable health services. The general approach of the project was to strengthen the technical, managerial, and financial capacity of government health services and the community to provide sustainable health care services. The project's major accomplishments include: the establishment of a highly successful revolving drug hnd (RDF) that has resulted in a readily available drug supply in the project area and beyond; increase in the contraceptive prevalence rate from 6% to 36 % due to trained community based reproductive health distributors (CBWs) who conducted house to house visits, distributed contraceptives and carried out health education in the communities; noticeably improved acute respiratory tract infection (ART) case management; the training of health staff in ARI, integrated management of childhood illnesses (IMCI), supervision and development of information, education and communication (IEC) materials; the involvement of church leaders and partners in the planning and implementation of project activities so that sustainability of some of the major intervention is now assured; and integration of activities with those of the Shenkolla ADP. Results from the final HFA showed that 100% or all eligible health facilities in the project area had essential drugs and equipment available during the assessment to provide expanded program for immunizations (EPI) and IMCI case management services, compared to zero for the baseline, that all had an established RDF scheme in place and that all health personnel had been trained in ART case management compared to zero trained for the baseline. The proportion of children with ARJ who received antibiotics rose from 18% to 89% and for whom two or more assessment tasks were performed fkom 16% to 73%. According to the KPC survey, the project met two out of three ARI objectives. One of the three objectives could not be measured as it was omitted from the survey. EPI results as measured by the KPC survey were less impressive. One of the three objectives could not be measured because the sample was all women of reproductive age who had received TT2, and the project did not meet its other two objectives, although significant improvements were made in raising immunization coverage rates. Data from the IHFA in relation to EPI indicated that health workers did improve their perfomance in relation to EPI activities. Many more children and mothers had their vaccination cards checked during sick child visits and the majority of health workers had correct knowledge of EPI calendar. Possible reasons for not achieving higher immunization coverage rates include: lost or misplaced cards, limited outreach activities, poor supervision and logistical problems. The project's only maternal health objective, other than family planning, was met, however, it had no impact on increasing the proportion of women who were assisted during their last delivery by a trained attendant. Practices surrounding childbirth are still steeped in local traditions, transport to health facilities, other than walking, is virtually non-existent, and the quality of health care services, especially delivery services remains very poor. These are probably some of the reasons why women prefer the assistance of a relative during labor rather than a health worker at a health facility. The project met all its sustainability and capacity building objectives. During its relatively short time line - less than four years - the SCSP was able to make considerable progress in achieving its two inter-related goals to reduce under-five and maternal morbidity and mortality and total fertility rates; and to build the capacity of partners to provide quality, sustainable health services. The project adapted innovative ideas to develop new initiatives, made concerted efforts to integrate its activities with those of the MOH and the Shenkolla ADP, explored available avenues in a timely fashion to sustain activities beyond the life of the program, utilized and strengthened existing resources to increase access and availability of health care services, included cornunity members in the delivery of health care services, Iimited its scope to that what was doable and feasible, md partnered very successfully with well established organizations in the project area. World Vision's strategy to establish and maintain ADPs in areas of great need compliments the overalI effort of a CSP. When fully integrated, the two programs can greatly benefit from each other, not only in terms of sharing its resources and expertise, but also in assisting each other with the attainment of goals, objectives and long-term sustainability endeavors. Many lessons were learned by WVEthiopia from its first experience with such integration, foremost of which is the need to have lines of authority and managerial issues clearly articulated and fol~owed, and supported by the powers at be. If done carefully, the integration of a CSP with an ADP can become the new paradigm for community based health activities for World Vision. B. Assessment of Results and Impact The findings below reflect the data collected from the three evaluation methodologies: an IHFA, the KPC survey, and qualitative assessment of program activities as described by the partners, communities and the staff. The MFA was designed to collect information on the quality of case management of the common causes of child morbidity and mortality. Information was also collected on the status of the health facility with respect its organization, availability of services in relation to maternal and child health, drugs, supplies and equipment, and the status of and support for health facility staff. The IHFA was conducted fiom 12-17 August 2001 in all of the five health facilities in the SCSP area. The KPC survey was conducted from 27 August to 6 September 2001 with a randomly selected sample of 300 mothers with children less than 24 months of age living in the targeted 28 peasant associations of Soro Woreda, It used a questionnaire originally designed by the Johns Hopkins University, School of Public Health in Baltimore, MD USA, that has been updated and modified to meet the needs of individual projects. The last part of the evaluation process was a qualitative assessment made of the program, activities, strategies and methodologies used by the project through a review of reports and other documents, and as perceived by the partners, the communities and the staff. For this evaluation exercise, 7 of the 28 PAS were randomly selected for field visits. In each PA, mothers, youth, PA members, and church elders were interviewed, using a semi-structured interview guide. In addition, health workers in 3 of the 5 health fxilities were interviewed, as were the supervisors, and persons responsible for the revolving drug fund (KDF). In total, more than 700 people were contacted or interviewed during the course of this evaluation. The table below indicates the extent to which SCSP met its primary objectives as described in the Detailed Implementation Plan (DIP). B.1, Results: Summary Table Table 1: Indicators Indicator 1. ARI 1.1 % of mothers able to give 3 key refmal signs for AN in children < 24 months (not a KPC survey question) 1.2 # of health workers involved in patient care received appropriate training for ART case management as measured by district supervisory report Project 20% (PI 1.3 # of health facilities with necessary drugs for / all I NA I 5/5 ARI case management in adequate quantity for 90% of the last 2 quarters of the year the next two years or who are not sure, who will 1 I I I 2. Family Planning 2.1 % of women who do not want another child in use a modern contrace~tive method I 1 I 1 12% 3. EPI 3.1 % of children 12-23 months hlly immunized 6% 3.2 % of mothers who received 2 doses of TT know that measles immunization should be given at 9 months (Not a KPC survey question) 36% 80% - mothers in focus W'JP discussions 60% NA 3.4 % of mothers with children < 24 months who 1 60% NA --- 7% 3% 15% NA All ~ 37% 4. ANC 4.1 % of mothers who had at least 2 ANC visits during last pregnancy 5. Capacity Building 5.1 # of health facilities in the SCSP area that assess the quality of ART, case management using methods and standards developed by BASICS 5.2 # of PA Development Committees that have a functioning health sub-committee with 25% female membership 5.3 # of church elders and pastors in PAS who have been brought together for sensitization 5.4 # of health stations and health center given 80% necessary medical supplies 5.5 # of health institutions with RDF scheme 6. Sustainability 6.1 % of health institutions with essential drugs and equipment to render EPI and IMCI case management services on a regdar basis 6.2 % of health institutions with regular programs for supervision/support 6.3 % of PAS with f~mctioning TBAs, CBDs, and all four * health facilities 100% 100 3 CHAs 6.4 % of PAS with health sub-committees 83% 4 NA NA NA regularly functioning 6.5 % of health institutions with a successfd 1 revolving hg scheme 84% none 0 0 0 NA 5/5 28/28 158 5 0 0 0 0 1 6.6 % of community health workers, clinics and NA 0 100% NA 4 515 100% 415 80% 85% 0 100% 0 80% health centers with functioning recording and reporting systems 6.7 % of TBAs and CHAs who continue to carry out their service for at least one year after taking I ORS and appropriate antibiotic use. NA 0 85% refresher course 6.8 % of beneficiaries with access to health services defined as within three hours walk 6.9 % of PAS with drug vendor sources trained in a One health facility - Kosha was constructed/supplied after the baseline survey. NA - Not Available 85% B. 2. Results: Technical Approach B.2.a. Overview of the Project This CSP was originally designed and fimded to take place in Eritrea. However, due to political changes in Eritrea, the government asked WVEritrea to leave the country by October 30, 1997 soon after the start of the project. As both the United States Agency for International Development (USAID) and WV desired to continue the cooperative agreement, a change in CSP site was approved. The new program description and budget supported CSP efforts for 33 months. Thus the SCSP was initiated in early 1998 in Ethiopia and was to end September 30,2000, but a no-cost one-year extension was granted, which extended the project to September 30,2001. The dead line for the cornpietion of the project was extended to November 30", 2001, because of the September 1 I attacks in the United States. NA The Shenkolla Child Survival project was a collaborative effort between the Ministry of Health of Ethiopia and World Vision to provide sustainable, high quality health services to a beneficiary population of 117,470, including 50,000 children under five and women of child-bearing age (WCBA). The project was located in one of the four woredas of the Hadiya Zone, Southern Nations and Nationalities Peoples Region (SNNPR), approximately 260 KMs south of Addis Ababa. It originally was intended to cover 20 PAS in Soro Woreda. However, after the preparation of the DIP, it was deemed feasible to expand the number of PAS to 28. The SCSP included interventions in ARI, immunization, maternal and newborn care, family planning, revolving drug hd and capacity building. Its two interrelated goals were: To accelerate the reduction of under five and maternalmorbidity and mortality and decrease the total fertility rate; To build the capacity of partners and to provide quality, sustainable health 85% NA 84% services, The general approach of the project was to strengthen the technical, managerial, financial and material capacity of government health providers and the community. 0 B.2.b. Progress by intervention The results of the program must be viewed in light of the unusual situation the project found itself in as a result of its abrupt departure from Eritrea and the thrust necessary to place it within the Shenkolla Area DeveIoprnent Program (ADP) in Ethiopia. Although the ADP manager was able to travel to the United States and attend a training workshop to learn about CSPs, the relationships between an ADP and a CSP were not well understood by the parties involved and this resulted in high turn-over of CSP staff, Nevertheless, the results achieved by the SCSP are impressive. The evaluation team made great efforts to understand the root elements that can be attributed to these achievements, both in terms of quantifiable and qualifiable data. There is no doubt that the placement of the SCSP within the well-established Shenkolla ADP was of great value and that the integration of the two programs has benefited both programs. The Shenkolla ADP has worked in the Soro Woreda for over ten years, is trusted by the authorities and the communities, has developed an infrastructure, and has a history of accomplishments. As the two programs were integrated, SCSP was able to conduct activities with very few staff and few 1ogisticaI resources. The ADP shared its office space and equipment, provided housing for staff, and supported CSP activities. In turn, the ADP has benefited fiom SCSPYs rigorous evaluation, monitoring and implementation methodologies. The SCSP worked very closely with the health authorities and personnel both at the zonal and woreda levels, included their staff in all training sessions and in the planning and implementation of new ventures, such as the revolving drug scheme (RDF). Therefore, the project did not create any parallel structures, nor developed new categories of health care providers that could not be absorbed into the existing health care infra-structure. Another contributing factor related to the success of the project was that the activities were relatively limited in scope and the area limited in size, making the activities doable with few resources. The choice of PAS was based on the availability of a health facility within 3 hours walk for the population. PAS without ready access, that is within a 3 hour walk to a health facility, were not considered for inclusion in the program. It became evident during the focus group discussion that the population's need had been addressed, their knowledge had increased and that they were willing and able to change behavior and practices. Probably the most influential conduit for this change was the work of the community health workers (CHWs) and in particular the community based reproductive health agents (CBWAs) who conducted house-to-house consultation, showed respect for the couples, and provided contraceptive supplies individually. By conducting regular information, education and communication (IEC) sessions, knowledge of beneficial health practices increased at the same time that drugs became readily available at health facilities through the RDF scheme. Thus trust was generated though the reinforcement of activities at different levels of program implementation. trash is buried." Acute Respiratory Illness (ARI) ~llhealth facility staff was trained in treatment of ARI, as part of an integrated management of childhood illnesses. Care-takers were taught to identify dangers signs of AN in their children and seek prompt treatment at a health facility. CBRWAs and Community agents (CA) were trained to recognize the danger signs of pneumonia in children and refer them to a health facility. A drama group was formed and trained as a conduit for health education messages. Through the RDF, health facilities had a regular stock of appropriate antibiotics available at health facilities. The project met all ARI objectives that could be measured. Inadvertently, a question on mothers' knowledge of 3 key referral signs for Am in children was left off the KPC, and could, therefore, not be determined. Nevertheless, mothers were asked during focus goup discussions to identify danger signs in their children and when there is need for referral. Mothers' knowledge was universally high. They mentioned sunken fontanel, sunken eyes, rapid breathing, chest in-drawing, dry skin, lack of appetite and listlessness. The IHFA revealed the following concerning AlU: Table 2: IHFA AXU Indicators I last 12 months 1 antibiotics 3. Proportion of health workers who see children and have received IMCI training in Expanded Program on Immunizations (EPI) Final Data 73% Indicator 1, Proportion of children with AM for whom two or more assessment tasks were performed 2. Proportion of ARI cases who received Training of MOH health facility staff on EPI. Training of MOR health facility staff on cold chain management. Printing and distribution of material and child health reporting and recording materials, including EPI cards. Health education at health facility and community levels on importance of immunization; including the six vaccine preventable diseases and tetanus toxoid (TT) for pregnant women and other WCBAs, age of measles immunization, and importance of retaining immunization card. Training of CHWs, CHAs, CBRHAs and TBAs- to provide health education on EPI. Provision of financial and logistic support for outreach activities, including supervision. Mobilization of communities to attend outreach sessions. Baseline Data 16% 18% 75% 89% 100% Provision of staff, logistical support and finances for National Immunization Days (NfD). r Training of health and development workers on IEC message and material preparation, using the "P" process as developed by the Johns Hopkins University - Center for Communication Program. Purchase of refiigerators to support the cold chain at MOH facilities. The IHFA showed the following: Table 3. IHFA EPI Indicators Indicator 1 Baseline Data I Final Data 1 1. Proportion of children who had their vaccination card checked during sick child 1 3% 54% visit 2. Proportion of children eligible for 1 vaccination who had never been vaccinated ( NA 25% 1 3. Proportion of mothers who had their 1 I I I vaccination card checked during sick child ( 0% / 4. Proportion of mothers who had never I 1 I knowledge of EPI calendar 6. Proportion of health facilities with up-to￾date immunization and patient registers received TT vaccination 5. Proportion of health workers with correct Among the five health facilities surveyed, one did not provide immunization services, while only two out of the remaining fow carried out an integrated EPI program, that is that immunization was daily available. The other two facilities provided EPI twice a week. Among the surveyed health facilities, four out of five had a working refrigerator, but only two out of the four working refrigerators had their temperature recorded for the past 30 days. All health facilities with working refkigerators had stocks of vaccines. Although the project was not able to meet its EPI objectives, considerable improvements were made. The reasons for not achieving higher immunization coverage rates include: Iost or locked up immunization cards, lack of defaulters follow-up policies and information systems, limited outreach activities, poor supervision, lack of transport, impassible roads during the rainy season, insufficient numbers of health staff and poor health worker performance. As will be described later in the report, adequate supervision of health workers is lacking at all levels, standards of performance are not measured and appropriate supervisory tools were not available until the last few months of project operation. NA 85% Maternal and Newborn Care Training of health facility staff in maternal and newborn care. Health education at the health facility and community levels on recognizing danger signs during pregnancy and delivery, having deliveries attended by a trained health worker or traditional birth attendant (TBA), nutrition d~~ring pregnancy, TT vaccination & harmful traditional practices like female genital mntilation (FGM). Training of TBAs to provide safe deliveries and to refer hig-risk cases to the health facility. Workshops for community leaders and women group representatives on maternal and child health (MCH). Provision of financial and logistic support for the MOH to conduct outreach sessions for immunizing children and WCBA. Mobilizing communities to attend o~~t reach sessions. Printing and distributing child and maternal health recording and reporting material to the MOH. Provision of iron Ifolate to RDF stock. The project met its only MCH objective, but had little impact on increasing the percentage of deliveries attended by a trained provider. The vast majority of mothers prefer to deliver at home with the assistance of a family member (46% for the final versus 63% for the baseline survey) or with the help from an untrained TBA (38% for the final versus 27% for the baseline). Trained TBAs attended 21% of the birth according to the final KPC. 46% of the mothers knew two or more dangers of pregnancy, compared to 26% for the baseline. The reasons for home delivery preferences are steeped in the local traditions and culture, and are also influenced by the quality of the services provided in the health centers. Many of the health centers are inadequately maintained, lack essential supplies, equipment and staff and in general do not provide a mother friendly atmosphere. There is virtually no public transport in the project area except along the main road leading to the Zonal capital. The evaluation team saw sick people being carried to and from the Jajura clinic on locally made stretchers held on the shoulders of four men; this being the only means of "ambulance" available for the vast majority of the population. Most of the so-called roads in the area are passable during the dry season by a four-wheel drive vehicle only and dwing the rainy season these roads become impassable. The most common other means for transport is a horse, donkey or mule. Any obstetrical emergency is referred to the Zonal hospital in Hosamah. According to the IHFA, the five health facilities in the project area averaged 22 deliveries per year with a range from 0-1 11, Of these, 12 women were referred to the next higher level health facility. "We now receive all sorts of services." Family Planning The major family planning activities included: Health education at the health facility and community level on the benefits of spacing child-births and delaying first pregnancy. The efforts included video shows at schools and other community gatherings. Training of CHWs on family planning. Providing CBRHAs with oral contraceptives, condoms and foaming tablets for distribution at the community level. Conducting sensitization workshops for community leaders and church elders. House-to-house cotmeling by CBMAs and individualized distribution of contraceptives. Collaboration with church leaders and elders to diffuse FP messages. Collaboration with Jajura Clinic in the facilitation of contraceptives. Remunerating CBRHAs for their efforts. Regular, although poor, supervision of CHWs. The FP component of the SCSP can be considered as one of the most enduring and successfil activities of the project. Data from the KPC surveys shows an increase in modem contraceptive prevalence from 6% to 36%. This extraordinary increase was verified through focus group discussions. Mothers, church elders and other community members reported the increased use of modern contraceptives because there is a strong desire to limit the number of children per family. The reasons given were: lack of adequate land for farming and food insecurity, improved mothers' health, children were no longer dying, and better family relationships. Communities credited CBRHAs for much of this success. CBRHAs through their IEC and house-to-house efforts have been able to change behaviors, while making contraceptives readily available, The Ethiopian Evangelical Church Mekane Yesus (EECMY) will continue to provide support for the CBMAs with the assistance of Pathfinder. EECMY will supervise the CBWAs, provide contraceptives, and pay them 60 birr each month when they come for restocking. A memorandum of agreement with the church has been signed to sustain the FP activities. The IHFA revealed that 415 health facilities in the project area had FP services, and that the average number of new acceptors was 88 during the past year. This number ranged fiorn 8-257. Each facility surveyed had an average of 9 CBRHAs responsible for family planning. All health facilities had adequate stocks of oral contraceptives, although stock control cards were not available in any of the clinics. Capacity Building Project activities included: Training of MOH staffs and community-level health workers in IMCI, supervision, development of educational materials, and RDF. Providing transport, financial support, and medical supply in the health facilities. Establishing RDF at 13 MOH facilities. Creating awareness about child survival and building support among church elders and community leaders. Facilitating the selection of CHWs and health sub-committees in each of the project PAS. The project met or exceeded all its capacity building objectives, although the PAS are not carrying out their functions and the health cornmitt ees are ineffective. PAS are political in nature and their turn-over is very high. They axe difficult to supervise or monitor, although they do express the desire for further training. Capacity building without a concurrent well-developed supervisory structure in place is of questionable value, as was shown during discussions with health workers and CHWs. The FTE team was shown a daily clinic attendance record, during an interview with one of the health workers, in one of the busier clinics. All six chddren that had been recorded during the previous day, had been diagnosed with pneumonia. When we questioned the health worker, she stated that she had been off that day and that her replacement fkequently completes the daily tally sheet by filling it in with one diagnosis. We questioned the supervisor of the clinic who was not aware of the problem, but promised to look into it. Other evidence that supervision is not working comes from the records kept by one of the trained TBAs. She proudly showed us her book, but when questioned about some of the data she had recorded, it was found that she had listed the number of birth as the number of infant who had died. Some of the health workers did improve their performance in relation to IMCI in health facilities and as observed through the HPA, but without adequate supervision it is doubtful that these gains can be sustained. CHWs appear to reap greater benefits from the capacity building activities, as discussed in previous sections. A mother, "Our children are no longer sick and dying". Lessons Learned - Shenkolla CSP 1. When given the opportunity religious leaders can be used to greatly enhance the diffusion and acceptance of health messages and practices. 2. When a CSP engages people with strong spiritual beliefs and commitments, the efforts of the project may be enhanced. 3. The provision of family planning services through the privacy of house-to-house visits, greatly improved family planning acceptance by the couples. 4. Training CHWs in the use of multiple channels of communications and approaches to service delivery, enhances attitude and behavior change. 5. It is better for a project to focus its activities in a manageable area ~mtil there is some evidence that: improvements have been made in key health indicators before attempting to expand into new areas. 6. Capacity building can not be accomplished through training alone, but must go hand in hand with policies and an infrastruct~m that support quality supervision. B.2.c. New tools/ approaches Revolving Drug Fund In Ethiopia, government health facilities are often in short supply of drugs, particularly in rural areas. Because of budgetary constraints, health facilities frequently consume their entire annual budget for drugs in a few months. As a consequence, wrong usage of dmgs and self-medication are prevalent, and unauthorized dispensing of drugs a common practice. Furthermore, the income from dispensing drugs in government health facilities is paid back into the central Treasury, and hence the refinancing of drug supplies is difficult. To address these problems, a revolving drug fimd PF) was established by an Africare CSP in the mid nineties in Butajira, a town 120 kms, from Gimbichu, during a NGO's phase out process to ensure the continued availability of essential drugs. This RDF began with little financing, but was miraculously sustained over the years with little outside support. The SCSP manager knew about the scheme and arranged for an experience￾sharing visit to learn about what did and did not work. Participants involved in the visit included staff fi-om the Woreda Council, health, education and finance offices, two community members, a member fi-om the Zonal health department and the SCSP manager. Upon return, the group recognized that such a scheme was doable and feasible and decided to establish a RDF scheme in Gimbichu. They formed a RDF committee with nine members, established written bylaws and initiated the scheme. A sub-committee - the technical committee made up of members of the main committee -was also formed. Members of the main committee include: the Chairperson - Woreda Council Administrative Officer, Vice Chairperson - Ministry of Education Officer, Secretary - Woreda Health Officer, Accountant - Woreda Finance Officer, Auditor - Chairperson Women's Affairs, and members: two community representatives, SCSP Manager, and head Gimbichi health center. The technical committee, which is made up of the head of the health center, the Woreda health office head and the education officer, meets every month, while the main committee meets every three months. All the drugs are centrally controlled by the Gimbichi RDF and distributed according to requisition. The chief of the Gimbichi health center, who is also the RDF's store-keeper makes the requisition for drugs. The requisition is then sehd to the Zonal health pharmacist who has the only authority to purchase the drugs fiom the central supplies. The SCSP and the governmental health authorities have worked closely together from the initiation of the RDF scheme. The health office provided one building block with three rooms at Gimbichu health center, and is paying the salary of the drug dispenser, while the SCSP renovated the building, supplied all the initial necessary drugs, supplies and materials to begin operation and supported the mming cost of the scheme for one year. Supervision and distribution of supplies is shared between the SCSP and woreda health authorities. Within a few months of the initiation of the RDF, it was expanded to two other government health facilities and to a third one soon thereafter. Expansion to another facility occurred about a year after the RDF' initiation because many dmgs in the store had rapidly approaching expiration dates. The RDF authorities, in consultation with the Woreda health powers that be, have decided to expand the RDF scheme even krther. Other health facilities in the Woreda will be able to obtain drugs on a credit basis. Health facilities will be provided with essential drugs on credit, repay the Gimbichu RFD when drugs are sold, and retain the profits with the hope that each health facility will eventually become self-sustaining. Findings from focus group discussion support IHFAs' findings that essential drugs are now readily and consistently available at health facilities and that the RDF scheme is functioning well in the Soro Woreda. The majority of mothers and other community members reported that they could now obtain drugs from the health centers, whereas before there was a chronic shortage. Antibiotics to treat ARI were available in all health facilities at the time of the IHFA. A side effect from the RDF is that it has put several plastic bag practitioners and established pharmacies out of practice in Gimbichu, because now there is a reliable drug source and the cost is cheaper than that of private pharmacies and the plastic bag doctors. The pharmacy sells the drugs at a 25% profit margin. Success can be attributed to the following: highly dedicated staff at the Gimbichi central RFD office, adequate supervision, relatively well functioning main and sub-committees, clear auditing and check writing policies, cheap and reliable supply of drugs, and high demand by the community. B.2.d. Outreach Strategies The project supported the following: Fuel, logistical and per-diem support for health workers Capacity building for social mobilization. Supervision of CHWs during outreach activities in SCSP sites. Health education during outreach sessions. For further information on outreach strategies see section below. B.3. Results: Cross- cutting approaches B.3.a. Community Mobilization The project's community mobilization efforts were built on the experiences and corninunity relations established by the Shenkollah ADP. Initially, community mobilization focused on the selection and training of CHWs to serve as links between communities, the health facilities and the project. Three different types of CHWs (12 CHAs, 40 CBRHAs, and 29 TBAs) received training to equip them with new knowledge and skills to serve communities. In addition, already existing, trained CHWs in the project area (I I CHAs and 22 TBAs) received a refresher training to reinforce their ability to function effectively, The project was responsible for mobilizing communities to select the CHWs and to establish health sub-committees, with technical guidance from the woreda health office and CSP staff. Communities participated actively in the selection process and as described earlier, have responded positively to the services provided by the CHWs. Demand for services from the CHWs by the commtmities resulted in the selection and training of more CHWs than originally intended. Because awareness was raised as a result of the CHW's IEC activities, demand for modem FP services increased. In response, the CSP in collaboration with the woreda and zonal health authorities introduced the house-to-house delivery of contraceptives and counseling through the CBRHAs. With the introduction of this service, the number of clients for each CBRH increased to about 6 to 18 clients. As mentioned earlier, CBRHAs have been able to impact FP prevalence in the project area. Other areas where impact of the CHWs activities was also noted include: improved latrines, household and personal cleanliness and other hygienic practices. Except for an one day daily allowance given to the CBRHAs when they come for monthly review meetings, CHWs are not remunerated and plans to sustain their motivation have not been formulated by the PAS. Although CHWs maintain that they will continue to function now that funding for CSP is no longer available, unless the ADP can continue to reach them, their services will most likely slack offgradually. All PAS interviewed stated that they wouId be willing to devise schemes to support the work of the CHWs, such as helping them during harvest time and collecting monetary contributions. The ADP or health facility supervisors must support these efforts, as it appears unlikely that the PAS are politically motivated to devise remuneration mechanisms on their own. Youtb To get a better understanding of youth's knowledge and behavior in relation to HIVIAIDS for the development of htue WE activities, the FTE developed a short focus group discussion guide for young people between the ages of 15 and 21 years of age. One hundred and two males and thirty females ages 15 to 30 participated in the discussions. In general, the youth had very good understanding of HIVIAIDS transmission and ways to protect themselves. Most of their information came fiom within their own groups, radios, CRWs, churches, schools and printed information. They recognized their own vulnerability, but were reluctant to discuss the reasons for this. They were more comfortable in discussing their own roles in limiting the spread of HIV, including their influence in providing information on HIVIAIDS and to mobilize their own peer groups on ways to avoid transmission. The youth unanimously declared that information and education was not directed at their age group and that they had few opportunities to discuss the topic in appropriate places. They mentioned the need for AIDS groups in schools and churches, testing sites in proximity to the villages, and educational materials targeted for youths. B.2.b. Behavior Changes IEC activities were carried in health facilities, at the community level and in churches, The final evaluation team (FET) did not observe any IEC session, but both qualitative and quantitative data suggest that the messages are heard and that behavior is changing, The CSP facilitated the training of 43 health facility, partner and project staff in the preparation of messages and materiaIs. This training resulted in the production of flyers and posters depicting relevant health messages. A facilitator experienced in the "P" process as developed by the Johns Hopkins University, Center for Comunication Programs conducted the ten-day training during which time they were given theoretical training. After the ten day training the trainees went to their project areas for one month to develop different kinds of IEC messages (posters, flyers, flip charts, drama, billboards, etc.) pretested the product in the community, incorporated the pretested materials into the message designed and came back for a three day presentation session to complete the training. During the three days presentation, all the participants commented on each others work and feedback was provided by the trainer. Most of the materials prepared are in use by the project. Many of the posters developed by the project can now be found in health centers in the woreda. At the community level, CHWs delivered IEC messages d~uing social and religious gatherings and house-to-house visits. The CWWs also used coffee ceremonies to elicit discussions on the different health messages. The coffee ceremony is a social occasion throughout Ethiopia where from 4-6 families (or more) gather every morning or evening to drink coffee and chat, It usually lasts more than one how in the morning and often extends Lo more than two hours during the evening and weekends. It takes place in different houses on a rotation basis and it is during this time that the villagers discuss issues and problems informally. Although, the FET was not able to review the content of the messages as depicted in the posters, mothers and other community members demonstrated good knowledge of danger signs in children, appropriate age for measles immunization, advantages of child spacing, dangerous traditional practices and reasons to take sick children to a health facility. The most remarkable behavior change centered around farnily planning practices, as mentioned earlier, Mothers and other community members also reported that they would no longer allow their girl children to go through FGM procedures, citing the dangers involved in those practices both related to the spread of HIV/AIDs and difficulties during pregnancy and child birth. Another possible behavior change relates to the common practice of polygamy in the area, Through the focus group discussions, it was learned that IEC on HIV/ADS has increased knowledge of HIV transmission, including the dangers of having multiple sexual partners. Many of the men interviewed, stated that they now have only one wife and that this was now the norm for most men. Mothers verified this finding and commented that in addition to the danger of HIV transmission, families could no longer support large families and multiple wives. Father's request "Please give us information about anthrax!" B.3.c. Capacity Building Approach Strengthening the PVO Organization No objectives were set in the DIP related to capacity building of the PVO. Nevertheless, the organizational capacity of the field office staff was enhanced through several training and information sharing activities. WVE staff attended training on conducting KPC and IHFA surveys. Information was shared on the RDF before the scheme was initiated in the project area. Staff was trained in the development and production of IEC materials, and they also participated in the training for supervisors and in the use of the BASICS tool and in IMCI training. Senior WVE staff took the summer quality assurance course at the Johns Hopkins Bloomberg School of Public Health in Baltimore. It is difficult to assess how the training affected PVO performance, since project staff turn-over was high and persistent and no assessment was made of staff training needs. However, there is ample evidence at the national PVO headquarters, regional office in Awasa and at the ADP site, that training for the surveys, the development of the DIP, the CSP's reporting and evaluating mechanism, the focus on quality interventions and "state of the art" technical approach has had a positive effect on other PVO activities. Other PVO projects are now required to carry out baseline and final surveys, health programs are now better integrated into the overall activities of the PVO, duplication of efforts are minimized and the importance of managerial effectiveness is stressed. The SCSP received numerous visits from WV/Washington staff to provide technical assistance and support for program activities. Strengthening Local Partner Organizations WVE has partnered with the Soro Woreda Health Office, the Zonal Health Department, Jajura Catholic Clinic, EECMY, Pathfinder International, BASICS and the communities. The five health facilities within the CSP area and the communities in the 28 PAS, have been the prime focus of attention in partnership and capacity building activities. The roles and responsibilities of each partner are as follows: Soro Woreda Health Office, Zonal Health Department and health facilities - Health services are provided through MOH facilities and outreach sessions. The project facilitated this work by building the capacity of the MOR to carry out the targeted interventions. The Woreda Health Office, together with Zonal Health Department is responsible for conducting health workers training and supervision. The health workers in the facilities are responsible for the supervision of the CHWs. The health information system (HIS) is also managed by the MOH. A Memorandum of Understanding was signed with the MOH that outlined their collaboration in the project. Communities - The roles and responsibilities of the communities were channeled through CHWs, the health sub-committees, church elders and community leaders. Along with the Zonal and Woreda health authorities and personnel, communities were the SCSP's primary partners, Each community was responsible for the selection of its CHWs and for establishing a health subcommittee in each PA within the project's target area. The selected and trained CHWs were responsible for providing IEC, mobilizing communities to participate in project activities and attend outreach sessions, referring patients, providing FP services, assisting with deliveries, and collecting and reporting data. The roles and responsibilities of the health sub-committees was never clearly defined, nor were they adequately trained to assume their outlined activities. On the other hand, the trained church elders have played a significant role in diffusing essential messages, in creating awareness and in sensitizing the communities. Their facilitating role was largely untapped until approached by the project to assist in mobilizing communities. Table 4: Supplies for CHWs I Item Recipient Total Delivery Kits Umbrellas Bicycles Hand Watch Walk Man Tape Recorder Radio TBAs CHWs Selected CHWs T Shirts and Caps GloveslBox 30 128 19 Selected CHWs Selected CHWs Selected CHWs Table 5: Major Activities Carried Out by CHWs 9 3 6 CHWs and Community Leaders TBAs Home Jajura Catholic Clinic - This clinic, nm by the Catholic mission, is involved in providing primary health care (PHC), except for the provision of modern contraceptives, The clinic has allowed the CBRHAs and CHAs to provide health education on family planning and other MCH topics in the clinic. The clinic also collaborated on the promotion of natural birth spacing as an acceptable form of family planning. Supervision of the CHWs in the clinic catchment area is done by CSP staff, rather than by the clinic or MOH staff. 3000 10 Activity CHW Remarks IEC CHAs Deliveries Family Planning EECMY - The SCSP collaborated with EECMY in the training of CBRHAs, making use of EECMY's experienced trainers. EECMY provided its CBRHAs with a substantial monthly transport allowance, which made it difficult for the SCSP to retain its CBRHAs' motivation. Recognizing that such an allowance may have been a mistake, EECMY will gradually wean the CBRHAs to accept a reasonable allowance until such time that the allowance can be substituted by reimbursement through social marketing. EECMY has agreed to phase in the SCSP area and to engage some of SCSP's trained CBMAs. They will provide their monthly stipend and contraceptive supplies. This agreement will ensure that the FP activities of the CBRHAs will be sustained well beyond the life of the project, Pathfinder International supported the efforts of EECMY. WVE has also agreed to support them with an one-time training of 100 CBRHAs to assist EECMY with their expanded coverage area. TBAs CBMAs TBAs Accomplishment CBRHAs Female 13564 4822 2400 Pills Condoms ppppp Male 7900 5007 3164 Total 21464 9829 5564 1622 New 1380 45 Repeats 6845 2 1 Total 8225 66 Strengthening Health Facilities The SCSP supported all. of the five health facilities in the project area, through a variety of programs. Health facility staff was trained in IMCI, RDF, supervision and IEC material development. Drugs were more readily available through the RDF. The project reinforced health facility staffs supervisory activities through financial and logistical support. The Gimbichu RDF office was furnished and supplied with project funds and the Kosha health center was also refi~rbished. An IHFA survey was camed out both at the beginning and end of the project that included all health facilities in the SCSP catchment area. The objectives of the surveys were to: assess the current knowledge and practices of health workers at out patient clinics regarding the assessment and management of sick women of child bearing age and children; identi@ baniers to quality case management for sick children; evaluate the adequacy of training and supervision of health workers; assess the availability of essential drugs, equipment and supplies; and to make relevant recommendations. In relation to the first objective to assess the current knowledge and practices of health workers in relation to their sick child and WCBA assessment skills, the proportion of children screened for severe ihesses rose from 2% to 6%, not an encouraging improvement. None of the children's nutritional status was assessed, a result similar to the baseline. Only 2% of the mothers were asked all key history questions about their child. There did appear to be a discemable increase in ARX and diarrhea task assessments. For AEU this rose from 16% to 73% and for diarrhea from 20% to 40%. In general, the IHFAs showed that despite training, health workers are not performing to standards at any of the health facilities and the quality of their work is below expectation. The IHFA revealed the following barriers to quality case management: shortage of health personnel, shortage of laboratory technicians, and inadequate supervisory guidelines and technique. All health facilities had at least one antibiotic in stock, however, it can not be discerned from the survey data which antibiotic was available. The FET reviewed antibiotic stock necessary to treat AN and found that it was available in three of the five facilities visited. One hundred percent of the health facilities experienced at least one stock out of drugs in the previous month. The most common stock-out drug was tetracycline eye ointment, and the most common stock-out of supplies were needles and syringes. In relation to the adequacy of training and supervision, data revealed that health workers continue to practice below standards and that whatever supervision is provided does not address the issue of quality performance or performance to standards. There is no doubt the recent trainings have positively affected health workers performance, but unless the whole supervisory system is overhauled and supported, quality services will not be provided by the current crop of health workers. Among the sick children seen in all health facilities, 44% of them were seen at the Jajma health clinic. The main reasons why Jajura is the magnet clinic are: consistent availability of services and medicines, attitude of the health care providers, hygienic environment of the clinic, and the availability of comprehensive health care services. I Out patient Fonnats 4000 4000 --- 8000 TT Cards 2000 1000 1000 1000 1000 6000 Table 6: CSP Material and Medical Support Support Motor Bike Generator Tape Recorder Outreach Activities Each health facility serves an average of 9 PAS (kebeles) and a population of 24,694. Major obstacles to outreach activities include: poor road conditions especially during the rainy season, long distances from the health facility, mountainous terrain, and seasonal impassable rivers. The majority of health facilities conduct outreach activities at least once a month. They use CHWs and megaphones to mobilize the communities and to provide health education, 1 EFT Cards FP Cards See the attached Report on the IHFA for further information. ---- 1 1 4 1 1 1 5 Mule I 1 Gimbichu HC 1. Strengthening Health Workers Performance The SCSP's role in strengthening health workers performance has mainly focused on training and providing material and financial support. All health workers currently staffing the health facilities have received at least one training activity through the project and several have participated in several training sessions. As mentioned earlier, the project was not directly involved in supervision, nor had the staff available to improve the quality of services provided. The project did facilitate the introduction of a supervisory tool developed by BASICS, but training on the use of the tool was not conducted until August of this year. During focus group discussions with supervisors, it was revealed that the tool is usehl, that supervisors are using it: at the zonal and woreda levels and that some feed-back to the supervisee is provided. It will be important to determine how these recent developments wiIl translate into improved performance in the future. 1000 800 Kosha HC 1 1 1000 Segie HC 1 1 1000 I 1000 1 Homaro HC 1 1 1000 Jajura HC 1 5000 800 .I Total 2 1 5 At ZonaI Level Supervision consists of a number of tools used to check off different items on the lists. The SCPS introduced the BASICS IMCI supervisory tool to supervisors only four months ago and they are just beginning to see the value in the tool. Feedback is provided on the spot verbally and in writing. At Woreda Level Woreda level supervisors were also recently trained using the BASICS' IMCI supervisory tool and appear to use the tool when supervision is possible. Verbal feedback is provided arid written feedback is given within four weeks after the supervisory visit. At health center level Supervision at the health centers is in most instances non-existent. Even at the health center level, supervisors did not have supervisory tools, did not have schedules, and in general did not carry out supervisory activities with their staff. At the community level The supervisors siinply look at the CHWs' reports, make comments and, at times, attempt to verify the information from other sources. No tool is used during these visits. Nevertheless, the CHWs commented that they are motivated by the supervisory visits. Constraints At all levels the following constraints were noted: shortages of staff, lack of transport, lack of finds for gasoline, and bad roads. Table 7: Training, Experience Sharing and XEC Activities Activity/Gxoup Health Staff Training ARI Facilitators FP Facilitators EPI Facilitators Health Sewice Management Training MCI AN for Health Workers IEC Training Planned 1 Accomplished 2 6 17 33 11 4 2 11 5 13 2 5 - 43 1 I 3 / ~ocal 1 8 11 ( Community Training 54 CHAs - New 6 P 5 4 12 Refresher training for CHAs Refkesher training for CH Ws Sensitization for Community Leaders Sensitization for Church Leaders 35 TBAs - New Refresher training for TBAs 35 128 248 100 Review Meetings IEC Drama Shows * Includes staff from other ADPs and NGOs 132 252 278 6 102 Video Shows Mothers' Day Posters Bill Boards Pamphlets B.3.d. Sustainability Strategies The key elements of the SCSP sustainability strategy as outlined in the DIP were: Training and transfer of skills to community level health worker (CBRHAs, CHAs and TBAs), who are selected by communities them selves. 12 7 8 12 56 - Promoting active involvement of other community groups, such as church leaders, mothers' group and community development groups. Organizing and developing counter part problem solving and decision-making teams, in particular, at the woreda, zonal and regional health levels. Providing initial resources for a revolving drug fund system. Strengthening other development initiatives in the area, including water provision, income generating activities, and infrastructure development (roads and 8 47 96 5 14 1100 2 0 5000 communication). 4 9 1100 10 5000 The key indicators for measuring progress towards meeting sustainability objectives, and actual progress achieved to date are: All health facilities have adequate supplies of vaccines and drugs for the management of ARI, diarrhea and malaria. All health facilities receive monthly supervisory visits by the woreda and CSP staff, but none of the clinics has an established supervisory program. 84.3% of the C13W7s are actively engaged in community programs in the project area. All PAS have established a health sub-committee and they have been oriented on their responsibilities. Three health facilities have established their own RDF ('Kosha, Sege and Humaro) on their own initiation from the Gimbichu RDF that was established by SCSP. The catholic clinic at Jajura has an established cost sharing system of it own. The RDF has been extended to nine health facilities outside the project area in the woreda, There is a plan to replicate to other woredas and to make Gimichu health center a distributor to other woredas as well. The RDF at Gimbichu is working in consortium with the government pharmacy. All CHWs and health facilities have a functioning recording and reporting systems. 53.6% beneficiaries have access to health services defined as within three hours walk. 5 of 6 drug venders trained in ORS, appropriate antibiotic use and FP. Though family planning is not part of their intervention, the Jajura Catholic clinic stores modern contraceptives for pick up by CHWs and allows CHAs and CBRHAs health education on family planing and other related topics. ECCMY and Pathfinder will continue the support CBRHAs' FP activities. The Shenkolla WV ADP has committed funds to support outreach activities to maintain the gains achieved, such as EPI and supervision. Lessons Learned 1. Clear understanding of roles and responsibilities of each party is essential to ensure successful collaboration in an RDF venture. 2. An in-coming project should carefully identify small but undeveloped key activities that have the potential for scaling up and strengthening. 3. A relatively short-term project cannot expect to reform a supervisory system because such an effort would require a major "quality assurance" paradigm shift. 4. It is essential for future CSP and ADP integration that lines of authority as well as roles and responsibility are clearly documented and agreed upon during the planning phase of a project, C. Program Management C.1. Planning The heads of the Soro Woreda Health Office and the Hadiya Zone Health Department, the Shenkolla ADP manager, WVE's health coordinator, and WV headquarter staff, jointly planned the project objectives and intervention strategies. The preparation of the DIP involved a wider group. WVE obtained technical assistance from WV Uganda for the training on the KPC survey. Three people from the Hadiya Zone health department, three fiom the Soro Woreda health office, two from the Gimbichu health center, one fiom the Shenkolla ADP, two from WVE headquarters, and one each from Seghie and Humaro health clinics participated in the training. Twenty people from the community also participated in the data collection. The two HFAs training and data collection activities involved staff from health centers, woreda and zonal representatives, ADP staff, CSP staff and WVE headquarter staff. An outside consultant with knowledge of EPInfo, carried out data analysis. Trainers were trained in IHFA through Save the Children USA, CSP at Negellie Borena, during the preparation of the DIP. Information from the KPC and the initial IKEA were used to revise objectives, strategies and interventions in the preparation of the DIP. Health authorities at the woreda and zonal levels complained that in spite of numerous requests to obtain a copy of the DIP, the requests were never granted. They stated that they, therefore, did not have a clear idea of the project's objectives and strategies, and that they were informed about activities after they had been planned. C.2. Staff Training The SCSP devoted more than adequate resources to training activities. Project staff attended a quality assurance summer course in the USA, two WVE staff attended CSP management training in the USA, staff participated in training courses conducted for and with partners, and staff organized courses in areas to update their knowledge and skills, such as the KPC and IHFA surveys, IMCI, supervision and the preparation of IEC materials. Unfortunately, some of these capacities were lost due to very high staff turnover. C.3. Supervision Since the SCSP is incorporated into the existing ADP management structure, the ADP and South regional development office management staff (the latter based in Addis Ababa.) have direct involvement in managing and supervising the CSP. Technical and supervisory support is also obtained fiom the Grant Division and the health coordination unit of the country head quarters' office. Under this structure, the responsibility for providing supervision, technical and managerial support to the project fiom the country office is diffused. Except for the technical support provided by the health coordination unit, including intermittent project visits by the technical health staff to fill the gaps created by the rapid turn over of project personnel, problem-solving support &om the country office was inadequate. The serious conflicts between the ADP and the CSP managers were not resolved in a timely fashion to avoid what appeared to be spill over to the rest of the CSP staff and before it became apparent to the partners. The CSP manger regularly supervised the IEC and MCH coordinators at the field level, but due to the frequent turn over of the CSP managers, the process of directing and supporting the project staff was impaired. C.4. Human Resources and staff management The high turnover problem among the project staff was noted during the mid-term evaluation and continued though out the life of the CSP. The only stable position in the CSP was that of the MCH coordinator. The frequent turnover of key personnel, coupled with the long time in finding replacements, caused delays in the implementation and interruption of activities. Availability of better job alternatives, the short program duration, and the remote program site that does not include provisions for the family of staff, were some of the reasons for fiequent turnover. But, according to headquarter staff and MCH coordinator, the major reason why the CSP managers refused to remain with the project was unresolved conflicts between the roles and responsibilities of the CSP staff vis ir vis the ADP staff and in particular, the ADP manager. Reasons for this conflict vary according to the source of information, one side blaming the autocratic attitude of the ADP manager, while others cite lack of understanding on the part of the CSP managers of how an ADP works. Whatever the reason, headquarter staff should have been more proactive and should have resolved these issues early on. Both the needs of the CSP and those of the GDP should have been considered during conflict resolution. However, according to a former CSP manager, their problems were not taken into consideration by headquarter staff, and they were never asked to explain their side of the story. Morale, therefore, was low for CSP staff in relation to the ADP manager, but high when they actively engaged in comrn~mity mobilization and health center activities. EventualIy, the ADP manager took over the direct role of managing the project and supervising project staff. The integration of the CSP in to the ADP has helped the program to be more cost￾effective, through the sharing of offices, administration, and financial management, Secretarial services, and logistics (warehouse, purchasing, lodging). Table 8: CSP's Staff Turnover Responsibility All project staff was employed through short-term contracts and the one remaining staff member found a job before the project terminated. CSP Manager IEC coordinator Driver C.5. Financial Management The integration of the project into the ADP administrative structure meant that the CSP finance management came under the ADP manager's authority. This included the approval of the budget and expenses and accessing needs for supplies from the warehouse. Although there were many advantages to this arrangement, it also created undo difficulties for the CSP managers since they did not have the authority to dispense funds necessary to implement activities. Rather all financial decision had to be approved by the ADP manager, a person who was not versed with the technical aspects of CSP activities. The primary mechanism established to ensure financial sustainability of the program is through the RDF scheme as previoudy described. The ADP manager has committed a budget for the continuation of some of CSP's activities, such as support for supervisory,, RDF and outreach activities to maintain the gains made through the project. 2 1 1 2 1 - - - - For further infomation on the ItDF see section B.2.c. The total budget expenditure as of mid August is US$302,346. At the end of the project the field had $22,092 remaining in unused, budgeted funds, and the total remaining unused funds, including headquarters, are $134,343. Even though the remaining amount seems quite large the project encompassed many countries initially, and was eventually transplanted to Ethiopia from Eritrea where it originally was awarded. This as well as the staff turnover made spending down difficult, even with the extensions it was granted. Moreover, some activities were taken up by other NGOs such as training conducted by John Snow, which were originally budgeted in the CSP. C. 6. Logistics The CSP had the use of one old vehicle that frequently broke down and was not well suited for the extremely difficult terrain it had to navigate. The ADP manager controlled the use of vehicles, as well as the procurement and distribution of equipment and supplies, such as motorbikes, TBA delivery kits, tape recorders, and IEC materials. Despite its advantages in cost-effectiveness, the combined management of logistics for all the ADP projects caused delays in licensing of the motorbikes and the procurement of materials for the CSP. Transport shortages delayed the implementation of a number of activities and caused a great deal of frustration on the part of project staff and its partners. C.7. Information Management. The CSP used MOH data collection foms for all interventions, rather than developing a parallel system. Additional tools (developed by BASICS), such as the supervisor checklist and the supervisory visit feedback formats (register) were introduced by the project to facilitate supervision and improve on the quality of services. MOH staff was trained in the use of these tools earlier this year, thus it was too earlier to assess their effectiveness or their appropriate usages, Data collection, compilations and reporting forms were available in all health facilities surveyed by the IHFA. This is a considerable improvement compared to the baseline IHFA survey, when it was noted that stock of essential drug cards and weekly notifiable disease forms were lacking in all four health faciiities. Nevertheless, further improvement is needed. Only 315 facilities had up to date immunization registries compared to 414 for the baseline survey, and 4/5 had a stock of child vaccine and TT cards. Data on morbidity, mortality and CHW activities was reported by CHWs to their supervisors monthly. The facilities compiled these reports and forwarded them to the woreda health office. The SCSP staff received the monthly activity and morbidity reports from the woreda health office. In general, little or no feedback is provided to health workers on the foms they submit to woreda and zonal authorities, nor is their supervision in relation to health information gathering and reporting adequate. Some do not understand the foms they fill out and some do not value the accuracy of the information they submit, Great improvement is needed not only in the quality of data collection, as noted earlier in relation to the mistakes made by one TBA, but also in the value and the use of data as a learning, quality improvement and management tool for health workers and health authorities. Table 9: Availability of liecords and Forms I S.N ( Registers and cards 1 Proportion of facilities with 1 I cards and registers (n=5) I . . - 3. C.8. Technical and Administrative support The project received technical support Erom the PVO headquarters for the preparation of the project proposal, the DIP, mid-term evaluation and final evaluation. In Up to date patient register Up to data immunization -- 4. 1 Stock of TT cards 5. 1 Cards and forms addition headquarter staff identified a consultant to aide in the KPC survey, provided training opportunities for CSP staff and resource materials on CS. Headquarter staff made frequent trips to Ethiopia to assist the SCSP with the managerial and technical aspects of the progarn. Table 10. Technical and Administrative Support I 100% 60% register Stock of child vaccine cards 80% 100% Support given by: 1. PVO head quarters 80% r~V~/~ddis Ababa 3. MOH 4. ECCMY Family Planning Type of support - Preparation of the project proposal - Preparation of the DIP - Identification of a consultant for a KPC survey - Identification of training opportunities for CSP staff - Provision of resource materials -Networking - Net working with different organizations at national level of MOW, FGAE, Pathfinder - Preparation of the baseline and final surveys, the DIP, and monthly, quarterly and annual reports - CSP and ADP management, financing and logistics - Recruitment and hiring of staff Training on MU, EPI, FP for CHWs Training of CBRHAs The project received adequate technical support from both the national and USA based PVO organizations. D. Other Issues Identified by the Team The SCSP's integration into the Shenkolla ADP in Ethiopia was both an advantage and a hindrance. Perhaps the most important advantages of such an integration was that the ADP had developed very good relationships with the coinmunities with whom they had worked over a number of years and were trusted by the woreda and zonal authorities. Furthermore, it meant that the project did not have to establish a new infrastructure in the area, nor use it very limited resources to set up and hrnish offices and living quarters for the staff. These advantages allowed the SCSP to start its activities in a timely manner after its bumpy beginning in Eritrea and considering the isolated and difficult to access area where the project was located. The Shenkolla ADP is a self-contained community located in mountainous terrain and is accessible only by a four-wheel drive vehicle or motorbike during the dry season. Communication fiom the project site to the outside world is through radio only. The nearest telephone is in Jajura, a village 30 minutes drive from the project site in good weather. The ADP constructed the roads into the area through their food for work program. Water is piped in from the mountains and electricity is produced through generators. All the buildings are constructed by the ADP, as there is no major village or city nearby. The project needed the ADP's resources and linkages. The difficulties of the integration centered on role confusion and bltmed lines of authorities in relation to the management of the ADP and the SCSP, If this confusion had been addressed early on and resolved amicably, staff morale would have been higher and output probably greater. The experiences of the integration lead to some important lessons learned, some of which have already been addressed. In summary, integration has tremendous advantages both for ADPs and CSPs in that they have similar and complimentary objectives, can reinforce each others strategies and programs, reduce costs and share resources, benefit from experiences and lessons learned if lines of authority and job descriptions are clearly articulated and understood by staff of both groups. WV's ADPs have long-term commitments to an area and can thus greatly influence the sustainability of any CSP. E. Conclusions, Lessons Learned and Recommendations Overail Conclusion During its relatively short time line - less than four years - the SCSP was able to make considerable progress in achieving its two inter-related goals to reduce under-five and maternal morbidity and mortality and total fertility rates; and to build the capacity of partners to provide quality, sustainable health services. The project adapted innovative ideas to develop new initiatives; made concerted efforts to integrate its activities with those of the MOH and the Shenkolla ADP; explored available avenues in a timely fashion to sustain activities beyond the life of the program; utilized and strengthened existing resources to increase access and availability of health care services; included comm~mity members in the delivery of health care services; limited its scope to that what was doable and feasible; and partnered very successfully with well established organizations in the project area. World Vision's strategy to establish and maintain ADPs in areas of great need compliments the overall effort of a CSP. When fully integrated, the two programs can greatly benefit fi-om each other, not only in terms of sharing its resources and expertise, but also in assisting each other with attainment of goals, objectives and long-term sustainability endeavors. Many lessons were learned by WV/Ethiopia fiorn its first experience with such integration, foremost of which is the need to have lines of authority and managerial issues clearly articulated and followed, and supported by the powers at be. If done carefully, the integration of a CSP with an ADP can become the new paradigm for community based health activities for World Vision. Children and Mother's Health I. Through the efforts of CHWs the knowledge of mothers has been greatly enhanced regarding danger signs in their children and the need to seek care, the timeliness and importance of EPI, family planning, and the dangers of harmful traditional practices. 2. There is adequate drug supply at the majority of health facilities, according to the mothers. 3. Mothers greatly valued the respect and sensitivity shown them by CHWs, especially in the promotion of family planning. 4. Community members recognized the improved health status of children as a result of the CHW efforts trained in the CSP. 5. The vast majority of mothers and other community members embraced the family planning knowledge and services they had received from the CHWs. 6. As a result of health education provided by CHWs, there is a marked change in attitudes toward female genital mutilation and other harrnfiil traditional practices. The Work of CHWs 7. CHWs were the most consistent agents of change in the community. 8. The role the CHW played gained them recognition and acceptability in the community. 9. The performance and commitment of CHWs is commendable in the face of negligible financial support &om the community and other partners. 10, In spite of the poor quality of supervision, supervisory visits have motivated the CBWs to remain engaged in their activities. 11. The communities seek out CBWs for problem solving and referrals. Capacity BuiIding 12. IMCI training improved the performance and function of facility based health workers, and increased their confidence to deliver quality case management. 13. The CSP has facilitated capacity building of key Zonal and Woreda health staff in the use of the BASICS' IMCZ supervisory tool. The RDF 14. The RDF is the result of a successful private-public partnership between MOH and WVE Shenkolla CSP that included the sharing of essential resources. 15. The CSP's RDF initiatives were motivated by the discovery of the small, but sustained RDF scheme initiated by an Italian NGO (SACIF) a few years ago. 16. WV-CSP took the opportunity to learn from the earlier RDF experience in the area, as well as information obtained on the Bamako Initiative. 17. The project, from the inception of the RDF scheme, included its partners in capacity building activities. 18. The success of the ICDF at Gimbichu has spawned a network of 13 other associated RDFs, both within and outside of the CSP catchment area. 19. The RDF bas notably increased access, affordability and convenience of the essential dmg supply. 20. The RDF has changed drug-purchasing patterns, thereby decreasing the need to use unregulated sources in the community. 2 I. The RDF has significantly improved IMCI case management by making antibiotics more readily available. 22. The success of the RDF is partially due to the commitment and honesty of a few key individuals, and clearly defined regulations and auditing practices. Supervision 23. Although supervision is carried out, the quality of this supervision is compromised by to the lack of supervisory skills and tools at the facility level. Furthermore, there is a lack of adequate transport and funds at the Zonal and Woreda levels to ensure regular supervision at all levels of the health care system. Church Elders 24. The success of the CSP activities convinced the church elders to support the project, especially in relation to family planning activities. 25. The church elders have linked Christian teaching to address HIVIAIDS, female circumcision, family planning and polygamy. 26. The church elders have extended to CHWs the opportunity to address the congregation with key household and health education messages. PAS 27. The PA administrators are essential in facilitating mobilization of communities to support CHW efforts, although their potential is not met and, therefore, needs to be strengthened. 28. The PA administrators are not yet engaged in folmulating plans to compensate CHWs for their voluntary service to the community. Youth 29. The youth's reluctance to address the prevalence of STDs among themselves is an impediment to their acknowledgement of their own vulnerability to HIV/ADS. 30. Knowledge about HIVIAXDS risk among youth is high, and educational messages are reaching them. 3 1. There is a strong desire among youth to be involved in the establishment of anti￾AIDS clubs in their communities. 32. The youth see themselves as the most effective vehicle for communicating information on HIVIAIDS to their peers and their communities. 33. HIVIAIDS IEC materials available in the CSP area are generic in nature and not targeted to youth. Health Committees 34. The health committees appear to be barely functioning, their roles are weak and their ability to influence health services is negligible. ADPJSCSP Integration 35. The goals of the CSP are compatible with the overall goals of the ADP, which are increased household agricultural production, increased household off-farm income and improved household health status. 36. The CSP benefited greatly fiom the overall established infrastructure of the ADP, as well as from the pre-existing relationships with government and community authorities. 37. The CSP has contributed greatly to the attainment of some ADP planned health activities in the CSP catchment area such as environmental sanitation, HIVJAIDS, EPT and family planning. 38. The high turnover of CSP staff was a result of unresolved conflicts between the CSP and ADP managers. Sustainabiliq 39. The sustainability of the project is enhanced by the continuing ADP support of the CSP activities, such as outreach EPI and supervision, as well as IEC at the community level. 40. CSP's family planning activities at the household level, including IEC and distribution of contraceptives will be continued through a "memorandum of understanding" signed with the PathfinderIEvangelical Church. 41. By adopting a focused approach to health interventions that resulted in documented improvement in health indicators in the community, some of the project's activities were expanded to neighboring PA. 42. Although EPI outreach has been facilitated by CSP transport, manpower shortage, rugged terrain and long distances are persistent barriers to effective outreach services, and threaten community compliance and sustainability. 43. The sustainability of some of the major components of the CSP is assured beyond the life of the project through: 1) WVE Shenkolla ADP's commitment to allocate budgetary support for outreach activities to maintain the gains achieved to date, such as EPI and supervision,; 2) increased mothers' capacity to identify danger signs in sick children, to seek care in a timely fashion and to readily access medicines; 3) expanding RDF scheme; 4) continued support for the CBRHs family planning activities SCSP Accomplishments 44. In spite of high turnover in PA, CSP and health facility personnel, the CSP has made remarkable achievements. Lessons Learned - Shenkolla CSP Clear understanding of roles and responsibilities of each party is essential to ensure successful collaboration in an RDF venture. An in-coming project should careklly identify small but undeveloped key activities that have the potential for scaling up and strengthening. A relatively short-term project cannot expect to reform a supervisory system because such an effort would require a major "quality assurance" paradigm shift. When given the opportunity the church eIders can greatly enhance the diffusion and acceptance of health messages and practices. When a CSP engages people with strong spiritual beliefs and commitments, the efforts of the project may be enhanced. The provision of family planning services through the privacy of house-to-house visits, greatly improved family planning acceptance by the couples. Training CHWs in the use of multiple channels of communications and approaches to service delivery, enhances attitude and behavior change. It is better for a project to focus its activities in a manageable area until there is some evidence that improvements have been made in key health indicators before attempting to expand into new areas. It is essential for future CSP and ADP integration that lines of authority as well as roles and responsibility are clearly documented and agreed upon during the pIanning phase of a project. Recommendations - Shenkolla CSP The ShenkolIa ADP and its partners should engage PA administrators in the design and pianning of meaningful compensation for CHWs, so that they will continue to fill important roles in their communities. The Shenkolla ADP and its partners are encouraged to make efforts to find means to obtain or develop youth-appropriate, IECBCC materials for use in anti-AIDS clubs, schools and churches. The Shenkolla ADP and its partners are encouraged to build the capacity of the church to engage in HIV/AIDS related activities. The Shenkolla ADP and its partners are encouraged to form new groups and to strengthen existing anti-AIDS clubs in schools and churches in the communities. Because of the essential contribution of the CHWs in their communities their supervisory system needs to be strengthened and improved. The current government attempt to create a career structure for the CHW needs to be strengthened through regularly scheduled, continuing education opportunities and policy changes. WVE and MOH together are encouraged to explore cost-recovery and income generating mechanisms that will strengthen the sustainability of community and facility based health services, such as the experience of Satussa ADP and Attat Hospital. WV, along with partners, has to explore ways to strengthen health committees so that they will function as a support to the CHW and link to the community. 9. The MOH is encouraged to immediately address staff and budget shortages in existing health facilities before considering expansion and upgrading of health facilities. 10. The ADP and its partners should maintain gains made through CSP by remaining focused on those strategies that have proven to be successfi~l such as: CRW training and supervision; RDF scheme; and outreach activities. 11. WV should not support the expansion of the health infrastructure until existing health facilities are operating with acceptable standards of performance. 12. When unresolved conflicts occur at the ADP level between CSP and ADP staff, then national headquarters should intervene quickly to arbitrate a resolution. 13. WVE with its partners should immediately initiate planning for a follow on CSP in the Shenkolla ADP area to be submitted to USAJD in the FY04 round. 14. In the event that a follow on project is granted, the new CSP should function as a learning center for multiple WVE ADPs, F. Results Highlights Revolving Drug Fund In Ethiopia, government health facilities are often in short supply of drugs, particularly in rural areas. Because of budgetary constraints, health facilities frequently consume their entire annual budget for drugs in a few months. As a consequence, unauthorized dispensing of drugs is a common practice, To address these problems, a revolving drug Eund (RDF) was established in the early nineties in Butajira, a town 120 kms, from Gimbichu. This RDF began with little financing, but was miraculously sustained over the years with little outside support. The SCSP manager knew about the scheme and arranged for an experience-sharing visit that included staff from the Woreda Council, health, education and finance offices, two coinmunity members, a member fiom the Zonal health department to learn about what did and did not work. As a result of this visit a RDF committee was formed with nine members, and the scheme was initiated. A sub-committee - the technical committee made up of members of the main committee - was also created. The SCSP and the governmental health authorities established a successfbl private-public partnership by working closely together from the initiation of the RDF scheme. The health office provided one building block with three rooms at Gimbichu health center, and paid the salary of the drug dispenser, while the SCSP renovated the building, supplied all the initial necessary drugs, supplies and materials to begin operation and supported the running cost of the scheme for one year. Supervision and distribution of supplies was shared between the SCSP and woreda health authorities. Within a few months of the initiation of the RDF, it was expanded to two other government health facilities and to a third one soon thereafter. The success of the RDF at Gimbichu eventually spawned a network of 13 other associated RDFs, both within and outside the CSP catchment area. It changed drug-purchasing patterns, thereby decreasing the need for unregulated sources in the communities. It also notably increased access and affordability of the essential drug supply. Family PIanning The FP component of the SCSP can be considered as one of the most enduring and successful activities of the project. Data from the KPC surveys showed an increase in modem contraceptive prevalence from 6% to 36%. This extraordinary increase was verified tlu-ough focus group discussions. Mothers, church elders and other community members reported the increased acceptance of family planning because there was a strong desire to limit the number of children per family. The reasons given were: lack of adequate land for fanning and food insecurity, improved mothers' health, children were no longer dying, and better family relationships. Comm~mities credited CBRHAs for much of this success. CBRHAs through their IEC and house-to-house efforts have been able to change behaviors, while making contraceptives readily available. The Ethiopian Evangelical Church Mekane Yesus (EECMY) will continue to provide support for the CBWAs with the assistance of Pathfinder. EECMY will supervise the CBRHAs, provide contraceptives, and pay them 60 birr each month when they come for restocking. Attachment A Attachment A: Shenkolla Evaluation Team Name Organization Position Zeleke Aklilu Soro MOH MCH Coordinator Larry Casazza WVUS Tech. Resource Team 'ubishet Degiiiet Bekele Degefu Deneke Necha Tom Hall Samuel Lechebo Helga Morrow Sisay Sinamo Kassahun Tezera Terefe Tafesse Doris Wurah WVE WVE Zonal MOH WS WVE Independent WVE - SBO WVE WVE AfricareEthiopia ADP Manager Health Coordinator Training Coordinator Program Officer MCH/FP Coordinator Consultant & Team Leader Health Officer Social Facilitator HTV/AIDS Social Facilitator Intern Attachment B Attachment B: Evaluation Methodology The data for this report was collected from the three evaluation methodologies: an IHFA, the KPC survey, and qualitative assessment of program activities as described by the partners, coinlilunities and the staff. The IHFA was designed to collect information on the quality of case management of the common causes of child morbidity and mortality, Information was also collected on the status of the health facility with respect its organization, availability of services in relation to maternal and child health, drugs, supplies and equipment, and the status of and support for health facility staff. The IHFA was conducted from 12-17 August 2001 in all of the five health facilities in the SCSP area. The KPC survey was conducted from 27 August to 6 September 2001 with a randomly selected sample of 300 mothers with children less than 24 months of age living in the targeted 28 peasant associations of Soro Woreda. It used a questionnaire originally designed by the Johs Hogkins University, School of Public Health in Baltimore, MD USA, that has been ~~pdated and modified to meet the needs of individual projects. The last part of the evaluation process was a qualitative assessment made of the program, activities, strategies and methodologies used by the project through a review of reports and other documents, and as perceived by the partners, the communities and the staff. For this evaluation exercise, 7 of the 28 PAS were randomly selected for field visits. In each PA, mothers, youth, PA members, and church elders were interviewed, using a semi-structured interview guide. Xn addition, health workers in 3 of the 5 health facilities were interviewed, as were the supervisors, and persons responsible for the revolving dmg ftmd (RDF). In total, more than 700 people were contacted or interviewed during the course of this evaluation. The final evaluation team developed the guides used for the focus group discussions, summarized the findings of those discussions and drew conclusions based on the summaries. These were then presented in plenary session and recorded. The entire team then formrllated the conclusions, lessons learned and recommendations as presented in this report. Attachment C Attachment C: List of Persons Interviewed Worid Vision/Ethiopia Headquarters - Addis Ababa Getachew W. Michael WVE National Director Teklu Wodajo Grant Division Director Mekuria Asfaw Staff Simon Heliso CBD - Director Tsegame Guracha Grants Division - Deputy Manager Tamirat Yirgu Communication - Manager Asrat Dibaber MICAH - Program Manager Terefe Tafesse HIV/AIDS - Social Facilitator Mexin Loha Relief - Deputy Manager Genechu Demissie MICAH - Program Officer World Vision/Ethiopia South Branch Office - Awasa Zerihun Beyene South Branch Director Bekako Duguma Program Manager Hadiya Zone Authorities Erimias Ashamo Acting Head Zone Health Department Abayneh Ekello Team Leader for Health Programs Terefe Lerebo Team Leader for Health Facillities Woreda Level Abayneh Erkelo Head Woreda Health Facilities Zeleke Aklilu Head Woreda Disease Control Ayile Lemma Head Health Center - Gimbichu Y isehak Lemma Sanitarian Lemlem Kifle Yesus Senior Public Health Nurse Partners Almaz Andebo Allan Weinstock Geira Baruda Doris Wurah Robert Kacoba Head Jajura Catholic Mission Clinic Advisor, JSI - Awasa Community Health Specialist, JSI - Awasa Afiicare - Intern Afiicare - Country Representative Regional Authorities Zeleke Gobi Deputy Head Regional Health Bureau Annex C. Interviews VHWs and Health Committee members from different PAS were interviewed together as a group. Eight Fathers were interviewed in Kosha Qualitative Assessment was conducted with a total of 438 persons