IRC Sierra Leone Child Survival Grant in Kono District CSHGP Cycle XIX 2003 – 2008 Mid-Term Evaluation Report September 8, 2006 Submitted by: Eric Sarriot, consultant (ericsarriot@gmail.com) International Rescue Committee Contacts: In Freetown : Yilma Robelle Country Health Coordinator yilmar@theirc.org In New York: Emmanuel d'Harcourt 122 East 42nd Street 12th Fl. New York NY 10168-1289 emmanueld@theirc.org In Kono : Bernadette Udo Project Manager bernudo@irc.sl IRC Kono CS 19 Mid-Term Evaluation Report Acknowledgements: Thank you to the IRC team from Kono, Freetown, the regional level and New York for the opportunity to learn about an exciting project in the challenging context of Sierra Leone. Much appreciation for the hearty welcome in Kono and the support in Freetown, the hospitality of the guest house, but also for the support for and openness to the rigors of an honest evaluation. As customary, this exercise was conducted as a team exercise. Many thanks go to the members of the evaluation team, project partners, DHMT and MOH staff, CARE, Kenema and other IRC project members. This report also acknowledges the IRC admin-finance and support staff who offered time and information. Finally, thanks to the communities who welcomed us, the community leaders we interviewed and the paramount chiefs who participated in the debriefing, and all those who welcomed and sometimes hosted the evaluation team in their Health Units. The Child Survival project in Kono is funded by the US Agency for International Development’s Child Survival and Health Grants Program (USAID/CSHGP). The Mid-Term Evaluation is a requirement of the Grants Program. Except when specified otherwise, opinions expressed here are those of the consultant, informed by the participatory process of evaluation. A-2 IRC Kono CS 19 Mid-Term Evaluation Report Table of Contents Table of Contents _____________________________________________________ A-3 List of acronyms______________________________________________________ A-4 Executive Summary ___________________________________________________ A-5 A. Introduction ___________________________________________________ A-12 B. Assessment of the progress made toward achievement of program objectives B-12 1. Technical approach _________________________________________________ B-12 2. Cross cutting strategies ______________________________________________ B-43 C. Program Management ___________________________________________ C-53 1. Areas of consensus __________________________________________________ C-54 2. Areas for resolution and improvement _________________________________ C-57 D. Mission collaboration ___________________________________________ D-59 E. Other issues identified by the team _________________________________ E-59 F. Conclusions and recommendations ________________________________ F-59 1. First recommendations from the workshop _____________________________ F-59 2. Final recommendations from the consultant_____________________________ F-60 G. Results Highlight _______________________________________________ G-76 H. Action Plan____________________________________________________ H-76 I. Revised Project Data Sheet __________________________________________ I-76 A-3 IRC Kono CS 19 Mid-Term Evaluation Report List of acronyms ACT Artesunate Combination Therapy ARI Acute Respiratory Infection BCC Behavior Change Communication BFV Blue Flag Volunteer CARE Cooperative for Assistance and Relief Everywhere, Inc. CBO Community Based Organization CHW Community Health Worker (include TBA, VHC, BFV, etc.) C-IMCI Community-Based Integrated Management of Childhood Illnesses CS Child Survival CSSA Child Survival Sustainability Assessment CSTS+ Child Survival Technical Support DHMT District Health Management Team DHO District Health Office DIP Detailed Implementation Plan DMO District Medical Officer, in charge of DHO DPT Diphtheria, Pertusis, Tetanus EOP End of Project HMIS Health Management Information Systems HQ Headquarters IEC Information Education and Communication IMCI Integrated Management of Childhood Illnesses IRC International Rescue Committee ITN Insecticide Treated Mosquito Net KPC Knowledge, Practice and Coverage LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MNC Maternal and Newborn Care MOHS Ministry of Health and Sanitation, Government of Sierra Leone NGO Non-Governmental Organization OR Operations Research ORS Oral Rehydration Solution ORT Oral Rehydration Therapy PCM Pneumonia Case Management PHU Peripheral Health Unit PVO Private Voluntary Organization QA Quality Assurance TA Technical Assistance TBA Traditional Birth Attendant TT Tetanus Toxoid UNICEF United Nations Children’s Fund USAID United States Agency for International Development VHC Village Health Committee WHO World Health Organization A-4 IRC Kono CS 19 Mid-Term Evaluation Report Executive Summary The International Rescue Committee (IRC)’s Child Survival Program (CSP) in Sierra Leone, funded by USAID and matched by CIDA and IRC, started in October 2003 and is set to run until September 2008. The main objective of the project is the sustainable reduction of child and maternal mortality in the catchment areas of twenty health facilities in the Kono District. It is implemented by IRC, in collaboration with the Kono District Health Management Team (DHMT), in 10 of the 14 chiefdoms of the district. Its intended beneficiaries include an estimated 17,000 children under 5 years of age and 24,000 women of reproductive age. The program focuses on five intervention areas: immunization, control of malaria, control of diarrhea, pneumonia case management and maternal and newborn care. Through active partnership with the DHMT and by maximizing national opportunities in the chiefdoms of intervention, the project has increased bednet coverage for children under five from 6% to 40% (compared to a national increase to 12.5% nationally). It has doubled complete immunization coverage, and helped rapidly translate the new policy for Intermittent Presumptive Treatment (IPT) of malaria for pregnant women into a success (the district has now reached 60% coverage in the project areas of intervention). One of its strongest features is a close and constructive partnership with the DHMT. This is shown, for example, by the project recently ending its Health Information Systems (HIS) data collection and reporting independently of the DHMT, in order to support the production of information at the DHMT level. This picture is however balanced on the down side by a lack of progress in a number of areas of intervention. The two summary figures (next page) present overall progress on all eight main objectives of the project as well as on secondary objectives. These figures reveal where progress has taken place (Bednets, IPT, Immunization, and Vitamin A supplementation), but also where it has failed to do so (Control of Diarrheal Diseases, Pneumonia Case Management (PCM), and Maternal and Newborn Care as a whole.) It also highlights areas where objectives and indicators will require adjustments for the second half of the project. A-5 Summary Figure: overall mid-term project achievements on principal and secondary objectives (The main indicators of the project are numbered and have a defined EOP target. Related but secondary objectives are also presented.) 0 20 40 60 80 100 1-Complete Immunization 2- Vit. A suppl. (6-59 months) [Yellow Fever Immunization] 3-Hand washing (4 times) [Hand washing (2 times)] [Increasing Fluid Intake of Sick Child] 4-Children <5 sleeping under ITN 5-IPT for Malaria during pregnancy 6- ARI signs recognition 7- TT coverage [Post-partum Iron suppl.] [Post partum Vit.A suppl.] 8- SBA for complicated deliveries [SBA for all deliveries] Main and Secondary Objectives Percentage Baseline Mid-Term EOP target (main objectives only) Several key factors, some external and some internal, have prevented the project from making good progress on some of its objectives so far. They start with the context, as IRC chose (with MOH and USAID encouragements) to focus its work on one of the districts which suffered the most from the 12 year war that devastated Sierra Leone1. The project has also gone through changes in key positions (project manager and BCC coordinator) and the long vacancy of the health coordinator position in Freetown, which combined have created excessive stress on staff and slowed activities. This goes with a number of management challenges which have been analyzed through this evaluation exercise. - Mid-Term: good time to check the road The Mid-Term Evaluation (MTE) has identified areas where redress is necessary: some objectives and indicators need to be redefined; essential strategies and activities to reach objectives need to be clarified; tools need to be set in place and capacity building accelerated, first at project level, then with DHMT; finally the roles and distribution of staff to carry out the activities need in-depth retooling. Improving the M&E and HIS systems, while keeping the HIS with the district team will be an essential step in giving the project its second wind. The Mid-Term evaluation has also identified areas in which project performance has been stronger. The project has been present in the field, from Peripheral Health Units (PHUs) to the household level. It has been a solid partner of the DHMT. As the project now introduces community-based treatment (CBT), it has an opportunity to reach great impact on the indicators that have already moved (some having already overreached their end of project (EOP) target), and address those that have lagged behind. Phase out is not the ‘ordre du jour’, but rather focus and acceleration of project efforts toward impact. However, this can be done with and through the very strong DHMT partnership, which itself is a strong sustainability prospect. The tools and functions which the project 1 Sierra Leone ranks last in the UNDP Human Development Index ranking. IRC Kono CS 19 Mid-Term Evaluation Report needs to develop are the tools and functions which the DHMT will continue implementing and will rapidly scale up to the entire district. The following are recommendations developed in the final section of this document, based on the review of strengths and weaknesses. Priority recommendations are highlighted in bold font: Recommendations for improving Child Health 1. Develop a comprehensive set of IMCI tools with the DHMT, for PHU, CHW and TBA, and CBT levels. 2. Train project staff in IMCI, by MOH IMCI trainers in accordance with MOH schedule. 3. Review and adapt the DHMT supervision plan, emphasizing the same IMCI strategy. This includes following up with PHU in-charges and also reviewing, adapting and monitoring their own plans. Additional recommendations derive from this: 4. Intensively focus community and facility-level immunization strategies on appropriate timing of immunization, particularly measles and DPT1. 5. Develop a plan with each PHU for improving systematic Vitamin A supplementation. 6. In collaboration with the DMO, this plan could include pilot zones for involving TBAs and possibly CHWs in community-based distribution of Vit. A. 7. As recommended by the evaluation team, even though the project has already reached its EOP goal, it should be ambitious and try to increase intensive bednet distribution. 8. Appropriate and timely treatment of malaria in under-fives needs to become a clear objective of the project (rather than a “secondary objective” only occasionally measured). 9. Proper management of diarrhea in children (oral rehydration and zinc treatment) should become a clear objective (with an indicator measured toward a target) of the project. 10. The PCM strategy of the project needs a total overhaul, which should be simply done by going through the IMCI strategy. Recommendations for improving Maternal and Newborn Care 11. The project should set as an objective to modestly but significantly increase skilled birth attendance at the PHU level. 12. The project should improve the district capacity (from management to facilities) to manage timely referrals of A-8 IRC Kono CS 19 Mid-Term Evaluation Report complicated deliveries. 13. The project should increase the proportion of communities with emergency transport plans in place and operating 14. The project should increase TBA detection and early referral of complicated deliveries 15. Just as child health needs a driving strategy, antenatal and postpartum care need tools and focus to improve quality, and coverage for PNC. 16. PNC efforts must take into account the role of TBAs (see above) and emphasize immediate breastfeeding, Vit.A supplementation, and Iron supplementation. Recommendations for improving cross cutting strategies 17. The capacity of the project to operate a useful HIS /HMIS for the benefit of the DHMT and its own objectives needs to be built rapidly. 18. The project HIS, integrated within the DHMT, needs to focus on performance gaps identified during the MTE, and on producing rapid analyses to guide management decisions. 19. After an initial and immediate effort to launch a HIS processes more supportive to management decisions, the system should probably be reviewed and improved at least on an annual basis. 20. The project should start implementing simple QA learning cycles with the DHMT. This will require that essential tools be developed (see above) and the project team becomes comfortable with their implementation and monitoring in the field. 21. The BCC strategy needs to be strengthened and put on tracks and the BCC manager needs to receive adequate support to do so. 22. In terms of community strategy, the consensus of the evaluation was on moving forward with community-based treatment and encouraging more outreach and CHW/TBA efforts. In the long run, the project should consider a more sustainable community mobilization and community organizing strategy, by building its own capacity or partnering with other IRC projects. It should consider reviving the CHW network development strategy. 23. The partnership with the DHMT will be key to the success of the project. This needs to be maintained and fostered. This is an essential role for the project manager. Discussions about further integrating the project and DHMT teams (possibly by moving the project inside the DHMT) are certainly interesting; this idea deserves further consideration. A-9 IRC Kono CS 19 Mid-Term Evaluation Report 24. The large picture of sustainability is in place through this partnership, but the project should keep an eye on specific issues which are coming up to threaten components of sustainability. Recommendations for improving project management 25. The field staff needs to be used more effectively, to build capacity rather than to implement and reach communities themselves. 26. The project manager, with the close support of the health coordinator as well as support from the CS team in NY and Nairobi, needs to carefully assess and strengthen the capacity of the staff to conduct specific tasks (i.e. supervision of technical areas versus supervision of geographic areas). The assignment of roles and responsibilities, and the number of positions required to effectively cover specific activities need to be reviewed based on operational plans and the sort of functional analysis initiated at the time of the evaluation. Budgets can be adjusted. 27. Managers need to be equipped to support Field Supervisors through tools and methods, plans, effective monitoring, coaching/supervision, as well as by protecting their staff from distraction and building a supportive environment. 28. Support and backstopping of the project needs to be strongly reinforced in a coordinated fashion between New York and Freetown. This means providing tools, supporting capacity building, helping with prioritization of action steps in the immediate future and close communication and monitoring. The health coordinator and the HQ child survival team should agree on a list of tools to support the manager and her team, and develop this within a short timeframe. 29. A monthly backstop supervision visit by the Health Coordinator in Freetown should take place for the foreseeable future to support the project management in this new phase. The project senior and activity managers will benefit from this coaching, ongoing and in the field (in PHU and community sites). 30. Financial, administrative and logistical support to the project needs to be greatly improved. The project manager needs a greater sense of the elements within her control, notably budgeting, procurement problem-solving, access to logistical resources and to technical assistance. This requires country office and field coordination involvement. Different A-10 IRC Kono CS 19 Mid-Term Evaluation Report perspectives exist on the root causes of the weaknesses identified (systemic vs. individual performance, and the role of cultural factors) but not on their existence and the necessity to diagnose and remedy them rapidly. 31. All efforts must be made to recruit the senior TA position recently advertised in a capacity building role, as well as timely recruitment of the CBT staff. 32. Communication within the project and with its support units needs to be improved, both in terms of hardware and software. A-11 IRC Kono CS 19 Mid-Term Evaluation Report A. Introduction The International Rescue Committee (IRC) solicited a consultant to perform a mid-term evaluation (MTE) of the Kono district child survival project, with the following objectives: o Evaluate the degree to which the program’s objectives have been achieved; o Evaluate the sustainability of current strategies and activities, particularly the role of IRC’s local partners; and o Make recommendations for the program’s final two years, as regards to: o Technical interventions o Management o Sustainability o Partnerships This evaluation was conducted from May 4 to May 17, 2006 in Freetown and the Kono district through a participatory evaluation approach. Findings from the evaluation are presented according to CSHGP guidelines in the next sections. Conclusions and recommendations are presented after a separate treatment of technical and managerial issues. B. Assessment of the progress made toward achievement of program objectives 1. Technical approach a. General overview The International Rescue Committee (IRC)’s Child Survival Program (CSP) in Sierra Leone, funded by USAID and matched by CIDA and IRC, started in October 2003 and is set to run until September 2008. The main objective of the project is the sustainable reduction of child and maternal mortality in the catchment areas of twenty health facilities in the Kono District. It is implemented by IRC, in collaboration with the Kono District Health Management Team (DHMT), in 10 of the 14 chiefdoms of the district.2 Its intended beneficiaries include an 2 The largest administrative unit at the sub-national level is the region; there are four regions in Sierra Leone. The second level administrative unit in Sierra Leone, under B-12 IRC Kono CS 19 Mid-Term Evaluation Report estimated 17,000 children under 5 years of age and 24,000 women of reproductive age; and also indirectly benefit some of the other 150,000 women and children in the district. The program focuses on five intervention areas: immunization, control of malaria, control of diarrhea, pneumonia case management and maternal and newborn care. The project was designed with key strategies in mind: - Community integrated management of childhood illness, including the provision of first-line treatment at the community level - The development and support of networks of community health workers - Promotion of zinc treatment and ORS for diarrhea Improving communication & reporting from PHU to DHMT - The promotion of insecticide treated bed nets - The promotion of intermittent presumptive malaria treatment for pregnant women - The promotion of immunization for children under five and at least 2 doses of tetanus toxoid vaccine for pregnant women - The promotion of hand washing with soap and water on key occasions in diarrhea prevention - The use of appropriate information systems to support the above activities. The program’s major partners include the beneficiaries themselves, as well as other community members, Community health workers, Ministry of Health and Sanitation (MOHS) staff including those from health centers; district health teams; relevant technical offices in the central MOHS office. Critical events on the project’s timeline Table 1 provides an overview of the project timeline, as developed by the CSP team for the evaluation startup meeting. Some first the Region and above the Chiefdom is the District. B-13 IRC Kono CS 19 Mid-Term Evaluation Report observations are possible at the level of inputs and outputs of the project: ƒ First year activities started naturally after the DIP presentation, although a ‘startup phase’ was probably extended by the addition of seven Primary Health Units (PHUs) and PHU areas to the initial 13 defined by the project. This decision was a commendable effort to maintain the size of the promised beneficiary population, after it was found that it was lesser than thought in the initial 13 areas. It nonetheless added a requirement for ‘startup’ efforts, with a baseline survey in these seven areas conducted between January and March 2005. (In hindsight, it could perhaps have been postponed until the project had an intervention method in place.) ƒ After 18 months, the project appears to have entered its most active phase (roughly between April and June 2005). At this point, the project’s five health promoters were each covering one PHU and its catchment area every week. Activities included follow-up on training of PHU in-charges and vaccinators, supervision of PHU staff independently or with the DHMT, participation in PHU outreach, joint or independent supervision visits to CHWs, participation in or organization of health talks, and household visits with CHWs or independently. ƒ After the summer of 2005, the project entered a series of transitions, leading to a slowdown of activities and a drastic reduction of field presence in the last six months. Some of these important transitions are the following: o A positive transition was the arrival of the current Health Coordinator. There had not been a stable Health Coordinator in post in Freetown up until that point. o This was followed by a transition in the leadership of the project. The expatriate project manager had a six week overlap with the new and current project manager before becoming regional coordinator of the community-based treatment project (CBT). o At the end of 2005 / beginning of 2006, this was followed by the departure of the previous BCC officer, and then a general redefinition of roles and responsibilities (promotion of health promoters to technical area supervisory roles, definition of objective managers (M&E, BCC, and CBT). o In the months preceding the MTE, the project also implemented the mid-term KPC data collection, and started preparation and recruitment for the newly funded community based treatment (CBT) program to complement the general child survival activities in the district. B-14 IRC Kono CS 19 Mid-Term Evaluation Report This timeline needs to be kept in mind in the analysis of the strengths and weaknesses of the program. - CSP Health Promoter/CBT supervisor at work with PHU staff B-15 IRC Kono CS 19 Mid-Term Evaluation Report Table 1. Rapid overview of the project timeline to the preceding quarter (developed by the CSP team for the MTE) Project Management & Partner Relations Project activities & deliverables Oct.- Dec. 03 ƒ Official Start of CS Program ƒ Held meetings with the DHMT's EPI point person and the disease surveillance person (Ongoing) Jan.- Mar.04 ƒ National staff hired. ƒ Base Line survey done with analysis completed ƒ LQAS covering 13 PHUs ƒ Sustainability Exercise Apr.- Jun. 04 ƒ DIP presentation to USAID ƒ Finalized DIP (Field & NY TU) ƒ Mortality Survey Jul.- Sep.04 ƒ Additional Staff hired ƒ 7 PHUs added ƒ Exit of BCC Supervisor ƒ In collaboration with the DHMT trained PHU staff on (a) Tickler System (b) Epidemics (c) Calculating Targets and coverages for vaccine ordering purposes Oct.- Dec.04 ƒ Eight staff including the manager on board ƒ Supported DHMT for and participated in two rounds of NIDs in October (8th, 9th,10th) and November (19th,20th, 21st) 2004 Jan.- Mar.05 ƒ Participated in the DHMT's EPI mobile campaign in Mafindor, Gbaneh Kandor, and Soa chiefdoms on the 26th, 27th 28th January 2005 ƒ Baseline survey among 7 PHUs (NY TU) ƒ QA System development started Apr.- Jun. 05 ƒ Conducted survey enquiring TBA obstacles to referrals ƒ The 2 child survival supervisors participated in the April 2005 NIDs exercise from the 8th-10th as supervisors. In addition, the program provided two vehicles to make this exercise a success. ƒ One child survival staff assisted the DHMT in facilitating the IMCI training on malaria, pneumonia/ARI and diarrhoea for PHU in-charges from the 13th to 23rd April 2005. ƒ Conducted TOT workshop for PHU staff for them to train CHWs on the use of flash cards to educate mothers and care givers on the 22nd and 23rd June 2005 ƒ IRC in collaboration with the DHMT conducted training for our seven new PHUs (in-charges and 10 vaccinators) on the 4th,5th, April 2005 on (a) EPI Defaulter tracing using the tickler system (b) calculation of targets and coverage for vaccine ordering purposes ƒ Distributed 2,050 treated bed nets to PHU communities on MOHS criteria Jul.- Sep.05 ƒ Current CS Prog Manager hired ƒ Current health coordinator hired ƒ Program exchange visit with CARE, USA Oct.- Dec.05 ƒ Exit of Expat CS PM ƒ Exit of BCC Officer ƒ 2nd Annual Report Delivered ƒ Held CBT strategic planning meeting on the 17th and 18th December 2005 ƒ Donated desktop computer to the DHMT to have one good reliable data base system in the district Jan.- Mar.06 ƒ NY TU visit ƒ In collaboration with the DHMT, developed the Community Based Distributors Curriculum ƒ Reorganization of roles and titles of project staff ƒ Start of CBT program ƒ 10 week radio broadcast of health messages (jingles) B-16 IRC Kono CS 19 Mid-Term Evaluation Report b. Progress by intervention areas Intervention areas fall under two main categories: child health on one side, and maternal and newborn care on the other. Within these categories, each section representing an intervention area will start with a summary table of progress on the main objectives, as measured through the KPC (graphic presentation is available in the Summary Figure of the Executive Summary). This will be followed by a review of the essential activities which contributed to the observed results, then assessment findings from the MTE, and conclusions. Update on contributions to USAID’s new PMP The Price of Equity The USAID Mission asked the IRC to base its child survival activities in Kono District for several reasons. Kono District was, like most of Eastern Sierra Leone, already neglected for decades before the war. As the heart of Sierra Leone’s diamond area, Kono was the site of particularly fierce and prolonged fighting during the civil war. During this period, the people of Kono had access to virtually no services, and by the time fighting ceased both the physical infrastructure and the human resource pool were largely gone. Kono suffered the highest amounts of displacement, and subsequently was one of the districts of highest IDP and refugee return. Thus starting child survival activities in Kono, one of the poorest and most devastated areas of one of the poorest countries on earth, helps to redress health status inequities overall but also within Sierra Leone. This choice promotes equity and is fully in line with the IRC’s commitment to serving people affected by conflict, but it has brought about considerable challenges. It has not been easy to hire qualified staff to a distant district with few schools. Transportation is a daily challenge, with some staff members having to spend hours on foot to reach remote health centers, and some areas inaccessible during the rainy season. Cost-recovery is almost impossible to implement given the generalized extreme poverty. As in other child survival projects, these obstacles are the inevitable price of equity. B-17 IRC Kono CS 19 Mid-Term Evaluation Report Childhood illnesses addressed by the IMCI strategy i) Immunization (20% level of effort) Table 2: Progress toward EPI objectives Objectives and indicators3 Baseline (2004- 2005) 4 Mid￾Term KPC (2006) 2008 Goal 1. Increase the proportion of children who are correctly immunized against tuberculosis, diphtheria, polio, tetanus, pertussis, and measles Indicator: % of children 12 to 23 months with recorded complete immunization before their first birthday 15% 30% 80% 2. Increase the proportion of children receiving supplemental vitamin A, which decreases mortality from measles and from other diseases Indicator: % of children 12 to 23 months who have received vitamin A within the last six months 4% 29% 50% ™ Increase coverage against yellow fever, as per MoHS policy. 30% 46% The project has closed 17% of the gap to complete immunization and 23% of the gap toward its target on this indicator.5 (It observes a similar 23% gap closure for Yellow Fever, a secondary indicator.) Progress will have to accelerate notably for the project to reach its objective on the main immunization indicator. The MTE provided useful analyses on important factors and possible solutions to this remaining gap. 3 The DIP presented 8 primary objectives (with specific indicators and EOP targets) and additional “secondary” objectives. In all following sections, primary objectives are numbered (1 to 8) and secondary ones appended to the table. 4 In the initial KPC survey (2004), full immunization coverage was 26%, as it only sampled in the first and more accessible 13 PHUs. Vitamin A coverage in those 13 PHUs was 15%. Numbers in this table combine all 20 areas to form a comparison baseline to the mid-term KPC. 5 The gap closure is used by USAID in its annual performance review of the CSHGP grantees. It takes the “gap” between a baseline measure and the maximal result (usually 100%) and calculates what proportion of this gap has been filled by the project. A gap closure of 30% is not unusual for most indicators by the end of a typical CSP. The gap closure to the project’s defined EOP objective can be examined at mid-term. By definition 100% of the gap to the project’s defined targets should be filled by the EOP. B-18 IRC Kono CS 19 Mid-Term Evaluation Report For Vitamin A, the gap closure to full coverage and to EOP target is correspondingly 26% and 54%. It is on track to reach this indicator by EOP. DHMT and PHU Colleagues review mid-term progress Contributing activities: ƒ Vaccines have been consistently supplied by the MOH and UNICEF. ƒ The DHMT’s Rapid Results Initiative (RRI) has created a lot of energy around this issue, which the project has contributed to and built upon. ƒ The CSP provided ad hoc support in logistics and distribution, both from Freetown to Kono, as from Kono to the 20 PHUs targeted by the project. ƒ The project has supported DHMT training of PHU staff in EPI, vaccine management, and the tickler system for identification of defaulters. It has also supported training by PHU in-charges of CHWs in defaulter tracing, and training on the dosage and timing of Vitamin A supplementation. ƒ The CSP organized the production and broadcast of a 10-week radio jingles campaign including EPI messages in Kono and Krio in March 2006. ƒ Finally, its ongoing field activities contributed to this area (monthly visits to all PHUs and surrounding communities / joint and independent supervision; joint (with PHU) and independent visit of CHWs/TBAs; coaching/on-job training; participation in and support to PHU outreach activities; and direct health education sessions in the communities by the CSP health promoters). Assessment findings: PHUs are generally stocked in vaccines and can manage the logistics of B-19 IRC Kono CS 19 Mid-Term Evaluation Report immunization at their facility level, with some gaps which may affect routine immunization. ƒ Based on the small sample of 8 PHUs visited during the MTE, with an arguably perfectible methodology, one PHU in eight (1/8) had a stock out on two observed antigens. The probability that at least one of 6 vaccines (DPT, BCG, Polio, Measles, TT, Yellow Fever) will be found missing in a PHU ranges from 4% to 17% depending on the method used (observation or registers). ƒ Most PHUs have a working cold chain equipment and those who do not had access to their own stock in neighboring more equipped PHUs. This is a workable arrangement for immunization campaigns and outreach until routine immunization can be offered in every PHU. ƒ All PHU in-charges or MCH aides interviewed during the MTE were able to correctly calculate the number of defaulters in their catchment area, and 5/8 could correctly calculate their antigen needs. (Six of the eight staff interviewed had benefited from training in EPI organized by the CSP/DHMT.) All PHUs are active in conducting outreach activities, although it sometimes falls below the useful level. ƒ PHUs have an informal working arrangement with vaccinators, which facilitate the organization of outreach. All eight (8) PHUs surveyed during the field assessment had a visible outreach plan with monitoring of the outreach activities. Figure 2 shows the frequency distribution of the number of monthly outreach efforts organized by these PHUs. For three (3) of these PHUs, the frequency of outreach (1-2 per month) was well below the minimum frequency needed to achieve adequate coverage (1/week or 4/month). Figure 2: Number of monthly outreach efforts organized by 8 PHU surveyed 1 4 3 0 1 2 3 4 5 10 4 1 to 2 Number of PHUs Access to first immunization is high and drop out from DPT1 to DPT3 is B-20 IRC Kono CS 19 Mid-Term Evaluation Report low. A large part of the immunization gap is due to untimely measles immunization, and untimely DPT1 immunization to a lesser extent (Figure 3). ƒ As demonstrated by the KPC 2006, and mentioned in earlier project reports, up to 30% points in the gap to full immunization is caused by inappropriately timed measles immunization (between 9 and 12 months). ƒ This can probably be largely explained by a combination of access problems and missed opportunities caused by the absence of routine immunization services available in all PHUs (as opposed to immunization days and outreach activities). At the household level, immunization did not rank high in the exposure of households to BCC messages, although it is recognized as important by the population. ƒ In the MTE rapid survey, recollection of immunization as a topic from health talks and other BCC activities only appeared four (4) times out of 102 spontaneous recalls. Completion of the immunization schedule was the spontaneously reported behavior following health talks in three (3) out of 48 reports. ƒ Community members cited immunization as an important contribution of CHWs in nine (9) out of 43 mentions. Figure 3: Immunization coverage by antigen6 90% 78% 80% 48% 35% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% BCG DPT1 DPT3 Measles Complete Not recorded No card Too late Too early Correct Discussion: The CSP has contributed actively in and supported the DHMT’s efforts 6 “Not recorded” means “absence of complete immunization” for the Complete bar. B-21 IRC Kono CS 19 Mid-Term Evaluation Report to expand full immunization and Vitamin A coverage. The project will be able to reach its EOP objectives if it can achieve the following: 1- Address the too frequent inappropriate timing of immunization (measles and DPT1). Untimely measles immunization accounts for nearly 30 percentage points in bridging the gap to complete immunization. 2- Go beyond training on defaulters tracing to actually identify, trace, and reach defaulters, by ensuring that they are brought to PHUs or to community outreach visits. This will require strengthening community and behavior change efforts to reach the 10% of the population who still do not access immunization services at all, and also a more general effort to motivate PHU staff to give more consideration to community and preventive activities. These two directions (community and PHU staff) work in synergy, as a more structured community and BCC strategy is one way to provide support to PHU staff, increase their motivation and job satisfaction generally, by being part in a successful strategy to bring change. This will have an impact on immunization but also on many other project objectives. 3- Examine opportunities for strengthening routine immunization, over and beyond outreach efforts. CSP BCC Manager presents the different communities of intervention B-22 IRC Kono CS 19 Mid-Term Evaluation Report ii) Control of malaria (25% of effort) Table 3: Progress toward Malaria objectives Objectives and indicators Baseline (2004- 2005) Mid￾Term KPC (2006) 2008 Goal 4. Increase the proportion of children less than 5 years of age who sleep under a correctly treated bed net Indicator: % of children 0-23 mos who slept under a correctly treated bednet the previous night 6% 40% 30% 5. Increase the proportion of women who receive intermittent preventive malaria treatment during pregnancy Indicator: % of mothers of children 0 to 11 months who got at least one intermittent malaria preventive treatment during their most recent pregnancy 0% (2004)* 54% (2005)* 60%* 30% ™ Increase the proportion of children less than five years of age who receive correct first-line treatment for presumed malaria within 24 hours. 11%7 * A note on IPT coverage: At the time of the first KPC, the MOH policy had not established SP as the drug of choice for IPT during pregnancy. The baseline for the first 13 PHUs is thus 0%. A new policy was adopted between the 2004 and the 2005 KPC, when seven additional PHU areas were assessed. At that point IPT coverage had already jumped to 54% for these 7 PHUs, as measured in the 2005 KPC which served as the baseline for the “new” areas. The 2006 KPC sampled in all PHU areas (“old” plus “new”). The difference between 2005 and 2006 is not significant, but of course the net overall result largely exceeds the original project ambitions. The project has already reached beyond its initial objectives for ITN coverage, thanks to aggressive efforts to join UNICEF and the DHMT’s efforts promoting free bednets for children completing their immunization and for women receiving their second TT immunization. Nationwide, UNICEF and the MOH report a bednet coverage having reached over 12% from 0.5% only two years ago.8 The DHMT and IRC’s efforts in Kono have notably exceeded this figure. Contributing activities: ƒ The project procured bednets and participated in the distribution of 7 28% of sick children with fever ultimately receive ACT treatment, but only 11% in a timely manner. 8 Interview with Dr Pat Roberts, MOH Director of PHC. Freetown. B-23 IRC Kono CS 19 Mid-Term Evaluation Report the UNICEF ITNs, through ad hoc support in logistics and distribution. ƒ It reinforced the bednet distribution policy at the PHU level (free to pregnant women after the second TT immunization, and free to children completing DPT3) ƒ Again, its ongoing field activities contributed to this area (monthly visits to all PHUs and surrounding communities / joint and independent supervision; joint (with PHU) and independent visit of CHWs/TBAs; coaching/on-job training; and direct health education sessions in the communities by the CSP health promoters). ƒ The project produced and sponsored a 10-week radio jingles campaign including malaria message in Kono and Krio in March 2006. Assessment findings: The project has an opportunity to show a substantial impact, by repeating the progress made in ITN coverage since its startup: ƒ The acceptability of use of bednets seems to be at a high level. National discussions are underway to balance a free net policy for priority beneficiaries and a social marketing approach for the rest of the population. The project is actively and closely following these discussions, although policies and strategies do not seem to be yet finalized. ƒ Some PHU in-charges are concerned about the continued supply of bednets, but the DHMT and UNICEF are confident that supply is not going to be a problem in the near future. MTE: documenting to learn – example of VHC meeting minutes The level of IPT coverage has been made possible thanks to the high utilization of ANC services, although gaps in quality at the facility level may be a constraining factor of further gains. It benefited from the adoption of a national policy in favor of SP during the first year of the project. Appropriate presumptive treatment of fever is only a secondary objective of the project. Given the burden of diseases caused by malaria in Sierra Leone and in Kono, it deserves a closer examination and should become a major objective of the project. B-24 IRC Kono CS 19 Mid-Term Evaluation Report Essential determinants of quality are in place in the PHUs surveyed during the MTE. ƒ ACT was observed as available in 7/8 PHU surveyed (5/8 according to the registers). ƒ A sampling of 10 registry records in each of 8 PHUs showed: o 88% compliance with standards for the distribution of ITNs; o 87% for ACT; and o 73% for IPT. But overall appropriate ACT treatment of children with fever is critically low. ƒ Although not measured at baseline, the percentage of children less than five years of age who receive correct first-line treatment for presumed malaria within 24 hours is at 11%. Nine children out of ten do NOT receive timely presumptive ACT treatment for their febrile episodes. ƒ Insufficient recording of treatment might contribute to this low number. Discussions with community members indicate that at least three factors continue to constrain improvements on appropriate treatment: o Physical access problems; o Financial access concerns; o Recourse to traditional (native) medicine. The CBT component of the project should play an important role, but its full implementation has not started yet.9 Discussion: The project and the DHMT have helped make Kono a success story in the national MOH/UNICEF free bednet strategy, and has supported the new policy in favor of SP for IPT during pregnancy. These are very positive result. The project has taken full advantage and maximized the benefits of national policies and efforts. For its two main malaria objectives (ITNs and IPT), the project has already passed its end of project objectives, which were set modestly at 30%. It has now an opportunity to continue this trend and consolidate these results. Impacting malaria could be a tremendous achievement of this project, and this first success can be used as the spearhead of efforts to strengthen the other areas. Resetting the 9 More details on the status of CBT implementation in the section on IMCI implementation. B-25 IRC Kono CS 19 Mid-Term Evaluation Report project objectives for ITN and IPT coverage was discussed during the evaluation. It would be a reasonable decision, but one that must be taken considering all the areas where the project is being asked to make additional efforts. iii) Control of diarrhea (20% of effort) Table 4: Progress toward Control of Diarrhea objectives Objectives and indicators Baseline (2004- 2005) Mid￾Term KPC (2006) 2008 Goal 3. Increase the proportion of caregivers who wash their hands with soap on key occasions Indicator: % of mothers of children 0 to 23 months who wash their hands on 4 key occasions (before food preparation or feeding child; after defecation or cleaning the child) Hand washing on 2 occasions. 2% 45% 3% 64% 30% ™ Proportion of sick children who receive increased fluids 28% 15% ™ Presence of soap in household 51% 79% ™ Increase the proportion of children less than five years of age whose diarrhea is treated with oral rehydration fluids. (ORS) ™ Increase the proportion of children whose intake of food and fluid is not reduced during illness. ™ Improve the proportion of children with diarrhea who receive zinc. ™ Increase the quality of care at PHUs for children with diarrhea. NA10 Contributing activities: The main project activities supporting this intervention have been: ƒ Monthly visits to all PHUs and surrounding communities with joint and independent supervision. ƒ Joint (with PHU in-charges) and independent visits of CHWs/TBAs; coaching/on-job training; participation in and support to PHU outreach activities. ƒ Direct health education sessions in the communities by the CSP health promoters. ƒ Production and diffusion of a 10-week radio jingles campaign including diarrhea message in Kono and Krio in March 2006. 10 KPC 2006 measure for ORS was not collected correctly and cannot be reported. B-26 IRC Kono CS 19 Mid-Term Evaluation Report The project has developed flashcard messages, including themes on diarrhea, to support community-level health education. The messages were developed based on some formative research and examination of practices and determinants of behavior. These materials have however not been produced yet, and still require insertion of graphic material, format definition and printing. Assessment findings: Message recollection is low as expected for handwashing, but quite low for the life-saving practice of oral rehydration. ƒ Diarrhea was mentioned as a key message 8 times out of 50 (16%) when household members interviewed during the rapid MTE survey were asked to recall main lessons learned from health talks. Handwashing and preparation of ORS/ORT were recalled 7 times out of 48 behaviors reported as following participation to health talks (respectively 6 times for handwashing and 1 time for rehydration). ƒ Handwashing practices have increased but only as measured on two occasions. The project’s main indicator (handwashing on four occasions) remains close to nil. ƒ Objectives in terms of promoting oral rehydration are only secondary and indicators have not been consistently measured. One indicator provides useful information however. Increasing fluid intake of the sick child remains low (from 28% at baseline to 15%). Although ORS is available, it is not systematically used in health facilities (or its use is not recorded). Checking project achievements with mothers ƒ ORS was available in all PHUs visited for the MTE rapid survey, but was only given to half the children with diarrhea, at least according to registers. Zinc was just procured by the project and has not been distributed yet. Discussion: The project’s impact on diarrheal diseases is essentially in the form of some hygiene improvements (presence of soap and partial progress on hand washing). There is no progress on promoting systematic increase B-27 IRC Kono CS 19 Mid-Term Evaluation Report in fluid intake of children with diarrhea. Although the baseline KPC indicated a high knowledge about ORS or SSS (76%), and ORS packets are widely available commercially, the only indicator measured on both occasions (increased fluids for the sick child) gives a worrying impression of what the situation is. While hand washing on four occasions is recommended, even the project’s baseline KPC suggests that some benefit of partial compliance can be found, particularly for children from 6 to 15 months. The project should continue promoting hand washing broadly, but may consider more specific action-research efforts to understand who washes hands, what the factors are (how are people dealing in the absence of water pumps for example), and what occasions are prioritized for hand washing. Based on this it could have more specific communication efforts to targeted groups. The project is not set up to conduct Operations Research effectively, but it could consider a Trial of Improved Practices (TIPS) approach, once its BCC component has been put on tracks (see cross-cutting section). Additionally, handwashing is constrained by water access problems (available from village pumps), and possibly customs and traditions, in spite of a heavy emphasis of health educators on general hygiene principles. It is noteworthy that this indicator has increased for 2- occasions of hand washing. This could reflect a change in motivation and intention, which has not yet overcome all constraints and reached the 4-hand washing target. The increased presence of soap suggests a similar process. The very poor water and sanitation conditions in which the population lives call for joint or integrated programming to try and address the behavioral and the environmental sides of the equation at the same time. In the meantime, there is one more benefit of exclusive breastfeeding in avoiding contact with fecal pathogens. However, even with successful promotion of frequent hand washing on all required occasions, the evidence suggests that the impact on diarrhea might range from 10% to 40% reduction in incidence. 11 11 According to the TRMs (www.childsurvival.com), a recent analysis of 21 controlled field trials related to point-of-use water treatment and safe water storage at the household level showed a reduction of 42 percent in diarrheal diseases compared with other groups. (Clasen and Cairncross, 2004). A recent literature analysis found that the single hygiene practice of hand washing with soap is able to reduce diarrhea incidence by over 40 percent (Curtis and Cairncross, 2003). Another recent study by the WHO/UNICEF Joint Monitoring Program concludes that safe water, basic sanitation and hygiene may account for as much as 88 percent of the disease burden due to diarrhea (WHO and UNICEF, 2000). However, a WHO review of the evidence (Hill et al., 2004) on mother and child health behaviors only claims 10 to 26% reduction in B-28 IRC Kono CS 19 Mid-Term Evaluation Report Considering the current situation (environmental and the difficulty in translating increased handwashing and presence of soap to systematic handwashing on 4 occasions), this will leave a large number of children vulnerable to diarrhea related complications and fatality. Improvements in access to water and sanitation (as achieved by a prior IRC project which left about a quarter of households in the target areas with access to water pumps) might yield larger impact, as might widespread zinc treatment, which reduces severity and duration by 25% and mortality by 50%, and which is more likely to be implemented successfully in the short term than the current behavior change strategy. Zinc was procured by the project around the time of the evaluation, and should be dispensed through the CBT strategy as through facilities. PHU staff will not be supportive of Zinc if it is a treatment available to CBDs and not to them. Part of improving PHU staff motivation is ensuring the perception that the community strategy reinforces their efforts, rather than undermine them. These provide directions for accelerating impact on diarrhea morbidity and mortality in the years remaining to the project. diarrhea prevalence based on the promotion of handwashing. http://www.coregroup.org/working_groups/imci_Evidence_review.pdf B-29 IRC Kono CS 19 Mid-Term Evaluation Report iv) Pneumonia case management (PCM) (15% level of effort) Table 5: Progress toward Pneumonia Case Management objectives Objectives and indicators Baseline (2004- 2005) Mid-Term KPC (2006) 2008 Goal 6. Increase the proportion of caregivers who recognize breathing-related danger signs Indicator: % of mothers of children 0 to 11 months who recognize rapid or difficult breathing as a danger sign for childhood illness 7% 30% ™ Increase the proportion of children less than five years of age with difficult breathing who receive correct treatment within 24 hours from authorized providers 5/12 (~42%)12 Contributing activities: The main project activities supporting this intervention have been those common to the other intervention areas: ƒ Monthly visits to all PHUs and surrounding communities with joint and independent supervision. ƒ Joint (with PHU in-charges) and independent visits of CHWs/TBAs; coaching/on-job training; participation in and support to PHU outreach activities. ƒ Direct health education sessions in the communities by the CSP health promoters. ƒ Production and diffusion of a 10-week radio jingles campaign including pneumonia messages in Kono and Krio in March 2006. Assessment findings: Problems exist with the indicators of performance for PCM: ƒ As for all projects the percentage recognition of fast or difficult breathing as a danger sign in the sick child is not on its own a satisfactory outcome measure, although it might be a necessary intermediary step to care seeking and proper treatment. The main issue is that it failed to be collected at mid-term. It might have effectively disappeared from the project’s radar screens until the mid-term. ƒ The secondary indicator (% children under five with difficult breathing receiving appropriate treatment) was not measured at baseline. It is also difficult to measure with precision as the 12 N is too small to consider this a valid estimate. B-30 IRC Kono CS 19 Mid-Term Evaluation Report denominator is small. In the MTE KPC, the denominator is only 12 children (only 5 of which were treated appropriately).13 There are important quality gaps in the management of ARI at the PHU level: ƒ Cotrimoxazole was available in all 8 PHUs surveyed during the MTE rapid survey, but the respiratory rate of children presenting with difficult breathing or cough was only recorded 29% of the time in the review of registry records of the MTE rapid assessment. And none of the PHUs were equipped with timers.14 Debriefing with DHMT in front of Paramount Chiefs At the community level, pneumonia-related topics (such as rapid breathing, difficult breathing) was never mentioned as a key message when household members interviewed during the rapid MTE survey were asked to recall main lessons learned from health talks. Recognition of danger signs for children was less than 50% for CHWs and TBAs encountered in the MTE rapid survey. At the project inputs level the procurement of timers to measure respiratory rate was bogged down in internal IRC logistics and procurement. Started late (in the last 6 months), this procurement had not been completed at the start of the MTE. Discussion: There is little information to assess what, if anything, the project has achieved at the population level in this area, although it may have supported a general effort in clinics to improve management of the 13 This indicator can be measured with reasonable accuracy, but this needs to be planned in the design of the study with an appropriate stratified sampling design. Even then, it can be difficult to get valid data given the biases such as the respondent’s desire to give the answer most pleasing to the interviewer. 14 200 timers have been procured since the end of the evaluation. B-31 IRC Kono CS 19 Mid-Term Evaluation Report sick child. The strategy to reduce pneumonia-related mortality needs to combine both family and community practices and quality of care at the point of service, usually PHUs. The introduction of CBT offers the project with an opportunity to drastically energize this area. Certain steps should be taken: i. Improved supervision and monitoring can track appropriate treatment at least at the PHU level. (Measure of the respiratory rate may be higher than the register-recorded 29%, but questions about the quality of treatment (compliance) remain warranted.) ii. In the 9 PHUs where community-based treatment is being initiated with CIDA funding, supervision, monitoring and reporting must be set to track compliance effectively on a monthly basis. iii. At the community level, in CBT and non-CBT PHU areas, repeated assessment (LQAS) and monitoring can help correct the limitations of small sample size in traditional surveys. Specific recommendations are made in the relevant section (starting with reviewing the main indicator for this intervention), but overall, PCM is an area, which will only improve through coordinated efforts in all cross-cutting areas: BCC, QA at facility level and the new opportunity opened by CBT. B-32 IRC Kono CS 19 Mid-Term Evaluation Report v) Implementation of the IMCI Strategy Malaria control and treatment, Pneumonia case management, Control of Diarrhea and the Immunization program are essential elements addressed by the IMCI strategy. At the national and district levels, IMCI is a national policy of Sierra Leone.; its implementation started last year. The DMO is a trainer in IMCI and has trained all district PHU staff with the support of UNICEF. But pending the availability of assessment forms, reporting tools, and systemic changes (starting with adoption of appropriate professional practices), systematic implementation of IMCI has yet to take place. This translates at the next level. For example, in the MTE rapid survey, less than half the CHWs and TBAs surveyed were able to identify the essential danger signs of the sick child, which are emphasized in IMCI protocols. In fact, the CSP health promoters themselves have not been trained in IMCI. While they have the nursing background and qualification to do their work and are familiar with many (maybe most) IMCI features, there has been no mechanism to insure that they are fully comfortable with the IMCI standards and able to supervise their implementation. For facility level work, neither DHMT nor CSP have the supervision checklists, teaching aids, or reporting forms corresponding to the IMCI algorithms (as noted previously timers are also missing). Finally, the HIS used by the project (and DHMT - see HIS section), continues to track basic delivery of services and facility statistics, but does not readily produce management-triggering information such as: ƒ number of defaulters; ƒ defaulters identified and treated; ƒ proportion of respiratory illnesses correctly diagnosed and treated by antibiotic; ƒ consumption and stockout of essential drugs; ƒ frequency and content (focus on essential messages) of PHU outreaches; ƒ Etc. A more systematic approach of the IMCI strategy would bring certain benefits to the project: ƒ As will be discussed in the cross-cutting section, the project’s BCC efforts are possibly too “scattered” or diluted to reach impact. A focus on IMCI tools and messages can provide a way to solve this B-33 IRC Kono CS 19 Mid-Term Evaluation Report problem. ƒ Important issues for the health of the newborn and the child, for example breastfeeding (with 29% early breastfeeding and 36% exclusive breastfeeding at baseline), which was discussed in the DIP, can be addressed through IMCI, without detracting the project from its core objectives. ƒ Health information can be produced to track implementation of and compliance with IMCI and to trigger management / supervision responses. ƒ The DMO would be highly supportive and the district would greatly benefit from the project serving as a spearhead in developing and field-testing actual IMCI materials. ƒ Such materials would also help structure the child health component of supervision visits, at both PHU and community levels. ƒ It will be important to advance both facility and community IMCI. This will require the full support of the DHO and DHMT to energize PHU staff. There is an opportunity for doing this, as the DHO is very much in support of this. Community based treatment (CBT) IRC has accessed funding from CIDA as part of its match to USAID’s grant to implement community-based treatment in nine of its 20 PHU areas. Implementation (defined as treatment of sick children at the community level) has not started yet, but initial steps have been taken: ƒ Preparation meetings with partners and early planning steps were conducted as scheduled. A storage room has been built in the CSP office. ƒ 133 CBDs (27 female, and 106 male) have been trained in 9 PHUs. ƒ Some of the possible candidates to fill the CSP positions opened by this component have been identified. ƒ Procurement of Zinc and essential drugs is underway. Recruitment of CBDs was based on criteria developed by IRC and partners (DHMT, Pharmacy Board, Local council and others). According to the project team, literacy was established as a criterion based on conditions of acceptability to the MOHS, given concerns about malpractice of illiterate people (e.g. Quacks) and the risks incurred. However, although non-literate people are often excluded on the assumption that they cannot report accurately and are less likely to provide high-quality care, this assumption has often been disproved by experience. B-34 IRC Kono CS 19 Mid-Term Evaluation Report The majority of women in these communities are illiterate (e.g. TBAs) and CHWs are predominantly men, leading to a high proportion of male CBDs. This could be a concern for the future. This component could not be further assessed but it has a lot of potential: ƒ This component has strong support from the DHMT, and would be scaled rapidly if successful; ƒ By operationalizing an important component of C-IMCI, it has the potential to accelerate the impact of the project, particularly as access problems continue to constrain care seeking by care takers; ƒ This is the level at which the community strategy can take a clearer shape. The challenge for the project will be not to neglect the PHU areas where CBT is not yet implemented. (It is ultimately expected to cover the 20 IRC supported PHUs.) Update on contribution to USAID’s new PMP Scaling up Community-Based Treatment The child survival program is still in the process of setting up and refining its key activities, but the recently begun community treatment program shows great promise for future scale-up. A number of child survival experts have suggested that scale-up works best when the activity is designed from the start for scale. Community treatment has several elements that will make future scale-up more feasible: First of all, it answers a need that is felt in all of Kono District –with a population of over 500,000– and most of Sierra Leone. Secondly, the activity has been designed with policy clearance and support from the Ministry of Health at both district and national level. Thirdly, the IRC is drawing on its worldwide experience, including that of projects already scaling up in Rwanda and DR Congo, to design a set of simple, inexpensive, and field-friendly tools that will make training and supervision feasible on a wide scale. Last, but not least, the IRC has already identified partner NGOs who are interested in participating in scaling up, including World Vision in Kono District and CARE in Koinadugu District. B-35 IRC Kono CS 19 Mid-Term Evaluation Report Maternal and newborn care (20% level of effort) The project’s MNC interventions can be divided into two categories: 1) Interventions ultimately targeting a reduction in maternal mortality, essentially skilled birth attendance and referral of complicated deliveries; 2) Interventions improving antenatal to postnatal care and the health outcomes of the newborn/young infant. Challenges for these two types of interventions vary and they are considered in order. vi) Targeting Maternal Mortality Table 6: Progress toward maternal health objectives Objectives and indicators Baseline (2004- 2005) Mid-Term KPC (2006) 2008 Goal 8. Increase the proportion of women experiencing delivery complications receiving care from a skilled attendant Indicator: % of mothers of children 0 to 11 months having experienced a serious delivery complications who delivered at a health facility 27%15 35% ™ Poportion of women who delivered with the help of a skilled birth attendant16 17% 12% The indicator defined for this objective is not one that can be measured easily or reliably by the standard KPC. Its measure is not available for the MTE. It would require an ad hoc study or a census￾based registration system. The MTE assessment provided valuable information on this component however. Contributing activities: The only project activities that might lead one to expect an improvement of the situation are those which support improved 15 Estimate based on small sample. 16 This indicator was not defined by the project as a secondary indicator. The two values (17% and 12%) can be considered an unchanged situation. B-36 IRC Kono CS 19 Mid-Term Evaluation Report referrals. These are in the form of essential inputs at the PHU and DHMT levels: ƒ VHF Radios have been procured for 11 PHUs of the original 13 PHUs, as well as for the Koidu government hospital in addition to radio relay to cover the distant PHUs. DHMT has very reasonably requested that power be provided by solar panels. The IRC has agreed to provide these, and identified the necessary funds. The panels are currently being procured. The project is also considering budget implications for providing radios to the remaining PHUs. Finally, radios have to be appropriately programmed, as this was not done before they were brought to Kono and installed in the facilities. ƒ An ambulance was procured, is now in Kono and has been used by the hospital. It was transferred to the DHMT shortly after the MTE. Assessment findings: Apart from the procurement steps described above, essentially nothing has happened to impact this component of the project. ƒ There is no indication that management of complicated cases has improved.17 ƒ General skilled birth attendance for all pregnancies has not progressed. ƒ No relevant monitoring and information systems have been put in place. ƒ The project’s maternal health strategy itself is not clearly defined. 17 In some areas, the project has put efforts in the revitalization of community structures and positive traditional practices in place before the war (Hammock System, sometimes modified as a “Blanket System” for lack of local traditional materials to make the hammock) through sensitization. (Formerly hammock was only used to carry very important persons Chiefs or the dead; the project reports that with CSP inputs this is now used for evacuation for referrals.) The extent to which this has occurred was not examined during the evaluation, but appeared modest from the feedback of community groups, community structures and TBA/CHWs. B-37 IRC Kono CS 19 Mid-Term Evaluation Report The MTE team spent some time identifying elements of the safe childbirth and delivery issue: ƒ Central role of TBAs: o Most deliveries are conducted by TBAs (64%; KPC 2006), and o Even in PHUs 2/3 of deliveries are conducted by TBAs. o There is some awareness of danger signs from TBAs and communities, but gaps are important. o Only 5 of 27 TBAs interviewed in our rapid survey reported actual referral of obstetric emergencies. They reported three cases of maternal death and three severe complications while they conducted the delivery themselves. In none of these six cases was a referral discussed. o TBAs are very popular, even among MOH staff, and receive considerable support from the community (in some cases through the construction of TBA houses). In summary, TBAs seem to have a central and unavoidable role but, at the moment, are under-used as allies in improving maternal care. ƒ Quality of skilled birth attendance: o SBA is defined as a delivery performed by a nurse or an MCH aide in a PHU setting. But there is little data on the actual skills of MCH aides in the district. o Four of the eight PHU visited had no delivery kit in stock. o Although no evidence is available, one of the factors proposed for explaining the proportion of TBA deliveries in facilities might be the greater experience of some TBAs and the lack of self-efficacy of MCH aides in their supervision. o The DHMT hopes to double the number of MCH aides available in PHUs over the course of the coming years. 20 new MCH aides having completed their training are expected to integrate the district in the summer 2006. o In our rapid survey, infection and hemorrhaging represent half of the complications seen at the PHU level (49%). But the data tentatively indicate that a number of complications are possibly missed (47 complications were reported, while the expected number (15% of deliveries) would place this at 86). In summary, there is little information on complications of delivery or on quality of care provided by trained PHU staff, and there are questions about the skills of the staff. ƒ Transport from community to facility and even referral from facility to district hospital remain important limiting factors. o The DMO reports a recent maternal death when he was B-38 IRC Kono CS 19 Mid-Term Evaluation Report contacted by cell phone for a case of prolonged labor and could not send any vehicle to transport the woman. o The operating theater of the district hospital has been closed for 6 months due to construction work. This in turn has substantially prolonged transport times and costs as patients need to be transported to another hospital two hours away. o The project has no record of actively supporting or organizing emergency transport plans at the community level. A number of communities use hammocks or blankets to transport women. Discussion MCH aides in PHUs are considered “skilled birth attendants,” but TBAs continue to perform the vast majority of deliveries. While TBAs may be “hard to control”, they are an ubiquitous presence at the community level, largely female (unlike the CHWs), dynamic and motivated, and gathering a lot of community support. Additionally their role has already been institutionalized by the PHUs themselves where they conduct 2/3 of the deliveries. The project’s strategy will be more effective if it includes a role for TBAs, even has the ultimate goal is to advance skilled birth attendance. This represents a tension for strategic design but one that cannot be avoided. Working with TBAs can be viewed as an intermediate step to increase skilled birth attendance for complicated deliveries in the short and medium term. One of the project’s strategies is to encourage community funds in collaboration with DHMT & PHUs. The MTE did not assess this intervention.18 The IRC has an EmOC project in Kenema, and some of its staff participated in the MTE. It is clear that there are natural linkages that need to be strengthened here, just as better integration with other IRC project can be beneficial (for example for community strategies). Strengthening and sometimes creating those linkages will require higher level (country program) intervention as the team is quite 18 In this strategy, a stipulated amount from cost recovery is allocated for community by the DHMT, and also proceeds from bednet sales (ITNs supplied by IRC – free for DPT3 & TT2 but sold at minimum cost to those who did not fall into this free category for maximum coverage to target beneficiaries) is also encouraged to be kept in for use in supporting health related activities. This fund is now being used to open chiefdom savings bank account for each PHU. It has been a slow process but still modalities have been put in place by the DMO and partners. B-39 IRC Kono CS 19 Mid-Term Evaluation Report stretched. In a later section, recommendations are presented for: 1. revising the project objectives 2. emphasizing skilled birth attendance at the PHU level 3. working with rather than against TBAs, to improve maternal care 4. making incremental progress in the management of referrals of complicated deliveries vii) Antenatal, Postnatal and Newborn Care Table 7: Progress toward maternal health objectives Objectives and indicators Baseline (2004- 2005) Mid-Term KPC (2006) 2008 Goal 7. Increase the proportion of women who receive at least 2 TT immunizations Indicator: % of mothers of children 0 to 11 months who have received at least 2 recorded TT immunizations 39% 43% 60% Maintain the proportion of women who take iron supplementation in the first month post￾partum to decrease the prevalence of maternal anemia. Improve the proportion of mother-children pairs for whom four essential actions (mother: check for fever and anemia; child: check weight and breathing) are taken within a week of birth. Increase the proportion of newborns who are put on the breast within an hour of birth. Increase the proportion of women given vitamin A within 6 weeks of delivery.19 35% 29% 49%20 31% 56% Table 7 shows no improvement on the indicators measured. Contributing activities: Apart from ongoing visits to PHUs, CHWs and TBAs and general information to the communities visited, there are no specific activities contributing to these objectives. Assessment findings: Data on ANC utilization (Figure 4) shows 60% of mothers having at least attended one PNC in last pregnancy as per card, and close to 19 The DIP label for this objective/indicator is “within one month of delivery”; the indicator measured in 2005 and 2006 was as stated in the table (6 weeks). 20 Data from KPC 2005 only. B-40 IRC Kono CS 19 Mid-Term Evaluation Report 100% per maternal report. (The self-reported proportion of women going for ANC appears constant, although there does seem to be an increase in the proportion of documented visits). There is a disconnect between the prevalence of at least one ANC contact per women, the high ANC distribution of FeFol (Iron and Folic Acid) at the PHU level (Figure 5), and the low population coverage with iron. Figure 4: Number of prenatal visits attended, per card and per maternal report. Source: 2005 and 2006 KPC surveys21 0% 20% 40% 60% 80% 100% Maternal Report 2004 Maternal Report 2005 Maternal Report 2006 Card￾confirmed 2004 Card￾confirmed 2005 Card￾confirmed 2006 % of respondents 0 1 2 3+ According to our rapid survey, quality of ANC/PNC is encouraging. Compliance with essential ANC/PNC standards ranged from 60% (albendazole during ANC) to 97% (measure of birth weight) (see next figure), although the dropout between women attending one ANC visit and those attending all required visits shows room for improvement. 21 2004 is only for “old” and 2005 is only for “new” PHUs—in both cases these are the baseline figures. B-41 IRC Kono CS 19 Mid-Term Evaluation Report Figure 5: Percent compliance with ANC/PNC standards based on a review of 10 registry records in the 8 PHUs of the MTE rapid survey 68% 95% 73% 86% 97% 62% 78% 0% 20% 40% 60% 80% 100% ANC deworming ANC FeFol ANC IPT ANC TT stock PNC birthweight PNC Fefol PNC VitA In the MTE rapid survey, recognition of danger signs for pregnancy and labor was unsurprisingly low in the general community (leaders and household members). More troubling is that a majority of TBAs and CHWs failed to identify correctly the danger signs (TBAs tended to do a little better than CHWs). This is comparable to the findings of the QA exercise conducted in 2005 by the CSP in one of the PHU areas, and corroborates our earlier finding on insufficient referral of complicated pregnancies and deliveries. As Figure 6 shows, lessons learned recalled by community members interviewed during the rapid survey of the MTE, only rarely involve the importance of delivering in a facility, and very little else. Discussion The issue in this area is one of lack of mobilization on the project’s side. While quality improvement is needed in facilities (where TT coverage remains low in spite of the availability of TT stock), it is also critically needed in communities, where the main issue is in improving outreach, notably through CHWs and TBAs. Considering the dire situation of child survival in Sierra Leone and in Kono, the next steps should focus on simple tested achievable results. This should drive the revision of the objectives for this area, eliminating those that are too complex to pursue at this stage (and have not been measured in the past). B-42 IRC Kono CS 19 Mid-Term Evaluation Report Figure 6 Themes of lessons learned recalled by community members during the MTE rapid survey22 0 5 10 15 20 25 30 Hygiene￾related IMCI-related messages Breastfeeding General/vague PHU deliveries HIV Compliance with treatmt 2. Cross cutting strategies a. Health Information System (HIS) Development and management of an effective HIS is one of the strategies of the project. It is also an essential strategy for capacity building and reaching all other objectives of the project. As stated during the evaluation, the objective is to operate a Health Management Information System with the DHMT. For the moment the project has collected monthly service data, which have been used to compile quarterly and annual reports. Some modest results have been achieved in producing information out of this. For example the project related service statistics related to census data about the catchment area of PHUs to track immunization coverage and drop out rates versus a 100% or 80% target. Another positive element is now the integration of the project’s HIS with the DHMT’s, so that only one data process is in place. This of course comes with a loss of control about the timeliness of the reports, which the CSP team, in particular the M&E manager, has had to deal with. The DHMT is highly appreciative of this step, and the accompanying transfer of a computer with capacity building activities led by a project intern in the use of databases. However, at this stage, the project does not produce and does not have the capacity to produce the kind of information required for making management decisions. For example, a lot of the most 22 N=106 mentions of topics by 66 household members selected during the rapid assessment through opportunity sampling. B-43 IRC Kono CS 19 Mid-Term Evaluation Report meaningful information gathered during the MTE rapid survey, could be produced routinely through a functional HIS. Some of the anecdotes of community conflict resolution on the part of the CSP health promoters are reportedly related to subsequent improvement in referrals for example. There is no capacity to infirm or confirm these stories based on the available information system. The merging of databases with the DHMT is fundamentally a good thing, but will only lead to frustration about administrative delays, unless the project can also help make the HIS more supportive of management decisions, by producing information out of data. This requires higher capacity than that currently available, by the admission of all actors. Questions and concerns have been raised about the appropriateness of PDAs in the Sierra Leonean environment. Although the evaluator is not a specialist23, some points can be made about this: - PDAs have been used in CS projects in other very harsh environments without defaulting; - Any information system (working at the level of a district) will require a degree of comfort with technology. The PDA technology is probably only marginally more advanced than commonly available tools (table PCs); but more advanced technology is also sometimes more user-friendly (i.e. as moving from DOS to Windows was); - On the other hand, PDAs eliminate an entire step of data processing, which is tedious, time-consuming, and error inducing; - In the end, the main issue facing the project is one of data management capacity, irrespective of technology. The CSP needs urgent support in developing clear monitoring tools, and in producing working templates of analysis for the HIS data. To build on a potentially winning strategy, particularly for the long term, this should be done with and within the DHMT. b. Communication for Behavior Change (BCC) Two observations can be made from considering the BCC elements related in the previous technical sections: ƒ Inputs into BCC have been limited. 23 IRC has access to a number of experts on this issue, working in challenging areas, through CSTS+ and the CORE group. B-44 IRC Kono CS 19 Mid-Term Evaluation Report While health promoters invested considerable energy reaching PHUs every week and from there reaching the most remote villages, frequently on foot24, there is only so much five (now four) people can do themselves. CHWs and TBAs have been met and encouraged, but have not been organized, directed and mobilized to achieve impact. Use of communication tools has been limited to a worthy 10 week radio campaign, with only minor financial sponsoring of the radio station. Our rapid survey conveyed the sense of some recall of the campaign, which had taken place just a month or two before. The production of flashcards in which the project team has spent notable energy, starting with formative research, then development, then testing was mentioned as an after-thought in the evaluation process. The production of this tool has not been finalized, and planning for its distribution and the method according to which it should be used is not clearly defined. ƒ The impact of the project seems to have been somewhat diluted by other messages at the community level. As Figure 6 shows, spontaneous recall of messages by community members relates to a project objective less than half of the time. (We did not assess whether even then, the message recalled fit specifically one promoted by the project.) Relevant messages recalled were most often not specific. There is not a cohesive BCC strategy focusing on covering a few, key essential messages. The project needs to find a way to intensify this area, and have a method (and tools) allowing cycles of communication and learning. One of the directions discussed during the MTE is to adapt CARE sister project’s strategy in forming community groups (community health clubs) to act as a catalysts in motivating community members individually or as groups in healthy behavioral practices linked with IRC/CSP intervention. The Community Health Clubs could be formed from representatives of existing committees within each village. c. Community Mobilization The project has worked with Chiefs, Paramount Chiefs, VDCs, CHWs, etc. Health promoters are well known and introduced in all communities. They have also supported and encouraged community 24 The 3 male promoters used motocycles (there are now only 2), but females are usually dropped at the PHU and from then proceed by foot if the car cannot stay with them. B-45 IRC Kono CS 19 Mid-Term Evaluation Report outreach by PHUs. No community structure is however organized and mobilized to facilitate or amplify the efforts of the project. One of the findings of the assessment has been the relative weakness and lack of mobilization of the VDCs. Their position as a potential partner for community mobilization is even more tenuous due to the fact that they lack recognition by traditional chiefs, who see them as competing for power. At the time of the MTE, the DHMT has pronounced itself in favor of engaging communities more aggressively through the Ward Committees, which are closer to communities, have access to the local councils’ and their resources and thought to be better partner. This will deserve careful consideration by the project. Community mobilization can play an important role for the project. The debriefing of the MTE led to a number of discussions of the pros and cons of an outreach model (strengthening efforts to reach from the health facilities to the communities) versus a community organizing model (working with community groups to build a community counterpart to the health professionals). The next table summarizes the plus and minuses of both approaches at this juncture for the project. Table 8: Contrasting Outreach and Community Organizing models Advantages of an Outreach model Advantages of a Community Organizing model Defaulter tracing Emergency transport and funding Health awareness Systematic registration efforts Supervise community treatment Sustain community treatment The close-distant community The distant-distant community “Pushing” services “Pulling” for quality We know where to start Who is the partner? VDC? Ward? TBAs? CHW networks? The consensus was that: ƒ The project is well placed to accelerate its outreach efforts, particularly through the CBT component, in addition to its past investment at the PHU level. Launching an extensive community mobilization effort now would be beyond the available resources of the project. ƒ For the long-term, community organizing would have specific advantages for accelerating progress, achieving objectives which are not going to be met solely through a facility + outreach strategy B-46 IRC Kono CS 19 Mid-Term Evaluation Report (see Table 8), and for sustaining achievements. It would additionally line very well with the USAID Mission’s current strategy, which emphasizes governance at the local level. ƒ IRC has other projects in Kono, which might be able to collaborate with the CSP. The SCOPE community mobilization program notably overlaps with the CSP in five villages. It is unfortunate that not more overlap was introduced in the design stage, but these five villages might be a testing ground for the remaining years of the project. (This overlap is small (5 villages out of 270 plus), so would not immediately lead to scaled impact, but could benefit learning by the IRC and the projects.) d. Capacity building and quality assurance - PHU and CHW i) Strengthening the PVO organization The CSP has a number of characteristics which make it a strong fit within IRC, but which also open opportunities for learning and strengthening of the organization. The developmental, partnership and capacity building nature of the project helps the organization move from a tradition of emergency assistance to a more long term development focus. By focusing on managing for results, and managing for sustainable results, the project can help the organization not only address acute community needs, but also emphasize using evidence in doing so. Other elements which IRC SL tries to advance are potentially being supported through the project. Community action planning will be informed by the CBT component and potentially an increase in community mobilization; while the IRC Program Framework, a conceptual tool, is being strengthened through the project’s emphasis on sustainable health outcomes.25 25 There are a number of other ways in which the project could help advance the IRC framework, if it can be implemented more effectively, for example in terms of: (i) Principles: *Rights: by promoting the right to health, and also by improving birth registration; *Participation: by empowering community providers to provide key services; *Capacity-building: by focusing on building the capacity of community, PHU, and DHMT partners; *Partnership: by implementing the project through local partners, rather than directly, and by involving national and local partners in the design, monitoring, and evaluation of the program; *Holistic programming: by linking with community development programs, or by liaising with World Vision integrated development efforts. (ii) Aims: *Saving lives: by decreasing maternal and child mortality; *Strengthening institutions: by building the capacity of the DHMT, among others, and providing a B-47 IRC Kono CS 19 Mid-Term Evaluation Report This is for the strengths and potential strengths. Opportunities for improvement and further strengthening of the organization are discussed in the Management section. ii) Strengthening local partner organizations DHMT The CSP has developed a strong and mutually profitable partnership with a highly energetic, competent and dedicated DMO and the DHMT. Achievements and areas for improvements (notably HIS and linking management to information) have been discussed in the technical sections. This partnership and strengthening of the local DHMT carries great promises for sustaining results. Update on contribution to USAID’s new PMP Civil Society Development The Child Survival Program is promoting civil society development by working with village leaders and other community members, and empowering community members to organize themselves and to provide key maternal and child health services. For example, the community treatment project will increase the capacity of community distributors to offer life￾saving treatment for common, deadly childhood illnesses, and will also increase the capacity of community leaders to monitor and respond to endemic and epidemic diseases. CHW networks The project initially planned to develop and support networks of CHWs. No specific activities have yet been implemented in this area. Opportunities for improving on this exist: ƒ CHW networks already exist. In our rapid survey 5/25 CHWs and 15/27 TBAs already belonged to network or associations. ƒ The launching of CBT will create a group of CBDs, which will have to be regrouped on occasion, for training, problem solving, etc. better HMIS. (iii) Promoting social cohesion: *by increasing community participation in health, and *by promoting the role of women as CBDs. B-48 IRC Kono CS 19 Mid-Term Evaluation Report ƒ As seen in previous sections, areas for improvement of CHWs/TBAs’ performance are numerous. Whether for training or launching new efforts (identification of danger signs, organizing emergency transport, intensive BCC efforts) CHWs and TBAs will be brought together on occasion. These occasions could open a door to strengthen the organization and networking of CHWs. iii) Strengthening health facilities and health worker performance Strengthening of health facilities and health worker performance has been addressed in the previous sections. This section will discuss the project’s Quality Assurance (QA) approach. The project brought an intern for two months in early 2005 to focus on this issue. The intern worked intensively with the project team and local partners to conduct four quality assurance / diagnostic activities in one PHU each. The topics addressed were: ƒ MNC: failure of TBAs to consistently identify prolonged labour or obstruction as a danger sign; ƒ Treatment of diarrhea; ƒ Malaria: IPT; ƒ PCM: technical competence of PHU staff. Although the evaluation did not concern itself with the evaluation of these specific QA efforts, they appear to have been conducted systematically, in a participative manner, with emphasis on capacity building. They led to a range of interventions, in the form of awareness raising (drama skits), improving reporting, testing and retraining, and on the job training. On the downside, however, this effort has left limited institutional memory. After some time, reports of these four activities were provided to the evaluator, but plans of action developed after these four pilots are non operational and could not be provided. None of the steps of QA have been duplicated in other PHUs, and the project does not have now the capacity to implement QA processes. The same person is now the objective manager for BCC, QA, and training, and the staff does not have a tool box to which to turn to implement simple QA learning cycles. The last limiting factor to institutionalizing QA at scale in the project is the need to produce and rapidly analyze data when conducting QA interventions. B-49 IRC Kono CS 19 Mid-Term Evaluation Report iv) Training Training activities conducted by the project are presented in Table 9. Most training activities have been conducted with or through the existing structures (DHMT, PHUs), which is a positive side for sustainability. Internal training has been limited. The project staff has learned basic computer skills. None has been trained in IMCI, or in MNC. e. Sustainability strategy The CSP has placed emphasis on sustainability early on. It was one of the projects which received TA from CSTS+ in 2003. It developed a ‘dashboard’ at that time, and recently revived efforts to track progress. One of the lessons it learned is that tracking progress toward sustainability happens at key times, but not necessarily on a monthly basis. Among the strengths of the project’s sustainability strategy is the partnership with the DHMT at all levels, and the work through local institutions or in reinforcement of national (MOH/UNICEF) programs. As the project will seek to adjust and re-energize its strategy, a new time will come to keep an eye on the sustainability question, for example: ƒ What is the viability of the free distribution of bednets? ƒ What are the implications of the absence of cost recovery for drugs sold through CBT? ƒ What happens if the DMO changes district or moves to Freetown? ƒ Once strengthened, how can CHW/TBA efforts and positive roles be maintained? The project is well placed to answer these questions, in the very specific context of Kono, where very basic environmental variables will remain the limiting factor of sustainability for a number of years of national reconstruction. As in other post-conflict settings, such as South Sudan or DR Congo where IRC has other projects, health services will need to rely on the continued infusion of resources for years to come. Communities in Kono, which have suffered catastrophic destruction and disruption, and which have limited economic options, cannot be expected to respond entirely on their own immediately to the high burden of disease and lack of infrastructure. Furthermore, the child survival program is well placed to demonstrate that such support is not prohibitively expensive, when calculated per beneficiary and compared with the sum of development aid. The child survival B-50 IRC Kono CS 19 Mid-Term Evaluation Report program will contribute to sustainability in the short term by (1) building systems that allow the efficient delivery of services, limiting the amount of resources needed; and (2) building the capacity of partners at all levels, from village-level workers to DHMT members, so that the departure of any one person, such as the DMO, does not disrupt progress. As the project will seek to adjust and re-energize its strategy, a new time will come to keep an eye on the sustainability question, for example: ƒ What are the conditions for the viability of the free distribution of bednets? ƒ What are the implications of the absence of cost recovery for drugs sold through CBT? ƒ What happens if the DMO changes district or moves to Freetown? ƒ Once strengthened, how can CHW/TBA efforts and positive roles be maintained? B-51 IRC Kono CS 19 Mid-Term Evaluation Report Table 9: Training conducted by IRC Kono CSP No TRAINEES TRAINING TITLE DATE OF TRAINING No OF DAYS EXPECTED TO TRAIN ACTUALLY TRAINED 1 PHU staff (In￾charges) Epidemic disease collection and compilation, calculation of target and coverage 19/8/200 4 by DHMT 2 14 13 2 Vaccinato rs Epidemic disease collection and compilation, calculation of target and coverage 24/8/200 4 by DHMT 2 16 15 3 CHWs/TB As (13 PHU) - Danger signs in pregnancy, labor, and after delivery. - Clean home deliveries - Recognition and reporting epidemic diseases. - Follow-up on EPI defaulters - Management of ORT corners - CHW record keeping. 28/9/200 4 PHU in￾charges Pre-post evaluatio n 3 TBA 3 days CHWs 2 days TBAs-155 CHWs￾155 TBAs-145 CHWs￾133 4 CHWs/TB As (7 PHUs) - Danger signs in pregnancy, labor, and after delivery. - Clean home deliveries - Recognition and reporting epidemic diseases. - Follow-up on EPI defaulters - Management of ORT corners CHW record keeping. 4/5/05 3 TBAs-100 CHWs￾100 TBAs-85 CHWs-74 5 PHU staff 7 added PHUs April 05 2 days In￾charge: 10 Vaccinato rs:9 In￾charge: 10 Vaccinato rs:9 6 PHU staff TOT for CBD: - Training techniques - Recognition and management of ARI, CDD, Malaria. - EPI promotion, incl. Vitamin A (1 and 2) 31 March 06 DHMT (2 sisters & Rogers) 1 day CBDs: 135 CBDs: 132 B-52 IRC Kono CS 19 Mid-Term Evaluation Report C. Program Management From the onset of the evaluation, management was designated as an area of particular emphasis for the work to be carried out by the consultant. Individual interviews were conducted with all manager-level staff in the project and with backstop staff from Freetown and New York, as well as with the IRC Country Director. For all staff directly involved with the project, these interviews were semi-structured to cover the following topics: 1- Main strengths and weaknesses of the project 2- Respective roles and responsibilities across levels (Kono, Freetown, New York) 3- Capacity in specific technical interventions 4- Financial management 5- Human resources management 6- Administrative and logistical support 7- Communication 8- Organizational learning A lot of lessons came out of this and are presented here under a format differing slightly from the one recommended in the MTE guidelines. In order to emphasize implications for organizational behavior, the lessons learned are organized as follows: 1- Areas where consensus dominates a- Strengths recognized by most or all. b- Weaknesses, which most if not all recognize need to be addressed. 2- Areas where consensus is weak. Those are identified as areas “for resolution and improvement.”26 26 The role of the evaluator is to look at a system and what it produces. When a weakness is found and reported, it does not mean that a specific person or position is blamed, even if that person holds a position critical to fulfilling the organizational function. There are many systemic factors which may hinder a person from fulfilling his or her role. The responsibility of the evaluator is to help the organization—the system—have clarity on what ultimately works and what doesn’t work based on the information available. The work of assessing staff competence, suitability or unsuitability to a function, and the work of arbitrating inter-departmental differences are the responsibility of senior management. This is why for areas of management where consensus is weak, the first step is “resolution.” This means resolving the differences in perspective in order to find a solution fitting the context and the identity of the organization. C-53 IRC Kono CS 19 Mid-Term Evaluation Report 1. Areas of consensus a. Strengths As noted previously, there is a good fit of the project with IRC’s evolution toward development and its program framework. The ultimate purpose of the project is understood and shared by all. The project operates in an acutely needy region of the lowest ranking country on the Human Development Index, with field staff that is qualified, committed, and energetic. This staff has skills and capacity building needs, but has gone through numerous and somewhat stressful phases and continues to perform with dedication and understanding of the local communities. The project benefits from motivated backstopping staff in New York and, for the last six months, in Freetown. Field staff is universally appreciative of the support they receive, if not the amount of it available. Mention is made of training provided by the NY TU (i.e. MS Excel); or the supportive and motivating approach of the health coordinator, who has played a key role in the definition of a new organizational structure which seeks to improve the project’s capacity building role vis-à-vis the DHMT. There is universal consensus on need for the recently advertised Senior Technical Advisor position for the project. There generally is as well now a consensus for reorienting the health promoters (who are now technical area specific supervisors) in a capacity building role, even if progressively. The project has already moved to an integrated HIS with the DHMT. Reports and documentation about the project are generally rapidly available and there is transparency about sharing information. The project operates within a strong organization culture, IRC, with the capacity to support ambitious projects and equip its staff to do so. b. Weaknesses There is consensus, if not absolute then general, about weaknesses in the following management areas: Human resources and staff management ƒ There have been numerous changes and stressors in the life of the C-54 IRC Kono CS 19 Mid-Term Evaluation Report project, particularly in the last six months: - change of project manager (considered by some as “traumatic and untimely”) with limited overlap time with her successor; - the long period without health coordinator in Freetown has finally been ended; the corollary is that the health coordinator is recent and has had to catch up with eight projects new to him; - departure of the BCC manager and reorganization of roles and titles of almost every staff bar the manager of the project. ƒ This has made the organization of the mid-term KPC and evaluation, and the launching of the CBT component more stressful and disruptive on the management and staff of the project in Kono than should be anticipated. Ability to dispatch the available human resources and maintain project activities has suffered as a consequence in the last six months. ƒ Not all steps are the easy on the road to Child Survival ƒ Although titles and positions have been redesigned and a new organizational chart developed for the project, these changes have not fully been thought operationally yet. For example the definition of technical area supervisors (EPI, Infectious Diseases) is on face value at odds with the project’s IMCI (integrated management) strategy. ƒ There are concerns about the breadth of capacity required from project staff, but in particular their managers. This is voiced as concerns about, or alternatively demand for increased capacity building and support. Some things are obvious, though the modes of remedy proposed vary: 1- the M&E / HIS management and production capacity needs strengthening, particularly to produce meaningful and timely analyses; 2- one objective manager covers BCC, QA and training. This is probably a job for two people, both of whom will need support. C-55 IRC Kono CS 19 Mid-Term Evaluation Report 3- The CBT component will benefit from having one single objective manager and one regional backstop. The same kind of focus might be beneficial for PHU-level interventions, mostly through simple QA, and for community / BCC efforts. These different components would reach CHWs operationally through the same health promoters / field supervisors. ƒ Learning and training for staff suffers from the lack of available options in country and the absence of a health coordinator in the past. Technical (and managerial) backstopping and supervision ƒ The project team currently needs more backstopping and support, particularly from Freetown. Most technical issues have managerial implications; so this support is both technical and managerial. There were two week-long field visits from the HQ backstop over the 12-month period preceding the evaluation, including one joint visit with the new Freetown-based health coordinator in January; there was also a visit by the regional CBT backstop immediately preceding the evaluation to complete the mid-term survey. There were no other visits from the Freetown health coordinator. ƒ New York and Freetown need to make their joint efforts at support more efficient. More work needs to take place to provide the project with the tools it needs; and more visits to the field are needed. The recommendation of this evaluator is for a monthly supervision visit from the health coordinator for the next 6 to 12 months. These supervision visits should include visits to PHUs and community sites, to provide coaching to the team in the field in addition to rather than simply to the Koidu office. This ongoing supportive supervision could help determine whether and when more visits from the NY or regional backstops would be most appropriate. Administrative, financial and logistical support ƒ The project does not have a logistician, petty cash manager, or even administrative assistant. The recent recruitment of a data clerk might help with this issue rather than the intended strengthening of M&E. ƒ Administrative and logistical support to the project is generally recognized as weak. This translates in procurement and programmatic delays, in mutual frustration between technical and administrative departments, and occasionally possibly costly mistakes. C-56 IRC Kono CS 19 Mid-Term Evaluation Report ƒ The project manager has to turn to too many nodes of control to resolve simple logistical issues. As vehicles and drivers are not assigned to the project, she has to turn to a Field Coordinator to obtain resources, but since the Field Coordinator has administrative and not programmatic oversight, she needs to combine this with contacting Freetown to resolve issues. This can be a frustrating way to carry out work and to expend management energy.27 Organizational learning ƒ Integration with other projects (i.e. EH, SCOPE) is not taking place and is perhaps not seriously attempted. ƒ Production and analysis of data (basic M&E and HIS data) is weak. There are no standard monitoring forms for any of the field staff activities. (They were tried once but discontinued.) The project team contributed to data collection for the KPC but not to analysis and reporting. Without the presence of the regional and NY TU staff, the evaluation would not have had the required KPC data to start with. ƒ The fact that two annual reports listed every activity “on target” is of concern, given the findings of this evaluation.28 2. Areas for resolution and improvement The following issues are not perceived in the same light by many actors. In some cases, they came to the attention of the evaluator but may not be on the team members’ radar screens. Financial, administrative and logistical management ƒ Initial mistakes in budget allocation and compliance were resolved rapidly once identified. But appropriate processes for effective and timely procurement, for catching possible errors, and for solving the root causes of possible errors are not in place to all parties’ satisfaction and to the good conduct of work. 27 A pooling of resources among projects is partly the issue here, although the system has also benefited the project, when it was able to start implementing activities before arrival of its own vehicles. 28 This is not uncommon for projects to be over-optimistic in their reports, when the focus is on showing implementation of activities without sufficient critical consideration of the effective contribution to results. An evaluation can be a useful step in focusing the attention of the M&E system more strongly on results. C-57 IRC Kono CS 19 Mid-Term Evaluation Report ƒ Use and understanding of budget vs. actual reports is not shared adequately at all the levels where it needs to take place. The project manager and Freetown need to be on the same page to make meaningful recommendations to senior management. There may be room for revising a budget and obtaining required approvals from USAID, but this cannot be initiated from the financial controllers without programmatic understanding, or from the program management without financial understanding. ƒ The dialectic between program management and administrative/financial oversight is not one of problem solving and progressive harmonizing. This is a serious systemic issue. Managers of different levels recognize the protective importance of regulations for the organization, but there is also a lack of clarity about what flexibility managers can have to make the project happen. There is a demand for “supportive control,” with coaching and close communication, rather than proscriptive control. Communication ƒ Staff at the level of managers and above have different views of acceptable protocols of communication between the field and higher levels, notably the NY TU, while the latter have been frustrated in their efforts to solicit more communication from the field. This needs resolution, bearing in mind that the US HQ backstop position is a non-negotiable of the CSHGP grant mechanism. HR management ƒ Due to the recent changes discussed previously, a mapping of the capacity, size, and skills of staff vs. its role in the implementation of the project remains to be shared among all who manage and support the project. The new roles for field staff have not been clearly planned out (that was “pending the MTE”). ƒ There is a perception that the scope of some job descriptions is too broad. ƒ There is some concern that recruitment delays can lead to losing good candidates for important positions in the project. ƒ Reports on staff morale are inconsistent. But stress at the management level probably transfers to staff. Information technology ƒ It is the opinion of this evaluator that the IT conditions of the C-58 IRC Kono CS 19 Mid-Term Evaluation Report project can be cheaply and rapidly brought to better standards, even in the context of Kono. Computers can be networked and share access to printers for example. Transfer of data between computers is oddly a problem in the office; M&E data reside for the most part on one MS Excel spreadsheet, which rapidly leads to computer crashes once accessed. Only the project manager’s computer has access to email, as there I only one Internet port available in the office. If objective managers and health promoters/supervisors are going to become greater capacity builders, their access to technical assistance and information might require improving connectivity in the office. D. Mission collaboration Collaboration is recognized by the Mission as satisfactory on the part of the project. The limiting factor is the limited staffing at the Mission level, the absence of a health sector, and recent strategic re-design activities with a substantial drop in funding. The Mission is pleased with the site selection and the level of information it receives from the project. The USAID representative in Sierra Leone recently visited the program site in Kono. It has, however, proven challenging for the IRC to generate sustained engagement with and knowledge of the Child Survival Program in Kono inside the USAID Mission (which has gone through a lot of changes, in the last year and does not have a PHN officer). E. Other issues identified by the team No other issues were identified by the team. F. Conclusions and recommendations 1. First recommendations from the workshop At the end of the field assessment week, the evaluation team members reviewed all findings and discussed the main recommendations to be made to the project (Table 10). These recommendations are amplified in the next section. F-59 IRC Kono CS 19 Mid-Term Evaluation Report 2. Final recommendations from the consultant Recommendations will address child health issues then maternal and newborn care and cross-cutting issues. Finally, recommendations for management will be presented. Priority recommendations are highlighted through boxes inserted in the text. Table 10. Main recommendations from the evaluation team. Malaria Pilot intensive bednet distribution ARI Re-define objectives Pilot community treatment Improve behavior change approach Diarrhea Improve behavior change approach EPI Focus on defaulters and timing of immunization MNC Re-think objectives and strategy Consider focusing on complications Cross￾cutting Improve HIS and other systems to improve ease of use and provide more structure and more guidance for field staff Increase the role of the DHMT and PHU staff in actual implementation of the program a. Recommendations for improving Child Health 1. Develop a comprehensive set of IMCI tools with the DHMT, for PHU, CHW and TBA, and CBT levels. 2. Train project staff in IMCI. Overall, systematic IMCI implementation is the key to focusing the project’s child health efforts in an effective and – not a small detail – in a sustainable manner. IMCI means more than addressing messages and practices recommended by IMCI. IMCI is meant to be an integrated health care management system. Developing this system is a high need and priority of the DHMT; it is also a high need of the project if it wants to achieve “sustainable reduction in child mortality.” ƒ A lot of training in IMCI has already taken place. The next training, to be organized with the DMO by MOH trainers, should include all project staff. This should happen sooner rather than later. ƒ The project needs to rapidly develop basic tools with the DHMT, such as: o Supervision checklists; o Job aids; F-60 IRC Kono CS 19 Mid-Term Evaluation Report o Corresponding monitoring and HIS forms. These tools should be developed within the context of IMCI implementation. ƒ The project can help the DHMT by suggesting emphasis areas. It would be ridiculous, for example, to start developing job aids for every IMCI practice. They are not needed. The project can look at gaps in performance (a number are presented in this report) and develop job aids on those topics. ƒ The priority should be to develop tools that are: o simple; o rapidly available, even if perfectible; o will be used repeatedly and consistently by the DHMT and project supervisors. ƒ The project’s role is to be a testing ground. These tools will not be perfect. Practice will identify the flaws, which can be corrected in a following phase, before the DHMT scales them to the entire district. (National scaling might be a following step, and would not be hard to imagine if the project is successful.) ƒ The BCC strategy needs to complement this and have a similar focus. This is discussed later. ƒ The project team will not be able to move speedily without intensive support and inputs from Freetown and New York, who need to coordinate and invest actively in the production of these tools. 3. Review and adapt the DHMT supervision plan, emphasizing the same IMCI strategy. This includes following up with PHU in￾charges and also reviewing, adapting and monitoring their own plans. ƒ Once trained or at least refreshed in IMCI, the CSP field supervisors29 will play a key role in coaching the institutionalization of quality IMCI with: o PHU in-charges and MCH aides; o CHWs and TBAs where appropriate (the set of tools is necessary to move from good advice to systematic reinforcing of the right practices); o CBDs. ƒ The QA strategy can play an important role in this. (Actually, consistent supervision based on set standard of cares and 29 These are the former health promoters, now technical area specific supervisors. Their role will remain to be the field frontline of the project. They will be referred as field supervisors, a functional description, in this section. F-61 IRC Kono CS 19 Mid-Term Evaluation Report emphasized by standard tools is already a QA approach in some sense.) This is discussed later. 4. Intensively focus community and facility-level immunization strategies on appropriate timing of immunization, particularly measles and DPT1. ƒ The immunization strategy fits totally within the IMCI strategy. In terms of reaching the project’s objectives, tracing defaulters is of course an important activity, which needs to be institutionalized in each PHU. But the more strategic step for the project and DHMT to reach 80% full immunization coverage will be to improve the timing of immunization. This should be a central point of the new BCC strategy (this includes CHWs, TBAs, and CBDs) and of the facility QA and supervision strategy. Mass media can be used. The message needs to get through. The project objective is within reach. ƒ Additional issues might be balancing immunization outreach efforts and the potential for strengthening routine immunization. Close effective monitoring of progress on the timing and coverage of immunization in each PHU, can help PHU in-charges and DHMT consider taking this direction. 3 priority recommendations for improving Child Health • Develop a comprehensive set of IMCI tools with the DHMT, for PHU, CHW and TBA, and CBT levels. • Train project staff in IMCI, in accordance with MOH schedule. • Review and adapt the DHMT supervision plan, emphasizing the same IMCI strategy. This includes following up with PHU in￾charges and also reviewing, adapting and monitoring their own plans. 5. Develop a plan with each PHU for improving systematic Vitamin A supplementation. 6. In concertation with the DMO, this plan could include pilot zones for involving TBAs and possibly CHWs in community-based distribution of Vit. A. ƒ The increase in Vitamin A coverage observed at mid-term might continue or might stop as there are gaps in the immunization schedule which lead care takers and care providers to neglect this important activity. F-62 IRC Kono CS 19 Mid-Term Evaluation Report ƒ Vitamin A supplementation practices (right time, right dose) and coverage need to be monitored. ƒ Improving this indicator can be achieved by a community distribution strategy. This should not be limited to CBT areas. The DMO is open to community strategies for improving Vit.A coverage. Since post-partum Vit. A coverage (the first dose) will hardly increase without a community strategy and TBA involvement, it makes sense to have a Vit.A coverage strategy rather than just a post-partum Vit.A. 7. As recommended by the evaluation team, even though the project has already reached its EOP goal, it should be ambitious and try to achieve intensive bednet distribution. ƒ This is an area where the project has already achieved results, and does not require a lot of additional know-how. Thousand of bednets are available in the district and will continue to be provided by UNICEF for the foreseeable future. ƒ Problem solving some of the issues with bednet distribution needs to move from discussion to action planning with the DHMT. (What happens if a child completes DPT3 through an outreach and does not receive a bednet on this occasion? Etc.) 8. Appropriate and timely treatment of malaria in under-fives needs to become a clear objective of the project (rather than a “secondary objective” only occasionally measured. ƒ Three main strategies can allow it to obtain notable results: 1- The initiation of Community-Based Treatment in the coming months; 2- Quality Assurance (with effective monitoring feeding into the HIS) following the IMCI guidelines in PHUs. 3- Intensive communication to households and communities, for example just before the rainy season. The project will certainly have a notable impact on the first cause of child mortality in Kono if it achieves 80% bednet coverage, and 30- 40% timely treatment of fever. 9. Proper management of diarrhea in children (oral rehydration and zinc treatment) should become a clear objective (with an indicator measured toward a target) of the project. ƒ The project can persist in promoting hand washing, particularly if it can be supported by parallel water and sanitation projects. More F-63 IRC Kono CS 19 Mid-Term Evaluation Report effective promotion of handwashing would require identifying why improvements are noted for handwashing on two occasions and are not happening on four occasions. This might require a small Operations Research (OR) effort, but this is probably not the time for the project to attempt this. However in the future, if such an OR can be set up and demonstrate reduction in incidence of diarrhea, this would provide valuable material for a communication effort based on realities in the district of Kono. ƒ What the project can do now is, in addition to continuing to promote hand washing, aggressively promote oral rehydration at PHU and particularly at community level. The CBT component which is about to start could help it achieve high results in this area. ƒ The project should fully direct its efforts to improving fluid intake of children suffering from diarrhea, through ORT, ORS or other available fluids (starting with breastfeeding), as recommended by IMCI and the DIP. ƒ Zinc should be added to the treatment options for diarrhea. 10. The PCM strategy of the project needs a total overhaul, which should be simply done by going through the IMCI strategy. The recommendations of the evaluation team are valid and direct: ƒ Re-define objectives; ƒ Start community treatment component; ƒ Improve (intensify) behavior change approach. The new project objectives for PCM can be captured by a combination of: ƒ Defining and improving quality of PCM at PHU level (which will happen if IMCI is implemented); ƒ Defining and improving quality of PCM at community level for CBT chiefdoms; (Targets for these first two indicators should be set high (70-90%). ƒ Effective communication to communities about danger signs (rapid or difficult breathing) leading to an improvement in the recognition of these danger signs; ƒ Rapid quarterly (more if possible) LQAS assessment of pneumonia cases treated in the community.30 b. Recommendations for improving Maternal and 30 As suggested by an evaluation team member, assessment of 2 randomly identified respiratory illness cases per PHU area, collected during ongoing supervision, would yield a sample of 40, largely sufficient for a trend to be analyzed. F-64 IRC Kono CS 19 Mid-Term Evaluation Report Newborn Care In terms of addressing maternal mortality, the project is not set to address effectively or even measure appropriately the current maternal care indicator (% of mothers of children 0 to 11 months having experienced a serious delivery complication who delivered at a health facility). The environment is going to remain a challenge in achieving high results over the short and mid-term. As a consequence, the project should focus on making incremental gains, supporting the DHMT long-term strategy for reducing maternal mortality in four main directions. 11. The project should set as an objective to modestly but significantly increase skilled birth attendance at the PHU level. 12. The project should improve the district capacity to manage timely referrals of complicated deliveries. 13. The project should increase the proportion of communities with emergency transport plans in place and operating. 14. The project should increase TBA detection and early referral of complicated deliveries To achieve progress in these directions, a number of actions need to be taken at different levels: ƒ MCH aides’ skills need to be assessed for and strengthened in management of delivery and identification of danger signs. This should be conducted in collaboration with the DHMT and the hospital. This is one activity where collaboration with the Kenema project would be beneficial. Evidence-based SBA tools and practices need to be introduced (such as clean delivery, safe delivery (use of partograph), prevention of newborn hypothermia, immediate breastfeeding, and active management of third stage of labor), and used to conduct any assessment; ƒ PHUs need to have the minimum equipment to perform safe deliveries, including prevention of post-partum hemorrhaging; ƒ MCH aides need to be strengthened and encouraged to take on more deliveries and to be more assertive in supervising TBAs conducting deliveries in PHUs. This can only take place if they are more confident and more skilled, and if they are monitored, supported, and rewarded for performance in this area. ƒ Detection of complicated deliveries in PHUs need to be monitored, reviewed regularly and increased. The role of effective monitoring F-65 IRC Kono CS 19 Mid-Term Evaluation Report within the district HIS is obvious here. More generally, the HIS needs to be upgraded to measure key obstetric outputs more effectively ƒ The transfer of the ambulance took place shortly after the MTE. The radios need to be made functional rapidly; this includes a sustainable power source. The budget might need to be revised to finalize this component and extend it to all PHUs. Discussions on this topic need to end, and decisions need to be followed by action.31 ƒ A monitoring a review process needs to be set in place at the DHMT level to support the hardware investments (radio, ambulance). This will involve the referral hospital. Simple forms can be created and the DHMT should follow up on each referral to assess its outcome. This will allow the CSP to build a database of emergency obstetric referrals in the 10 chiefdoms. While this will not solve the problem of management of obstetric emergencies, this will build the capacity of the local health system to understand the extent and precise nature of the problem (sources of most referral, birth attendant delivering the woman, place where delivery was initiated, timeline of the referral / nature of delays, outcome for the mother and the child). ƒ A reasonably tight timeline should be set for completing all steps (hardware and software). Setting the most optimal timeline should be part of the action plan, which will follow the evaluation. ƒ The BCC strategy needs to involve community groups and TBAs, to inform about danger signs and the benefits of facility deliveries. (Of course this requires that benefits can be found in facility deliveries. See above.) ƒ TBAs need to be actively involved and turned into allies of the project. ƒ As a joint step, there needs to be organization of, promotion of and monitoring of the development and working of community emergency transport plans, based on clear signals of danger to the mother and child. ƒ All these action steps should target the four directions listed above, and correspond to clear indicators of performance. These indicators should be set modestly for the time being. But progress in those four directions would be key to sustainably reduce maternal mortality as the system and the environment move forward. Role of TBAs 31 If IRC lacks capacity in this area, it can contract with PACTEC, another NGO already very active in West Africa. See http://www.pactec.org/index.htm. Contacts: (US) Paul Abbott pabbott@pactec.net; (Senegal) Souleymane Kouyate skouyate@pactec.net. F-66 IRC Kono CS 19 Mid-Term Evaluation Report The end game and objective should be to increase skilled birth attendance, at the PHU level. TBAs will only support this trend if there is some quid pro quo which they can benefit from, by expanding their roles in other areas. It will be easier to displace their role than suppress it. The following are options which should be considered seriously by the project with the DHMT, for the involvement of TBAs: ƒ Some areas of training are controversial for TBAs, but at a minimum, the TBA capacity to identify danger signs must be greatly increased, through training and active supervision in PHUs. Home Based Life Saving Skills (HBLSS) training has pros and cons. The question is between more rapidly identifying danger signs and referring women who need to be referred, versus legitimizing the role of TBAs when the poject should try to promote skilled birth attendance. ƒ They can play a role as ‘early detectors’ of pregnancy and delivery danger signs. This would require a plan, with training and motivation, for example by special recognition for referring rather than delivering complicated cases. ƒ Train and encourage role (possibly through recognition at the PHU level) in mobilizing communities to have emergency transport plans. The potential role of ward committees needs to be considered in this. ƒ TBAs’ role in antenatal, post-partum and newborn care can also be considered, as promoters of breastfeeding, Vitamin A, checking on first immunization; Iron (FeFol); TT promotion; birth preparedness; encouraging PNC; early breastfeeding; Vit A; first Immunization. ƒ (Some TBAs might be recruited to provide more female CBDs.) 4 priority recommendations for improving Maternal and Newborn Care • The project should set as an objective to modestly but significantly increase skill birth attendance at the PHU level. • The project should improve the district capacity to manage timely referrals of complicated deliveries. • Just as child health needs a driving strategy, antenatal and postpartum care need tools and focus to improve quality, and coverage for PNC. • PNC efforts must consider the role of TBAs (see above) and emphasize immediate breastfeeding, Vit.A supplementation, and Iron supplementation. 15. Just as child health needs a driving strategy, antenatal and postpartum care need tools and focus to improve quality, and F-67 IRC Kono CS 19 Mid-Term Evaluation Report coverage for PNC. ƒ Improving TT coverage needs to fit within an effort to monitor and improve the quality of ANC coverage, and in this case look at compliance with Iron and Folate supplementation during ANC visits. ƒ Recognition of pregnancy danger signs needs to be emphasized practically through supervision tools, job aids and BCC strategies from the PHU level to CHWs/TBAs and communities. 16. PNC efforts must consider the role of TBAs (see above) and emphasize immediate breastfeeding, Vit.A supplementation, and Iron supplementation. ƒ Once again consistency between approaches to care providers and care takers will be key to reaching impact. ƒ The area of Maternal and Newborn Care is the most challenging for the project, and the one that has progressed the least. Actually not at all. ƒ The project needs to carefully reconsider what it wants to achieve, and line up inputs, resources and activities behind the desired objectives. These recommendations are options, all of which have design and management implications. c. Recommendations for improving cross cutting strategies 17. The capacity of the project to operate a useful HIS /HMIS for the benefit of the DHMT and its own objectives needs to be built rapidly. 18. The project HIS, integrated within the DHMT, needs to focus on performance gaps identified during the MTE, and on producing rapid analyses to guide management decisions. ƒ This will require concrete inputs from New York and from Freetown, and the strengthening of the project’s own capacity in the field. ƒ The production of routine health services statistics (i.e. number of consultations provided; number of cases of fever treated) needs to be distinguished from the production of rapid analyses referring to the ongoing efforts of the project and the DHMT (i.e. to answer questions such as “Are we improving timeliness of measles immunization? And where?”). This will be key in ultimately reviving the QA approach. F-68 IRC Kono CS 19 Mid-Term Evaluation Report 19. After an initial and immediate effort to launch an HIS process more supportive to management decisions, the system should be reviewed and improved at least on an annual basis. 5 priority Recommendations for improving cross cutting strategies • The capacity of the project to operate a useful HIS /HMIS for the benefit of the DHMT and its own objectives needs to be built rapidly. • The project HIS, integrated within the DHMT, needs to focus on performance gaps identified during the MTE, and on producing rapid analyses to guide management decisions. • The project should start implementing simple QA learning cycles with the DHMT. This will require that essential tools be developed (see above) and the project team becomes comfortable with their implementation and monitoring in the field. • The BCC strategy needs to be strengthened and put on tracks and the BCC manager needs to receive adequate support to do so. • The partnership with the DHMT will be key to the success of the project. This needs to be maintained and fostered. This is an essential role for the project manager. Discussions about further integrating the project and DHMT teams (possibly by moving the project inside the DHMT) are certainly interesting; this idea deserves further consideration. 20. The project should start implementing simple QA learning cycles with the DHMT. This will require that essential tools be developed (see above) and the project team become comfortable with their implementation and monitoring in the field. This recommendation can only be implemented progressively, as tools get developed. Suggestions that it start with CBT are appropriate, but in a second phase, simple QA methodologies should be taught and applied in all areas (more TA will be needed) building on the information coming from the HIS. 1- collect management-supportive information (HIS) through ongoing supervision; 2- identify priorities for QA (one or two problems unresolved through ongoing supervision and efforts); F-69 IRC Kono CS 19 Mid-Term Evaluation Report 3- problem solve with DHMT and PHU, or with DHMT and TBAs/CHWs/Communities; 4- agree on remedial and monitoring approach; 5- implement remedy; 5- review progress, and scale to more PHUs; 6- repeat cycle on next problem. 21. The BCC strategy needs to be jump-started and the BCC manager needs to receive adequate support to do so. ƒ The BCC strategy needs to focus on key messages (through IMCI or the MNC strategy). Specific coaching and on-site technical assistance will probably be needed to get the process rapidly and effectively underway. ƒ It needs to have supportive material. At this point the project should have two tracks: (i) develop quality material it sees as needed; (ii) procure as rapidly as possible existing material, as long as it respects the standards being promoted. This could start by obtaining the materials already developed by the CARE project and others. ƒ Some of the communication tools (visual aids) most needed are about: - measles and DPT1 timeliness; - identification of maternal health danger signs; - IMCI management of fever and cough; - Vitamin A supplementation; - Oral Rehydration;… In each case the messages should be specific about what the behavior is, who should be doing it, and when. ƒ The project should learn to outsource development and production and be responsible for rapid implementation. ƒ A simple BCC strategy needs to be developed identifying channels of communication, including community partners to be mobilized, and how communication tools will be used. ƒ Communication can be simple and direct at this stage. ƒ Once again the role of monitoring will be essential in turning this into a successful endeavor. ƒ The BCC manager needs to be able to focus on this activity and have his capacity built. (This goes with a general review of roles – see management recommendations.) ƒ The project can operate through series of campaigns, which need to be focused, consistent, implemented through multiple channels of communication at the same time, and monitored carefully. ƒ Decisions need to be made rapidly about what elements of the F-70 IRC Kono CS 19 Mid-Term Evaluation Report CARE model are appropriate to implement at this stage. 22. In terms of community strategy, the consensus of the evaluation was on moving forward with community-based treatment and encouraging more outreach and CHW/TBA efforts. In the long run, the project should consider a more sustainable community mobilization and community organizing strategy, by building its own capacity or partnering with other neighboring projects. It should consider reviving the CHW network development strategy. ƒ This recommendation will be easier to implement as the project builds its credibility as well the confidence of community distributors and DHMT members by focusing on the “low-hanging fruit” (getting results through straightforward interventions, and documenting these results through a simple and functional HIS). ƒ As the project moves to CBT, and examines the role it wants to give to CHWs and TBAs, it might still decide to work on strengthening CHW networks. This may open a new vision for its community strategy. As will be discussed in the management section, the question is who in the project can move this forward? 23. The partnership with the DHMT will be key to the success of the project. This needs to be maintained and fostered. This is an essential role for the project manager. ƒ This partnership also needs to be furthered, going to a partnership of permission to one of genuine joint implementation. Discussions about further integrating the project and DHMT teams (possibly by moving the project inside the DHMT) are certainly interesting; this idea deserves further consideration. ƒ At the same time, the partnership will work best in conjunction with the design and implementation of effective supervision and information systems, for which the project manager will require support from NY and Freetown. 24. The large picture of sustainability is in place through this partnership, but the project should keep an eye on specific issues which are coming up to threaten components of sustainability. ƒ The distribution of free bednets and free drugs carries an inherent risk for sustainability, in spite of current reassurances by donor agencies. ƒ As the project continues to review sustainability periodically (annually or bi-annually rather than monthly), it should focus on F-71 IRC Kono CS 19 Mid-Term Evaluation Report these specific questions and suggest action steps. d. Recommendations for project management The project team and TA colleagues started an essential work of looking at staff role from the perspective of objectives pursued and activities needing to take place. This activity needs to be continued and finalized. Human resources management cannot be dissociated from a consideration of the project’s strategies. Following are some general advice for each level 25. The field staff need to be used more effectively, to build capacity rather than to implement and reach communities themselves. 26. The project manager, with the close support of the health coordinator as well as support from the CS team in NY and Nairobi, needs to carefully assess and strengthen the capacity of the staff to conduct specific tasks (i.e. supervision of technical areas versus supervision of geographic areas). The assignment of roles and responsibilities, and the number of positions required to effectively cover specific activities need to be reviewed based on operational plans and the sort of functional analysis initiated at the time of the evaluation. Budgets can be adjusted. 27. Managers need to be equipped to support Field Supervisors through tools and methods, plans, effective monitoring, coaching/supervision, as well as by protecting their staff from distraction (such as the daily emergencies, which occur in running any office, but which do not impact the ultimate outcomes) and building a supportive environment. ƒ All this means more work through health workers and care providers; more coaching and less direct implementation. ƒ New functions cannot mean an end of geographic focus. If five health promoters covered 20 PHUs, can four now do the same? ƒ Given the range of recommendations (technical focus, cross cutting efforts), it is not reasonable to think that individuals can master all new tasks at the same time. The number of field staff, the number of managers, and their ability to conduct the work need to be carefully considered and supported. ƒ Previous recommendations on the procurement of tools, and the operationalizing of effective M&E and HIS systems are relevant here. 28. Support and backstopping of the project needs to be strongly F-72 IRC Kono CS 19 Mid-Term Evaluation Report reinforced in a coordinated fashion between New York and Freetown. This means providing tools, supporting capacity building, helping with prioritization of action steps in the immediate future and close communication and monitoring. The health coordinator and the HQ child survival team should agree on a list of tools to support the manager and her team, and develop this within a short timeframe. Update on contributions to USAID’s new PMP Visibility of Project and PVO grantee The project has increased its visibility over the last two years in a variety of ways. For example, it has become a key partner of UNICEF in Kono by increasing bednet distribution and initiating UNICEF-supported strategies such as community treatment. The project has helped to increase the visibility of child survival generally, which remains quite neglected in Sierra Leone, through contact and communication with key MOH officials such as the director of primary care and the director of planning, and by partnering and collaborating with CARE. If the program goes further in its thinking and decides to share a common office with the DHMT within the hospital setting, additional visibility will be a collateral advantage of such a move. 29. A monthly backstop supervision visit by the Health Coordinator in Freetown should take place for the foreseeable future to support the project management in this new phase. The project senior and activity managers will benefit from this coaching, ongoing and in the field (in PHU and community sites). 30. Financial, administrative and logistical support to the project needs to be greatly improved. The project manager needs a greater sense of the elements within her control, notably budgeting, procurement problem-solving, access to logistical resources and to technical assistance. This requires country office and field coordination involvement. Different perspectives exist on the root causes of the weaknesses identified (systemic vs. F-73 IRC Kono CS 19 Mid-Term Evaluation Report individual performance, and the role of cultural factors)32 but not on their existence and the necessity to diagnose and remedy them rapidly. ƒ This will require diagnostic steps and problem solving. Some of the solutions could involve the detachment of members of the administrative staff to the project for a while, or the direct recruitment of administrative support staff by the project. ƒ Basic procedures for managing ongoing activities where vehicles are used need to be set in place and recourse to different nodes of control needs to be reduced. 31. All efforts must be made to recruit the senior TA position recently advertised in a capacity building role, as well as timely recruitment of the CBT staff. ƒ This position should focus intensively on improving strategic coherence and capacity building, starting mostly through further development of the HIS. ƒ Once this position is filled, the burden placed on the health coordinator for field supervision will be able to be reduced. 6 priority recommendations for improving project management • The field staff needs to be used more effectively, to build capacity rather than to implement and reach communities themselves. • The capacity of the staff to conduct specific tasks (i.e. supervision of technical areas versus supervision of geographic areas) needs to be carefully assessed and strengthened where needed. The assignment of roles and responsibilities, and the number of positions required to effectively cover specific activities need to be reviewed based on operational plans and the sort of functional analysis initiated at the time of the evaluation. Budgets can be adjusted. • Managers need to be equipped to support Field Supervisors through tools and methods, plans, effective monitoring, coaching/supervision, as well as by protecting their staff from distraction and building a supportive environment. effect of these cultural and interpersonal factors. See also Footnote 15. 32 Development workers know that interpersonal issues can affect business processes, with expatriate and national staff facing different challenges. The art of management is providing systems which reduce the F-74 IRC Kono CS 19 Mid-Term Evaluation Report • Support and backstopping of the project needs to be strongly reinforced in a coordinated fashion between New York and Freetown. This means providing tools, supporting capacity building, helping with prioritization of action steps in the immediate future and close communication and monitoring. The health coordinator and the child survival team should agree on a list of tools to support the manager and her team, and develop this within a short timeframe. • A monthly backstop supervision visit by the Health Coordinator in Freetown should take place for the foreseeable future to support the project management in this new phase. The project senior and activity managers will benefit from this coaching, ongoing and in the field (in PHU and community sites). • Financial, administrative and logistical support to the project needs to be greatly improved. The project manager needs a greater sense of the elements within her control, notably budgeting, procurement problem-solving, access to logistical resources and to technical assistance. This requires country office and field coordination involvement. Different perspectives exist on the root causes of the weaknesses identified (systemic vs. individual performance, and the role of cultural factors) but not on their existence and the necessity to diagnose and remedy them rapidly. 32. Communication within the project and with its support units needs to be improved, both in terms of hardware and software. ƒ In terms of hardware: o The project IT infrastructure needs to be brought to a working standard. o Email access needs to be given to all managers. o Fax communication can be used for rapid messaging when email is failing. o Transfer of files between computer and access to data needs to be improved. ƒ In terms of software, the immediate action steps after the MTE between the project manager and the TA staff need to be implemented in collaboration with the Freetown office. F-75 IRC Kono CS 19 Mid-Term Evaluation Report G. Results Highlight There was no results highlight developed at this time. H. Action Plan Please see attached file. I. Revised Project Data Sheet I-76 2006 2007 2008 Activities Indicators 41234123 A. Increase Quality of care at thePHUs 8 indicators defined and progress made on 5 of them . 1.Set up periodic Quality assurance monitoring of the health facilities. Teams in place and reports available 2.Develop quality measures to monitor each activitity (specified interventions)at the PHUs Specified indicators per objective available 3.Intergrate supervision plans into one with the DMHT Supervision plan in place ,supervision checklist report show atleast one visit/month /PHU one IRCstaff/DHMT. 4.Advocate for supplies through the DMHT facilities well equiped as per QA assessment reports. B.Increase childhood and maternal immunization coverage Number of children fully immunized before their first birth day. Number of pregnant women who received atleast TT two during their pregnacy and well documented Intensify on job training of PHU staff to monitor and how to calculate the coverage of EPI Population figures and calculated coverage available at each PHUs Maintain EPI coverage report data at every PHU and give feedback to the community health workers Reports well posted on the walls at every facility ,monthly meetings held with the CHWs Continue to support DHMT to plan and carryout outreach immunization Outreach team in place in each PHUs ,reports available Organize Quality assurance teams involving the DHMT EPI supervisor to monitor and improve the documentation of vaccine records and cold chain management (safe vaccine handling and practice). Teams in place reports available. Implement Quality assurance team with the DHMT EPI supervisor ehance and mointor the use of vaccine supply forms (stock balances and order). Forms available at the PHUs Continue to support the DHMT and also help to advocate for funds to assist in transportation supply of vaccines No stock outs of fuel and availability of funds to maintainance of vihicle. Involve communities (TBAs,CHW, community groups) in ares of low coverage and create awareness of timely EPI calander. Number of community sensitization session carriedout, % community providers aware of population figures and household coverage. Number of community providers who understand the calender. C.Increase the propotion of care givers of children <5yrs of age who seek care for respiratory illness at the health facility and CBD Number of children treated for suspected pneumonia at health facility and CBD . .Improve the case management of pneumonia at the health facility level QA reports on % of children treated at health center who have a respiratory rate documented . Number of PHU provided with ARI timers. .Improve the case management of pneumonia at the community level QA reports on % of children treated by CBD who have a respiratory rate documented. Number of CBD able to use ARI timers correctly . Implement QA team involving the DHMT to monitor drug use and case management at the PHU and community. Quality assurance supervision visits and data report on improved practice Develop key messages in case management of respatory illness at the PHU and Community Number of key messages developed ,proprtion of health workers who understand and practice use. Expand community based care treatment using community providers Number of PHU areas with community treatment available. D.Increase the propotion of care givers of children <5yrs of age who seek care for fever at the health facility and CBD Number of children treated for fever at health facility and CBD . .Improve the case management of malaria at the health facility level QA reports on number of children treated at health facility who are correctly catagorized for fever. .Improve the case management of malaria at the health facility level QA reports on number of children treated by CBD who are correctly catagorized for fever. Implement QA team involving the DHMT to monitor drug use and case management at the PHU and community. Quality assurance supervision visits and data report on improved practice Develop key messages in case management of malaria at the PHU and Community Key messages developed and proportion of health workers who understand and practice use. Appoint one CS staff /DHMT member to take responsibility for ITN distribution Team in place and number of ITN distributed. SIERRA LEONE CHILD SURVIVAL PROGRAM MTE ACTION PLAN OCTOBER 2006 - SEPTEMBER 2008 Objective : Sustainable reduction of maternal and child mortality Involve community distributors in distribution of ITN at the comunity level. Number of TBAs,CBD and CHW involved in distribution , reports on number of ITN distributed within the villages. Continue to monitor proper utilization and use of ITN Proper use documented through QA visits reports. Set strategies to distribute free mosquito net complimenting other activities ( free net on getting DPT 3 ,Free net on 3 ANC visit). Indicators set and reports on improved coverage E.Increase use of ORS and Zinc for children with diarrhoa Proportion of children <5yrs who receive ORS and zinc when they have diarrhoea Set up ORT coners in PHUs HIS documents increased presciption and use of ORS at the health facilities and community. Expand on going community case management of diarrhoea HIS documents on increse use of community distributors On going QA teams to monitor ORS and Zinc use in case management of diarrhoa at PHU and community Number of children documented correctly treated for diarrhoea. Develop key messages on dehydration QA household visits documents that most households have received health promotion messages F:Increase prorpotion of deliveries attended by skilled birth attendants Proportion of children between age 0-23 months whose births were attended by skilled birth attendant. PHU staffs to sentize the TBAs in the community on the importance of health facility deliveries Record reports on number of TBAs reached per PHU. % increase in facility deliveries. Train PHU staff through supervision visit on proper record documentation of ANC Number of pregnant women attending the ANC with proper birth plan. Improve timely referals of complicated deliveries Number of VHF radios well functioning at the PHUs for communication,% of health workers who can use partograph and practice proper referal documentation, report on analysed cases , causes and feedback of outcome management . Documentation on ambulance utilization. Work with the DHMT to introduce strategies to improve ANC and PNC coverage using community IMCI . 6 indicators defined to improve community mobile outreach maternal clinics Proportion of WCBA reached through outreach supported clinics, Number of TBAs intergrated in outreach clinics. Involve the community groups in planing strategies for community transport. Link with other partners and income generating community group to strategize the approach . Improve cross cutting strategies Continue to improve the quality of HIS intergrated with the DHMT M&E staff recruited on board, Both IRC/DHMT M&E staff can produce timely monthly reports analysed and interprented . Improve the management of the program atleast monthly field visit by the country office , reports on monthly meetings with the DHMT and CS staffs. Involve the DHMT to introduce strategies to improve quality of health delivery system Monthly reports on Supervision check list used to monitor performance, atleast monthly visit by DHMTto the PHU ,Monthly supervision visit of PHU to community providers .. QA reports on improved documentation and care service delivery. HIS reports on increase in attendance and timely referals. Implement quality assurance process trainings for CS staffs and DHMT Number of well completed supervision checklists, atleast two IRC staffs /DHMT staff who can analyze and intepret the data.Report on Monthly meeting held DHMT/IRC to give feedback from data and supervision .% of PHU staff who can use and complete the checklist during visits. Quaterly Post analyzed data displayed at all levels. Improve behaviour change strategies Review the capacity of the BCC manager, Number of exchange visits carriedout between CARE /IRC -Koinadugu/Kono. Atleast three staffs able to use BEHAVE framework to strategize key messages. Number of key messages developed. Number of sessions of media transmission. Improve and maintain partnership with the DHMT/IRC teams Reports on Monthly meetings attended by DHMT/IRC members, Plan of action put in place to address gaps. Agreement on performance key indicators signed at all levels . Improve project management at all levels Atleast a monthly field visit by country office , Support communication by NY tech. team , each staff have defined performance oriented job description. Number of CS staff able to perform geared to 8 key indicators. Improve finacial ,administrative and logisitic support Monthly reports on BVAs,monthly functioning tracking procurement system,weekly field coordination meetings in place ,Annual evaluation of staff performance. Child Survival and Health Grants Program Project Summary Nov-08-2006 International Rescue Committee (Sierra Leone) General Project Information: Cooperative Agreement Number: GHS-A-00-03-00012 Project Grant Cycle: 19 Project Dates: (9/30/2003 - 9/29/2008) Project Type: Standard IRC Headquarters Technical Backstop: Emmanual D'harcourt Field Program Manager: Bernadette Udo Midterm Evaluator: Eric Sarriot Final Evaluator: USAID Mission Contact: Seydou Doumbia Field Program Manager Information: Name: Bernadette Udo Address: Phone: Fax: E-mail: bernie4002@yahoo.com  Alternate Field Contact: Name: Dorice Manasseh Address: Phone: E-mail: Dorice.Manasseh@theirc.org  Funding Information: USAID Funding:(US $): $1,456,795  PVO match:(US $) $499,126  Project Information: Description: The Child Survival project’s goal is the sustainable reduction in child and maternal mortality.  To achieve its goal, the program is using three major inter-related strategies: integrated management of childhood illnesses at the community, and household level; use of data from health information systems to identify priorities and measure progress; and use of quality assurance tools to improve the quality of care and the performance of systems.  The specific intervention areas are immunizations, control of diarrheal diseases, pneumonia case management, malaria control, and maternal and newborn health.  Location: Kono District, Eastern Province, Sierra Leone  Project Partners Partner Type Subgrant Amount Kono District Health Management Team Collaborating Partner    General Strategies Planned: Advocacy on Health Policy Strengthen Decentralized Health System Information System Technologies  M&E Assessment Strategies: KPC Survey Health Facility Assessment Organizational Capacity Assessment with Local Partners Participatory Learning in Action Lot Quality Assurance Sampling Appreciative Inquiry-based Strategy Community-based Monitoring Techniques Participatory Evaluation Techniques (for mid-term or final evaluation)  Behavior Change & Communication (BCC) Strategies: Mass Media Interpersonal Communication Peer Communication Support Groups  Groups targeted for Capacity Building: PVO Non-Govt Partners Other Private Sector Govt Community US HQ (General) US HQ (CS unit) Field Office HQ CS Project Team  PVOs (Int'l./US) (None Selected)  National MOH Dist. Health System Health Facility Staff Other CBOs CHWs  Interventions/Program Components: Immunizations (20 %)  (IMCI Integration)  (CHW Training)  (HF Training)   - Classic 6 Vaccines   - Vitamin A   - Surveillance   - Cold Chain Strengthening   - Injection Safety   - Mobilization Pneumonia Case Management (15 %)  (IMCI Integration)  (CHW Training)  (HF Training)   - Pneum. Case Mngmnt.   - Case Mngmnt. Counseling   - Recognition of Pneumonia Danger Signs   - Zinc   - Community based treatment with antibiotics Control of Diarrheal Diseases (20 %)  (IMCI Integration)  (CHW Training)   - Hand Washing   - ORS/Home Fluids   - Feeding/Breastfeeding   - Care Seeking   - Case Mngmnt./Counseling   - Zinc Malaria (25 %)  (IMCI Integration)  (CHW Training)  (HF Training)   - Training in Malaria CM   - Adequate Supply of Malarial Drug   - Access to providers and drugs   - Antenatal Prevention Treatment   - ITN (Bednets)   - Care Seeking, Recog., Compliance   - IPT   - Community Treatment of Malaria   - ACT Maternal & Newborn Care (20 %)  (IMCI Integration)  (CHW Training)  (HF Training)   - Emerg. Obstet. Care   - Neonatal Tetanus   - Recog. of Danger signs   - Newborn Care   - Post partum Care   - Integr. with Iron & Folate   - Normal Delivery Care   - Birth Plans   - STI Treat. with Antenat. Visit   - Home Based LSS   - Control of post-partum bleeding   - Emergency Transport Target Beneficiaries: Infants < 12 months: 4,000 Children 12-23 months: 3,500 Children 0-23 months: 7,500 Children 24-59 months: 8,500 Children 0-59 Months 16,000 Women 15-49 years: 21,200 Population of Target Area: 91,868 Rapid Catch Indicators: Indicator Numerator Denominator Percentage Confidence Interval Percentage of children age 0-23 months who are underweight (-2 SD from the median weight-for-age, according to the WHO/NCHS reference population) 0 0 0.0% 0.0 Percentage of children age 0-23 months who were born at least 24 months after the previous surviving child 0 0 0.0% 0.0 Percentage of children age 0-23 months whose births were attended by skilled health personnel 11 94 11.7% 6.5 Percentage of mothers of children age 0-23 months who received at least two tetanus toxoid injections before the birth of their youngest child 41 95 43.2% 10.0 Percentage of infants age 0-5 months who were exclusively breastfed in the last 24 hours 0 0 0.0% 0.0 Percentage of infants age 6-9 months receiving breastmilk and complementary foods 0 0 0.0% 0.0 Percentage of children age 12-23 months who are fully vaccinated (against the five vaccine-preventable diseases) before the first birthday 28 94 29.8% 9.2 Percentage of children age 12-23 months who received a measles vaccine 45 94 47.9% 10.1 Percentage of children age 0-23 months who slept under an insecticide-treated bednet the previous night (in malaria-risk areas only) 75 189 39.7% 7.0 Percentage of mothers who know at least two signs of childhood illness that indicate the need for treatment 0 0 0.0% 0.0 Percentage of sick children age 0-23 months who received increased fluids and continued feeding during an illness in the past two weeks 17 94 18.1% 7.8 Percentage of mothers of children age 0-23 months who cite at least two known ways of reducing the risk of HIV infection 0 0 0.0% 0.0 Percentage of mothers of children age 0-23 months who wash their hands with soap/ash before food preparation, before feeding children, after defecation, and after attending to a child who has defecated 5 189 2.6% 2.3 Comments for Rapid Catch Indicators                                                                                                 We did parallel sampling using LQAS. TT, Delivery and danger signs: asked only of mothers of children 0 - 11m. Vaccination: asked only of mothers of children ages 12-23m. Sick child: asked of mothers of children 0-59m.                                                                             International Rescue Committee Sierra Leone Program 35 Fraser Street Freetown Health Unit 122 East 42nd Street New York, NY 10168 October 31, 2006 Ms. Nazo Kureshy Child Survival and Health Grants Program Global Health Bureau United States Agency for International Development Dear Ms. Kureshy, Please find attached the mid-term evaluation report and annexes for the IRC’s child survival program in Kono, Sierra Leone (CS XIX) as well as a print-out of the CSHGP data form. We are passing on the report exactly as it was finalized by the external evaluator, Eric Sarriot. The IRC feels that this evaluation was conducted in a professional and participatory manner. Staff and partners at all levels of the system including beneficiaries, health center managers, the District Health Management Team, and IRC staff in Kono, Freetown and New York had meaningful input into the findings and recommendations, while benefiting from the external evaluator’s experience and facilitation skills. We feel that the attached report accurately represents the strengths and weaknesses of the program and are committed to carrying out the recommendations. We feel that the evaluation has helped to further build IRC’s child survival capacity, both in Sierra Leone and elsewhere. We have laid out in the attached action plan how we intend to carry out the report’s recommendations over the next two years. Please note that the action plan is a work in progress—we intend, over the next two months, to detail some of the proposed action steps further. Following is a discussion of our response to the report’s priority recommendations, described in bold type in the executive summary of the report. Please note that recommendations are numbered to correspond to those in the evaluation report. Improving child health 1. Develop a comprehensive set of IMCI tools: We understand this to be both a technical and a managerial recommendation, with the tools helping us to improve and focus staff performance as well as insuring IMCI implementation in Kono. However, we note that only two DHMT members have been trained as trainers, and limited MOH and DHMT resources allocated to IMCI implementation. In this context, we do not feel that the project is in a position to fully implement facility IMCI. We do feel that we can help the DHMT put in place key components of facility IMCI, particularly those that mirror community case management practices, such as improving the assessment and treatment of children with diarrhea and pneumonia. We have begun the process of developing a set of tools including supervision checklists and performance criteria, specifically designed for IMCI implementation at both community and facility levels. Many of these tools are being adapted from existing IMCI materials. We expect this and the following two recommendations to be completed by early 2007. Page 2 October 31, 2006 2. Train staff in IMCI: We will be working with the two DHMT who have received IMCI training to insure our staff achieves IMCI competence through a variety of techniques, favoring practical, on-the-job training. We will work to insure that the tools developed for recommendation 1. are compatible with and are used in this training, as well as the official MOHS IMCI documents obtained by the Child Survival Coordinator. 3. Review the DHMT and PHU supervision plans: We will work with the DHMT and PHU staff to assess and review current supervision performance and goals. For us to effect meaningful change, however, will also require progress in addressing recommendation 23, and increasing the participation of the DHMT and PHU staff in the project more generally. Improving maternal health 11. Set as an objective to modestly but significantly increase skilled birth attendance: Rapid catch indicator “Increase the percentage of children age 0–23 months whose births were attended by skilled health personnel” has replaced the indicator “Increase the proportion of women experiencing obstetric complications who receive skilled care.” Activities and process indicators will be oriented towards achieving the new objective. The first task will be to define who is a skilled birth attendant, as this has, in Sierra Leone, often and incorrectly been taken to include trained traditional birth attendants. 12. Improve the district capacity to manager referral of complicated deliveries: We intend to implement this recommendation initially by (1) finding functional power solutions for PHU radios, and (2) helping the DHMT set up systems to monitor referral performance. Our longer term objective will be to assess and improve quality of obstetric care at the PHUs and District Hospital. We will seek to do this as possible initially within the limited financial and human resources available, and will seek additional resources. 15. Develop an antenatal and post-partum care strategy: Health Unit advisors will work with the Health Coordinator and project management staff, as well as with the DHMT, to develop specific objectives and corresponding tools related to antenatal and post-partum care, to insure that we set and achieve concrete objectives in this area. The strategies to achieve this will parallel those for recommendation 1. 16. Include TBAs in post-natal care and emphasize immediate breast-feeding, vitamin A and iron supplementation: This recommendation emerged as a consensus during the evaluation process and will be initiated in the next three months. TBAs will be trained and supported to provide these essential post-natal actions, and project and DHMT staff will monitor their effectiveness in improving coverage and quality of post-natal care. Improving cross-cutting strategies 17. Develop an improved health management information system that also serves the DHMT: This is a priority recommendation, and we will work to improve it as soon as possible. The challenge remains, however, of hiring a qualified monitoring and evaluation manager for the project. In the meantime, we have made some progress already by having a project manager from another IRC country program give on-site support during a month-long visit. A summer Page 3 October 31, 2006 intern also worked to build the capacity of project and DHMT staff in Excel use. In addition, the Ministry of Health and Sanitation is actively working to improve the national health management information system. 18. The project information system needs to focus on performance: We will work to insure that the information system will facilitate collection and analysis of process and performance indicators, such as the number of community supervisions carried out by PHU staff, or the quality of obstetric care at facilities. 20. Implement quality assurance learning cycles with the DHMT: This is an important recommendation and one which we are committed to implement. We face challenges in doing so, however. A previous attempt, facilitated by an intern, received staff support at the time but no subsequent follow-up, as documented in the evaluation report. As per the recommendation, we will use the newly-developed tools and improved HMIS to re-energize quality monitoring and improvement. The IRC is also reviewing its quality assurance approach worldwide, and we hope that this process will give us new tools in our efforts to promote a quality assurance culture among our staff and our partners. 21. Improve behavior change communication strategies: This is a priority for the IRC in Sierra Leone as in other countries. As with quality assurance, previous attempts to address this issue have failed: a key staff member was trained in the BEHAVE framework but failed to pass this knowledge on to others, or to translate it into his own work. We will consider two strategies to improve project performance in this area: (1) organizing more frequent cross-visits with our CARE counterparts in Koinadugu, where behavior change communication has been more successful, and (2) bringing a consultant for an extended period of time to improve the capacity of several key staff in this area. 23. Maintain and foster our partnership with the DHMT, and integrate project and DHMT teams: IRC and the DHMT have agreed to share offices and internet space. This has not yet taken place because the DHMT has not yet secured a new office space and the modalities of engagement are not yet worked out. In addition, we will work to address and resolve the issues that have prevented PHU staff from taking a more active role in the program. We will also be working to make sure the close working relationship translates into more DHMT participation in day-to-day supervision activities. Improving project management 25. Use staff to build capacity rather than carrying out community health activities directly: This recommendation is directly related to recommendation 23. We will work to address the underlying reasons why IRC staff have been not been able to build capacity instead of implementing directly, including addressing partner motivation issues and improving IRC staff capacity. The Health Unit will be working closely with the Health Coordinator and the Child Survival Coordinator to build staff capacity. 26. Monitor and improve staff performance, and adjust the organigram as needed: This will be addressed in part through recommendations 1., 15., and 18. The following is the current working organigram developed in response to the mid-term evaluation. Page 4 October 31, 2006 27. Managers need to be equipped to support Supervisors: This recommendation is related to recommendations 1., 15., 18. and 26., and will be prioritized for implementation over the next six months. 28. Coordinate Freetown and New York support: We will work to insure that technical support is coordinated, through email communication and monthly phone calls. We expect that the development of staff monitoring tools, as well as implementation of recommendation 29., will make coordination easier to implement. 29. The Freetown Health Coordinator should visit Kono at least monthly: The Health Coordinator, with support from the Country Director, is working to make this a reality. 30. Improve financial, administrative, and logistical support: This recommendation was made in the context of long-standing administrative issues in the IRC Sierra Leone program. IRC senior managers in-country have been addressing these complex problems. They report significant improvements in the last few months and will continue to improve the support systems necessary for the project to function effectively. 31. Recruit an expatriate on-site Technical Advisor: This person has been recruited. The position has been re-titled as “Child Survival Coordinator.” We look forward to working with your local partners, but also with the USAID CSHGP, CSTS, and other partners as we implement these recommendations. Sincerely, For the International Rescue Committee: Dorice Manasseh CS Coordinator Kono Dr. Yilma Robelle Health Coordinator Freetown Katharine Haxall CS Program Manager New York Dr. Emmanuel d’Harcourt CS Sr. Technical Advisor New York