Child Survival 20 – Mali Scaling-Up Community-Based Services In the Sikasso Region of Mali Cooperative Agreement No.: GHA-A-00-0003-00 Final Evaluation Report 30 September 2004-30 September 2009 Prepared by: Circe Trevant, Consultant and FE Team Leader Edited and reviewed by: Boubacar Sidibe, Health Coordinator, Save the Children, Mali Animata Diarra, Project Coordinator, Save the Children, Mali Eric Swedberg, Senior Director Child Health & Nutrition, Save the Children Sharon Lake-Post, Senior Specialist, Communications, Save the Children Submitted by: Save the Children Federation, Inc. 54 Wilton Road Westport, CT 06880 Telephone: 203/221-4000 Fax: 203/221-4056 Contact Person: Eric Swedberg, MPH, Child Health and Nutrition Advisor Submitted to: USAID/GH/HIDN/NUT/CSHGP December 31, 2009 This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of Save the Children and do not necessarily reflect the views of USAID or the United States Government. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 i ACRONYMS ACT Artemisinin-based Combination Therapy ADAC Association for Development and Community ANC Antenatal Care ARI Acute Respiratory Infection ASACO Management committees for health center’s in Mali’s system ATN Assistance Technique Nationale (Bilateral project for national level health programming in Mali) BC Behavior Change BCC Behavior Change Communication DA District Advisor DV Dépôt de Vente au niveau CSCom (Drug Depot in Health Center) CAP Couple Année Protection (couple years of protection) CDD Control of Diarrheal Diseases CHW Community Health Worker CPM Chiefs de Postes Médicaux (Health Center Heads) CPCV Coopératives des producteurs de cotton villageois (Cotton Producer Associations) CPR Contraceptive Prevalence Rate CS-20 “Scaling-Up Community-Based Services in the Sikasso Region of Mali project as the 20th cycle of PVO CSH Grants Program CSRef District Health Office and Hospital (Centre de Santé de Reference) CYP Couple Years of Protection DA District Advisor DHO District Health Officer DIP Detailed Implementation Plan DRC Depot Repartiteur de Cercle (Regional Drug Depot) DK Drug Kit FP Family Planning HIS Health Information System HZ Health Zone IPT Intermittent Presumptive Treatment - malaria in pregnant women ITN Insecticide Treated Net M&E Monitoring and Evaluation MCH Maternal and Child Health MOH Ministry of Health MTE Midterm Evaluation OR Operation Research ORS Oral Rehydration Solution PDSEC District Development Plan PMI President's Malaria Initiative PO Plan Operationnel (Operational Plan) PVO Private Voluntary Organization Relais Community Health Volunteers in the Mali health system – Drug Kit Managers for the project CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 ii RTA Rapport Trimestriel d’Activité (CsCom and CSRef ) – Quarterly Reports SC Save the Children Federation, Inc. SDSES Service de Développement Social et de l’Economie Solidaire (Social Services) SP Sulfadoxine-Pyrimethamine TOT Training of Trainers USAID United States Agency for International Development VDK Village Drug Kits WRA Women of Reproductive Age CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 iii A. PRELIMINARY INFORMATION Executive Summary Goal/Objective/Result Framework: CS-20, “Scaling-up Community-based Services in the Sikasso Region of Mali” was designed as an expanded impact project to reinforce community￾based management of childhood illnesses. The goal was to obtain a sustained reduction in under￾five mortality in Sikasso with two specific objectives: 1) To increase the use of key health services and improved child health practices at the village level; and 2) To increase the capacity of local entities (RHOs, DHOs, and community organizations) to assume responsibility for health activities while adopting innovative CS-20 approaches. The project was designed to build on existing relationships with the MOH in three districts (Bougouni, Yanfolila, and Kolondieba) and was expanded to two additional districts (Yorosso, and Selingue). The coverage reached 84 health zones (HZ) in 464 villages serving a total population of 1,037,418 of which (234,456 women of reproductive age (WRA), 177,887 children under five years of age). USAID funding of $2.5 million was matched by $833,330 from Save the Children (SC), from October 2004 to September 2009. Technical/Cross-cutting interventions: For the control and prevention of childhood illnesses and promotion of use of modern family planning (FP) methods, CS-20 elected the following levels of efforts: malaria (35%), diarrhea (20%), pneumonia (20%) and family planning (25%). The Sikasso Region has high morbidity and mortality resulting from low access to health services, poor service quality, and inappropriate health care seeking behaviors due to low recognition of illness danger signs. CS-20 was implemented using three key strategies: (1) the establishment of a network of village drug kits (VDKs); (2) a behavior change (BC) strategy aimed at recognizing illness danger signs, and increasing care seeking behavior; and (3) in￾service training of health workers and ongoing formative/integrated supervision and support by the regional, district and health center (HC) teams. CS-20 also applied and scaled up innovative approaches to increase access and improve quality of services by introducing zinc and Artemisinin-based Combination Therapy (ACT) in the VDKs for community case management (CCM) of diarrhea and simple cases of malaria. CS-20 used a community focused BC approach that was guided by formative research, based on interpersonal and small group communication, reinforced by the use of local radios that disseminated messages for each target group. The main strategy for community mobilization was to use the community health workers (CHWs)elected by village leaders to conduct health talks, counsel of mothers, and they in turn were supported by the Oversight Committees (OCs), a sub group of the village health teams. CS-20 used the results of the zinc and Artemisinin-based Combination Therapy (ACT) effectiveness trial to inform national policy about incorporating zinc into diarrhea case management in all five districts as well as introducing ACT in the VDKs. These studies meant to advocate for strengthening the community-based care seeking behavior from VDKs (serving as first line of treatment) in addition to the local community HCs. Instead of being a direct implementer, CS-20 involved the Ministry of Health (MOH) as principal owner of project activities with support from SC DAs operating as facilitators and resources for capacity building at the regional, district and community levels. The CS-20 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 iv approach was uniquely designed to increase access to care by establishing VDKs as the first line of treatment at the community level and to improve the correct management of sick children not treated in HFs. The strategy also sought to reinforce links between communities and their associated HCs through an improved referral system. DK Managers learned to recognize danger signs for the critical cases of malaria, diarrhea and acute respiratory infection (ARI) for referral to the HCs. Lastly, CS-20 sought to make community data available to the MOH health information system (HIS) where health staff could use it to improve health service and deliver it in a more timely fashion. Prior to CS-20, this community data had not been routinely collected for use by the health system. Below is a sample summary of the progress made towards most CS￾20 objectives: ƒ CS-20 established 477 VDKs in 464 villages and trained over 928 (CHWs). ƒ 90% of these VDKs had no stock-out of ORS and none had FP methods stock-out from July to September 2009. ƒ Zinc was added to the national list of essential medicines for the management of diarrhea and was also introduced at the VDK level. ƒ Malaria: Increase from 25.8% to 82.8% the percentage of children under 5 who were treated at a VDK or Health Center. Increase in number of children < 5 (from 8.4% to 70.6%) and pregnant women (from 25.9% to 66.2%) sleeping under insecticide treated nets (ITNs) the previous night; 82.5% of the households surveyed had at least one bed net. ƒ Diarrhea: Increase in number of children < 5 with diarrhea in last two weeks treated with ORS, home fluid and/or increased fluids from 17.5 % to 67.7%. ƒ FP: Increase in the number of women who use a modern method of FP (from 5.1% to 14%). ƒ ARI: Increase from 25% to 44.7% of children < 5 with rapid breathing who visited the VDK for treatment and were referred to the HC. ƒ Increase in the number of women who know where the VDK is located from 30.5% to 74.5%. ƒ Progress reflected by a slight increase in the number of mothers who know three or more child illness danger signs (60.7% to 62.7%). Some lessons learned and conclusions include: ƒ IR1: Access to Health Care: Access to health care has increased with the establishment of the VDKs. The VDKs have become the first source for health care and information in the villages. This local access to drugs has made care more affordable since there is no transportation cost to reach CSComs and no consultation fees from DK Managers. ƒ IR2: Quality: The CHWs are very knowledgeable about the treatment protocols for simple cases of malaria and diarrhea at the community level and consistently and correctly refer critical cases to the CSCom. On the job trainings and supervision visits have positively contributed to improved diagnosis and treatment. In addition, District Advisors (DAs) check for accuracy of data collected by CHWs during supervision visits and CSCom Managers provide feedback during monthly meetings at the HCs. ƒ IR3: Demand: Mothers know more about childhood illness danger signs and they seek care and purchase treatments at the VDKs. There is increased demand for modern FP method at the community DKs and CSCom. CSComs have also seen an increase in prenatal care visits. Consequently, more women can receive ITNs, and sulfadoxine-pyrimethamine (SP). CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 v ƒ IR4 and 5: Capacity Building of Partners and Enhanced Political Environment: CS-20 has engaged partners at all levels (community, district, regional levels) and trained them in the treatment protocols for diarrhea, malaria, and referral of ARI. The OCs and the CHWs have acquired good management skills of the VDKs including the distribution of profits from drug sales. In the HZs where CWHs are members of the ASACO (health boards), CSComs management is more efficient. The public health system has also embraced the VDKs model and restocking of those DKs is integrated into most CSComs workplans. Recommendations: Whereby the analysis proved that significant progress was made, and that the CS-20 design was effective and is replicable, some recommendations follow below to address pending challenges faced by the MOH: ƒ The health boards (ASACOs) should be more involved in the management of the HCs and VDKs so that they work with the DK Managers to assure timely restocking of the DKs. One example is that the health boards (ASACOs) can ensure that CSComs do not mark-up prices of drugs sold to VDKs. ƒ Standards of performance should be applied to all health boards so that they follow the same codes for quality care in all HZs uniformly. This strategy will help to make them more competitive and accountable for quality of care to their constituencies. ƒ Improve data management and use by establishing a consistent and timely review of data at all levels (i.e. at the community level with the DK Managers and OCs, at the HC level with the HC Chiefs and ASACOs, and at the district and regional levels) for better planning and decision making. There was a general sense that although CS-20 had collected a great deal of data and improved community data contribution to the HIS, the information was underused for planning purposes. ƒ The MOH should recognize CHWs’critical role in the health system as health educators, community mobilizers, providers of the first line of treatment for simple cases of childhood illnesses, and as community data collectors. More specifically, the MOH should formalize the CHWs by applying a sustainable incentive strategy to maintain their continued involvement. ƒ In order to address stock-outs in the DKs, the DK Managers should work more closely with the CSCom Managers to combine their orders with those of corresponding HCs in their HZs. This would require CHWs to submit their monthly data on a timely basis and allow time for the CSComs to process the community data for better planning. ƒ The strategy of establishing VDKs should be scaled up at the national level using modified selection criteria (i.e. use distance rather than population density plus distance) to increase local access to quality health care. ƒ Local partners should have a contract of performance to hold them accountable and ensure a high standard of delivery of health services in the HZs. ƒ The regional department of planning (DSES) should integrate the supervision of the DK Managers into their field visits. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 vi Table 1: Summary of Major Project Accomplishments Project Inputs Activities Outputs Outcome Project Objective #1: To increase the use of key health services/improved child health practices at village level Radio time to disseminate health messages IEC and BCC materials (in Bambara) Training (CHWs, HC Chief, CSRef Team, OC, Midwives) VDKs (Oral rehydration solution (ORS) supplies, malaria treatment, FP supplies (ovrette, pilplan and condoms), paracetamol (syrup and comprime), tetracycline (1%), zinc) Data collection tools for CHWs, and supervision in Bambara: ƒ Drug sale records ƒ Drug stock log ƒ Birth and death log ƒ BCC sessions and counseling logbook ƒ Referral logbook ƒ Growth monitoring logbook CsCOM: Training materials for HC staff, monthly report form (Bambara), IEC support materials (French and Bambara), continuous supply of zinc, financial support for supervision visits of the CHWs CS-20 Advisors: Work support, Motorcycle/helemet, Supervision log , IEC support materials (Boite a image), Training modules for CHWs), District computers (one per district), Rain coats, USB key storage key CSRef: Training materials for the CSRef staff in Bambara and French, Continuous supply of zinc Technical Assistance: ƒ HQ backstop TA with field visits, training (in emergency nutrition protocol), case management of malaria with ACTs, CCM diarrhea with zinc. ƒ Operation Research CS-20 obtained additional support for the zinc activities through American Idol funding. The USAID mission provided additional support for ACT roll out in the VDKs. Operation research (OR) on zinc, ACTs Dissemination of health messages via radio broadcasts Reproduction of IEC/BCC support materials CHWs conduct village health talks, personal counseling, household visits Advocacy efforts to introduce zinc, ACT in VDKs and also recognize the critical role of CHWs as key players for increasing access to health care. Training of district, regional and CHWs in CS-20 activities and refresher training on prevention and treatment of malaria, diarrheal diseases, ARI, and FP Monitoring and supervision of CHWs by the DAs and HC staff DAs attend monthly meetings held at CSComs 928 CHWs trained (2 per village) 477 DKs installed in 5 districts (464 villages) Increased availability of select MCH services Improved quality of select MCH services Increased demand for appropriate health services 14% of all women using a modern FP method 44.7% of children < 5 with difficult or rapid breathing who visited DK are referred to a HC 62.7% of mothers who know three or more childhood danger signs of illness 74.5% of mothers who know where the DK is located 59.7% of women who state that ITNs are very important for children under 5 90% of the VDKs had no stock-out of ORS and 100% had no stock-outs of FP methods (oral contraceptives) in the past three months (July to September 2009) 82.8% of children with fever in the last two weeks used DKs or HC CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 vii Project Inputs Activities Outputs Outcome Project Objective #2: Increase the capacity of (RHOs, DHOs, and community organizations) to assume responsibility for health activities CsCOM: Training materials for staff, monthly report forms (Bambara), IEC support materials (French and Bambara), continuous supply of zinc, financial support for supervision for the first 4 years. CSRef: Training materials for the CSRef staff in Bambara and French, Continuous supply of zinc. Organization of OCs Train OCs in the management of the VDKs DAs conducted supervision visits of HCs Increased local capacity to implement CS-20 services and activities DHOs complete two integrated supervisory visits of HCs in the last year. DHOs, RHOs, and HC staff conducted monthly meetings to monitor the HC activities. 70% of OCs were supervised by health center staff. 80% of management committees (ASACOS) participate in monthly meetings of the DK Managers at the HCs. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 viii CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 ix TABLE OF CONTENTS Acronyms..................................................................................................................................................ii A. Preliminary Information...................................................................................................................iv B. Overview of the Project ....................................................................................................................1 CS-20 Results Framework ....................................................................................................................3 C. Data Quality: Strengths and Limitations...........................................................................................4 D. Presentation of Project Results .........................................................................................................6 E. Discussion of the Results................................................................................................................11 F. Discussion of Potential for Sustained Outcomes and Global Learning..........................................24 G. Lessons Learned and Recommendations:.......................................................................................27 H. Required Annexes...........................................................................................................................31 Annexes Annex 1: Results Highlight Annex 2: List of Publications and Presentations Related to the Project Annex 3: Project Management Evaluation Annex 4: Workplan Table Annex 5: Rapid CATCH Table Annex 6: Final KPC Report Annex 7: CHW Training Matrix Annex 8: Evaluation Team Members and Their Titles Annex 9: Evaluation Assessment Methodology Annex 10: List of Persons Interviewed and Contacted During Final Evaluation Annex 11: Special Reports (optional) Annex 12: Updated Project Data Form Annex 13: Grantee Plans to Address Final Evaluation Findings Annex 14: Grantee response to Final Evaluation Findings CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 1 B. OVERVIEW OF THE PROJECT Goal/Objective/Results Framework: CS-20, “Scaling-Up Community-Based Services in the Sikasso Region of Mali was designed as an expanded impact project to reinforce community-based management of childhood illnesses. The goal was to obtain a sustained reduction in under-five mortality in Sikasso with two specific objectives: 1) To increase the use of key health services and improved child health practices at the village level; and 2) To increase the capacity of local entities (RHOs, DHOs, and community organizations) to assume responsibility for health activities while adopting innovative CS-20 approaches. The Results Framework summarizes key expected results with corresponding indicators (see Table 1 for more details). The five (5) Intermediate Results (with selected indicators) include: ƒ IR1: Increased availability of selected MCH services (80% availability of various drugs in DKs); ƒ IR2: Improved quality of selected MCH services (60% of children and 70% of women treated with correct doses of anti-malarials); ƒ IR3: Increased demand for appropriate health services (80% of mothers will know the danger signs indicating when to take their sick child for care, 60% of mothers will know how to access DKs); ƒ IR4: Increased local capacity of regional and District Health Offices to implement CS-20 innovations (80% of DHOs will conduct monthly meetings of the HCs); and ƒ IR5: Increased capacity of communities (village and HC management committees) to address priority needs of children under five years old (60% of OCs will be supervised at least 1/month by the HC). Location/ Target Group: CS-20 was implemented from October 2004 to September 2009 in five (5) districts in the Sikasso Region: Yorosso, Selingue, Bougouni, Yanfolila, Kolondieba reaching 84 HZs in 464 villages serving a total population of 1,037,418 of which (234,456 were WRA, 177,887 children under five years of age). SC matched USAID funding of $2.5 million by $833,330. Technical/Cross-Cutting Interventions: For the control and prevention of childhood illnesses and promotion of use of modern FP methods, CS-20 elected the following levels of efforts: malaria (35%), diarrhea (20%), pneumonia (20%) and FP (25%). The Sikasso Region has high morbidity and mortality resulting from low access to health services, poor service quality, and inappropriate health care seeking behaviors due to low recognition of illness danger signs. CS-20 was implemented using three key strategies: (1) the establishment of a network of VDKs with essential medicines; (2) a BC strategy aimed at recognizing illness danger signs, and increasing care seeking behavior; and (3) in-service training of health workers and ongoing formative/integrated supervision and support by the regional, district and HC teams. CS-20 also applied and scaled up innovative approaches to increase access and improve quality of services by introducing zinc and ACTS in the VDKs for CCM of diarrhea and simple cases of malaria. CS-20 used a community-focused BC approach that was guided by formative research, based on interpersonal and small group communication, and was reinforced by the use of local radio broadcasts targeted for each group. For example: (1) for families: use of the VDKs and CSComs for first treatment rather than self-medicating, and compliance with treatment; (2) for DK Managers: sale of correct doses and appropriate counseling to mothers; and (3) for health staff: provision of correct doses and appropriate counseling. All groups were encouraged to adopt healthy behaviors (preventive and treatment for childhood illnesses), to recognize danger signs for illnesses, and to use insecticide treated bednets, especially for children under 5 and pregnant women. Additional messages targeted CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 2 pregnant women encouraging them to obtain prenatal care (SP, ITNs, vaccination, etc.) at the nearest CSComs, and all WRA were counseled to use modern methods of FP, including condom use by men. The main strategy for community mobilization was to use the CHWs elected by Village Chiefs to conduct health talks and home visits, and to counsel mothers. CS-20 used the results of the zinc and ACT effectiveness trial to inform national policy about incorporating zinc into diarrhea case management in all five districts, as well as introducing ACT in the VDKs. These studies were designed to advocate for strengthening the community-based care￾seeking behavior from VDKs as the first line of treatment in addition to local HCs. Project Design: Instead of being a direct implementer, CS-20 involved the MOH as the principal owner of project activities, with support from CS-20 DAs operating as facilitators and as a resource for capacity building at the regional, district and community levels. Ten (10) CS-20 technical (district) advisors were involved in onsite/on the job training of CHWs, and integrated supervision with the CSCom Managers who were directly responsible for the DK Managers. The CHWs were trained to refer all critical cases (malaria, diarrhea, ARI) to the HCs (CSCom). CS-20 also assisted in furnishing support materials (such as BCC health education materials and data collection tools) and followed up with supervision visits to ensure proper data collection and management of the VDKs. The CS-20 approach was unique and designed to increase access to care, improve quality of care by establishing the VDKs as first line of treatment, and promote correct management of sick children not seen in HFs. The strategy also further sought to reinforce links between the communities (the DK Managers) and corresponding local HCs through an improved referral system. DK Managers learned to recognize danger signs for critical cases of malaria, diarrhea and ARI for referral to the HCs. Lastly, CS-20 sought to make community data available to the MOH HIS where it could be used for better decision making to improve service. Prior to CS-20, community data had not been routinely collected nor used consistently by health staff. Mission Collaboration: The USAID/Mali Mission has supported CS-20 activities from the design through implementation stages. In October 2004, the Mission provided a two-month advance of $70,446 to ensure the timely start-up of CS-20 activities. In addition, the Mission facilitated the introduction of ACT at the community level when the national policy changed to remove chloroquine as the first line of treatment in CCM of simple malaria. The Mission also obligated funds ($250,000) to support scale up of ACT in the DKs. SC has been an active member in all Mission-led meetings for its health partners including the annual review of the USAID Mali health strategy. The Mission will continue to support CS-20 through its no-cost extension phase and provide additional funding through the President's Malaria Initiative (PMI) to increase the availability of ACTs in the Sikasso Region. CS-20 Results Framework SO1: Increased practice of key household behaviors • 30% of children under 5 years old slept under a treated bed net the previous night. • 50 % of pregnant women slept under a treated bed net the previous night. • 30% of children < 5 with diarrhea in last two weeks were given ORS, appropriate home-based solution and/or increased fluids. SO2: Increased use of services at HCs and in the community (village drug kits) • 60% of children under five with fever who used drug kits or HCs. • 70% of pregnant women will receive malaria intermittent treatment. • 8% of all women use a family planning method. IR1: Increased availability of select MCH services • 50% of all villages have drug kits. • 50% of all villages have associations selling ITNs. • 80% of drug kits had no stock-out of FP methods in last 3 months. • 80% of drug kits had no stock-out of ORS in last 3 months. IR2: Improved quality of select MCH services • 50% of FP clients report they are satisfied with the services they received. • 60% of children <5 with difficult or rapid breathing who visited Drug Kits are appropriately referred to a HC IR3: Increased demand for appropriate health services • 80% of mothers know three or more childhood danger signs of illness. • 60% of mothers know where a drug kits is located. GOAL: Sustained reduction in under five mortality and morbidity IR4 and 5: Increased local capacity to implement CS-20 activities • HCs, District Medical Directors, MOH and SC collaborate in writing 2 articles for publication about DKs approach. • MOH and partners adapt and distribute an Operations Manual for drug kits. • 50% of DHOs will conduct at least two integrated supervisions of the HCs each year. • 60% of Oversight Committees will be supervised at least 1/month by the HC • 60% of management committees participating in the monthly meetings of the DK Managers at the HCs CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 3 C. DATA QUALITY: STRENGTHS AND LIMITATIONS CS-20 produced a sizable amount of data from multiple sources including: a) community data through the CHWs registers; b) monthly reports from CS-20 DAs; and c) CSCom monthly and quarterly reports. The table below summarizes the various dimensions in the data quality assessment. Table 2: Data Quality Assessment Data Quality Dimension Comments/observations Methodology (collection method) Data was generated and processed through various methods: KPC surveys, OR, CHWs logs, CSCom logs, and monthly summary logs produced by the DAs. The CS-20 M&E Officer entered routine information monthly into a central database using specific indicators. The CHWs used collection forms, notebooks/logs that were designed by CS-20 M&E Officer with specific indicators, as well as indicators from the MOH. The CSComs used data forms that feed into the health information system; some of the data collected by the DK managers have been integrated into the CScom forms. Accuracy/Reliability/ Quality The quality of the data from CHWs is not uniformly of the same quality due to low varying CHW literacy levels; some CHWs did not read or write French or Bambara well. As a result, their logs were not always filled out completely or accurately. At the CSCom level, there were also some gaps in the routine data for some indicators, otherwise, for the most part the data was complete. The definition of the indicators was standard. Data Usage The routine data was used by project staff during their monthly meetings to assess progress and make adjustments in activities as needed. The same analysis also fed the annual reports. However since the MTE results, it was noted that the data was not applied to its fullest potential for decision-making. It was recommended that the amount of data be streamlined for better use at all levels (project and MOH) especially integrating it into the HIS. Since most of the project indicators did not fit into the HIS, the challenge remains for the project and the HZs to make use of the information at their levels. Accessibility The project database contained routine data collected monthly and was available to project staff for use. The survey results and OR data were disseminated to all partners. The annual reports were also shared widely. Integrity and Soundness Both MOH and CS-20 indicators use internationally accepted statistical formulas to measure morbidity. For the purpose of this evaluation, the data was adequate to measure progress of CS-20 objectives. The remaining challenge (also noted in the MTE) is how to support a more timely analysis of the data to inform decision-making both at the HZ and district levels. The accuracy and reliability only come into question as far as the community data since low literacy rates of some CHWs may have affected accurate reporting. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 4 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 5 Box 1: Examples of Data Application for Decision Making Typhoid Fever Outbreak in the Village of Djene In 2006, during a supervision visit in the village of Djene, the DA noticed from the DK Manager's records, that there were more than eight deaths of children under 5, from high fever. CS-20 notified the CSCom Chief and together with the CSRef Chief, the project launched an investigation. After interviewing the parents about probable causes for the fever, the District Medical Chief sent a technical team to the village to investigate further, as parents attributed the fever to a spiritual cause rather than a medical cause. Their findings showed that the source of drinking water was contaminated with fecal matter and that typhoid fever was the cause of death. The team immediately intervened by treating the well and all water points in the neighboring villages. In addition, the DK Managers received training on water treatment and sanitation. Without the data from the DK Managers, this investigation would not have been promptly initiated and more deaths could have resulted. Outbreak of Diarrhea in Sibirila In 2008, data from the Sibirila VDK showed a high sale of zinc tablets suggesting that diarrhea was a problem in the area. The DAs for the zone went to investigate and found that there was an epidemic. Immediately, CS-20 notified the CSCom Chief and the district. A technical team including the HC Chief for the area, was sent to investigate and to intervene; they treated the drinking water with chlorine and at the same time treated all the diarrhea cases in the village. Further investigation revealed that the water source was contaminated; the water source is a rainwater catchment system that the village built to catch rainwater since there is a chronic water shortage in the area. It was noted that the village had been advised to relocate to another area because of this water problem but no one had relocated. Subsequently, the District Health Chief approached some partners in the region and asked them to look into resolving this water shortage issue. Since then, one company has agreed to intervene and has started a hydraulic project to increase access to potable water in the region. If it were not for the prompt analysis of the data from the DK Managers, the epidemic could have been more severe. These two examples illustrate the timely application of CS-20 data by health officials in two different HZs, to improve the children's health and reduce the potential number of community deaths. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 6 D. PRESENTATION OF PROJECT RESULTS Table 3. Revised CS-20 Project Indicators Objectives/Activities # Indicators Source Baseline Final Target IR1:. practice of key household behaviors 1 % of children < 5 who slept under an ITN the previous night KPC 8.4 70.6 30 IR2. Increased use of services at HCs and in community (VDKs) 2 % of pregnant women who slept under an ITN the previous night KPC 25.9 66.2 50 3 % of children < 5 with fever in the last two weeks used DK or HCs KPC 25.8 82.8 60 4 % of children < 2 with watery diarrhea in the last two weeks were treated with ORS, an appropriate home based solution and/or increased fluids KPC 17.5 67.7 30 5 % of pregnant women will receive malaria intermittent preventive treatment (SP) KPC 7.4 41.3 70 6 % of all women using a FP method KPC 5.1 14% 8 IR2. Increased availability of select MCH services in the community 7 % of 244 new DKs planned to be created are installed 100 (477) 95 8 % of DKs that had no stock-out of FP methods in last 3 months. MOH M&E system 100 80 Improved quality of select MCH services 9 % of children < 5 with difficult or rapid breathing who visited DK are referred to a HC KPC 25 44.7 60 Increase demand of appropriate health services (using knowledge attitude and assess as proxies of demand) 10 % of mothers who know three or more childhood danger signs of illness KPC 60.7 62.7 80 11 % of mothers who know where DK is located KPC 30.5 74.5 60 12 % of women who state that ITNs are very important for children under 5 KPC 32 59.7 80 IR 3: Increased capacity of 5 DHOs and regional MOH to support CS-20 activities 13 % of DHOs complete two integrated supervisory visits of HCs in the last year. MOH 20 50 14 % of DHOs will conduct monthly meetings to monitor the HC activities. MOH 40 80 IR-4: Increased capacity of community (village and HC Board) to effectively address health needs of mothers and children. 15 MOH and partners adapt and distribute an Operations Manual for DKs and ITNs. FE Draft manual 16 % of OCs that will be supervised at least 1/month by the HC Capacity Assess￾ment 20 70 60 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 7 Objectives/Activities # Indicators Source Baseline Final Target 17 % of management committees participating in the monthly meetings of the DK Managers at the HCs Capacity Assess￾ment 0% 80% 60 18 % of DK that had no stock-out of ORS in last 3 months (July to September 09) M&E system - 90 80 Table 4: Key Indicators by District 1, CS-20/Sikasso Yorosso Kolondièba Bougouni Yanfolila Sélingué Base￾line 2004 Objec￾tive 2009 FINAL 2009 Indicators 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2009 % of children <5 slept under an ITN the previous night 18.6 15.0 5.5 15.2 2.1 18.6 5.8 27.4 10.2 23.9 8.4% 30% 20% % of children <5 slept under a net the previous night (treated/not) 40.8 61.0 11.4 71.3 8.3 77.1 11.7 65.1 29.7 78.6 25.9% 70.6% % of households in possession of at least one bed net 79.8 86.9 33.7 92.7 32.1 85.9 40.0 89.6 45.3 92.0 82.8% % of pregnant women slept under mosq. net 7 nights/week during most recent pregnancy within past 2 years 46.4 59.0 21.2 37.3 15.0 86.1 20.4 46.5 26.8 58.9 25.9% 50% 66.2% % of children <5 with fever in the last 2 weeks who used DKs (DKs) or HCs 21.4 67.4 16.5 63.6 36.0 70.0 23.2 82.9 41.0 50.0 25.8% 60% 69.8% % of children <2 with watery diarrhea in the last two weeks were treated with ORS, an appropriate home-based 25.7 58.0 18.4 95.8 14.0 36.4 14.8 73.3 10.2 70.6 17.5% 30% 67.7% CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 8 Yorosso Kolondièba Bougouni Yanfolila Sélingué Base￾line 2004 Objec- FINAL tive 2009 2009 Indicators 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2009 solution and/or increased fluids % of pregnant women who received malaria intermittent preventive treatment or chemoprophylaxi s during most recent pregnancy 59.7 68.1 71.6 79.4 72.2 68.4 69.3 88.2 74.3 77.6 7.4% 70% 72.9 % of women who received at least one dose of SP during last pregnancy 36.6 44.4 36.6 42.2 43.8 41.3% % of pregnant women who received iron supplementation during most recent pregnancy 47.8 68.9 65.5 79.1 62.2 91.8 58.7 92.6 66.3 95.1 77.5% % of all women using a modern method of FP 7.6 27.3 7.0 12.5 3.2 10% 3.5 14% 4.0 6.7 5.1% 8% 14% % of children <2 with difficult or rapid breathing who visited DK are referred to a HC No DK 50.0 2 w/ IRA @ DK 63.6 2 w/ IRA @ CP 60.0 0 w/ IRA @ DK 50.0 No DK 0 25% 60% 44.7% % of mothers living 5 km or more from a HC know where a No DK 81.7 34.5 65.1 23.1 58.9 44.3 83.4 No DK 83.3 30.5% 60% 74.5% CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 9 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 10 Yorosso Kolondièba Bougouni Yanfolila Sélingué Base￾line 2004 Objec￾tive 2009 FINAL 2009 Indicators 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2004 2009 2009 DK is located % of women state that ITNs are VERY important for children <5 28.1 57,9 24.4 46,6 34.2 74.1 35.5 62,4 38.0 57,7 32% 80% 59.7% % of women state that ITNs are VERY important for pregnant women 27.2 64,4 22.9 67,4 29.3 74.0 33.3 87,2 37.7 66,4 NA 71.9% E. DISCUSSION OF THE RESULTS To summarize, CS-20 addressed all of its objectives and achieved almost all of its targets while exceeding some. Of the ten indicators measured at baseline and final household surveys, seven exceeded their targets while the other three exceeded their baseline values but did not reach their target. In 464 villages, 477 VDKs were installed with a principal DK Manager and a substitute, both trained in CS-20 interventions. In addition, 75% of the women interviewed knew the location of DK in their villages. While mothers’ knowledge about illness danger did not reach the desired target (80%), the results of the household survey show that there is a slight increase from 60% to 62% in knowledge. It is not clear whether this reflects poor formulation of the question during the interview or the fact that mothers’ knowledge did not increase significantly. E1 Malaria Management and Prevention Indicator (Household Survey) Baseline Final Target 1. % of children < 5 who slept under an ITN the previous night 8.4 70.6 30 2. % of pregnant women who slept under an ITN the previous night 25.9 66.2 50 3. % of children < 5 with fever in the last two weeks used DK or HCs 25.8 82.8 60 4. % of households in possession of at least one bed net NA 82.8 NA 5. % of women state that ITNs are VERY important for children <5 32 59.7 80 6. % of women state that ITNs are VERY important for pregnant women NA 71.9 7. % of pregnant women who received malaria intermittent preventive treatment or chemoprophylaxis during most recent pregnancy 7.4 41.3 70 Routine CS-20 M&E Data August 08 to July 09 8. # of long lasting nets distributed to children < 5, and pregnant women (CSCom) 68,389 9. Pregnant receiving 2 doses of IPT (CSCom) 15,731 10. # of sick children < 5 who the DK Manager referred to CSCom for signs of malaria OR severe malaria 2,111 The malaria management and prevention activities were based on a compelling BC strategy that consisted of dissemination of specific health messages on fever case management, recognition of danger signs, counseling mothers of children with fever, promotion of use of long lasting treated bednets for children under 5 and pregnant women, promotion of ANC visits, and SP treatment for pregnant women. As a result, all malaria objectives exceeded their targets except for the IPT coverage for pregnant women which progressed from baseline but did not achieve target. For example, the percentage of children < 5 and pregnant women sleeping under ITNs the previous night rose from 8.4% to 71%, and 26% to 82%, respectively. The household survey also reported that 83% of the households interviewed possessed at least one net (whether the net was long lasting was not specified). The routine data from last year alone show that between August 2008 to July 2009, 68,369 long lasting nets were distributed to children up to 5 years and pregnant women in all five districts. As for care seeking behavior, 22,262 cases of fever were seen and treated at the CSCom as well. The referral data from the routine data also shows that DK Managers referred a combined 2,111 cases of suspected severe cases of malaria to the CSCom. From the final CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 11 household survey, again we see that in the week prior to the survey, the percentage of mothers of children with fever who sought care either from DKs or CSComs increased from 26% to 83% exceeding the 60% target. More cases of fever were seen and treated at the CSCom instead of at the VDKs since last year when the MOH policy changed and shifted ACT availability to HCs only. As a result, ACTs were removed from the VDKs except in Kolondieba where ACT distribution at the village level was successfully piloted the year before. In the other districts as well as nationally, ACT stock-outs were also noted during the final evaluation (FE). Fortunately, the situation is changing. In August 2009, the MOH changed the policy again to allow ACT back at the VDKs as long as the DK Managers are trained on the new ACT brand protocol. Over the next six (6) months, under the no-cost extension, the USAID mission will fund restocking of the VDKs with ACTs for the Sikasso Region. While IPT coverage did not quite achieve the 70% target, this indicator progressed dramatically nonetheless from 7% at baseline to 41% at final. The routine data is similarly encouraging showing that from August 2008 to July 2009, 15,731 pregnant women received two doses of IPT during their ANC visit at the CSCom. Despite this encouraging number, the routine data for the same period show that twice as many pregnant women (31,851) received a long lasting net during their ANC visit. These data support the household survey result of IPT indicator missing the target. Similarly, the percentage of mothers who think that it is very important for children < 5 to sleep under net did not reach the target of 80%; nonetheless, there was significant progress since that indicator almost doubled from baseline to final: 32% to 60%. This shows that mothers have appreciated the utilization of ITNs for their children < 5 and that they are increasingly become more cognizant of the link between the use of the net and the prevention of malaria. E2. Family Planning Results Indicator (Household Survey) Baseline Final Target % of “all” women using a FP method 5.1 14 8 % of DKs with no stock-outs in the last 3 months - 100 80 The FP intervention was very successful as its objective of WRA reporting using a modern family method almost tripled to 14% from 5% at the baseline assessment, and exceeding the 8% target previously set. This achievement was highly appreciated at all levels (HCs, district and regional) during the FE interviews. All of those interviewed noted that more women were requesting some type of FP method and that the HIS data was showing higher use than the national level of 5%, in all CS-20 districts. The results of the household survey further show that Depo Provera (51%) was most in demand by the women at the HCs and that pills were the second choice with Contraceptipe Prevalance by Method: DK vs Health Center (CSCom) 0.13 0.22 0 02 0 01 0.02 0 09 0 0.05 0.1 0.15 0.2 0.25 DK CSCom DK CSCom DK CSCom Pilplan Ovrette Comdom CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 12 32% of women claiming it as their modern FP method in use. The MTE found that CS-20 needed to refocus the FP strategy towards men since sometimes husbands refuse to allow their wives to use FP. At the baseline, it was noted that men were concerned about promiscuity, religious issues, and generally felt that FP was not natural. Therefore, it was recommended that the FP strategy be broadened to enlighten them and get them more involved in learning more about the benefits of FP for their wives' and children's health. Unfortunately, due to lack of time and resources, it was not possible to add additional activities to focus on men and religious leaders, as was accomplished in CS-20 Ségou. Regardless, the FP messages reached the women and demand increased. Of all of the five districts,Yorosso made the most dramatic progress from 7.6% to 27.3 %, while Selingue (the smallest district) progressed slightly (from 4% to 6.7%). Furthermore, the routine M&E data shows that none of the DKs had stock-out of oral contraceptives in the last three months prior to the FE (June –August 2009). No other constraints were noted for FP. When comparing the contraceptive prevalence rate by method, we noticed that for Pilplan and condoms, it was higher at the CSCom level (HC) instead of the VDKs. The routine data also shows that higher sales Pilplan and condoms at the CSCom than at DKs. E3. Diarrhea Management and Prevention Results Indicator Baseline Final Target 1. % of children < 2 with watery diarrhea in last two weeks were treated with ORS, appropriate home based solution and/or increased fluids 17.5 67.7 30 Routine CS-20 M&E Data August 08 to July 09 2. # of children <5 w/ diarrhea treated by the DK 7,500 3. # of children <5 w/ diarrhea treated by the CSCom 5,404 4. # of children <5 w/ diarrhea treated with zinc by DK 7,224 5. # of children <5 w/ diarrhea treated with zinc by CSCom 5,216 6. # of sick children < 5 who the DK Manager referred to CSCom for danger signs of diarrhea 182 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 13 % of (Children < 2) Use of ORS/Other Fluid During Diarrhea in Last 2 Weeks 26 18 14 15 10 58 96 36 73 71 0 20 40 60 80 100 120 Yorosso Kolondièba Bougouni Yanfolila Sélingué % 2004 2009 ORS Treatment at DK Level (August 08-July 09) 96.6 95.6 88 88 91 48.8 0 20 40 60 80 100 120 Bougouni Kolondièba Yanfolila Sélingué Yorosso Total % Case management of diarrhea was equally successful. The objective was tripled from 17% at baseline to 68% at final, which surpassed the 30% target set by CS-20. In all five districts, significant progress was made. The cases of diarrhea reported by the mothers by district are as follows: Kolondieba (25%), Bougouni (16%), Yorosso (19%), Yanfolila (34%) and Selingue (19%). Of the mothers who sought assistance outside the home, those from Yorosso sought the DK most (49%), then Bougouni and Yanfolila (47%), Kolondieba (22%) and from Selingue, the least (16%). The routine data for last year shows that zinc was utilized more at the DK level than at the CSCom level. 96 70 96 0 10 20 30 40 50 60 70 80 90 100 DK CSCom Total % of Diarrhea Treated with Zinc: DK vs CSCom (August 08-July 09) CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 14 At the time of the FE, both ORS and zinc were in stock at most of the DKs. There was no stock-out noted at the CSComs and the CSRef depots. Regardless, the DK Managers and the DAs have reported that the use of ORS has increased in all districts. CS-20 was consistent in providing an enabling environment for case management of diarrhea at the community level. OR on the use and acceptance of zinc for the control of diarrheal diseases (CDD) conducted at the initial stage of CS-20, made it possible to scale it up at all project sites. Zinc was also added on the national essential medicine list. In addition, supplemental funding from American Idol foundation facilitated a continuous supply of zinc at the CSCom, CSRef and the VDKs. Also, in all districts the same strategies were used in terms of the messages regarding use of ORS/zinc via radio, village health talks, DK Managers counseling mothers on danger signs. E4. ARI Management and Prevention Results Indicator Baseline Final Target 1. % of children < 5 with difficult or rapid breathing who visited DK are referred to a HC 25 44.7 60 2. % of mothers who know 3+ danger signs 60 62 80 The focus of this intervention was on the referral capacity of the DK Managers, mothers' knowledge of child illness danger signs, proper care seeking behavior, and whether or not those referred made it to the HC. DK Managers were trained how to identify rapid breathing, how to counsel mothers, recognize danger signs, and conduct proper referral. BC messages for better care seeking were disseminated via radio, village health talks, and counseling of mothers. National health policy does not support ARI community-based management; therefore, there was not drug treatment involved with this intervention. As a result, the improvement on this indicator was lower than hoped for and the target was not reached. However, referral of children from the DK to CSCom increased from 25% at baseline to 45% at final. According to the household survey data, in addition to seeking care from DKs, some mothers sought care directly from the CSComs without going through the DK Managers. In Bougouni, of the 61% of mothers who sought care outside the home, 25% went directly to the HC; in Konlondieba (25% of 80%), Yanfolila (24% of 64%); Selingue (12% of 58.3), and Yorosso (24%). Referral Compliance: The below graph indicates that referral compliance was best for diarrheal cases (87% compliance), least for cases of ARI (32%) and for fever (39%). This compliance shows mothers who were referred to the CSCom and actually arrived there according to the routine data collected (October 2008 to July 2009). Yet, it was noted that sometimes mothers want to follow-up but significant barriers exist with respect to care seeking, and/or getting to the HCs once a referral was made. Key barriers that were mentioned during the evaluation include: the household decision maker (husband/father) did not agree to care seeking for the child, the high cost of transportation to bring the child to the HCs, and competing household demands on mothers' time. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 15 Referral Compliance for All Diseases (October 08-July 09) 1093 2111 182 351 832 159 0 500 1000 1500 2000 2500 ARI Fever/Mal Diarrhea Referred Complied Referral Compliance Rate for All Diseases to CsCom by DK Managers for Children (0 to 5 yrs): Oct 08-Jul 09 0 20 40 60 80 100 120 Bougouni Kolondièba Yanfolila Sélingué Yorosso Total ARI Fever/malaria Diarrhea Another indicator that was helpful in order to understand compliance of mothers with referral is their knowledge of childhood illness danger signs: limited progress was made (from 60% at baseline to 62% at final) and the 80% target was not reached. From the baseline, the findings showed that a lack of knowledge of danger signs also contributed to poor care seeking behavior. It is not clear what the barriers in obtaining knowledge were. In all districts, women had access to radios and they reported listening to the radio mostly in the morning. CS-20 had contracts with radio stations in the region to disseminate health messages, in addition to the village health talks conducted by the DK Managers. During the MTE, it was recommended that messages around ARI be reinforced as single (focused) messages and through one-on-one counseling with mothers. At that time, the weaknesses of the ARI intervention were beginning to show. Since the Center for Research on Child Survival has made ARI a priority, it is expected that further guidance will be provided on case management of ARI. Utilisation of products by the mothers 0 50 100 % ORS Zinc Paracet Pills Condom ORS 77 36 76 78 66 Zinc 62 44 68 67 48 Paracet 77 65 80 90 81 Pills 26 18 28 51 47 Condom 9 2 17 44 40 Yorosso Kolondièb a Bougouni Yanfolila Sélingué 31% 60% 74% 0% 20% 40% 60% 80% Baseline (2004) Objective Final (2009) % of Mothers Interviewed Living 5 km+ from HC & Know Location of Village DK CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 16 % of Mothers Interviewed Who Know at Least Three Child Illness Danger Signs 61% 80% 63% 80% 70% 60% 50% 40% 30% 20% 10% 0% Baseline (2004) Objective Final(2009) Utilization of products at VDKs: In all five districts, routine data from 2008-2009 show that demand is consistent for ORS products, zinc, and Paracetamol from the VDKs. E5. Capacity Building of Partners Results Indicator Baseline Final Target % of OCs that were supervised by the HC 20 75 70 % of management committees participating in the monthly meetings of the DK Managers at the HCs 0 80 60 In December 2009, Save the Children conducted a capacity assessment of a representative sample of VDK oversight committees (OCs) in order to compare their status to the baseline capacity assessment conducted in 2005. Twenty oversight committees (4 in each district) were randomly selected and focus groups were conducted with the membership using interview guides. Focus groups were used although it was not too formal because of the limited size of the groups surveyed and the availability of the members. The final survey of the organizational capacity of the OCs indicates that there has been significant progress made in the capacity of the OCs to function effectively. The baseline study found that in the majority of the cases, the management tools were not up-to-date and were not accessible during the study team visit. The final study found that 100% of the OCs reported that they had the four types of management tools and 60% of the tools were readily available to be inspected and were properly kept and up to date. In the baseline only 36% of OCs included women in their membership while in the final assessment 55% included women who most often functioned as the treasurers of the OC. Oversight Committees were able to articulate their role in planning and conducting health activities in their villages especially their responsibilities to plan for the restocking of the VDKs, health promotion and the promotion of ITNs. 90% of OCs reported to have received training in management, 80% in monitoring, 95% on their roles and CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 17 responsibilities, and 55% on disease prevention and control. At baseline, 11% of OC reported to have received a supervisory visit from the staff in the nearest health center while in the final survey 75% had been visited. As in the baseline survey the large majority of OCs were satisfied with the work of the DK managers. While in the baseline, the majority of OCs reported that there are no meetings between their organization and the Health Center team during village health activities, in the final survey 50% of OCs report to have met with the health staff during their outreach visits in their villages. Similarly while the baseline survey found that feedback to the village was rare, in the final survey 70% of OCs reported giving feedback to villages leaders. The baseline study revealed weak collaboration of groups or associations by the OC for the implementation of village health activities, while the final survey found that 65% of OCs worked with other groups or associations of women, young people, market-gardeners, traditional obstetricians, protectors of newborns, hunters, and village associations for certain health activities. Most OC positions did not have a set time period for renewal at baseline. In the final survey the majority of OCs plan to renew their membership every 3 to 5 years. E6. Village Drug Kits Results Table 5. Performance of Village Drug Kits by District District Drug Kits Performance (Score 50 to 89) Selingue 28 26 (93%) Yorosso 56 55 (89%) Bougouni 182 137 (90%) Yanfolila 103 103 (99%) Kolondieba 109 109 (94%) Total 478 432 (90%) Table 6. Overall Performance of Village Drug Kits Drug Kits Performance Level Score # of DKs % Very High 65-89 281 58.8 High 50-64 147 30.8 Medium 30-49 28 5.8 Weak < 30 22 4.6 Total 478 100.0 Overall, 90% (428 out of 478) of the DKs scored between 50 to 89 points, translating to a "High" or "Very High" performance level. The score was set up on a point system to measure the physical status of the DK itself, and included: 1) whether they contained most of the medicines (at least 7 out 9); 2) whether the tools (i.e. data collection forms were filled out); 3) whether the CHWs were literate, or had been replaced; and 4) whether the OC and health village team were active. The maximum number of points that a DK could score was 89; if the criteria were not CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 18 met, then there were no points awarded. In Selingue, Yanfolila, and Kolondieba, more than 90% of the DKs performed well. Yorosso was at 89% and Bourgouni performed at 90%. When the scores from all the five districts were pooled together, only 59% of the DKs performed very high and 31% performed high. At the lower end, almost 5% of the kits were found to be weak. During the course of the FE, most of the DKs were experiencing some stock-outs of common medicines such as antimalarials and paracetamol syrup for children. This issue is discussed more extensively in other parts of the report. The MOH (district, region and HC levels) overwhelmingly felt that the VDKs were an effective intervention and that this model could be replicated nationally. Factors Affecting Achievement: 1. Partnership with the communities: The collaboration with the CHWs was most notably the biggest factor that contributed to the success of CS-20. The CHWs were hand-picked by the villages using criteria provided by CS-20 for the successful management of the VDKs. The CHWs successfully mobilized the community around the VDKs to influence care seeking behaviors by mothers/women. They provided the first line of treatment for childhood illnesses such as malaria, and diarrhea. In addition to treatment, the CHWs also served as counselors, conducted home visits, collected morbidity and mortality data on children under five, and data on modern FP method use that was eventually integrated in the CSCom quarterly reports. 2. Capacity building: The excellent training and supervision strategy also provided an enabling environment for the CHWs to thrive. Each supervision visit was an opportunity for the DAs and the CSCom Managers to provide on-the-job training and continuing education for the CHWs. The DKs interviewed during the FE knew the correct protocol for treatment of child with fever, cough and for diarrhea. Continously, the DAs provided support to the CHWs in their community outreach efforts. 3. Malaria treatment policy change: After Chloroquine was discontinued as the first line of treatment, another policy change in ACT administration halted its availability at the community level; as a result, there was a major stock-out at the DK level and a disruption of treatment of simple malaria at the DK level. 4. Restocking of the VDKs: While this was not directly measured in terms of its impact on achievements, it was clear during the FE that restocking the DKs was a major issue. Many DKs ran out of paracetamol (syrup form) for infants. The DK Managers expressed frustration in having to rely on the CSComs to restock them regularly and in a timely fashion. Some CSComs in turn, had trouble receiving the DK orders on time in order to combine them with their central orders. In some instances, DK Managers had to restock directly from the CsRef (this was not encouraged), or from a private provider. In some cases, the DA themselves would procure the drugs from the CsRef. The entire restocking procedure should be revisited and integrated into the regional drug supply chain system. Despite these challenges, the DK Managers who were interviewed remain motivated oveall. Their BCC materials were in good conditions and project staff reported few non-performing DK Managers. 5. Literacy of the CHWs: The low literacy level of some CHWs also affected their performance in maintaining records, and data collection. Consequently records from these CHWs were difficult to decipher and not very useful. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 19 6. Attendance of monthly CSCom meetings: Some CHWs did not attend monthly meetings regularly. One of the most common barriers cited by the CHWs for lack of their attendance to the CSCom monthly meetings, was a lack of transportation. Consequently, their monthly orders to restock the DKs could not be integrated into the CSCom orders, and their data could not be used for planning purposes in a timely fashion. 7. Strategy to increase access to health services: The establishment of the VDKs was the key strategy for increasing access to health services. A total of 477 DKs were set up in 464 villages and 2 DK Managers (principal and aide) were trained per village. An OC was established to supervise the activities of the DK Managers and manage the funds generated by the sale of the medicines (Artesumate and Amodiaquine combination in a pilot study, paracetamol tablets and syrup, zinc , aureomycine 1% drops for eye infections, condoms, and oral contraceptives). During the FE, there was a stock-out of paracetamol (syrup), and ACT in almost all of the DKs. There was some debate at the national policy level about making ACT available at the village level for fear of mismanagement of treatment. This issue was resolved within the last two months of the project (August 2009) with a protocol to train and supervise the CHWs in the management and distribution of ACT. CS-20 has already trained all of its CHWs on the use of ACT. Zinc is continuously available through special private funding obtained by SC which is supplying all of the main drug depots at the CSRef (district) level. 8. Oversight Committees: The final survey of the organizational capacity of the OCs indicates that there has been significant progress made in the capacity of the OCs to function effectively. The OCs are critical to providing a strong link between the village leadership, the ASACOs, other community groups, and the drug kit managers. 9. ASACO: The involvement of the health boards in the running of the HCs and CS-20 activities was not uniform in all districts. It was clear that in the districts where the ASACOs were involved (i.e. participation at CSCom monthly meetings, provided incentives to CHWs such as a meal), the CHWs were more engaged, and the CSComs were more efficiently run with less stock-outs. 10. Joint Supervisory visits: CS-20 facilitated the joint supervisory visits of the VDKs with the CSCom and the CSRef Chiefs whenever possible. Capacity Building/Human Resources Strategies (skills-based training and supervision) • CS-20 invested a great deal of resources and effort into the cascade training approach, printing of support materials and funding of supervision activities. The first level of training (training of trainers - TOT) at the regional level, targeted the MOH district health staff teams (CSRef), and SC’s DAs in the five districts. This training was held once at the beginning of the project. At the district level, HC staff trained the DK Managers, and the OCs. The topics of the training for the HC staff and the DK Managers included malaria prevention and control, pneumonia case management, diarrhea case management, and FP. The DAs provided follow-up support supervision and on-the-job training at the community level for the DK Managers. • Based on the various interviews at the community level, there is evidence that the training was effective since the DK Managers were knowledgeable about the treatment protocol, the health messages, and their responsibilities. The DK Managers also mentioned that they were satisfied with the training because they learned how to provide health advice to mothers as well as treatment and referral for sick children. Ad-hoc, onsite training was provided during CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 20 supervision visits. The only challenge encountered was the high turnover of staff at the district and CSCom levels, which led to loss of capacity/momentum, and the need to retrain new staff as well as start new relationships with the CHWs and HC Managers. Close follow￾up was necessary to ensure that new staff were well-oriented on CS-20 innovations. • Since there was no pre- or post-test training, informal assessment of their knowledge on treatment protocols through direct observation of the CHWs' interactions with the mothers, was a useful method to assess the effectiveness of the training. While this was not measured systematically, there is enough evidence to show that the training was effective through the CS-20 indicators which progressed well beyond target for increased used of ORS, and use of modern FP methods. Health Systems Strengthening and Information Management System Strengthening One of the project's objectives was to improve quality of services by strengthening the HIS with more data that would better inform the decision-making process. For this reason, CS-20 designed community data collection forms which would be useful to the health system. The data was to be used to better assess causes of morbidity and mortality of children under five, and help the system design programs to address these problems. This was successfully achieved through the life of the project despite some minor delays related to the delivery of data to the HCs, and the integration of the data at the regional level. During the interviews held at the regional and district levels, health staff highly appreciated the data and felt that it was useful for them to know what was happening at the community level. Quality Assurance CS-20 mainly used training and supervision for quality assurance at the district level. The emphasis was on correct training to manage severe malaria, diarrhea and respiratory infections, manage malaria in pregnancy, administer the right treatment protocol per WHO guidelines, and promote the use of ITNs. The project also placed two DAs per district (10 in total) to provide technical assistance to the district health team as needed through ad-hoc/refresher training, joint supervision visits of the DK Managers, and also monitor the restocking of the DKs. At the community level, the DAs ensured that the right drugs were available in the VDKs. CS-20 created an OC per village to oversee the activities of the DK Managers; the members of these committees came from the village health teams and were selected by the communities themselves using criteria developed by CS-20. More specifically, DAs also worked directly with the CHWs to ensure that they followed the correct treatment protocols, and provided on-the-job training as needed. Policy Dialogue and Advocacy at the Local/National Level CS-20 contributed to the national policy dialogues to introduce ACT as the first line of treatment for malaria at the community level and zinc for the management/treatment of diarrhea. OR conducted by CS-20 showed that introducing ACT and zinc at the community level could greatly improve the management of malaria and diarrhea at the community level. Subsequently zinc was added to the list of essential medicine and ACT was introduced into the VDKs. Another contribution of CS-20 was the demonstration of the invaluable role that CHWs play in the community health system. CHWs were the main partners at the community level, and they managed the DKs and administered the first line of treatment for simple malaria and diarrhea cases. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 21 Role of Key Partners Partner Role Result of Collaboration District Health Services Responsible for training CSCom Chiefs, participated in integrated supervision and provided feedback through monthly meetings, and collated data from various HZs. The CSRef was also responsible for restocking the CSComs who in turn would restock VDKs. They were also supposed to integrate CSCom data into quarterly data forms to be sent to regional HIS system. For the most part, the collaboration was positive as CS-20 DAs worked closely with district health teams on integrated supervision issues, restocking of the CSComs, and introduction of zinc and ACTs into the VDKs. The challenge arose when the CSRef ran out of supplies and thereby caused a negative chain reaction in the restocking of the CSComs and VDKs. When the national policy changed for free ITN distribution, the district health team was also very instrumental in making sure that the ITNs were available for the CS-20 HZs. Turnover of district staff also caused some disruption for CS-20 activities, as new chiefs had to buy into CS-20 innovations and activities. The MTE had recommended a checklist or turnover package to assist newly hired staff and shorten the learning curve. The same recommendation stands. HCs (CSComs) Responsible for training CHWs, supervision through monthly meetings and provided feedback, collated community data villages served by their HZs. They are also responsible for placeing orders to restock the DK. This collaboration was generally productive. Most CSComs Chiefs were engaged in CS-20 activities. The frequent stock-outs experienced by HCs caused some disruption for VDKs. With drug stock-out (paracetamol, ACTs), visitation of the DKs decreased. Also when the staff turned over, the new staff had to get oriented to CS￾20 activities and develop new relationship with the DK Managers who had to learn the new management styles of the CSCom Chiefs. This was not always a smooth transition. It is hard to quantified the direct impact on the project, but it was noted during the interviews that not all CSCom Chiefs collaborated well with CHWs. As a result management of data quality/exchange and restocking of DKs became challenges in some HZs. Mayors A protocol of collaboration was established between the CS-20 project and the mayors to define the areas of collaboration and their support for project activities. The result of the collaboration was not very obvious. In many cases, the mayor was also the head of the ASACO (health management board for the zone). If the ASACO was active, then it meant that the mayor was active. Otherwise, the contribution was insignificant. Association of Community Health (ASACO) The Boards of Directors are included in all the activities at the community level. They facilitate the drug supply of the CHWs and take part in their supervision. CS-20 lacked a strong collaboration with the ASACOs; the same issue was brought up during the MTE. As the Board of Directors of the CSCom, they should have played a more central role in CS-20 activities. For example, they could have helped with the restocking issues faced by most DKs, facilitated communication between the CHWs and the CSCom Chiefs, and take part in supervision of the DK Managers. It is recommended that ASACOs play a more central role in any health project with a community component. Oversight This three-person committee The final survey of the organizational capacity of the OCs CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 22 Partner Role Result of Collaboration Committees was created by CS-20 at the village level and included a treasurer and representatives from youth and women's groups. Their role was to oversee the operation of the VDKs, help the DK Managers with projecting DK restocking needs, and conduct supervisory visits. indicates that there has been significant progress made in the capacity of the OCs to function effectively. The OCs were critical to providing a strong link between the village leadership, the ASACOs, other community groups, and the drug kit managers. CHWs and Village Authorities CHWs were key in implementing CS-20 activities (health talks, counseling, home visits, outreach) at the village level. They managed the DKs, provided first-line treatment for CCM of diarrhea and simple malaria while referring all critical/serious cases to the CSComs. They also collected community data (births, morbidity and mortality) that they submitted monthly to CSComs. This relationship was very successful. Despite the lack of tangible/continuous incentives, the CHWs were consistent in the DK management and remained interested throughout the life of CS-20. The CHWs conducted health talks, provided the first-line of treatment for CCM of diarrhea and simple malaria, made referrals for severe cases and for IRA, counseled mothers, conducted household visits, and collected community data. Their contribution is clearly exhibited through the achievement of CS-20 indicators; including care seeking by mothers at the DKs, purchased drugs, increased knowledge about dangers signs, seeking treatment for children at the CSComs once danger signs were recognized, and the increased use of ITNs for children under 5 and pregnant women. Regional MOH Office The MOH through the National Division of Health and the Regional Health Division supported the development of training manuals and the implementation of training courses. Joint supervision visits were organized with the national and regional authorities in CS-20 districts. Information exchange was accomplished through the workshops and trainings organized at the regional or national levels. They regularly invited CS-20 staff to share project results. The Regional MOH office was a great supporter of CS-20 activities. They facilitated all the relationships with the district health teams who in turn motivated the health zones (CSComs) to implement CS-20 innovations. Overall Design Factors that Influenced Results The overall project design with the MOH as the main program implementer was key for ownership of the innovative approaches brought by CS-20, and central in going to scale geographically. CS-20 operated with a very limited staff structure, with the ten DAs providing technical assistance on an ad hoc basis with support and coordination with a small staff at SC's Sikasso office. This staffing model facilitated the complete integration of CS-20 activities into CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 23 the MOH structure without significant overhead costs. Over the life of the project, the advisors helped to seal some gaps among the district MOH office, the CESComs and the communities as they interacted and coordinated supervision activities at all levels. This strength in the design of CS-20 has significant implications for future projects at scale, in terms of data quality, financial sustainability and quality of care and services. For example, making the advisors directly available at the district level, reduced overhead/operating costs and freed more funding for project activities. The integrated supervision was another element in the design that also helped to reduce operating costs since it promoted collaboration among the various levels (regional, district, CSCom, and village). The formation of a “health team1 ” at the village level was more symbolic than functional. It really served to recognize the role that health leaders play in communities rather than served as a specific function. From that health team, a functional unit of three members, the “OC’ was created to supervise the VDKs. This part of the design of the DKs management did not contribute much to the overall management of the VDKs although the village authorities and representatives were trained in management skills. It is not clear whether or not the DKs would have had fewer stock-outs if the OCs had been more effective. Another example of the design that is worth keeping for future scale up initiatives is that CS-20 indirectly maintained a continuous demand on the regional and ultimately the national drug supply system through the VDKs. With the CHWs adding their monthly orders to HCs’ orders, it forced the system to “try” to respond to increased demand for drug and care from the communities. Theoretically, this could help the regional MOH improve its planning scheme by having a better picture of the morbidity status of the communities it serves and design interventions to respond more adequately to their needs. F. DISCUSSION OF POTENTIAL FOR SUSTAINED OUTCOMES, CONTRIBUTIONS TO SCALE, EQUITY, COMMUNITY HEALTH WORKER MODELS, AND GLOBAL LEARNING F1. Progress Toward Sustained Outcomes The design of CS-20 emphasized complete involvement of local partners, deliberately assigning an advisory role to SC staff, and designating partners as CS-20 implementers. CS-20 made it clear from the very beginning that this role would build partners’ capacity so that they could have complete ownership of project activities. CS-20 was also designed as an expanded impact project to replicate and extend the model of community health care using the VDK. Since the sustainability of these DKs depend on their integration into the larger health system and oversight by MOH and HC personnel, CS-20 mapped out project activities to increase local health partners’ capacity to do just that. The initial input such as training and stock for the DKs, served as a springboard and incentive to start the process. Monthly meetings and quarterly visits of CS-20 advisors, district health personnel and DK Managers helped to establish relationships and troubleshoot the system, allowing the local health system to find its own solutions. 1 The health team was formed by CS-20 project and comprised of village leaders, representatives of various groups in the community: youth, women, and associations. The village health team is not a formal structure of the communities. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 24 While this approach did not solve all of the challenges which arose (for example stock-out are still an issue for many DKs), local health partners realize that they have to find a solution on their own. It is clear that health services will continue at the community level, that the DK Managers will continue their activities, that mothers of children under five know where the DKs are located and they know to demand more services, and have learned to recognize illness danger signs and how to better manage fever, diarrhea, and ARI at home. The other major contribution of CS-20, is that it has demonstrated on a large scale, the critical role that CHWs play in the Malian health system. The data show that without these individuals, the system cannot be functional since it is the CHWs who do the outreach at the village level (i.e. through village health talks, household visits, provide first-line of treatment for simple childhood illnesses, and referral to HCs). Further, throughout the life of CS-20, CHWs (DK Managers) have remained largely active with few or no incentives. Very few DK Managers have defaulted and more than 75% of the DKs will continue to function once CS-20 withdraws completely from the picture. The challenge now being addressed by project partners, is finding a way to formally recognize the CHWs in the public health system. With CS-20 activities continuing in Sikasso due to the ongoing support provided by the regional health system after the project end date, the achievements and impact of CS-20 activities are expected to be sustained. With the upcoming no-cost extension through March 2010 and the Mission support to increase stocks of ACTs, VDKs will be able to restock ACTs and help increase DK use, product sales, and referrals to HCs. CS-20 capacity building package was substantial and comprehensive providing training in all aspects of disease management, BC, and business management of the DKs. This will help the local partners maintain a good standard of practice since they have been trained in local governance and financial management, in addition to the refresher training they received on technical interventions. The project goal had been for all activities to be integrated in MOHs as part of an effort to improve access to health care at the community level. Hence the CS-20 support package included cascade training whereby the regional and district MOH staff were trained to become trainers of the HZ staff, who in turn trained the CHWs. In addition, CS-20 reinforced supervision skills and promoted integrated supervision activities, as well as support tools to facilitate these visits. Data collection tools were also developed to increase access to community data that could be integrated into the national HIS. In sum, the project design was well thought out and implemented from the beginning, ensuring that key staff and partners had the necessary information and tools to implement a high-quality project designed to meet the identified community needs. The strategy of implementing a TOT approach to develop of cadre of well-equipped staff, combined with regional efforts focused on advocating for the DKs to be financially sustainable (with SC providing the initial stock) and the communities positioned to maintain them after the project ended, positioned project activities for success. Supervision of project activities by CS-20 Advisors gradually decreased over time, allowing partners to integrate DK supervision visits into their routine activities and to maintain communication with the DK Managers, while using the data collected for better decision-making for improved community health services. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 25 During the FE, the appreciation and the critical importance of the DKs were recognized as essential to increasing access to care in the most disenfranchised communities. DK Managers and the CSCom Chiefs will continue to work on options to restock the kits in a more timely fashion. F2. Contribution to Replication or Scale-up: a) Project Design: CS-20 was designed as an expanded impact project to replicate a community health care model using the VDK strategy and extend it geographically to five (5) districts to cover the entire region of Sikasso. The mentoring strategy was successful as it positioned/enabled the local MOH to take ownership of project activities and implement them directly. As a result, the number of CS-20 staff members was kept to a minimum of two DAs per district with their main responsibility being to build the capacity of local partners via training and supervision. This is an excellent model since it allowed CS-20 to have a small central office as the DAs were placed directly in the districts. b) OR on zinc: The zinc OR study at the beginning of the CS-20 was hugely effective. The communities in Sikasso widely recognized the benefits of zinc for diarrhea case management in the pilot project and subsequently, zinc was added to the national essential drug list as well as the VDKs in all five districts in Sikasso. As a result of CS-20's efforts, zinc is nationally recognized as a treatment for diarrhea case management in Mali. c) OR on the Effective Use of ACT: This OR activity started in Kolondieba (in 2006) through CS-20's introduction of ACTs for treatment of malaria by the DK Managers. These managers were trained on the ACT treatment protocol and the VDKs were subsequently stocked with ACTs. The results of the trial led to national discussion of making ACTs available at the community level. In August 2009, the National Malaria Control Program finally accepted the dispensing of ACTs through the community-based agents (DK Managers) as long as they were trained. Subsequently, the USAID Mission in Mali has agreed to support this activity to make ACTs available to all five districts in Sikasso Region through the VDKs. d) CHWs: CS-20 has demonstrated that there is an expanded role for CHWs to play in the case of the management of childhood illnesses. Since the beginning, CS-20 has been advocating at the national level to recognize this expanded role for CHWs. All CHWs in the project intervention areas in the five districts of Sikasso have been trained on treatment protocols for ACTs, zinc, DK management and counseling skills for the promotion of better care seeking behaviors. Currently, the national debate is around how to incorporate the CHWs into the health system. While the statistics show that access to health services through CHWs is less than 10% in the Sikasso Region, CS-20 has demonstrated that it can be significantly increased to even more than 75% once CHWs are engaged and trained properly. SC will continue to contribute to the national dialogue on the design of a sustainable strategy to bring the CHWs onboard in a more formal way in order to increase access to health services in rural areas of Mali. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 26 F3. Attention to Equity CS-20 deliberately chose to address geographic equity by intervening in Yorosso. Despite a decentralized health system, Yorosso still faces major limitations to the provision of health care to its population due to distance and poor infrastructure, that translates into low quality of services for a low-density population that is widely dispersed. Historically, the international community had neglected the district; when CS-20 began, SC was the only international PVO operating there. The other four districts of Sikasso had previously benefited from SC’s interventions through activities such as education, school health, strengthening of the local MOH system; as well as from interventions from other international organizations. The inequities in Yorosso were such that there were no VDKs, the CHWs had not been trained, there was no ACTs nor zinc available, and referral to the HCs was very poor. Because of CS-20 interventions, Yorosso District's needs have become better known and have attracted other donors. For example, Save the Children/Korea has started a five-year project to reinforce the activities of CS-20 and added a nutrition component with the support of UNICEF and SC's child sponsorship program. Currently in Yorosso, there are 56 VDKs and 112 trained DK Managers, where before there were none. As a result, the demand for services has increased and community health has improved. F4. Contribution to Global Learning CS-20 made several contribution to global learning including: a) the establishment of zinc as an effective treatment in the case management of diarrhea, based on the pilot study conducted by SC's Child Health and Nutrition Advisor in Madagascar in 2008; b) when trained properly, CHWs play a significant role in the public health system to increase access to health services in rural/remote areas; and c) VDKs are effective in increasing access to local health care and improve quality of care; the scale-up strategy was successful as it allowed local partners to implement project activities, with CS-20 staff playing an advisory role, thereby decreases costs and allowing for better integration into partners’ system. This model is worth replicating globally or throughout Africa as it minimizes the presence of external implementers in the field and encourages greater integration and strengthening of existing systems, thereby facilitating project phase out. F. LESSONS LEARNED AND RECOMMENDATIONS: The CS-20 project has been a positive experience for the Sikasso Region and for advancing innovative community-based strategies for case management of common childhood illnesses. The introduction of zinc for CDD was a great achievement as zinc is now on the essential national drug list. Furthermore, CS-20 showed that CHWs are essential to increasing access to health care at the community level and that the health pyramid cannot function without their dissemination of health messages within the community itself. CS-20 also showed that it takes very little to motivate the CHWs and that a small investment in recognizing their critical role in the Malian public health system would go a long in saving lives of children under five and decreasing morbidity due to common childhood illnesses. CS-20 also learned that the health boards are vital to making the community health care system work. They control how the HCs run and without their leadership, the VDKs and the HCs cannot CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 27 be run efficiently. For example, in the HZs where the health boards are more active in the management of the HCs and the VDKs, the DKs have fewer stock-outs and if they do have a stock-out, the ASACO intervenes to provide solutions. In addition, when the ASACOs provide a small incentive to the DK Managers like a meal at the monthly meetings, DK Managers are motivated to attend the meetings, complete their monthly reports and submit the community data to the HCs in a timely fashion. However, if the DK Managers know that when they go to HCs for monthly meeting and to drop off their community data, if the drugs are not available restocking the VDKs, the Managers tend to miss these monthly meetings and not turn in their monthly data logs. If the ASACO intervenes ahead of time through more effective communications and a system to maintain a link with the DK Managers, there would be a better exchange of data and minimal drug stock-out at the village level. For these reasons, the ASACOs need to recognize their governance roles and take their responsibilities seriously by learning and using standards of good, sustainable, business management. Contrary to what the MOH believes, once DK Managers are trained, they are competent to offer case management of simple malaria by dispensing ACTs at the village level; the Kolondieba case study proved this until the ACT supply ran out. Despite this, CS-20 staff followed the scale￾up strategy to offer quality of care in all of its intervention areas, and trained all of the DK Managers on administering ACTs. With just a brief refresher course on the new ACT formula, the DK Managers will be able to safely dispense ACTs as soon as they can restock their DKs. It was also noted that when the HCs integrated the community data collected by the DK Managers, it allowed the head nurse to do an additional assessment of the morbidity and mortality of children under five in the HZs, as well as the use of modern FP methods. During the interviews with HZ and District Chiefs, many confirmed the increase in oral contraceptive method use by women, and that injectables were the most favored method. Even in the absence of external motivation or incentives, the DK Managers maintained a consistently positive and proactive attitude and interest in their activities (i.e. health talks, household visits, and community mobilization). The CHWs play a critical role in BC and care seeking behaviors (especially in the correct use of mosquito nets). CS-20 strategy to let the local health system implement program activities and manage VDKs with the DK Managers is a good strategy because it allows the local system to integrate the DKs as part of their health care delivery system instead of viewing it as a parallel, separate project. For any future innovation that will take place in the health system, advocacy efforts should target stakeholders at all levels; under CS-20, the focus on advocacy directed toward the health boards (ASACOs) could have been more tangibly developed and implemented. Finally, the project team felt that the communities could have been more clearly advised of CS￾20 phase-out. They believe that there should have been more frequent, formal reminders of the exit plan. The lesson that comes from this, is the degree to which the team perhaps did not completely understand that the ownership of and implementation of CS-20 activities belonged to the local health system rather than SC. This is an essential paradigm shift at the local level that SC is very cognizant of and consistently works to ensure. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 28 ƒ IR1: Access to Health Care: Access to health care has increased with the establishment of the VDKs. The DKs have become first point of contact for health care seeking as well as serving as resources for health information. This local access to drugs has made care more affordable since there are no transportation costs needed to reach CSComs and no consultation fees from DK Managers. ƒ IR2: Quality: The CHWs are very knowledgeable about the treatment protocols for simple cases of malaria and diarrhea at the community level, and consistently and correctly refer critical cases to the CSComs. On-the-job trainings and supervision visits have positively contributed to improved diagnosis and treatment. In addition, DAs check for accuracy of data collected by CHWs during supervision visits and CSCom Managers provide feedback during monthly meetings at the HCs. ƒ IR3: Demand: Mothers know more about childhood illness danger signs and they seek care and purchase treatments at the VDKs. There is increased demand for modern FP methods at the community DKs and CSComs, and CSComs have also seen an increase in prenatal care visits. Consequently, more women are able to receive ITNs and SP. ƒ IR4 and 5: Capacity Building of Partners and Political Environment: CS-20 has engaged partners at all levels (community, district, regional levels) and trained them in the treatment protocols for diarrhea, malaria, and referral of ARI. The OCs and the CHWs have acquired good management skills of the VDKs including the repartition of the benefits from the sale of the drugs. In HZs where CWHs are members of the ASACO (health boards), CSCom management is more efficient. The public health system has also embraced the VDK model and restocking of those DKs has been integrated into most CSCom workplans. Recommendations: Whereby the analysis proved that significant progress was made, and that the CS-20 design was effective and is replicable, some recommendations follow below to address pending challenges faced by the MOH: ƒ The health boards (ASACOs) should be more involved in the management of the HCs and VDKs so that they work with the DK Managers to assure timely restocking of the DKs. One example is that the health boards (ASACO) can make sure that CSComs do not mark-up the prices of drugs sold to VDKs. ƒ Standards of performance should be applied to all health boards so that they follow the same codes for quality of care in all HZs uniformly. This strategy will help to make them more competitive and accountable for their obligations for quality of care to their constituencies. ƒ Improve data management and use by establishing a consistent and timely review of data at all levels (i.e. at the community level with the DK Managers and OCs, at the HC level with the HC Chiefs and ASACOs, and at district and regional levels) for better planning and decision making. There was a general sense that although CS-20 had collected a wealth of data and improved community data contribution to the HIS, the information was underused for planning. ƒ The MOH should recognize CHWs’critical role in the health system as health educators, community mobilizers, providers of first line of treatment for simple cases of childhood illnesses, and community data collectors. More specifically, the MOH should formalize the CHWs by applying a sustainable motivation strategy to maintain their continued involvement. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 29 ƒ In order to address stock-outs in the DKs, the DK Managers should work more closely with the CSCom Managers to combine their orders with that of corresponding HCs in their HZs. This would require CHWs to submit their monthly data on a timely basis and allow time for the CSComs to process the community data for better planning. ƒ The strategy of establishing VDKs should be scaled up at the national level using modified selection criteria (i.e. use distance rather than population density plus distance) in order to increase local access to quality health care. ƒ Local partners should have a contract of performance to hold them accountable to their obligations to assure standard delivery of health services in the HZs. ƒ The DSES (regional department of planning) should integrate the supervision of the DK Managers in the scope of work of their field visits. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 30 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 31 G. REQUIRED ANNEXES Annex 1: Results Highlight Annex 2: List of publications and Presentations Related to the Project Annex 3: Project Management Evaluation Annex 4: Workplan table Annex 5: Rapid CATCH Table Annex 6: Final KPC Report Annex 7: CHW Training Matrix Annex 8: Evaluation Team Members and their Titles Annex 9: Evaluation Assessment Methodology Annex 10: List of Persons interviewed and contacted during Final Evaluation Annex 11: Special Reports (optional) Annex 12: Updated Project Data Form Annex 13: Grantee Plans to Address Final Evaluation Findings Annex 14: Grantee response to Final Evaluation Findings ANNEX 1: RESULTS HIGHLIGHT Scaling up Zinc for Community Case Management of Diarrhea Problem: Malaria, pneumonia, and diarrhea account for approximately 60% of the child mortality in Sikasso Region. Understanding and behaviors around case management of these diseases especially diarrhea and care, are also poor. The 2004 baseline household survey confirmed that only 15% (9% in Sikasso) of households surveyed in the DHS III had soap, ash, or other detergent for washing hands; and only 8% in Sikasso had water, soap, and a basin for hand washing. While 68% of mothers knew about ORT, only 12% with a child with diarrhea in the last two weeks gave ORS to their child, and another 22% gave homemade solution. Instead, to treat diarrhea, mothers give inappropriate treatments, including pills and syrups (26%) and traditional treatments (36%) or no treatment at all (17%). More positively, 54% of all mothers with a child with diarrhea knew to give more liquids than usual. (Detailed Implementation Plan, CS-20, 2005). The project’s input: The CS-20 project was designed to increase access to care by introducing the DKs at the community level, improve health seeking behaviors and recognize illness danger signs. After piloting the introduction of zinc in DKs in Bougouni District, the response was very favorable: community demand for zinc increased as well as for ORS. In light of these results, CS-20 went to scale and introduced zinc in four other districts in the Sikasso Region. With additional funding from American Idol, SC was able to train the DK Managers and stock all of the DKs, and the regional and HC depots with zinc. The magnitude of the intervention: The World Health organization (WHO) and the United Nations Children’s Fund (UNICEF) recommend zinc as a treatment for acute diarrheal disease in addition to ORS. The protocol is a 10-14 days treatment. CS-20 adopted this protocol and went to scale to stock 477 drug kits in 464 villages. In addition, all the health centers (82 health district centers) received a stock of zinc as well. In total, 177,887 children under five years old in all five districts of Sikasso region have benefited from this intervention. Specific results: The CS-20 zinc intervention for the management of diarrhea led the MOH to add zinc to its list of essential medicines and zinc is recognized as a critical treatment in the management of diarrheal disease in Mali. Through CS-20, communities were able to access zinc at a nominal price. The benefits of using zinc for diarrhea control are widely recognized by the population in Sikasso region. Despite the fact that the treatment period is much longer than most treatments, acceptance among caregivers was high. The blister packs’ design facilitated correct administration of the dose prescribed. Introducing zinc also facilitated counseling for better nutrition, management of other childhood illnesses including promotion of hygiene. In summary, most children received the minimum recommended dose of zinc and the compliance level was high. The final household survey also confirmed that because of the availability of zinc in the DKs, the objective of treating with ORS or some other fluids tripled from 17% at baseline to 68% at final, surpassing the 30% target set by the project. At the global level, Save became instrumental in leading other NGOs efforts to adopt zinc as part of their program interventions in management of diarrhea at the community level. For example, Save the Children led a technical assistance team in Madagascar to introduce the use of zinc in the management of diarrhea. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 32 ANNEX 2: LIST OF PUBLICATIONS RELATED TO THE PROJECT Publications: 1. Winch PJ, Doumbia S, Kante M, Male AD, Swedberg E, Gilroy KE, Ellis AA, Cisse G, Sidibe B (2008). Differential Community Response to Introduction of zinc for Childhood Diarrhea and Combination Therapy for Malaria in Southern Mali. Journal of Nutrition. 138 (3): 642-645. 2. Winch PJ, Gilroy KE, Doumbia S, Patterson AE, Daou Z, Diawara A, Swedberg E, Black RE, Fontaine O (2008). Pilot introduction of zinc supplementation for childhood diarrhoea management in Bougouni District, Mali. Journal of Health, Population and Nutrition. 26(2): 151-162. 3. Winch PJ, Gilroy KE, Doumbia S, Patterson AE, Daou Z, Coulibaly S, Swedberg E, Black RE, Fontaine O (2006). Short Communication: Prescription and Administration of a 14-Day Regimen of zinc Treatment for Childhood Diarrhea in Mali. American Journal of Tropical Medicine and Hygiene 74(5): 880-883. 4. Patterson AE, Winch PJ, Gilroy KE, Doumbia S. Local terminology for medicines to treat fever in Bougouni District, Mali: implications for the introduction and evaluation of malaria treatment policies. Trop Med Int Health 2006;11:1613-24. 5. Ellis AA, Winch P, Daou Z, Gilroy KE, Swedberg E. Home management of childhood diarrhoea in southern Mali-Implications for the introduction of zinc treatment. Soc Sci Med 2007;64:701-12. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 33 ANNEX 3: PROJECT MANAGEMENT EVALUATION During the midterm evaluation (MTE), the project management was thoroughly reviewed and it was noted that the project was on track with spending, training, logistics, planning and had weathered extremely well the changes in leadership that occurred three times during the life of CS-20. At the time of the FE, the coordinating unit had been there since the mid-term review in 2007. Information Management: From the MTE, one recommendation was made to more systematically integrate community data into the HC (CSCom) monthly report. Since then, there was some modification made with the DK Managers' data collection tool to address this issue. Specifically, it became evident that not all the information collected by the DK Managers could be integrated into the CSCom report. For example, when it came to the malaria cases (children with fever who received ACT or paracetamol), it was clear that they could not be added as such to the CSCom malaria case confirmed cases. However, the FP data (i.e. # of oral FP products) sold by the DK Managers was already very useful and could be easily integrated into the CSCom data and district health system level. Consequently, the indicators or data points were reviewed and the HC Managers selected the indicators of interest to them. While this change was a great improvement, it did not automatically resolve other data collection related issue. One issue that has remains is the delay encountered in receiving the data from the DK Managers. Some reported not having sufficient funds for transportation, as a constraint. The data was not directly used to test new approaches, however the DAs could easily spot sale trends of medicines, and villages where the VDKs were used more. Furthermore, the DAs compile information monthly using a form that DK Managers fill out about their activities and the sales activities. They use this information during their monthly supervision meetings to troubleshoot with the DK Managers onsite. The forms are brought back to the office and given to the M&E Officer who inputs them into a CS-20 database. (Please see Section C for the specific examples on how the data was used for decision making.) Planning: The yearly workplan was used as a guide to plan field activities, training and keep track of the target. The staff found it very useful in keeping track of the many different project components. Supervision of Project Staff: There has been no change since the MTE. The staff is lean with 10 DAs for the 5 districts, (1) M&E Officer, (1) Capacity building/training officer, (1) Regional Health Advisor, (1) Project Coordinator, and (1) Finance Officer. The staff appreciates their independence to operate in their respective districts with full support from the coordinating unit in the Sikasso office. The staff felt that monthly meetings were adequate and necessary to regroup, share experiences and troubleshoot. There was no suggestion made for improvement. Human Resources and Staff Management: The staff described their relationship as a family. They felt a cohesion and collaboration that kept them glued as a unit during project management changes. It was noted during the FE that despite turnover in field staff, CS-20 implementation was on schedule and that new staff were quickly nurtured, trained and because of their level of experience, quickly adapted to CS-20. Staff also reported supporting each other (i.e. more CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 34 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 35 experienced DAs or those who had been with CS-20 since the beginning supporting new DAs during supervisory visits with the DK Managers and community health education events). At the time of the FE exercise, only one staff had left the team: the Training/Capacity Building Advisor who was replaced internally by one of the DAs two months prior to the FE. Because of the no-cost extension, some staff is expecting to be retained. There was not a concrete plan to map out transitions to other paying jobs. Staff are aware that other positions were opening with program expansion in Sikasso and some had already applied. Financial Management: There was no time to discuss the financial management issues with the Financial Officer during the FE. The M&E Officer who was sitting in for the Project Coordinator confirmed that all the financial transactions were tracked in the Bamako office. With respect to specific activities, there was still money available to stock the DKs and the CSCom drug depots with zinc beyond Sept 30th. Additional funding from the Mission for the no-cost extension will allow restocking the DKs with ACTs since the policy has changed to make it available at the community level. Some staff mentioned that cutting back on supervision budget this past year to allow partners to adjust to the transition process was challenging for partners who did not have the funds to conduct supervision of the DK Managers. While this budget cut for the transition period has been known all along by the partners, they still express the need to have another project to support supervision activities. Financial sustainability is not a new issue. All along, CS-20 has been discussing with partners how to plan and integrate supervision costs within their system so that when the project support ends,partners can continue with supervision activities. It was noted during the FE interview of the regional MOH office in Sikasso, that integrated supervision is under consideration. During the FE exercise, it was not confirmed that adequate resources had already been secured for this. Regarding funds to restock the DKs, suggestions were made to work with the cotton association, women's groups and/or community leaders should there be a lack of funds to buy the products. CS-20 paid for the initial medicine stock and the sustainability plan was clear from the beginning to use the 20% markup as a way to generate more capital for DKs. Logistics: The logistics system in place has served the project well. Staff expressed complete satisfaction. ANNEX 4: WORKPLAN TABLE Objectives/Activities Obj. Met Activity Status Advocacy/Mobilization DIP workshop Yes Completed in 2005 Agreement with regional health office Yes An agreement was signed with Sikasso Regional MOH office Contacts with national partners Yes Monthly meetings of Cs-20 staff Yes Monthly (last Thursday) meetings Participate in regional PRODESS Yes CS-20 health advisor attended PRODESS (CROCEP) meeting in March 2009 Regional Launching of CS-20 Yes All key stakeholders at Regional MOH were well implicated in all CS-20 activities District launch of CS-20 Yes All key stakeholders the (5) Districts MOH office were well implicated in all CS-20 activities Installation of CS-20 advisors Yes 10 District Advisers were distributed over the (5) districts. Launching in 67 HZs Yes CS-20 operated in 84 HZs Selection/creation of DK and OC (532) 95% of kits are operational 477 DKs were installed; 95% are operational, while less than 5% are weak. The number of villages meeting the eligibility criteria for VDKs was less than originally projected. 477 OCs created 70% of the membership of the OCs are women. Organizations of village general assemblies for the replacement of OCs offices by stressing greater participation of women Continuation of the renewal of the offices of the OCs 55% of the membership of the OCs are women. OC members are chosen by Village Chiefs who overwhelmingly choose men for these positions. As a result, it has been impossible to meet that objective. Develop contract with mayors (82) Yes 59 contracts were signed with 59 mayors Organize meeting with Cotton Producer Associations NA This activity was annulled since program strategy was changed because of new MOH policy for free ITN distribution Develop MOU with ITN marketing Association NA Same as above. Monitoring and evaluation Doer/Non Doer Formative research Yes See Formative research report (DIP) KPC Studies Yes See KPC report (DIP) Organizational capacity assessment Yes See Org. capacity report (DIP) Finalize various HIS forms Yes CS-20 put in place 10 data collection forms for CHWs, DAs, and internal reporting???? Mid term evaluation Yes See Mid term evaluation report; Sept, 2007 Final Evaluation Yes See Final Evaluation report, Sept 2009 Documentation Annual reports Yes CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 36 Objectives/Activities Obj. Met Activity Status Operation manual for DKs and DK Managers The draft document will be finalized in November 2009 during the no cost extension period. Dissemination of FE results TBA To be completed in November 2009 National Roundtable of lessons learned TBA To be completed November 2009 OR Study Documentation Yes See zinc study report HCs, District Medical Directors, MOH and SC collaborate in writing two articles on DK and ITN promotion approaches. This activity was dropped due to lack of time Supervision and Support Quality Monthly supervision of HC nurses to the villages: 70% of Oversight Committees will be supervised at least once/month by HCs. HC staff visit villages at different times; not specific only for OCs. This was not measured consistently. Quarterly supervision of HCs by DHOs Yes DHOs staff visit HCs at different times. This was not measured consistently Annual Supervision of National MOH (Malaria and RH Program) Only in Kolondièba by Dr Barasson Diarra, PNLP Restocking of DKs at the HC drug Depots (# of Stock-outs of DKs) 80% of DKs had no stock-out of (now ACT) in the last three months. 80% of DKs had no stock-out of FP methods in last three months. 80% of DKs had no stock-out of ORS in last three months. Restocking DKs have had stock-outs because many HC drug depots also have stock-outs and consistently; Some DK Managers don’t have enough money to restock and they are not consistent in placing their orders. There have been stock-outs of ACT in the CSComs but it has been difficult to determine the cause Constant stock-outs of spermicides and condoms in district depots. Stock-outs of ORS in Kolondieba District due to low financial reserves of district depot. Restocking of ITNs and kits by the marketing companies (# of Stock-outs of Cotton Producer Association) NA This activity was annulled with new national policy of free ITN distribution 60% of Management Committees participate in monthly meetings of DK Managers at the HCs. DAs sensitize Management Committees on importance to participate in monthly CHW meetings at HCs Not measured; overwhelmingly management committees have failed to attend meetings due to lack of funds for transportation and meals. CS-20 did not financially support these monthly meetings. In selected HCs, some ASACO made the effort to support some meetings; but it was not consistent. 50% of DHOs complete two integrated supervisory visits of HCs in the last year. No There was not enough money in the budget to support this activity. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 37 Objectives/Activities Obj. Met Activity Status Participation of DAs in integrated supervision visits of the districts to the CSCom Behavior Change Communication Develop contract with local radios Yes Using an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. Identify and collect BCC materials Yes The materials were identified and collected from MOH and other NGO partners. Monthly meetings of DK Managers at HCs Partial Most DK Managers adhered to monthly schedule. Promotion and sales of ITNs and reimpregnation kits by cotton producer association. NA MOH policy changed to free ITN distribution Monthly outreach visits organized by HC head nurse Yes 70% of VDKs were visited during outreach visits. Conduct Village BC sessions (weekly talks) – 10-15 households visits /month by DK Managers Yes - Two (2) education sessions per month on BC in the village; home visits are conducted to: check on sick child, and to find out if the referred child made it to the CSComs, and also to observe if the mothers are practicing new behaviors. Use rural radio in the diffusion of key messages Yes Using an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. 80% of mothers know three or more danger signs of childhood illness. Organization of health education sessions by CHWs for parents of children < 5 years old on recognition of danger signs in children Yes The DAs tested the level of knowledge of the mothers of children less than five during their supervision visits and the education sessions. The DK managers organized at least 2 education sessions per month on the recognition of danger signs of childhood illnesses 60% of mothers know where a DK is located. Organization of the health education sessions by CHWs for parents of children under 5 years old on the location of the DK, its purpose, contents and use Yes All health education sessions are also integrated with the sesntitization activities of the DK location, its use, and its purpose. Prevention and case management Malaria 30% of children <5 slept under an Insecticide Treated Net (ITN) the previous night 50% of pregnant women slept under an ITN the previous night Yes Free ITNS are distributed to: a) children less than 1 year old having completed their vaccination; b) pregnant women with at least (1) ANC visits Each pregnant woman receives an ITN during the first ANC visit CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 38 Objectives/Activities Obj. Met Activity Status Organization of BCC sessions by the head nurses and the CHWs with the target populations on the importance of ITN use Broadcasts of key ITN messages on the local radio stations Sensitization sessions are held during immunisation outreach activities on the improtance of the use of ITNs by pregnant women and children under 5. 60% of children <5 with fever in the last two weeks treated through DKs or HCs Organization of health education sessions by CHWs for the mothers of children of under 5 years old on the recognition of signs of fever, and on the location of the DK, its purpose, contents and use Broadcasts of messages on signs of fever and the promotion of the DK on the local radio stations The removal of chloroquine from the DKs in June 2006 contributed to a large decrease in the use of the DKs. Currently, as ACTs are progressively introduced, DK visits are increasing again. 70% of pregnant women will receive intermittent preventive treatment with Fansidar Organization of health education sessions by CHWs for pregnant women on importance of SP as preventive treatment for malaria and proper interval between the two doses Local radio broadcasts of messages on importance of SP for pregnant women as preventive treatment for malaria and proper interval between the two doses Yes Currently in Sikasso, sulfadoxine-pyrimethamine (SP) is not free. This is a real barrier for pregnant women and their adherence to the preventive treatment and their ANC visits. Education messages focused on 2 doses of SP for pregnant women. 80% of women state that ITNs are very important for children <5. Organization of BCC sessions by head nurses and CHWs with the target populations on importance of ITN use Local Radio broadcasts of key ITN messages Yes Sensitization sessions on the importance of ITNs for children under 5 during monthly outreach visits in the villages. Using an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. The radio stations kept a register on all the broadcasts that were reissued after the release of the initial broadcast. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 39 Objectives/Activities Obj. Met Activity Status Prevention and case management of Diarrhea 30% of children < two years with watery diarrhea in the last two weeks were treated with ORS, an appropriate home-based solution, and/or increased fluids. Organization of health education sessions by CHWs for parents of children < 5 years old on the recognition of signs of diarrhea, feeding during diarrheal episodes, and importance of ORS and zinc for treatment Organization of the health education sessions by CHWs for parents of children under 5 years old on the location of the DK, its purpose, contents and use Local radio stations Broadcasts of messages on signs of diarrhea and the promotion of zinc Yes Although zinc has been added to list of essential medicines and is in the Policies Standards and Procedures, it is still not available in the national pharmacy. SC with private funding is stocking the Regional Sikasso drug depot with zinc so that it can stock the HCs depots. Restocking DKs with zinc is not a problem in Sikasso. Using an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. The radio stations kept a register on all the broadcasts that were reissued after the release of the initial broadcast. Family Planning 8% of all WRA using a FP method Organization of BCC sessions by the head nurses and the CHWs with men and WRA on the importance of child spacing and the availability of FP methods in the DKs and HCs Local radio broadcasts of messages on the advantages of child spacing and modern FP method availability at the DKs and HCs 50% of FP clients report they are satisfied with the services they received. Yes Oral FP products and condoms are available a the DKs; Injectables are available at the HCs and are a preferred FP method among women. Use of condoms is low in men. Using an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. The radio stations kept a register on all the broadcasts that were reissued after the release of the initial broadcast. This was not measured. Acute Respiratory Infections (ARI) CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 40 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 41 Objectives/Activities Obj. Met Activity Status 60% of children <5 with difficult or rapid breathing who visited DKs are appropriately referred to a HC. Organization of health education sessions by CHWs for parents of children under 5 years old on the recognition of ARI signs. Head nurse and DAs sensitize the CHWs on referral of ARI cases to the CSComs Local radio broadcasts of messages on ARI danger signs and reference to the CSComs . Yes CS-20 used an annual contract with (9) local radio stations, pre-registered messages on all the 4 interventions with a schedule developed by CS-20 were disseminated. Training: See Annex 7 for all CS-20 training completed ANNEX 5: RAPID CATCH TABLE (TO BE COMPLETED IN JANUARY 2010) This table should display Rapid CATCH baseline, MTE (if collected) and final estimates. Indicate with an asterisk (*) those final estimates that are significantly different from the corresponding baseline estimates. Statistical significance can be ascertained by analyzing the confidence intervals. If the confidence intervals for the baseline and final estimates do not overlap, then the estimates are significantly different. You can see the confidence intervals on the project data form. Contact MCHIP SO3 team if you have any questions. An example of the Rapid CATCH table is provided below: Indicator Baseline Estimate Final Estimate Target % children (0-23 mos) who are underweight % children (0-23 mos) who were born at least 24 months after the previous surviving child % children (0-23 mos) whose births were attended by skilled health personnel % of mothers of children (0- 23 mos) who received at least two tetanus toxoid injections before the birth of their youngest child % of infants (0-5 mos) who were exclusively breastfed in the last 24 hours % of infants (0-5 mos) with only response = A (breatsmilk) for Rapid CATCH Question 13 % of infants (6-9 mos) receiving breastmilk and complementary food % of children (12-23 mos) who are fully vaccinated against the five vaccine preventable diseases before the first birthday % of children (12-23 mos) who received a measles vaccine % of children (0-23 mos) who slept under an insecticide treated bednet the previous night (in malaria risk areas only) % of mothers who know at least CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 42 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 43 two danger signs of childhood illness that indicate the need for treatment % of sick children (0-23 mos) who received increased fluids and continued feeding during an illness in the past two weeks % of mothers of children (0-23 mos) who cite at least two known ways of reducing the risk of HIV infection % of mothers of children (0-23 mos) who wash their hands with soap/ash before food preparation, before feeding children, after defecation, and after attending to a child who has defecated ANNEX 6: FINAL KPC REPORT – EXECUTIVE SUMMARY The final KPC survey for CS-20 was carried out during May 24 to July 28, 2009 in the five districts of Sikasso Region (Bougouni, Kolondieba, Yanfolila, Yorosso and Sélingué). Similar to protocol used for the baseline survey in 2004, this quantitative survey used a representative sample of 2,874 children (6 to 59 months) measuring their weight and height, administered a health questionnaire to 1,481mothers and 1,283 men on hygiene, prevention of childhood illnesses (malaria, acute respiratory infection, diarrhea), and use of modern family planning methods including their care seeking behaviors. The same indicators measured at baseline were measured in the final survey to compare how they progressed after five years of program interventions to address behavior change related to malaria, diarrhea management, acute respiratory disease, family planning, nutrition and proper hygiene practices. The standard rapid CATCH indicators were used. Quantitative and qualitative data were collected through three types of questionnaires administered in 238 villages, and 64 health districts. Socio-demographic and economic characteristics of the surveyed households are as follows: a) Low education level: less than 27% in Bougouni, Kolondieba and Yanfolila, 56.9% in Yorosso and 40.8% in Selingué. In all the districts, nearly 95% women live with the biological father of their child. Most houses were made of bricks with tin roofs, and the floor was unfinished with sand in more than 80% of the households and with cement in 26.5% in Yoross,o and 17% in Yanfolila. Outdoor traditional latrines were used in more than 80% households. Generally, basic hygiene of the dwellings was rather good in 36.89% of the households surveyed. Wood remained the principle source of energy for cooking fuel. Disease prevention Malaria: more than 85% of the households surveyed in all the five districts have mosquito nets. Kolondiéba and Sélingué had higher net ownership: 92.7% and 92% respectively. The use of insecticide treated mosquito nets (children under five years old slept under an insecticide treated net the night before the survey) improved from 8.4% to 70.6%. Among pregnant women, use of mosquito net also improved between 2004 and 2009 except in Yorosso and Kolondieba with rates at 46.4% to 59% and from 21.2% to 37.3% respectively. Elsewhere, ITN use doubled in Yanfolila and Sélingué, and quadrupled in Bougouni. Malaria in pregnant women: The proportion of pregnant women on intermittent presumptive treatment (IPT) hardly changed. In Yanfolila, IPT rate went from 69.3% in 2004 to 88.2% in 2009. In Bougouni on the other hand, the IPT rate slightly dropped from 72.2% in 2004 to 68.4% in 2009. Iron supplementation for the prevention of anemia in pregnant women improved however in all five districts; in Yanfolila and Bougouni, iron supplementation increased almost twice as much from 58.7% to 92.6% and from 62.2% to 91.8% respectively from 2004 to 2009. Knowledge about prevention of childhood illnesses such as malaria also improved. For example, the percentage of mothers who think that ITNs are very important for children less than 5 years against malaria improved in all five districts. In Bougouni, this indicator doubled from 34.2% in 2004 to 74.1% in 2009. Sélingué recorded the lowest increase in knowledge (38% to 57.7%). Similarly, the proportion of women who think that use of ITNs for pregnant women is good against malaria also improved most in Kolondieba and Yanfolila (22.9% to 67.4% and from 33.3% to 87.2%) respectively. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 44 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 45 Care seeking for children under five with fever at the village level (visitation of the drug kit manager) improved considerably between 2004 and 2009. Apart from Sélingué where 50% of the cases visited the drug kit manager, all the other districts recorded a visitation rate higher than 63%. The best rate was recorded in Yanfolila with 82.9%. Management of diarrhea: Use of ORS for treating diarrhea in children less than 2 years old, also clearly improved in all five districts, but in particular in Sélingué where the rate increased from 10.2% in 2004 to 70.6% in 2009, and in Kolondieba (18.4% to 95.8%). The lowest rate was recorded in Yorosso with 25.7% in 2004 to 58% in 2009. Family planning: Use of modern contraceptive methods for birth spacing in women of reproductive age increased significantly. For example, in Bougouni the rate quadrupled (from 3.2% in 2004 to 17.6% in 2009) as well as in Yanfolila (3.5% to 18.8%); it tripled in Yorosso (7.6% to 30.3%). ANNEX 7: CHW TRAINING MATRIX Yorosso Bougouni Kolondieba Yanfolila Selengue # Theme # # # # 112 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) 358 201 206 56 110 Community Case management of malaria with ACTs 360 207 206 55 95 Case Management and Essential Nutrition Action (ENA) Protocol 232 112 204 38 CHWs/Assistant s VHT V 107 Case management of diarrhea with Zinc supplementation 361 217 206 55 Surveillance Committee VHT V 112 Management System for Drug Kits 354 176 204 52 10 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) 27 16 13 5 3 Case Management and Essential Nutrition Action (ENA) Protocol 21 6 0 3 Chiefs (CSCom) Gov P 11 Case management of diarrhea with Zinc supplementation 32 16 14 5 0 Case Management and Essential Nutrition Action (ENA) Protocol Matrones 21 6 0 3 (CSCom) Gov P 10 Case management of diarrhea with Zinc supplementation 31 17 15 5 DK managers, DV CSCom) Gov P 11 Case management of diarrhea with Zinc supplementation 17 16 15 5 Nurse Aids (CSCom) Gov P 11 Case management of diarrhea with Zinc supplementation 27 15 15 5 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 46 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 47 Yorosso Bougouni Kolondieba Yanfolila Selengue # Theme # # # # 1 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) 1 0 1 1 1 Case management of diarrhea with Zinc supplementation (Training of trainers) 0 1 1 0 CSRef - Doctors Gov P 1 Essential Nutrition Action (ENA) Protocol 1 1 1 1 1 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) CSRef - Nurses Gov P 1 Case management of diarrhea with Zinc supplementation (Training of trainers) 1 1 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) CSRef – 1 1 1 1 Midwives Gov P Case management of diarrhea with Zinc supplementation (Training of trainers) 1 CSRef – Manager DRC Gov P 1 Case management of diarrhea with Zinc supplementation 1 1 1 1 Social Workers Gov P 1 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) 1 1 1 1 New HC Chiefs (CSCom) Gov P 4 Community based management for (malaria, diarrhea, ARI, FP, prenatal care, and health education) 6 5 5 0 ANNEX 8: EVALUATION TEAM MEMBERS AND THEIR TITLES Name Title Abdoussalam Tièmogo Idrissa Doumbia Yorosso District Advisors Moussa Kouyaté Selingue District Advisors Gninè Samaké Sékou O. Sidibé Bougouni District Advisors Moussa Coulibaly Mamadi Keita Kolondieba District Advisors Nantenin Traore Mamadou Sanogo Yanfolila District Advisors Sikasso Region- MOH Dr Drissa B. Ouattara Souleymane Traoré Direction Régionale de Santé de Sikasso Direction Régionale du Développement Social et de l’Economie Solidaire/Sikasso Mme Touré Fatoumata Dialy Touré Programme National de Lutte contre le Paludisme (PNLP) Dr. Boubacar Sidibe SC Health Coordinator Modibo Bamadio SC M&E Officer, Bamako Bamody Diakité CS-20 Capacity Building Officer Zana Daou CS-20 M&E Officer Eric Swedberg Technical Support, Save the Children Headquarters Circe Trevant External Consultant CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 48 ANNEX 9: EVALUATION ASSESSMENT METHODOLOGY 1. Phase 1: A quantitative study (typical household survey: Knowledge, Practice and Coverage survey): May to July 2009, by an external consulting firm. It was the same firm that carried out the baseline survey. The report of that survey was finalized in August 2009. The survey was a repeat of the baseline household survey with the objective to assess whether or not the project met the target for each of the indicators set at baseline. 2. Phase 2: In the second phase, in September, a 2nd team met to carry out a participatory qualitative exercise. The team members included partners (MOH), project staff, an external consultant and HQ staff. (See Annex for list of participants with accompanying affiliations/titles). a. First, the project health advisor presented the findings of the household survey. b. Next step, it was clear that some questions remained as to the contextual factors behind those results. c. In order to facilitate the interviews, the target groups were identified and then in small groups, guides were developed for the interviews in the field. In plenary session, the guides were finalized allowing all team members to give their input. d. Because of several constraints (time, transportation, rainy season), it was decided that instead of visiting all 5 districts in depth, three teams would be formed to visit the five districts while doing in-depth interviews in three districts and doing less intense work in two districts that had already benefited from previous interventions. The groups then divided in three. e. After the selection of the target groups to interview, the team used site selection criteria by selecting two HZs per district, one performing well and the other not. Within each district, similar performance criteria were used to select a performing and then a less performing village (two villages per HZ). Since this was done in a plenary session, each DA has a chance to voice their opinion about performance of their areas and selected the sites. f. The evaluation team was divided so that the group that was the furthest from the regional office, would only do one district (plus the regional office). The other two groups each took two districts. g. The target groups included: 1) Regional Officer; 2) District Health Team; 3) Mayors; 4) DK Managers; 5) ASACO members; 6) local leaders/mayors 7) HC Head Nurse; 8) District Health Chief; h. Each team spent four days in the field for the interviews; then the information was synthesized by target group; then in small group, the information was synthesized by theme: 1) Access; 2) Quality; 3) Demand for health care; and 4) Capacity building of partners. In plenary sessions, the findings were validated. i. Subsequently, in the last plenary session, lessons learned and recommendations were formulated. j. In Bamako, the staff prepared a presentation and a debriefing session was held with USAID and it was attended by the Malaria Advisor (USAID health team), consultant, Project Advisors, HQ Health Advisor and SC Country Director. k. CS-20 staff conducted an assessment of the capacity of project partners to see how they have evolved over the past five years. This was conducted in December 2010. . CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 49 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 50 l. CS-20 staff plans to conduct a debriefing session with regional, district and community stakeholders on the findings of both evaluations (quantitative and qualitative). ANNEX 10: LIST OF PERSONS INTERVIEWED AND CONTACTED DURING FINAL EVALUATION District: Yorosso; Health Zone Koumbia; Village: Barena N° Name Role 01 Dr Benoit Traore Doctor 02 M Mamadou Diarra Chief SDS- ES 03 Mme Alice Dako Manager DRC 04 Siaka Goita Président ASACO 05 Boubacar G Sissoko CPM 06 Lamine Goita Chief de Village 07 Ibrahima Goita Advisor village 08 Timoté Goita Advisor village 09 Karim Traore Advisor village 10 Aly Goita Advisor village 11 Danaya Goita Manager CP 12 David Nantaga Goita Member CS 13 Djibril Traore Member CS Village: Dorosso 14 Moussa Goita Chief Village 15 Bakariba Goita Advisor village 16 Lassina Sanzare Advisor village 20 Lassina Goita Advisor village 21 Doullaye Dembele Advisor village 22 Diahara Goita President –of women group 23 Adama Goita Village Youth 24 Siaka Goita Village Youth 25 Zoumana Goita President CS 26 Salif Goita Treasurer CP District: Yorosso; Health Zone: Karangana; Village: Bénigorola 27 Srima Soumano CPM 28 Adama Kone President ASCO / FELASCOM 28 Boureima Goita Village Chief 29 Souleymane Goita 30 Djémo Goita Advisor 31 Sekou Goita President CP 32 Mariam Dao Trasurer CP 33 Hamidou Cisse Manager DK 34 Sadou Kone Village Chief 35 Adama Kone President DK 36 Korotoum Kone Presidente women group 37 Oumar Traore Youth Secretary CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 51 38 Hamidou Dembele Secretary DK 39 Mariam Dembele Manager DK 40 Drissa Goita Manager DK District: Kolondiéba Health zone : Kolondiéba central Village :Kolon foulala No Name Role 1 Sounkao Konaté Village Chief 2 Lanceini Doumbia Advisor 3 Amadou Konaté Advisor 4 Boulaye Konaté Advisor 5 Seydou Konaté President C.S 6 Zakaria Diakité Secretary 7 Sékourou Diakité Treasurer 8 Ramata Diallo Assistant manager 9 Drissa Konaté Manager principal Health zone :Kolondiéba central No Name Role 1 Alhousseyni Diallo 2 Dramane Fané President ASACO 3 Moussa Doumbia S.G Mairie 4 Moussa Dagnogo Secretary Info ASACO 5 Aly Koné Relais District: Kolondiéba Health Zone : Bougoula No Name Role 1 Bakary Koné Representant Mairie/Santé 2 Karim Koné President ASACO 3 Drissa Koné Community health worker 4 Daouda Traoré Manager DV 5 Minata Koné Midwife 6 Abdou Koné CPM District: Kolondiéba Health Zone: Bougoula No Name Role 1 Soumana Traoré Village Chief 2 Adama Traoré Advisor 3 Sidiki Koné Advisor 4 Konimba Traoré Manager Principal 5 Noumounké Traoré 6 Ousmane Traoré President 7 Moussa Koné Secretary 8 Djatou Traoré Treasurer CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 52 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 53 District: Kolondiéba Health Zone: Bougoula No Name Role 1 Bakary Koné Village Chief 2 Tiédjougou Koné Advisor 3 Salif Koné Advisor 4 Sidiki Koné President Surveillance committee 5 Kadiatou Coulibaly Treasurer C.S 6 Daouda Koné Main Manager 7 Oumar Koné Assistant Manager District: Kolondiéba Health Zone: Kolondiéba Central No Name Role 1 Abou Traoré Main manager 2 Adjara Konaté Assistant Manager 3 Noumouké Koné Treasurer 4 Moussa Traoré Secretary 5 N’Saly Koné Advisor 6 Ladjie Koné Village Chief District: Kolondiéba Health Zone: Kolondiéba Central No Name Role 1 Abou Traoré Main manager 2 Adjara Konaté Assistant Manager 3 Noumouké Koné Treasurer 4 Moussa Traoré Secretary 5 N’Saly Koné Advisor 6 Ladjie Koné Village Chief 7 Moussa Doumbia Advisor District: Referral Health center - Bougouni No Name Role 1 Brema Bamba Doctor Chief 2 Karounga Keita Manager - SIS 3 Doulaye Coulibaly Main focal person CS-20 4 Djelika Coulibaly Member of CS-20 team ANNEX 11: SPECIAL REPORTS (OPTIONAL) If appropriate, include special reports, papers, presentations or analyses produced by the project. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 54 ANNEX 12: PROJECT DATA FORM Will be completed in January, 2010. Updated and printed from www.childsurvival.com/projects/dipform/login.cfm CSHGP Project Data Form The CSHGP project data form is used to capture critical project information to make CSHGP reporting easier at both the project and portfolio levels. There are extended help files to guide you in filling out each subform to keep your project information up to date. We strongly encourage you to carefully read through the help files for each page to assist in properly filling out the entire form. The most extensive of these help files is the one that provides detailed explanations on each project intervention component. This help file can be printed out by clicking here. CSHGP grantees are required to update Rapid CATCH project data in the online form twice during the life of the project, at the time of the DIP submission, and again at the time of the final evaluation. PVOs/NGOs are encouraged to also enter Rapid CATCH data at the time of the mid term evaluation, however, this is no longer a requirement. It is a requirement to print out the completed online forms and include them with your corresponding DIP report. After filling out this online form and submitting your information, you will be presented with an option to 'Print this Form'. This will generate a printable version of your project form which should be included with the corresponding report you are submitting. The CSHGP team thanks you in advance for keeping this information up to date! Select PVO and enter Password to access active Project Information: PVO: Password: Forgot Password? CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 55 CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 56 ANNEX 13: GRANTEE PLANS TO ADDRESS FINAL EVALUATION FINDINGS Findings and recommendations from the final evaluation will be incorporated into the design of the follow-on project in 2010. CS-20 Mali, Sikasso, Final Evaluation, Save the Children, December 2009 57 58 ANNEX 14: GRANTEE RESPONSE TO FINAL EVALUATION FINDINGS (OPTIONAL) – SEE PREVIOUS COMMENTS