INTERNATIONAL EYE FOUNDATION NSANJE DISTRICT HEALTH MANAGEMENT TEAM IMPROVED CHILD SURVIVAL IN NSANJE DISTRICT, MALAWI THROUGH COMMUNITY BASED INTERVENTIONS AND STRENGTHENING OF THE HEALTH DELIVERY INFRASTRUCTURE FINAL EVALUATION REPORT, DECEMBER 31, 2006 Implementing a XVIII Child Survival Project October 2002 - September 2006 Cooperative Agreement HFP-A-00-02-00027-00 Presented to: Child Survival and Health Grant Program United States Agency for International Development Document prepared by: Garth Osborn, Team Leader/External Evaluator Contacts: John M. Barrows, Director of Programs International Eye Foundation 10801 Connecticut Avenue, Kensington, MD 20895 Tel (240) 290-0263; Fax (240) 290-0269; jbarrows@iefusa.org Geoffrey Ezepue, MD, Country Director International Eye Foundation, Malawi P.O. Box 2273, Blantyre, Malawi ief@Malawi.net DECEMBER 31, 2006 Final Evaluation Report i ACKNOWLEDGEMENTS Special thanks go to the following individuals and organizations for their contributions to this Final Evaluation: • Edna Tembo, Geoffrey Ezepue, and John Barrows for their work in preparing for this evaluation exercise, their participation throughout, and the sharing of their invaluable insights on Nsanje and this child survival project. • The members of the Final Evaluation Team who contributed their time, input, and long days to this effort. The members of the FE Team were: • Mr. Mhango, Deputy District Health Officer, MOH • Beatrice Thaulo, District Nursing Officer, MOH • Mathew Kalaya, District Environmental Health Officer, MOH • Kenneth Ndau, District MCH Coordinator, MOH • Mwachumu Reukia, IMCI Coordinator, MOH • George Matope, Malaria Coordinator, MOH • Steven Nkuma, District AIDS Coordinator, MOH • Richard Martin, Senior Health Surveillance Assistant, MOH • Maclean Sosono, Executive Director, Friends of AIDS Trust (FAST) • Pius Nakoma, Nsanje District Coordinator, Family Health International (FHI) • Edna Tembo, Project Manager and Acting MCH and HIV/AIDS Coordinator, IEF • Frank Chola, Project Administrator, IEF • George Meckiseni, Monitoring and Evaluation Advisor, IEF • Hopson N. Tiringamawa, IMCI Coordinator, IEF • Evalister Nyirenda, Administrative Assistant, IEF • Geoffrey Ezepue, Country Director, IEF • John Barrows, Program Director, IEF/Headquarters • The leadership and staff of the Ministry of Health who provided invaluable perspectives on this project and the communities in Nsanje. • FAST and FHI for supporting Maclean Sosono and Pius Nakoma in their participation on the FE Team. • The many community members, volunteers, health workers and staff who responded to our many questions with openness and a view towards the future. Garth Osborn, FE Team Leader Final Evaluation Report ii TABLE OF CONTENTS A. SUMMARY …………………………………………………………………………... 1 B. ASSESSMENT OF RESULTS AND IMPACT OF THE PROGRAM ……………… 3 1. Results: Summary Chart …………………………………………………………. 3 2. Results: Technical Approach …………………………………………………….. 5 a. Overview ………………………………………………………………………. 5 b. Progress Report by Intervention Area …………………………………………. 7 c. New Approaches ……………………………………………………………… 19 3. Results: Cross-cutting Approaches ………………………………………………. 19 a. Community Mobilization ……………………………………………………... 19 b. IEC and Behavior Change Interventions ……………………………………... 22 c. Capacity Building Approach ………………………………………………….. 23 d. Sustainability Strategy ………………………………………………………… 28 C. PROGRAM MANAGEMENT ……………………………………………………….. 29 1. Planning …………………………………………………………………………… 29 2. Staff Training ……………………………………………………………………… 30 3. Supervision of Program Staff ……………………………………………………… 30 4. Human Resources and Staff Management ………………………………………… 31 5. Financial Management …………………………………………………………….. 32 6. Logistics …………………………………………………………………………… 33 7. Information Management ………………………………………………………….. 33 8. Technical and Administrative Support ……………………………………………. 34 9. Mission Collaboration ……………………………………………………………... 34 10. Management Lessons Learned …………………………………………………… 34 D. OTHER ISSUES ……………………………………………………………………… 35 E. CONCLUSIONS AND RECOMMENDATIONS ……………………………………. 36 ATTACHMENTS ………………………………………………………………………... 42 A. Evaluation Team Members and Titles …………………………………………… 42 B. Final Evaluation Methodology …………………………………………………... 43 C. Persons Interviewed and Contacted ……………………………………………... 48 D. Accomplishments, Challenges, Lessons Learned & Recommendations …........... 49 E. Focus Group Discussion Report …………………………………………………. 54 F. Rapid CATCH Indicators ………………………………………………………... 62 Final Evaluation Report iii ACRONYMS ARI Acute Respiratory Illness ARV Anti Retro Viral Drugs BCC Behavior Change Communication BCI Behavior Change Intervention BF Breast Feeding BFSG Breast Feeding Support Group CHAM Christian Hospitals Association of Malawi CHV Community Health Volunteers (Including all project-trained volunteers and members of VHCs, ITN committees and support groups) CI Confidence Interval CSMC Child Survival Management Committee DD Diarrheal Disease DEC District Executive Council DHMT District Health Management Team DHO District Health Officer DHS Demographic and Health Survey DIP Detailed Implementation Plan DOSA Discussion Oriented Self Assessment DRF Drug Revolving Fund DTC District Technical Committee EBF Exclusive Breast Feeding EDHMT Extended District Health Management Team EPI Expanded Program for Immunization FAST Friends of AIDS Trust (A Malawian CBO) FE Final Evaluation FGD Focus Group Discussion FHI Family Health International GMV Growth Monitoring Volunteer GOM Government of Malawi HF Health Facility HFA Health Facility Assessment HH/C IMCI Household and Community Integrated Management of Childhood Illness HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome HMIS Health Management Information System HSA Health Surveillance Assistant HQ Headquarters IEC Information, Education, and Communication IEF International Eye Foundation IMCI Integrated Management of Childhood Illness IPT Intermittent Presumptive Treatment IR Intermediate Result ITN Insecticide Treated Net KPC Knowledge, Practice, Coverage Survey LQAS Lot Quality Assurance Sampling MCH Maternal Child Health MK Malawian Kwacha (monetary unit of Malawi valued at 139 K to US$1 during the FE.) MOHP Ministry of Health and Population MTE Midterm Evaluation N/A Not Applicable/Available Final Evaluation Report iv NDH Nsanje District Hospital NGO Non Governmental Organization OCAT Organizational Capacity Assessment Tool ORS Oral Rehydration Solution ORT Oral Rehydration Therapy OTJ On The Job OVC Orphans and Vulnerable Children PDI Positive Deviance Inquiry PLWA People Living With AIDS PMTCT Prevention of Mother-to-Child Transmission (of HIV/AIDS) PSI Population Services International PVO Private Voluntary Organization SD Standard Deviation SO Strategic Objective SP Sulphadoxine-pyrimethamine (Fansidar) TBA Traditional Birth Attendant TH Traditional Healer U-5 Under Five UNICEF United Nations Children’s Fund USAID United States Agency for International Development VAC Vitamin A Capsule VCT Voluntary Counseling and Testing VHC Village Health Committee W&S Water and Sanitation Final Evaluation Report IEF/DHMT Nsanje Malawi 1 C.A.: No: HFP-A-00-02-00027-00 A. SUMMARY The International Eye Foundation (IEF) has been implementing a USAID-funded child survival project in Nsanje District, Malawi from October 2002 to September 2006 in partnership with the Nsanje District Health Management Team (DHMT), the health facilities and the district’s 513 villages. Nsanje District is located in the far southern tip of Malawi. Its estimated 2005 population was 237,000, with a project beneficiary population of 97,500, including 42,500 infants and children under the age of five (18%) and 55,000 women of reproductive age (15-49 years) (23%). Infants and children 0-23 months and 24-59 months represent 44% and 56% of the beneficiary infant and child population, respectively. The project goal was to reduce infant and child mortality in the district. Its strategic objective was, “families and caregivers with young children increase the practice of healthy behaviors and seek medical care from quality sources.” To achieve this, the project focused on the following four results: 1. Strengthened effectiveness of district health management systems for quality childcare. 2. Improved health provider skills in prevention and management of childhood illness. 3. Increased community participation, ownership, and demand for health services. 4. Increased availability and accessibility to quality preventative and curative health services. The project focused on the following child survival interventions and strategies at the level of effort projected in its Detailed Implementation Plan: • Reduced mother to child transmission (MTCT) of HIV/AIDS (Establish VCT services, train health workers, promote exclusive breastfeeding, introduce “EBF role models,” and BCC). 20% • Improved Nutrition (HH/C IMCI, Hearth Positive Deviance model, exclusive breast feeding promotion, vitamin A/iron campaigns, and BCC). 20% • Strengthened EPI (HH/C IMCI, training health workers, and immunization/vitamin A campaigns, and BCC). 5% • Improved Diarrhea case management and prevention (HH/C IMCI, training volunteers, training health workers, community-based promotion, strengthen ORT Corners, and BCC). 15% • Improved Pneumonia case management (HH/C IMCI, training health workers, and BCC). 20% • Improved Malaria case management and prevention (HH/C IMCI, training volunteers, training health workers, bed net promotion, training shopkeepers, and BCC). 20% At the household level the project saw its most notable achievements in expanded coverage of ITNs, EPI and VCT, and increased use of ORS to treat children experiencing diarrheal disease, as well as a drop in malnutrition in children under-five. More mothers received SP during their most recent pregnancy and knew effective ways to prevent the transmission of HIV. The project saw little or no change in VAC coverage, care seeking in response to IMCI danger signs, exclusive breastfeeding, or proper feeding of the sick child. Final Evaluation Report IEF/DHMT Nsanje Malawi 2 C.A.: No: HFP-A-00-02-00027-00 Among community volunteers and health workers, the VHCs, GMVs and HSAs are well established in the communities and competent in carrying out their responsibilities. Between half and two-thirds of health workers are demonstrating competence in using IMCI protocols when treating sick children at health facilities. Supportive supervision of the facility-based health workers, the VHCs and the GMVs is now strong, while supervision of the HSAs has diminished in frequency, perhaps due to the increase in their numbers and the difficulties inherent in reaching them since they spend most of their time out in the communities where communication is difficult. While the project increased the capacity of the DHMT in planning, implementing, monitoring and evaluating its services, it was not able to achieve a significant impact on the numbers of: stock outs of IMCI drugs at health facilities, health facilities providing daily immunization services, or static and mobile under-five clinics. In addition, the project achieved the following: • The spectacles shop and the cost wards, which were conceived and piloted by the project as part of its sustainability strategy, have both been successful. Responsibility for their oversight is now fully vested within the DHMT, which is exploring other income generating strategies. • Nsanje was one of the four of Malawi’s 27 districts to implement comprehensive IMCI, a major accomplishment in a challenging setting. • Community mobilization strategies were particularly popular and successful, having established a motivated and coordinated set of community health volunteers, committees and support groups. The Final Evaluation Team summarized the project’s accomplishments: 1. The communities have become more actively involved in their own health. 2. The DHMT has increased its capacity to plan, coordinate, supervise and monitor the health care services in Nsanje District. 3. The connection between the health care system and the communities is stronger and deeper due to the committed work of the MOH, volunteers and leaders. 4. People appear very committed and motivated to continuing the work of this project. 5. Most importantly, the Final Evaluation Team believes that as a result of this project, fewer women and children are getting sick or dying from disease. Final Evaluation Report IEF/DHMT Nsanje Malawi 3 C.A.: No: HFP-A-00-02-00027-00 B. ASSESSMENT OF RESULTS AND IMPACT OF THE PROGRAM 1. Results: Summary Chart Strategic Objective/Intermediate Results Baseline EOP Indicators Rate CI Target Rate CI SO: Families and Caretakers with Children Under Five Years of Age Practice Healthy Behavior and Seek Care From Quality Providers % of children 12-23m who are fully immunized before their first birthday 63%1 94/150 55-71% 85% 80% 103/129 70-90% % of children 12-23m who receives measles vaccine before 1st birthday 70% 105/150 62-77% 85% 80% 103/129 70-90% % of children 6-23m who received VAC within 6 months of the survey date 89% 196/221 84-93% 95% 85% 181/213 78-92% % of caretakers who treat children 0-23 months with ORT during their last diarrhea episode 55% 77/139 47-64% 75% 73% 65/89 58-88% % of children 0-23m who slept under an ITN the night prior to the survey2 17% 51/299 13-21% 60% 70% 209/300 61-79% % of women who took SP to prevent malaria during her last pregnancy 75% 225/299 70-80% 90% 86% 220/256 81-90% % of caretakers who took children 0-23 months to health workers for diarrhea, fever, or difficult breathing after recognizing illness symptoms 84% 51/61 72-92% 95% 74% 81/109 65-82% % of children 0-5 m who are exclusively breastfed 55% N/A 44-66% 75% 57% 50/87 42-72% % of children 6-23m who consumed the same amount of foods during most recent episode of reported illness3 49% 68/139 40-58% 75% 46% 41/89 24-68% % of children 6-23 months who are underweight (-2SD from median WFA WHO/NCHS reference) 39.5% N/A 33-45% 30% 27% 57/213 19-35% % of household that possess a bed net 29% 86/299 24-34% N/A 76% 227/300 71-81% 1 The baseline rate was measured without taking into account whether all the vaccinations were received prior to the first birthday. In computing the rate at the final evaluation only those children 12-23 months of age who received all of their immunizations prior to their first birthday were counted as being fully immunized. 2 The baseline and FE rates presented here are computed the same way, without regard as to whether the net was dipped. This is not consistent with how the related RAPID Catch indicator is measured, which requires that only ITNs that have been dipped be counted, the rate for which was 60% (181/300; CI: 50-70%). (See Attachment F for the related Rapid CATCH indicator.) 3 A review of the baseline rate and how it was computed found that this indicator only covered children with diarrhea, which is consistent with how the rate presented here for the FE was computed. For children with fever, ARI-symptoms and/or DD, the FE rate was 33% (54/164; CI: 23%-43%) Final Evaluation Report IEF/DHMT Nsanje Malawi 4 C.A.: No: HFP-A-00-02-00027-00 IR1. Strengthened Organizational Effectiveness and Health Management Systems % of facilities that have a stock out of essential medical supplies (e.g., ORS, SP, Cotrimoxazole)4 N/A N/A N/A 36% 5/14 N/A % of facilities by zone that have received at least 1 supervisory visit using observation of health worker practice in the 3 months prior to the facility assessment 67% N/A N/A 95% 93% 13/14 N/A % of HSAs by zone received at least 1 supervisory visit using observation of performance and feedback in 3m before assessment 89% N/A 90% 58% 7/12 N/A % of VHC/GMVs by zone that have received at least 1 supervisory visit by HSA using a checklist in the 3 months prior to the facility assessment N/A N/A 60% 67% 6/9 N/A % of VHCs that are established, trained that meet at least 1 time per quarter as verified in village records 70% N/A 90% 100% 9/9 N/A % of CSP annual work plan activities completed on time N/A N/A 95% 85% 154/181 N/A % increase in self-earned revenue from hospital sustainability activity per year. 0% N/A 15% N/A N/A IR2. Improved Prevention and Management of Childhood Illness % of sick children 0-5 years health cards were checked for immunization, VA status and growth monitoring.5 N/A N/A 95% 58% 96/165 N/A % of sick children 0-5 years who present with fever and are correctly assessed, counseled & treated for febrile illness/malaria. N/A N/A 95% 63% 66/104 N/A % of sick children 0-5 years who present with difficulty breathing and or cough and are correctly assessed, counseled & treated for ARI N/A N/A 90% 67% 44/66 N/A % of sick children 0-5 years who present with diarrhea and are correctly assessed, counseled & treated for diarrhea N/A N/A 90% 59% 19/32 N/A % of HSAs demonstrate competence in EPI vaccination and GM protocols at time of assessment N/A N/A 80% 100% 12/12 N/A % of HSAs demonstrate competence in counseling VHC & GMV in promotion of home care practices (ORT, malaria, ARI, W & S) at time of assessment N/A N/A 80% 100% 12/12 N/A % of GMVs demonstrate competence in counseling mothers and families in home care practices (ORT, malaria, ARI, BF, W & S) at time of assessment N/A N/A 70% 89% 8/9 N/A % of sick children referred by GMVs received attention by HF N/A N/A 70% N/A N/A # of DRFs established & operational according to the new strategy. N/A N/A 60% N/A N/A 4 This indicator did not specify the length of time covered. During the three-month period prior to the HFA, 43% (6/14) of health facilities had a stock out of cotrimoxazole, 57% (8/14) had a stock out of Fansidar, and 43% (6/14) had a stock out of ORS. In sum, 36% (5/14) of health facilities had a stock out of at least one of these essential medical supplies within the previous three months. 5 The way this indicator is written implies that the growth monitoring is done by card review, rather than by weighing the child during the health assessment. The result for this indicator as written is 58% (96/165). When this same indicator is measured based on the number of assessments where the health worker actually weighed the child, the rate is 41% (68/165). Final Evaluation Report IEF/DHMT Nsanje Malawi 5 C.A.: No: HFP-A-00-02-00027-00 Strategic Objective/Intermediate Results Baseline EOP Indicators Rate CI Target Rate CI IR3. Increased Availability & Accessibility to Quality Preventative & Curative Health Services % health facilities that provide daily immunization services N/A N/A N/A 23% 3/13 N/A % of planned (17,000) ITNs sold N/A N/A 85% 376% 64,000/ 17,000 N/A % of planned persons tested and counseled by NDH VCT unit in the 3 months prior to the assessment (600/yr) N/A N/A N/A 9,543 VCT 1/04- 6/06 N/A % of mothers of children 0-23 months who know at least 2 ways of reducing the risk of HIV/AIDS 58% 52-64% 80% 65% 195/300 56-74% % scheduled Under 5 clinics (static and mobile) conducted in the 12 months prior to facility assessment 80 N/A N/A N/A6 N/A IR4. Increased Community Participation & Demand for Preventative & Curative Services % of mothers with children 0-23 months able to demonstrate correct use of ORS 52% 46-58% N/A 67% 202/300 58-76% % of mothers with children 0-23 months who report hand washing with soap/ash before food preparation, before feeding children, after defecation, and after attending to a child who has defecated. 3% 1-5% N/A 9% 26/300 4-14% 2. Results: Technical Approach a. Overview The Strategic Objective (SO) of IEF’s CS-18 project, “Improved Child Survival in Nsanje District, Malawi, Through Community-Based Interventions and Strengthening of the Health Delivery Infrastructure” was: “Families and caretakers with young children increase the practice of healthy behaviors and seek medical care from quality sources.” To reach this SO the project planned four Intermediate Results (IRs): IR1. District Organizational Effectiveness and Management Support for Quality Child Care Strengthened, to be achieved by: • Strengthening planning, training, supervision, and evaluation skills. • Increasing inter-sector coordination. • Introducing new financial sustainability strategies. IR2. Health Provider Skills in Prevention and Management of Childhood Illness Improved, to be achieved by: • Increasing inter-sector understanding of Household/Community IMCI (HH/C IMCI). • Increasing the skills of health facility providers at all levels. • Increasing the skills of community volunteers. 6 The FE did not have access to data necessary to measure this indicator. Final Evaluation Report IEF/DHMT Nsanje Malawi 6 C.A.: No: HFP-A-00-02-00027-00 IR3. Availability and Accessibility to Quality Preventative and Curative Health Services Increased, to be achieved by: • Strengthening under-five and ante-natal clinic services. • Increasing availability to malaria ITN/IPT. • Evaluating and improving the Drug Revolving Fund strategy. • Improving nutrition by adopting the Hearth strategy. • Expanding HIV/AIDS prevention, testing and counseling services. IR4. Community Participation, Ownership, and Demand for Health Services Increased, to be achieved by: • Improving community mobilization and support to Health Surveillance Assistants (HSA) and Community Health Volunteers (CHV) in prevention and promotion activities. • Increasing access to appropriate and quality care and information by trained CHVs. • Improving district communication skills using the BEHAVE Framework. The project covers all of Nsanje District, which is located in the southern tip of Malawi. The majority of Nsanje’s population are subsistence farmers living below the poverty line. Nsanje’s unofficial population was 237,000 in 2005, based on data collected from the Village Health Committees. The total beneficiary population was 97,500, including 42,500 (18%) infants and children under the age of five, and 55,000 (23%) women of reproductive age (15-49 years). IEF’s partners in this project have been the District Health Management Team, the MOHP facilities, and the district’s 513 villages. The Ministry of Health and Population (MOHP) infrastructure consists of the DHMT, Nsanje District Hospital, one mission hospital, 11 health centers and eight health posts. The project focused on the following child survival interventions and strategies at the level of effort projected in the DIP: • Reduced mother to child transmission (MTCT) of HIV/AIDS (Establish VCT services, train health workers, promote exclusive breastfeeding, introduce “EBF role models,” and BCC). 20% • Improved Nutrition (HH/C IMCI, Hearth Positive Deviance model, exclusive breast feeding promotion, vitamin A/iron campaigns, and BCC). 20% • Strengthened EPI (HH/C IMCI, training health workers, and immunization/vitamin A campaigns, and BCC). 5% • Improved Diarrhea case management and prevention (HH/C IMCI, training volunteers, training health workers, community-based promotion, strengthen ORT Corners, and BCC). 15% • Improved Pneumonia case management (HH/C IMCI, training health workers, and BCC). 20% • Improved Malaria case management and prevention (HH/C IMCI, training volunteers, training health workers, bed net promotion, training shopkeepers, and BCC). 20% Final Evaluation Report IEF/DHMT Nsanje Malawi 7 C.A.: No: HFP-A-00-02-00027-00 b. Progress Report by Intervention Area Reduced Mother to Child Transmission of HIV/AIDS (20%) Discussion of Results and Outcomes: The project’s HIV/AIDS intervention activities focused on reducing MTCT of HIV/AIDS by supporting VCT services, training health workers, BCC, and promoting exclusive breastfeeding. Baseline EOP Indicators Rate CI Target Rate CI % of planned persons tested and counseled by NDH VCT unit in the 3 months prior to the assessment (600/yr) N/A N/A N/A 9,543 VCT 1/04- 6/06 N/A % of mothers of children 0-23 months who know at least 2 ways of reducing the risk of HIV/AIDS 58% 52-64% 80% 65% 195/300 56-74% % of children 0-5 months who are exclusively breastfed 55% 44-66% 75% 57% 50/87 42-72% The DHO reports that 9,543 individuals received VCT services at MOH facilities from when the services were initiated in January 2004 up to June 2006 and the demand is continuing to grow as a result in part of health education provided by project-trained health workers and volunteers. The recent availability of ART in Nsanje District for individuals at third or fourth stage AIDS is probably increasing the demand for VCT services as well, as people recognize the potential relief these drugs can afford. As of July 1, 2006 – 507 people living with AIDS (PLWA) were receiving ARVs: 97% (507/525) of projected capacity of the system in Nsanje District.7 A variety of channels were used to communicate HIV/AIDS messages into the communities. Twenty-seven PMTCT Support Groups were formed and trained in July 2005. Each group is made up of mothers who work together to encourage women of reproductive age to go for VCT, provide health education, do referrals and coordinate with health workers and other community volunteers. The project also setup 65 clubs involving teachers, students and school aged children who are not in school. The participants were trained in the HIV/AIDS messages and how to communicate them to their peers. Health education was provided through “Open Days” or community fairs, PLWA support groups, and high-risk groups who have received training from the project, including bicycle drivers and commercial sex workers. In addition, 490 traditional leaders and 32 DEC members have been trained on HIV/AIDS and taken a more visible and proactive role in educating their communities. One of the benefits of all this education and training on HIV/AIDS has been a perceived willingness on the part of community members to discuss and confront the issue, especially dangerous traditional practices, such as polygamy and ‘sexual cleansing.’ The project also provided training to MOH workers on syndromic treatment of STIs in March 2004 and PMTCT in November 2005. 7 The MOH set the cap on the number of people who can enroll in ART in Nsanje District at 25 individuals per month for the first quarter and then 50 individuals per month for every month after that, resulting in a cap of 525 enrollees at the end of the first year: July 1, 2006. Final Evaluation Report IEF/DHMT Nsanje Malawi 8 C.A.: No: HFP-A-00-02-00027-00 Factors Affecting Achievement of Program Objectives and Outcomes: The project faced numerous challenges implementing its HIV/AIDS intervention, the most tragic of which was the loss of its HIV/AIDS Advisor to the disease in May 2006. The project originally proposed to establish a mobile unit to provide VCT services in the district through the DHMT, however, this turned out to be contrary to MOH policy. Further, MOH policy required that government-employed VCT counselors attend the government-run training program to be fully certified to practice. As a result, the start of VCT services in Nsanje was delayed and the project’s role vis-à-vis VCT services in the district became supportive, focusing on promotion, rather than the establishment of the services themselves. Only seven VCT counselors were trained for Nsanje District, which is substantially fewer than the two counselors per health facility as set out in MOH policy. In addition, of these seven VCT Counselors only four or five are still practicing in the district, which are not enough to meet the growing demand for services. While the FE found positive change in mothers’ knowledge of ways to prevent HIV transmission, the project fell short of its target. Even with the project focus on health education and training, many of the high-risk causal factors continue or have worsened in recent years, including the traditional taboos around discussing reproductive health, the large number of migrants to/from Mozambique, and the increasing levels of poverty. The FE found no appreciable change in the rates for exclusive breastfeeding. Per FGDs, the BFSG members said they are educating mothers about the importance of EBF and mothers said that they were practicing EBF appropriately. The limited change in both this knowledge and practice indicators could be due in part to delays in initiating the health education activities. Lessons Learned: • On sensitive topics such as AIDS, the newly educated can become effective educators and advocates, especially when they already have credibility in their community and with their peers. The support they need from the system is limited to facts about the disease and culturally appropriate messages to communicate. • Achieving positive changes in knowledge and behaviors at the community level can take years, especially in isolated, under-resourced locations such as Nsanje, whose people tend to be more reticent about discussing sensitive topics and less amenable to behavior change. Recommendation: • Attention needs to be directed towards training and placing more VCT Counselors in the district so the growing need and demand can be met, especially in the more rural areas. Final Evaluation Report IEF/DHMT Nsanje Malawi 9 C.A.: No: HFP-A-00-02-00027-00 Improved Nutrition (20%) Discussion of Results and Outcomes: The project proposed to improve infant and child nutrition through HH/C IMCI, Positive Deviance Hearth, exclusive breastfeeding promotion, vitamin A/iron campaigns, and BCC. Baseline EOP Indicators Rate CI Target Rate CI % of children 6-23m who received VAC within 6 months of the survey date 89% 196/221 84-93% 95% 85% 181/213 78-92% % of children 0-5 months who are exclusively breastfed 55% 44-66% 75% 57% 50/87 42-72% % of children 6-23 months who are underweight (-2SD from median weight-for-age by WHO/NCHS reference) 39.5% 33-45% 30% 27% 57/213 19-35% % of sick children 0-5 years health cards were checked for immunization, VA status and growth monitoring. N/A N/A 95% 58% 96/165 N/A % of HSAs demonstrate competence in counseling VHC & GMV in promotion of home care practices (ORT, malaria, ARI, W & S) at time of assessment N/A N/A 80% 100% 12/12 N/A % of GMVs demonstrate competence in counseling mothers and families in home care practices (ORT, malaria, ARI, BF, W & S) at time of assessment N/A N/A 70% 89% 8/9 N/A The areas that demonstrated the most improvement were supervision of GMVs by the HSAs, GMV counseling of mothers and families in home care practices, and malnutrition rates in children under five years of age. Rates of VAC coverage and EBF practice showed no appreciable change during the project, with VAC coverage remaining relatively high at 85%. While baseline rates for growth monitoring during clinical assessment of the sick child were not available, fewer than two-thirds of the health workers checked the sick child’s immunization and vitamin A status, and weighed the child when brought to a health facility for care. The project initiated the PD Hearth strategy in Nsanje District in January 2005 by first providing an orientation to Area Development Committees, VHCs, agricultural extension workers, and health workers. That same month GMVs and HSAs were trained in the intervention, with the HSAs becoming Hearth Supervisors. Based on the results of a GM screening later that winter, 38 children from two villages (Nyanjikhwi and Kaudzu) were identified and recruited for participation. Of these children, 22 were within the normal range, 12 were moderately malnourished, and four were severely malnourished, as measured by weight-for-age. Positive deviants were identified whose choice of foods and feeding practices were studied and shared with the other participant families during the Hearth sessions. Three months after the completion of the Hearth sessions, the project reviewed monthly growth monitoring registries for 32 of the 38 participants to assess change and found the following results: Final Evaluation Report IEF/DHMT Nsanje Malawi 10 C.A.: No: HFP-A-00-02-00027-00 Pre Hearth Three Months Post￾Hearth Hearth Participants Weight-for-Age # % # % Normal 22 58 21 66 Moderately malnourished 12 32 10 31 Severely malnourished 4 10 1 3 Total 38 100 32 100 The report prepared by IEF and the FE FGDs found that the concept of PD Hearth was very popular with the community and health workers who participated. The two main challenges were the lack of food, due to the drought, and the lack of support from male heads of households. In response the report recommended: (1) repeating the Hearth sessions in the two pilot villages for the children who did not improve, and (2) conducting a one-day training session for men whose children are malnourished, to teach them about nutrition. A repeat session was done in the area during Summer 2005. However, the training of men was not done due to the resignation of the Officer-in-Charge. The project also helped to recruit and train mothers to participate in Breast Feeding Support Groups (BFSGs). These groups educate communities on EBF and complementary feeding, do home visits, make referrals and coordinate their work with the other community volunteers. Their work is complemented with that of the Growth Monitoring Volunteers who also provide health education, make referrals, do home visits, and coordinate their work with the other community volunteers, as well as do growth monitoring. Finally, the project has supported and participated on the district Targeted Nutrition Technical Committee that is charged with coordinating both the relief and long-term nutrition activities that are being implemented by a variety of organizations in the district, and has contributed a shipment of vitamin A for distribution throughout the district. Factors Affecting Achievement of Program Objectives and Outcomes: There were several factors that affected the achievement of the project’s nutrition interventions. The delayed start of PD Hearth, within two years of the end of the project, did not allow the project enough time to orient the key players; complete and assess the results from a pilot; and then expand the strategy to the rest of the district. While the project did complete the pilot, there was not sufficient time for scale-up and therefore, the strategy had little or no impact on the project’s nutrition indicators. Hearth was further complicated by the recent drought, which had greatly limited the quantity and number of options of food available in the district. The drought also led to the distribution of outside food aid in Nsanje, which undercut the PD Hearth strategy by exposing the community to free foods that are not locally sustainable. This external food aid also probably contributed to the lowered rate of malnutrition found in the project. The rate of malnutrition is also consistent with the rates reported in the most recent DHS survey 2004. Lessons Learned: • PD Hearth should be avoided in locations that are experiencing acute food shortages, especially when external food aid is being offered for free. • Nutrition programs need to be coordinated at the district and community levels to ensure they are not working at cross purposes. Final Evaluation Report IEF/DHMT Nsanje Malawi 11 C.A.: No: HFP-A-00-02-00027-00 • When selecting an implementation strategy one of the key factors to consider is whether there is sufficient time to have an impact. If not, then the objectives need to be changed or the strategy needs to be adapted or dropped. Recommendation: • Project staff should meet with GOAL (another international PVO doing nutrition and food distribution programs in Nsanje) to discuss what the project has done on PD Hearth, with a view towards transferring this strategy over to them. Strengthened EPI (5%) Discussion of Results and Outcomes: The project’s EPI intervention focused on HH/C IMCI, training of health workers, immunization/vitamin A campaigns, and BCC to expand coverage. Baseline EOP Indicators Rate CI Target Rate CI % of children 12-23m who are fully immunized before their first birthday 63% 94/150 55-71% 85% 80% 103/129 70-90% % of children 12-23m who receives measles vaccine before 1st birthday 70% 105/150 62-77% 85% 80% 103/129 70-90% % health facilities that provide daily immunization services N/A N/A N/A 23% 3/13 N/A The project succeeded in increasing coverage for both fully immunized children and those who have received measles vaccine before the first birthday. The project also trained staff in safe immunization technique and cold chain maintenance early in the life of the project. In addition, the project helped respond to two vaccine-preventable disease outbreaks by supporting three MOH ‘catch-up campaigns’ covering all five standard under-five vaccines in December 2004, April 2005 and August 2005. This support was in the form of assistance in planning, funding transport, supervision, and community mobilization. In addition, the project has promoted immunization coverage through its training and support of the GMVs and other health volunteers to provide health education and defaulter follow up and referral in the communities. Factors Affecting Achievement of Program Objectives and Outcomes: Two outbreaks of vaccine-preventable disease were reported in Nsanje District in 2004: 23 cases of measles and two deaths due to neonatal tetanus. Both outbreaks occurred in isolated rural communities near the border with Mozambique, a border that has been very porous since the 1980s when large numbers of Mozambican refugees began arriving and mixing in with the native population. Seasonal migration back and forth for months at a time by both Malawians and Mozambicans continues to this day, complicating attempts at expanding EPI coverage and increasing the potential for further outbreaks of vaccine-preventable diseases. EPI coverage was further limited due to periodic stock outs of vaccines and vaccine-related supplies, which the project was not in a position to directly resolve. The project also attempted to expand EPI coverage by increasing the number of health facilities in Nsanje District providing immunizations during their open hours, adopting a ‘no missed opportunities’ strategy. MOH policy, however, restricts the use of vaccines from previously opened vials, which made a ‘no missed opportunities’ strategy Final Evaluation Report IEF/DHMT Nsanje Malawi 12 C.A.: No: HFP-A-00-02-00027-00 impractical for the majority of health facilities in Nsanje District that have relatively small client populations. Lesson Learned: • Of knowledge, practice and coverage, the indicators that tend to take the most time and effort to improve, are those related to coverage. The project was able to initiate its work on the EPI intervention earlier than on some of the other interventions and as a result, achieved improvements in coverage, even though not all of the EPI strategies were ultimately successful. In sum, it is important to recognize and allow for sufficient time for these changes to occur, especially at the community level. Improved Diarrhea Case Management and Prevention (15%) & Improved Pneumonia Case Management (20%) Discussion of Results and Outcomes: The project focused on IMCI, training of volunteers, BCC, and community-based promotion to address both diarrheal disease and pneumonia in children.8 Baseline EOP Indicators Rate CI Target Rate CI % of mothers with children 0-23 months who report hand washing with soap/ash before food preparation, before feeding children, after defecation, and after attending to a child who has defecated. 3% 1-5% N/A 9% 26/300 4-14% % of caretakers who treat children 0-23 months with ORT during their last diarrhea episode 55% 77/139 47-64% 75% 73% 65/89 58-88% % of mothers with children 0-23 months able to demonstrate correct use of ORS 52% 46-58% 67% 202/300 58-76% % of children 6-23m who consumed the same amount of foods during most recent episode of reported illness 49% 68/139 40-58% 75% 46% 41/89 24-68% % of caretakers who took children 0-23 months to health workers for diarrhea, fever, or difficult breathing after recognizing illness symptoms 84% 51/61 72-92% 95% 74% 81/109 65-82% % of sick children 0-5 years who present with diarrhea and are correctly assessed, counseled & treated for diarrhea N/A N/A 90% 59% 19/32 N/A % of sick children 0-5 years who present with difficulty breathing and or cough and are correctly assessed, counseled & treated for ARI N/A N/A 90% 67% 44/66 N/A The project achieved measured improvement in prevention of diarrheal disease in the home, with more mothers reporting hand washing at all four appropriate times (before food preparation, before feeding a child, after defecating, and after attending to a child who has defecated).9 In 8 In the DIP the project proposed to develop a revolving drug fund to increase access to affordable and safe IMCI￾related drugs throughout the district. This strategy was dropped per recommendation in the midterm evaluation. 9 This is inherently a difficult question for mothers to understand and respond to since they have to list all four hand washing situations. In fact, it might be one of the few indicators where the knowledge rate is lower than the related practice rate. The project had originally approached this as an ‘or’ question rather than an ‘and’ question, which obviously changed the rates dramatically. Final Evaluation Report IEF/DHMT Nsanje Malawi 13 C.A.: No: HFP-A-00-02-00027-00 addition, FGDs found that many of the project-trained volunteers and committees are promoting improved hygiene and sanitation in the household and community, which might be contributing to improved DD prevention as well. The project’s results on home-based prevention and care of the child with DD and/or ARI symptoms were more mixed. Knowledge of when and how to prepare ORS improved, while feeding of and care seeking for the sick child stayed the same or diminished. Contrasting results were raised in the FGDs, where mothers and health workers both noted that the demand for health care at MOH facilities has increased markedly. When asked specifically why mothers might not be feeding their children when they are sick, they noted the belief that this might help the child stop vomiting and that it was too difficult to feed a child when s/he has no appetite. Once at the health facility, consistency of the case management for children with diarrhea and/or ARI symptoms is mixed as well, with rates for correct assessment, counseling and treatment that could be improved upon. Specific to DD, the project helped stock the ORT corners in 2003 and re-supply them in 2006. (DD and ARI are discussed further below in the section on IMCI.) Improved Malaria Case Management and Prevention (20%) Discussion of Results and Outcomes: The project focused on IMCI, training volunteers, bed net promotion, training shopkeepers and BCC for its malaria intervention. Baseline EOP Indicators Rate CI Target Rate CI % of planned (17,000) ITNs sold N/A N/A 85% 376% 64,000/ 17,000 N/A % of household that possess a bed net 29% 86/299 24-34% N/A 76% 227/300 71-81% % of children 0-23m who slept under an ITN the night prior to the survey 17% 51/299 13-21% 60% 70% 209/300 61-79% % of children 6-23m who consumed the same amount of foods during most recent episode of reported illness 49% 68/139 40-58% 75% 46% 41/89 24-68% % of caretakers who took children 0-23 months to health workers for diarrhea, fever, or difficult breathing after recognizing illness symptoms 84% 51/61 72-92% 95% 74% 81/109 65-82% % of sick children 0-5 years who present with fever and are correctly assessed, counseled & treated for febrile illness/malaria N/A N/A 95% 63% 66/104 N/A % of women who took SP to prevent malaria during her last pregnancy 75% 255/299 70-80% 90% 86% 220/256 81-90% The coverage and proper use of ITNs in households increased dramatically, with 17,600 ITNs distributed and 64,000 sold. Now, over three-fourths of households own at least one ITN and over two-thirds of children slept under an ITN the night prior to being surveyed. Mothers, health workers and community members agreed that the demand for ITNs continues to grow. The project has promoted the six-year old UNICEF-funded ITN retreatment program that has been going on in Nsanje District every November. Project support has been directed through the Final Evaluation Report IEF/DHMT Nsanje Malawi 14 C.A.: No: HFP-A-00-02-00027-00 150 ITN Committees and directly to the DHMT in the form of training, community mobilization, and assistance with logistics and planning. Rates for home based care, care seeking and health facility case management of malaria and febrile illness in children were consistent with those found for DD and ARI. The project did achieve positive progress in IPT coverage. As per the MTE recommendation, the project investigated reasons for low coverage of IPT in March 2005. The leading reason why mothers reported not taking SP during their pregnancy was due to a fear of side effects, specifically, fear of abortion and nausea. Others could not see a reason to take a drug when they were not sick. The project is promoting a DOTS approach tied to ANC visits. (Malaria is discussed further below in the section on IMCI.) Factors Affecting Achievement of Program Objectives and Outcomes: ITNs are being provided through various sources and distributed or sold through various mechanisms using a variety of different pricing schemes. PSI is selling ITNs to clients attending antenatal care clinics at the health facilities at a subsidized rate of 50 MK ($0.36 cents) (Malawi Kwacha) each. The health worker keeps 10 MK and the rest is used to purchase the next supply of nets. Since 2003, 53,000 ITNs have been sold through this system in Nsanje District. While this system is being managed and monitored entirely by PSI, they share reports with the DHMT and this project. Starting in 2004, the MOH and IEF have provided 5,000 and 6,000 ITNs, respectively. These ITNs were distributed to the HSAs who then distribute them to the 150 ITN Committees the project has organized and trained. These nets are then sold at 100 MK each ($0.71 cents), with the ITN Committee keeping 20 MK per ITN sold and the rest is returned to the district administration via the HSAs and DHMT for purchasing replacement stocks. Finally, the MOH has received 17,600 ITNs it will be distributing to the ‘poorest of the poor’ as determined by the communities themselves. While ITN coverage has greatly increased, the variety of strategies and pricing mechanisms used has created some confusion in the communities and resulted in mismanagement within the system. Some people have purchased ITNs for 100 MK; others have purchased the same type of nets for 50 MK; and still others will be receiving an ITN for free. The MOH worked closely with village headmen to develop a list of the poorest persons who will be eligible for free nets. As people become aware of the availability of the free nets, a disincentive to purchasing ITNs can result, undermining the viability of both the revolving fund and the incentives for increasing ANC coverage. The accounting system for the project and MOHP ITN sales is managed by the MOHP Malaria Coordinator and was set up by his predecessor. Neither had any prior experience or training in the development and management of revolving funds. The system does not account for the number of ITNs brought into the district, distributed to the HSAs, distributed from the HSAs to the ITN Committees, or sold by the ITN Committees. It does not account for the inventory on hand at each ITN Committee. The system is not independently reconciled or audited. In reviewing what is recorded (the amount of money returned by the ITN Committee to the HSAs), it appears that many of the ITN Committees are holding most of the cash, making it impossible to reorder new inventory, even though they all reported stock outs at the time of the final evaluation. The ITN Committees also lack a clear and consistent system for tracking the number Final Evaluation Report IEF/DHMT Nsanje Malawi 15 C.A.: No: HFP-A-00-02-00027-00 of ITNs they have received or sold, and their cash on-hand and turned over to their HSAs. Six of the seven ITN Committees interviewed during the final evaluation were unable to agree internally on the number of ITNs they had sold over the previous three months. Discussions have already been held about how to deal with any discrepancies. Solutions and punitive measures have been identified, including garnishing wages of the HSAs. The district lacks an overall vision for both its short-term and sustainable long-term ITN coverage goals, so it can maximize the opportunities resulting from this supply of ITNs. It is not clear what level of coverage they hope to achieve at the household or community level, nor do they have a plan for coordinating distribution so it will support the establishment of a sustainable supply system into the future. This level of planning and goal setting is currently beyond the capacity of the DHMT, but could be a focus of outside technical support, so the DHMT can maximize this unique, time-limited opportunity. Recommendations: • The sale of ITNs should be stopped immediately, until an effective and transparent accounting and control system can be designed and put into place. This system needs to be designed by a trained accountant or an organization with relevant experience and expertise. (NOTE: This recommendation does not pertain to the free distribution of ITNs noted above.) • The shortfalls of the ITN accounting system are due primarily to inadequate design. Even though some individuals probably benefited financially from these design flaws, the system that is in place is not sufficiently attuned and managed to identify these individuals or to quantify any amount taken. Therefore, it suggested that punitive measures not be pursued against individuals or committees at this time. Integrated Management of Childhood Illness (IMCI) Discussion of Results and Outcomes: Baseline EOP Indicators Rate CI Target Rate CI % of HSAs demonstrate competence in counseling VHC & GMV in promotion of home care practices (ORT, malaria, ARI, W & S) at time of assessment N/A N/A 80% 100% 12/12 N/A % of GMVs demonstrate competence in counseling mothers and families in home care practices (ORT, malaria, ARI, BF, W & S) at time of assessment N/A N/A 70% 89% 8/9 N/A % of children 6-23m who consumed the same amount of foods during most recent episode of reported illness 49% 68/139 40-58% 75% 46% 41/89 24-68% % of caretakers who treat children 0-23 months with ORT during their last diarrhea episode 55% 77/139 47-64% 75% 73% 65/89 58-88% % of caretakers who took children 0-23 months to health workers for diarrhea, fever, or difficult breathing after recognizing illness symptoms 84% 51/61 72-92% 95% 74% 81/109 65-82% % of sick children 0-5 years health cards were checked for immunization, VA status and growth monitoring. N/A N/A 95% 58% 96/165 N/A Final Evaluation Report IEF/DHMT Nsanje Malawi 16 C.A.: No: HFP-A-00-02-00027-00 Baseline EOP Indicators Rate CI Target Rate CI % of sick children 0-5 years who present with fever and are correctly assessed, counseled & treated for febrile illness/malaria N/A N/A 95% 63% 66/104 N/A % of sick children 0-5 years who present with difficulty breathing and or cough and are correctly assessed, counseled & treated for ARI N/A N/A 90% 67% 44/66 N/A % of sick children 0-5 years who present with diarrhea and are correctly assessed, counseled & treated for diarrhea N/A N/A 90% 59% 19/32 N/A % of facilities that have a stock out of essential medical supplies (e.g., ORS, SP, Cotrimoxazole) N/A N/A 36% 5/14 N/A The project exceeded its targets for improving the quality of counseling provided by the HSAs to the VHCs and GMVs on home care practices, and by extension from the GMVs to mothers and families. For home-based care of the sick child ORS use increased while feeding showed no change from baseline. The percentage of mothers who took their child to a health worker in response to the IMCI-related danger signs declined, which was consistent with what was found in FGDs, where IMCI danger signs were not as prominently mentioned as other health education messages.10 Once at the health facility about one third of health workers were completing the general assessment tasks of checking immunization and VA status, as well as growth monitoring. Between 59% and 67% of health workers were completing the assessment, counseling and treatment tasks and questioning specific to diarrhea, ARI and fever. Just over one third of health facilities noted experiencing a stock out in ORS, SP and/or cotrimoxazole within the past three months. The project provided orientation, education, and training to an impressive array of community members, local leaders, volunteers, committee members, government officials, agricultural extension workers, and health workers on HH/C IMCI and facility IMCI, as appropriate. Between October 2003 and August 2005, the project providing training and orientation on HH/C IMCI to 1,693 individuals, exceeding the number projected, and 493 VHCs, 96% of the number projected. 10 Per IEF: “The decline in the percentage of mothers who took their children to a health worker in response to IMCI-related danger signs could be due to two related factors. First, the baseline percentage may be an overestimation resulting from variation in how baseline and final KPC questions were asked and indicators were measured, as noted elsewhere in this report. Secondly, a decline may be the result of more mothers treating their children at home as instructed by C-IMCI messages, e.g., treatment of diarrhea with ORT, and referring only when the illness is serious rather than going to the health center for any sign of illness.” “I can educate about danger signs – what can be done at home and what needs to be done at a health facility.” A project trained HSA Final Evaluation Report IEF/DHMT Nsanje Malawi 17 C.A.: No: HFP-A-00-02-00027-00 HH/C IMCI Training Trainees Number of Trainees Planned Actual Individuals HSAs 171 184 BFSG Members 513 530 GMVs 400 459 TBAs 173 146 MOH Support Staff 80 80 Drug Vendors 200 152 Traditional Healers 142 142 Total Individuals 1,679 1,693 Groups VHCs 514 493 Total Groups 514 493 In addition, members of the Area Development Committees, District Executive Council, Traditional Authorities, and Agricultural Extension Workers received orientation to HH/C IMCI. SC and DHMT staff completed five sets of supervisory visits of the VHCs on HH/C IMCI between October 2004 and July 2005, which averaged five days in length where they would meet with 18-24 VHCs each time. For the facility-based IMCI, the project trained 13 individuals in IMCI supervision (October 2003), 13 in TOT skills (November-December 2003), and ten as IMCI Facilitators. Ninety health workers were trained in IMCI between June 2003 and July 2005, and were then given refresher training in February and March 2006. Ten quarterly supervisory visits were done between August 2003 and April 2006 jointly by the DHMT and IEF staff: meeting with an average of 24 health workers at eight facilities over five days per quarterly supervision. Initially the health workers shunned and even feared these sessions, but this gradually changed as they came to realize that the tone of the sessions had become more supportive and less punitive. The project also worked with the DHMT to establish and lead a district IMCI Coordinating Group to plan and oversee these activities, provided laminated IMCI wall charts for 13 health facilities, and developed and provided referral forms, which have yet to be put into consistent use. Project and MOHP staff reported that the IMCI experience has improved the quality of care provided in the community and at health facilities, and increased the DHMT’s skills in planning, administering, supervising, monitoring and evaluating health care services and training. They also agreed that it helped strengthen the relationships between the DHMT, the hospital, the health facilities, the HSAs and the volunteers working out in the communities. Factors Affecting Achievement of Program Objectives and Outcomes: The project has faced several challenges in its implementation of comprehensive IMCI, some of which have implications for continuation of this strategy. The first and primary challenge has been the lack of sufficient human resources. The MOH has experienced a ‘brain drain’ in its IMCI trainees, many having moved away. The district will need to replace these individuals at some point through recruitment and another round of the eleven-day IMCI training, which is very expensive and time consuming. There are three individuals at the district level MOH responsible for Final Evaluation Report IEF/DHMT Nsanje Malawi 18 C.A.: No: HFP-A-00-02-00027-00 supervising IMCI, but they have limited access to vehicles and fuel, and there is some doubt as to the district’s ability and motivation to support these activities on their own, without further support and assistance from the project. IEF staff have initiated a discussion during the final evaluation on how to make the IMCI training more manageable and cost effective so it can be sustained at the district level. The possibility of providing IMCI training as part of in-service duties, changing the scheduling of the training, or cutting back the number of training days or facilitators, might potentially result in some cost savings and limit the impact of days missed from work for the trainees. These options should be considered further. The project had planned in its DIP to use a drug revolving fund to resolve the chronic shortages of IMCI drugs in the district’s health facilities. It was later discovered that this strategy was not consistent with GOM policy and as a result had to be dropped. IEF experimented with drug revolving funds (DRF) in its previous CHAPS project in Chikwawa District without much success. The idea was to locate a DRF in an area lacking a health center in order to “fill the gaps” in basic service delivery. The intervention was included in the proposal for this child survival project as a carry over from the Chikwawa program on insistence by staff and later the MOHP decided against supporting this effort. The project then changed its focus over to supporting the DHMT and health facilities to improve the accuracy and rationalize their drug orders. In response to a recent policy change by the MOH, health centers are supposed to initiate the ordering of drugs directly to the central supply rather than going through the DHMT, which might be contributing to the number and duration of stock outs. As noted above, at least one-third of health facilities continue to experience stock outs of three of the essential drugs and supplies for implementing IMCI (cotrimoxazole, Fansidar, and ORS). This might be explained, at least in part, due to an increased demand for services created by IMCI and C/HH IMCI as well as the lack of a rationale system for basing ordering on demand. Many health facilities are often faced with the logic of, “you got one box last month and you will get the same this month.” In general, the direct restocking of drugs from outside the district has led to some improvement in the consistency of the supply. The question is now whether this system is sensitive enough to recognize subtle changes in demand and adjust accordingly on a monthly basis. Lessons Learned: • Developing comprehensive IMCI at the district level is an excellent capacity building exercise, the results of which can have broader implications, impacting the ability to plan, administer and monitor other activities. • IMCI must be implemented comprehensively with clinical training, drug supply, and community components. Supportive supervision must begin immediately after training in order to be successful. • The ultimate solution for ensuring broad and consistent use and coverage of IMCI throughout Malawi is the incorporation of IMCI into the training curricula for all types of health workers who provide care for the sick child and creating in service training that can be sustained by the DHMT themselves Final Evaluation Report IEF/DHMT Nsanje Malawi 19 C.A.: No: HFP-A-00-02-00027-00 Recommendations: • As one of the first of Malawi’s 27 districts doing comprehensive IMCI, IEF and the Nsanje DHMT need to focus their remaining time on analyzing the results of their work on IMCI and sharing it with the national MOH, the other NGOs working in the health sector, and the donors. • The DHMT needs to monitor the percentage of its health workers responsible for case management of sick children trained in IMCI. Once this rate drops below a certain point a new training needs to be planned and budgeted for. This will need to be driven by “IMCI champions’ at the local and national levels. • Options for cutting the costs of the eleven-day IMCI training without jeopardizing the quality of care or the certification requirements need to be brainstormed, tested, and analyzed. The results of this effort need to be shared and debated with national and international authorities at the policy levels. • The training of the HSAs should be designed on a CIMCI and performance-based curricula that is coordinated with the clinical IMCI training. • The DHMT could deal with the stock outs by setting up an emergency supply to fill gaps during the month. c. New Approaches The project’s cost wards and spectacles shop are new approaches that are being piloted in this project. (See Section d. Sustainability below.) 2. Results: Cross-cutting Approaches a. Community Mobilization Discussion of Results and Outcomes: Baseline EOP Indicators Rate CI Target Rate CI % of VHC/GMVs by zone that have received at least 1 supervisory visit by HSA using a checklist in the 3 months prior to the facility assessment N/A N/A 60% 67% 6/9 N/A % of VHCs that are established, trained and that meet at least 1 time per quarter as verified in village records 70% N/A 90% 100% 9/9 N/A The project set up a community level structure that is tied to the health care system at both the community-health facility as well as the community-DHMT levels through reporting, training, referral, feedback and supervision. It is made up of the following: • Volunteers who work individually: Growth Monitoring Volunteers (GMVs) and traditional birth attendants (TBAs); • Volunteers who work in support groups: Prevention of Mother to Child Transmission (PMTCT) and Breastfeeding (BF) Support Groups; and, • Volunteers who work on committees: Insecticide-Treated Net (ITN) and Village Health Committees (VHCs). Final Evaluation Report IEF/DHMT Nsanje Malawi 20 C.A.: No: HFP-A-00-02-00027-00 Community Volunteer – Health Care System Structure During the FE FGDs it was apparent that mothers, volunteers and health workers were well aware of the various parts of this system that exists in their community, knew who the volunteers were in their communities, and understood their responsibilities. Relationships between the health care system and the volunteers appeared to be mutually respectful and positive. Mothers have greater responsibility for the health of their families and a voice in how health care services are determined and managed in their communities through their VHC representatives and the other volunteers. One issue the project has grappled with since the beginning was finding the best mix of community volunteers and defining their responsibilities to ensure clarity and avoid redundancy. As can be seen in the following table, the responsibilities of the PMTCT and BF support groups are very similar and as a result, in several communities these two groups are in fact one. Also in reviewing their respective responsibilities, VHCs have the same responsibilities as the two types of support groups, except for doing the home visits, and could probably have been consolidated, resulting in some savings in community time and energies. Community ITN Committees GMVs & TBAs VHCs PMTCT & BF Support Groups HSAs (MOHP) Senior HSS (MOHP) Zonal Supervisor (MOHP) Environmental Health Officer (MOHP) PMTCT Coordinator (MOHP) Malaria Coordinator (MOHP) Final Evaluation Report IEF/DHMT Nsanje Malawi 21 C.A.: No: HFP-A-00-02-00027-00 Summary Responsibilities of Support Groups Committees Individual Volunteers Community Volunteers11 PMTCT BF VHC ITN GMV TBA TH Health education/promotion X X X X X X X Community mobilization X Refer clients to health facility X X X X X Provide care (i.e., assist in deliveries) X Home visits X Work with other volunteers X X X X X X X Conduct meetings/record minutes X X X Community planning/problem solving X Data collection/reporting X X Manage funds X Sell/distribute supplies X X Volunteers are appointed to these positions by the village chiefs with the consensus of other local leaders and community members. Most said they were motivated to continue serving because they valued the prestige that accompanied the position, what they had learned through the training, and the opportunity to improve the health of their neighbors. The FE also found that the groups and committees were meeting regularly at least once per month to share their experiences, receive feedback and plan future activities. Their level of motivation appeared to be high for continuing their work after the conclusion of the project as each type of volunteer listed future activities they planned to do that were doable with locally available resources and in addition to their regular responsibilities as volunteers. One capacity building strategy the project has used, which has been particularly useful, has been to have the more successful and active VHC members meet with their peers from other communities. This helps build connections between communities and demonstrates in a practical way the value of local solutions and support mechanisms. Factors Affecting Achievement of Program Objectives and Outcomes: Most of the supervision and support of the volunteers (especially the Malaria Coordinator’s support of the ITN Committees and the PMTCT Coordinator’s support of the PMTCT and BF Support Groups), is being driven by project rather than DHMT staff, due to limited DHMT personnel, finances and transport. While it is possible, or even probable, that many of the volunteer activities will continue beyond the life of this project, with little or no DHMT supervision, there will likely be some diminishment without it. As can be seen in the schematic above, much of the supervisory responsibilities fall on the HSAs, who are responsible for overseeing and supporting the GMVs, the VHCs, and the TBAs. When HSAs move away or change jobs, as many have in Nsanje, it takes time for the replacement to learn this system. Frequently, the project has had to respond to community expectations that went beyond the capacity and mandate of this project. Nsanje District has a history of receiving humanitarian assistance for Mozambican refugees and Malawians going back to the late 1980s and continuing 11 This list is based on a set of job descriptions prepared by project staff in response to a MTE recommendation that the volunteer roles be clearly defined. “I have a better relationship with the village health committee; we are more effective and I can provide better supervision.” A project-trained HSA Final Evaluation Report IEF/DHMT Nsanje Malawi 22 C.A.: No: HFP-A-00-02-00027-00 to today. As a result, some communities have grown used to receiving food, medical services, and other commodities for free from international sources. In addition, some volunteers have quit shortly after completing their training (for which they received per diem), when they discovered that they were not going to benefit financially from volunteering, even though this had been stated during the volunteer recruitment and training. Due to a variety of factors, Nsanje District is never far from requiring and receiving humanitarian assistance, which changes the environment for how long-term sustainable development can be accomplished. Lessons Learned: • Volunteer and committee responsibilities need to be clearly defined and communicated widely at the start and throughout the life of the project to avoid unrealistic expectations and confusion. This is especially critical when there are many different types of volunteers and committee/groups, such as in Nsanje, and where their responsibilities overlap. • Many of the volunteers are motivated by a wish to serve their communities and the respect they earned through their participation. This incentive is something even the poorest, most isolated communities can provide. • There tends to be less turnover in community volunteers than in MOH staff (including the HSAs), making volunteer support an excellent long-term investment for achieving primary health care goals. Recommendations: • The HSAs and CHVs play very important roles in extending the reach of MOH services to the villages. It is important to continue to track the number of active HSAs and CHVs by location so their numbers can be replenished when necessary. • Where feasible, the practice of having successful VHCs visit new or less motivated VHCs should be continued. This is an incentive for the stronger VHCs and a great learning opportunity for the others. b. IEC and Behavior Change Interventions Discussion of results and outcomes: The project used a cascade approach to disseminate its health education messages – recruiting, training and supervising volunteers, health workers, community leaders, and people at risk to convey the project’s health education messages to their peers, neighbors, clients and constituents. CSMC members and HSAs were trained in adult learning styles and facilitation skills which they passed on to the VHCs and the other volunteers they supervise. The project also supported “Child Health Days” (day long public health community festivals) and general health education sessions led by health workers and project volunteers through community meetings, dramas and home visits, as well as the distribution of health education posters and other materials. The project was particularly strategic and effective in bringing together the Traditional Authorities and other community leaders to discuss harmful practices, such as ‘sexual cleansing’ (Kupita Kufa), and involve them in communicating the messages to their constituents. Project staff were trained in the BEHAVE Framework by the IEF Director of Programs in June 2005. Based on this experience the staff identified a target group (fishermen and business women) and hosted a training session with 57 participants where they discussed HIV/AIDS and Final Evaluation Report IEF/DHMT Nsanje Malawi 23 C.A.: No: HFP-A-00-02-00027-00 brainstormed the risk behaviors in these target groups and identified strategies for behavior change including drama shows, ‘stopping nocturnal activities,’ group discussions, and forming a committee to fight against HIV and AIDS. Factors Affecting Achievement of Program Objectives and Outcomes: There was no follow-up report on whether the piloted Behave Framework activities were done and if so, whether they resulted in behavior change. Since this pilot was done late in the project’s life there was little opportunity to attempt using the framework on the project’s other interventions. As noted in the previous section, each of the project-trained volunteers and groups provide health education, however, each type of volunteer is limited to a specific topic area. For instance, TBAs educate on proper care seeking for delivery, breastfeeding, complementary feeding, and HIV/AIDS; Breastfeeding Support Group members educate on VCT and HIV/AIDS. However, none of the volunteers are expressly charged with educating mothers on IMCI danger signs. This is consistent with what was found in the FGD and KPC results at the final evaluation. Lessons Learned: • When there are several types of volunteers, each of which specializes in a relatively narrow topic, it is easy for a project to miss opportunities to educate and promote behavior change. • It is important to involve the local leadership in project planning and implementation, especially those related to sensitive topics, such as HIV/AIDS. They bring the necessary credibility to the project’s health education messages. Recommendation: • Review training curricula and incorporate the IMCI danger signs into all volunteer and health worker training and make sure they are being reinforced during supervision. c. Capacity Building Approach Strengthening the PVO Organization Going through the process of successfully designing, implementing, monitoring and evaluating a CSHGP-funded child survival project is a tremendous learning experience for a PVO. The technical and administrative rigor required not only builds the capacity of PVOs to do child survival projects more effectively, many of the skills developed and experiences can be carried over to other areas. More specifically, IEF has increased its abilities in: • The use of qualitative and quantitative assessment tools, especially LQAS and the BEHAVE Framework. • Project planning through the use of both qualitative and quantitative data and the involvement of as many key stakeholders as possible. • Building working partnerships with district and health facility level MOH and communities. By actively involving local partners at each step of the way, much has been passed on to the MOH and communities, as noted in the following sections. Final Evaluation Report IEF/DHMT Nsanje Malawi 24 C.A.: No: HFP-A-00-02-00027-00 • Identifying and addressing its own staff’s capacity building needs, as noted throughout the Management section below. These are skills they will take with them when they move within IEF and beyond. Strengthening the Local Partner Organization: District MOH Discussion of Results and Outcomes: A primary focus of this project was to ‘strengthen organizational effectiveness and health management systems’ in Nsanje District so the partnering MOH would be well prepared to take on the responsibility for sustaining project improvements beyond the conclusion of the this USAID-funded cooperative agreement. In February 2004, the project implemented an Organizational Capacity Assessment involving thirteen participants. The project adapted the Organizational Capacity Assessment Tool (OCAT) developed by PACT for use in this exercise. As noted in its report on the results, “…this method relies on self-assessment and the findings are valid only to the degree that team members are objective, candid, and knowledgeable about their organization.” It should not, therefore, be considered as a baseline or evaluation tool, but rather a mechanism for generating discussion on what capacity building means in the context of a particular project and identifying self-perceived capacity building needs. Based on a scoring continuum from 1-100, the tool looked at how the respondents viewed MOH capacity (100 being high capacity and 1 being low) and to what extent they were in consensus in their assessment (100 being in complete agreement and 1 having no agreement on capacity at all). Capacity Building Topics Scoring Capacity Consensus Governance 79 70 External Relations 75 77 Program Development/Service Delivery 68 66 Management Practices 68 59 Sustainability 64 68 Financial Resource Management 62 61 Human Resource Practices 58 45 This tool identified human resource practices, financial resource management and sustainability as the capacity building areas requiring the most discussion and attention. The project used several strategies to build the organizational capacity of the MOH, including traditional training; on-the-job training through joint IEF-MOH planning, implementation and evaluation of project activities; development of supervisory checklists; regular coordination; and joint planning and piloting of sustainability strategies. The motivation of the DHMT to directly involve stakeholders in the development of their annual plans and budgets increased over the course of the project. The DHMT used to request written reports from the NGOs working in Nsanje District. These would be incorporated into the District’s annual District Implementation Plans. Now the district convenes an annual three-day planning meeting, which is modeled on this project’s DIP Workshop, that includes all the NGOs working in the district to prepare its DIP. The project also promoted the establishment of cost wards and a spectacle shop at the district hospital as ways to generate revenues without diminishing the quality or availability of services. Final Evaluation Report IEF/DHMT Nsanje Malawi 25 C.A.: No: HFP-A-00-02-00027-00 Both of these systems have been adopted and are being entirely run by the DHMT; the district has started to explore other revenue generating strategies based on this experience. The project presented the option of establishing a zonal based supervisory system to the DHMT, which it adopted and instituted. It focused on improving the quality and consistency of the supervision provided within the system. The project led a collaborative process to update and adapt supervisory checklists, and provided training in supportive supervision for the MOH managers throughout all levels within the District. Post-training follow-up found that six of the seven Zonal Supervisor trainees had worked to ‘reform’ the VHCs in their areas and ten of the 11 health facility In-Charges had formed ‘health center management teams’ at their facilities. Integrated supervision was done in December 2004, April 2005, August 2005, and March 2006 jointly by IEF and MOH staff to ensure retention. While the project did not specifically commit in the DIP to strengthen the skills of the other NGOs working in Nsanje it did establish mutually beneficial relationships with FAST (Friends of AIDS Support Trust – a Malawian NGO doing VCT, AIDS peer education, and working with orphans and vulnerable children) and Family Health International (FHI). Together they coordinate their community level work through the VHCs and their district level work together with the MOH. Factors Affecting Achievement of Program Objectives and Outcomes: The primary challenge for the DHMT at this time and into the future is whether it has the motivation, momentum and resources necessary to continue on this course of improvement, especially with its responsibilities for supervision and coordination. Most of the supervision of the health facilities is done by the MOH Coordinators, who have responsibility over the different project CS interventions (i.e., IMCI, HIV/AIDS, malaria, etc.). However, their salaries are paid from the clinical services they provide rather than their work as Coordinators. The number of NGOs working in Nsanje has also continued to grow, putting further strains on the MOH which is charged with coordinating and overseeing all of the health services in the District. Lessons Learned: • A good working relationship between the partners is critical to the success of a project. This takes time and regular contact to develop or redevelop when there is a change in staffing. Ideally this relationship needs to be established prior to starting work at the community or health facility levels. • Creating supervision zones was beneficial to organizing the supportive supervision system. • Per the DHMT: New NGOs desiring to do health programs in Nsanje need to first coordinate with the DHMT to ensure that their plans are consistent with the District Implementation Plan. Sufficient time needs to be dedicated to this process. NGO programs need to be measurable and evidence based to work in Nsanje District. Final Evaluation Report IEF/DHMT Nsanje Malawi 26 C.A.: No: HFP-A-00-02-00027-00 Strengthening Health Facilities and Health Worker Performance Discussion of Results and Outcomes: Baseline EOP Indicators Rate CI Target Rate CI % of facilities by zone that have received at least 1 supervisory visit using observation of health worker practice in the 3 months prior to the facility assessment 67% N/A 95% 93% 13/14 N/A % of HSAs by zone received at least 1 supervisory visit using observation of performance and feedback in 3m before assessment 89% N/A 90% 58% 7/12 N/A % of HSAs demonstrate competence in EPI vaccination and GM protocols at time of assessment N/A N/A 80% 100% 12/12 N/A % of HSAs demonstrate competence in counseling VHC & GMV in promotion of home care practices (ORT, malaria, ARI, W & S) at time of assessment N/A N/A 80% 100% 12/12 N/A % of sick children 0-5 years health cards were checked for immunization, VA status and growth monitoring. N/A N/A 95% 58% 96/165 N/A % of sick children 0-5 years who present with fever and are correctly assessed, counseled & treated for febrile illness/malaria N/A N/A 95% 63% 66/104 N/A % of sick children 0-5 years who present with difficulty breathing and or cough and are correctly assessed, counseled & treated for ARI N/A N/A 90% 67% 44/66 N/A % of sick children 0-5 years who present with diarrhea and are correctly assessed, counseled & treated for diarrhea N/A N/A 90% 59% 19/32 N/A For health facility and health worker capacity building, the project focused on improving supervision, referral system, and overall quality of care, as well as establishing IMCI case management. The project used an adapted version of the BASICS Integrated Health Facility assessment (IHFA) at baseline and final, as well as a project-designed Health Facility Quarterly Checklist to assess and monitor progress on health facility and health worker capacity building. However, baseline rates were not available for some of the IHFA indicators and there appeared to be some discrepancy in how rates were being measured from the baseline assessment to the final evaluation surveys. The project succeeded in turning around the perception of supervision as a punitive activity. In FGDs and interviews, the HSAs, health workers and EDHMT staff reported that supervision is now viewed as a positive experience, where it used to be avoided due to fears of punishment. Supervision is now being provided consistently to the health facilities, but less so to the HSAs. One reason for this is the difficulty in reaching many of the HSAs since they spend most of their time outside the health facilities working in the communities. To its credit the Government of Malawi has established paid Health Surveillance Assistants (HSAs) at the community level who are responsible for providing health education, supervising community health volunteers/VHCs, distributing health supplies, reporting on vital health events, and community mobilization. There are currently 184 HSAs working in Nsanje District. The quality of work done by the HSAs has also been a success of the project, as they have Final Evaluation Report IEF/DHMT Nsanje Malawi 27 C.A.: No: HFP-A-00-02-00027-00 demonstrated competence in vaccination and growth monitoring protocols and home care counseling skills in support of the VHCs and GMVs. While the project reached its target in the number of health workers trained in IMCI, changes in their skill level were less impressive, with only half to two-thirds of health workers completing the necessary tasks in assessing, treating and counseling at health facilities. This is probably due to the need for additional supervision and the ‘brain drain.’ Several of the health workers observed and interviewed during the final evaluation had not received IMCI training since they only arrived in these positions after the project’s IMCI training. Overall, health workers and mothers reported an increased demand for services at health facilities due to greater satisfaction in the quality of care provided. Mothers said they were seeking services at their local health facility more frequently because they had a better understanding of danger signs, the attitudes of health workers had improved, and they had increasingly come to believe that going to traditional healers was not helpful. Factors Affecting Achievement of Program Objectives and Outcomes: The main challenges to sustaining these accomplishments are the turnover in health facility staff, drug stock outs, and the distribution of HSAs throughout the district. While the project reached its target for the number of health workers trained in IMCI, many of them have since moved away. The district has no formal system in place for tracking how many of its health workers who have been trained in IMCI have left, so training of replacement can be planned and budgeted. Just over one third of the health facilities experienced a stock out of ORS, cotrimoxazole and/or Fansidar prior to the final evaluation. Reasons for these stock outs could be due to the recent transfer of responsibility for ordering drugs from the DHMT over to the health facilities, and the increased used of medicines resulting from the increased demand for services. Lessons Learned: • The role of the HSAs is very important to the continued success of project activities, as they are the face of the MOH in the communities. It is important that the MOH and communities continue to identify sustainable incentives and ways to reward the HSAs, and the DHMT continues to provide supervision, training, and feedback. Training The project successfully completed all of the trainings it set out to implement in its Training Plan from the DIP, except for the drug revolving fund that was dropped per recommendation in the MTE. The project also met or exceeded a majority of the targets it set for itself for the number of trainees per topic. Project training was provided jointly by MOH and IEF staff, with both actively involved in planning, curriculum development, logistics, training and monitoring the results through joint supervision, in an effort to transfer and institutionalize these skills within the DHMT. These responsibilities have increasingly been taken on by the MOH over the life of the project, with the most recent IMCI training only requiring minimal support from IEF. The evidence that project training has led to improvements is noted throughout this report. Summary highlights include: HSAs are effectively counseling and supporting the VHCs and Final Evaluation Report IEF/DHMT Nsanje Malawi 28 C.A.: No: HFP-A-00-02-00027-00 community volunteers; health workers are providing higher quality case management for sick children; the quality and consistency of supervision has increased; and drug vendors, TBAs and traditional healers better understand danger signs and are making referrals to health facilities. The DHMT has expressed its commitment to continuing to provide training, noting that the Coordinators are prepared to include it in their annual plans and budgets. With the limited financial resources, the high costs of providing the training (HW IMCI especially), and the high demands on coordinators’ time, it is difficult to see how all of the training will continue at the current levels. Lesson Learned: • Traditional didactic training by itself is not sufficient in building organizational capacity. It needs to be complemented with on-going on-the-job training, feedback and supportive supervision before changes in practices and organizational culture can occur and be institutionalized. d. Sustainability Strategy The project planned two sustainability strategies: a spectacles shop and cost wards, both of which have been setup at the Nsanje District Hospital and are now running entirely under the direction of the DHMT. IEF provided the Hospital with an initial supply of reading glasses and distance frames; helped set up a spectacles display in the OMA’s office; and trained a retired Ophthalmic Medical Officer to manage it. From August 2005 to July 2006, 97 pairs of reading glasses at 700 MK ($5) each and four pairs of distance frames/lenses at 2,400 MK ($17) each were sold for a total of 77,500 MK ($550) in gross revenues. New supplies of glasses are available on consignment from Blantyre. The cost of administering the spectacles shop are minimal and it builds nicely on IEF’s organizational expertise in eye care, providing an excellent potential for expansion to other district hospitals nation-wide. Cost wards were established by the project in the maternity section of the Nsanje District Hospital in 2005 and additional ones have been setup for general inpatient services in 2006, in response to the initial success of this strategy. The cost wards are private rooms that clients can use for an added fee (100 MK ($0.71) per night without food or 500 MK ($3.57) with food, plus a 500 MK deposit). The DHO reports that the costs wards have proven very popular and are continuously occupied. The national government has told the DHO that the revenues generated must be turned over to the national treasury, which would be a loss to the DHMT’s budget and therefore defeat the purpose of the strategy. In response the DHO is trying to setup a bank account to hold the revenues so they can be retained by the district and used to cover its own expenses. Financial reports were not made available to the FE Team so it was not possible to determine costs, revenues, and how the revenues were being used. The cost wards provide patients with a choice, which they can decide to take advantage of or not, without any impact in the quality of health care services they receive. As a result of the success of both strategies, the DHO is more in tune with the needs and preferences of their clients and has greater confidence to test other options that generate revenues without compromising quality Final Evaluation Report IEF/DHMT Nsanje Malawi 29 C.A.: No: HFP-A-00-02-00027-00 or access. They are now considering a ‘fast track service lane’ at the hospital, where clients in need of non-emergency services can get expedited services. A formal sustainability phase-over plan has not yet been prepared. However, the project has been discussing and acting on phase-over activities since the DIP and more recently since IEF’s Program Director led IEF and MOH representatives through a structured discussion on sustainability in 2004. During the FE the DHO and Deputy DHO voiced a strong commitment to support continued project activities, particularly in the planning, training and supportive supervision. Some concerns were voiced by the DHMT Coordinators about their ability to continue to cover the costs of the training (especially the IMCI, which is very expensive), supervision, and community support within the context of their current budget and other, competing responsibilities. There needs to be a driving force at both the district and national levels for IMCI to continue in the district. The project has helped to build demand for MOH services, as noted throughout this report, and the community is engaged through their volunteers, committees and support groups to influence health care services in their area. B. PROGRAM MANAGEMENT 1. Planning Inclusiveness of the Program Planning Process: Together with the MOHP, IEF invested over three months in preparing the project DIP, which involved joint needs assessment, data collection, analysis, design, planning, and writing. Several of the senior IEF staff as well as members of the DHMT who were involved in preparing the DIP still work in Nsanje. The consensus among them was that the DIP and subsequent planning processes were inclusive and participatory. All senior project and MOHP staff had copies of the DIP, which they reportedly turn to frequently when reviewing progress and developing annual work plans. As a result of its participation in the preparation of the project DIP, the DHMT reports that it has increased its ability to take a more linear, inclusive, and data-driven approach to developing its own annual District Implementation Plans and budgets, and has since started inviting all of the NGOs working in Nsanje to participate in this process. This increased capacity will probably become even more important in the future as the Government of Malawi continues on its course towards decentralization – transferring national level oversight responsibilities from the national to district level government. Practicality of the DIP Work Plan and Gaps in the DIP: The project recognized that it was overly ambitious in the number and nature of activities it planned to implement and in the indicator targets it set in the DIP. Most of the IEF staff who started this project came from IEF’s Community Health Partnerships project (CHAPS) in neighboring Chikwawa District. As a USAID Mission-funded, MCH project, the administrative and programmatic requirements tended to be less cumbersome than those of the CSHGP. Many of the project activities required much more time to get approval, plan and initiate than had been assumed in the development of the DIP Work Plan. For instance, HH/C IMCI required much more time than originally planned to orient the communities, the traditional authorities and the health workers. Final Evaluation Report IEF/DHMT Nsanje Malawi 30 C.A.: No: HFP-A-00-02-00027-00 Further, the project experienced some false or delayed starts with strategies that turned out to be contrary to national MOHP policy. For instance, the MOHP’s open vial policy limited the district’s ability to increase the number of clinics providing EPI services during open hours. The drug revolving fund strategy and the mobile VCT unit were not permissible under MOHP policy, even though the DHMT accepted or assured the project otherwise during the development of the DIP. In addition, some of the project objectives required substantial inputs from other entities, which the project had little or no influence over. Examples included the ITNs provided by PSI and the MOH that were necessary to reach the project targets for increased access to and use of ITNs; the increased use of VCT services, which were to be provided entirely by the MOH and other NGOs; and the increased EPI coverage that relied heavily on the MOH’s supply of vaccines. In addition, the large nationally managed malaria and HIV/AIDs programs required the DHMT to reschedule established work plans. The National Aids Commission required that all training be completed using their training curricula for VCT counselors at a central location outside the district delaying the implementation of the VCT program. The National malaria program required retraining of the DHMT Malaria Coordinator soon after he received IMCI training in the district. There has been an inherent conflict between relief and development in Nsanje. Over the life of this project and before, Nsanje District has been living on the edge of a public health, nutrition, and economic emergency, necessitating the need to distribute emergency donated food, bednets, and medicines for free. This has undercut the project’s ability to do sustainable PD Hearth and revolving funds for bednet distribution. 2. Staff Training IEF project staff received training on a variety of topics, including PDI Hearth, the BEHAVE Framework, STI syndromic treatment, PMTCT, IMCI Supervisor Training, LQAS, proposal writing, quality assurance, and supportive supervision. Each of these topics was closely related to either a technical or administrative need of the project. After each training the project staff implemented activities that related directly to the topics, for instance development of a BEHAVE Framework targeting high-risk behaviors in fishermen and the use of LQAS for monitoring and a mini KPC survey to produce data. In addition, project staff identified the following training topics they felt would have been useful: IMCI supervisor and facilitator training,12 EPI INFO software, financial management, and developing and managing partnerships, particularly with public sector agencies. The challenges have been that it is costly to send staff to trainings and difficult to cover their responsibilities while they are away. 3. Supervision of Program Staff The supervisory system that IEF has developed over the course of this project is strong and has been well run within and between three levels: HQ in the United States, the IEF Country Office 12 NOTE: IEF’s current IMCI Advisor was not a certified IMCI Facilitator or Supervisor. Final Evaluation Report IEF/DHMT Nsanje Malawi 31 C.A.: No: HFP-A-00-02-00027-00 in Blantyre, and the field program in Nsanje. One of the benefits of being a smaller NGO is the opportunity to develop close working relationships between all of the staff over time. A smaller size allows an organization to make adjustments and target capacity building to each individual. Since the project staff are relatively few in number in Nsanje, they have constant daily contact with one another. In addition, formal planning meetings are held monthly to review work plans and project financial needs for the next month. Each staff person has a signed position description that details his/her responsibilities and the lines of authority. Supervisory relationships throughout the IEF staff are clear and well understood by all. Several persons from the IEF headquarters were directly involved in planning, training, and evaluation of the project, over the course of the project. (Supervisory relationships within the MOHP are described in the section on capacity building of partners.) 4. Human Resources and Staff Management Personnel Policies and Procedures: IEF has developed an employee policy manual specifically for its programs in Malawi and provided a copy to each of its staff along with a signed position description. The MOHP has essential personnel policies and procedures in place, however, this was not a focus of this project. Staff Morale and Turnover: Staff morale was relatively positive during the final evaluation even though some IEF employees will be loosing their jobs after the conclusion of this project. There was no evidence that staff morale was a particular problem during the project for the current staff. As a result of staff turnover, there was a protracted period in 2004 and again at the time of the final evaluation when not all of the key positions in IEF were filled. Responsibilities had to be moved from person to person, (i.e., IMCI Advisor had to take over the malaria intervention; the Project Manager had to take over EPI and nutrition when the MCH Advisor left and was not replaced; the Project Manager also had to take over the HIV/AIDS intervention when the HIV/AIDS Advisor passed away in early 2006.) Neither the MCH nor the HIV/AIDS Advisor positions were occupied at the time of the final evaluation. Recruiting and retaining the IMCI Coordinator was the most problematic, with three different individuals hired for short periods before resigning. As certified IMCI trainers they tend to be in high demand and as a result they will often follow the best salaries and work locations. While staff noted that they were able to cover these responsibilities adequately, replacing key staff repeatedly during the life of a project can often result in lost momentum while the new staff get oriented and caught up, especially in doing the joint activities with the MOHP staff. Reasons for staff departures included the death of the HIV/AIDS Advisor in 2006 and the difficulties inherent in working in Nsanje. Nsanje is considered a ‘hardship’ post by many Malawians from the cities because of the difficulty in travel (lack of a tarmac road between Nsanje and Blantyre), mosquitoes, heat, and the lack of schools, health care, and other common conveniences for themselves and their family members. The other positions were held by individuals who had worked for IEF for several years in Malawi and as a result, had grown used to working together. IEF also experienced turnover at its headquarters in the U.S., as the original Final Evaluation Report IEF/DHMT Nsanje Malawi 32 C.A.: No: HFP-A-00-02-00027-00 backstopping person left to go to another PVO. Her responsibilities were then taken over by the Director of Programs who has several years of experience backstopping CSHGP grants. This situation is mirrored in the MOH, where the DHO, Deputy DHO, and several of the DHMT staff involved with this project changed over the life of the project. It takes time to introduce and orient new partner staff to the project and gain their support and involvement, especially on the capacity building objectives, which often need to be adjusted to fit the unique needs of the new staff. IEF is preparing letters of recommendation and offering assistance in preparing and editing resumes for its staff who will be looking for new jobs after the conclusion of this project. There are many international, national and local organizations working in Malawi in the health sector looking for individuals with proven experience working on USAID and other internationally funded programs. 5. Financial Management Every month the IEF project staff in Nsanje meet to plan and budget for their activities for the following month. They prepare a request for funds that is sent to IEF/Blantyre for approval, and is then forwarded to IEF HQ in the U.S. for a draw down. Nsanje District lacks a formal bank, so the Project Administrator has had to travel three hours one-way to Blantyre to do banking transactions and collect cash transfers, at least two times each month. The national banks closed in Nsanje district several years ago due to increased armed robbery and safety concerns. The IEF Project Manager and Project Administrator in Nsanje prepare monthly financial reports using Quick Books and submits them along with the original receipts to the IEF Country Director in Blantyre. The Country Director reviews the report and accompanying documentation; completes a bank reconciliation; and then sends the financial and bank reconciliation reports to the accountant at IEF HQ for final internal review. Headquarters uses these financial reports from the field to prepare its quarterly reports to USAID. The accountant from HQ makes annual trips to Blantyre where he reviews the receipts and other documentation to ensure compliance with USAID rules and regulations. No budget or variance reports are provided back to the field for review, although there is agreement on an annual budget for the year. The project budget did have to be amended due to the difficulties in getting a tax waiver for purchased vehicles, an impediment to all the international NGOs working in Malawi. This meant that the project was not able to purchase the second vehicle for its work on VCT or the motorcycles for use by the MOH. Even with these changes, the project fully expects to meet its cost share requirements and spend down all the allotted USAID funds. The financial and budgeting skills within the MOHP are limited, as found in reviewing the accounting systems for the ITN revolving fund and the spectacles shop. Financial reports for the cost wards have not been made available to the project or to the FE Team. The project has offered to provide technical assistance on financial planning and budgeting, but this offer was not accepted. Final Evaluation Report IEF/DHMT Nsanje Malawi 33 C.A.: No: HFP-A-00-02-00027-00 6. Logistics Logistics are an on-going challenge in Nsanje. The main road into the district is unpaved; the secondary roads are difficult to travel on during the dry season and often impassable during the two annual rainy seasons. The “eastern bank,” a large section of Nsanje District, is cut off from the majority of the district due to a flooded bridge so drivers have to take a detour that takes several hours. The lack of a bank in Nsanje District has meant staff have to make frequent trips to Blantyre often carrying large amounts of cash, which can be dangerous. This was compounded by the fact the project had only four functioning vehicles during most of the project and the maintenance and fuel costs increased annually, making it difficult for the four Advisors to do their regular field work. The MOHP faces the same difficulties with their lack of vehicles and other equipment. The MOHP placed short wave radios in the health centers and the district hospital, which has been a big help in making referrals, but some have broken down and they do not have access to the necessary spare parts. 7. Information Management The project established a system for tracking its activities and inputs, each of which was tied directly to one or more project results and/or strategic objective. Project staff used this information to review and report on project progress, plan future activities, and inform feedback to stakeholders. Progress at the community level, particularly related to changes in knowledge, behaviors and access to care, was tracked through the use of four mini KPC surveys using LQAS the project implemented between the midterm and final evaluations: • All project interventions (April 2004) • Breastfeeding and nutrition (March 2005) • Malaria (August 2005) • Diarrhea, water/sanitation, pneumonia, EPI and breastfeeding (December 2005) The report for the April 2004 survey included an extensive set of recommendations for project improvements. In addition to these KPC surveys, the project also did studies on BCC, PD Hearth and QA, which are discussed in their respective sections of this report. The project also sought to strengthen the MOHP’s HMIS through training, the donation of 27,250 Health Passports (required permanent child health cards), and the development of a set of supervisory checklists and village-based registry books. The project provided training to 50 MOHP staff in October 2005 on the data collected and reported on in the nine registries used in their facilities. The project has developed a set of supervisory checklist formats for use by the MOH during supervisory visits with the VHCs, TBAs, GMVs, BFSGs, HSAs and traditional healers, however, they have not been tested or institutionalized within the MOHP. It sum, it is doubtful that the MOHP has the capacity to manage anything more than a very elementary HMIS At the time of the final evaluation, it’s own computerized HMIS had not been functioning for the previous six months since the computer the data was stored on crashed and Final Evaluation Report IEF/DHMT Nsanje Malawi 34 C.A.: No: HFP-A-00-02-00027-00 had been sent to Lilongwe for repairs. There was no backup. The district’s HMIS is being run by three HSAs, none of who have experience in database design or management or expertise in health statistics. Further, there is little use of data in decision making by the health facilities. As reported in the project’s SP study of May 2005 found that only 1/8 (13%) of the health facilities surveyed said they were using health information for decision-making. 8. Technical and Administrative Support The project has received technical support from the MOHP, Family Health International and the Malawi College of Nursing for facilitation of the IMCI training; training provided by an independent consultant from South Africa on PD Hearth; and training provided by IEF to the MOHP on BCC. Each of these topics related directly to the project strategies and was timed appropriately to coincide with their implementation. Backstopping for this project has been provided both from IEF headquarters and IEF’s Malawi Country Office in Blantyre. Initially the HQ Child Survival Coordinator and later the HQ Director of Programs provided support through regular email and phone communication: reviewing reports, responding to requests for information, and giving feedback. This was usually provided to the Country Office in Blantyre, which then relayed it to the staff in Nsanje. In addition, HQ staff made annual trips to the project to provide training, technical support, and share new tools and materials. The HQ Director of Finance and Administration made annual trips to the Country Office in Blantyre to review and provide feedback on financial and administrative systems and conduct an audit. The Country Director who is officed in Blantyre provided support to the project through frequent and regular phone communication with the Project Manager, meetings with the Project Administrator during his bi-monthly trips to Blantyre to transact banking business, and site visits to Nsanje (totaling an estimated 10% of his time during the life of the grant) where he participated directly on key activities during the life of the project. 9. Mission Collaboration IEF staff have collaborated with the USAID mission in Lilongwe by sharing key documents and reports with the mission during the life of the project and through their attendance at monthly ‘Synergy Meetings,’ where representatives from other NGOs, donors and government agencies come together to coordinate activities and plans.13 The project has also invited mission staff to attend project evaluations and was briefed after the MTE. A briefing after the FE was cancelled due to conflicts in schedules. The mission has not informed IEF of any instances where it has used project data. 10. Management Lessons Learned • IEF prevented potential morale problems at its conclusion by keeping staff informed throughout the life of the project about the status of funding appeals and making sure they 13 USAID’s office is a four-hour drive from IEF’s Country Office in Blantyre and eight hours from the CS project site in Nsanje. Final Evaluation Report IEF/DHMT Nsanje Malawi 35 C.A.: No: HFP-A-00-02-00027-00 understood the organization’s need for donor support as a prerequisite for jobs to continue. When staff learned from leadership that there were not sufficient funds to continue the project beyond the conclusion of this cooperative agreement, the news was disappointing but was better understood and accepted if they had not been kept informed. • Achieving sustainable improvements in public sector and community capacity in locations that are currently receiving or have recently received humanitarian assistance requires close coordination between the implementing organizations and the public agencies charged with their oversight. Otherwise, they can end up working at cross-purposes, increasing dependency and leading to unintended consequences. • District level MOHP authorities are not always aware of national level policies, especially new or recently changed ones. It is important that program plans be vetted with national level authorities when developing the DIP to ensure project compliance with national policies and plans. • The costs of staff turnover can be high – programmatically, financially and in terms of lost time and momentum. NGOs have to weigh competing factors when hiring staff. There is a preference for hiring people from the project area who are familiar with and used to living in the area. However, in the more isolated areas there tend to be fewer candidates with the necessary technical qualifications and experience. NGOs that have offered salaries significantly higher than what the local market can sustained can be accused of ‘raiding the MOH’, which can undercut the very services the NGOs are trying to improve. Yet, salary is an important factor in recruiting and maintaining a qualified project workforce. • Project plans need to be revisited and reassessed when staffing changes occur in the NGO or local partners. • Having worked successfully with a respected international NGO, on a USAID-funded project, brings with it much credibility when staff leave to go seek employment options elsewhere. In addition to being helpful for a departing employee, this can be an effective incentive for recruiting qualified new hires as well. D. OTHER ISSUES Contributing to Scale: (See section on IMCI above.) Civil Society Development: The project contributed to civil society in Nsanje District through its development and support of BF and PMTCT Support Groups as well as Village Health Committees at the community level. In FGDs the participants stated that they now have a better understanding of health and are more effective users of the services available to them. They also understand their roles and responsibilities as members, and at the time of the FE appeared to be highly motivated to continue their work and take on other projects that will benefit their communities. The project was also successful in involving other sectors (i.e., agriculture and education) and community based organizations (i.e., FAST) in its project activities, through coordination and participation in training, planning, monitoring and evaluation. Finally, the project also sought to address the problems of ‘sexual cleansing’ by enlisting the support of the District Development Committee and the District Assembly on this issue. Final Evaluation Report IEF/DHMT Nsanje Malawi 36 C.A.: No: HFP-A-00-02-00027-00 Equity: This project has actively sought to work with the most disenfranchised members of Nsanje population. A majority of the members of the PMTCT and BF Support Groups and VHCs are women. The project has worked with PLWA and OVCs, as well as adolescents through its HIV/AIDS intervention. E. CONCLUSIONS AND RECOMMENDATIONS At the household level the project saw its most notable achievements in expanded coverage of ITNs, EPI and VCT, and increased use of ORS to treat children experiencing diarrheal disease, as well as a drop in malnutrition in children under-five. More mothers had received SP during their most recent pregnancy and knew effective ways to prevent the transmission of HIV. The project saw little or no change in VAC coverage, care seeking in response to IMCI danger signs, exclusive breastfeeding, or proper feeding of the sick child. Among community volunteers and health workers, the VHCs, GMVs and HSAs are well established in the communities and competent in carrying out their responsibilities. Between half and two-thirds of health workers are demonstrating competence in using IMCI protocols when treating sick children at health facilities. Supportive supervision of the facility-based health workers, the VHCs and the GMVs is now strong, while supervision of the HSAs has diminished in frequency, perhaps due to the increase in their numbers and the difficulties inherent in reaching them since they spend most of their time out in the communities where communication is difficult. While the project increased the capacity of the DHMT in planning, implementing, monitoring and evaluating its services, it was not able to achieve a significant impact in the frequency of stock outs of IMCI drugs at health facilities or the number of health facilities providing daily immunization services. Even though the project did not reach all of its targets and not all of the interventions achieved their full potential, positive improvements were seen on its strategic objective, “Families and caretakers with children under five years of age practice health behavior and seek care from qualified providers,” and each of its four intermediate results: IR1. District organizational effectiveness and management support for quality child care strengthened. IR2. Health provider skills in prevention and management of childhood illness improved. IR3. Availability and accessibility to quality preventative and curative health services increased. IR4. Community participation, ownership, and demand for health services increased. Through the Five Stars exercise, the FE Team concluded that the project had achieved the following summary achievements: 1. The communities have become more actively involved in their own health. 2. The DHMT has increased its capacity to plan, coordinate, supervise and monitor the health care services in Nsanje District. 3. The connection between the health care system and the communities is stronger and deeper due to the committed work of the MOH, volunteers and leaders. Final Evaluation Report IEF/DHMT Nsanje Malawi 37 C.A.: No: HFP-A-00-02-00027-00 4. People appear very committed and motivated to continuing the work of this project. 5. Most importantly, the Final Evaluation Team believes that as a result of this project, fewer women and children are getting sick or dying from disease. In addition, the project also achieved the following: • The spectacles shop and the cost wards, which were conceived and piloted by the project as part of its sustainability strategy, have both been successful. Responsibility for their oversight is now fully vested within the DHMT, which is exploring other income generating strategies. • Nsanje was one of the four of Malawi’s 27 districts to implement comprehensive IMCI (clinical training, drug supply, community training with supportive supervision), a major accomplishment in a challenging setting. • Community mobilization strategies were particularly popular and successful, having established a motivated and coordinated set of community health volunteers, committees and support groups. The project faced numerous constraints and challenges, many of which were beyond the control and scope of this project: • High staff turnover within the project and in the senior levels of the DHMT resulted in some lost momentum. • The shorter duration of the project (four versus five years) and the delayed start of some of the intervention strategies, meant that PD Hearth and the BEHAVE Framework did not reach their full potential. • The district’s history of receiving humanitarian assistance led to untenable expectations from some communities. • The cost of training health workers in IMCI is high and probably not sustainable by the DHMT as the training is currently designed and being implemented in Malawi. • Seasonal migration between Malawi and Mozambique, vaccine stock outs, and the national MOHP opened vial policy probably limited the project’s potential impact on EPI coverage. • Problems in the management of the ITN supply and accounting system, as designed and administered by the MOHP, are jeopardizing the potential benefits of the large number of ITNs available to Nsanje District. The lessons learned and best practices identified by the FE Team included the following: HIV/AIDS: • On sensitive topics such as AIDS, the newly educated can become great educators when they already have credibility in their community and with their peers. The support they need are the facts about the disease and culturally appropriate messages to communicate. • Achieving positive changes in knowledge and behaviors at the community level can take years, especially in isolated, under-resourced locations such as Nsanje, whose people tend to be more reticent about discussing sensitive topics and less amenable to behavior change. Final Evaluation Report IEF/DHMT Nsanje Malawi 38 C.A.: No: HFP-A-00-02-00027-00 Nutrition: • PD Hearth should be avoided in locations that are experiencing acute food shortages, especially when external food aid is being offered for free. • Nutrition programs need to be coordinated at the district and community levels to ensure they are not working at cross purposes. • When selecting an implementation strategy one of the key factors to consider is whether there is sufficient time to have an impact. If not, then the objectives need to be changed or the strategy needs to be adapted or dropped. EPI: • Of knowledge, practice and coverage, the indicators that tend to take the most time and effort to improve, are those related to coverage. The project was able to initiate its work on the EPI intervention earlier than on some of the other interventions and as a result, achieved improvements in coverage, even though not all of the EPI strategies were ultimately successful. In sum, it is important to recognize and allow for sufficient time for these changes to occur, especially at the community level. IMCI: • Developing comprehensive IMCI at the district level is an excellent capacity building exercise, the results of which can have broader implications, impacting the ability to plan, administer and monitor other activities. • IMCI must be implemented comprehensively with clinical training, drug supply, and community components. Supportive supervision must begin immediately after training in order to be successful. • The ultimate solution for ensuring broad and consistent use and coverage of IMCI throughout Malawi is the incorporation of IMCI into the training curricula for all types of health workers who provide care for the sick child and creating in service training that can be sustained by the DHMT themselves. Community Mobilization: • Volunteer and committee responsibilities need to be clearly defined and communicated widely at the start and throughout the life of the project to avoid unrealistic expectations and confusion. This is especially critical when there are many different types of volunteers, committees and groups, such as in Nsanje, and there their responsibilities overlap. • Many of the volunteers are motivated by a wish to serve their communities and the respect they earned through their participation. This incentive is something even the poorest, most isolated communities can provide. • There tends to be fewer turnovers in community volunteers than in MOH staff (including the HSAs), making volunteer support an excellent long-term investment for achieving primary health care goals. Final Evaluation Report IEF/DHMT Nsanje Malawi 39 C.A.: No: HFP-A-00-02-00027-00 IEC/BCC: • When there are several types of volunteers, each of which specializes in a relatively narrow topic, it is easy for a project to miss opportunities to educate and promote behavior change. • It is important to involve the local leadership in project planning and implementation, especially those related to sensitive topics, such as HIV/AIDS. They bring the necessary credibility to the project’s health education messages. Capacity Building of the MOH • A good working relationship between the partners is critical to the success of a project. This takes time and regular contact to develop or redevelop when there is a change in staffing. Ideally this relationship needs to be established prior to starting work at the community or health facility levels. • Creating supervisory zones was beneficial to organizing the supportive supervision system. • Per the DHMT: New NGOs desiring to do health programs in Nsanje need to first coordinate with the DHMT to ensure that their plans are consistent with the District Implementation Plan. Sufficient time needs to be dedicated to this process. NGO programs need to be measurable and evidence based to work in Nsanje District. Capacity Building of Health Facilities and Health Workers: • The role of the HSAs is very important to the continued success of project activities, as they are the face of the MOH in the communities. It is important that the MOH and communities continue to identify sustainable incentives and ways to reward the HSAs. Training: • Traditional didactic training by itself is not sufficient in building organizational capacity. It needs to be complemented with on-going OTJ training, feedback and supportive supervision before changes in practices and organizational culture can occur and be institutionalized. Project Management: • IEF prevented potential morale problems at its conclusion by keeping staff informed throughout the life of the project about the status of funding appeals and making sure they understood the organization’s need for donor support as a prerequisite for jobs to continue. When staff learned from leadership that there were not sufficient funds to continue the project beyond the conclusion of this cooperative agreement, the news was disappointing but was better understood and accepted if they had not been kept informed. • Achieving sustainable improvements in public sector and community capacity in locations that are currently receiving or have recently received humanitarian assistance requires close coordination between the implementing organizations and the public agencies charged with their oversight. Otherwise, they can end up working at cross-purposes, increasing dependency and leading to unintended consequences. • District level MOHP authorities are not always aware of national level policies, especially new or recently changed ones. It is important that program plans be vetted with national level authorities when developing the DIP to ensure project compliance with national policies and plans. • The costs of staff turnover can be high – programmatically, financially and in terms of lost time and momentum. NGOs have to weigh competing factors when hiring staff. There is a Final Evaluation Report IEF/DHMT Nsanje Malawi 40 C.A.: No: HFP-A-00-02-00027-00 preference for hiring people from the project area who are familiar with and used to living in the area. However, in the more isolated areas there tend to be fewer candidates with the necessary technical qualifications and experience. NGOs that have offered salaries significantly higher than what the local market can sustained can be accused of ‘raiding the MOH’, which can undercut the very services the NGOs are trying to improve. Yet, salary is an important factor in recruiting and maintaining a qualified project workforce. • Project plans need to be revisited and reassessed when staffing changes occur in the NGO or local partners. • Having worked successfully with a respected international NGO, on a USAID-funded project, brings with it much credibility when staff leave to go seek employment options elsewhere. In addition to being helpful for a departing employee, this can be an effective incentive for recruiting qualified new hires as well. The following recommendations were developed by members of the FE Team, intended for IEF: • Project staff should meet with GOAL (another international PVO doing nutrition programs in Nsanje) to discuss what the project has done on PD Hearth, with a view towards transferring this strategy over to them. • As one of the four of Malawi’s 27 districts doing comprehensive IMCI, IEF and the Nsanje DHMT need to focus their remaining time on analyzing the results of their work on IMCI and sharing it with the national MOH, the other NGOs working in the health sector, and the donors. • Options for cutting the costs of the eleven-day IMCI training without jeopardizing the quality of care or the certification requirements need to be brainstormed, tested, and analyzed. The results of this effort need to be shared and debated with national and international authorities at the policy levels. The following recommendations were developed by members of the FE Team, intended for the Nsanje DHMT: • Attention needs to be directed towards training and placing more VCT Counselors in the district so the growing need and demand can be met, especially in the more rural areas. • The sale of ITNs should be stopped immediately, until an effective and transparent accounting and control system can be designed and put into place. This system needs to be designed by a trained accountant or an organization with relevant experience and expertise. (NOTE: This recommendation does not pertain to the free distribution of ITNs noted above.) • The shortfalls of the ITN accounting system are due primarily to inadequate design. Even though some individuals probably benefited financially from these design flaws, the system that is in place is not sufficiently attuned and managed to identify these individuals or to quantify any amount taken. Therefore, it suggested that punitive measures not be pursued against individuals or committees at this time. • The DHMT needs to monitor the percentage of its health workers responsible for case management of sick children trained in IMCI. Once this rate drops below a certain point a new training needs to be planned and budgeted for. This will need to be driven by “IMCI champions’ at the local and national levels. • The HSAs and CHVs play very important roles in extending the reach of MOH services to the villages. It is important to continue to track the number of active HSAs and CHVs by location so their numbers can be replenished when necessary. Final Evaluation Report IEF/DHMT Nsanje Malawi 41 C.A.: No: HFP-A-00-02-00027-00 • Where feasible, the practice of having successful VHCs visit new or less motivated VHCs should be continued. This is an incentive for the stronger VHCs and a great learning opportunity for the others. • Incorporate the IMCI danger signs into all volunteer and health worker training curricula and make sure they are being reinforced during supervision. The project planned two sustainability strategies: a spectacles shop and cost wards, both of which have been successfully setup at the Nsanje District Hospital and are now running entirely under the supervision of the DHMT. A formal sustainability phase-over plan has not yet been prepared, however, the project has been discussing and acting on phase-over of project activities since 2004 in concert with the DHMT. The primary challenge the local partners face in sustaining project activities at their current levels is the need for on-going support and development of the workforce – both health workers and community volunteers. IEF will use the best practices and lessons learned from the project in several ways. The IEF headquarters will assist the staff to develop and present a presentation of the results, lessons learned, and recommendations to the Ministry of Health headquarters, the National IMCI Coordinator, and the USAID Mission during the next program visit to Malawi. IEF also discussed with staff writing and submitting an article to a Malawi journal for publication on an important aspect of the project such as IMCI. At the IEF headquarters the final report and findings will be shared with staff and presented to the Board of Directors. Short news articles will be prepared to post on the IEF web site and IEF will continue to participate in the CORE Group activities. The project strategies that pose the greatest potential for scale-up or expansion are the cost wards and spectacle shops. Both strategies have been successful at the district level and as a result, the DHMT leadership has been more than willing to take over the management and support responsibilities. This experience has spurred the DHMT to explore other revenue generating strategies that, like these two, do not create disincentives or diminish access to or quality of the health care services available in the district. IEF has also planned expansion of refractive error screening and provision of glasses to other districts subject to funding availability. IEF has also sought to initiate a national debate on how best to implement IMCI training so it can be financed and administered by the districts. As currently designed, the eleven-day training is costly, time-consuming, and beyond the financial means of the majority of districts in Malawi. With the high turnover in trainees many districts such as Nsanje are experiencing, many will need to do regular repeat trainings as untrained workers replace trainees. Moreover, those persons trained in IMCI might not be able to use their skills in case management if the district does not support the practice or implement comprehensive IMCI. Such a situation is a lost opportunity and a waste of resources. Upon review of this project’s experience establishing comprehensive IMCI in Nsanje District, members of the FE Team felt that the IMCI training could be done in a streamlined, less-costly manner without jeopardizing quality and should be explored further at the policy level. Final Evaluation Report IEF/DHMT Nsanje Malawi 42 C.A.: No: HFP-A-00-02-00027-00 ATTACHMENTS Attachment A Evaluation Team Members and Titles • Mr. Mhango, Deputy District Health Officer, MOH • Beatrice Thaulo, District Nursing Officer, MOH • Mathew Kalaya, District Environmental Health Officer, MOH • Kenneth Ndau, District MCH Coordinator, MOH • Mwachumu Reukia, IMCI Coordinator, MOH • George Matope, Malaria Coordinator, MOH • Steven Nkuma, District AIDS Coordinator, MOH • Richard Martin, Senior Health Surveillance Assistant, MOH • Maclean Sosono, Executive Director, Friends of AIDS Trust (FAST) • Pius Nakoma, Nsanje District Coordinator, Family Health International (FHI) • Edna Tembo, Project Manager and Acting MCH and HIV/AIDS Coordinator, IEF • Frank Chola, Project Administrator, IEF • George Meckiseni, Monitoring and Evaluation Advisor, IEF • Hopson N. Tiringamawa, IMCI Coordinator, IEF • Evalister Nyirenda, Administrative Assistant, IEF • Geoffrey Ezepue, Country Director, IEF • John Barrows, Director of Program, IEF/HQ • Garth Osborn, External Consultant and FE Team Leader Final Evaluation Report IEF/DHMT Nsanje Malawi 43 C.A.: No: HFP-A-00-02-00027-00 Attachment B Final Evaluation Methodology The final evaluation of International Eye Foundation’s Child Survival Project in Nsanje District, Malawi was implemented during the summer of 2006 by a team of 19 individuals from FHI, FAST, the district MOH, and IEF’s headquarters, Malawi Country Office, and Nsanje field program, as well as an external consultant who acted as the team leader. Per USAID, the purpose of final evaluations of CSHGP projects is to determine: • Whether project goals and objectives have been met. • The Effectiveness of the technical approach. • Lessons learned. • Strategies for disseminating lessons learned. Fieldwork was completed between July 5 and 17, 2006 as detailed in the following schedule and was done in two phases: data collection and review of findings, discussed below. Date Time/Activity Wednesday, July 5 • Arrival in Nsanje. • Planning meeting with staff Thursday, July 6 (National Holiday) • KPC/HFA data cleaning • Start interviews with project staff. • Review HMIS Friday, July 7 • Orientation Meeting • Interviews with FHI and FAST • Prepare KPC/HFA results for presentation • Select communities for site visits and split up the FE Team Saturday, July 8 • Presentation and discussion on the KPC/HFA results • Training on FGDs and interviewing Sunday, July 9 • Finalization and copies made of community FGD questionnaires • Interviews with project staff Monday, July 10 • Day one of community visits (See detailed table below.) • Debriefing and troubleshooting on day one community visits Tuesday, July 11 • Day two of community visits • Interviews with district MOH staff Wednesday, July 12 • Day three of community visits • Interviews with district MOH staff Thursday, July 13 • Debriefing and discussion on community visits. Friday, July 14 • Discussion on findings, recommendations and lessons learned Final Evaluation Report IEF/DHMT Nsanje Malawi 44 C.A.: No: HFP-A-00-02-00027-00 Date Time/Activity Saturday, July 15 • Discussion on findings, recommendations and lessons learned • Five Stars exercise Sunday, July 16 • Preparation of field report for presentation. Monday, July 17 • Wrap up and presentation of project findings, recommendations and lessons learned. • Travel back to Blantyre. Data Collection: Six data collection methods were used during the final evaluation to collect both quantitative and qualitative information about the project: 1. Review of Project Documents 2. Knowledge Practice Coverage (KPC) Survey 3. Integrated Health Facility Assessment 4. Focus Group Discussions (FGD) 5. Informational Interviews 6. Review of Health Management Information System (HMIS) Data To ensure that all the relevant topics were covered with each respondent group, the following table was developed and used to design and adapt the final evaluation questionnaires. Final Evaluation Report IEF/DHMT Nsanje Malawi 45 C.A.: No: HFP-A-00-02-00027-00 RESPONDENTS Moms of <5’s VHCs ITN Committees BF Support Groups PMTCT Support Groups GMVs District Authorities DHO/DDHO EDHMT Members Dist Hospital Staff Health Facility Staff HSAs Other NGOs IEF Project Manager IEF M&E Advisor IEF IMCI Advisor IEF Project Admin. IEF Country Director IEF Program Director USAID TOPICS: ARI/CDD X X X X X X X X X X X X Malaria X X X X X X X X X X X X X - ITNs X X X X X X X X X X X X X X IMCI X X X X X X X X X X X X X - HW X X X X X X X X X X X - HH/C X X X X X X X X X X X X Nutrition X X X X X X X X X X X X X X - PD Hearth X X X X X X X X X X X X X X - Breastfeeding X X X X X X X X X X X X X X X - VAC X X X X X X X X X X X X X X HIV/AIDS X X X X X X X X X X X X X X X X EPI X X X X X X X X X X X X MOH Training/ Supervision X X X X X X X X X X X BCC/IEC X X X X X X X X X X X X X X X X X X MOH Capacity X X X X X X X X X X X X X X X X Sustainability X X X X X X X X X X X X X X X X X X X X MIS/Data use X X X X X X X X X X X X X X X X X X Project Planning X X X X X X X X X X X X X Personnel X X X X X X X X X X Finance/Budget X X X X X X X X X X X X X Logistics X X X X X X X X X X X X X X X X X X TA X X X X X X X X X X X X X X X X X DATA COLLECTION METHODS Document Review X X X X X X X X X X X X X X X X X X X KPC X IHFA X X X X FGDs X X X X X X X Interviews X X X X X X X X X X X X HMIS Review X X X X X X X X X X X X X X X X Review of Project Documents: The following documents were provided by project staff, for review by members of the FE Team: • Final Evaluation Guidelines (October 2005) • Detailed Implementation Plan (2003) • Annual Reports (2002, 2003 and 2005) • Midterm Evaluation Report (2004) • Sustainability Discussion Report (2004) • Project Quarterly Reports (4th quarter of 2003, 1st & 4th quarters of 2004, and 2nd and 4th quarters of 2005) Final Evaluation Report IEF/DHMT Nsanje Malawi 46 C.A.: No: HFP-A-00-02-00027-00 • PD Hearth Evaluation Report (2005) • PD/Hearth Approach Training Report (2004) • Organizational Capacity Assessment Results (2004) • LQAS Baseline Survey Report (2004) Knowledge Practice Coverage (KPC) Survey: A KPC survey was conducted by project staff in June and July 2006 targeting mothers of children under the age of two, including 101 questions and covering each of the project’s CS interventions and the 13 Rapid CATCH indicators. Thirty-cluster sampling was used with a sample size of 300. The data was entered into EPI INFO for Windows, cleaned and tabulated by project staff. The results of the project and Rapid CATCH indicators were shared with the project staff and FE Team members for analysis and discussion. Integrated Health Facility Assessment: The four components of the BASICS IHFA were implemented in June and July 2006 at 14 health facilities, including observation of 165 children who were brought for care of fever, pneumonia and/or diarrhea. The results were entered into EPI INFO for Windows, cleaned and tabulated by project staff. The results for the project indicators were shared with the project staff and FE Team members for analysis and discussion. Focus Group Discussions (FGD): The FE Team first established a set of criteria and characteristics for selecting communities that would help to ensure as representative and inclusive a picture of Nsanje District as possible. These included geographical distribution across the district, the presence/lack of an ITN committee, the presence/lack of a health facility, and as broad a range as possible in populations. Twelve communities were selected that fit these criteria: six zones were represented; six communities had an ITN committee and six did not; six communities had a health facility and six did not; and the population range was 312 – 3,256. Date of Visit Community Name Zone Presence of an ITN committee? Presence of a health facility/type? Population Chataika No HC 952 Nkhudche Boma Yes None 995 Malemia No HC 505 Monday, July 10 Chimtombwe South Yes None 312 Chisi No HC 1195 Jonikisi Tengani Yes None 442 Jambo No HC 1,827 Tuesday, July 11 Natchitchi Bangula Yes None 498 Mwanabvumbe No HC 3,256 Alufazema Makhanga Yes None 401 Kandodo No Mission-run Hospital 650 Wednesday, July 12 Sandramu Mlolo Yes None 180 The FE Team then developed FGD questionnaire guides for each of these respondent groups based on the questionnaires used in the midterm evaluation and the results of the final evaluation KPC survey and IHFA. A brief refresher training was provided to the FE Team members, the questionnaire guides were reviewed, and final adjustments made. Final Evaluation Report IEF/DHMT Nsanje Malawi 47 C.A.: No: HFP-A-00-02-00027-00 The FE Team interviewed or led FGDs with each of the following respondent groups: • 233 mothers of children under five in twelve communities. • 9 health facility workers (Seven Nurses and two Medical Assistants) • 7 Growth Monitoring Volunteers • 19 Health Surveillance Assistants • 106 Village Health Committee members from 12 VHCs (58 women and 48 men) • 46 ITN Committee members from seven ITN Committees (24 women and 22 men) • 20 Breast Feeding Support Group Members • 14 PMTCT Support Group Volunteers After the community visits, the FE Team met to review their notes from the FGDs and share them with the rest of the FE Team. (See Attachment E for the FGD Report and Questionnaires.) Informational Interviews: FE Team members prepared questionnaire guides and interviewed each of the 19 individuals listed in Attachment C. Review of HMIS Data: FE Team members reviewed data collection tools and reports as noted in Section VI. G. Information Management. Review of FE Results: After the data was collected and shared amongst its members, the FE Team met as a group to discuss these results and come to consensus on a draft set of accomplishments, challenges, lessons learned and recommendations. (See Attachment D.) This analysis phase ended with the Five Stars Exercise, the results of which are summarized in the following project achievements: 1. The communities have become more actively involved in their own health. 2. The DHMT has increased its capacity to plan, coordinate, supervise and monitor the health care services in Nsanje District. 3. The connection between the health care system and the communities is stronger and deeper due to the committed work of the MOH, volunteers and leaders. 4. People appear very committed and motivated to continuing the work of this project. 5. Most importantly, the Final Evaluation Team believes that as a result of this project, fewer women and children are getting sick or dying from disease. Finally, the fieldwork portion of the FE concluded with a presentation on the preliminary results to local key stakeholders in Nsanje. Final Evaluation Report IEF/DHMT Nsanje Malawi 48 C.A.: No: HFP-A-00-02-00027-00 Attachment C Persons Interviewed and Contacted • Medson Semba, District Health Officer, MOH • Mr. Mhango, Deputy District Health Officer, MOH • Kenneth Ndau, District MCH Coordinator, MOH • Mathew Kalaya, District Environmental Health Officer, MOH • Mwachumu Reukia, IMCI Coordinator, MOH • Mr. Phuma, District AIDS Coordinator, MOH • George Matope, Malaria Coordinator, MOH • Hanerick Kaufu, Data Entry Clerk (HMIS), MOH • Rose Kaliza, MCH/Nutrition Coordinator, MOH • Alexander Chikwangusala, Ophthalmic Medical Officer, MOH • Amos Nailosi, District AIDS Coordinator, District Assembly • Maclean Sosono, Executive Director, Friends of AIDS Trust (FAST) • Pius Nakoma, Nsanje District Coordinator, Family Health International (FHI) • Edna Tembo, Project Manager and Acting MCH and HIV/AIDS Coordinator, IEF • George Meckiseni, Monitoring and Evaluation Advisor, IEF • Hopson N. Tiringamawa, IMCI Advisor, IEF • Frank Chola, Project Administrator, IEF • Geoffrey Ezepue, Country Director, IEF • John Barrows, Director of Programs, IEF/HQ In addition, focus group discussions were held with the following: • 233 mothers of children under five in twelve communities. • 9 health facility workers (Seven Nurses and two Medical Assistants) • 7 Growth Monitoring Volunteers • 19 Health Surveillance Assistants • 106 Village Health Committee members from 12 VHCs (58 women and 48 men) • 46 ITN Committee members from seven ITN Committees (24 women and 22 men) • 20 Breast Feeding Support Group Members • 14 PMTCT Support Group Volunteers Final Evaluation Report IEF/DHMT Nsanje Malawi 49 C.A.: No: HFP-A-00-02-00027-00 Attachment D SUMMARY OF PROJECT SUCCESSES, CHALLENGES, LESSONS LEARNED, AND RECOMMENDATIONS Activities Successes Challenges Lessons Learned/ Recommendations IR1. District Organizational Effectiveness and management support for quality child care strengthened. • Joint planning • Joint training • Joint supervision • Checklists developed • Cost wards and specs shop • Provided feedback to DHMT • Good relations between IEF and MOH • Cost wards and spectacle shop are open and being used. • DHMT-EDHMT-NGO communication and collaboration improved • Improved supervision from MOH Coordinators to the health facilities. • Increased motivation on the DHMT to involve stakeholders in planning. • Large and growing number of service providers to coordinate in the district. LL: New NGOs desiring to do health programs in Nsanje need to first coordinate with the DHMT to ensure that their plans are consistent with the District Implementation Plan. Sufficient time needs to be dedicated to this process. LL: NGO programs need to be measurable and evidence based to work in Nsanje. IR2. Health Provider skills in prevention and management of childhood illness improved. 2.a. Increase inter-sector understanding of HW and HH/C IMCI. • Training of Area Dev’t Committees, District Exec Council, Traditional Authorities and Ag Extension Workers in CIMCI. • District IMCI Coordinating Group formed. • Involvement at national level IMCI coordination. • Nsanje is recognized as a leading IMCI district in Malawi. • There is a broad based understanding of IMCI at the district level. • Good working relationships have been established between Government and NGOs. • The support of traditional leaders. • Initial expectations were high from the community, Ag Extension workers, and the Government. LL: Establishing good coordination with the MOH, other government ministries, and NGOs working in the district is a critical first step to program design and later implementation. 2.b. Increase the skills of health facility providers in HW and HH/C IMCI. • Joint planning w. MOH. • Joint training w. MOH • Joint supervision w. MOH • IEC activities • Supplies provided (i.e., referral forms, health passports, ORS, etc.) • Referral system re￾introduced. • The target for health workers trained in HF IMCI (96) was reached. • Increased client satisfaction has led to increased demand for MOH services. • Improved referral and record keeping. • Health facility staff have become more welcoming of supervision as it has become more • High MOH staff turnover. • Drug stock outs. • Initial fears of supervisory visits by health facility staff. • Lack of feedback on referrals. • Concern about the impact on access to care at health facilities while health workers are at trainings. • The cost of HW IMCI R: The district needs to track the number of health workers trained in IMCI so future trainings can be appropriately timed, planned and budgeted. R: Options need to be explored for cutting the high costs of IMCI training, through efficiencies and alternative scheduling. Any changes need to be raised with the national level Final Evaluation Report IEF/DHMT Nsanje Malawi 50 C.A.: No: HFP-A-00-02-00027-00 Activities Successes Challenges Lessons Learned/ Recommendations supportive. training is high and might not be sustainable. MOH to ensure continued compliance with the national/international standards of IMCI. R: Since Nsanje is one of the first districts in Malawi to initiate comprehensive IMCI, IEF and the DHMT should pursue opportunities to share the experiences and lessons learned at the national level. R: Exploring the reasons for the drug stock outs and pursuing solutions should continue to be a high priority of the DHMT. 2.c. Increase the skills of HSAs and CHVs in HH/C IMCI. • Joint Training • Joint Supervision • IEC/BCC Activities • VHC Exchanges • Good coordination at the community level (HSAs-community) • Increased capacity for IEC, recognize/refer for danger signs, community mobilization and problem solving. • Increased demand for health facility services. • Greater community involvement in health issues. • Increased acceptance of HSAs by communities • Communities understand the roles of the HSAs and CHVs. • As some HSAs move out of the district or change jobs, access to their services diminishes until they can be replaced. • Supervising the large numbers of volunteers and HSAs • Initial high expectations from volunteers which can lead to some attrition after training. R: The HSAs and CHVs play very important roles in extending the reach of MOH services to the villages. It is important to continue to track the number of active HSAs and CHVs by location so their numbers can be replenished when necessary. R: Supportive supervision of the volunteers needs to be made a high priority for the HSAs. LL: It is important to invite local leadership participation in project activities, especially those related to sensitive topics, such as HIV/AIDS. They bring the necessary credibility. LL: The responsibilities of volunteers need to be clearly defined and communicated at the start of volunteer recruitment and training activities to avoid unrealistic expectations. Final Evaluation Report IEF/DHMT Nsanje Malawi 51 C.A.: No: HFP-A-00-02-00027-00 IR.3. Availability and accessibility to quality preventive and curative health services increased. 3.a. Strengthen under five and ANC services. • Training provided • Procured health profiles • Joint supervision with checklists. • Campaigns (VAC, measles, etc.) • Supplies donated (VAC, etc.) • Immunization coverage is high. • Improved community mobilization skills • GMVs can weigh and plot on charts • More clinics provide ANC • Supportive supervision has created a “Team Spirit” among the HSAs. • Opened vial policy has limited the number of clinics that can provide 7 day/wk EPI. • Vaccine stock outs. • Cultural beliefs that impede access to ANC. • Seasonal migration between Malawi and Mozambique complicating EPI and ANC coverage. LL: Seasonal migration needs to be taken into account when planning and setting targets for EPI strategies. 3.b. Increase availability of malaria ITN/IPT services. • Training and supervision • Promoted, provided and distributed ITNs • ITN Committees established • Net dipping • IPT assessment done • Increased demand for ITNs. • 64,000 ITNs distributed/sold. • Increased use of ITNs for children. • Management of supply chain • Stock outs of nets at district level. • Price variation has potential for confusing communities and leading to disincentives that could undermine demand. R: The distribution of ITNs should be stopped temporarily until a transparent accounting system can be put in place that will track the transfer of ITNs and cash throughout the system. R: Realistic targets need to be identified that will rationalize the ITN distribution system, helping to ensure the greatest possible coverage for those in need. 3.c. Improve nutrition services by adopting PD Hearth strategy • Orientation provided to district on PDI/Hearth strategy. • Training • Positive Deviance Inquiry • Hearth groups in 4 villages • District coordination through Targeted Nutrition Program (TNP) meetings. • IEC on breastfeeding • VAC distribution • People motivated by PD Hearth process and results achieved. • Improved complementary feeding and EBF. • Competing NGO strategies on food distribution. • Late start for Hearth limited the project’s ability to expand. LL: Nutrition programs need to be coordinated at the district and community levels to ensure they are not working at cross-purposes. LL: PD Hearth requires significant resources (time and intensity of effort) to succeed. R: The MOH should consider implementing PD Hearth in more areas, based on the success to date. Final Evaluation Report IEF/DHMT Nsanje Malawi 52 C.A.: No: HFP-A-00-02-00027-00 3.d. Expand HIV AIDS prevention, testing and counseling services. • HF and community training • Supervision and Monitoring • Support of VCT at district level. • Health education through “Open Days,” peer educators, PLWA support groups, traditional authorities (TAs), drama groups, bicycle drivers, commercial sex workers, etc. • Some harmful traditional practices (e.g. ‘sexual cleansing’) have been lessened • There is an increased willingness to discuss and address HIV/AIDS in the district. • There is an increasing demand for VCT. • Some traditional leaders have taken a visible role in educating their communities on HIV. • Traditional taboos about discussing AIDS and sex. • High risk factors: poverty, migration, and fear/denial. • VCT counselor shortage is resulting in unmet demand. • Stock outs of reagents for VCT. R: Attention needs to be directed towards training and placing more VCT Counselors in the district so the demand can be met. LL: Community leaders need to be enlisted as advocates, especially on sensitive topics such as HIV/AIDS. The educated can become the educators. R: Exploring the reasons for the stock outs and pursuing solutions should continue to be a high priority of the DHMT. IR4. Community participation, ownership, and demand for health services increased. 4.a. Improve community mobilization and support to HSAs and CHVs prevention and promotion activities. • HSAs trained and supervised • VHCs trained and supervised • GMVs, BFSGs, PMTCT groups trained and supervised. • 514 VHCs reactivated or formed. • Trained in CIMICI: 184 HSAs, 514 VHCs, 400 GMVs, 173 TBAs, 80 MOH support staff, and 162 drug vendors. • Training in CIMIC: Area Dev’t Committees, District Exec Council, Traditional Authorities and Ag Extension Workers. • Other community members (drug providers, “bicycle riders”, “free lancers” trained in health education messages. • Inter-connectedness between HSAs, VHCs, volunteers with HFs is strengthened and communities are empowered. • Access to parts of the project area are difficult to access during part of year due to bad roads and bridges. • Competing expectations in community due to different NGO and government activities and methods of work, e.g., food distribution, free ITNs. • Initially, the lack of clarity in roles/responsibilities of volunteers led to some confusion and attrition due to unrealistic expectations. LL: It is important to first establish a common approach to project planning and implementation amongst the MOH and the NGOs prior to working with the HSAs and CHVs at the community level. LL: Community and volunteer expectations need to be understood and discussed during recruitment and beyond. R: Where feasible, the practice of having successful VHCs visit new or less motivated VHCs should be continued. This is an incentive for the stronger VHCs and a great learning opportunity for the others. LL: Many of the CHVs are motivated by a wish to serve their communities and the respect they will earn by participating. This incentive is something even the poorest, most isolated communities can provide. Final Evaluation Report IEF/DHMT Nsanje Malawi 53 C.A.: No: HFP-A-00-02-00027-00 4.b. Increase access to appropriate and quality care and information from trained CHVs. • Mobile U5 clinics and catch up campaigns • CIMCI Training • IEC/BCC on danger signs through home visits, drama groups, festivals, “open days” • Referral system established • ITN committees formed • PD Hearth • Improved community practices, e.g., ORT, danger sign recognition • Increased demand for services. “The community will work with you if you empower them” • Increased immunization coverage • Changed cultural practices, e.g., sexual cleansing and willingness to seek health services. • Increased availability of ITNs through ITN Committees • Physical access by road and bridges difficult during part of year. • Limited resources (manpower, supplies). • Seasonal migration to and from Mozambique. • Family decision making on seeking health services, e.g., limited money and religious beliefs. R: The role of the HSAs is very important to the continued success of project activities, as they are the face of the MOH in the communities. It is important that the MOH and communities continue to identify incentives and ways to reward the HSAs that are sustainable. 4.c. Improve district communication skills using BEHAVE framework. • Trained staff in BEHAVE framework • Field tested application of BEHAVE framework to HIV/AIDS high risk groups, • Trained community, e.g., fish buyers, sex workers, bicycle riders • Trained HSAs and VHCs • Distributed materials and posters • Organized successful gathering of traditional authorities and leaders to discuss changing harmful practices, e.g., “Kupita Kufa “sexual cleansing” and promotion of VCT • Strong cultural belief systems that can conflict with positive health care practices. LL: Traditional beliefs and practices can be changed through dialogue with traditional authorities. R: Continue use of BEHAVE framework methods and the process and use of behavioral analysis and incorporate into IEC activities. Final Evaluation IEF/DHMT Nsanje Malawi 54 C.A.: No: HFP-A-00-02-00027-00 Attachment E Summary Results from Community Visits, Focus Group Discussions and Key Informant Interviews, CS/Malawi Final Evaluation, July 2006 Health Facility Staff Nine health facility staff were interviewed, including seven nurses and two medical assistants. 1. What are the constraints you face in practicing IMCI case management? • Demand for care at health facilities has increased, resulting in larger caseloads for the limited number of current staff. • Stock outs of necessary drugs and supplies. • Staff report they are not receiving feedback on their referrals. 2. What support do you need to be able to continue practicing IMCI case management? The staff request: • More supervision. • More training and refresher courses. • Increased supply of drugs and supplies to meet the increased number of clients. 3. Which types of cases are referred to your facility from HSAs and from the community? How do you track these referrals to your facility? How many referrals do you receive every month and has there been any change in the number over the past year? The number of referrals from the communities is increasing. The most frequent health problems are pneumonia, malaria, and malnutrition. They are not tracking these referrals. 4. How many referrals did you make during the previous three months? Do you receive a report back on the referrals you make? They are not tracking referrals made and they do not receive feedback on them. 5. What were the most common problems in making referrals? Most of the clients lack transportation or the money to pay for it. The lack of referral forms and feedback are also considered impediments to making referrals. 6. How can the timeliness, monitoring, and accuracy of the referrals be improved? • Develop transportation options. • Provide feedback to the health facilities making the referrals. • Several of the health facility radios used for making referrals are broken and need repair. Final Evaluation IEF/DHMT Nsanje Malawi 55 C.A.: No: HFP-A-00-02-00027-00 7. What changes have you seen in the quality of services delivered by HSAs, GMVs and other volunteers in the last year? Can you provide an example? • The HSAs are competent to assess and refer sick children when necessary. Their attitudes have improved too. Health Surveillance Assistant Nineteen HSAs (12 males and seven females) were interviewed who were working in eleven different locations/health facilities. 1. What are you most proud of as an HSA working in this community? • “I refer people to the health facility because we have a better understanding of danger signs and we are better organized in the community now.” • “I have a better relationship with the village health committee; we are more effective and I can provide better supervision.” • “I can educate about danger signs – what can be done at home and what needs to be done at a health facility.” 2. Can you provide a story of a positive experience you have had with CIMICI in your community? • The community is empowered: A mother brings her sick child to the HSA and tells him he has blood in stool. “What do I do?” Child is referred to health facility for treatment. • The HSAs are more effective, confident, and credible: A HSA identifies an outbreak of diarrhea, goes to the health facility to get ORS and chlorine to bring back to the community for distribution. • The community is better able to practice prevention: Seeking ITNs and if there is a stock out they request more from the hospital. 3. What are the constraints you face in practicing HH/C IMCI? • Community expectations for free food and allowances for training. • Some cultural and religious beliefs promoted by traditional healers and churches. • Some VHCs and CHVs have not been trained and are reluctant to organize. • NOTE: Lack of transportation was not mentioned as a constraint by the HSAs. 4. What support do you need to continue practicing HH/C IMCI? • Training for HSAs and VHCs. • Materials, supplies and some mention of allowances. • Transport and bicycle spare parts. • Increase the number of HSAs – wide variation in the number of HSAs across the district. Final Evaluation IEF/DHMT Nsanje Malawi 56 C.A.: No: HFP-A-00-02-00027-00 5. How can the timeliness, monitoring, and accuracy of the referrals be improved? • Improve communication: mobile phones. • Improve transport: Bicycle ambulances. • Improve coordination: o Within communities: VHCs to HSAs. o Between organizations: NGOs, MOH, Churches, etc. • Improve referral system: o More forms. o Recognition/acceptance 6. In your opinion, how can the distribution of ITNs be improved in the community? • The ITNs need closer oversight and inventory control. Their use needs to be monitored. • The management and sale of the nets and the use of the revenues are not transparent or managed consistently from community to community. • The supply is not consistent: “We need to make ITNs available all the time.” • Rumor that many HSA Supervisors are bypassing the regular HSAs in order to sale the nets themselves for personal profit. Village Health Committees (VHC) Focus group discussions were held with 12 VHCs, which included 58 women and 48 men in total. 1. What are you most proud of as a member of this committee? VHC members valued their position as community leaders; their ability to assess and refer patients; and their contribution to disease prevention through the sharing of health education messages. 2. How often does your committee meet? Do you keep written records of your meetings? All of the VHCs report meeting monthly, at a minimum, and most were maintaining some form of written records of their meetings that included lists of attendees and topics covered. 3. What was your motivation for joining the committee? Members from each of the VHCs said they were appointed by the chief and other community members. Many were motivated by altruism and the opportunity to learn and educate about health. 4. In your opinion, how can the quality and coverage of health services be improved? Final Evaluation IEF/DHMT Nsanje Malawi 57 C.A.: No: HFP-A-00-02-00027-00 • Pressure needs to be exerted on health facility staff to improve their performance. (One health worker had been found drinking on the job and was confronted by community members.) • Some health facilities close between work shifts, which creates problems when patients come from longer distances or need urgent care. One VHC said they could help plan the timing of clinic hours so they best meet the needs of the community. • Request that the HSA Supervisor refill a vacated HSA position. 5. Who comes to supervise you in completing your activities? How often do you receive supervision and how has this support been helpful to you? The VHCs report receiving supervisory visits from their HSAs at least once a month, at a minimum. The HSAs provide encouragement, refresh on training content, and deliver supplies. 6. How do you help and support the GMVs, BFSGs, PMTCTGs, and ITN Committee members? The HSAs organize joint health activities that involve the volunteers and community committees, such as ITN promotion, growth monitoring and health education. They also hold joint meetings for planning and troubleshooting. 7. Can you provide examples of how the GMVs, BFSGs, PMTCTGs, and ITN Committees have been able to change health practices in the community? Increases in breastfeeding, use of ITNs, and use of modern family planning methods, as well as a decrease in the traditional practice of ‘sexual cleansing.’ 8. What are your plans for the next year? Nearly all of the VHCs were able to articulate specific future plans, including: • Construction of latrines and sanitation activities • Starting community gardens. • Continuing their health education and community mobilization activities. Village ITN Committees The FE Team interviewed 24 women and 22 men who were members of seven Insecticide Treated Net (ITN) Committees 1. What are you most proud of as a member of this committee? • They are most proud of the nets they have been able to sell in their communities and how that has contributed to the decline in malaria cases. • The commission they receive is also appreciated. Final Evaluation IEF/DHMT Nsanje Malawi 58 C.A.: No: HFP-A-00-02-00027-00 2. How often does your committee meet? Do you keep written records of your meetings? All of the ITN committees interviewed reportedly meet at least once a month, however, less than half (3/7) of these committees were able to produce any written records of their meetings or the nets they had received/distributed. 3. What was your motivation for joining the committee? Most of the members were placed on the committee by the chief and other members of the community. 4. To which members of the community do you usually provide ITNs? Nets are available to everyone through the ITN Committee at 100 K each. However, they are currently out of stock. (Nets from PSI are sold by the health centers at 50 K during ANC visits.) 5. How do you track distribution and sale of ITNs? How is money accounted for and kept for safekeeping? What happens to the money after it is collected? Committees had various answers to this question: through registration, or kept by the ITN Committee Treasurer or HSA. 6. How many ITNs were distributed in the past three months? In only one of the seven ITN Committees were members able to agree on the number of nets they had distributed, implying the lack of a transparent accounting system with checks and balances. 7. Who comes to supervise you in completing your activities? How often do you receive supervision and how helpful was this support? Supervision of the ITN Committees is provided by the HSAs, at least once per month. 8. How can the activities of the ITN Committee be improved? The primary request from the committees was for help in maintaining records of the nets they have received and sold, as well as the commission they have earned. They also asked for more nets to respond to the unmet demand and for the cost of the nets to be lowered so more could afford to buy them. There was also interest expressed by some groups in setting up an account to better manage the money and use the money for income generating activities. Final Evaluation IEF/DHMT Nsanje Malawi 59 C.A.: No: HFP-A-00-02-00027-00 Breastfeeding Support Groups/ PMTCT Groups & Growth Monitoring Volunteers Members of the FE Team interviewed 20 Breastfeeding Support Group (BFSG) members, seven Growth Monitoring Volunteers (GMV) and 14 Prevention of Mother to Child Transmission Support Group (PMTCT) Volunteers. 1. What are you most proud of as a GMV, BFSG, PMTCT volunteer? The opportunity to provide health education was most frequently mentioned as a point of pride by all volunteers. Many also noted that they were proud of being able to effectively monitor children for danger signs and appropriately refer those who are sick for medical care. Many enjoy the respect they received as volunteers as well as the responsibility that comes with volunteering. 2. What health education messages do you convey to the community the most frequently? The volunteers most commonly communicate health education messages about sanitation, hygiene, exclusive breastfeeding, and HIV/AIDS prevention, which relate most closely with the type of group they have joined or volunteer they have become. The danger signs associated with childhood illness were not mentioned as frequently. As possible, those individuals supporting and supervising these volunteers should consider adding these messages to the ones they are already teaching to the volunteers and asking them to convey them to the community. 3. Who comes to supervise you in completing your activities? How often do you receive supervision? Most of the volunteers said they were being supervised by the HSA at least once per month. 4. How has the community supported your work? • Complying with their health education messages. • Assisting on sanitation projects, growth monitoring, and other project-related activities. • Participation in meetings. 5. Do you think the health activities of the HSAs, VHCs, ITN committees, and the other volunteers have been able to change health practices in the community? Can you provide an example? All agreed their work was leading to changes in practices in their communities. The most frequently mentioned examples included decreased incidence of diarrheal disease resulting from improved sanitation, decreases in malaria due to the distribution of bednets, and an increased willingness to go to a health facility for care. 6. What activities do you have planned for next month? Final Evaluation IEF/DHMT Nsanje Malawi 60 C.A.: No: HFP-A-00-02-00027-00 Each of the volunteers voiced a commitment to continuing their work (health education and mobilizing for small volunteer projects) and a majority had plans that could potentially be done in the community without outside support. Mothers The FE Team held 12 focus group discussions in 12 communities with a total of 233 mothers, involving primarily mothers of children under five years of age. 1. Do you have meetings where someone talks to you about health? How often are these meetings? And who gives these talks? Women in nearly all of the communities said they attend meetings at least once a month where someone talks to them about health. 2. How have you been able to apply what you learned? If you could not apply it, why not? Mothers listed sanitation projects (water, latrines, environmental cleanup), exclusive breastfeeding, and sleeping under bednets as ways they had applied what they had learned in health education meetings. (NOTE: IMCI danger signs were only mentioned minimally or not at all, raising the need for these messages to be focused on in the future.) 3. In your community, what are the activities of the HSAs, VHCs, GMVs, BFSGs, and PMTCTs? In each of the FGDs with mothers, the respondents were aware of who their HSA is and in most communities, the respondents were aware of who the other volunteers and what their roles are, implying good outreach and communication. 4. How can the HSA, VHC, GMV, BFSG, and PMTCT Committee better support health activities? They would like to see their HSA and the volunteers more frequently – doing home visits, hosting meetings, and providing trainings. 5. Are more community people using the health facility now compared with a year ago? If yes, why do you think this is so? There was general consensus across all of the FGDs with mothers that they were using their local health facilities more frequently due to: increased awareness of danger signs, improved access to quicker care, and a growing sense that going to traditional healers was a waste of money. Final Evaluation IEF/DHMT Nsanje Malawi 61 C.A.: No: HFP-A-00-02-00027-00 6. Why are some mothers not taking their children to health facilities when they are sick? Can you provide some reasons? 14 The primary reasons mothers mentioned for this were due to the distance to facilities and lack of money for fees. Fear was also mentioned, but it was not possible during the FGD debriefing to clarify why they were afraid. While it appears that mothers might be opting to go to health facilities over traditional healers more than before, traditional healers remain a familiar source of health care and will likely continue to be so into the future. 7. Why do some mothers provide less food to their children when they are sick? Provide some reasons.15 The main reasons suggested for why some mothers are cutting back the amount of food provided to a sick child were the belief that it would help the child stop vomiting and the sick child usually refuses to eat due to a lack of appetite. 14 This question was included to further explore the results on one indicator measured in the KPC baseline and final surveys. While relatively high at the final evaluation KPC (74%), the percentage of caretakers who took children 0- 23 months to a health workers for diarrhea, fever or difficult breathing after recognizing illness symptoms, decreased by ten percentage points from the baseline KPC rate of 84%. 15 These questions were included in the FGDs to better understand possible reasons why there was no significant movement on the project indicator, percent of children 6-23 months of age who consumed the same amount of foods during their most recent episode of reported illness, from the baseline (49%) to the final KPC survey (46%). Final Evaluation IEF/DHMT Nsanje Malawi 62 C.A.: No: HFP-A-00-02-00027-00 Attachment F Rapid CATCH Indicators Final Evaluation KPC Survey June-July 2006 Rapid Catch Indicators % N/D CI 1. Percentage of children age 0–23 months who are underweight (-2 SD from the median weight-for-age, according to the WHO/NCHS reference population) 27% 57/213 19%-35% 2. Percentage of children age 0–23 months who were born at least 24 months after the previous surviving child 90% 183/204 84%-96% 3. Percentage of children age 0–23 months whose births were attended by skilled health personnel 64% 191/300 56%-72% 4. Percentage of mothers with children age 0–23 months who received at least two tetanus toxoid injections before the birth of their youngest child 71% 213/300 62%-80% 5. Percentage of children age 0–5 months who were exclusively breastfed during the last 24 hours 57% 50/87 42%-72% 6. Percentage of children age 6–9 months who received breastmilk and complementary foods during the last 24 hours 79% 42/53 63%-95% 7. Percentage of children age 12–23 months who are fully vaccinated (against the five vaccine-preventable diseases) before the first birthday 80% 103/129 70%-90% 8. Percentage of children age 12–23 months who received a measles vaccine 80% 103/129 70%-90% 9. Percentage of children age 0–23 months who slept under an insecticide-treated net (in malaria risk areas) the previous night 70% 209/300 61%-79% 10. Percentage of mothers with children age 0–23 months who cite at least two known ways of reducing the risk of HIV infection 65% 195/300 56%-74% 11. Percentage of mothers with children age 0–23 months who report that they wash their hands with soap/ash before food preparation, before feeding children, after defection, and after attending to a child who has defecated 9% 26/300 4%-14% 12. Percentage of mothers of children age 0–23 months who know at least two signs of childhood illness that indicate the need for treatment 73% 220/300 66%-80% 13. Percentage of sick children age 0–23 months who received increased fluids and continued feeding during an illness in the past two weeks 22% 40/186 14%-30%