Mid-Term Evaluation of The Tiweko Tose Child Survival Project August 19 - 28, 2002 WR Malawi/CCAP Livingstonia Synod Hospitals Child Survival Project “Tiweko Tose” means “All of Us Together” This Mid-Term Evaluation was completed in compliance with Cooperative Agreement # FAO-A-00-00-00050-00 for World Relief, 7 East Baltimore St, Baltimore MD 21202 Tel. (443) 451 1900 E-mail: worldrelief@wr.org Executive Summary The Tiweko Tose (All of US Together) Child Survival Project (2000-2004) is located in Mzimba and Rumphi districts of northern Malawi. The three project areas include the health service catchment areas around the Ekwendeni, Embangweni, and David Gordon Memorial hospitals of the CCAP Synod of Livingstonia (SOL). The target group comprises children 0-5 year olds (36,732) and women of childbearing age (32,185). The project goals are to 1) reduce morbidity and mortality in children under 5 and women of childbearing age; 2) strengthen the capacity of the SOL to implement Child Survival interventions; and 3) empower communities to improve their health. The major project strategy is to work in partnership with SOL and the MOH to help communities create 300 Care Groups and train 3,000 community volunteers as behavior change agents. Care Group volunteers make regular visits to more than 30,000 households to explain and promote key messages for each child survival intervention. The project intervention mix includes Nutrition (25%), Malaria control (15%), Pneumonia case management (15%), maternal and newborn care (15%), child spacing (15%), and STI/HIV/AIDS prevention (15%). The project also includes capacity building of SOL to strengthen its organizational capacity to manage child survival activities and to increase/reinforce other community-based and health facility-based services, e.g., traditional birth attendants (TBAs), drug revolving funds (DRF), insecticide-treated bednets (ITN), and community based distribution agents (CBDAs). These activities are conducted in collaboration with, and as part of, the MOH district health system. The mid-term evaluation took place from August 19-28, 2002. The evaluation team of 15 persons included external evaluators, World Relief personnel, project staff, SOL PHC directors, and MOH representatives. The evaluation team reviewed CSP quarterly surveys and other project reports. Three teams visited the project areas to hold discussions with mothers’ groups, Care Groups, project promoters, community leaders, and health center personnel. The project is to be commended for its strategy of conducing quarterly surveys and for having completed an internal review in preparation for the MTE. These activities made it possible to complete the MTE for a complex and dispersed project in a timely manner. The evaluation team found that the project is making excellent progress in achieving its primary objectives. The project has met nearly all mid-term targets, and exceeded end of project targets for four of eleven objectives. For example, in collaboration with the SOL CBDA program, the project has increased contraceptive use of pills, depo-provera, and condoms from 23% to 50%. The project has also demonstrated that the Care Group structure can be successfully implemented in Malawi as a social movement for behavior change. Discussions at all levels indicate perceived and real increases in referrals and reduced mortality. For example, the percentage of children under five years treated the same day or the next day for fever (suspected malaria) has increased from 35% to 52%. The project must now concentrate on consolidating and sustaining its success. For example, the project should strengthen project monitoring, improve information exchange with partners, and diversify strategies for sustainability of Care Groups. Based on the MTE findings and discussions, ten major recommendations and strategies for implementation were agreed upon and discussed with project partners. These are summarized on the following page. Major Recommendations 1) Standardization of Indicators: CSP needs to clarify the definition of project indicators and ensure that their measurement is compatible with the baseline KPC survey and international standards. 2) Drug Revolving Funds: The three Health Units in collaboration with CSP and the MOH should evaluate the status of the DRF program to identify and correct problems with the supply of essential medicines at the community level. 3) Obstetrical Emergency Transportation Plans: CSP should encourage communities to develop community-wide emergency transportation plans in addition to family-specific plans, especially for complications of pregnancy. CSP, the three health units and the MOH should also strategically place bicycle ambulances in a few communities. 4) Incentives for Care Group Volunteers: CSP should identify and promote best practices for incentives to Care Groups. These might include 1) services provided to Care Groups from the community; 2) recognition of well performing Care Groups by the MOH and Health Units; 3) exchange visits between Care Groups; and 4) income generating activities. 5) Behavior Change Communications (BCC) targeted to Men: CSP should target more BCC to men and community leaders. Possible strategies include 1) special training of village leaders; 2) revitalizing “Mphala,” a tradition of men to boy communication; 3) encouraging men’s discussion groups; and 4) team visits by volunteers to difficult households. 6) BCC Materials and Training: CSP should document best practices in Behavior Change Communications materials and make these widely available. These materials might include 1) durable picture codes for Care Group volunteers; 2) written materials in the local language for volunteers and village leaders; 3) orientation programs for community leaders; and 4) appropriate audio-visual equipment for Health Units. 7) Program Integration CSP and Synod: CSP and the three Health Units should consider how promoters might provide integrated, cost-effective support and supervision for all community-based Synod health initiatives. 8) Project Expansion and Replication: CSP, the three Health Units and the Synod should examine the geographic reach of its current programs to 1) determine how to reach inaccessible areas within the current CSP project area and 2) discuss options with the Ministry of Health for expanding services to other communities and health areas. 9) Exit Strategies and Local Sustainability: CSP should consider diversifying strategies to sustain Care Groups. In addition to strengthening Village Health Committees, alternatives might include 1) regular meetings between Chief volunteers and village headmen; 2) Care Group Zone Committees; and 3) increased links to HSAs and health centers. 10)Health Information System: CSP should strengthen project monitoring and information exchange. Specifically, CSP should 1) adopt standardized tools to supervise promoters and Care Groups, 2) improve information feedback to communities; 3) increase information exchange with the MOH and within Synod Health Units; and 4) improve the mapping of project areas and activities as part of the health district system. Figure 1: Map of Project Area Abbreviations and Acronyms AIDS Acquired Immune Deficiency Syndrome ARI Acute Respiratory Infection BCC Behavior Change Communication CBDA Community Based Distribution Agent CCAP Church of Central Africa Presbyterian CG Care Group CSP Child Survival Project DGMH David Gordon Memorial Hospital DHMT District Health Management Team DHO District Health Officer DIP Detailed Implementation Plan DRF Drug Revolving Fund EBF Exclusive Breast Feeding EOP End of Project EPI Expanded Program on Immunizations GMC Growth Monitoring Clinic HC Health Center HH House Hold HIS/MIS Health Information System/Management Information System HIV Human Immunodeficiency Virus HQ Headquarters HSA Health Surveillance Assistants IEC Information, Education and Communication IFA Iron Folic Acid IMCI Integrated Management of Childhood Illness ISA Institutional Strengths Assessment ITN or ITM Insecticide-Treated Nets (or Materials) KPC Knowledge, Practice and Coverage Survey LLW Lilongwe (Malawi’s Capital City) MCH Maternal and Child Health MOH Ministry of Health MTE Mid – Term Evaluation ORS (ORT) Oral Rehydration Solution (and Therapy) PCUSA Presbyterian Church United States of America PHC Primary Health Care PVO Private Voluntary Organization SOL Synod of Livingstonia SP Sulfadoxine Pyrimethamine (Fansidar) STI Sexually Transmitted Infections TA Technical Administrative support TBA Traditional Birth Attendant TTCSP Tiweko Tose Child Survival Program USA United States of America USAID United States Agency for International Development VCT Voluntary Counseling and Testing VHC Village Health Committee WatSan Water and Sanitation WCBA Women of Childbearing Age WR World Relief Table of Contents Executive Summary Major Recommendations Map of project area Abbreviations and Acronyms Tables and Figures I. Introduction............................................................................................................................1 A. Brief overview of the project B. Scope of Work for the Mid-Term Evaluation C. Assessment Methodology II. Technical Approaches...........................................................................................................4 A. Overall Project Approach B. Project Objectives, Indicators and Progress C. Technical Interventions D. New tools and innovative approaches III. Cross Cutting Approaches ..................................................................................................17 A. Community Mobilization B. Communications for Behavior Change (BCC) C. Capacity Building D. Sustainability and Exit Strategies IV. Program Management .........................................................................................................29 A. Planning B. Human Resources and Staff Management C. Financial Management D. Logistics Management E. Information Management (Health Information System) F. Technical and Administrative support V. Conclusions and Recommendations...................................................................................34 A. Conclusions B. Summary of Recommendations C. Results Highlights Attachments: A. MTE Team Members and Resources Persons B. Mid-Term Evaluation Methodology C. MTE Questionnaires for Field Visits Discussions D. MTE Schedule and Contact List E. Action Plan for Implementation of MTE Recommendations Tables and Figures Figures: Figure 1: Map of Project Area Figure 2: Project Intervention Mix Figure 3: Children <5 yrs. treated same/next day for fever Figure 4: Children < 5 yrs. treated same/next day for rapid, difficult breathing Figure 5: Children < 5 yrs. sleeping under a bednet (ITN) Figure 6: Bednets (ITNs) retreated within last 12 months Figure 7: Reported Cased by a Health Surveillance Assistant Figure 8: Reported Deaths by a Health Surveillance Assistant Figure 9: Children < 36 months weighed at GMC sessions Figure 10: Pregnant/Lactating women receiving Iron Folic Acid (IFA) Figure 11: Married women using modern contraception Figure 12: Married women using condom during last intercourse Figure 13: Pregnant women receiving 2 or more SP doses Figure 14: The Protected Community Award Figure 15: The Care Group Structure Figure 16: Project, Partner and Community Structures Figure 17: Links from Care Groups to HSAs and Village Headmen Figure 18: Project Organizational Chart Tables: Table 1: Summary of Field Visits to the Three Project Areas Table 2: Field Program Summary Table 3: KPC and Quarterly Survey Results Table 4: What Volunteers Like and Don’t Like About Their Work Table 5: What Promoters Like and Don’t Like About Their Work Table 6: Key Messages for the Technical Interventions Table 7: Comparison of Community-Based Activities in the Three Project Areas Table 8: Sustainability Objectives for Synod of Livingstonia Hospitals and Health Services Table 9: Sustainability and Capacity-Building Objectives (Community Groups) Table 10: Human Resources for Tiweko Tose Midterm Evaluation of the Tiweko Tose Child Survival Project Page 1 I. INTRODUCTION A. Brief overview of the project The Tiweko Tose Child Survival Project (2000-2004) is located in Mzimba and Rumphi districts of northern Malawi. The three project areas include the health service catchment areas around the Ekwendeni, Embangweni, and David Gordon Memorial hospitals of the CPAP (Central Church of Africa, Presbyterian) Synod of Livingstonia (SOL). The target group comprises children 0-5 year olds (36,732) and women of childbearing age (32,185). The project goals are to: 1) Reduce morbidity and mortality in children under 5 and women of childbearing age; 2) Strengthen the capacity of the SOL to implement Child Survival interventions; and 3) Empower communities to improve their health. The major project strategy is to work in partnership with SOL and the MOH to help communities create 300 Care Groups and train 3,000 community volunteers as behavior change agents. Care Group volunteers make regular visits to more than 30,000 households to explain and promote key messages for each child survival intervention. The project intervention mix includes Nutrition (25%); Malaria control (15%); Pneumonia case management (15%); Maternal and newborn care (15%); Child spacing (15%), and the prevention of STI/HIV/AIDS (15%). The project also includes capacity building of SOL to strengthen its organizational capacity to manage child survival activities and to increase/reinforce other community-based and health facility-based services, e.g., traditional birth attendants (TBAs), drug revolving funds (DRF), insecticide-treated bednets (ITN), and community based distribution agents (CBDAs). These activities are conducted in collaboration with, and as part of, the MOH district health system. B. Scope of Work for the Mid-Term Evaluation The mid-term evaluation provides an opportunity for all project stakeholders to take stock of accomplishments to date, and to listen to the needs of beneficiaries at all levels – mothers, other community members and opinion leaders, health workers, health system administrators, local partners, other organizations, and donors. The mid-term evaluation focuses on the process of program implementation, using data and information from the program’s monitoring system to: · Assess progress towards achievement of objectives or yearly benchmarks; · Assess progress in implementing the project approach and work plan; · Assess if the approach used and interventions are sufficient to reach desired outcomes; · Identify barriers to achievement of objectives; and · Provide recommendations to improve the project. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 2 C. Assessment Methodology The mid-term evaluation took place from August 19-28, 2002 at Mzuzu, the project headquarters, and the three project areas (see Figure 1 - Map of Project Area). The evaluation team consisted of fifteen persons. External evaluators included the team leader, two MOH representatives from Mzimba and Rumphi districts, and representatives from USAID and World Relief/USA. The team also included two representatives from the World Relief Lilongwe office and three PHC directors/coordinators from the principle project partner, the CCAP synod. Five project personnel participated, including the project director, deputy director, and three project area coordinators. In addition, five health educators from the project served as resource persons and logistical coordinators for the field visits. The methodology for the MTE followed the USAID guidelines as summarized in Annex C. The evaluation team reviewed project documents, as well as the project monitoring surveys. The results of the baseline KPC survey and quarterly surveys were extremely valuable in assessing the achievement of project objectives. Many child survival projects do not complete a mid-term KPC, so World Relief’s methodology of conducting quarterly surveys is to be commended. In addition, the team reviewed the results and recommendations of an internal MTE review that the project personnel had completed in preparation for the MTE. The internal review and the quarterly surveys made it possible to complete the MTE for a complex and dispersed project in a timely manner. The evaluation team was divided into three groups for visits to the three project areas to hold discussions with mothers’ groups, Care Groups, project promoters, community leaders, and health center personnel (see Table 1). Project and MOH evaluators were asked to visit and evaluate project areas where they did not normally work. Table 1: Summary of Field Visits to the Three Project Areas DGM Embangweni Ekwendeni Totals Health Facilities Luwuchi (Synod) Chitimba (MOH) Kalikumbi (Synod) Ekwendeni (Synod) 12 HSAs interviewed Local Leaders: Chakaka Kambundi Mtuzuzu Enyezini 55 community leaders Promoters Group of 12 Group of 13 Group of 13 38 of 45 Care Groups Chakaka, Chiweta Chitimba, Zowo Kawaza Chizimya Dunduzu Engcongolweni 100 volunteers Mothers Chakaka, Chitimba Nkhombwa Kawaza Chizimya Dunduzu Engcongolweni 150 mothers Midterm Evaluation of the Tiweko Tose Child Survival Project Page 3 The group discussions (see Annex D for the questionnaires) focused on the following topics: · The effectiveness of the Behavior Change Communication methods; · The organization, effectiveness and sustainability of Care Groups; · The relationship between Care Groups, community leaders and health facilities; · The organization of medical emergency transportation; · Problems encountered in participation in this project; · Knowledge of and satisfaction with project activities; and · Suggestions for improvement of project activities and sustainability. Following the field visits, the three teams consolidated their findings to answer the following questions: • Is the project achieving its objectives? • Is the project implementing its work plan satisfactorily? • Is the approach used sufficient to meet objectives? • What are the strengths and challenges to achievement of objectives? • What are recommendations to improve the project? Based on findings and discussions, ten major recommendations were agreed upon, as well as specific suggestions and strategies for implementation of the recommendations. The evaluation results were presented and discussed with a large group of project partners and with USAID. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 4 II. TECHNICAL APPROACHES A. Overall Project Approach The Tiweko Tose Child Survival Project (CSP) is an ambitious project, the largest World Relief (WR) has ever attempted. It is also a very complex project for several reasons. First, it is essentially three projects in one, since the project includes three project areas in two northern districts of Malawi (see project map in Figure 1). Second, most child survival projects are implemented in partnership the communities and/or MOH as the principle partner. In the case of CSP, however, the CCAP Synod of Livingstonia is the principle partner. This doubles the number of capacity-building interactions and complexity of the project, i.e., project to SOL, project to community, project to MOH, SOL to community, SOL to MOH, and MOH to community. Third, the SOL has a long history of community-based health initiatives that includes a variety of community-based structures and volunteers. While this work includes many positive contributions, it also dramatically increases the number and complexity of interactions that project personnel and Care Group volunteers must work with and through. The project approach is based on community volunteers organized into Care Groups and supervised by project-paid promoters and health educators. This involves three key activities: 1) Helping communities create 300 Care Groups of 10-12 volunteers (male and female); 2) Training 3,000 community CG volunteers as Behavior Change Communication agents; 3) Encouraging and monitoring CG volunteers as they make weekly visits to 30,000 households with BCC messages. Tiweko Tose has adopted and adapted the Care Group strategy that was pioneered by World Relief’s Vurhonga Child Survival Project in Mozambique. The Vurhonga project found that the Care Group structure transformed behavior change from individual decisions to a social movement. Care Groups became a source of encouragement and social support for volunteer mothers. They became a well-accepted and sustainable community institution. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 5 B. Project Objectives, Indicators and Progress 1. The Field Program Summary Table The TT CSP Detailed Implementation Plan (DIP) indicates that the project will spread its efforts on across all of the USAID-funded child survival interventions. (see Table 2). It was agreed during DIP review that immunizations and diarrhea interventions would be integrated with general nutrition and Maternal and New born care. Table 2: Field Program Summary Intervention % of Total Effort (a) Immunization 0% General Nutrition (including breastfeeding) Immunization Control of Diarrheal Diseases 9% 3% 3% Vitamin A 5% Micro nutrients (iron folate) 5% Control of Diarrheal Diseases 0% Pneumonia Case Management 15% Control of Malaria 15% Maternal and Newborn care 15% Child Spacing 15% STI/HIV/AIDS Prevention 15% Others (specify) 0% Total 100.% (a) Percent of total effort (from USAID and PVO match funding) for each intervention. Figure 2: Project Intervention Mix Midterm Evaluation of the Tiweko Tose Child Survival Project Page 6 2. Progress in achieving project objectives The project is to be commended for the quarterly surveys that it conducts. The sampling procedures for these quarterly surveys involved three stages of sampling: 1) Each of the 45 promoters randomly selects one Care Group (45 Care Groups) 2) Within each Care Group two volunteers are selected (90 Volunteers) 3) All households for those two volunteers are visited/interviewed (900 households) While these quarterly surveys do not replicate the sampling procedures of the baseline KPC there are quite adequate for monitoring purposes. In fact, World Relief’s previous experience in the Vurhonga I project demonstrated that the methods used are quite comparable to the results of the KPC cluster sampling. The sampling size per project area (approximately 300 households) should also be sufficient to permit comparisons between project areas. The evaluation team found that the project is making excellent progress in achieving its objectives (see Table 3). The project has met nearly all mid-term targets, and exceeded some end-of- project targets. There are, however, several definition and measurement problems with how some objectives are defined or how they were measure during quarterly surveys. Objective 7: “Increase from 36% to 50% the proportion of mothers exclusively breastfeeding 0-6 months infants.” The baseline of 36%, which is consistent with the national average, was determined by asking if the child was being exclusively breastfed AND by conducting a 24-hour recall (which resulted in 46% exclusive breastfeeding) and a 7-day recall (which resulted in 36% exclusive breastfeeding). The quarterly surveys, however, are based a 24-hour recall. The project recognizes the need to define the measurements of the quarterly surveys to match the baseline KPC. Objective 8: “Increase from 23% to 40% the percentage of WCBA (Women of Child-Bearing Age) who use a modern method of contraception (pill, condom, Depo-Provera). The project has opted to limit BCC for child spacing to married women. The KPC baseline and quarterly surveys were also limited to married women. The project should redefine this objective to clarify the focus on married women. Objective 9: “Increase from 17.5% to 30% the percentage of sexually active women who stated they used a condom during their most recent sexual intercourse (for contraception or HIV prevention).” The project has limited BCC for condom use to only married women. The KPC baseline and quarterly surveys were also limited married women. The objective needs to be redefined and focused on married women. Objective 10: “Increase to 80% the proportion of families who have an emergency transport plan in place before delivery.” The baseline of 9.5% was determined by asking households to identify actions to be undertaken in case of an obstetrical emergency. These actions involved “logistical issues” (availability of transportation) and “permission issues” (permission to seek care). However, during quarterly survey, households were simply asked, “Does your family have a plan for emergency transportation in case of an obstetrical emergency?” This resulted in a higher response rate that is not comparable to the baseline. Recommendation 1: Standardization of Indicators: CSP needs to clarify the definition of project indicators and ensure that their measurement is compatible with the baseline KPC survey and international standards. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 7 Table 3: KPC and Quarterly Survey Results PROJECT OBJECTIVES KPC 2000 1 st Survey Sep ‘01 2 nd Survey Dec ‘01 3 rd Survey May ’02 4 th Survey Jul ’02 MTE Targets EOP Targets 1. Increase from 35.4% to 90% the percentage of children < 5 yrs who are treated the treated the same day or the next day for fever (suspected malaria) at an appropriate health facility. 35.4 52.0 56.2 56.0 52.1 45.0 90.0 2. Increase from 27.6% to 50% the percentage of children < 5 yrs who are treated the same day or next day for rapid, difficult breathing (with or without fever) at an appropriate health facility. 27.6 52.0 42.7 65.0 55.0 30.0 50.0 U5s 13.8 26.1 23.1 15.0 50.0 3. Increase from 8.5% to 50% the number of children <5 yrs & pregnant women sleeping under a bed net. Preg. Women 8.5 15.0 18.3 23.3 20.1 15.0 50.0 4. Increase from 62% to 75% the percentage of bed nets that will be retreated within the last twelve months. 62 63.0 39.2 55.0 62.2 65.0 75.0 5. Increase from 65% to 90% the number of children 0-35 months weighed regularly in growth monitoring and counseling (GMC) sessions. 65 61.0 59.8 74.04 65.2 75.0 90.0 6. Increase from 3% to 30% the percent of pregnant or lactating women who receive daily IFA supplements. 3 * 66.0 58.0 5.1 10.0 30.3 7. Increase from 36% to 50% the proportion of mothers exclusively breastfeeding 0-6 months infants. 36 76.0** 80.7** 72.4** 82.1** 40.0 50.0 8.Increase from 23% to 40% the percentage of WCBA who use a modern method of contraception (pill, condom, Depo-Provera). 23 36.0 47.7 50.3 50.1 25.0 40.0 9.Increase from 17.5% to 30% the percentage of sexually active women who stated they used a condom during their most recent sexual intercourse (for contraception or HIV prevention). 27.5 12.0 21.4 17.1 22.0 20.0 30.0 10. Increase to 80% the proportion of families who have an emergency transport plan in place before delivery. 9.5 * 75.2** 84.1** 86.3** 10.0 80.0 11. Increase from 30% to 60% the number of pregnant women who receive at least 2 doses of SP during pregnancy. 30.6 * 66.7 56.0 63.0 40.0 60.0 * = Data not collected during the first survey. ** = Quarterly method not the same as the baseline Midterm Evaluation of the Tiweko Tose Child Survival Project Page 8 C. Technical Interventions The project emphasizes key messages at the household level for all child survival interventions. These interventions are introduced in phases according to the following schedule: · Malaria and Pneumonia (April 2001) · Nutrition and Exclusive Breastfeeding (August 2001) · HIV/AIDS and Child Spacing (April 2002) · Maternal and Newborn Care (Oct-Dec 2002) These activities as outlined in the DIP are appropriate and sufficient to meet project objectives. At the time of the MTE, a total of 5 interventions have been phased in and all related activities have been carried out. The technical nature of the interventions has not been altered from the strategies proposed in the DIP. However the sequence and timing of implementation of some activities and interventions has changed. The project strategy is to train staff and volunteers in each intervention and gradually implement them one by one. This provides volunteers an opportunity to fully learn the material and to concentrate on training households for those messages. World Relief has found that by teaching in smaller units, people have sufficient time to internalize and put into practice the information they are learning. Also, interventions can be timed during their greatest need, e.g., introduce malaria training at the beginning of the malaria season. The emphasis of this project is on transmitting key behavior messages for the technical interventions. This includes social behavior and communication strategies including interpersonal communication, peer communication, and support groups. The project does not generally provide services or training for service providers, since it is the role of the Synod PHC and MOH to train service providers. However, due to the program’s integrative nature, some of these service providers participate in project training sessions. For example, TBAs are trained if they are members of a Care Group. Health Surveillance Assistants and Home Craft Workers are also invited to join promoters during their training sessions. 1. Malaria and Pneumonia Malaria and pneumonia together are the major cause of under-5 mortality, accounting for over half of the deaths of children nationwide (malaria/anemia, 35%; pneumonia, 16%). Malaria is among the top 10 preventable communicable diseases in Malawi and the leading cause for outpatient visits (Malawi MOHP Health Facility Survey; 20-25 November 2000.) The project baseline KPC survey found that symptoms and malaria and pneumonia were quite common as reported for illnesses in children 0-23 months in the previous two weeks: · 58.6% reported fever · 59.3% reported cough · 27% reported malaria Midterm Evaluation of the Tiweko Tose Child Survival Project Page 9 · 17.9% reported fast/short quick breaths · 19.2% reported difficult breathing Malaria control and pneumonia case management are addressed as one intervention because they share a similar symptom (fever) and primary strategy (rapid recognition and treatment of suspected cases). They were the first set of interventions to be introduced by the project in April of 2001. The Behavior Change Communication (BCC) component for malaria and pneumonia focuses on community understanding of the importance of preventing malaria transmission and responding to cases with prompt and appropriate treatment. The messages include: 1) Identify adverse effects of fever (Malaria). 2) Explain the importance of care seeking same day or next day. 3) Explain the importance of under-five children and pregnant mothers sleeping under bed nets. 4) Explain the importance of retreating bed nets. 5) Identify adverse effects of pneumonia. 6) Explain the importance of care seeking same day or next day. Mothers and promoters perceived the key messages for malaria as the easiest to understand and to obtain cooperation from other family members prompt care-seeking behavior and purchase of bednets. As is evident in the following graphs, the project has met its MTE target for malaria and already surpassed its EOP target for pneumonia case management. The plateau and slight decrease in referrals during the past six months may simply be a statistical “blip” in the surveys, or evidence of the deteriorating economic and food security situation in Malawi. Figure 3: Figure 4: The project has also met its MTE target for bednets, with utilization increasing from 8.5% to 23% for children and 20% for pregnant women. The project needs to redouble its efforts and collaboration with community bednet volunteers for this activity in order to reach its objective of 50%. While the re-dipping of bednets has improved during the past six months, the project and Synod should consider switching to permanently treated nets. Unfortunately, at this time the Malawi malaria policy has not yet adopted the use of permanently treated nets. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 10 Figure 5: Figure 6: There were also numerous comments from communities and health facilities about decreases in deaths from malaria and pneumonia in communities. These decreases were often attributed to the activity of Tiweko Tose in collaboration with Drug Revolving Funds (DRFs) and health facilities. One Health Surveillance Assistant (HSA) provided the team a copy of his written records for the past six years documenting the increased referrals and decreases in deaths for malaria, pneumonia, diarrhea and malnutrition. While the records of one HSA are insufficient to confirm the trend, the records from health facilities and the MOH health information system should be able to confirm the trend. Unfortunately, the project has not made use of this information. It is recommended, therefore, that the project and MOH work together to examine health facility data for the past four years to establish whether the trend shown below is real. Figure 7: Figure 8: Recommendation 2: Drug Revolving Funds: The three Health Units in collaboration with CSP and the MOH should evaluate the status of the DRF program to identify and correct problems with the supply of essential medicines at the community level. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 11 2. Nutrition and Exclusive Breastfeeding The objectives for this component are to: 1) Increase from 65% to 90% the number of children 0-35 mo. weighed regularly in growth monitoring and counseling (GMC) sessions. 2) Increase from 3% to 30% the percent of pregnant or lactating women who receive daily iron/folate supplementation for a minimum of 3 months. 3) Increase from 3.8% to 30% the percent of children 6-23 months old who will receive weekly Iron supplement. 4) Increase from 12.9% to 50% the percent of children 6-59 months old who will receive appropriate dose of Vitamin A capsules (2 per year.) 5) Increase from 36% to 50% the proportion of mothers exclusively breastfeeding 0- 6 month infants. The project staff feel that the nutrition intervention has begun to influence some longstanding socio-cultural issues that constitute underlying causes of malnutrition e.g. breast-feeding while pregnant, some foods (e.g. eggs, pumpkins) being taboos when taken by certain age groups and genders, exclusive breastfeeding, and breastfeeding first colostrums. The fact that some husbands are becoming jealous and complaining that their wives are giving more protein-rich foods to the children (and less to them) may be viewed as a positive “negative” indicator! The results in achieving the primary nutritional objectives have been mixed. The project received 50,000 Vitamin A capsules from Sight for Life. These were distributed to children in collaboration with MOH and Synod PHC, and coverage is now at 80%. The project also participated in the child health week in June 2002 and National Immunization days for measles and vitamin A in July 2002. Attendance at growth monitoring clinics had improved to meet the MTE objective, but recently decreased. It is thought that this is due to the current food shortages and the mothers need to spend more time in hunting for food. Nutritional supplementation of pregnant and lactating women improved dramatically, only to fall sharply due to a temporary stock out of the products in the MOH. Figure 9: Figure 10: Midterm Evaluation of the Tiweko Tose Child Survival Project Page 12 The Hearth approach is theoretically sound and appropriate and for reducing malnutrition given the rural Malawi’s current state of food security, poverty and deprivation. One village headman in DGMH remarked: Most nutritious foods we used to eat in the past are no longer eaten but are still available today. With the coming of Hearth we hope that mothers will start feeding children those foods to prevent children from malnutrition. The Hearth intervention was originally scheduled to begin in April 2002. However, due to the food crisis it was rescheduled to a future date. Being a new intervention it was thought that to establish its credibility it should be implemented at a time when chances of success are higher. Meanwhile, World Relief has assessed the health impact of food shortage in the project area, and assisted 5,500 families with 131.45 metric tones of white maize. Following a recent visit by the World Relief President and his Vice to the project area, WR Malawi is preparing to mobilize additional resources to respond to this emergency situation. It should also be noted that key messages for immunization and diarrhea control have been integrated with this component and are tracked both in the quarterly surveys. For example, diarrhea has reduced from 20% (KPC, 2000) to 13% (KPC, July/02). There has been an increase in children completing their immunizations by their first birthday from 74% (KPC, 2000) to 90% (KPC, May 02). 3. HIV/AIDS and Child Spacing The stated objectives of these components are to: 1) Increase to 1000 the number of people taking VCT in 6 months in all the three areas. 2) Increase to 30% the percentage of sexually active women who report using a condom during their most recent sexual act for either contraception or prevention against HIV/AIDS and Sexually Transmitted Infections. 3) Establish 5 post-test clubs to promote healthy lifestyles and advocate VCT in each of the 3 project areas. 4) Increase from 23% to 40% the percentage of WCBA who use a modern method of contraception (pill, condom, Depo-Provera). However, as was mentioned previously, the project and Synod have opted to limit BCC for family planning and condom use to only married women/couples. According to the project staff, the KPC baseline and quarterly surveys for these indicators have been limited to married women. The project needs, therefore, to clarify the definition of these objectives and indicators (see Recommendation #1). The key messages and lesson plans for this component were developed, just like the other project components, through a series of focus group discussions. The key messages are as follows: Midterm Evaluation of the Tiweko Tose Child Survival Project Page 13 · Knowing ones HIV status through Voluntary Counseling and Testing (VCT) helps to reduce further HIV transmission and prolongs life. · Men, Women, Boys and Girls should know at least 5 signs of Sexually Transmitted Infections (STIs) and upon seeing them they should seek medical treatment immediately. · Vwira, Chokolo, Nevayi. 1 Local Injections and bathing of dead bodies are some of traditional practices promoting the spread of HIV/AIDS. · Mothers who are HIV positive and choose to breastfeed should breastfeed exclusively · Married couples should use condoms for STI/HIV/AIDS prevention and contraception · Use of modern methods of family planning improves health, economic and social status of individuals and families. · Condoms when properly stored and used can effectively prevent HIV/AIDS and pregnancy. The implementation of this component began in May 2002, and has been proceeding as planned. This is component addressee one of the most cultural-sensitive areas because it involves negotiating socio-cultural factors that predispose people to HIV/AIDS including wife inheritance and local injections. Volunteers, however, are convinced that one-on-one interaction with households (mothers) is the ideal for such sensitive issues. At the same time, the evaluation team found that it has been difficult to obtain compliance for Voluntary Counseling and Testing (VCT). For example in a typical Care Group only four of the twenty-two volunteers had accepted to seek VCT. Progress for these project components is shown in the following graphics. The increases in modern contraception, from 23% to 50%, are truly impressive. The project should verify that the survey methodology for the quarterly and KPC baseline are truly comparable. The project also needs to determine what percent of Women of Child Bearing Age (WCBA) are married women. Assuming that the percentage is relatively high, this will help the project and Synod justify its approach in limiting BCC to married women. Figure 11: Figure 12: 1 Vwira means that a husband’s younger brother is secretly requested by the parents to have sex with his wife without his knowledge so that this family gets a child. This is usually done in families without children. Chokolo is inheriting a brother’s wife after the death of the husband, and Nevayi is a locally or commercially made razor blade. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 14 4. Maternal Health and Newborn Care The objectives of this project component are to: 1) Educate families about maternal risks; 2) Increase the number of deliveries by trained personnel, including trained TBAs; 3) Ensure that families have an emergency plan for referrals. 4) Increase the number of pregnant women receiving SP and Iron Folic Acid (as part of the nutrition and malaria components) The project strategy is to train Care Group volunteers will educate mothers (and specifically pregnant women) that every pregnancy is a risk to the mother, and complications cannot be predicted. Therefore, every pregnant woman should have at the least a trained TBA attending the birth, or even to go to the hospital if possible. The provision of Iron Folic Acid (IFA) as a nutritional supplement and Sulfadoxine Pyrimethamine (Fansidar) as a malaria prophylaxis for pregnant women have progressed well as shown in Figures 13 and 10. However, a recent temporary shortage of IFA by the MOH has had a negative impact on this the IFA component. Figure 13: The Maternal Health and Newborn Care component will officially begin later this year. The evaluation team, however, identified the emergency transport component as an issue for the MTE, because there was some question about whether the focus should be at the family or community level. The project objective and indicator is to “Increase to 80% the proportion of families who have an emergency transport plan in place before delivery.” As mentioned earlier, there is a definition problem as to how this indicator is measured during quarterly surveys. This has resulted in a higher positive response rate that is not comparable to the definition used during the baseline KPC (see Recommendation #1). The team asked mothers, volunteers, promoters and community leaders about the definition of a medical emergency, about the steps in deciding if and when to transport someone for referral, and about the available physical means of transportation in the community. The team was told that the decision process for emergency transportation is generally not a major problem, but that it does require the following steps: 1) A pregnant woman tells her husband; 2) The husband tells the grandmother; 3) The grandmother tells the TBA; 4) The TBA decides whether referral to the hospital is needed; 5) If so, the husband is responsible for arranging the transportation; and 6) The community, including Care Group volunteers, help organize the transportation. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 15 The team also learned that methods of transportation, primarily bicycles and ox carts, exist in the community, but not as part of an organized emergency transportation plan. It is recommended, therefore, that the project continue its focus on the family, and expand it to build a community￾wide emergency transportation plan, especially for complications of pregnancy. The following steps for implementation should be considered: Figure 14: 1) The Chief Volunteer(s) for the Care Groups in each community and their promoter meet with the village headman (or VHC) to discuss the need for a community-wide emergency transport plan for complications of pregnancy. They suggest and receive approval from the Village headman or VHC to develop a community plan. 2) Care Group volunteers provide BCC at the household level on Maternal Health and Newborn care, including the need for a family specific and community-wide emergency transportation plan. They collect information and ideas, e.g., identifying households with of carts that are willing to make them available for emergency transportation 3) The information collected by volunteers is compiled and discussed with the promoter during their regular meetings. They prepare a draft action plan, which identifies the means of transportation and the contact persons to implement the plan. 4) The proposed plan is presented to the village headman (or VHC) for discussion and approval. 5) The village headman presents the plan at a community meeting, and the plan is officially put into action. 6) Optional: CSP in collaboration with the Synod and MOH commends the community for this achievement and presents them with an achievement certificate or banner. 2 Recommendation 3: Obstetrical Emergency Transportation Plans: CSP should encourage communities to develop community-wide emergency transportation plans in addition to family-specific plans, especially for complications of pregnancy. CSP, the three health units and the MOH should also strategically placing bicycle ambulances in a few communities. 2 Cotton twill banners (19” x 25”) similar to the one shown on this page are produced by the SANRU III project in DR Congo at a cost of less than $5 per banner). Yearly “Badge-A-Mini” badges may optionally be used to update the banner each year, i.e., to ensure that community renews their commitment to the plan once a year. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 16 4. Special outcomes, unexpected successes or constraints The following are examples of special outcomes and successes of the project: a) Care Groups in some areas are also used to mobilize communities for other development activities taking place in the areas e.g. agriculture and community development activities. Volunteer communal gardens to raise funds to meet Care Group financial needs and establish village banks. This is also aimed at helping the destitute in terms of food and money b) Village bank at Chindoka Care Group in Embangweni and the idea has potential for replication in other Care Groups and areas. Funds are meant to support families to access treatment at the hospital when they do not have money and they repay afterwards. c) Open days to exhibit child survival activities and lobby for local support staged by volunteers with support of promoters. d) Zone committees (in Embangweni) or central committees (in DGMH) that oversee the activities of individual Care Groups in their respective areas. Their major role is to support Care Groups by ensuring that they are active and discharge their duties diligently and liaising with village headmen on how best to improve Tiweko Tose activities. In the absence of strong village health committee structure this is a potential surrogate. This will also facilitate HIS as they are a potential feedback loop with the community. e) Exchange visits between Care Groups and between promoters are also very important. This helps in problem solving and information sharing. Unexpected constraints that the project has encountered are 1) food shortages; 2) devaluation of the local currency (Kwacha) against currencies of major trading partners; 3) staff illness; and 4) road accidents involving project staff and other resources. D. New tools and innovative approaches The implementation of this project in partnership with the CCAP Synod of Livingstonia is an innovation for child survival projects. The project builds on the existing infrastructure and service history of health services of the church. The Care Group structure represents a new and innovative approach to community organization in Malawi. While this concept was “imported” from World Relief’s Vurhonga Child Survival project in Mozambique, the establishment and adaptation of Care Groups in the volunteer intensive environment of Malawi, has required the project to be the persistent pioneer for this approach. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 17 III. CROSS CUTTING APPROACHES A. Community Mobilization 1. Care Groups This project is based on the highly successful Care Group model that World Relief pioneered in its Vurhonga Child Survival Project in Mozambique. Community volunteers are organized in a Care Group structure as shown in the Figure 15. In brief, the Tiweko Tose Project Director and Deputy Director oversee the work of three Area Coordinators and five Health Educators, who in turn train and supervise 45 Promoters. Each Promoter works with five to eight Care Groups of 10-15 volunteers each. The project anticipates training approximately 3,000 volunteers. Volunteers, male or female, are to be respected by the community, honest, and committed to serving their neighbors. Literacy is optional provided at least some volunteers in every care group can read and write to help with record keeping. The volunteers make weekly visits to ten households for behavior change communications, i.e., a total of 30,000 visited each month. Figure 15: Care Group Structure In addition to home visits, the volunteers mobilize communities for attendance at outreach clinics for growth monitoring and immunization sessions. The volunteers also frequently assist with non-technical duties, e.g., weighing children. In this way, community volunteers serve as a link between clinical points of service and villages. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 18 The MTE focused on the work of Care Groups and their long-term sustainability in discussions with mothers, community leaders, promoters and, of course, the volunteers themselves. Mothers were very appreciative of the work of Care Group volunteers. They generally noted, “We have had Health Surveillance Assistants and community health workers before. But, now health messages are arriving at the household level” Most mothers received a home visit once a week or every two weeks. In particular they appreciated the opportunities for one-on-one discussion. They also noted, however, the absence of picture codes by many of the volunteers. Specific problems that they identified in their participation in the program included: · Husbands who think child spacing will increase the promiscuity of women; · Wives who are suspicious that a female volunteer may be after her husband; · Husbands who think that their wives don’t respect him anymore, because they give more of “his” protein-rich food to their children. Mothers suggested that the work of volunteers might be improved by 1) targeting young people and men with messages; 2) improving the availability of medicines in the Drug Revolving Funds; 3) training more mothers to become volunteers; 4) and doing something to help prepare the community for the next hungry season. The MTE team asked Care Group volunteers what they like and didn’t like about their work. Their responses are summarized in Table 4. Table 4: What Volunteers Like and Don’t Like About Their Work What they like. . . What they don’t like. . . · When households respond positively · Being part of a community · Uniforms · Status recognition · Gained knowledge · Promiscuity and alcoholism has decreased (because they are kept busy doing other things) · Improved family planning · Improved relationships with husbands who have become more supportive · Encouraged to set an example for the rest of the community (seeking cure for an STI) ·Not enough time to do their own work ·Not enough money for soap, etc. ·Want to receive an allowance by going somewhere for more training ·Difficult households that think BCC is obscene ·Being chased by drunken husbands ·“Friends” who make fun of their work ·People who think they are being paid by CSP ·Fear that volunteers are disrupting their families by encouraging family planning Volunteers had many good things to say about their promoters. They said that the encouragement that they received was the most important support. In addition they mentioned that promoters provided assistance with health message delivery; helped with communication with the hospital; and helped to resolve misunderstandings between volunteers. Volunteers also appreciate the support that they receive from community leaders, especially the authority that headmen provide them to visit households. They also noted the positive contributions of community leaders in community mobilization; dealing with difficult families; providing volunteers an opportunity to speak during funerals and at village meetings. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 19 With regards to Health Surveillance Assistants, however, the quality of the relationship was quite variable. Some Care Groups consider, in fact, that they support the HSA more than the HSA supports them. Community Leaders and health facility personnel were also quite positive about the work of Care Groups. They noted many positive changes in health and in changed behaviors in their communities. They generally cited the need for incentives and motivation of Care Groups as an important concern for sustaining their effectiveness. When asked about incentives, they mentioned not only the traditional incentives from the project (caps, T-shirts, and umbrellas), but also the importance of incentives from the community and the health system, e.g., public recognition of the work of Care Groups by community leaders, and Synod or MOH representatives. Care Groups were asked how they would sustain their work when the project ends. Strong dynamic Care Groups responded, "We are volunteers, our role is to help, we cannot stop." On the other hand, weaker Care Groups begged the question by responding that they would "form a committee to discuss the issue when the project ends." Other suggestions for sustaining Care Groups and providing incentives were as follows: · Volunteers in mature CGs should become village “consultants;” · Increase partnerships with other programs; · Income Generating Activities could help to sustain CG activities; · Incorporate Drug Revolving Funds, bednet distribution and CBDs into Care Group; · Provide loans to CGs to purchase seed and fertilizer; · Increase piece work (gardening, irrigation) to make money for the Care Group; · Social activities to solidify group dynamics, e.g., Bible studies The evaluation team was impressed with the level of motivation of Care Groups, especially with incentives that have been community generated, rather than project dependent. These best practices should be documented and shared between the three project areas. It is also a key component of the following recommendation. Recommendation 4: Incentives for Care Group Volunteers: CSP should identify and promote best practices for incentives to Care Groups. These might include 1) services provided to Care Groups from the community; 2) recognition of well performing Care Groups by the MOH and Synod Health Units; 3) exchange visits between Care Groups; and 4) income generating activities. 2. Promoters The 45 project paid promoters are the key personnel who are responsible for training and supervising Care Group volunteers. Promoters live in the same geographic area in which they work, exemplifying their commitment to the community and modeling that which they teach. This strategy also ensures that even remote communities lacking easy access to curative services will have access to promoters and volunteers. In the long-term a promoter may train a volunteer in to take up the role of the promoter for a specific skills area. This might be especially helpful in areas that are seasonally inaccessible. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 20 Promoters are trained independently in the three catchment areas, each area having 11-17 promoters. The curriculum consists of pictures, songs and stories that communicate the BCC messages in a format that engages the villagers. The health educators train promoters in the lessons, modeling for them exactly how the promoters will later teach the volunteers in their care groups. Each promoter-trainer is supervises up to eight Care Groups of 12-15 volunteers each. Promoters meet weekly with health educators to discuss common challenges and review upcoming lessons to be shared with the Care Groups. According to the DIP, promoter performance is to be evaluated formally twice a year using a performance-based criteria system that includes personal example, knowledge of interventions, training skills, performance of volunteers, results of the HIS for the households of that promoter’s volunteers, and relationships with other health workers. However, this system has not yet been fully implemented, and is noted as a part of the recommendation to improve project information management. Table 5: What Promoters Like and Don’t Like About Their Work What they like. . . What they don’t like. . . · Status recognition · Opportunities for personal and spiritual growth · Supervision and support from health educators · Gained knowledge through ongoing training · Salaries arrive on time from the Central Office · Major spare parts for bicycles are provided · Good working relationship with some HSAs · No holidays · Bikes bogging down in sand · High expectations and “Too much respect” · Lack of protective clothing · Insecurity in traveling to some communities · Being chased from some households · Difference in progress of Care Groups In addition to the importance of support from the project, promoters also noted the importance of community leaders in sustaining the work of Care Groups, e.g., in dealing with difficult households; mobilizing the community; trouble-shooting problems with volunteers, and occasionally providing food. B. Communications for Behavior Change (BCC) The project recognizes that effective BCC must be taught in small, easily digestible messages. The project has appropriately and carefully formulated BCC messages based on focus group discussions in the targeted communities. These messages, summarized in Table 6, were found to be technically correct and in keeping with MOH policies. People appreciate this educational approach because it is very participatory where grannies, fathers, mothers and women without children, as well as the youths get involved. For instance one granny commented thus: “We thank you for bringing these sessions at HH level because we can also learn and share what we have. Previously only women with children were called to an under five mobile clinic and we were left out! Do you think that elderly people do not need these type of health services?” Midterm Evaluation of the Tiweko Tose Child Survival Project Page 21 Table 6: Key Messages for the Technical Interventions Malaria and Pneumonia 1) Identify adverse effects of fever (Malaria). 2) Explain the importance of care seeking same day or next day. 3) Explain the importance of children under five and pregnant mothers sleeping under bed nets. 4) Explain the importance of retreating bed nets. 5) Identify adverse effects of pneumonia. 6) Explain the importance of care seeking same day or next day. Nutrition and Supplementation 1) Pregnant/lactating women and children under age of five should take adequate nutritious foods such as Yellow, Greens, Brown and White foods. 2) Pregnant/lactating women and under five children should take adequate iron and Vitamin A supplements. 3) All women of childbearing age, men and grannies should know the nutrition related complications in pregnancy/lactating and under fives. Exclusive Breast Feeding 1) Babies should exclusively breastfeed from soon after birth until six months 2) Breast milk contains all the nutrients required for a child from birth to six months 3) Colostrum protects the baby from infections from birth up to later stage. 4) Introduce solid foods after six months and continue breast-feeding for a minimum of two years whether mother is pregnant or not. Growth Monitoring and Counseling 1) All children under five should be weighed every month. Suspect a problem if there is no weight gain for two consecutive months 2) All at risk children should receive special care. 3) Parents and guardians should attend the under-5 clinics to be counseled on under-five childcare. Disease Control 1) Sick or recovering children should be given food and breast milk more than usual. 2) Children with diarrhea should be given fluids/ORS frequently, 3) Wash hands after contact with faces and before handling foods, 4) All immunizations should be completed by the child's first birthday. HIV/AIDS and Child Spacing 1) Knowing ones HIV status through Voluntary Counseling and Testing (VCT) helps to reduce further HIV transmission and prolongs life. 2) Men, Women, Boys and Girls should know at least 5 signs of Sexually Transmitted Infections (STIs) and upon seeing them they should seek medical treatment immediately 3) Vwira, Chokolo, Nevayi, Local Injections and bathing of dead bodies are some of traditional practices promoting the spread of HIV/AIDS 4) Mothers who are HIV positive and choose to breastfeed should breastfeed exclusively 5) Married couples should use condoms for STI/HIV/AIDS prevention and contraception 6) Use of Modern Methods of Family Planning improves health, economic and social status of individuals and families. 7) Condoms when properly stored and used can effectively prevent HIV/AIDS and pregnancy. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 22 There is ample evidence that BCC messages are resulting in new ways of doing things: · Increased attendance at health facilities and for antenatal care; · Volunteers referring mothers to the hospital even when they can only pay in kind; · Mothers refusing to go to traditional healers; · Traditional Healers bemoaning their ‘dwindling market’; · Traditional healers who become volunteers. · Openly breastfeeding child even when mother pregnant; · Increasing attendance at growth monitoring clinics; and · Couples opting for Voluntary Counseling & Testing. One issue that was identified by the project team was the possible need for more BCC directed to men. The evaluation team pursued this issued and found that some men perceive BCC as women’s issues. This base is reinforced if volunteer providing BCC happens to be female. At the same time, the evaluation team received numerous requests from village headmen for additional messages and orientation meetings for community leaders, who are primarily men. Based on these observations, the evaluation team collected ideas for increasing targeting of BCC to men. The organization of special meetings or discussion groups for men (and community leaders) outside of the household is an obvious possibility. There is, for example, a community tradition called “mphala” which involves the oral communication of wisdom from men to boys. It was suggested that this tradition might be revived with an emphasis on BCC messages. Recommendation 5: Behavior Change Communications (BCC) targeted to Men: CSP should target more BCC to men and community leaders. Possible strategies include 1) special training of village leaders; 2) revitalizing “Mphala,” a tradition of men to boy communication; 3) encouraging men’s discussion groups; and 4) team visits by volunteers to difficult households. One of the complaints from volunteers and promoters is the current lack of teaching pictures. The project intends to provide each of the 2950 community volunteers with a set of picture code teaching pictures. Unfortunately, there have been a number of delays in the reproduction of these materials. In an attempt to standardize teaching materials, the project has been trying, unsuccessfully, to obtain materials used by the MOH. The evaluation team received numerous requests from community volunteers and leaders for additional BCC materials. Given the relatively high level of literacy in the Northern part of Malawi, the requests for written materials are justified. The project and Synod have has recently obtained a digital camera to document their work. Recommendation 6: BCC Materials and Training: CSP should document best practices in Behavior Change Communications materials and make these widely available. These materials might include 1) durable picture codes for Care Group volunteers; 2) written materials in the local language for volunteers and village leaders; 3) orientation programs for community leaders; and 4) appropriate audio-visual equipment for Health Units. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 23 C. Capacity Building 1. Capacity Building of World Relief World Relief underwent an Institutional Strengths Assessment (ISA) in February 2002. The ISA was conducted by external evaluators from CSTS and CONCERN Worldwide, and included participation by all relevant World Relief headquarters staff, and 3 out of the 4 Child Survival Program field staffs. The ISA explored six interactive areas of overall organizational capability. The core element of the instrument was comprised of a questionnaire on which respondents rated the headquarters health division on multiple indicators within each area. The team complemented these findings with written recommendations from the field unit and recommendations and discussion of the headquarters staff over several hours of group meetings facilitated by the evaluators. As the ISA constitutes a first step in organizational strengthening, PVO’s are allowed to keep their results confidential, however World Relief would like to share a summary of what they learned from this exercise. The outside facilitators observed: ß Since World Relief has considerable experience of implementing Child Survival programs, they can draw on their experiences with other programs when confronting issues arising in new ones. ß Present staffing of the Health Technical Unit represents a diverse array of expertise that offers a range of resources in technical areas to field projects; they provide both direct, on￾site support and long distance technical support. ß There has been little recent turnover of key project staff in Health Unit or at Field Sites. Majority of Health Unit members moved with the organization to Baltimore, ensuring continuity in International Health Programs. ß There is an established system of progress reports going from the field to health unit; the health unit is able to source the input of consultants when appropriate. ß The health technical unit and larger organization clearly recognize weaknesses and are taking clear steps toward addressing those issues. ß There is a high degree of agreement that the support received from the health unit has increased the capacity of field staff to implement quality programs. ß Much discussion focused upon the issue raised most frequently by the field—improving the communication between projects. Several strategies were devised to address this issue and are in the process of being implemented. Overall, the ISA was deemed helpful to both HQ and field staff in both identifying areas in need of improvement and also in pointing out many strengths. When given the opportunity to reflect on Midterm Evaluation of the Tiweko Tose Child Survival Project Page 24 World Relief’s health portfolio, the World Relief staff and the external evaluators were impressed by their overall performance. A relatively small Headquarters staff was deemed successful in managing a portfolio of 17 discreet projects in 12 countries. 2. Capacity Building of the Synod of Livingstonia (SOL) The project is designed to build the capacity of the SOL to carry out child survival interventions in the overall context of their health system. This is particularly important since the SOL is the sole representative of the MOHP in their catchment areas. Capacity building of SOL is not simple since each of the three hospitals has a unique history of community project development and implementation, reflected in different core competencies and different community-based programs and groups. The DIP noted, “The project is not working in an organizational and program vacuum but rather in organizational and community settings that are richly diverse.” This diversity of SOL’s programs is even more evident in the following table. It was compiled as a MTE exercise to compare and rate the work of the three project areas. SOL has a long history, dating back more than ten years, of community-based initiatives including drug revolving funds (DRFs), bednet sales, community-based distribution agents, and water/sanitation committees. According to the CSP project staff, some of these structures, e.g., DRF, and VHCs, are only marginally functional, while others are functioning relatively well, e.g., growth monitoring clinic volunteers. This table also shows the strengths and weaknesses of the three project areas. Table 7: Comparison of Community-Based Activities in the Three Project Areas Abbrv. Description Since Embangweni Ekwendeni DGMH GMC Growth Monitoring Clinic Volunteer 10+ yrs ++ ++ +++ VHC Village Health Committees 10+ yrs + + ++ TBA Traditional Birth Attendants 10+ yrs ++ +++ + DRF Drug Revolving Fund Committees 10+ yrs ++ ++ + CBDA Community-Based Distribution Agent 10+ yrs ++ +++ - Wat/San Water and Sanitation Committee 8-10 yrs +++ ++ ++ HIV/AIDS HIV/AIDS home based care volunteer 8-10 yrs +++ ++ + Bednet ITN bednet volunteer 5-6 yrs ++ ++ + EBF Exclusive Breast Feeding volunteer 5-6 yrs ++ ++ - Care Group Care Group Volunteers 2 yrs ++++ ++++ ++++ For purposes of the MTE discussion, the SOL structure of health facilities and community-based volunteers was combined with the CSP structure, as shown in Figure 16. This graphic also shows the position and role of community leadership and the potential role of Village Health Committees in coordination of these activities. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 25 Figure 16: Project, Partner and Community Structures Capacity building takes place primarily at two levels 1) capacity building of SOL and 2) capacity building of communities. The project objectives and indicators for these components are shown in the Tables 8 and 9. The SOL sustainability table indicates that about half of the activities have been implemented, but that the documentation of their current status is weak. For example, The project would needs to improve its mapping of health areas in order to calculate and document its objective that 90% of beneficiaries in the project area will live within 5 km of a trained volunteer. In addition, some may be too ambitious. For example, the objective of having a SOL PHC staff member attend at least 80% of the VHC meetings seems overly ambitious. It is suggested, therefore, that the project review and revise these indicators as part of improving its information management system (Recommendation #10). Midterm Evaluation of the Tiweko Tose Child Survival Project Page 26 Table 8: Sustainability Objectives for Synod of Livingstonia Hospitals and Health Services Objectives Indicators/ Data Sources Planned Activities MTE Status 1. 90% of beneficiaries in the project area will live within 5 km of a trained volunteer providing health information and access to essential drugs/medical advice. 1. SOL will select volunteers through the VHCs. CSP will train and monitor the volunteers 1. Implemented, but undocumented. The project needs to improve its mapping of health area to calculate and document this objective. All beneficiaries will have access to primary health care centers. 2. By end of project, SOL will write and have funded from outside donors a grant proposal for interventions related to the sustainability of essential MCH interventions. 2. WR will provide technical assistance in proposal writing. 2. Not yet implemented. 1. 80% of care groups will have met at least four times in the last six months, 1. CSP staff will establish and monitor care groups, VHCs and AHCs. 1. Probably OK, but not well documented. Project monitoring of CGs and promoters should be improved. 80% of community groups will maintain linkages with the SOL and the MOHP. 2. SOL PHC staff will attend at least 80% of the meetings of VHCs and District Health Management Team (DHMT). 2. CSP will promote joint meetings of DHMT 2. DHMT meetings are taking place as planned (Mzimba district). Most VHCs are nonfunctional and too numerous for SOL to attend. The indicator should be redefined. 1. 80% of beneficiaries will report that they are basically satisfied with their last visit to the SOL. 1. CSP will encourage communities to express needs, acting as intermediary when required. This has been achieved. 94% of households say they were satisfied with their last visit. 80% of community members will expect and demand provision of basic services. 2. Churches, community groups, or VHCs will approach SOL concerning disrupted or poor quality services. 2. Build cooperative problem-solving skills through VHCs. Undocumented. According to the DIP, most of the capacity-building objectives directed at the community level are to be implemented during the last two project years. It is understandable, therefore, that the MTE status of these indicators is unknown or undocumented. However, some of the indicators are overly complex and would be extremely difficult to measure, e.g., “At their last meeting, 75% of the VHCs address at least 3 of 5 responsibilities.” On the other hand, the graduation of Care Groups which is an extremely important and easy to measure capacity-building indicator is not included in the chart. It is recommended, therefore, that the project review and redefine its capacity-building indicators as part of improving its information management system (Recommendation #10). Midterm Evaluation of the Tiweko Tose Child Survival Project Page 27 Table 9: Sustainability and Capacity-Building Objectives (Community Groups) Objectives Indicators/Data Sources MTE Status VHCs will monitor, plan and evaluate maternal and child health for their areas. 1. 95% of villages will have established VHCs by the EOP 2. 80% of VHCs will have met at least once in the last 2 months 3. At their last meeting, 75% of the VHCs address at least 3 of 5 responsibilities (specified in partnership section.) Staff reports, VHC meeting records, M&E Not documented. Indicator #3 would be extremely difficult to measure. Churches in the project area will actively promote behavioral changes specified in the interventions. 1. Church care groups will be established in 65% of the communities in the project area. 2. Leaders from 60% of the congregations in the communities in with church care groups will participate regularly (at least half of the meetings over a quarter-year) in the meetings. 3. 60% of the participating pastors will have delivered an intervention related health message from the pulpit in the previous month. 4. 65% of the churches with educational programs (Sunday Schools, etc) will deliver intervention related health messages to their learners in the previous three months. Church care group meeting records, M&E results. This component of the program has not yet been implemented. Planned for post￾MTE. Develop community accountability and sense of efficacy for maternal and child health. 1. Increasing numbers of community members report instances of sickness, malnutrition, and neglect to leaders, volunteers or promoters. 2. CSP staff or volunteers, or SOL staff address at least one non-project related village meeting each quarter in which at least two are held. 3. Community Advisory Boards meet quarterly. Supervisor and promoter records, M&E, visits to care groups. Undocumented. Indicator #2 would be extremely difficult to measure. The project should consider alternative indicators. Each hospital has developed its own PHC related programs and approaches. The CSP is the first community-based program to link all three hospitals, which has caused the community to consider developing Synod-wide programs. An important discussion during the MTE concerned the potential advantage of reorganizing and integrating the support and supervision structure of SOL for its community-based volunteers with that of the Care Groups and Promoters structures. More information of possible scenarios is provided in the next section of this report. Recommendation 7: Program Integration CSP and Synod: CSP and the three Health Units should consider how promoters might provide integrated, cost￾effective support and supervision for all community-based Synod health initiatives. 3. Expansion and Replication The project and Synod are striving to reach all the areas served by their units. However, some areas are difficult to reach or served seasonally. The project and the Synod are already discussing ways to improve access in difficult-to-reach project and non-project Synod areas. For example, there are several areas within Embangweni and DGMH, which are seasonally cut off from access by promoters. The project and Synod are considering hiring an additional health educator and several promoters. However, since the project budget is constrained, this might be an opportunity to consider moving an existing promoter to a new area, and transferring their current Care Groups to an HSA/promoter (see Recommendation # 9). Midterm Evaluation of the Tiweko Tose Child Survival Project Page 28 Any consideration for expansion to new project areas within Synod catchment areas or into health areas managed by the MOH, will add another level of complexity to an already complex project. It is not only a question of identifying additional resources, but also one of improving project management to handle additional responsibilities. While long-term planning could (and should) begin now, it is recommended that the project implement the MTE recommendations before making decisions about project expansion and replication into new areas. Recommendation 8: Project Expansion and Replication: CSP, the three Health Units and the Synod should examine the geographic reach of its current programs to 1) determine how to reach inaccessible areas within the current CSP project area and 2) discuss options with the Ministry of Health for expanding services to other communities and health areas. D. Sustainability and Exit Strategies The exit strategies for this project naturally focus on how best to support the work of Care Group volunteers. However, because the SOL has its own community-based activities in the project area, the question is not how the program will phase-out but how the innovations introduced through the CSP will be phased-in to the ongoing PHC programs and SOL structure. An important first level sustainability strategy is directed at Care Groups and their eventual “graduation.” A Care Group “graduates” when 60% of its members have correctly answered oral test questions. The project has not yet implemented this policy, and is still considering whether to graduate Care Groups “by intervention” or after all interventions are introduced. The combined average score of all the volunteers in a Care Group cannot fall below 60%, or they must retake training and testing. The objective is to make Care Groups as independent as possible, with the “Chief Volunteer” becoming the internal “promoter” of the Care Group. This approach worked well for Vurhonga in Mozambique, and should work equally well in Malawi. A second level sustainability strategy is directed at promoters. The project hopes that since promoters were recruited locally, that they will stay in their communities and continue supporting Care Groups after project completion, either on a volunteer basis, through locally generated incentives, or through some sort of yet-to-be-identified SOL-salaried mechanism. The primary strategy as proposed in the DIP, however, is based on having Village Health Committees take over the supervision and supporting role now provided by promoters. Unfortunately, since most VHCs are currently nonfunctional, except in the DGMH project area, VHCs will require a major revitalization effort. The evaluation team spent considerable time, therefore, examining other options, and decided that the key to continuing the work of Care Groups does not depend on being linked to a committee. Rather it depends on receiving regular encouragement and technical supervision. In particular, the evaluation discussions found that the links between Care Groups and the village headman (see Figure 17) might, in fact, be an acceptable substitute for the VHC. As mentioned earlier, the village headman is already playing a key role in authorizing Care Groups to function, in helping them deal with difficult households, and in providing them opportunities to provide BCC at community functions. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 29 Figure 17: Links from Care Groups to HSAs and Village Headmen Similarly, it was found that links between Care Groups and some HSAs have developed to the point where some HSAs are seen as an important source of technical information and supervision from the health center. Therefore, one exit strategy scenario would be for the project to transfer the “moral encouragement” support for Care Groups to village headman, and the “technical support” for Care Groups to HSAs. This is, however, only one of several possible exit strategies identified by the evaluation team. Other strategies that should be considered are: 1) Train project promoters to become HSAs. These HSA/promoters could then be paid by the MOH, but seconded to the Synod to continue their work in also supporting Care Groups. 2) Train existing HSAs to become promoters. This is, in fact, already happening in some project areas where HSAs participate in training camps with promoters. The project might consider moving an existing promoter to a new project area and allowing the HSA to continue the support to Care Groups in that geographic area. 3) Train project promoters to also be supervisors of Synod community-based health initiatives. A weakness in the current Synod system is in the infrequent supervision and support from the hospital to the community level. By having the promoter supervise and support other Synod volunteers would be a way to integrate the two programs and 4) Encourage more “Zonal Committees.” Care Groups in Embangweni organized a larger coordinating Zonal Committee with representatives from a dozen Care Groups. The leadership power of this group might be sufficient as a replacement for the promoter, if combined with strong links to village headmen and/or VHCs. Recommendation 9: Exit Strategies and Local Sustainability: CSP should consider diversifying strategies to sustain Care Groups. In addition to strengthening Village Health Committees, alternatives might include 1) regular meetings between Chief volunteers and village headmen; 2) Care Group Zone Committees; and 3) increased links to HSAs and health centers. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 30 IV. PROGRAM MANAGEMENT A. Planning Following the approval of the CSP proposal in May 2000 monthly meetings were held with representatives from the three Synod hospitals to develop a consensus for the project action plan. Additional planning meetings were also held with MOH staff from Mzimba and Rumphi District hospitals. Project objectives and activities are well understood by staff and local partners. The fact that representatives from both the Synod and the MOH participated full-time in the MTE, is in itself a good indication of the good collaboration and dialogue among partners. Project staff members have also attended the District Health Management Team meetings held regularly in Mzimba district, and somewhat less frequently in Rumphi district. B. Human Resources and Staff Management Figure 18 shows the organizational chart for the project. Table 19 summarizes the vast number of human resources that are collaborating in this project. The numbers of personnel are generally adequate except in several project areas where the project has identified a possible need for an additional Health Educators and promoters. Figure 18: Project Organizational Chart Midterm Evaluation of the Tiweko Tose Child Survival Project Page 31 Key personnel job descriptions, policies, and procedures are in place. Conditions of employment are defined by the Synod of Livingstonia Health Department policies, which are also the same for hospitals in which the program is being implemented. Staff recruitment and selection is done in liaison with relevant hospitals and are salaried by the project. Staff allowances (meals, housing, upkeep, etc) and gratuity is also provided by the project. Base hospital personnel report to respective hospital management teams through PHC Directors. Disciplinary issues are also handled in consultation with hospitals. This system appears to be running adequately. The area coordinators, health educators and accounts assistants went through a one-week basic computer course for Microsoft Word and Excel at the Ekwendeni CCAP Lay Training Center. The course was specially tailored to project needs, and has been quite helpful in improving report writing, budgeting, and financial management. Over and above whatever has been done the project recognizes a need for staff to be trained in qualitative research to build the capacity of communities to make use of HIS to evaluate progress (see HIS). Staff supervision is structured and provided according to the organizational chart (Figure 18). The periodicity of supervision at all levels has been good, including supervision of volunteers by promoters, of promoters by health educators, of health educators by area coordinators, and of area coordinators by central project staff. Feedback during supervision visits is provided to staff in a participatory and constructive atmosphere that fosters team building and cohesion. The supervision system could, however, be improved with more quantitative monitoring of volunteers and promoters using information that is already available and, in most cases, being collected (see HIS for the specific recommendation). Supervision information could be the basis of calculating simple composite indicators to monitor the performance of both volunteers and promoters. For example, a the work of a promoter could be monitored through the following individual indicators (or combined into one composite indicator): · Level of participation of volunteers in Care Group meetings and by the number · Number of household visits by volunteers · Results of quarterly surveys · Number of supervision visits to Care Groups by promoters There is an excellent spirit of cohesion and collaboration among project personnel. This has improved over time with team building exercises such as the baseline KPC, DIP preparation, and census taking in project areas. Healthy personnel relationships are also credited as a major reason for the very negligible levels of staff turnover, i.e. only two staff members have left due to personal reasons. There have been a few reported problems with the relationship and information flow between project personnel and the other hospital staff. The project is aware of this problem is continually exploring ways of improving staff conditions of service and relationships to hospital staff within the existing Synod policy framework. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 32 Table 10: Human Resources for Tiweko Tose Org Affil Position Duties No. Pay Status % Project director Oversee project 1 Paid 100% Deputy Director Oversee project 1 Paid 100% Area Coordinator Coordinate activities /supervise promoters 3 Paid 100% Health Educator Develop curriculum; train/supervise promoters 5 Paid 100% Administrative assistant Office Work 1 Paid 100% Bookkeeper Track finances 3 Paid 50% Driver Drive vehicles 4 Paid 100% Tiweko Tose CS Project Field Staff Promoter Train/supervise volunteers 43 Paid 100% Volunteer BCC to households 2950 Volunteer 10% Church Leader BCC at church meetings TBD Volunteer 10% DRF volunteer Provide first treatment 150 Volunteer 10% ITN volunteer Sell bednets/retreatment 120 Volunteer 10% CBDA Provide contraceptives 300 Volunteer 10% VHC members Link to care groups, GMC 2000 Volunteer 10% Community Health Workers Trained TBA Safe home deliveries 95 In-kind Pay 50% Synod Health Coord. Coordinate Health work Synod-wide 1 SOL 15% Synod PHC Coordinator Coordinate Synod-wide PHC activities 1 PCUSA 25% Hospital PHC Dir/Coord Oversee hospital PHC programs 3 15% MCH Coordinator Oversee MCH Activities in two hospitals 2 40% Reproductive Hlth Coord Oversee Reproductive Health in 1 hospital 1 40% Water/Sanitation Coord Oversee Water & Sanitation in 2 hospitals 2 5% Development & Relief Dir Oversee development work Synod-wide 1 Gov’t via SOL 10% Water Program Manager Coordinate water programs Synod-wide 1 PCUSA 5% Principal Clinic Superint. Technical Supervision/ Health Care 2 30% Clinical Officers Technical Supervision/ Health Care 7 20% Medical Assistants Health Care 15 20% Community Health Nurses Health Care 3 Gov’t SOL 80% SOL Employees Nurse Midwives Health Care 40 40% District Health Officer Oversees all health work in the District 2 5% MCH Coordinator Coordinate District MCH activities 2 5% Environmental Hlth Office Oversee H.S.A. supervisors; EPI activities 2 5% Health Surveillance Super Coordinate & reports activities of H.S.A. 3 20% Gov’t Employees (Mzimba and Rumphi Districts) Health Surveillance Asst Environmental, EPI, & growth monitoring 40 Govt. 20% WR Country Representative Project support 1 WRC 10% Malawi Prog Support Office Health Project support 1 WRC 15% Dir of International Health Overall guidance to health programs 1 Paid 15% Dir of MCH Overall CS program coordination 1 Paid 20% CS Program Specialist Technical Support to Tiweko Tose 1 Paid 25% WR USA Program Assistant Admin Support to Tiweko Tose 1 Paid 25% Midterm Evaluation of the Tiweko Tose Child Survival Project Page 33 C. Financial Management The project receives quarterly transfers directly from World Relief USA. A consolidated financial report in addition to a narrative report is prepared by the project and submitted monthly to World Relief Lilongwe. Budgeting is done annually with minor periodic adjustments introduced arising situations. The system, according to all those involved, is running adequately and smoothly. The CSP contracted Graham Carr, a locally based audit firm, to review the project accounting system in February 2001 to ensure that financial standards are in line with the Synod Treasury and with USAID guidelines. Their findings complied with the USAID guidelines have resulted in an improved accounting system. D. Logistics Management As the DIP so aptly states, “The project is person-intensive, not commodity-intensive.” Given the three project areas and the large number of personnel involved in this project, logistics are very important to maintaining a smoothly operating project. Vehicles and motorbike have facilitated provision of extension. All necessary personnel were provided two weeks of motorcycle training and exams. Each promoter is provided a pushbike with the understanding that ownership remains with the project. The promoters are responsible for minor repairs. Major repairs, especially new tires, are provided by the project. This system seems to be working quite well, especially in the flatter terrain of Ekwendeni and Embangweni. There are a number of project areas that are rendered inaccessible during the raining season because of the rising rivers. The project is looking into the possibility of stationing a promoter and volunteers on both sides of the river in order to ensure better continuity of services. E. Information Management (Health Information System) The quarterly surveys conducted by the project provide an excellent means of monitoring project activities. These are to be encouraged. However, there is some confusion in the definition of project indicators and standardization of their measurement. (see Recommendation #1 in “Project Objectives, Indicators and Progress.” In addition to the quarterly surveys, chief volunteers keep Care Group registers and submit monthly reports to their respective promoters. Promoters on their Friday meetings give reports and submit them to Area Coordinators and Health Educators who in turn serves as inputs into monthly reports submitted to central office by hospitals. These data have also been used to report to local leaders and solicit their support to mobilize communities. As noted above, however, there is a need to improve the HIS and monitoring system by using this information to compile performance indicators for volunteers and promoters. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 34 As noted in the section on “Malaria and Pneumonia” there is some evidence from the records of one Health Surveillance Assistant decreases in cases and deaths due to malaria, pneumonia and diarrhea. The project should increase information exchange with the MOH health and the Synod Health Units for the health areas where it is operating. For example, an examination of health facility data for the past four years could establish that the HSA reported decreases in deaths due to malaria and pneumonia are real. It should be noted, however, that the actual number of cases received by a health facility might have actually increased due to project referrals. Finally, the project could be more map-friendly. Because of the importance of the geographic location of services in this project, and the need to identify logistical barriers, it is recommended that the project increase its use of maps to monitoring project areas and activities. While maps of each project area were attached to the DIP, there is little evidence that they are being used for project planning and reporting. Careful mapping of Synod and MOH managed health areas will also be important for discussion of project expansion in collaboration with the MOH. Maps will also facilitate the presentation of the project to visitors and evaluators. Recommendation 10: Health Information System: CSP should strengthen project monitoring and information exchange. Specifically, CSP should 1) adopt standardized tools to supervise promoters and Care Groups, 2) improve information feedback to communities; 3) increase information exchange with the MOH and within Synod Health Units; and 4) improve the mapping of project areas and activities as part of the health district system. F. Technical and Administrative support Almost all of the anticipated needs for technical assistance and special training identified in the DIP took place as planned and were, according the project staff, “quite beneficial, expeditious and timely.” This included TA from external sources, WR USA Headquarters, Synod Health and PHC Coordination, Vurhonga CSP in Mozambique, and USAID Malawi. The project has also benefited from and appreciates the close and frequent support from the WR Malawi Office and Headquarters. An average of 15% of the time has been devoted to the program. Future needs for TA have already been identified in the DIP for the formation of HIV/AIDS post-test clubs, follow-on grant proposal writing, and end-of-program evaluation. In addition, based on the MTE evaluation it is suggested that the project seek TA for improving it supervision and monitoring tools. Since the work of World Relief’s Vurhonga project in Mozambique has been particularly strong in this area, it is suggested that the project send someone, perhaps the Deputy Project Director, to visit the project in Mozambique. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 35 V. CONCLUSIONS AND RECOMMENDATIONS A. Conclusions This project is a very complex child survival projects because it 1) is essentially three projects areas combined into one project; 2) is implemented through the health services of the Synod of Livingstonia; and 3) is introducing a new concept in community-based BCC in an environment that is already historical rich with community initiatives. The project is to be commended for its strategy of conducing quarterly surveys and for having completed an internal review in preparation for the MTE. These activities made it possible to complete the MTE for a complex and dispersed project in a timely manner. The evaluation team found that the project is making excellent progress in achieving its primary objectives. The project has met nearly all mid-term targets, and exceeded end of project targets for four of eleven objectives. The project has also demonstrated that the Care Group structure can be successfully implemented in Malawi as a social movement for behavior change. The project must now concentrate on consolidating and sustaining its success. Based on the MTE findings and discussions, ten major recommendations and strategies for implementation were agreed upon and discussed with project partners. B. Summary of Recommendations (see Executive Summary for the official versions) 1) Standardization of Indicators: Clarify the definition of indicators and their measurement. 2) Drug Revolving Funds: Evaluate the status of the DRF program to correct problems. 3) Obstetrical Emergency Transportation Plans: Encourage communities to develop community-wide emergency transportation plans in addition to family-specific plans 4) Incentives for Care Group Volunteers: Identify best practices for incentives to Care Groups, especially services provided to Care Groups from the community 5) Behavior Change Communications:: Target more BCC to men and community leaders, including revitalizing “Mphala,” a tradition of men to boy communication 6) BCC Materials and Training: Document best practices in BCC make these materials widely available, including durable picture codes for Care Group volunteers 7) Program Integration CSP and Synod: Consider how promoters might provide integrated, cost-effective support and supervision for all community-based Synod health initiatives. 8) Project Expansion and Replication: Determine how to reach inaccessible project and discuss options with the Ministry of Health for expanding services. 9) Exit Strategies and Local Sustainability: Diversify strategies to sustain Care Groups in addition to VHCs, e.g., regular meetings between Chief volunteers and village headmen 10)Health Information System: Strengthen project monitoring to improve supervision of promoters and Care Groups and improve information exchange with the MOH and Synod. Midterm Evaluation of the Tiweko Tose Child Survival Project Page 36 C. Results Highlight The Mid-Term Evaluation asked mothers, Care Group volunteers, project promoters, community leaders, and health center personnel if they knew any local proverbs that might aptly describe the work of the Tiweko Tose Child Survival project. Given the rich heritage of proverbs in the Malawian culture, it is not surprising that there were some very interesting proposals. Here are the ones that the evaluation team found to be particularly appropriate. One proverb described a primary objective of the project: Children of today are the leaders of tomorrow. Another proverb noted the importance of responding to behavior change communications. The “weapons” in this proverb were considered to be synonymous with “your children.” If you delay in taking action, someone may snatch your weapons from you. A third proverb spoke of improving the health of the community while resources, like project promoters, are available: Bask in the sun while it is shining. Another proverb spoke of the household visits and health messages from Care Group volunteers: The roar of a lion is heard from afar, while that of a cat is near. A fifth proverb spoke about the importance of collaboration between the four project partners – the community, World Relief, the Synod of Livingstonia and the Ministry of Health. One head cannot carry a roof. And finally, this last proverb creates another very vivid image of the importance of partnership. You can’t kill lice with one thumbnail. It takes two to crush a louse. Annex A: Baseline information from the DIP (There have been no substantial changes in this project since approval of the DIP) I. Executive Summary The TT CSP Detailed Implementation Plan (DIP) indicates that the project will spread its efforts on across all of the USAID-funded child survival interventions. It was agreed during DIP review that immunizations and diarrhea interventions would be integrated with general nutrition and Maternal and New born care. Field Program Summary Intervention % of Total Effort (a) % of Total USAID Funds in US $ Immunization 0% 0 General Nutrition (including breastfeeding) Immunization Control of Diarrheal Diseases 9% 3% 3% 90,000 30,000 30,000 Vitamin A 5% 50,000 Micro nutrients (iron folate) 5% 50,000 Control of Diarrheal Diseases 0% 0 Pneumonia Case Management 15% 150,000 Control of Malaria 15% 150,000 Maternal and Newborn care 15% 150,000 Child Spacing 15% 150,000 STI/HIV/AIDS Prevention 15% 150,000 Others (specify) 0% 0 Total 100.% 1,000,000 (a) Percent of total effort and USAID funding for each intervention. Tiweko Tose is a NEW program that began October 1, 2000 and will continue through the end of September 2004. USAID will provide $1,000,000 and World Relief (WR) will contribute $333,334 over the life of the project. II. Program Goals and Objectives The project goals are to: 1) Reduce morbidity and mortality in children under 5 and women of childbearing age; 2) Strengthen the capacity of the SOL to implement Child Survival interventions; and 3) Empower communities to improve their health. Three primary areas of intervention contribute to the realization of said program goals include 1) Malaria and Pneumonia; 2) Nutrition and Micronutrients; and 3) Reproductive Health (Child Spacing, Maternal and Newborn Care, and STI/HIV/AIDS). Specific objectives and strategies for each of the intervention areas are as follow. Malaria/Pneumonia CSP activities will produce the following results: Result #1: Improved rapid recognition and treatment of suspected malaria and pneumonia cases. This will be achieved through education at household level, improving treatment in community (DRF volunteers) and health facilities (IMCI training), and by improving drug supply. Additionally, the CSP will direct the procurement and overall distribution (sales) of ITNs for the entire project area, by working with Area Coordinators as they train and supervise their ITN volunteers. In the community, ITN committees will help organize the periodic re-treatment of nets and sales of nets in the community. (People are allowed to buy nets on credit and pay in kind (with grain from harvest.) Area Coordinators will liaise with local retailers and shop keepers to assure that drugs are used as wisely as possible for treatment of malaria and pneumonia, and encourage them to get training to be able to assess serious illness (rapid, difficult breathing) and refer to health center for treatment. Objectives: · Increase from 35.4% to 50% the percentage of children <5 who are treated within 24 hours (same or next day) for rapid, difficult breathing (with or without fever) at an IMCI￾trained health facility or given first dose by a trained DRF PCM volunteer. · Increase from 30% to 60% the number of pregnant women who receive at least 2 doses of malaria prophylaxis SP (Sulfadoxine Pyrimethamine) during pregnancy; and 80% who get 1 dose. Result #2: SOL personnel will treat according to IMCI guidelines UNICEF and MoHP are training health personnel in Mzimba District (includes Ekwendeni and Embangweni) in IMCI. Result #3: Prevent malaria through promotion of ITNs via: · Increased awareness among community members and health workers about the importance of malaria prevention in children and pregnant women, · Presence of sustainable revolving fund for purchase of nets and insecticide operated by members of the community. · An established system for re-treatment of nets in villages, so that at least 75% of nets are retreated every year. · Increased use of nets, so that 50% of children <5 and pregnant women sleep under a net regularly. Objectives for malaria prevention · Increase from 8.5% to 50% the number of children < 5 and pregnant women sleeping under a bednet. · Increase from 62% to 75% the bednets retreated within the last 12 months. Result #4: Improved access to treatment by training one DRF volunteer per area (group of villages- DRF will be centrally located to be a reasonable walk for most women). Nutrition and micronutrients objectives: · Increase from 65% to 90% the number of children 0-35 mo. weighed regularly in growth monitoring and counseling (GMC) sessions. · Increase from 3.0% to 30% the percent of pregnant or lactating women who receive daily iron/ folate supplementation for a minimum of 3 months. · Increase from 3.8% to 30% the percent of children 6-23 months old who will receive weekly iron supplement. · Increase from 12.9% to 50% the percent of children 6-59 months old who will receive appropriate dose of Vitamin A capsules (2 per year.) · Increase from 36% to 50% the proportion of mothers EBF 0-6 month old infants. Although diarrhea is not an explicit area of intervention, mothers will be taught relevant messages as they pertain to nutrition. For example, exclusive breastfeeding will be touted as an effective means of preventing diarrhea in infants under six months of age. Volunteers will also promote hygienic practices such as hand washing after defecation and before food preparation and feeding. Mothers will learn how to hydrate and feed their sick children via the immediate provision of Oral Rehydration Salts (ORS) or home available fluids and offering of small frequent meals. Signs of serious illness such as blood in stool, fever, or persistent diarrhea (lasting 7 days, or less if condition is poor) will be taught as indications for referral to a health center. During recovery from illness, mothers will give their children additional feedings. Reproductive Health strategies include promotion of child spacing, maternal and newborn care and HIV/AIDS. In order to increase use of modern methods of family planning to 40%, volunteers will educate mothers and the promote use of both female and male CBD volunteers (who will educate men) and distribute condoms and pills. As a result of this promotion, 30% of sexually active women will report using a condom during their most recent sexual intercourse. The CSP staff will train volunteers to provide initial information to mothers; more detailed counseling and screening will be done by trained nurses at monthly mobile clinics and at the SOL. Volunteers will follow up on defaulters and new users. Maternal and newborn care objectives are: · Increase from 30% to 60% the number of pregnant women who receive at least 2 doses of malaria prophylaxis SP (Sulfadoxine Pyrimethamine) during pregnancy. · Of women who do not deliver in hospital, increase from 38% to 70% those delivered by a trained and supervised TBA. · Increase to 50% the proportion of families who have an emergency transport plan in place before delivery. The CSP will provide in-service training for active, untrained TBAs, and volunteers will educate families about every pregnancy being a potential maternal risk. AIDS program results and activities are: to educate community members and increase awareness for voluntary counseling and testing. The CSP will also increase to 1,000 every 6 months the number of people getting HIV voluntary counseling and testing at the SOL. Secondly, we want to improve social support for Persons Living With AIDS (PLWAs); by starting 5 post-test clubs in each area (support groups for post-HIV test takers, regardless of status). III. Program Location The project area for Tiweko Tose includes three separate geographic areas, each served by one of three hospitals run by the Synod of Livingstonia (SOL) of the Church of Central Africa, Presbyterian (CCAP) in Northern Malawi. SOL hospitals are integrated into the Ministry of Health and Population (MoHP) and operate in portions of Mzimba and Rumphi health districts. At present, each hospital has its own PHC program. The SOL specifically requested that the CSP cover all three catchment areas in order to increase their organizational capacity for continuing community work. Headquarters for the SOL Health programs, including Tiweko Tose CSP, is centrally located in Mzuzu, a two-hour drive from the border of the furthest project area. Ekwendeni, in the northern part of Mzimba District, has 10 PHC areas and is near Mzuzu, the third largest town in Malawi. Embangweni borders Zambia south and west, and has 16 PHC areas and three health centers: Kalikumbi, Mabiri, and Mpasazi. David Gordon Memorial Hospital (DGMH) in the isolated Livingstonia catchment area is 3 hours north of Ekwendeni in the northeast of Rumphi District; DGMH has nine PHC areas, three health centers at Luwuchi, Mlowe and Tcharo and a health post at Zunga. Tcharo and Zunga are only approachable by boat or by foot, as there are no roads serving them. Tiweko Tose staff conducted a census in February 2001 and found that the total population equaled 161,596. Though the census yielded a similar total population to the proposal estimate, there were many more children than estimated in the proposal. There are 68,917 potential project beneficiaries; 36,732 0-5 year olds (22% of population) and 32,185 childbearing age (CBA) women (20% of general population/ 44% of females); An additional 6,400 babies will be born each year (25,600 over 4 years). Ekwendeni; 8,422 households Program Site Population: Children and Women Age Female Male Total 0-11 months 942 968 1,910 1-5 years 3,828 3,545 7,373 6-14 years 5,367 5,458 10,825 15-49 years 9,113 8,248 17,361 Over 49 years 2,573 1,979 4,552 Total 21,823 20,198 42,021 Embangweni; 20,489 Households Age Female Male Total 0-11 months 1,852 1,883 3,735 1-5 years 5,941 5,984 11,925 6-14 years 9,900 10,462 20,362 15-45 years 14,942 13,978 28,920 Over 45 years 5,115 4,117 9,232 Total 37,750 36,424 74,174 David Gordon Memorial; 7,634 households Age Female Male Total 0-11 months 1,188 1,428 2,616 1-5 4,028 5,145 9,173 6-14 5,827 5,793 11,620 15-45 8,130 7,601 15,731 Over 45 3,255 3,006 6,261 Total 22,428 22,973 45,401 IV. Program Design The overall program design of Tiweko Tose is community-based, volunteer-driven, and builds upon the strengths of World Relief’s partnership with SOL. SOL strengths include that they have established credibility in the project area, have already fielded some PHC programs, and will continue to be the predominant provider of health care services long after the conclusion of the CSP grant. The major project strategy is to work in partnership with SOL and the MOH to help communities create 300 Care Groups and train 3,000 community volunteers as behavior change agents. Care Group volunteers make regular visits to more than 30,000 households to explain and promote key messages for each child survival intervention. The project intervention mix includes Nutrition (25%); Malaria control (15%); Pneumonia case management (15%); Maternal and newborn care (15%); Child spacing (15%), and the prevention of STI/HIV/AIDS (15%). The project also includes capacity building of SOL to strengthen its organizational capacity to manage child survival activities and to increase/reinforce other community-based and health facility-based services, e.g., traditional birth attendants (TBAs), drug revolving funds (DRF), insecticide-treated bednets (ITN), and community based distribution agents (CBDAs). These activities are conducted in collaboration with, and as part of, the MOH district health system. Tiweko Tose has adopted and adapted the Care Group strategy that was pioneered by World Relief’s Vurhonga Child Survival Project in Mozambique. The Vurhonga project found that the Care Group structure transformed behavior change from individual decisions to a social movement. Care Groups became a source of encouragement and social support for volunteer mothers. They became a well-accepted and sustainable community institution. V. Partnerships The SOL is the implementing partner of Tiweko Tose CSP. A signed agreement between WR and the Synod is in Annex D. Child Survival staff relate both to the PHC department of each hospital as well as to the Tiweko Tose Project Director and Deputy Director, who provide central leadership for CSP implementation. Only the Project Director is a WR employee, seconded to SOL for the duration of the grant. The SOL continues to serve as the major health care provider in the catchment area via its hospitals, health centers and mobile clinics that are integrated into the government health system via the Christian Health Association of Malawi. Each hospital will continue to be responsible for TBAs, CBDAs, DRFs, and ITN committees in conjunction with applicable government guidelines. (Training of TBAs and CBDAs is regulated and implemented by the government.) However, because such community-based health workers and volunteers are unevenly distributed and the CSP is likely to increase demand for them, the Synod PHC Coordinator will assist the hospitals to seek funds for training additional people, supplying drug kits and the like. The role of the MOHP does not change as a result of the CSP. It continues to work in concert with the SOL to address the health needs of the population. The MOHP pays salaries of designated hospital and health center staff and provides commodities through Central Medical Stores including basic drugs, vaccines, micronutrients, and contraceptives. A major goal of Tiweko Tose is to empower communities to improve their health. Consequently, every village in the catchment area is a partner in the CSP. The CSP is not something that will happen to the villages; rather it is commitment at the village level that will give life to Child Survival. Villages will field the extensive network of volunteers that serve as the conduit for behavior change communication. Additionally, villages will take turns helping to host training camps for the promoters who train their volunteers. Formal agreements in writing at this stage would not be conducive to continuing to build rapport, but villagers have had input with regard to plans for Tiweko Tose from the beginning. No activities will take place without the approval of village leadership. VI. Health Information System From Sections II: E and II: H of the DIP (“Information Management” and “”), describe the program's proposed health information system and the mechanism for program monitoring. A. Information Management 1) Quarterly HIS Monitoring and Evaluation: The quality, effectiveness and coverage of the interventions are monitored through these quarterly rapid assessment surveys (the M&E plan), which use KPC survey questions to monitor most knowledge, practices and coverage indicators. Some questions may be asked every quarter while others may be rotated, keeping the survey short (10 minutes to complete) and focused. Every three or four months, one Care Group from each of the 43 promoters is randomly chosen, then two volunteers are randomly selected from the group, and all the families that they are responsible for are surveyed, approximately 25 households per care group. To preserve objectivity, the promoters are assigned to different areas to conduct the home surveys. The resulting data set of 1,000+ interviews is hand-tabulated and summarized by the promoters in the field, then analyzed and reported project-wide to chart progress on objectives. Promoters and supervisors will also use this time as an educational opportunity; the volunteers benefit from immediate feedback, and the results are left with each promoter for them to review with their Care Group. 2) Census Information: The initial census of all mothers and children in the project area provides a highly reliable estimate of the number of beneficiaries included in the program. The census also prevents the inadvertent exclusion of poorer or more marginalized groups from the project interventions. The census also provides the baseline for the health information system maintained through the care group structure. The HIS will maintain accurate records of population-based vital events. 3) Health Information System: The existing HIS in the project area varies by hospital as well as by District. UNICEF is introducing community registers in Mzimba District but not Rumphi. Health facilities have registers of under-fives and women of reproductive age—but lack data from private practitioners, TBAs and others. Little feedback is given to the community. In villages where records are already being kept, the statistics initially can be compared with those tallied by care groups. The care groups will of course be sharing their data with the village on a periodic basis. If the data corroborate over time, the Care Group could limit its data collection to only those stats not recorded elsewhere, relying on the existing village system for supplemental data that they would discuss in their Care Group meeting. Data collected by the care groups will be limited to vital events (births, deaths, pregnancies, families visited) and those mothers and children who require follow-up by volunteers. The simplicity of the data collected and relatively few number of households that each volunteer is responsible for, allows for verbal reports by the volunteer to the care group secretary. B. Monitoring and Evaluation There are four processes – formal evaluations, intervention specific research, cycles of organizational learning, and the promotion of personalized feedback loops – which World Relief and the Synod of Livingstonia will combine to provide a richly detailed picture of the Tiweko Tose program. 1) Formal Evaluative and Monitoring Cycles: These include the proposal and DIP reviews, the midterm (August 2002) and final evaluations (September 2004) led by external evaluators and the three KPC surveys that punctuate the progress of the project. 2) Intervention Specific Research: Each of the project interventions will require specific qualitative approaches to inform program and BCC strategy. The Hearth program, for instance, will use 24 hour diet recalls in each village to identify locally sustainable nutritious diets, that are continually assessed to take into account regional and seasonal changes. The feedback loop for intervention specific research is short and extremely practical. Knowledge about health services and the needs and practices of people will be used to revise objectives and plans at several key junctures: 1) during objective-setting at the DIP, after the KPC survey, 2) during development of curriculum, (Training of Trainers, TOT, with promoters, and when they train Care Groups) and 3) after midterm evaluation, monitoring visits, and to a lesser extent after each quarterly assessment. 3) Organizational Learning Cycles: Each workweek will have meetings when promoters, supervisors and technical staff discuss progress, results and problems. These meetings function as quality assurance (QA) meetings where project staff identify opportunities for improvement, define problems, establish desired outcomes and plan steps to achieve them. In monthly cycles, the Project Director or Deputy Director will attend District Health Management Team meetings to share feedback. Director and/or Deputy Director also attend monthly Synod Health Department meetings. The M&E and training camps mark quarterly cycles of information gathering and organizational reflection. The Project Director is responsible to collect and analyze data on limiting factors and will plan corrective action and monitor results. 4) Personalized Feedback Loops: Personal feedback is essential to the cultural fit and flexibility of program interventions as well as to continuous cycles of quality assessment. All Tiweko Tose staff and project beneficiaries speak Tumbuka, greatly enhancing communication and cultural understanding. In Tiweko Tose, all Area Coordinators, Health Educators, promoters, and volunteers will live in the geographic area they are serving. Thus there will be close supervision of promoters and volunteers through frequent interaction and observation, guided by supervisory forms and project indicators. The Project Director, Deputy Director, and Area Coordinators will regularly liaise with PHC staff at each hospital to maintain open lines of communication and understanding. A number of monitoring and evaluation tools have already been described in the context of their use. Those that have not yet been created will be developed and pre-tested by the management team in conjunction with those people who will be using them. They include performance Evaluation forms for all levels of staff and records for monitoring of activities and measuring progress towards objectives. Annex B: MTE Team Members and Resources Persons ______________________________________________________________________________ NAME POSITION AFFILIATION EVALUATION TEAM Dr. Frank Baer External Consultant Baertracks, USA Raymond E Mkandawire Assistant Environmental Rumphi District Health Officer Hospital (MoHP) Ketwin C Kondowe District Environmental Mzimba District Health Office Hospital (MoHP) Kalinde Chindebvu Monitoring & Evaluation USAID Malawi Officer Mission Anbrasi Edward Maternal & Child Health World Relief HQ Programs Director Young Soko Primary Health Care David Gordon Director Memorial Hospital (SOL) Jeffrey Mwala Primary Health Care Embangweni Hospital Coordinator (SOL) Marko Chirwa Soil Food & Health Ekwendeni Hospital Communities Coordinator (SOL) Stella Kasirye Country Director World Relief Malawi Georgina Chinula Project Support World Relief Malawi Officer Joyce Ng’oma Area Coordinator TTCSP Embangweni Agnes Hara Area Coordinator TTCSP Ekwendeni Chimwemwe Lungu Area Coordinator TTCSP DGMH Paul Mkandawire Deputy Director TTCSP Mzuzu Victor Kabaghe Director TTCSP Mzuzu RESOURCE PERSONS Rodney Nyirenda Health Educator DGMH Fletas M Ziyewo Health Educator DGMH Glyn K Gondwe Health Educator Ekwendeni Hospital Simon Sibanda Health Educator Embangweni Hospital Esther Chirambo Health Educator Embangweni Hospital Annex C: The Mid-Term Evaluation Methodology The MTE is an opportunity to evaluate accomplishments to date, and to listen stakeholders at all levels, e.g., mothers, community leaders, health workers (both project and MOH), local partners, and donors. The MTE usually includes an external consultant and other PVO and resource persons. The mid-term evaluation focuses on the process on program implementation to: $ Assess achievement of objectives, benchmarks, project approach and work plan; $ Determine if the approach used and interventions will be sufficient to reach desired outcomes; $ Identify barriers to achievement of objectives; and $ Provide recommended actions to sustain and improve the project. I. Technical Interventions A. Program Interventions (Nutrition, Malaria, Pneumonia, MCH, child spacing, STI/HIV/AIDS). 1. Compare activities to those as proposed in the work plan; explain changes if any. 2. Assess progress toward benchmarks or intermediate objectives 3. Discuss effectiveness including special outcomes, unexpected successes or constraints 4. Propose recommendations for improvement and/or next steps 5. Describe new tools, innovative approaches; operations research or special studies and actions taken. B. Cross Cutting Approaches (address each section as applicable) 1. Community Mobilization (Care Groups of Volunteers and Church Leaders) a. What kinds of community mobilization activities have been undertaken? b. How has the community responded? How have project plans been refined program? c. What community barriers exist for program interventions? How have these been addressed? d. What other factors impact program implementation (security, politics, roads, mass media)? 2. Communications for Behavior Change (BCC) a. Is the program's approach to behavior change appropriate and effective? b. Are the messages technically up-to-date? Have any essential messages been omitted? c. How is behavior change being measured? Describe tools used (appropriate? effective?) d. How have stakeholders used the BCC data to reinforce or promote other behavior changes? 3. Capacity Building (World Relief , SOL, and other Local Partners) a. Identify local partners and discuss organizational capacity building efforts planned for them. b. Discuss the roles and responsibilities of each partner and any changes from the DIP c. Describe Institutional Strengths Assessments (formal or informal) that have been conducted. d. How have organizational capacities changed? What interventions contributed to those changes? e. What are the primary challenges for further capacity building of partners? f. Health Facilities and Health Worker Strengthening (1) Discuss linkages between health facilities, communities and the project. (2) Assess approaches for strengthening health facility and worker performance. (3) How have assessment results been used to address performance gaps? (4) Discuss training strategies, progress in knowledge and skills, and effectiveness. C. Sustainability Strategy 1. What is the progress in meeting the sustainability objectives articulated in the work plan? 2. Has the groundwork for the exit strategy been laid with project staff and local partners? 3. What approaches exist to build financial sustainability? 4. What does the community say about sustaining services through alternative funding sources? 5.Program Management 6. Planning a. What groups have been involved in program planning? b. Is the work plan submitted in the work plan on schedule? c. Are the program's objectives understood by, all staff, local level partners, and the community? d. Do all parties have a copy of the program's objectives and the monitoring and evaluation plan? e. To what extent are monitoring data used for planning and/or revising program implementation? 7. Human Resources and Management, Training and Supervision a. Discuss the program's personnel management system b. Are key policies, job descriptions, and procedures in place for all positions at all levels? c. Describe staff training to improve knowledge, skills and competencies d. How is trainee performance in new skill areas monitored? Are resources adequate? e. Are numbers, roles, and workload of personnel and frequency of supervisory visits appropriate? f. How effective is supervision, leadership, planning and problem solving to support staff? g. Describe the morale, cohesion, working relationships and impact on program implementation. h. Describe staff turnover and its impact on program implementation. 8. Financial and Logistics Management a. Discuss the management and accountability for program finances, budgeting, and financial planning for sustainability of both the program and local NGO partners. b. What impact have logistics and procurement had on the implementation of the program? c. What logistics challenges will the program face during the remainder of the program? 9. Information Management a. Is there a systematic way of collecting, reporting and using data at all program levels? b. Does the program use data to inform management decision-making? Describe methods and use? c. Describe types and frequency of data generated, and who collects and analyzes the data. d. Does the program use and support other existing data collection systems? If so, describe. 10. Technical and Administrative support a. Discuss types, sources and timeliness and benefits of external technical assistance. b. What is the anticipated technical assistance needs for the next two years? c. Discuss PVO/headquarters and regional support, and the approx. time devoted to this program. Annex D: MTE Questionnaires for Field Visits Discussions Questions for Health Facilities and HSAs 1) Are you familiar with the Tiweko Tose Child Survival Project? If so, what are its main objectives and activities? How effective do you think the project has been in reaching its objectives? Is there anything the project is doing that you are unhappy with? 2) Are you receiving information from the project and/or community? If so what information? If not what information would you like to receive? 3) What outreach exists from you health facility into surrounding communities? Which outreach activities are assisted by Care Group volunteers? How effective is that collaboration, how can it be improved? 4) Has this project assisted your health facility? If so, how? How effective was the assistance that you received? How might it be improved? 5) What is the role of HSA with respect to Care Groups and promoters? How might this be further developed? 6) Did HSAs from your health facility attend a project organized training? If so how effective was this training? 7) What gaps, problems have you encountered with at your health facility? What gaps, problems have you encountered with your work with Care Groups? 8) Do you have any other comments, questions or recommendations for this project? Also, try to identify: Special stories about the work of this project Local proverbs that illustrate the work of this program Questions for Local Leaders and Village Health Committees 1) Are you familiar with the Tiweko Tose Child Survival Project? If so, what are its main objectives and activities? How effective do you think the project has been? Is there anything the project is doing that you are unhappy with? 2) Compare the health of your community now with two years ago. Has the project made other non-health contributions in your community? 3) What role do you play in this project? Probe: in facilitating project work in your communities 4) What do you think about Care Groups? What contributions have you made to the work of Care Groups? 5) Are men adequately receiving Behavior Change Communications? If so, what methods are being used? If not, how could this be improved? 6) What do you consider to be a medical emergency? Who do you do for these cases? Who should be in change of organizing an emergency transportation system? 7) What structures can the community put into place to sustain the work of Care Groups? 8) Do you have any other comments, questions or recommendations for this project? Also, try to identify: Special stories about the work of this project Local proverbs that illustrate the work of this program Questions for Groups of Mothers 1) Compare the health status of your family now with what it was two years ago? To what do you attribute these changes? 2) Have you been visited by a Care Group volunteer? If so, what did she/he do? What did you find to be the most useful? What do you like about these visits? What do you not like about these visits? What contacts have you had with the Care Group volunteer outside of a visit to your home? 3) Have you changed any health behavior as a result of these visits? If so, explain? 4) Are men adequately receiving Behavior Change Communications? If so, what methods are being used? If not, how could this be improved? 5) What do you consider to be a medical emergency? Who do you do for these cases? Who should be in change of organizing an emergency transportation system? 6) Are there any problems you experienced as a result of participating in this program? 7) Do you have any other comments, questions or recommendations for this project? Also, try to identify: Special stories about the work of this project Local proverbs that illustrate the work of this program Questions for Care Groups 1) Compare the health status of your family now with what it was two years ago? To what do you attribute these changes? 2) How many visits did the promoters make in the last month? 3) What types of support do you receive from the promoters? What support has been the most useful? 4) What support do you receive from the headman? What support has been the most useful? 5) What support do you receive from the HSA, what was useful? What support has been the most useful? 6) Are men adequately receiving Behavior Change Communications? If so, what methods are being used? If not, how could this be improved? 7) What do you consider to be a medical emergency? Who do you do for these cases? Who should be in change of organizing an emergency transportation system? 8) What do you like about being part of a Care Group ? What do you not like about being part of a Care Group ? Are there any problems you experienced as a result of participating in this program? 9) How do you plan to sustain the Care Group when the program is completed? 10) Do you have any other comments, questions or recommendations for this project? Also, try to identify: Special stories about the work of this project Local proverbs that illustrate the work of this program Questions for Promoters 1) Compare the health status of your family now with what it was two years ago? To what do you attribute these changes? 2) How many visits did the health educators make in the last month? 3) What types of support do you receive from the health educators? What support has been the most useful? 4) What support do you receive from the central office? What support has been the most useful? 5) What support do you receive from local leadership? What support has been the most useful? 6) What support do you receive from the other programs? What support has been the most useful? 7) How do you collaborate with the health centers and HSAs? 8) What do you like about being a promoter? What do you not like about being a promoter? Are there any problems you experienced as a result of participating in this program? 9) Are men adequately receiving Behavior Change Communications? If so, what methods are being used? If not, how could this be improved? 10) What do you consider to be a medical emergency? Who do you do for these cases? Who should be in change of organizing an emergency transportation system? 11) Do you have any other comments, questions or recommendations for this project? Also, try to identify: Special stories about the work of this project Local proverbs that illustrate the work of this program Annex E: MTE Schedule and Contact List Day Activity Aug 19 Monday Team Leader arrival in Malawi Preliminary discussion of evaluation methods and schedule Aug 20 Tuesday Evaluation team orientation Review evaluation objectives; Brief review of survey results; Develop questions for field visits; Field Visits to three project areas Project staff will visit other areas where they do not normally work. Aug 21 Wednesday Ekwendeni Ekwendeni Enyezini Embangweni Kalikumbi HC Mtuzuzu David Gordon Mem Hosp Luwuchi HC Chitimba HC Chakaka Kambundi Aug 22 Thursday Ekwendeni Dunduzu Engcongolweni Embangweni Kawaza Chizimya David Gordon Mem Hosp Chakaka Chiweta Chitimba Zowo Aug 24 Friday Discuss findings; Draft tentative recommendations Aug 25 Sunday Combined Field Reports; Consensus on Recommendations Aug. 26 Monday Review Findings and Recommendations with entire team Anbrasi and Stella depart Aug 27 Tuesday Project Management Assessment with WR/LLW Aug 28 Wednesday Debriefing with local authorities in Mzuzu Travel to LLW Aug 29 Thursday Debriefing with USAID Team Leader departure from Malawi List of Contacts during Field Visits DGM Embanqueni Ekwendeni Health Facilty Personnel Luwuchi Health Center, Synod Chitimba, Health Center, MOH Kalikumbi Health Center, Synod Ekwendeni Hospital, Synod 12 HSAs interviewed Local Leaders: Chakaka Kambundi Mtuzuzu Enyezini Approx 55 community leaders Promoters 12 13 13 38/45 Care Groups Chakaka Chiweta Chitimba Zowo Kawaza Chizimya Dunduzu Engcongolweni Approx. 100 volunteers Mothers Chakaka Chitimba Nkhombwa Kawaza Chizimya Dunduzu Engcongolweni Approx 150 mothers Annex F: Action Plan for Implementation of MTE Recommendations Summary of Recommendation (See Executive Summary for Official Recommendations) Who is Responsible for Action? Proposed Completion Date 1. Standardization of Indicators: Clarify the definition of indicators and their measurement. Project staff & WRM, SOL Health/PHC Coordinator October 31, 2002 2. Drug Revolving Funds: Evaluate the status of the DRF program to correct problems. DHMT, Project Staff & SOL PHC November 30th, 2002 3. Obstetrical Emergency Transportation Plans: Encourage communities to develop community-wide emergency transportation plans in addition to family￾specific plans Project staff &SOL PHC March 31th, 2003 4. Incentives for Care Group Volunteers: Identify best practices for incentives to Care Groups, especially services provided to Care Groups from the community DHMT, Project staff, SOL PHC & Local Leaders November 30th, 2002 5. Behavior Change Communications:: Target more BCC to men and community leaders, including revitalizing “Mphala,” a tradition of men to boy communication Project staff, promoters, Care groups, HSAs, SOL PHC, Local leaders& Men End of FY2003 6. BCC Materials and Training: Document best practices in BCC make these materials widely available, including durable picture codes for Care Group volunteers CSP staff, WRM & MOH End of FY2003 7. Program Integration CSP and Synod: Consider how promoters might provide integrated, cost￾effective support and supervision for all community￾based Synod health initiatives. MOH, Project Staff, SOL PHC, Local Leaders, Promoters, care groups End of Project, 2004 8. Project Expansion and Replication: Determine how to reach inaccessible project and discuss options with the Ministry of Health for expanding services. Synod Secretariat, MOH, CSP, SOL Health/PHC Coordinator, SOL PHC End of Project, 2004 9. Exit Strategies and Local Sustainability: Diversify strategies to sustain Care Groups in addition to VHCs, e.g., regular meetings between Chief volunteers and village headmen MOH, Project Staff, SOL PHC, Local Leaders, Promoters, care groups April 30th , 2004 10. Health Information System: Strengthen project monitoring to improve supervision of promoters and Care Groups and improve information exchange with the MOH and Synod. Projects staff, SOL PHC, Promoters & care groups April 30th , 2003