CS-15 Malawi Final Evaluation Cooperative Agreement No.: FAO-A-00-99-00050-00 The Partnership for Equity, Access and Quality (PEAQ) Project: Ensuring the Pathway to Survival Balaka District, Southern Region, Malawi 30 September 1999 – 30 September 2003 Submitted By: Save the Children Federation, Inc. 54 Wilton Road Westport, CT 06880 Telephone: 203/221-4000 Fax: 203/221-4056 Contact Person: Eric Swedberg, MPH, Child Survival Specialist Report Prepared by Marc Debay On behalf of the Final Evaluation Team December 31, 2003 CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children i Table of Contents A. Summary................................................................................................................................ 1 B. Assessment of Results and Impact of the Program .............................................................. 1 1. Results: Summary Chart ....................................................................................................... 1 2. Results: Technical Approach ................................................................................................ 5 a. Overview of Project.......................................................................................................... 5 b. Immunization ................................................................................................................... 7 c. Maternal and Newborn Care ............................................................................................. 8 d. Child Spacing ..................................................................................................................10 e. STI/HIV/AIDS................................................................................................................11 f. Breastfeeding/Diarrhea/ARI/Malaria ...............................................................................15 3. Results: Cross-cutting Approaches.......................................................................................17 a. Community Mobilization .................................................................................................17 b. Behavior Change Communication ...................................................................................18 c. Capacity Building............................................................................................................19 d. Sustainability ...................................................................................................................22 C. Program Management......................................................................................................... 23 1. Planning ..............................................................................................................................23 2. Staff Training ......................................................................................................................23 3. Supervision of Program Staff...............................................................................................24 4. Human Resources and Staff Management............................................................................24 5. Financial Management.........................................................................................................25 6. Logistics..............................................................................................................................25 7. Information Management.....................................................................................................26 8. Technical and Administrative Support .................................................................................26 9. Manageme nt Lessons Learned .............................................................................................27 D. Conclusions and Recommendations.................................................................................... 28 E. Results Highlights................................................................................................................ 30 ANNEXES..................................................................................................................................... 31 A. Team Members and Persons Interviewed........................................................................... 32 B. Methodology ........................................................................................................................ 33 C. Documents Reviewed........................................................................................................... 36 D. Final Evaluation Schedule ................................................................................................... 37 E. Group Interview Guidelines................................................................................................ 38 F. Field Visits Findings ............................................................................................................ 42 G. Implementation Timeline – FY00 to FY03 ......................................................................... 50 H. Quarterly Input Monitoring Indicators .............................................................................. 52 I. Number of Trainees by Topic and Target Group at Midterm and Project End ............... 54 J. Training Events Attended by SC staff................................................................................ 55 K. Objectives Monitoring Indicators ....................................................................................... 56 L. Selected HMIS Data ............................................................................................................ 59 M. Final Knowledge, Attitudes, and Practices in Health Survey Report N. Final Health Facility Assessment Report CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children ii Acronyms List AIDS Acquired Immune Deficiency Syndrome ARI Acute Respiratory Infection BA Bicycle Ambulance BBA Benzyl Benzoate Application BCC Behavioral Change Communication BCG Tuberculosis Vaccine BF Breastfeeding BFHI Breastfeeding Health Initiative CAC Community AIDS Committee CBDA Community Based Distribution Agent CHAM Christian Hospitals Association of Malawi CHBC Community Home-Based Care CMM Community Male Motivators CORE Child Survival Collaborations and Resources Group CPR Contraceptive Prevalence Rate CS Child Survival CSHGP USAID’s Child Survival and Health Grants Program DAC District AIDS Coordinator DACC District AIDS Coordination Committee DHMT District Health Management Team DHO District Health Office DHPCC District Health Program Coordination Committee DIP Detailed Implementation Plan DRF Drug Revolving Fund EBF Exclusive Breastfeeding EPI Expanded Program for Immunization EPMU Expanded Project Management Unit FE Final Evaluation FP Family Planning GM Growth Monitoring GTZ German Technical Cooperation HBC Home-Based Care HBCC Home-Based Care Counselors HC Health Centre HIS Health Information System HIV Human Immunodeficiency Virus HMIS Health Management Information System HO Home Office of Save the Children HSA Health Surveillance Assistant HZC Health Zone Coordinator IEC Information, Education, and Communication IFA Iron Folate IMCI Integrated Management of Childhood Illnesses KAP Knowledge, Attitudes and Practices KPC Knowledge, Practice and Coverage LQAS Lot Quality Assurance Sampling CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children iii MCH Maternal and Child Health MNC Maternal and Newborn Care MOHP Ministry of Health and Population MTE Midterm Evaluation NAC National AIDS Committee NGO Non Governmental Organizations OVC Orphans and Vulnerable Children ORS Oral Rehydration Solution ORT Oral Rehydration Therapy PEAQ Partnership for Equity, Access and Quality PLG Program Learning Group of Save the Children PMU Project Management Unit QA Quality Assurance SC Save the Children Federation Inc., USA SC/Malawi Save the Children Federation, Inc./Malawi Field Office SHA Senior Health Advisor SM Safe Motherhood SP Sulfadoxine-Pyrimethamine STI Sexually Transmitted Infection TBA Traditional Birth Attendant TOT Training of Trainers TT2 Tetanus Toxoid Vaccine 2nd doses TTV Tetanus Toxoid Vaccine VAC Village AIDS Committee VCT Voluntary Counseling and Testing VHC Village Health Committee VHV Village Health Volunteer CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 1 A. Summary The Partnership for Equity, Access and Quality (PEAQ) Project started in October 1999 and ended in September 2003. The total funding from USAID was $1 million from the Child Survival and Health Grants Program (CSHGP), to which Save the Children US (SC) matched $333,000. In April 2001, the USAID/Malawi mission added $200,000 to this initial funding under the Leadership In Fighting an Epidemic (LIFE) Amendment. The Final Evaluation (FE) of the PEAQ Project was conducted by a team of ten members from SC, the Balaka District Health Office and the Mangochi District Health Office, and one external evaluator. The entire team contributed to the design, implementation and analysis of the evaluation, relying on the DIP and the MTE reports to develop the methodology and analyze their findings, and using data collected by the PEAQ Project through the baseline and final surveys, project monitoring quarterly reports, and the Balaka District HMIS. The PEAQ Project aimed at strengthening the health system of the entire Balaka District. The district was separated from the Machinga District one year before project inception, and now has a total population of 267,512 inhabitants, the majority (94%) of which live in rural areas. The three goals of the Balaka PEAQ Project were to: (1) Achieve a sustainable reduction in infant, child, and maternal mortality and morbidity; (2) Expand and refine a model for district health system strengthening; and (3) Adapt a district-to-district model for project replication and scaling up. The overarching approach of the PEAQ Project was one of district health system strengthening and capacity building, with an emphasis placed on community mobilization for behavior change. The PEAQ Project design included joint SC/DHO planning, the implementation of the national HMIS in the district, and intensive training activities for health workers and community members. Building on its successful experience in a similar project, the DHO staff of the nearby Mangochi District was called upon to act as a “Living University” for the Balaka DHO staff. The FE Team found evidence of increased coverage of the following key interventions known to be effective in reducing infant, child and maternal mortality and morbidity: - Immunization against preventable childhood diseases. - Vitamin A supplementation of children under five. - Iron and folic acid supplementation for pregnant women. - Intermittent preventive treatment of malaria in pregnancy and presumptive treatment of fever among children under five with sulfadoxine-pyrimethamine. - Assistance at delivery by qualified/trained birth attendants and early referral in case of complications. - Exclusive breastfeeding for infants up to six months of age. - Treatment of diarrhea with Oral Rehydration Therapy. - Birth spacing and prevention of unwanted pregnancies. - Early and effective treatment of sexually transmitted diseases. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 2 The FE Team also identified that the following features of the PEAQ Project had contributed to the success of the district health system strengthening approach: - Explicit partnership agreement with the DHO. - Joint planning of activities by SC and the DHO. - Focus on community mobilization and behavior change communication. - Decentralized management of selected activities to Health Zone Coordinators covering the catchment areas of one or two health centers. - No direct involvement of SC staff in the implementation of activities without designated MOHP counterparts. The following lessons were learned through the PEAQ Project and were observed by the FE Team: - With appropriate support and resources, the Balaka DHO can achieve high level of coverage of the essential interventions to reduce infant, child and maternal mortality and morbidity. - The district-to-district Living University approach has been effective in replicating and scaling up the successful Mangochi district health system strengthening project. - The formation and mobilization of the current District and community level HIV/AIDS structures is labor intensive and time consuming, and not as effective as expected. - Operations research and documentation of innovative interventions require careful planning, expertise and dedicated human and other resources. Given the success of the PEAQ Project in increasing the coverage of key interventions through a district health system strengthening approach, the Balaka DHO and SC should carefully define the human, financial, and transportation resources required to sustain these results. The DHO and SC should then develop a phase-over plan that specifies the equipment that SC can dispose of and the resources that SC can continue to provide under its other projects in the District. Through its continued presence in the District, SC should support the DHO to further develop the behavior change communication strategy and the health and management information system, strengthen the current district HIV/AIDS structures, and create new voluntary counseling and testing (VCT) centers. At the national level, the MOHP and its partners should ensure the regular supply of vaccines and other essential health products. Finally, USAID/Malawi and USAID/GH/CSHGP should consider the district health system strengthening approach and the district-to-district replication mechanism of the PEAQ Project for future health sector programs in Malawi and other countries. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 1 B. Assessment of Results and Impact of the Program 1. Results: Summary Chart FINAL INDICATOR BASE￾LINE Target Actual Immunization 1 Percent of children age 12-23 months fully immunized by their first birthday (card verified) 46% 75% 50% 2 Percent of mothers of children age =< 24 months who received 2 or more doses of tetanus toxoid during their last pregnancy (card verified) 39% 90% 61%** 3 Percent of children age 6-24 months who ever received Vit A (card verified) 31% 40% 39% (p = 0.07) Breastfeeding Promotion 4 Percent of children (0 – 5 mo.) who received only breast milk in the last 24 hours 28% 20% 57%** Maternal and Newborn Care 5 Percent of mothers with children age =< 24 months who received iron/folate supplementation during their last pregnancy 64% 80% 87%** 6 Percent of mothers with children age 24 months or under who received at least one dose of SP during their last pregnancy 72% 80% 77% 7 Percent of mothers with children age 24 months or under who report last delivery assisted by a health worker or trained TBA 62% 75% 73% (p = 0.07) Child Spacing 8 Percent of married women 15-49 years old using modern contraceptives 21% 30% 29% (p = 0.07) STI/HIV/AIDS Prevention 9 Percent of women 15-49 years old who report condom use with non￾regular partner 38% 50% 61% CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 2 FINAL INDICATOR BASE￾LINE Target Actual 10 Percent of women 15-49 years old who know of three ways to prevent STI/HIV 17% 60% 12%* 11 Percent of men 15-54 years old who report condom use with non￾regular partner 25% 70% 57%* 12 Percent of men 15-54 years old who know of three ways to prevent STI/HIV 24% 60% 20% 13 Percentage of female youth 15-19 years who delay sexual debut 21% 30% 29% 14 Number of persons tested in VCT centers/month 0 350 10 15 Percent target orphans/vulnerable children (OVC) with access to community services (OVC program) 0 50% NA 16 Percent STI cases treated according to national standards (STI treatment protocol) 19% 75% 38%* 17 Percent of households receiving help in caring for chronically ill young adults (15-49) in a population survey 0% 50% 9% 18 Percent of male youth 15-19 years who delay sexual debut 19% 30% 38%* CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 3 FINAL INDICATOR BASE￾LINE Target Actual Control of Diarrheal Disease 19 Percent of children age 24 months or under who have had diarrhea in the two weeks prior to the survey and who have been treated with Oral Rehydration Therapy (ORT) 65% 80% 77% 20 Percent of children age 24 months or under presenting at health facility with diarrhea who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 59% 80% Not Measured Control of Malaria 21 Percent of children age 24 months or under with fever in the two weeks prior to the survey who obtained appropriate treatment 31% 50% 23% 22 Percent of children age 24 months or under presenting at health facility with fever who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 44% 70% Not Measured Pneumonia Case Mgmt. 23 Percent of children age 24 months or under with cough and difficult/rapid breathing in the two weeks prior to the survey who obtained medical care from a health facility 50% 70% 43% 24 Percent of children age 24 months or under presenting at health facility with cough and difficult/rapid breathing who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 44% 70% Not Measured OTHERS 25 50% of villages have formal alarm and transport plans 0 50% 50% 26 80% of DHMT meetings discuss current HMIS information and make management decisions based on the data 0 80% 100% 27 Each of the 9 trained HC QA teams investigate and solve at least 3 problems per year after training 0 9 0 28 A Functioning DHMT-NGO District Coordination Comm. (DHPCC) NO YES NO 29 A District Fundraising Committee established and mobilizes resources for at least 2 projects per year NO YES YES 30 Quarterly HMIS report produced and disseminated using computerized HMIS data NO YES NO 31 80% of villages with functioning VHCs i.e. VHCs meeting at least one every two months (6 times/year) 0% 80% 91% Operations Research 32 Documented results of working with Traditional Healers and initiation leaders NO YES NO CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 4 FINAL INDICATOR BASE￾LINE Target Actual 33 Document results of Living University Approach NO YES NO 34 Document Results of Birth Plan Use OR Bicycle Ambulance NO YES YES * p < 0.05. ** p < 0.005 The baseline values were recalculated to ensure that the same indicator definition was used for both baseline and final. This resulted in several changes to the baseline values and targets which are below the baseline value. For example indicator 4 on exclusive breastfeeding. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 5 2. Results: Technical Approach a. Overview of Project The implementation of the Partnership for Equity, Access and Quality (PEAQ) Project started in October 1999 and ended in September 2003. The total funding from USAID was $1 million from the Child Survival and Health Grants Program (CSHGP) to which Save the Children US (SC) provided a match of $333,000. In April 2001, the USAID/Malawi mission added $200,000 to this initial funding under the Leadership In Fighting an Epidemic (LIFE) Amendment. The goal of the PEAQ Project was to strengthen the health system of the entire Balaka District. This district was separated from the Machinga District one year before the project started this district now has a total population of 267,512 inhabitants,1 the majority (94%) of which live in 647 villages in rural areas. At the time of the final evaluation (FE), the Balaka health system was comprised of one district hospital (84 beds; capacity to provide emergency obstetric care) and nine Health Centers (outpatient clinic with maternities; four from the Ministry of Health and Population (MOHP) and five from the Christian Hospitals Association of Malawi (CHAM). The District Health Office (DHO) personnel include six Clinical Officers (including the District Health Officer), five medical assistants, three Registered Nurses, 29 Enrolled Nurses, one Health Inspectors, ten Health Assistants, and 146 Health Surveillance Assistants (HSAs). A variety of community-based volunteers also provided health services in Balaka, including 80 trained Community Based Distribution Agents (CBDAs), 44 trained Community Male Motivators (CMMs), 90 active Traditional Birth Attendants (TBAs), 591 active Village Health Committees (VHCs—each composed of about 10 members and a chief), 888 active Village Health Volunteers (VHVs), 34 Drug Revolving Funds groups (DRFs), 58 Community-based HIV/AIDS counselors, and 80 Home Based Care counselors (HBCCs).2 The three goals of the PEAQ Project were to: (1) Achieve a sustainable reduction in infant, child, and maternal mortality and morbidity; (2) Expand and refine a model for district health system strengthening; and (3) Adapt a district-to-district model for project replication and scaling up. The overarching approach of the PEAQ Project has been one of district health system strengthening and capacity building, with the emphasis placed on community mobilization for behavior change. SC and the Balaka DHO worked in close partnership, with SC providing technical, managerial and financial support to the Balaka DHO and health centers staff. The PEAQ Project design included joint SC/DHO planning, the implementation of the national HMIS in the district, and intensive training activities for health workers and community members. This health systems strengthening approach also implied that the project interventions were 1 Projection estimate for 2002 used in the Malawi Health Management Information Bulletin, Special Issue 2002. 2 The number of community-based health workers at the beginning and at the end of the PEAQ Project is discussed in the relevant sections of the report and presented in Annex B-Methodology. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 6 implemented according to existing MOHP policies, that they complemented other efforts and programs in the district, and that they aimed at the entire population of the district, that is, 267,512 inhabitants in 2002. Building on its successful experience in a similar project, the nearby Mangochi DHO staff was called upon to act as “Living University” for the Balaka DHO staff. The PEAQ Detailed Implementation Plan (DIP) includes eight technical interventions: Immunization, Breastfeeding Promotion, Control of Diarrheal Disease, Pneumonia Case Management, Control of Malaria, Maternal and Newborn Care, Child Spacing, and STI/HIV/AIDS Prevention. The DIP also includes Integrated Management of Childhood Illnesses (IMCI) as a strategy to support the first five interventions cited above. The “Estimated Program Effort and USAID Funding by Intervention” table included in the DIP is reproduced below. The LIFE Amendment provided additional funding primarily to the STI/HIV/AIDS intervention. Intervention % of Total Effort USAID Funds in $ Immunization 10% $100,000 Breastfeeding Promotion 10% $100,000 Control of Diarrheal Disease 10% $100,000 Pneumonia Case Management 10% $100,000 Control of Malaria 10% $100,000 Maternal and Newborn Care 25% $250,000 Child Spacing 10% $100,000 STI/HIV/AIDS Prevention 15% $150,000 Total 100% $1,000,000 These eight interventions or Technical Approaches are discussed below in five sections covering one or several interventions. Each section begins with a summary table of the midterm and final targets for the key indicators of the interventions being discussed, and the corresponding baseline, midterm and final results. Most baseline and final result values presented in these tables were obtained from the two Knowledge, Attitude and Practices (KAP) household surveys conducted by the PEAQ Project at the beginning and the end of the project (baseline in November 1999 and final in July 2003). The midterm values come from a Lot Quality Assurance Survey (LAQS-- conducted in August 2001). Each summary table is followed by a brief discussion of the interpretation of the available data in terms of success of the project. The factors of success and the constraints identified by the Final Evaluation Team (hereafter, the FE Team) are then presented, followed by the lessons learned and the next steps. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 7 b. Immunization Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 1 Percent of children age 12-23 months fully immunized by their first birthday 46% 55% 60% 75% 50% 2 Percent of mothers of children age =< 24 months who received 2 or more doses of tetanus toxoid during their last pregnancy (card & self report) 80% 85% 85% 90% 87% 3 Percent of children age 6-24 months who ever received Vit A 31% 20% 46% 40% 39% The midterm and final KAP surveys demonstrate a definite increase in the immunization coverage of pregnant women. The percentage of children completely immunized by their first birthday is based on information collected from the growth monitoring card, and therefore might underestimate the true coverage. The Tetanus Toxoid Vaccine (TTV) coverage is based on information from the TTV card and from recall by the caretakers. The estimates of immunization coverage from the DHO services statistics are consistently lower than those from the KAP surveys (see section C.7). The major factors that explain the discrepancies between the two estimates seem to be underreporting of services statistics.3 The midterm and final KAP surveys also demonstrate a definite increase in the coverage of children 6-24 months with Vitamin A supplements. The baseline coverage of children 6-24 months was erroneously calculated as 10% at the beginning of the project and therefore the midterm and final targets were set at 20% and 40%, respectively. Factors of Success The PEAQ Project contributed to the above success by: 1. Increasing the number of outreach clinics from 108 to 135 through the provision of transportation to health workers (push bicycles and motorcycles). 2. Improving the cold chain through the training of six technicians and providing gas, paraffin, refrigerators, and other supplies and equipment. This support increased both the quality and accessibility of EPI services. 3. Training of 25 mid-level EPI managers, which improved the overall management of the EPI program. 4. Enhancing community mobilization though the training of the VHC members and VHVs on the importance of full immunization and the recommended immunization schedule for pregnant women and children under one year of age (see sections B.3.a and B.3.b). The provision of Vitamin A supplements to TBAs during supervision visits may have contributed to the increase in coverage of this intervention among children under five. 3 One source of underreporting seems to be that health workers often fail to check off the “completely immunized” column on the EPI tally sheets when appropriate. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 8 Constraints Throughout the duration of the PEAQ Project, the Balaka District experienced frequent stock-outs or insufficient supplies of several antigens, primarily of BCG, due to stock-outs or power shortages at the national level. Vitamin A stock-outs also occurred frequently. Lessons Learned With appropriate support and resources, the Balaka DHO can achieve high levels of immunization coverage. In order to reach the national target however, the DHO needs to have a constant supply of all vaccines. Next Steps The DHO should maintain all of the necessary cold chain equipment and means of transport to sustain the current levels of immunization services beyond the end of the PEAQ Project. The DHO and SC should reconcile the immunization coverage estimates from the KAP surveys and the services statistics to better understand the true immunization coverage rates and related determinants to improve the monitoring of immunization activities, and to continue to increase immunization coverage. The DHO is considering home-to-home visits to increase the immunization coverage rates in the district. An alternative strategy is to target areas known to have low coverage through the organization of periodic immunization campaigns in those areas. c. Maternal and Newborn Care Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 5 Percent of mothers with children age =< 24 months who received iron/folate supplementation during their last pregnancy 64% 75% 77% 80% 87% 6 Percent of mothers with children age 24 months or under who received at least one dose of SP during their last pregnancy 72% 75% 63% 80% 77% 7 Percent of mothers with children age 24 months or under who report last delivery assisted by a health worker or trained TBA 62% 70% 70% 75% 73% 25 50% of villages have formal alarm and transport plans 0% 25% 59% 50% 50% The midterm and final KPC surveys demonstrate an increase in the three individual-level indicators selected for the maternal and newborn care intervention. The indicators for iron/folate supplementation and the preventive malaria treatment with sulfadoxine-pyrimethamine (SP) are based on verbal reporting and the antenatal card and the TTV card, respectively. The indicator for delivery with a qualified/trained attendant is based on the answers from the respondents. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 9 Indicator # 25 is measured by the presence of one working bicycle ambulance (BA) for a group of about ten villages that can access it in case of medical or obstetrical emergency. Factors of Success The project contributed to achieving objectives #5, #6 and #7 above through: 1. Training and supervision of TBAs (34 TBAs received initial training and 29 TBAs received refresher training according to the national curriculum), and their receipt of supplies of iron/folate supplements and SP tablets (until the recent change in the MOHP policy to discontinue the practice of distributing SP through TBAs) 2. Inclusion in the VHC and VHV training curriculum of key messages on the importance of antenatal and of delivery care by trained health workers and TBAs. 3. Increase the number of outreach service delivery points offering antenatal services including provision of iron and SP tablets from 108 to 135. The FE Team found that the PEAQ Project also improved: 4. The referral system for obstetric emergencies through establishment of alarm and transport system at the village level (see below), training of TBAs in recognition of danger signs and criteria for referral, establishment of a radio communication system in two health centers, and repair and maintenance of DHO vehicles. 5. The infection prevention practices at the health centers and the TBA levels through specific training and provision of supplies (kerosene for sterilization in the health centers; antiseptics to TBAs). The PEAQ Project provided six new and revitalized 21 existing BAs, which together are accessible to about 300 of the most isolated villages. VHVs are typically in charge of the BA and ensuring that the community is aware of its existence and how to access it in case of medical emergency. The FE Team also noted that health workers in all health centers received a five-day training in life saving skills as part of the safe motherhood program of the MOHP. Constraints There are still about 30 active TBAs who have not yet been trained. There are still frequent stock outs of iron and SP at the national level. There is a concern that some pregnant women receiving antenatal care services from mission health facilities fail to buy SP and iron tablets because they are too expensive. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 10 The DHO reports late referrals by TBAs among the contributing factors of the approximately 20 maternal deaths that occurred last year in the Balaka District Hospital. However, this data should be confirmed through systematic maternal death audits. Lessons Learned Increasing the number of outreach service delivery points for antenatal care, training, and support of TBAs can improve the use of essential maternal and child care services. Next Steps Sustaining and improving the above results requires continued community mobilization and education, continued supervision and provision of supplies to TBAs, and continued availability of transport for outreach clinics. Some mission facilities include the cost of SP and iron in their antenatal services fees so that the client does not have to pay for these items separately. Facilities with this policy do not seem to have the problem cited above. d. Child Spacing Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 8 Percent of married women 15-49 years old using modern contraceptives 21% 25% 18% 30% 29% The baseline and final KAP surveys demonstrate a definite increase in the prevalence of modern contraception among women 15-49 years old. The Balaka health services statistics show slightly lower prevalence values by zone, probably due mainly to underreporting by some health centers and by CBDAs. Factors of Success The PEAQ Project contributed to the achievement of the above objective in three different ways: 1. Training of 40 CMMs and 80 CBDAs. 2. Training of 146 HSAs as co-family planning providers. 3. Increasing the number of outreach clinics offering integrated FP services from 35 to 128. 4. Mobilization of communities by trained VHCs and VHVs. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 11 Constraints HSAs were asked to stop providing Depo-Provera injections through an MOHP directive. If HSAs were still allowed to give Depo-Provera injections, shown by the DHS and other FP reports as the preferred contraceptive for most Malawian women, the achieved contraceptive prevalence would most likely be higher. Five of the nine HCs in Balaka do not provide family planning services because of their religious affiliation. Many women in the Balaka District would like to use contraceptive methods but still do not have access to family planning services. Lessons Learned The increase in the number of trained CMMs, CBDAs, co-family planning providers HSAs, and integrated outreach clinics has contributed to addressing the demand for family planning information and services, and increasing the utilization of modern contraceptive methods. Next Steps Health Center staff should support the CMMs and CBDAs in their locations through supervision and provision of supplies. Additional CBDAs and CMMs should be trained and supervised, especially in areas covered by the five CHAM health centers. Integrated FP outreach clinics should be maintained and new ones should be created. The DHO should find ways to increase the number of providers of Depo-Provera, the preferred family planning method in Balaka, for instance by involving private maternity and other clinics. e. STI / HIV/AIDS Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 9 Percent of women 15-49 years old who report condom use with non-regular partner 38% 30% NA 50% 61% 10 Percent of women 15-49 years old who know of three ways to prevent STI/HIV 17% 30% 16% 60% 12% 11 Percent of men 15-54 years old who report condom use with non-regular partner 25% 60% NA 70% 57% 12 Percent of men 15-54 years old who know of three ways to prevent STI/HIV 24% 30% 29% 60% 20% 13 Percent of female youth 15-19 years who delay sexual debut 21% 25% NA 30% 29% 18 Percent of male youth 15-19 years who delay sexual debut 19% 25% NA 30% 38% CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 12 The baseline and final KAP surveys show that both indicators 9 and 11 increased. However, the sample size for these two indicators is very small, and this change is only statistically significant (p < 0.05) for men. The two surveys also show that indicators 10 and 12 slightly decreased during the period of the project, but this difference is not statistically significant. Indicator Base- Midterm Final line Targ. Res. Targ. Res. 14 Number of persons tested in VCT centers/month 0 0 8 350 10 15 Percent target orphans/vulnerable children (OVC) with access to community services (OVC program) 0% 0% NA 50% NA 16 Percent STI cases treated according to national standards (STI treatment protocol) 20% 20% NA 75% NA 17 Percent of households receiving help in caring for chronically ill young adults (15-49) in a population survey 0% 0% NA 50% 9% The four objectives and indicators above were added to the project key indicators with the LIFE Amendment. HIV testing (indicator 14) was available in the Balaka hospital at the beginning of the project but primarily for inpatients and blood donor screening, and probably without proper counseling when needed. The VCT center of the District Hospital began its activities in June 2003. As the PEAQ Project did not directly establish community services for Orphans and Vulnerable Children, this indicator (15), assumed 0% at baseline, was not measured at the end of the project. The baseline estimate for indicator 16 came from a review of the STI registers in health centers to check the adequacy of the prescribed treatment to the recorded diagnosis. No such review was made for the final evaluation. The Balaka health services statistics show an increase in the monthly number of STIs treated in the District Hospital and Health Centers in 2002 (see Figure 2 in Annex L). This suggests that the STI training of health workers in 2001 and 2002 and the availability of STI drugs increased the coverage of STI cases treated, presumably according to the national STI treatment protocols. The baseline KAP survey includes a question to measure indicator 17, but the value is not presented in the report. The question asked does not specify the type and source of care. The final target of 50% probably referred to the proportion of households with a chronically ill young adult who receive help in caring for that person, rather than the proportion of households receiving care for chronically ill young adults. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 13 Factors of Success The PEAQ Project established the foundation for increased HIV/AIDS prevention, care and mitigation activities in Balaka through: - Formation of the Balaka District AIDS Coordination Committee (DACC) in 2000. - Training of 36 DACC members in 2002 (Leadership training conducted by NAC trainers; Training for Transformation, a three-day course developed under the SC/Malawi STEPS (previously COPE) project). - Sensitization of nine CACs and training of the members of four CACS in 2002; - Initiating the training of 135 VACs. The PEAQ Project supported the dissemination of the key HIV/AIDS messages at the community level through: - Inclusion of these messages in the training curriculum for VHCs and VHVs; - Various activities held during the District AIDS days. The PEAQ Project supported the training of various community volunteers: - 40 HIV/AIDS counselors (among the total of 80 trained counselors in the district); - 92 Initiation counselors; and - 80 Home-Based Care Counselors. The PEAQ Project supported the training of health workers in; - STI case management (two trainers; 20 trainees and seven received refresher training; another training is scheduled before the end of the project); - HIV/AIDS counseling (12 health workers received a six-week training as Hospital Based Counselors); - Integration of STI/HIV/AIDS into reproductive health services (68 trainees); and - Integration of HIV/AIDS in the BFHI (two trainees). One VCT center began its activities in the District Hospital in June 2003 when a first batch of 600 HIV tests was provided by SC to the DHO. The necessary equipment is also in place and 12 staff members have received training in counseling. Among the 33 tests already performed at the time of the FE, 12 were positive and 21 were negative. Constraints The District AIDS structures are composed of persons with other responsibilities and commitments, and many DACC and CAC members seek allowances or other forms of compensation. As a result, the DACC members can hardly manage to meet on a regular basis or conduct other activities expected from them such attending or conducting training events. This may slow down the overall progress on HIV/AIDS activities in the district. The national policy to provide uniform training for DACC members resulted in such delays that this training was only conducted in Balaka in 2002. The training of some VAC members by DACC/CAC members has only recently begun. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 14 There is no full-time HIV/AIDS coordinator at the DHO level to motivate and facilitate the work to be done by the DACC and the CACs. As the DACC is not functional, some NGOs are conducting HIV/AIDS activities in certain areas of the Balaka District without the DHO or the DACC even being aware of their work. The late availability and authorization to use the LIFE Amendment funds have delayed the beginning of some HIV/AIDS activities of the PEAQ Project. The late development of the national VCT guidelines and approval of VCT activities in Balaka have delayed the establishment of VCT centers in the district. The District Hospital is just beginning VCT activities and there is still no dedicated room for this service yet. Only one laboratory technician is trained in performing the rapid HIV tests. There is no VCT service in the health centers yet. The Health Centers still experience (and document) stock outs of the main STI drugs. Lessons Learned The formation and mobilization of the DACC/CACC/VAC structure is labor intensive and time consuming, and this structure has not been as effective as expected. The development of national policies and guidelines is a lengthy process that must be factored into projects that rely on their availability. Next Steps All DACC members must commit to making the DACC/CAC/VAC work despite their other non￾HIV related duties. The DACC should liaise with the NAC to get the necessary resources for intensifying HIV/AIDS activities in Balaka District. The DHO should consider appointing a full-time coordinator for all the HIV/AIDS activities of the health sector in the district. The DHO and the DACC should continue supporting the VCT center that was newly established in the Hospital, and establish additional centers in the health centers. Non-clinicians could be offered the necessary counseling training to avoid increasing the workload of the limited number of clinicians available in the district. Health workers should be trained in VCT testing procedures. The DHO should ensure a continuous supply of HIV rapid tests in the VCT center(s). The DHO should plan for training additional community-based HIV/AIDS counselors, and implement the integrated HIV/AIDS/BFHI training program in the health services. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 15 SC and DHO should complete the planned training of health workers in HIV/AIDS case management before the end of the PEAQ Project. SC should continue supporting the DHO HIV/AIDS activities through its other projects in the district, particularly the Adopt-a-Village project using the STEP strategy, the School Health and Nutrition Project, and the HIV/AIDS in Schools Project. f. Breastfeeding/Diarrhea/ARI/Malaria Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 4 Percent of children who received only breast milk to at least 4 months of age 28% 15% 68% 20% 57% 19 Percent of children age 24 months or under who have had diarrhea in the two weeks prior to the survey and who have been treated with ORT 65% 75% 59% 80% 77% 21 Percent of children age 24 months or under with fever in the two weeks prior to the survey who obtained medical care from a health facility 31% 40% NA 50% 23% 23 Percent of children age 24 months or under with cough and difficult/rapid breathing in the two weeks prior to the survey who obtained medical care from a health facility 50% 55% NA 70% 43% The baseline and final household surveys show a remarkable and statistically significant (p < 0.005) increase in the practice of exclusive breastfeeding of infants up to four months old. The baseline and final KAP surveys demonstrate an improvement in health care seeking behavior for children below 24 months of age for diarrhea. No improvement appears for cough and difficult/rapid breathing or for fever. Treatment with ORT for indicator 19 includes the use of Oral Rehydration Salts and Salt/Sugar Solutions. Medical care from a health facility for indicator 21 includes treatment received from DRFs. The quality of the case management of diarrhea, fever, acute respiratory infections (indicators 20, 22, and 24 in Annex K) was not measured at the end of the project because the IMCI intervention was dropped at midterm. Factors of Success The training curriculum of the VHCs/VHVs, TBAs, and interactive drama groups has been expanded to include the importance of immediate and exclusive breastfeeding, and key messages about health care seeking for diarrhea and fever. Establishing 34 DRFs in remote areas increased the availability of drugs to treat diarrhea and malaria. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 16 Constraints The PEAQ Project did not support IMCI training as initially planned in the DIP. This decision was made when the midterm evaluation team realized that the DIP was too ambitious for the available human and financial resources. Also, the Balaka District had not been considered by the MOHP in the first series of districts to implement IMCI. Only two health workers were trained in “Integrated BFHI and HIV/AIDS.” National BFHI master trainers were not available to conduct the training activities planned in the district. The MOHP supported training of health workers in malaria case management, but the last training events were held in 2001 and many trainees left the district since then. Case management of pneumonia has greatly improved at the Balaka Hospital under another project (case fatality rate decreased from 26% at the beginning of the project in 2001 to 7% currently), but this pilot project does not include health centers or the community health volunteers. ARI was not included in the training of community volunteers provided under the PEAQ Project– this is shown in the above results. DRFs do not have drugs to treat pneumonia. There are still 17 DRFs that were established but their members have not yet been trained. No one was available at Balaka Central to conduct this training. Lessons Learned The VHCs/VHVs/TBAs were effective in promoting increased care seeking for malaria and diarrhea. The DRFs are effective in increasing the use of health services and require continued support. BFHI master trainers are not available to meet the need for training in breastfeeding at the district level. Next Steps SC and the DHO will continue training the remaining DRFs as planned until the end of the PEAQ Project. All the health workers should be trained and supervised in IMCI. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 17 3. Results: Cross-cutting Approaches a. Community Mobilization Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 31 80% of villages with functioning VHCs i.e. VHCs meeting at least one every two months (6 times/year) 0% 40% NA 80% 91% Indicator 31 above shows increased community mobilization at the village level. The final result value is that reported in the 2002 Balaka District Health and Management Information System. Factors of Success The PEAQ Project helped establish new or has reactivated a total of 591 VHCs in the 647 villages of the Balaka District, and trained the members of these community-based structures (11 per VHC). The project also established and trained the members of 34 DRFs among these VHCs. The formation of the DACC/CAC/VAC structure and training of its members is discussed in section B.2.e. The PEAQ Project supported the training of various community volunteers in key BCC messages (888 VHVs, 91 CBDAs, 40 CMMs, 87 TBAs, 16 drama groups, 92 initiation counselors, 58 HIV/AIDS counselors and 80 HBC counselors). The PEAQ helped increase the number of HSAs (from 120 to 146 currently) and their capacity to supervise community volunteers through refresher training (three days; key messages), supervision, and provision of bicycles. The PEAQ introduced and supported Health Zone Coordinators (four DHO staff and two SC staff) to conduct and supervise the training of community volunteers, supervise the HSAs, and facilitate community mobilization activities such as the FP and the HIV/AIDS Days. Constraints Although about 80% of VHCs are reported to be active at the end of the PEAQ Project, the percentage of VHVs active may be lower, depending on the activity considered. For instance, VHVs often lack scales to conduct growth monitoring activities in their villages. The QA training and subsequent follow up activities proposed in the DIP did not involve communities in the problem solving exercises conducted at the health center level (see also constraints in section B.3.c). CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 18 Lessons Learned VHCs need stronger guidance and supervision to conduct specific and effective activities in their communities. Next Steps The DHO and HCs should increase the number of VHCs in underserved areas. The Health Centers should find ways to involve HSAs in the supervision and support of VHCs and VHVs. The DHO should support the training of Health Zone Coordinators and HSAs in community mobilization. The QA training and implementation in health centers should include community members as part of the QA teams. b. Behavior Change Communication Baseline, Midterm and Final Results About half of the PEAQ indicators related to Technical Approaches presented in Section B.2 assume some change in knowledge and behavior of the target population, and are therefore relevant to this section. Factors of Success The PEAQ Project established numerous community-based structures to conduct BBC activities. The PEAQ Project trained a large number of community volunteers, who in general seem to have appropriate knowledge of most of the key messages of the project. Constraints The PEAQ Project used national training materials to train most VHC members and VHVs. These materials primarily include information on water and sanitation, but do not cover all of the PEAQ interventions, and do not focus on behavior change. Many VHCs still seem to focus on water and sanitation activities (particularly cholera) although their training curriculum was modified to address this concern raised at midterm and VHC members seem to have appropriate knowledge of the key PEAQ messages. The PEAQ Project has not had sufficient education materials to provide to health workers and community volunteers trained in key BCC messages. A review of the existing materials at the national level was made after the midterm evaluation, but most of them were too expensive for the PEAQ budget (except for a few items about FP that were available for free). CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 19 The collaboration with, and capacity building of the IEC service of the Balaka DHO has not been as fruitful as with the other DHO services. Lessons Learned At midterm, the PEAQ Project staff realized that the initial plans for behavior change were too ambitious and that a revision of the BCC strategy needed to include: - Identification and prioritization of the changes in behavior likely to have the highest impact; - Extension of the scope of activities of the VHCs from water and sanitation to a series of key behaviors to be changed with involvement of the community; and - Evaluation of the effectiveness of the current BCC interventions including drama groups and others. Next Steps After the MTE, sele cted SC and DHO staff (program managers and zone coordinators) received training in the BEHAVE framework for planning behavior change activities. They conducted preliminary studies with four communities to identify a set of priority behaviors that they were willing to address. These findings can lead to practical plans of action involving the communities. The approach should now be presented and discussed with the EPMU and the action plans implemented as appropriate. c. Capacity Building Baseline, Midterm and Final Results Indicator Base- Midterm Final line Targ. Res. Targ. Res. 25 50% of villages have formal alarm and transport plans 0% 25% 59% 50% 50% 26 80% of DHMT meetings discuss current HMIS information and make management decisions based on the data 0% 40% 50% 80% 100% 27 Each of the 9 trained HC QA teams investigate and solve at least 3 problems per year after training 0 5 4 trained 9 0 28 A Functioning DHMT-NGO District Coordination Comm. (DHPCC) NO YES YES YES NO 29 A District Fundraising Committee established and mobilizes resources for at least two projects per year NO YES NO YES YES 30 Quarterly HMIS report produced and disseminated using computerized HMIS data NO YES YES YES NO 31 80% of villages with functioning VHCs i.e. VHCs meeting at least once every two months (six times/year) 0% 40% NA 80% 91% 32 Documented results of working with Traditional Healers and initiation leaders NO NO NO YES NO 33 Document results of Living University Approach NO YES NO YES NO 34 Document Results of Birth Plan Use OR Bicycle Ambulance NO NO YES YES YES CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 20 Indicators 25 to 31 measure various aspects of the capacity of the DHO and communities in the Balaka District to implement CS activities. Indicator 25 is discussed in section B.2.c, indicator 31 is discussed in section B.3.a and indicators 26 and 30 are discussed in section C.7. Factors of Success The FE Team found that the PEAQ Project successfully built the capacity of the DHO and communities in Balaka to plan and implement child survival and other health activities. This result was achieved through the following main strategies: 1. The DHO and SC developed, agreed upon and updated as appropriate an explicit partnership agreement defining their respective roles and responsibilities in the implementation of the PEAQ Project. 2. The PEAQ Project created and used an effective organizational structure including direct involvement of the DHO staff in the planning and implementation of project activities through the expanded project management unit (EPMU), and decentralized management of certain activities through the Health Zone Coordinators. 3. The PEAQ Project increased the capacity of the DHO and Health Center staff to hold meetings at the district and health center levels where general management and programmatic issues are discussed. 4. The PEAQ Project increased the capacity of the Balaka District to train its health workers and volunteers by using trainers from the district for all training activities and by supporting numerous and varied training activities. 5. Specific training activities, discussed in other sections, increased capacity of health workers and volunteers to mobilize communities and deliver child survival and other interventions. 6. The PEAQ Project helped establish various community-based structures such as the VHCs, DRFs, the bicycle ambulance committees, and the DACC/CAC/VAC. 7. The PEAQ Project improved transport capacity for DHO and Health Center staff through the provision and maintenance of push bikes, motorbikes, fuel, and vehicles. 8. The PEAQ Project successfully implemented an innovative model for replication and scaling up of district-level health strengthening activities from Mangochi to Balaka District. 9. SC has identified and initiated the documentation of several activities and models of potential significance outside the Balaka District: the district-to-district Living University model, the use of bicycle ambulances as part of a community level alarm and action plans, and the promotion of individual birth plans at the community level. 10. The Hospital Advisory Committee is meeting regularly and has been successful in obtaining resources for the District Hospital. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 21 SC/Malawi recognizes the positive effect of the PEAQ Project on its capacity to plan, implement and evaluate child survival and other projects. The various requirements of the USAID Child Survival and Health Grants Program such as the preparation of a detailed DIP and the organization of external midterm and final evaluations, improved the skills and the professional standards of all the staff involved in these exercises. This in turn improved the capacity of SC/Malawi to prepare, implement and evaluate projects in technical areas sometimes quite different from child survival. SC/HO also recognizes that the district health system strengthening approach of the PEAQ Project and the district-to-district Living University strategy for replication and scaling up have already positively influenced other projects’ design and implementation in Malawi and other countries. Constraints The PEAQ Project helped the formation and facilitated a few meetings of the DHMT-NGO Committee, but attendance has always been low and the last meeting of the committee was in 2002. The existing Hospital Advisory Committee does not seek funding for public health or community health activities, and the DACC is not actively seeking additional resources for the HIV/AIDS activities in the district. Only four Health Centers received training in Quality Assurance problem solving methods and tools but they have not regularly applied them. The two QA trainers left the district and were not available for coaching and following-up on the QA teams formed during the initial training. Some of the trained Health Center staff also left the district without transferring their skills to the replacement staff. In general, communities have not been fully involved in the problem solving exercises conducted in the Health Centers. The PEAQ Project had difficulty with the planning and follow up on the otherwise promising documentation of innovative interventions, as proposed in the DIP. While the initial plan to work with Traditional Healers was dropped at the midterm, the Initiation Leaders were trained by the PEAQ Project but without documentation of the results of this activity. The Living University approach was “highlighted” in the MTE report and presented at a meeting of the CORE group, but not documented in a format suitable for larger dissemination within SC or with other development organizations both within and outside of Malawi. A qualitative analysis of the sociocultural determinants of the development and use of birth plans was completed and birth planning was included in the training curriculum of TBAs. However, the extent and the potential results of this intervention are not documented. Collection of data on the utilization and potential results of bicycle ambulances and community alarm plans began, but was interrupted. Lessons Learned Health Workers and Community Volunteers have the capacity to conduct successful child survival activities but need continued support and resources. The district-to-district ‘Living University’ has been a successful approach to replicate the district health system strengthening success achieved in the Mangochi District. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 22 Conducting operations research or full documentation of innovative interventions require detailed planning and dedicated human and other resources. Next Steps The DHO should maintain the now well-established Zonal and EPMU meetings. The DHO should maintain and update health workers capacity though refresher courses. The DHO should replace the 2 SC Health Zone Coordinators. The QA/problem solving strategy should be revised to include a more active and effective involvement of the communities served by the Health Centers. SC should capture the information on the PEAQ Project innovative interventions that can still be documented before the end of the project. d. Sustainability The strength of the design of the PEAQ Project based on district health system strengthening and capacity building is its potential for sustainable results. This is confirmed by the FE Team findings that the PEAQ successfully increased the coverage of a series of maternal and child survival interventions while building the capacity of the Balaka DHO, Health Centers, and communities to plan and implement them. Nevertheless, the FE Team identified insufficient financial resources at the DHO level to provide the inputs necessary to sustain the PEAQ Project results, including the maintenance of vehicles and motorbikes, fuel, laboratory reagents, training and supervision. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 23 C. Program Management The implicit goal of the management of the PEAQ Project is the successful implementation of the planned activities and an increased capacity of the Balaka DHO and communities. In this perspective, this section examines the strengths and weaknesses of the key aspects of the management of the PEAQ Project within SC from HO to Balaka, within the DHO and its Health Centers, and at the community level. The focus is on identifying lessons learned and next steps. 1. Planning Strengths The joint DHO/SC planning of the PEAQ Project activities has been successful. Examples of joint planning activities are: the preparation of the DIP during a participatory workshop (a copy of the final DIP was given to all DHO and health center staff), the midterm and final evaluations, the regular meetings of the PMU and of the EPMU, the development of detailed quarterly activity plans, and the day-to-day planning of activities at the level of program managers and SC counterparts. Weaknesses The DIP turned out to be too ambitious, and significant adjustments of the objectives and work plan were made after the MTE (see Midterm Evaluation Action Plan, June 2002). 2. Staff Training Strengths The PEAQ Project provided numerous training opportunities to health workers and community volunteers. This training was conducted according to the national curriculum and guidelines. Most SC/Balaka and DHO staff received some training during the duration of the project (see Annexes I and J). Weaknesses The relatively high turnover of the DHO and SC staff makes it difficult to keep the specific knowledge and skills needed for project implementation. SC and the DHO often had difficulty finding qualified trainers for the various training events that the PEAQ Project supported. The PEAQ Project made photocopies of the MOHP recommended materials for all trainers and participants because no other material was available or affordable. This constant reproduction at the local level did not always result in good quality training materials. The PEAQ had too small a budget to offer outside training opportunities to all SC staff, and some staff members did not benefit from having received specific, external formal training during their many years of service on the PEAQ Project. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 24 3. Supervision of Program Staff Strengths The EPMU planned and conducted regular joint DHO/SC supervision visits of community volunteers. The SC/Balaka staff in general has maintained good teamwork and collaboration, and the PEAQ Project Coordinator has provided constructive support to the professional staff. Weaknesses The supervision system implemented during the PEAQ Project is too costly for the current DHO budget. 4. Human Resources and Staff Management Strengths The PEAQ Project has provided good support to the DHO staff through training, allowances linked to specific project activities, and provision of means of transportation to conduct project activities. The PEAQ supported the salary of two Health Zone Coordinators where no appropriate DHO staff was available. The SC and the DHO Health Zone Coordinators were supervised in the same way. As these two positions need to be filled at the end of the project, four prospective replacement persons for these two positions are currently in a 1-year training program at the Malawi College of Health Sciences. SC personnel policies are generally clear and appropriate. The SC/Balaka team has regular “Monday meetings” that constitute a useful forum for most day-to-day project implementation issues. Overall, the PEAQ Project benefited from qualified staff with no major extended vacancies of critical positions. SC staff members have left the PEAQ Project for significantly higher level positions, suggesting that their time on the PEAQ helped them advance in their career. SC has several new projects in Balaka and elsewhere in Malawi and currently examines how to provide career transition opportunities to the senior staff of the PEAQ Project who wish to continue working with SC. Weaknesses Both SC and the DHO experienced high turnover of staff and the consequences of having to compensate through longer hours for the vacant positions while the recruitment process was under way and to retrain the replacement staff. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 25 The vacant positions for a second trainer during more than a year (since mid-2002) and for an office manager since March 2003 have led to work overload among the other PEAQ Project staff members. 5. Financial Management Strengths SC maintained full transparency with the DHO on the PEAQ budget available for program activities in the Balaka District. Typically, the PEAQ Project Coordinator would propose a budget to the DHO for agreed upon activities, which is then signed by both parties. The PEAQ financial pipeline at the SC/HO level showed that the overall grant has been spent as expected and that there will be no significant remaining funds. Weaknesses There have been persistent problems with the allowances awarded to the DHO staff, which has prevented or slowed down many project activities. Many MOHP staff refuse to conduct activitie s with the level of allowances approved by USAID and offered by SC. The higher allowances offered by various organizations working in Balaka with the same DHO staff as the PEAQ Project create competition among them in that respect. The SC PEAQ Project Coordinator had limited signatory power for disbursing project funds, which may have created an unnecessarily heavy managerial burden when a large number of activities were implemented at the same time. 6. Logistics Strengths The PEAQ Project provided various commodities and equipment to the DHO and the communities implementing project activities: TBA kits, HIV rapid tests kits, motorcycles for HZC, bicycles for HSAs, bicycle ambulances, training materials, etc. Weaknesses Delays in provision of some supplies (HIV tests, TBA kits, HBC kits) have slowed down the beginning of the corresponding activities. Issues of accountability and the corresponding control procedures have sometimes delayed the maintenance of the motorbikes used by the DHO and health centers to conduct project activities. Over the duration of the project, both SC/Balaka and the DHO have at times lacked vehicles and other means of transport to conduct project activities. Each Program Manager and Health Center should have at least one motorbike at the very minimum. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 26 7. Information Management Strengths The PEAQ Project has effectively supported the implementation of the new national Health and Management Information System (HMIS) system in the Balaka District. The project supported the training of health workers and provided assistance in the use of data at the DHO and health center levels. As a result, timely and quality programmatic data have been appropriately used by the EPMU and in other training and supervision settings. Annex L presents selected HMIS data for year 2002 and the first quarter of year 2003. TBAs, CBDAs, CMMs, VHCs report their activities to HSAs who supervise them. DHO and health center supervisors sometime use this data to give feedback to community volunteers on their work and challenges. Health Zone Coordinators collect and report HMIS data for their zone. They have played an important role in the establishment of the HMIS and the use of the data at the health center level. Quarterly reports of the PEAQ Project were regularly prepared by program managers and zone coordinators for the EPMU meetings. The overall PEAQ Project quarterly reports have been completed regularly throughout the duration of the project. The PEAQ Project successfully conducted timely baseline and final KAP surveys and Health Facility Assessments, and a Lot Quality Assurance Survey at midterm. Weaknesses There are still delays in the transmission of the HMIS reports from the health centers to the DHO. Also, most indicators are higher in the household survey than in the HMIS, suggesting that the data in the latter are incomplete. Finally, HMIS supervisors have noted that the activity registers are sometimes not properly tallied before inclusion of the data in the HMIS reports. There is still a need for training and supervision of health centers and DHO staff and for provision of supplies to improve quality, timeliness and completeness of the HMIS data collection, analysis and use. There is also a need for specialized technical assistance for data analysis, quality control and reporting at the district level. The series of household, health facility, and lot quality assurance surveys are not fully documented and the comparative analyses are not complete (use of somewhat different indicators in each survey and lack of calculation of tests of statistical significance). 8. Technical and Administrative Support Strengths The majority of technical assistance was provided through the Living University strategy, which involved visits of the staff from Balaka DHO and SC to the Mangochi District, and technical CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 27 assistance visits from the Mangochi staff to the Balaka District. This strategy appeared very useful to both Mangochi “teachers” and Balaka “students.” SC/Balaka received frequent support from the Lilongwe offices for financial and administrative management. The SC/Malawi Senior Health Advisors and the SC Africa Regional Advisor also provided technical support on a continuous basis, and on critical areas such as the baseline and final surveys, the preparation of the DIP, the MTE and the FE evaluations, the training in the BEHAVE framework and subsequent formative research, and the establishment of VCT centers. The PEAQ Project received continuous technical and managerial support from the SC/HO through the Child Survival Specialist, who also made a series of productive visits with extensive presence in the project site area. These visits included the DIP preparation, the MTE and FE evaluations, and the preparation of a revised work plan based on the MTE recommendations. The CS specialist also provided much appreciated assistance with the preparation of the various reports, budget allocation decisions, and identification and recruitment of consultants. The CS Specialist, as a co-facilitator of the CORE South Africa BEHAVE training in February 2002, provided technical assistance in behavior change to three of the Malawi-based project staff. This led to a revised BCC strategy for the project following the MTE. Weaknesses The PEAQ Project would have benefited from timely specialized technical assistance in the few areas in which it did not perform as expected. Among such areas are the implementation of the HIV/AIDS interventions and the various plans of documentation and operations research. SC/Balaka team would have benefited from reviews of the management style and procedures by senior SC managers to ensure continuous improvement of the implementation of project activities and professional development. 9. Management Lessons Learned The preparation, adoption and update of a clear Partnership Agreement between SC and the DHO have been effective in building capacity, mutual trust and transparent financial management. Decentralizing the management of certain DHO activities through Health Zone Coordinators is effective. However, this organizational structure relying on the competence and attitude of the individual Health Zone Coordinators has led to project delays or poor performance in certain areas of the district where the Zone Coordinator was absent or did not cooperate. Rigorous management of even limited financial and logistical resources allows the implementation of most of the PEAQ Project activities. The maximum amount allowed at each level of signatory power needs to be adjusted to the actual value of the local currency to avoid delays and undue burden on the managers. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 28 D. Conclusions and Recommendations The PEAQ Project was implemented in the entire Balaka District (population 267,512 inhabitants in 2002) from October 1999 to September 2003. The three main goals of the project were to: (1) Achieve a sustainable reduction in infant, child, and maternal mortality and morbidity; (2) Expand and refine a model for district health system strengthening; and (3) Adapt a district-to-district model for project replication and scaling up. These goals were achieved as follows: (1) The baseline and final assessments of the PEAQ Project show definite increases in the coverage of interventions known as effective in reducing infant, child and maternal mortality and morbidity. These include: - Supplementation of children under five with Vitamin A; - Supplementation of pregnant women with iron and folic acid; - Intermittent preventive treatment of malaria in pregnancy and presumptive treatment of fever among children under five with sulfadoxine-pyrimethamine; - Assistance at delivery by qualified/trained attendants and early referral in case of signs of complication; - Exclusive breastfeeding up to four months of age; - Treatment of diarrhea with Oral Rehydration Therapy; - Birth spacing and prevention of unwanted pregnancies; and - Early and effective treatment of sexually transmitted diseases. (2) The PEAQ Project built on the achievements and lessons learned from a previous successful district health system strengthening project implemented by SC and the MOHP in the Mangochi District. The key features of this model that were successfully implemented in Balaka are: - Explicit partnership agreement with the DHO; - Joint planning of activities by SC and the DHO; - Focus on community mobilization and behavior change communication; - Decentralized management of selected activities to Zone Coordinators covering the catchment areas of one or two health centers; and - No direct involvement of SC staff in the implementation of activities without designated MOHP counterparts. (3) The PEAQ Project built upon the achievements of the previous project in the nearby Mangochi District by involving its experienced staff in formal and informal mentoring relationships with the Balaka District staff. This technical assistance was provided through two-way exchange visits and participation in project activities, primarily training and management. Staff members from both districts found their participation in this process beneficial to their work performance and personally enriching. The approach also seems to have been an efficient and time saving way to launch the PEAQ Project with the appropriate tools and know-how. A complete documentation of this approach as a strategy for replicating and scaling up successful district health strengthening projects is still warranted. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 29 The most important achievement of the PEAQ Project is to have achieved the above results among the entire population of the Balaka District while building the capacity of the district health system. It is more significant that this was accomplished right after its constitution as a separate district. The PEAQ Project provided the DHO and Health Centers with human resources, technical and management expertise, training, transportation and various equipment and supplies. This support has improved the performance of health services and enabled communities to address their health needs through community-based structures and volunteer networks. The PEAQ Project provides the following lessons: - With appropriate support and resources, the Balaka DHO can achieve a high level of coverage of the essential interventions to reduce infant, child and maternal mortality and morbidity. - The formation and mobilization of the DACC/CACC/VAC structure is labor intensive and time consuming, and it has not been as effective as expected to support intensified HIV/AIDS activities in the district. - The behavioral change strategy and activities in the district need to focus on the behaviors most likely to have a positive impact and involve communities in this process of change. - The district-to-district Living University approach has been effective in replicating and scaling up the successful Mangochi District health system strengthening project. - Operations research and documentation of innovative interventions require careful planning, expertise and dedicated human and other resources. Given the success of the PEAQ Project in increasing the coverage of key interventions through a district health system strengthening approach, the Balaka DHO and SC should carefully define the human, financial, and transportation resources required to sustain these results on the basis of the recent project experience. The DHO and SC should then develop a phase-over plan that specifies the equipment that SC can dispose of and the resources that SC can continue to provide under its other projects in the district. Through its continued presence in the Balaka District, SC should provide support to the DHO to further develop the BCC and community mobilization strategy and the HMIS, strengthen the current district HIV/AIDS structures, and create new VCT centers. At the national level, the MOHP and its partners should ensure the regular supply of vaccines and other essential health products. The DHO should advocate for these supplies at the level of the Reproductive Health Unit of the MOHP. Finally, USAID/Malawi and USAID/GH/CSHGP should consider the district health system strengthening approach of the PEAQ Project and include its district-to-district replication mechanism for future health sector programs in Malawi and other countries. SC will continue to use and share lessons learned from the PEAQ Project with the broader development community in Malawi through national forums with other PVOs/NGOs and the MOHP. One opportunity for sharing this experience is the MSH led health district strengthening project that will include both the Balaka and Mangochi Districts. The lessons learned will also be incorporated into the implementation of new SC health programming in Malawi. The PEAQ experience will also be shared within SC globally through presentations during the annual program learning group (PLG) meetings. The 2003 PLG focused on district health strengthening and included the PEAQ experience. Further documentation of the Living University approach used in PEAQ will be shared in the upcoming 2004 PLG. The new mini-University DIP workshop of CS grantees is another opportunity to share SC’s program experience with PVOs. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 30 E. Results Highlights Balaka District Health Systems Strengthening Save the Children Federation, Inc. (SC) and the Malawi Ministry of Health implemented the Partnership, Equity, Access and Quality (PEAQ) Project from September 1999 to September 2003. The project covered the entire Balaka District, which was cleaved off from the Machinga District in 1998 with a total population of about 259,664 inhabitants. The PEAQ Project provided the Balaka District Health Office (DHO) and Health Centers with human resources, technical and management expertise, means of transportation and various equipment and supplies. This support improved the performance of the health centers and strengthened community-based structures and volunteer networks. In less than four years, the coverage of a wide range of interventions increased significantly for the following indicators: Indicators 1999 2003 Children 6-24 months who ever received Vitamin A 31% 39% Percent of children 0 – 5 mo. Who received only breastmilk in the last 24 hours 28% 57% Mothers of children =< 24 months with 2 or more doses of tetanus toxoid 39% 61% Mothers of children =< 24 who received iron/folate during last pregnancy 64% 87% Mothers of children =< 24 with last delivery by health worker or trained TBA 62% 73% Married women 15-49 years using modern contraceptives 21% 29% Two other important achievements of the PEAQ Project in terms of increasing the capacity of the Balaka District are (1) the successful implementation and use of data from the new national Health and Management Information System, and (2) the regular meetings of 80% of the village health committees. The final evaluation team identified the following factors of success of the PEAQ Project: - Explicit partnership agreement between SC and the DHO. - Systematic joint planning of activities by SC and the DHO. - Strong focus on community mobilization and behavior change. - Decentralized management of selected activities to the Health Center level. - No direct involvement of SC staff in implementation of activities. Also, the PEAQ Project built upon the achievements of a similar project in a nearby district by involving its experienced staff in mentoring relationships with the Balaka District staff. This technical assistance was provided through two-way exchange visits and participation in project activities. This district-to-district replication strategy has been an efficient and time saving way to launch the PEAQ Project with the appropriate tools and know-how. Given the success of the PEAQ Project in increasing the coverage of key interventions while building the capacity of the Balaka DHO, USAID/Malawi, USAID/GH/CSHGP, SC and other PVOs should consider the health system strengthening approach of the PEAQ Project and its district-to-district replication mechanism for future health sector programs in Malawi and other countries. ANNEXES Annex A. Team Members and Persons Interviewed Annex B. Methodology Annex C. Documents Reviewed Annex D. Final Evaluation Schedule Annex E. Group Interview Guidelines Annex F. Field Visits Findings Annex G. Implementation Timeline-FY00 to FY03 Annex H. Quarterly Input Monitoring Indicators Annex I. Number of Trainees by Topic and Target Group at Midterm and Project End Annex J. Training Events Attended by SC Staff Annex K. Objectives Monitoring Indicators Annex L. Selected HMIS Data CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 32 Annex A. Team Members and Persons Interviewed Name Position Team members Marc Debay Independent Consultant/Team Leader Catherine Chiphazi District Health Coordinator (PEAQ Project Coordinator, SC-Mal.) Maggie Kambalame Health Trainer (PEAQ) Joseph DeGraft-Johnson Africa Regional Health Advisor (SC) David Patterson Senior Health Advisor (SC-Malawi) Eric Swedberg Child Survival Specialist (SC-HO) Emmanuel Mwale District Health Officer, Balaka District B.J. Miso Healh Zone Coordinator, Balaka District Richard Chola AEHO, Mangochi District Alex Nyambi Clinical Officer, Balaka District Teresa Mwinjiro District Nursing Officer, Balaka District District debriefing participants C.M. Kang’ombe Clinical Officer Hillary Mnjemu Health Zone Coordinator G.B. Anafi Skin Coordinator L. Mtemang’ombe EN/M O. Tonde T.O E. Mwambwerachaje SEN C. Kamende N/Midwife AAK Nkhoma M/A T.M.Mwinjiro N/Midwife Dubulao W. Moyo Clinical Officer D.G. Chando AEHO Rose Mbawala N/Midwife Joyce Chingota N/Midwife Banks Mateketa CM Blessat Mbandambanda SECHN Ethel Kachala N/Midwife Kondwani Nkanaunena Data Officer SC Gloria Chalira Program Assistant SC Master Chitabwino Zone Coordinator SC Mike Nkhoma HMIS Coordinator Thomson Kajombo DEHO Ruth Kasiyamphanje AIDS Coordinator Blessings Chitsime MCH Coordinator J.D. Salanaye Accountant USAID Debriefing participants Cheryl Kamin USAID HPN Officer Mexon Nyirongo USAID Development Assistance Specialist/Health & Population Alfred Chirwa HIV/AIDS Project Management Specialist Joseph DeGraft-Johnson Africa Regional Health Advisor (SC) Catherine Chiphazi District Health Coordinator (SC-Malawi) David Patterson Senior Health Advisor (SC-Malawi) Marc Debay Independent Consultant/Team leader Others Justin Okopu SC Director of Malawi Office CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 33 Annex B. Methodology The Final Evaluation (FE) of the PEAQ Project was conducted according to three principles of the USAID/GH/HIDN/NUT Guidelines for Final Evaluation of the Child Survival and Health Grants Program projects. The FE was: 1. Participatory The FE was conducted by a team of 11 members from SC, the Balaka District Health Office, the Mangochi District Health Office, and one external evaluator, Team Leader and author of the report (see team composition in Annex A). The entire team contributed to the design, implementation and analysis of the evaluation. 2. Based on the project design The FE Team systematically referred to the DIP and the MTE report to develop the methodology and analyze their findings. 3. Based on evidence and data The FE Team used the following sources of data: results from the baseline and final Knowledge, Attitude and Practice Surveys and the Health Facility Assessments conducted by the PEAQ staff; data from the Balaka District HMIS; findings from various interviews conducted during field visits; and the project monitoring data collected by the SC/Balaka. The fieldwork of the FE lasted eight days, from Tuesday July 28th to August 7th, 2003 (see schedule in Annex D). After an orientation on the principles of the evaluation by the Team Leader, the FE Team discussed and agreed upon the final team composition and availability, and the overall schedule of work. The FE Team members also shared their personal expectations for the FE, which included obtaining the “big picture” of the overall health impact of the project, drawing lessons from the partnership and capacity building efforts with the DHO, drawing lessons from the district health system strengthening “model,” documenting project highlights, innovations or successes, defining the Next Steps for Balaka, and producing a good FE report. The FE Team then reviewed the overall progress towards the specific objectives of the project, and identified the main factors of success, the lessons learned, and the way forward. These discussions were conducted according to the list of topics adopted as outlined in the present report (sections A through C). The FE Team spent half a day on each category of topic (Technical Approach, Cross-cutting Approach, and Program Management). When appropriate, the relevant corresponding program manager for the topic discussed was invited to participate in the discussion. During these first two days of overall program review, the FE Team also noted the issues for which further information could be collected during the two days of field visits. The FE field visit guidelines were built on those developed for the MTE, but modified according to the specific target groups and the few additional questions selected by the FE (see Annex E). For the two days of field visits, the FE Team divided into two groups with a similar number and composition of external evaluators and SC and DHO staff. Each team interviewed the same target groups in two zones tentatively selected as more and less successful on the basis of the team’s CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 34 overall understanding of their ethnic composition, accessibility to health services, levels of education and wealth, and strength of the community-based activities. The target groups for the visits were the Health Center teams (including the HC In-charge, the Health Zone Coordinator, Health Surveillance Assistants, and any other health staff), members of the VHC and VHVs of two villages, members of two DRFs, two TBAs, a group of two CBDAs and two CMMs, and members of one drama group. The HC In-Charge in each zone was informed of the FE Team visit one or two days in advance and invited the selected target groups to come to the Health Center for the interview. The team agreed that the group discussions would be conducted with the representatives of the target groups available at the time of the visit. A description of the samples of sites and interviewees is presented below with estimates of the corresponding Total. Sampling Unit Realized Sample Total Zone 4 6 Health Center Team 3 9 Community Health Committee members ~10 591 Community Health Volunteer ~10 888 Drug Revolving Funds 4 34 CBDA ~10 80 Community male motivators ~6 44 TBAs (trained) ~10 90 HIV/AIDS motivator ~10 58 Drama groups 4 - After the two days of field visits, each group summarized its findings for each zone visited according to the main topics relevant to the Technical and the Cross-Cutting Approaches of the PEAQ Project. The entire FE Team then discussed these findings once they were written up and included in one table (see Annex F). No specific conclusion was drawn from the field visit findings alone; rather, the team referred to these findings as appropriate during the various discussions that took place during the following days. The FE Team gathered and reviewed together all the training and IEC materials available to the PEAQ Project. This exercise provided the opportunity to discuss how the training curricula and activities were developed. The FE Team also constructed the overall timeline of the main project events and activities to discuss the pace of project implementation and identify potential delays and lapses in human resources (see Annex G). The Team Leader and the representative of the Mangochi District conducted a group interview of the Balaka CS-15 staff to investigate their motivation and satisfaction with their job, and review issues that had not been addressed during the previous group discussions and field visits. The interview guidelines are included in Annex E, and the findings from these interviews are included in the various sections of the report as appropriate. The Team Leader and the representative of the Mangochi District also conducted an interview of the Balaka District Health Officer. These findings are also included in the report, where appropriate. Prior to the FE, the PEAQ Project conducted the final household surveys and a health facility assessment, and prepared all the relevant data on project activities. These two surveys were conducted according to the same methodology as the baseline surveys. The entire FE Team discussed the preliminary results of the two final surveys and selected the key data needed for the CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 35 FE. A few FE Team members also reviewed the data available from the national and Balaka HMIS, and selected the key data directly relevant to the FE (see Annex L). The last day of the FE Team work consisted in the presentation of the main findings and recommendations to the entire SC, DHO and Health Center staff involved in the PEAQ Project implementation (see debriefing participants in Annex A). The discussions that followed each presentation were an excellent opportunity for all the SC/Balaka and DHO staff to contribute to the FE, and also to make decisions about future activities in the district such as continuing the EPMU meetings and continuing the collaboration with Mangochi DHO as a Living University. The meeting was also an opportunity for the SC/Balaka and DHO staff to express their appreciation of their fruitful collaboration and of the results of their efforts. At the end of the debriefing day, a rapid anonymous survey was conducted which included three simple questions: Was the final evaluation “fair”? Was the methodology “OK”? Do you have any other comment? The majority of the 29 respondents found that the evaluation was fair and appreciated the overall methodology. This evaluation of the FE also gave the opportunity to many respondents to share useful constructive feedback that was included in the final report, as appropriate. The above FE methodology had the advantage of encouraging full participation of most PEAQ staff, giving each one the opportunity to bring valuable information and perspective to the discussions and analyses and learn about the project achievements. The methodology has limitations such as the difficulty to cover all the topics with a large group of members of different background in a relatively short period of time. Also, the sample of sites and interviewees for the field visits was not necessarily representative, and only general interview guidelines could be developed in the two days available to the entire team before conducting the field visits. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 36 Annex C. Documents Reviewed Detailed Implementation Plan, March 2000. Knowledge, Attitudes and Practices in Health Survey 1999, March 2000. Quality Assurance Assessment of Health Facilities in Balaka District 1999, March 2000. First Annual Report, November 2000. Midterm Evaluation Report, February 2002. Mid-term Evaluation Action Plan, June 2002. Third Annual Report, October 2002. Child Survival 15 – Malawi LIFE Amendment, April 2001. Quarterly Report - January to March 2003. Report on EPMU meeting held from 21st to 23rd of May, 2003. Malawi Health Management Information Bulletin. Special Issue 2002. MOHP, Planning Department, Health Management Information Unit, June 2003. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 37 Annex D. Final Evaluation Schedule DAY DATE ACTIVITY Monday July 27 Briefing at SC Office in Lilongwe Travel to Balaka Tuesday July 30 Team Meeting with District Health Office Team building and development of evaluation work plan Review of PEAQ Program Management Wednesday July 31 Team meeting: Review of Technical Approaches Review of Cross-Cutting Approaches Thursday August 1 Field visits by two groups: --Health centers and community members in Phimbi --Health centers and community members in Phalula Friday August 2 Field visits by two groups: --Community members in Balaka Central --Health centers and community members in Kalembo Saturday August 3 Team meeting Review of field visits findings and of household survey results Sunday August 4 Rest Monday August 5 Team meeting Collection and Analysis of additional data Group interview with PEAQ staff Tuesday August 6 Team meeting Collection and Analysis of additional data Meeting with DHO Preparation of District Debriefing Wednesday August 7 District-level Debriefing Thursday August 8 USAID Debriefing CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 38 Annex E. Group Interview Guidelines The field visits and group interviews constitute one component of the final evaluation of the PEAQ Project. They aim at providing a snapshot of the results of the project in four of the six zones of Balaka District. Two zones are selected as the most “successful” ones while the two others are selected as being less “successful.” Comparing the findings in the four zones will specify these differences and help identify the main factors of project success. For each group interview, record: Name of the interviewers Date and time of interview Name of village Number of persons interviewed, per category Health Center Team 1. What do you see are the greatest strengths or accomplishments of this project? 2. How have your practices or activities changed since the project started? 3. What training have you received since the project started? How did it help your work? 4. What are the key messages for each intervention? ARI_____________________________________________________________ Malaria__________________________________________________________ Diarrhea _________________________________________________________ Immunization ____________________________________________________ Breastfeeding_____________________________________________________ Family planning___________________________________________________ Safe motherhood __________________________________________________ STI_____________________________________________________________ HIV/AIDS _______________________________________________________ 5. What are the main activities conducted to promote these messages and behaviors in the community? 6. What outreach activities do you conduct? Who is responsible for these activities? How often do you carry out each of these activities? 7. Who comes for supervision? Do they use a supervisory tool? When was the last time you were supervised and what problems did you discuss? 8. What outreach activities are provided in your zone by the DHO? How these outreach clinics are complementing the HC services? (Probe for type services provided; regularity; quality; any specific issues when the CS-15 ends) 9. Are the necessary drugs and supplies available? Has there been any stock out for any drugs/antigens during the past 5 months? (Probe for co-trimoxazole; SP; ORS; Vaccines; Iron/Folic Acid; condoms; pills; Depo-Provera; STI drugs) CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 39 10. Do you have DRF in your catchment area? If yes, how do the DRFs get their drug supply? What mechanisms do you have for supervision of DRF? 11. Are the fees for the drugs a constraint to their utilization? (Probe Iron/Folic Acid, SP). 12. What is the HMIS at the HC/Zone level? Who collects information, who analyzes it, who uses it, does it reflect what you are doing? (Look at some forms and reports; ask for examples of how the data are used) What are the trends in utilization of services? (Probe specifically malaria and FP). 13. Who in the communities does not access health services? What are the barriers? 14. How did the hunger last year affected the children’s health? (Probe on malnutrition, breastfeeding, use of health services) 15. Do you think many women in your zone would like to use FP but do not? Why? What can be done to respond to their needs? 16. Is the HC/Zone fully staffed? What is the staff turnover? 17. What plans are there for continuing these activities beyond the end of the project? Which will be the most important activities to maintain and why? (Observation: do they feel like the project belongs to them) VHCs and VHVs 1. What are the most important health problems you are addressing? Which of your activities have made the biggest difference to health in the village? 2. What are the key messages for each of these interventions? ARI_____________________________________________________________ Malaria__________________________________________________________ Diarrhea _________________________________________________________ Immunization ____________________________________________________ Breastfeeding _____________________________________________________ Family planning___________________________________________________ Safe motherhood__________________________________________________ STI_____________________________________________________________ HIV/AIDS_______________________________________________________ 3. How did you learn about these messages? 4. How do you help people in your village learn about these messages and adopt the appropriate behaviors? 5. When were you last supervised and what did you discuss? 6. What kinds of health information or reports are collected in the village? 7. Is there as DRF in the village? What is your role in supporting the DRF? 8. Community AIDS support systems –What activities are done by the HBC volunteers? What support groups are available in the community and what are their roles? How are orphans taken care of and by whom? 9. How often does the VHC meet? How many times last year? When was the last meeting? 10. What plans do you have to maintain the activities we have discussed after the project ends? CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 40 DRFs 1. How were you trained in DRF? Duration of training? Content? 2. What patients (children or everybody) do you see at your DRF? 3. How do you handle patients with inadequate or no money to buy drugs? 4. What do people prefer between the health facility and DRF in relation to drugs which they dispense? 5. What types of drugs are available? How is the price determined? How many different drug dispensers have been trained in this village? 6. Who keeps the money for the DRF? Drug box? What control measures are in place for the program in terms of money and drugs? 7. How do you make sure that there is a consistent supply of drugs? 8. Where do you order drugs from? 9. How do you record and report your activities? (Look at forms and reports; discuss utilization, sales, trends) 10. What are the key messages you educate mothers on when they come to your DRF? TBAs 1. How long have you been delivering babies? How many deliveries did you do last month? 2. Where were you trained? How long did your training last? 3. What advice and care do you give to a pregnant woman? (Probe about birth planning) 4. What advice and care do you give to a woman during labor & delivery? 5. What advice and care do you give to a woman soon after delivery? (Probe on how the cord was cut and what care was given; Immediate and Exclusive breastfeeding; counseling on STI/HIV/AIDS) 6. What kinds of medications do you give to the mother? (Probe about SP) 7. If a mother would die what would you do? (Probe for dialogue with district health personnel) 8. Which symptoms/cases would you refer to the health center? (Probe about newborn) 9. Where do you refer them? Do they go? If not, why? 10. How do they go? (Probe for alarm and transport; bicycle ambulance) 11. Can you show us your referral form and how you fill it out? What is supposed to happen with this form? In practice does this happen? 12. Do they have any feedback either from the mother or the health center? 13. How do you replenish your TBA kit? 14. What is the nearest health center? Has the nurse from the HC ever visited you? Have you been visited by any other nurse? How frequently? When was the last? 15. What challenges do you face? 16. Do you think many women in your zone would like to use FP but do not? Why? What can be done to respond to their needs? 17. Do you keep any records? Can we look at them? CBDAs and CMMs 1. Where were you trained? How long did your training last? 2. Do you serve both male and female clients? What are the challenges associated with each? 3. What is your total number of clients? Male/female? How many new clients do you see in the last month? 4. What methods do you counsel your clients on? CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 41 5. For male motivators: What methods do you counsel men on? (Probe: vasectomy) How do you motivate other men to use modern contraceptives? 6. What contraceptives do you supply? Where do you get your supplies from? Have you had any stock outs? 7. How do you handle clients who chose methods you can’t supply? 8. What are some of the side effects of oral contraceptives? 9. When and to whom do you refer clients? Do people go when referred? And do you get any feedback? 10. Please demonstrate how a condom is put on? 11. What material do you use in counseling your clients? Can we look at them? 12. What are the relationships between CBDAs and male motivators? 13. How do you work with HSAs and HC staff 14. Do you keep any records? Can we see them? 15. What challenges do you face? 16. Do you think many women in your zone would like to use FP but do not? Why? What can be done to respond to their needs? Drama Groups 1. Training received 2. Activities conducted 3. Messages promoted 4. Response/Reaction of the community HIV/AIDS and Initiation Counselors 1. Training received 2. Activities conducted 3. Successful results 4. Major challenges Interview Guideline for PEAQ Staff 1. What are your main achievements since on the job? 2. Can you describe your job? 3. What are the main difficulties you are facing in your job? 4. How do you collaborate with your colleagues? 5. How does your direct supervisor help you in your job? 6. What are your plans when the proje ct ends? CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 42 Annex F. Field Visits Findings Phimbi Balaka Central [Community level only] Phalula Kalembo Technical Approaches Immunization Trained VHC/VHV have resulted in increased awareness and coverage of EPI/GM VHCs have good knowledge. They involve religious and community leaders. They do not specify when the immunization needs to be completed when communicating with parents VHCs have good knowledge Appropriate messages are communicated to the target populations Cold chain has been strengthened at the HC level from training of technicians and provision of maintenance tools, cold chain equipment, kerosene, gas cylinders, etc No midlevel EPI manager training There are 12 outreach sites for U5 clinics conducted by HSAs once per month There are 12 outreach sites for U5 clinics; 6 of 12 also provide family planning services The HC experienced BCG stock outs No vaccine stock outs reported There have been stock-outs of some antigens from time to time. Recently, there was no vaccines for 2 months. Maternal and Newborn Care Improved infection control from provision of kerosene and TBA training TBAs are confident and have a good understanding of their work Stronger knowledge of risk factors than danger signs at the HC Improved referrals from community to HC from TBA training, bicycle ambulances The TBAs expressed a need for bicycle ambulances Birth planning messages were not mentioned at the HC Improved referrals from HC to Hospital from provision of radio to HC by SC Provision of radio to HC by SC Some mothers do not deliver with trained providers because of the cost in the CHAM facility. Increased number of deliveries at HC CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 43 Phimbi Balaka Central [Community level only] Phalula Kalembo TBAs trained in 2000-2001 (including refresher): --know referral criterias (primi-gravidae, short height or limps, previous scar); --advise to eat well, reduce work, prepare for birth (clothes, razor blades, parafin for lamp, potential transport); --advise to stay in lateral position, eat, void, not push before being told; --advise on immediate and exclusive breastfeeding. TBAs trained by GTZ (4 weeks) and refreshed by CS-15 (1 week) give advice: -- to pregnant women: personal hygiene, go to hospital if risk factors; --on birth plan: prepare 2 clothes; --on labor: rest on her side, eat porridge, frequent voiding, place baby to breast immediately after delivery; examines placenta --on HIV/AIDS: reduce # partners, treat STIs, TBAs have adequate knowledge of danger signs and generally have a good understanding of their work; They counsel on HIV/AIDS, STIs, EBF, and SM TBAs use gloves. TBAs do 8 to 36 (exception!) deliveries per month. TBAs are supervised once per month and go to the Hospital every month; replenish kit (gloves, drugs) when supervised by TBA coordinator or CS-15 staff Kit is replenish by the HC staff during meeting at HC or supervision TBAs report maternal death to TBA coordinator. Report maternal death to DHO Refer to Phimbi or Utare; Would refer for slow progress, transversal lie, bleeding, flabby or absence/high-pitch cry newborn. Refer to Hospital for excessive bleeding, anemia, cord prolapse, failure to deliver placenta, premature, not crying baby, jaundice, failure to suck, not passing stools. Usually get feedback from hospital on referred women; use referral form (not seen). TBAs used to provide SP (at 5 and 8 months); they provide IFA but there are shortages. TBAs used to give SP at 5 and 8 months; provide iron and postpartum Vitamin A. TBAs keep good records. TBAs face problems during some deliveries from lack of water and the absence of anybody accompanying the women in labor TBAs’ challenges are they are far from HC and often lack supplies CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 44 Phimbi Balaka Central [Community level only] Phalula Kalembo Family Planning Improved geographical accessibility from outreach clinics last year (1 DHO and 6 HC) No new outreach clinics. There are 6 outreach clinics – recently due to the ambulance and motorcycle breakdown the nurse provider used own money to get to the clinics Male motivators trained in June 2003 know all main methods and ways to talk to men about FP; would like demonstration and education materials, and some identification sign like uniform; keep records (register of activities) and send report to HSAs; face difficulty in motivating males in using condoms and vasectomy, and in transport. 1 CMM trained in Jan 2002 Use register of activities and report of activity/supply to HSA or HSA??? Registers 10-15 activities per months (individual counseling and health talks) One district staffed FP clinic is supposed to be held in the town on a monthly basis but has been irregular FP messages are clear and appropriate; VHCs also had good knowledge CBDAs trained by SC in July 2001 in FP and STI: --distribute free pills, condoms and foam tablets to women; --face problems of misconception and beliefs about FP, lack of communication with HC (lack of referral forms; no feedback; difficulty to get supplies), a too large area to cover, lack of supplies, lack of identification sign (card) and lack of incentives; --have flipchart, a demopenis, a drugbox, and umbrella. 1 new CBDA trained in June 2003 and 1 CBDA trained by GTZ in 1996: --provides pills, spermicides, condoms, and counsel on other methods and refer to hospital --serve men and women. --face some resistance from the community even though there is demand at the individual level. --is aware of misconception about contraceptives (“pill do not dissolve but accumulate in stomach”) --do not give clear answer about side effects and contraindications of pills. --get supplies from MOHP; experience stock outs. --use register of activities and report of activity/supply to HSA --need demonstration and visual education materials. --“work hand-in-hand” with CMMS --will miss training, supply, and supervision at project end Some CBDAs are quite active and competent; Other CBDAs have lacked motivation and some have stopped their activities Client compliance with referral is low – most just use the CBDA services The 14 CBDAs originally active have reduced to 9. Only 6 are very active - Generally good knowledge Good demonstration of condom use They “work hand-in-hand” with CMMs and refer clients to HSAs Challenges include lack of transport (push bicycle) and supervision CBDA/CMM Volunteers: --Average 6-7 clients per month --Encourage male leadership and participation in FP --Stock out of condoms for two months --Generally good knowledge --Good demonstration of condom use --They work hand-in-hand and refer clients to HSAs --Challenges include lack of transport (push bicycle) and supervision CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 45 Phimbi Balaka Central [Community level only] Phalula Kalembo There are still many women who would like to use FP but do not because of the large distance to FP services; also resistance from religious beliefs and misconceptions. TBAs do not think there is unmeet need in family planning Many women would like to use FP but their husbands refuse Some Muslims believe FP is prohibited by their religion Sexually Transmitted Disease Increased in number and quality of STI case management at HC level from training of HC staff in syndromic approach, improved drug availability, partner notification system, etc. HC staff very appreciative of training and are very active. Staff very appreciative of training – “I’m a STI manager” HC documented stock outs in several STI drugs There is an erythromycin stock out and syringes are not available – they use the EPI syringes Appropriate messages Appropriate messages HIV/AIDS Recently trained HIV/AIDS counselor already sees about 14 clients a month at the HC HIV/AIDS counselor / CHBC trained in 2001-2002 by GTZ has following activities: --was introduction by village chief as taking care of chronic diseases; --conduct group talks and house-to-house visits; --more and more people are asking for VCT and she refers them to Balaka. --if patients open: talk about VCT and how not to infect people; --currently cares for 3 patients: washing, drugs (CS-15 provided kit); --covers 4 villages, but thinks that there should be more counselors in the area; --faces challenges: transport; lack of pain killers; no education materials. HIV/AIDS counselor: --gives talks during U5 clinics, OPD --notes that the HIV+ clients she has counseled are living positively --challenged with timely transport of VCT samples --only referred 3 clients for testing since 2001. Counseling training resulted in a marked attitudinal change towards HIV/AIDS clients 3-4 VCT samples sent to Balaka per month except in the last six months when reagents were not available in Balaka VHCs have good knowledge VHCs have good knowledge Orphan care in the communities No orphan care in the communities CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 46 Phimbi Balaka Central [Community level only] Phalula Kalembo Malaria/ARI/Diarrhea ARI messages lack clarity at the HC level ARI messages lack clarity at the HC level Malaria messages clear For diarrhea they are promoting the use of other rehydration fluids but did not mention ORS VHC/VHVs have appropriate diarrhea and malaria messages Breastfeeding TBAs know importance of Immediate and Early Breastfeeding TBAs know importance of Immediate and Early Breastfeeding Messages are appropriate Messages unclear on exclusive BF but clear on immediate BF VHCs have appropriate knowledge Some mothers fail to EBF during the hunger period last year Cross-cutting Approaches Community Mobilization Major activities and accomplishments of VHC/VHV are in the area of cholera control (education, sanitation, epidemic surveillance) and Growth Monitoring --Main activities of VHVs are domestic hygiene and growth monitoring. --They are doing house visiting and community meetings VHCs have raised funds for transport and welcoming visitors through gardens or member contributions. They are very confident and clear about their activities 90% of VHCs trained 46% of VHVs trained VHC/VHV meet regularly (average 8 times last year) Strong support for HSAs now in the zone compared to before the project when they were working “alone” Have appropriate knowledge of most intervention areas Have appropriate knowledge of most intervention areas Focus on sanitation is a large part of their work Focus on sanitation is a large part of their work BCC VHC/VHV and HC staff were quite knowledgeable about most key messages of the project VHC/VHV know major messages, although not much was mentioned about SM and early malaria treatment VHCs had a good knowledge of messages The main behavior change activities of VHC/VHV are: health talks and house-to￾house visits CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 47 Phimbi Balaka Central [Community level only] Phalula Kalembo HC staff are conducting health talks in HC and outreach clinics Health workers did not have very clear messages on ARI, BF and immunization Drama groups (10 members) trained in interactive drama, key health messages and script are active in the area; well received by the population. Drama group (10 members) trained in Sept 2002 on messages, script, and interactive drama: --do research in the community before developing script; --meet twice a week to prepare; perform twice a month; --currently have two plays related to health: 1 on HIV/AIDS, and 1 on marijuana; --will continue at the end of project. Use of drama groups is important – there are two groups Drama presentations in schools The drama groups are well organized Use of drama groups is important: Also use flipcharts Weekly drama presentations during FP clinics and U5 clinics They are well organized and have a schedule for the next six months Also have used house to house visits to spread messages DRF DRF members were trained for 5 days in April 2003 in signs and symptoms and in dispensing drugs for malaria, scabies, conjunctivitis and in record keeping. Trained in February 2003 during 5 days on malaria, scabies, conjunctivitis, diarrhea, pneumonia, AIDS. 5 DRFs in zone and are supported/supervised by the HSAs 9 DRFs were established and are supported/supervised by the HSAs Training was 6 days DRF members see adults and children, mainly for malaria and colds; they deliver messages on hygiene, how to take drugs, and malaria prevention; they accept partial payments but always provide full treatment. DRF see mainly children (3/4); provides health message at first visits; and refer if no improvement; provides SP, aspirin, eye ointment, BBA, ORS, paracetamol, condoms; refer to village chief if no money, but give the entire treatment. Appropriate understanding of when to make referrals The manager interviewed only mentioned malaria Drugs are replenished the through the district pharmacy DRF well appreciated by the population because of the otherwise long distances to the HC. Community appreciates DRF because it is closer than HC Community feels SP is expensive (almost 3 K/tablet) The treasurer keeps the money and the HSA checks during supervision visits. One treasurer was selected but she died DRFs are well managed – established in 2000 DRFs are well managed DRF reorder to the DHO when ¼ of drugs remain; no drug box. Reorder from Hospital Drugs are replenished through the district pharmacy\ Price of drugs determined by the committee based on cost, transport, and price in local pharmacies. Price determined by cost and transport Drugs are sold slightly above cost Drugs are sold slightly above cost DRF record on register and report on forms, but these were not properly maintained Records well kept; mainly SP is provided. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 48 Phimbi Balaka Central [Community level only] Phalula Kalembo Capacity Building VCH/VHV have been trained in CS messages and activities Both health workers and community volunteers benefited from numerous trainings Both health workers and community volunteers benefited from numerous trainings HC have increased their capacity to deliver CS interventions through training in FP, STI, cold chain, HMIS, HIV/AIDS counseling HC staff has also been trained and supported in implementing QA problem solving activities (last topic was low use of FP and led to opening of new outreach clinics) The QA training helped to address the problem of women taking SP due to the cost. They decided to include the cost of SP in the initial consultation fee. They have not used QA since then due to lack of coaching and turn over of staff. HC has improved transport capacity (motorbikes, fuel) that allows to reach underserved areas: 6 monthly outreach clinics have recently been established HC has increased capacity to hold zonal meetings linked to EPMU meetings; HMIS data are used in this process. EPMU quarterly meetings have been important to the staff to compare their work with other zones in the district and as learning opportunities HC has experienced stock outs (SP, Vit A, Iron, BCG, STI drugs, others) of major drugs last year as a result of problems at the national level DHO does not supervise the HC DHO holds a monthly outreach clinic in the zone The zone has been supervised by the DHO and FP Coordinator but not regularly; HMIS is more regular TBAs are not regularly supervised by the nurses due to transport difficulties The zone has been supervised by the DHO and FP Coordinator but not regularly; HMIS is more regular; Supervisors do not use checklists TBAs are not regularly supervised by the nurses due to transport difficulties but they are visited by HSAs HSAs visit VHC/VHV every two weeks HSAs regularly supervise the VHCs/VHVs CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 49 Phimbi Balaka Central [Community level only] Phalula Kalembo HMIS All HC staff trained in new national HMIS; Zone coordinator plays major role in increasing quality and use of HC data They receive regular supervision from the district They receive regular supervision from the district Each staff member responsible for collecting and reporting data for its activity; all staff meets monthly to review data; report to district and review of all zones results on a quarterly basis; The HC staff explained some trends but were unclear on decisions taken in response to these trends The HC staff explained some trends in FP and malaria from their HIS VHV/VHV report to HSAs on head count, diseases surveillance, households with pit latrines, etc. VHC/VHV keep good records of vital events etc. from their catchment areas Sustainability Main support needed by HC staff to continue project activities are training, transport and outreach clinics. The HC staff feel they will be able to continue to conduct local trainings in the zone VHV/VHV will continue their activities at the end of the project VHC/VHV feel they can continue after project ends because they have knowledge and skills VHCs will have fund raising activities VHCs will have fund raising activities Miscellaneous Observations Geographical barriers (river, distance) are main caused for underserved populations A small percentage of the population (<10%) do not use health services for religious reasons Food crisis last year had impact on children; HC set up feeding center, still active. Several positions in the HC are vacant, mainly because most MOHP staff do not want to work in remote areas. The HC lack one medical assistant and the zone lacks one HSA; Low turn over of staff The HC is adequately staffed According to MOHP norms the zone lacks 12 HSAs; Low turn over of staff CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 50 Annex G. Implementation Timeline – FY00 to FY03 Fiscal Year (October to September) 2000 2001 2002 2003 Selected Project Activity / Landmark I II III IV I II III IV I II III IV I II III IV Management Awards CS￾15 LIFE SC Office operational X X X X X X X X X X X X X X X X Planning KAP HFA DIP wko DIP App. LQA Act.P lan KAP HFA External evaluation: Midterm / Final MTE FE SC/DHO Partnership Agreement / Update X X Number of zones established 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 Human Resources1 SC Senior Health Advisor 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 SC CS-15 Coordinator 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 SC Trainer 1 1 1 1 1 1 3 3 3 3 5 5 2 2 2 2 2 2 2 2 2 2 4 4 4 4 4 4 SC Data Officer 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 3 DHO (Tamaona / Mwale) 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 DNO 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 Office manager 1 1 1 1 1 1 1 1 1 1 1 1 1 Training of Health Workers and Community Volunteers VHCs/VHVs X X X X X X X X X X DRFs X X X X X X X X Interactive drama X X X X X X X X X X X X X X TBAs X X X X X X X X X X X X X X CBDAs/CMMs X X X X X X X X X X X X X X HIV Counselors X X X X X X HBC Counselors X HMIS X X X STI X X X CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 51 Fiscal Year (October to September) 2000 2001 2002 2003 Selected Project Activity / Landmark I II III IV I II III IV I II III IV I II III IV QA X X BEHAVE X DAC/CAC (Training for Transformation) X X HMIS National official launch X X X X X X X Quaterly Balaka District HMIS reports X X X 1 Numbering corresponds to the successive staff members recruited for that position CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 52 Annex H. Quarterly Input Monitoring Indicators Indicators Baseline Target 1999 Final 2003 Number % reached Increase the number of outreach clinics 108 135 128 105% Train Village Health Volunteers 82 888 1140 78% Increase number of HSAs 120 146 150 97% HSA Refresher 120 119 120 99% Train health center staff in monitoring and evaluation at HC level 0 9 9 100% Train cold chain technicians 1 6 6 100% Organize mini-immunization campaigns in all five health zones 0 ? 2 0% Train Village Health Committees 0 299 321 93% Train Drama groups (about 10 persons/group) in interactive drama 0 16 25 64% TOT in interactive drama 0 7 7 100% Train TBAs Initial 34 32 106% Refresher 58 87 90 97% Establish maternal audit teams 0 2 9 22% Train Core FP providers 0 89 150 59% Train CBDAs – initial 3 80 121 66% - refresher 11 No plan Train CBDA Supervisors 10 24 42% TOT of CBDAs 0 3 2 150% Train man-to-man motivators N/A 44 75 59% Train man-to-man motivators (Refresher) 0 20 75 27% TOT of man-to-man motivators 0 4 4 100% Integrate FP services into outreach clinics 35 ?? 128 % Establish District AIDS Coordinating Committee 0 1 1 100% Establish Community AIDS Committees per zone 0 4 15 27% Establish Village AIDS Committee 0 0 120 0% Establish voluntary counseling and testing centers in each of the five health zones 1 1 3 33% Train clinical officers, medical assistants and nurses in syndromic management of STIs 3 20 33 61% STI refresher 33 7 33 21% Organize anti-AIDS days 0 3 5 60% Organize FP days 0 1 6 17% Train traditional healers in STI/HIV/AIDS recognition and preventive messages 0 0 0 0% Train Anankungwis in STI/HIV/AIDS recognition and preventive strategies 0 102 75 136% Establishment of community HIV/AIDS support groups per zone 0 0 0 0% Establish DRFs 25 34 42 81% Train DRF volunteers 75 297 135 220% CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 53 Indicators Baseline Target 1999 Final 2003 Number % reached Villages with access to functional Bicycle Ambulance 66 162 321 50% Functioning Village Health Committees ? 591 647 91% Sensitization of local leaders in STI/HIV/AIDS 0 150 150 100% Train community Based HIV/AIDS counselors 0 58 140 41% Train HBCC (Initial) 80 100 80% HMIS Data – Jan. – Mar. 2003 Immunization of children (U/1 year) BCG 77% 65.5 80% DPT3 63% 60.3 80% Polio 3 66% 52.0 80% Measles 52% 51.0 80% Fully Immunized 46% 35.4 80% Vitamin A given to under five children 38% 66.2 80% Percentage of pregnant women who received at least TT2 vaccine 80% 72.8 90% Percentage of pregnant women provided iron and folate (at least 1 dose) 66% 17.1 4 doses 80% Family Planning CPR 21% 11.7 30% Percentage of deliveries attended by trained personnel 52% 44.6 80% Percentage of deliveries attended by trained TBA 13% 29.3 5% Percent of children who received only breast milk up to at least 4 months of age 10% 20% Sensitization of local leaders in STI/HIV/AIDS 0 150 150 78% Train community Based HIV/AIDS counselors 0 58 140 78% Train HBCC (Initial) 80 100 78% CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 54 Annex I. Number of Trainees by Topic and Target Group at Midterm and Project End TOPIC TARGET GROUP MID TERM FINAL Planned Actual Planned* Actual % Target TBA Initial Community Volunteers 18 17 32 34 106% TBA Refresher Community Volunteers 18 0 90 87 97% HSA Refresher Community Volunteers 59 62 150 146 97% CBDA supervisors Health Personnel 12 6 24 15 63% CBDA Initial Community Volunteers 35 29 121 80 66% CBDA Refresher Community Volunteers 14 11 14 11 79% VHCs Community Volunteers 321 52 321 299 93% VHVs Community Volunteers 840 625 1140 888 78% STI case management Community Volunteers 15 2 33 20 61% STI TOT Health Personnel 2 2 2 2 100% Core FPP HSA 50 43 100 89 89% Initiation Counselors Community Volunteers 75 40 75 92 123% HBCC Community Volunteers 100 80 100 80 80% HMIS Health Personnel 4 2 62 58 94% HMIS TOT Health Personnel 3 4 3 4 133% QA TOT Health Personnel 8 3 8 3 38% QA teams Health Personnel 4 4 9 4 44% FP Providers (by MOHP) Health Personnel 5 2 10 2 20% DRF (Committees) Community Volunteers 20 3 45 34 76% CACs Community Volunteers 20 0 40 9 23% DACC (Sub committees) Multisectoral 5 0 5 4 80% CMM Community Volunteers 30 20 75 40 53% Cold Chain Technician Health Personnel 5 6 5 6 120% Drama Groups (Interactive) Community Volunteers 12 10 25 16 64% TOT in Interactive Drama Health Personnel 7 7 7 7 100% Monitoring and Evaluation Health Personnel 9 9 9 9 100% STI refresher Health Personnel 15 7 15 7 47% HIV/AIDS Counselors Health Personnel 12 12 24 12 50% HIV/AIDS Counselors Community Volunteers 70 20 140 58 41% STI/HIV/AIDS/RH Integration Health Personnel 0 0 75 68 91% Infection prevention Health Personnel 0 0 75 68 91% HMIS Health Personnel 3 4 75 62 83% EPI Mid level managers Health Personnel 0 0 27 25 93% BCC Strategy Development Health Personnel/ CS-15 staff 0 0 18 12 67% TFT for Zone Coordinators Zone Coordinators 0 0 6 5 83% DACC TFT Multisectoral team 40 15 40 36 90% DRF TOT Health Personnel 11 0 11 11 100% New FP logistic system orientation Health Personnel 0 0 7 7 100% Integrated BFHI and HIV/AIDS Health Personnel 0 0 2 2 100% * Final targets as adjusted after the midterm evaluation. CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 55 Annex J. Training Events Attended by SC Staff DATE TRAINING EVENT PEAQ STAFF April 2000 Training of Trainers Health Trainer January 2001 LQAS Data Officer, Health Trainer April 2001 Quality Assurance Health Trainer September 2001 Report writing workshop District Health Coordinator October 2001 STI Syndromic Approach Health Trainer February 2002 BEHAVE Framework, South Africa District Health Coordinator, Senior Health Advisor December 2002 Integration of Reproductive Health and STI/AIDS Health Trainer March 2003 Qualitative Research Data Officer September 2003 Community mobilization Zone coordinator (MOHP), Health Trainer CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 56 Annex K. Objectives Monitoring Indicators INDICATOR BASE- MIDTERM FINAL LINE Target Actual Target Actual Immunization 1 Percent of children age 12-23 months fully immunized by their first birthday 46% 55% 60% 75% 50% 2 Percent of mothers of children age =< 24 months who received 2 or more doses of tetanus toxoid during their last pregnancy 80% 85% 85% 90% 87% 3 Percent of children age 6-24 months who ever received Vit A 31% 20% 46% 40% 39% Breastfeeding Promotion 4 Percent of children who received only breast milk up to at least 4 months of age 28% 15% 68% 20% 57% Maternal and Newborn Care 5 Percent of mothers with children age =< 24 months who received iron/folate supplementation during their last pregnancy 64% 75% 77% 80% 87% 6 Percent of mothers with children age 24 months or under who received at least one dose of SP during their last pregnancy 72% 75% 63% 80% 77% 7 Percent of mothers with children age 24 months or under who report last delivery assisted by a health worker or trained TBA 62% 70% 70% 75% 73% Child Spacing 8 Percent of married women 15-49 years old using modern contraceptives 21% 25% 18% 30% 29% STI/HIV/AIDS Prevention 9 Percent of women 15-49 years old who report condom use with non-regular partner 38% 30% NA 50% 61% 10 Percent of women 15-49 years old who know of three ways to prevent STI/HIV 17% 30% 16% 60% 12% 11 Percent of men 15-54 years old who report condom use with non-regular partner 25% 60% NA 70% 57% 12 Percent of men 15-54 years old who know of three ways to prevent STI/HIV 24% 30% 29% 60% 20% 13 Percentage of female youth 15-19 years who delay sexual debut 21% 25% NA 30% 29% 14 Number of persons tested in VCT centers/month 0 0 8 350 10 15 Percent target orphans/vulnerable children (OVC) with access to community services (OVC program) 0 0 NA 50% NA 16 Percent STI cases treated according to national standards (STI treatment protocol) 20% 20% NA 75% 50% 17 Percent of households receiving help in caring for chronically ill young adults (15- 49) in a population survey 0% 0% NA 50% 9% 18 Percent of male youth 15-19 years who delay sexual debut 19% 25% NA 30% 38% Control of Diarrheal Disease CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 57 INDICATOR BASE- MIDTERM FINAL LINE Target Actual Target Actual Percent of children age 24 months or under who have had diarrhea in the two weeks prior to the survey 28% 41% 24% 19 Percent of children age 24 months or under who have had diarrhea in the two weeks prior to the survey and who have been treated with Oral Rehydration Therapy (ORT) 65% 75% 59% 80% 77% 20 Percent of children age 24 months or under presenting at health facility with diarrhea who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 59% 70% NA 80% Not Measured Control of Malaria Percent of children age 24 months or under with fever in the two weeks prior to the survey 40% 50% 44% 21 Percent of children age 24 months or under with fever in the two weeks prior to the survey who obtained medical care from a health facility 35% 40% NA 50% 51% 22 Percent of children age 24 months or under presenting at health facility with fever who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 31% 55% NA 70% 23% Pneumonia Case Mgmt. Percent of children age 24 months or under with cough and difficult/rapid breathing in the two weeks prior to the survey 5% 23% 23 Percent of children age 24 months or under with cough and difficult/rapid breathing in the two weeks prior to the survey who obtained medical care from a health facility 50% 55% NA 70% 43% 24 Percent of children age 24 months or under presenting at health facility with cough and difficult/rapid breathing who were treated correctly (in accordance with the Malawi Standard Treatment Guide) at the health center level 44% 55% NA 70% Not Measured OTHERS 25 50% of villages have formal alarm and transport plans 0 25% 59% 50% 50% 26 80% of DHMT meetings discuss current HMIS information and make management decisions based on the data 0 40% 50% 80% 100% 27 Each of the 9 trained HC QA teams investigate and solve at least 3 problems per year after training 0 5 4 trained 9 0 28 A Functioning DHMT-NGO District Coordination Comm. (DHPCC) NO YES YES YES NO CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 58 INDICATOR BASE- MIDTERM FINAL LINE Target Actual Target Actual 29 A District Fundraising Committee established and mobilizes resources for at least 2 projects per year NO YES NO YES YES 30 Quarterly HMIS report produced and disseminated using computerized HMIS data NO YES YES YES NO 31 80% of villages with functioning VHCs i.e. VHCs meeting at least one every two months (6 times/year) 0% 40% NA 80% 91% Operations Research 32 Documented results of working with Traditional Healers and initiation leaders NO NO NO YES NO 33 Document results of Living University Approach NO YES NO YES NO 34 Document Results of Birth Plan Use OR Bicycle Ambulance NO NO YES YES YES CS-15 Malawi, The Partnership for Equity, Access and Quality (PEAQ), Final Evaluation Save the Children 59 Annex L. Selected HMIS Data Figure 1. Coverage of Selected Interventions by Zone, Balaka, 2002 0 10 20 30 40 50 60 70 80 90 100 Zone 6 Kalembo Phalula Mbera Phimbi Percentage Fully Immunized <1 Vit. A TTV Delivery in Health Centers Family Planning Average of Five Indicators Figure 2. Monthly STI Cases Treated in Health Facilities, Balaka, 2002 0 500 1000 1500 2000 2500 Jan￾02 Feb￾02 Mar￾02 Apr￾02 May￾02 Jun￾02 Jul￾02 Aug￾02 Sep￾02 Oct￾02 Nov￾02 Dec￾02 Jan￾03 Feb￾03 Mar￾03 Number of Cases Treated Hospital Health Centers