APPLYING LESSONS-LEARNED TO DEVELOPING SUSTAINABLE CHILD SURVIVAL AND MATERNAL CARE SERVICES WITH THE AGRICULTURAL ESTATES OF PRESS COMPANY IN KASUNGU, MALAWI CS-XIV Cooperative Agreement FAO-A-00-98-0017-00 FINAL EVALUATION REPORT Project Location: Kasungu District, Malawi Project Duration: September 29, 1998 to July 30, 2003 Submitted to: U.S. Agency for International Development USAID/GH/HIDN Washington, DC 20523-3700 Submitted by: Project HOPE – The People-to-People Health Foundation, Inc. Millwood, VA 22646 (540) 837-2100 Phone (540) 837-1813 Fax Prepared by: Bonnie L. Kittle External Evaluator Contact Person: Petra Reyes, ScD, MPH Regional Director, Africa preyes@projecthope.org APRIL 2003 i ACRONYMS ARI Acute Respiratory Infection BCC Behavior Change Communication CBDA Community Based Distribution Agent CDD Control of Diarrheal Diseases CHAPS Community Health Partnership CHC Compound Health Committee (made up of compound residents living on an estate as opposed to a traditional village) CS Child Survival E/32 Estate 32 E/81 Estate 81 EPI Expanded Program for Immunization Epi Info A word processing database and statistics program for public health epidemiological use DHO District Health Officer DHMT District Health Management Team FGD Focus Group Discussion FP Family Planning HIS Health Information System HIV/AIDS Human Immuno-deficiency Virus/Acquired Immuno-Deficiency Syndrome HSA Health Surveillance Agent IEC Information, Education and Communication IMCI Integrated Management of Childhood Illnesses KPC Knowledge, Practices, and Coverage (survey) MCH Maternal and Child Health MIS Management Information System MRM Mother Reminder Materials MOHP Ministry of Health and Population NGO Non-Governmental Organization ORDP/GRV Oral Reydration Distribution Point/Growth Monitoring Volunteer ORS Oral Rehydration Salt PAL Press Agriculture Limited PMTCT Prevention of Mother to Child Transmission RHO Regional Health Officer SP Sulfadoxine-Pryrimethamine STD Sexually Transmitted Disease TBA Traditional Birth Attendant T4T Training for Transformation UNICEF United Nations International Children Emergency Fund VCT Voluntary Counseling and Testing VHC Village Health Committee ii Table of Contents Acronyms i Table of Contents ii Executive Summary 1 Chapter One 7 A. Results: Summary Chart 7 B. Results: Technical Approach 8 1. Overview of the project 8 2. Progress by Intervention Area 9 3. New Tools or Approaches; Operations Research or Special Studies 23 Chapter Two 24 A. Planning 24 B. Staff Training 24 C. Supervision of Program Staff 25 D. Human Resources and Staff Management 26 E. Financial Management 26 F. Logistics 27 G. Management Information 27 H. Technical and Administrative Support 28 I. Management Lessons Learned 29 J. Other Issues Identified by the Team 29 K. Conclusions and Recommendations 30 1. Achievement of Objectives 30 2a. Achievements 30 2b. Constraints 30 3.Lessons Learned 30 Attachments 1. Team Members and their titles 2. Evaluation Methodology 3. List of People Interviewed 4. Training Table 1 Executive Summary Overview of the Project The Kasungu Child Survival Project was implemented in the Kasungu District of Malawi by Project HOPE in partnership with Press Agriculture Limited and the Ministry of Health. This four year project, which was initiated in 1998, sought to reduce maternal and child mortality and morbidity among 30,000 laborers and their families living on 34 tobacco and coffee estates and in 22 villages surrounding the estates. This goal would be achieved by implementing activities in the following intervention areas: control of diarrheal disease, immunization, breastfeeding, Vitamin A and iron folate, acute respiratory infection, malaria, family planning/maternal health and STDs/HIV/AIDS. The objectives of the project can be found in the results table shown on page 1 of the report. In the fourth year, 2002, the project was extended by 12 months, to September 2003. Then in July, 2002, Press Agriculture Limited was bought out by another Tobacco Company, Limbe Leaf, who took control of all but the 4 coffee estates. At this time, about 40% of the laborers were let go along with all but 4 of the HSAs and many of the community health volunteers. Limbe Leaf management did not commit to continuing the child survival activities as they had been designed and implemented. Rather, they committed to building additional clinics and providing curative care from these fixed facilities. During the last 6 months of the project little community-based preventive activities occurred due to these changes and a lack of transportation for the HSAs. Main Accomplishments The Kasungu Child Survival Project was very ambitious and sought to break new ground by partnering with a private sector enterprise, Press Agriculture Limited. Despite these challenges and many others, the project’s accomplishments are noteworthy. They include: ¾ Access to health education and various health care services has been increased in the project area through the training and support of scores of community health workers. These include 40 Traditional Birth Attendants (TBAs), 29 Community Based Distributor Agents (CBDAs), 54 Oral Rehydration Distribution Point Managers (ORDPs), 37 Breastfeeding Support Group members, 28 T4T volunteers, 20 community counselors, 65 drama group members, 340 Compound Health Committee (CHC) members on 34 estates and 290 Village Health Committee members in 29 villages. ¾ The quality of health on the targeted estates and in the district has improved by building capacity of various health care providers, providing drugs and medical supplies, building/up￾grading infrastructure (1 maternity, 2 multi-purpose shelters and 1 Voluntary Counseling and Testing (VCT) center). ¾ A strong and mutually supportive relationship between the project and the District Management Health Team was established, paving the way for continued support of health services on the estates. ¾ Although no empirical evidence is available to support this assertion, most estate managers report that worker morale is higher, laborers are more productive and fewer man-days are 2 lost for health-related reasons, because health services are closer and workers have adopted more healthful behaviors. Highlights of Baseline and Final KPC Survey Results Keeping in mind that the final KPC survey did not follow the standard 30 cluster random selection method and therefore the data is less reliable, it appears that the majority of project beneficiaries acquired key knowledge and adopted many healthful practices related to all but one objective. While the project did not attain all of the levels of achievement it set for itself, improvements across all of the 8 interventions were noted. Most noteworthy is the vaccination coverage, with 81% of all children under age 2 being completely vaccinated according to their vaccination cards. Use of ORS in the case of diarrhea surpassed the objective, increasing from 66.5% to 87%. Care-seeking behaviors in cases of suspected ARI rose from 63.9% to 84.1% and professional attendance at deliveries increased from 54% to 89%. Men’s understanding of STD symptoms increased from 20% to 64.5% and both men and women’s understanding of ways of transmitting HIV/AIDS increased significantly. Given these results, it is evident that even in the face of significant challenges and a very ambitious program, the project was a success. Priority Conclusions ™ When two organizations that have different mandates decide to go into partnership, they first need to agree on the benefits that each partner will gain from the joint effort. In other words, they need to identify the “value-added” to their partnership. In the case of the Kasungu Child Survival partnership the “value added” was that improved health and closer access to health services would lead to fewer man-days lost, which would contribute to higher production. ™ When working with a for-profit business, project activities (training events, meetings, and service provision) need to be planned/scheduled in consultation with the company managers, so that the demands of the work schedule can be taken into consideration (peak season vs. off season availability of workers). ™ It is not accurate to assume that a for-profit partner will have fewer financial/resource constraints than traditional partners such as the Ministry of Health. ™ In the project design stage, an in-depth study of the management structure of the partner needs to be conducted so that the project can be designed to fit seamlessly into that structure. ™ When two organizations go into partnership, the roles and responsibilities of all the stakeholders need to be clearly and specifically defined, assigned and communicated to all the stakeholders in a very transparent manner early on in the project. 3 Chapter One Assessment of Results and Impact of the Program A. Results: Summary Chart Objectives of the Project Baseline KPC 1999 Mid Term 2000 Modified1 KPC 2003 Diarrheal Disease Control % % % Increase from 66.5% to 75% the percent of mothers that administered ORS for a child w/ diarrhea 66.5 61.1 87 Increase from 11% to 70% the percent of mothers/caretakers who can name at least 3 practices that they can undertake to prevent diarrhea 11 36 53 Expanded Program of Immunizations (EPI) Increase from 66% to 90% the percent of children 12-23 months of age that are completely vaccinated 66 79.8 81 Breastfeeding Increase from 11% to 25% the percent of mothers that exclusively breast feed to 4 months of age 40.9 80 Vitamin A/Iron Folate Increase from 12.7 to 75% the percent of children older than 6 months of age that received Vitamin A supplementation in the 6 months prior to the KPC survey 12.7 45.3 46 Increase to 75% the percent of women receiving iron folate at every pre-natal visit 88.1 83.3 Acute Respiratory Infections (ARI - pneumonia) Increase from 63.9% to 85% the percent of mothers who sought treatment for their child’s cough, rapid or difficult breathing 63.9 54.3 84.1 Increase from 25.7% to 40% the percent of mothers that can name 3 danger signs of respiratory infection that would cause them to seek advice 56.4 41 Malaria Increase from 19.5 to 50% the percent of mothers that can name at least 3 ways to correctly manage a child with fever 19.5 26.1 37 Increase from 12.7 to 40% the percent of mothers that could 12.7 21.7 62 1 As a result of the change of management from PAL to Limbe Leaf in September 2002 a significant portion of the target population was thought to have moved off the estates and been replaced by new laborers. As a result, Project Hope in consultation with BHR/PVC decided not to conduct a final KPC since the results would not accurately reflect the results of the project. During the qualitative part of the final evaluation the team determined that it would be valid to survey the people who reside in the villages targeted by the project, since they had remained in the area during the life of the project. One hundred mothers were therefore interviewed and the results of this survey are presented here. While these data are not comparable to those of the baseline and mid-term KPC surveys, they provide a reasonable idea of the results of the project. See annex C for additional information. 4 name at least 3 signs of severe malaria Increase from 4.3% to 40% the percent of mothers that could name at least 3 appropriate ways to prevent malaria 4.3 12.5 42 FP/Maternal Care Increase from 32.9 to 75% the percent of women who retained their ante-natal card of their last pregnancy and received at least 2 antenatal visits. 72.62 18%3 94%4 Increase from 54 % to 90% the percent of women attended by a trained TBA, midwife or doctor during their last delivery (revised at final) 54 72 89 Increase from 53.2% to 60% the percent of women using a modern method of family planning (revised at final) 53.2 58.5 48 HIV/AIDS and STDs Increase from 28% to 50% the percent of women that can name at least 3 STD symptoms (revised at final) 28 17.5 32.6 Increase from 20% to 50% the percent of men that can name at least 3 STD symptoms (revised at final) 20 13.3 64.5 increase from 64.3% to 75% the percent of men who had experienced a STD symptom in the last 12 months that sought treatment at a health facility (revised at final) 64.3 68 Increase from 27.8% of men and 14.8% of women to 40% the percent or men and women that can name at least 45 correct ways of transmitting HIV/AIDS 28♂ 14.8♀ 73.3♂ 72♀ Increase from 10% of men and 18.5% of women to 40% the percent of men and women that can name at least 4 correct ways of avoiding HIV/AIDS 10♂ 10♀ 18♂ 19.4♀ 36.4♂ 21♀ B. Results: Technical Approach 1. Overview of the project The Kasungu Child Survival Project (KCSP) was implemented by Project HOPE in partnership with Press Agriculture Limited (PAL) and the Ministry of Health and Population (MOH). This four year project began in September 1998 but which was granted a one year no-cost extension through September 2003. The KCHP seeks to reduce maternal and child morbidity and mortality on 34 targeted tobacco and coffee growing estates run by PAL and 29 villages that border these estates. The target population was estimated to be 30,000 in 1998. The project goal was achieved by: 1) improving the quality of case management of priority conditions at the health facility level and increasing the number of outreach clinics providing preventive services; 2) training, supporting and supervising a cadre of 14 community health promotion agents (called Health Surveillance Assistants – HSA), financially supported by PAL, who provide community 2 Card retention only 3 card retention only 4 card retention and at least 2 ante-natal consultations 5 This objective was revised since the second two correct responses were only added to the list of key messages in the final year of the project. 5 education and organization; 3) strengthening community capacity by establishing Compound and Village Health Committees; 4) training and supporting a cadre of community health volunteers who raise awareness among the target groups, and promote the desired behaviors. The project implemented activities in each of eight technical areas. These included: Control of Diarrheal Disease (10%); Immunization (10%); Breastfeeding (10%); Vitamin A and Iron (10%); Acute Respiratory Infection (10%); Malaria (10%); Family planning and maternal care (20%) and HIV/AIDS/STD (including VCT and PMTCT) (20%). During the life of the project the financial viability of PAL deteriorated. Finally, in August 2002, just when the no-cost extension period was to begin, several divisions of Press Agriculture Limited, were taken over by Limbe Leaf (LL), another tobacco company, while the four coffee estates targeted by the project remained under the management of PAL. During the first months after the take-over, Limbe Leaf management let most of the management go, along with a large portion of the laborers. The estate population was reduced from about 23,000 to 12,000. The 14 HSAs were among those made redundant. The Medical Assistant and the nurse-midwife resigned. At the time of the final evaluation (March 2003), four of the HSAs and the Medical Assistant had been hired back and a second Medical Assistant had been hired. Limbe Leaf had reorganized the estates into 3 zones, each with 11 estates. Currently they have one clinic for each zone (2 of the 3 exist and function) which are staffed by a LL-employed Medical Assistant. Each of the four HSAs, also LL employees, now cover 11 estates rather than 2. Limbe Leaf employs a Medical Officer, Dr. Jaap Vrijberg, who oversees the clinic attached to the Limbe Leaf factory in Lilongwe, and who now oversees and advises management regarding the estates based health care system. For the immediate future, Limbe Leaf plans to continue supporting clinic-based activities only. 2. Progress by Intervention Area (see footnote 1 for an explanation of the modified KPC survey) Diarrheal Disease Control Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Diarrheal Disease Control % % % Increase from 66.5% to 75% the percent of mothers that administered ORS for a child w/ diarrhea 66.5 61.1 87 Increase from 11% to 70% the percent of mothers/caretakers who can name at least 3 practices that they can undertake to prevent diarrhea 11 36 53 Discussion The project successfully achieved the first DDC objective primarily by training and supporting 54 community health volunteers, called Oral Rehydration Distribution Point (ORDP) Managers. The ORDP Managers raised awareness among caregivers regarding the benefits, preparation and administration of ORS and served as a community-based ORS sachet distributor. As a result not only was knowledge and demand increased, but access to ORS packets was also improved. ORS packets are provided free of charge and are re-supplied by the Kasungu District Hospital. Clinic- 6 based health professionals, trained by the project in IMCI, also promoted the use of ORS packets. During health fairs organized by the project diarrheal disease control was also discussed and the HSA’s treatment of the estates’ water sources, also re-enforced this message. The evaluation team concluded that, while a great deal of progress was made, the second objective was not fully achieved because the standard of knowledge required 3 correct responses, whereas the women interviewed tended not to expand on their initial answer. Almost half (48%) of the 100 mothers interviewed sited drinking clean water as a means to avoid diarrhea and slightly less (43% and 42%) mentioned covering food and washing hand before eating respectively. Lessons Learned The evaluation team feels that making ORS packets available on each estate compound was the key to the success of this component. Obviously making the packets available free of charge also increased access. Special Outcomes, Successes or Constraints The key to the continued success of this activity is the re-supply of the ORDP Managers. Now that each HSA has to cover 11 estates, ensuring timely re-supply will be a challenge. Furthermore, with the retrenchment in September 2002, the number of ORDP Managers was reduced to 35. There are now 17 estates that do not have an ORDP Manager at all and there are only 18ORDP Managers among all of the 29 villages. Sustainability In order for this activity to continue, a trust-worthy means to re-supply the ORDP Managers with ORS packets needs to be identified. This was done during an end-of-project meeting attended by the Limbe Leaf Medical Officer, one of the two Medical Assistants, the 4 Area Managers and 24 Estate Managers. If the plan is put into action then re-supply should be continued on a regular basis. Expanded Program of Immunization Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Expanded Program of Immunizations (EPI) Increase from 66% to 80% the percent of children 12- 23 months of age that are completely vaccinated 66 79.8 81 Decrease from 21.2% to 20% the dropout rate for immunizations 21.2% Discussion This objective was achieved through the awareness raising efforts of the ORDP Manager; by his efforts to register all children under age five and promote attendance at the U5 in clinics static facilities as well as in the outreach clinics. This objective was measured by checking the child’s vaccination card, so the fact that mother’s tend to keep the card, also helped to measure achievement of this objective. In addition to the ORDP Managers, the achievement of this 7 objective was made possible by the work of the HSAs, the Medical Assistant and Nurse￾Midwife, who conducted U5 clinics on a regular basis both at the fixed facilities and in locations closer to the estates. The project also facilitated outreach activities by constructing two multi￾purpose buildings where U5 clinics are also conducted. Once the Area and Estate managers understood the purpose and importance of childhood immunizations, they too contributed to the achievement of this objective but releasing the mothers from work (without docking their pay) to attend U5 clinics. It was not possible to track the drop out rate due to the fact that regular immunization services were not being provided on the estates during the last 6 – 9 months of the project. Lessons Learned A key to improving immunization coverage is ensuring the presence and proper functioning of the cold chain. During the first half of the project one of the estate clinics did not have a refrigerator. It was only after the District Health Officer provided one that immunizations commenced in Eastern Area. It was also learned that not only does awareness need to be raised amongst the caregivers, but the managers of the estates also need to be made aware of the benefits of providing health care services on the estates to the business. This is measured in terms of fewer man-days lost. Special Outcomes, Successes or Constraints The lack of a refrigerator for 24 months of the project was a major constraint. Then during the last 6 months of the project (from September 2002 – March 2003) the lack of staff, Medical Assistant, Nurse-Midwife and HSAs, prevented services from being provided. Sustainability Limbe Leaf is committed to continuing curative clinic-based services. Therefore their commitment to providing clinic-based preventative services for children is not clear. For EPI activities to continue the cold chain must be present and properly maintained and a system to ensure re-supply of vaccines from the Kasungu District Hospital must be devised and implemented. At the end of project meeting, such a system was identified. It just remains to be implemented. The newly hired Medical Assistant needs training in refrigerator maintenance (training was provided by the project to the other Medical Assistant and Nurse-Midwife). Breastfeeding Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Breastfeeding Increase from 11% to 25% the percent of mothers that exclusively breast feed to 4 months of age 40.9 76 Discussion The modified KPC survey conducted among mothers of children ages 12 – 36 months living in four of the project’s 29 target villages asked the mothers 7 questions pertaining to breastfeeding. The sample included 100 mothers. Fifty-six percent (56%) of the children were under 24 months 8 of age and 44% were between 24 and 36 months. Seventy-two percent (72%) of the mothers were currently breastfeeding. Eighty-one percent (81%) had heard of exclusive breastfeeding and of those, 70% were able to define the term correctly. Fifty-two of the 100 mothers could name 3 advantages of exclusive breastfeeding. Seventy-six of the mothers (76%) interviewed who had ever breastfed said they had exclusively breastfed their infant, with almost half (42%) saying they continued the practice for 6 months. The self-reporting nature of this survey question calls into question the survey result, especially since the project only promoted the practice through health education and exclusive breastfeeding has a reputation of being a difficult practice to promote. On the other hand, it is note-worthy that fully 70% of mothers could accurately define the practice and 52% could name 3 advantages. That in and of itself is a significant achievement. Lessons Learned The evaluation team learned that self-reporting as a means of measuring behavior change is not nearly as reliable as tangible evidence. Team members also recognized that in the absence of specific carefully monitored activities to promote exclusive breastfeeding, such as breastfeeding support groups made up of mothers practicing the behavior, it is highly unlikely that such a high percentage of mothers actually breastfed their child exclusively. And finally the staff learned that if awareness raising is the only activity related to a component, then the most appropriate objective is one that measures increased knowledge, not practice. The team’s discussions of this component also brought forth the idea that behaviors that only require the decision and effort of the nuclear family, may be easier to promote among estate residents than in traditional village settings. This is because estate residents are not surrounded by the usual numbers and types of people who traditionally influence decisions regarding infant feeding (mothers, grandmothers, sisters etc). In the absence of these people each family is more at liberty to consider and perhaps take the advice of the community volunteers and health care providers. The team also noted that there seem not to be any very strong or religiously linked traditions regarding administration of other liquids or foods to infants, which would mitigate against the practice. Special Outcomes, Successes or Constraints If even half of the mothers interviewed during the final evaluation were honest in reporting their exclusive breastfeeding practices, this would be a major success of the project. The level of knowledge regarding the advantages of the practice is also noteworthy. The biggest weakness of this component, despite it’s apparent success, is that the project never developed a convincing strategy (other than health education) to more assertively promote this behavior. Such a strategy, accompanied by activities whose outcomes could be accurately monitored and measured would lend credibility to the final results. Sustainability If traditional birth attendants, family planning promoters, HSAs and health care providers on the estates continue to encourage mothers to exclusively breastfeed their infants then adoption of this behavior may continue. With the take over of Limbe Leaf, however, the numbers of TBAs has 9 been reduced from 40 to 32. Fourteen estates have no trained TBA at all and 11 estates are without a family planning promoter. Four HSAs cover all 34 estates instead of 14. All of this means that there are far fewer “change agents” in the community promoting and supporting exclusive breastfeeding. Vitamin A and Iron Folate Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Vitamin A/Iron Folate Increase from 12.7 to 75% the percent of children older than 6 months of age that received Vitamin A supplementation in the 6 months prior to the KPC survey 12.7 45.3 46 Increase to 75% the percent of women receiving iron folate at every pre-natal visit 88.1 83.3 Discussion Although the project increased consumption of Vitamin A significantly, it did not reach it’s stated objective. This may be attributed to stock outs of Vitamin A from time to time throughout the life of the project. Related to stock outs, it should be noted that 77% of the surveyed children had received at least one dose of Vitamin A, even if it wasn’t in the last 6 months. Furthermore the fact that many mothers stop coming to U5 clinics when their child has been completely vaccinated also reduces the opportunities of providing Vitamin A to all children under age 5. The data related to iron folate consumption is a bit misleading. The mid-term KPC results rely on self-reporting by the mother whereas the modified KPC surveyors consulted the antenatal card to report provision of iron folate. The fact that only 18% of mothers could produce their antenatal card weakens this data significantly, however. Now that the MOH has developed a new antenatal passport that contains more information and is valid for more than one pregnancy, it is more likely that mothers will keep their card (passport). The project helped train a total of 40 TBAs to provide antenatal consultations, conduct safe deliveries and refer at-risk cases to the district hospital. A TBA kit was provided to all trained TBAs and iron folate is one of the products a trained TBA is supposed to have. When interviewed, however, only one of the 8 TBAs had a supply of iron tablets. This points to a re￾supply problem, which is related to the lack of supervision of the TBAs by the MOH. In the past, joint, Project HOPE/MOH, supervisory visits were conducted and re-supply was taken care of at this time. Since December 2002, when the project person in charge of family planning and maternal care resigned, there have been no MOH supervisory visits. Supervision and re￾supply remain major challenges. Lessons Learned Activities that rely on the provision of products, such as Vitamin A, from the MOH (and in turn from UNICEF or other donors) are subject to stock outs, which are beyond the control of the project. 10 Activities that require regular support, such as the constant re-supply of materials and products, (iron folate) are a particular challenge to project implementers. Practical, feasible systems for re￾supply need to be developed and tested during the life of the project. Results that seem impressive at first glance, may be less so if the denominator is particularly low. Special Outcomes, Successes or Constraints Stock out of Vitamin A and the challenge of re-supplying TBAs were particular constraints for the project. Sustainability The Limbe Leaf Medical Officer is considering conducting a one-day workshop for HSAs to help them organize their work among the 11 estates. Furthermore, the estate managers have agreed to transport the HSAs around to the different estates, while waiting for the motorcycles provided by Limbe Leaf, to arrive. If this can happen, then HSAs and Medical Assistants will be able to continue conducting U5 clinics where Vitamin A is distributed. ORDP Managers should promote continued attendance even among children who are completely vaccinated. The LL Medical Officer, the District Health Officers and the Medical Assistant from Estate 81 proposed a plan to ensure monthly re-supply of all of the community health volunteers, including the TBAs. If this plan is implemented, TBAs should have iron tablets to distribute to pregnant women. Acute Respiratory Infections Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Acute Respiratory Infections (ARI - pneumonia) Increase from 63.9% to 85% the percent of mothers who sought treatment for their child’s cough, rapid or difficult breathing 63.9 54.3 84.1 Increase from 25.7% to 40% the percent of mothers that can name 3 danger signs of respiratory infection that would cause them to seek advice 56.4 41 Discussion The project was able to achieve these objectives primarily through intensive health education provided by the 14 HSAs and the 2 facility-based health care providers. It is also likely that knowledge regarding the signs and symptoms of ARI were fresh in the minds of the respondents at the time of the evaluation since the project had recently developed and distributed Mother Reminder Materials (MRM). The MRM depict eight symptoms for which professional health care should be sought, and among these were the ARI signs. Further to this the project trained both of the clinic health care providers in IMCI and both Project HOPE and PAL provided the necessary drugs. This improved their abilities to diagnose and treat ARIs, which would have encouraged mothers to come for treatment in a timely manner. Lessons Learned 11 Through discussions of this component, the project staff recognized that there can be several barriers to one behavior change and a successful behavior change strategy needs to address all of the major barriers. Thus, simply increasing the number of health care facilities and increasing proximity to the users may not result in increased use, if, for example, the fee for drugs is too high. Four consultations observed during the final evaluation showed that all patients paid between 64% - 83% of their daily wage to treat one child. (MK 35 and 45 for drugs; daily wage is MK 54 or 60 cents per day) Special Outcomes, Successes or Constraints As part of the Memorandum of Understanding between Project HOPE and PAL, PAL agreed to ensure emergency transport from the estates to the Kasungu District Hospital. To this end PAL designated a vehicle to be used as an ambulance. Unfortunately the cost of renting the vehicle was too high and it was rarely used for its designated purpose. Further to this one of the Area Managers used the vehicle as his personal means of transport so it’s availability was questionable as well. Sustainability Once again the continuation of this activity will depend on the continued awareness raising efforts of the HSAs and the effective treatment by clinic staff. While one IMCI-trained Medical Assistant returned to his post and is running the clinic on Estate 81, the Medical Assistant on Estate 17 has not been trained in IMCI and may therefore be less qualified to accurately diagnose and treat ARIs. The availability of antibiotics, which will be provided by Limbe Leaf is another issue that needs to be addressed if we expect mothers to continue their care-seeking behavior. Malaria Objectives of the Project Baseline KPC 1999 Mid￾Term 2000 Modified KPC 2003 Malaria Increase from 19.5 to 50% the percent of mothers that can name at least 3 ways to correctly manage a child with fever 19.5 26.1 37 Increase from 12.7 to 40% the percent of mothers that could name at least 3 signs of severe malaria 12.7 21.7 62 Increase from 4.3% to 40% the percent of mothers that could name at least 3 appropriate ways to prevent malaria 4.3 12.5 42 Discussion In addition to promoting diarrheal disease control, monitoring children’s growth, and teaching the benefits of childhood immunizations, the ORDP Manager was also in charge of educating the estate residents about the causes, management and prevention of malaria. Significant progress was made on all three objectives and only the first objective was not fully attained. Once again, this could be attributed to the required level of knowledge (3 correct responses). TBAs also play a role, by encouraging pregnant women to attend antenatal consultations where malaria prophylaxis is made available. 12 Lessons Learned When interviewed for the final evaluation, none of the ORDP Managers mentioned any aspect of malaria management or prevention as one of their tasks. Neither did they sight any malaria messages as part of their health education program. From this the team concluded that the ORDP Managers see other aspects of their job as taking precedence over malaria. It was also sighted that it is common for a worker to take his/her cues from the title of their position and assume that the title describes their primary task. Hence, the ORDP Managers focused on promoting diarrheal disease management and much less on malaria. Although the project did not plan on distributing impregnated mosquito nets as part of the malaria control strategy, when nets became available at the district hospital, the project took the opportunity to promote their use among estate residents. They made 300 available to the Compound Health Committee who were supposed to sell them and use the funds to replenish the supply. Two important lessons were learned from this. One is that it pays for a project to be flexible enough to take advantage of unexpected opportunities and resources such as these when they arise. However, lesson number two it is also necessary to carefully plan the new strategy before implementing it. Unfortunately without guidance and supervision from the project, many HSAs gave out bednets on credit and never recuperated the money so replenishing the supply was not possible. As one of the malaria prevention measures, the project promoted the practice of clearing the compound area of tall grass and stagnant water. This practice met with resistance, which the staff examined. The staff concluded that since the laborers are only temporary residents on the estates, they do not feel any ownership of the land and are therefore disinclined to expend their energy to make improvements. All three of the objectives are knowledge based. It would have been more meaningful if one of the 3 had related to a practice. Special Outcomes, Successes or Constraints While the project did not seek to reduce the incidence of malaria, a significant obstacle to the prevention of malaria is the existence on almost all estates of large bodies of water (usually reservoirs) where mosquitoes breed. To deal effectively with this scenario would require a larger more comprehensive mosquito control program, such as spraying, that would have to be implemented by Limbe Leaf. Sustainability The aspect of this component that is likely to continue is the treatment of a child with mild fever at home by the caregiver. This is because aspirin is available in shops on most estates and the other practices are free. Care seeking in the presence of severe fever may also continue since the estate clinics are nearby and have the necessary drugs to treat malaria. Aside from a certain demand for mosquito nets created when the nets were made available, it is doubtful that estate residents will take many of the other malaria prevention measures. Project staff report, however, that some CHC have taken it upon themselves to purchase quantities of mosquito nets from the district hospital and continue to sell them on the estates and in the villages. Unfortunately there is no monitoring system related to this activity, so no data is available. 13 Family Planning and Maternal Care Objectives of the Project Baseline KPC 1999 Mid Term 2000 Modified KPC 2003 Family Planning/Maternal Care Increase from (32.9%) 72.6 % to 75% the percent of women who retained their antenatal card of their last pregnancy and received at least 2 antenatal visits. (corrected at final) 72.66 18%7 94.4%8 Increase from (35.2%) 54 % to (50%) 90% the percent of women attended by a trained TBA, midwife or doctor during their last delivery (corrected at final) 54 72 89 Increase from (44%) 53.2% to 60% the percent of women using a modern method of family planning (corrected at final) 53.2 58.5 48 Discussion The objectives of this component deserve some examination to better understand the final results. As is usually the case, the proposal along with the objectives was written one year before the baseline KPC was conducted. The results of the baseline KPC conducted in early 1999 revealed that some of the objective levels (percentages) were not the same as was presented in the proposal. Rather than present the new data in the DIP by changing the levels of achievement of the objectives, however, both the baseline data and the end-of-project levels of achievement remained unchanged. As a result, the midterm evaluation team observed and remarked on the discrepancy between the results of the 1999 KPC results and the percentages that were presented as the objectives in the DIP. Thus, for example, while the proposal says that 35.2% of deliveries were attended by a trained caregiver, the Baseline KPC found that 53.2% of women received such assistance. Because the initial finding (35.2%) was so low, the end-of￾project objective was set at 50%. Consequently, the objective appears to have been achieved even before activities got underway. To address this issue in the table above the percentages presented in the DIP, but that are different from the findings of the baseline survey, are presented in parentheses. The baseline survey data and some revised end-of -project levels of achievement are presented normally. As the footnote indicates, the baseline KPC survey did not collect and correlate the information regarding antenatal card retention and antenatal consultation attention. This occurred, more than likely, because the objectives were changed between the submission of the proposal and writing of the DIP and then again upon recommendation of the DIP reviewers. The final revised 6 Card retention only 7 card retention only 8 card retention and at least 2 ante-natal consultations 14 objectives of the project were presented in an annex of the first Annual Report where they were easily and consistently over-looked by various people associated with the project. Of the three objectives it is clear that with the training and support of 40 Traditional Birth Attendants and the hiring of a nurse-midwife to work in one of the estate clinics, more women had access to the services of a trained caregiver during deliveries. The problem related to the first objective has more to do with card retention than having attended 2 antenatal consultations. Because the older antenatal card is only useful during the life of the pregnancy, most women discard it after the birth and some health facilities keep the card. Hence while 90% of women said that had received an antenatal card, out of 100 mothers interviewed only 18 still had the card. Amongst those, 17, or 94.4%, had had at least 2 antenatal consultations. The family planning objective does not fairly reflect the success of this activity since re-supply of contraceptives was disrupted when Limbe Leaf fired 10 of the 14 HSAs, whose job it is to replenish the supply. Hence during the six months prior to the evaluation, the 26 Community￾based Distribution Agents (CBDA) trained by the project were not able to re-supply their clients. Furthermore, with the retrenchment and hiring of new laborers, many FP clients left and were replaced by people who were not acceptors. Lessons Learned By following the sequence of the changes in the objectives, the project staff began to understand the importance of the formulation of the objectives and of up-dating the objectives and levels of expected achievement based on the results of the baseline data survey. The up-dated information is then incorporated into the DIP, which then becomes the “base document” for the project. Not having done this caused much confusion among staff and backstop people right up to the last quarterly report. Staff also learned such lessons as associating only one practice with each objective. Special Outcomes, Successes or Constraints Two constraints hampered achievement of the first and last FP/MC objectives. These are the combination of two practices (card retention and antenatal consultation attendance) into one objective. Since card retention is low, it is difficult to get a true sense of attendance at antenatal clinics. With regard to family planning services, the static shown reflects the disruption in the supply system rather than the level of family planning use prior to the change in company management. Sustainability If the Limbe Leaf Medical Officer and estate managers can help the HSAs re-organize their work so they can visit each of their estates at least once per month, and if the Medical Assistants can re-establish the supply system, then this service is likely to continue. This is also true because demand for contraceptives is quite high. 15 STD/HIV/AIDS (VCT & PMTCT) Objectives of the Project Baseline KPC 1999 Mid Term 2000 Modified KPC 2003 HIV/AIDS and STDs Increase from (21.3%) 28% to 50% the percent of women that can name at least 3 STD symptoms (corrected at final) 28 17.5 32.6 Increase from (23.4%) 20% to 50% the percent of men that can name at least 3 STD symptoms (corrected at final) 20 13.3 64.5 Increase from (32%) 64.3% to 75% the percent of men who had experienced a STD symptom in the last 12 months that sought treatment at a health facility (corrected at final) 64.3 68 Increase from 41.4% to 75% the percent of men who had an STD symptom in the last 12 months that informed his partner Increase from 27.8% of men and 14.8% of women to 40% the percent or men and women that can name at least 49 correct ways of transmitting HIV/AIDS 28♂ 14.8♀ 73.3♂ 72♀ Increase from (19.6%) 10% of men and (14.5%) 10% of women to 40% the percent of men and women that can name at least 4 correct ways of avoiding HIV/AIDS 10♂ 10♀ 18♂ 19.4♀ 36.4♂ 21♀ Discussion For an explanation regarding the percentages in parentheses, please refer to the discussion presented in the FP/Maternal Care section. The evaluation team discussed the interpretation of these results extensively. It was particularly interesting to analyze why so few women, as compared to men, could name 3 STD symptoms. The team came up with several hypotheses. First, it could be that women are more uncomfortable talking about diseases that affect the genitals. They may also be shyer when responding to the surveyor and therefore less likely to provide comprehensive responses. It is also possible that since men are perceived has having more risky behaviors (multiple partners) that the CBD Agents and HSAs concentrated this message on men and less on women. It was also speculated that while men easily talk about STDs amongst themselves, women are less likely to share such information during interpersonal contacts. The third and fourth objectives were not measured during the final KPC due to an oversight. The final evaluation team revised the fifth objective, having to do with ways of transmitting HIV/AIDS. It was decided that the end-of-project level of achievement should be two correct 9 This objective was revised to require only two correct answers since the third and fourth correct responses were only added to the list of key messages in the final year of the project. 16 responses rather than four, since two of the four correct answers (having to do with mother to child transmission) had only been introduced during the last 8 months of the project. The level of achievement of this objective is attributed to the fact that not only did the HSAs, CBDAs and T4T people help to raise awareness, but a huge multi-channel national campaign to combat the AIDS pandemic was launched and supported the objectives of the project. Regarding the final objective, ways to avoid contracting HIV/AIDS, it seems as though once again the men out performed the women. It should be noted, however, that 39% of the women interviewed were able to sight 3 means of avoiding contracting the disease, a significant enough achievement. Lessons Learned When reviewing these objectives and the results, the project staff recognized the importance of being realistic about the level of achievement anticipated (number of correct responses per objective). Obviously the higher the level the more difficult it is to achieve. It was also recognized that each objective needs to be accompanied by a corresponding key message. So if the objective requires 4 correct responses, then the health education sessions also need to focus on those four key messages. In other words, the BCC strategy needs to directly support achievement of the objectives. In the case of the Kasungu Child Survival and Mother Care Project, the BCC strategy was not as supportive as it could have been. Special Outcomes, Successes or Constraints Although it was not an objective, the final evaluation team measured the use of condoms. Thirty percent (30%) of the women interviewed said they had used a condom during their last sexual contact whereas 50% of the men reported having used a condom (and a few produced condoms from their pockets!). The VCT/PMTCT add-on component was a particularly note-worthy success of the project. This sub-component of the STD/HIV/AIDS component was initiated in February 2002 when the renovations on a district hospital out building were started to transform the structure into a VCT center. Two project staff members were hired to execute this component, one of whom was an IEC specialist with prior experience with HIV/AIDS, and the other a trainer with a technical background in nursing and reproductive health. During a particularly short period (which included the months of uncertainty regarding the status of PAL) the following outcomes were realized: VCT/PMTCT Program Achievements • PAL medical assistant and nurse midwife were trained to provide VCT services • 20 volunteers were trained as HIV counselors • 10 district hospital health care providers were trained in counseling techniques • VCT services established on Estate 81 clinic 17 • A building was renovated on the hospital premises to serve as the VCT center • VCT services offered at the District Hospital 8 hours per day five days per week • 5 district hospital maternity service providers were trained in MTCT • PAL nurse midwife trained in MTCT • 5 district hospital nurses trained in STI diagnosis and management • establish MTCT services at Kasungu District Hospital • Pilot breastfeeding support groups on the estates • Conducted 4 sensitization seminars with senior management at PAL involving all managers on CS estates • Established a working group for development of health policies • 160 drama group members (6 adult groups and 10 adolescent groups) trained in interactive/participatory performance methods and issues pertaining to VCT/ PMTCT/ STD/HIV/AIDS • Drama group performances conducted for large numbers of estate residents and managers to raise awareness and promote healthful practices regarding VCT/ PMTCT/ STD/HIV/AIDS • Awareness campaigns conducted on VCT and PMTCT • 2 MOH staff trained as trainers in PMTCT and psychosocial counseling • 5 day orientations on PMTCT and VCT conducted for the following: ƒ 22 District AIDS committee (DAC) members ƒ 32 TBAs ƒ 65 hospital support staff ƒ 26 CBDAs ƒ 19 nurses and clinicians from health centers and district hospital. Although the project had only been underway for a year, the final KPC survey revealed that 54% of the women interviewed could name two ways that a pregnant woman could pass HIV/AIDS on to her child. Twenty-four percent (24%) of the women knew that HIV positive pregnant women should give birth at the district hospital where preventive medicine would be administered. Eighty-two percent (82%) knew where to get HIV/AIDS testing and 12% of the women had already been tested. According to the VCT trained Medical Assistant, during the first week that VCT services were available at the Estate 81 clinic, 18 people had come for testing, most of whom were women. Sustainability Demand for information and condoms is high among the estate’s residents and this demand will provide the momentum needed for STD/HIV/AIDS prevention activities to continue. Both of the estate clinics have a supply of condoms and condoms are available in many stores on the estates. Keeping up with the demand is the challenge as supplies often run out. Much of the stigma attached to condom use has been reduced and thanks to the dramas many more people are aware of the disease and what measures to take to avoid contracting STD/HIV/AIDS. Interested and concerned people will continue to seek out advice. 18 The VCT/PMTCT activities are fully integrated into the district hospital plan of action and with support from UNICEF the VCT center will continue to function and PMTCT services will be provided. 3. New Tools or Approaches; Operations Research or Special Studies Special Studies During the first half of the project five special studies were carried out. These include: 1) a qualitative study among estate residents regarding cultural, practical and knowledge barriers to successful exclusive breastfeeding; 2) a qualitative nutrition study to help develop a nutrition component to address malnutrition amongst children; 3) a Health Services Assessment conducted by the Quality Assurance Project in March 2000 to assess the quality of service delivery by all levels of health care providers and volunteers involved directly or indirectly with the project; 4) a Partner Capacity Assessment carried out in December 1999 by the Development Center to determine the ability of PAL to implement and support health care services on the estates; and 5) the Iron Folate study conducted by the project in April 2001. The results of the breastfeeding study were not useful to the project because the study implementers (project staff) did not know enough about analysis of focus group discussions to get viable information. On the other hand, the experience proved to be a positive learning opportunity for the project staff. The Nutrition Study resulted in the decision not to initiate nutrition activities to address malnutrition. The project was deemed complex enough without adding ninth technical intervention. The Health Services Assessment results came out just prior to the mid-term evaluation and after many of the project health care providers and volunteers had already been trained. None of the staff members present at the end of the project were present at the beginning or even mid-term, so it is unlikely that training conducted during the second half of the project took the results of the assessment into consideration. It should also be noted that the MOH requires that all “officially” recognized health care providers be trained using the MOH-approve curriculum. As a result, any weaknesses identified in health care provision would need to be address through on the job training. The Partner Capacity Assessment was useful in identifying what PAL needed to do to strengthen it’s ability to support the health delivery system established under the project. Unfortunately due to financial constraints (which ultimately led to the demise of the company) and the need to focus on production issues prevented PAL from addressing all of the issues brought out in the report. PAL was able however, to identify a counterpart for the project which facilitated communication and some action taking, and they took steps to improve communication between PAL senior management and the estate level managers regarding the project. This effort improved relations between the HSAs and the Area and Estate managers that helped the project gain acceptance and run more smoothly. 19 The midterm evaluation recommended that an iron folate study be carried out to determine what percentage of pregnant women received iron during antenatal consultations. This information was necessary to establish a baseline upon which to evaluate this activity. The study was designed and implemented by the project HIS person and was carried out in April 2001. New Tools In the third and fourth years of the project, funds were secured to participate in a multi-country effort to design and produce Mother Reminder Materials (MRM). While project staff participated in testing the materials, the activity was spearheaded by CHANGE, a USAID￾funded BCC group. The end product was a poster that has a calendar at the bottom, a mirror in the middle and eight small drawings depicting the signs and symptoms that require medical attention. The purpose of the MRM is to remind mothers to take their children for treatment when any of the signs are present. Several hundred MRMs were distributed to mothers on the estates and project staff explained the purpose and use of the poster. As part of the final evaluation the team assessed mother’s understanding of the MRM, particularly the pictures. The survey revealed that a few of the pictures (dehydrated baby, baby with fever and baby with cough) are not well understood. While fully 83% of the mothers had seen the MRM, only 15% of the respondents recognized that it is more than just a calendar. When distributing the MRM mothers were instructed to hang it in the living area, but better visibility. When the evaluation team visited, 76% of respondents had the MRM hanging in the living room and 81% were in good condition. On the other hand only 9% of mothers had marked the MRM, and indication of it’s use to recall well baby clinic dates or other important events. 20 Chapter Two Program Management A. Planning At the time of the final evaluation no one who had participated in the development of the DIP was still associated with the project. Therefore little is known about the process. The present Country Director was hired just after the DIP was developed and her recollection is that the DIP format newly developed that year, was far too detailed and complicated. Rather than add items to the DIP, she suggested that it be simplified. B. Staff Training The project has very limited unrestricted funds and therefore does not have the resources to invest in staff development. During the first half of the project, only five formal staff development opportunities were provided for a total of 18 days. During the second half of the project however, additional opportunities for staff training became available through the add-on component VCT/PMTC. The following table shows the types of training provided to staff during the second half of the project. Staff from other Project HOPE/Malawi projects also attended most of the training events sighted below. STAFF TRAINING 2001 – 2003 TOPIC TRAINEES TRAINER Duration Integrated BFHI Infant feeding and HIV/AIDS Training 1 Project HOPE staff (CSMC) 4 MoHP Staff, 6 Project HOPE staff (CSMC) 2 weeks Training for Transformation 2 Project HOPE staff (CSMC) Staff Development Institute Lecturers 2 weeks HIV/AIDS Facilitating the Change Training 3 Project HOPE Staff (CSMC) Vusisizwe 1 Week Stepping Stones Training 2 Project HOPE Staff (CSMC) Action Aid Malawi Specialists 1 Week Integrated BFHI Infant feeding and HIV/AIDS Health Care and Community Services Training 1 Project HOPE Staff (CSMC) 3 Project HOPE Staff, 1 MoHP Staff, 1 Mzuzu University Trainer 12 days Integrated Infant Feeding and HIV/AIDS Psychosocial Counseling to prevent Mother to Child Transmission of HIV/AIDS 2 Project HOPE Staff and MoHP Nurses/ Clinicians Project HOPE and Linkages Project 6 Weeks Training of Trainers Integrated Infant Feeding and HIV/AIDS Psychosocial Counseling to prevent Mother to Child Transmission of HIV/AIDS 2 Project HOPE Staff and 2 MoHP Nurses/ Clinicians Project HOPE and Linkages Project 12 days 21 TOPIC TRAINEES TRAINER Duration Learn to ‘BEHAVE’ behavior change programming 1 Project HOPE Staff (CSMC) CORE Group for CSMC South Africa 5 days Strategic Sanitation and Hygiene Promotion 1 Project HOPE staff Consultant for Concern Universal 6 days Behaviour change Interventions Training 18 Project HOPE staff 6 CSMC 4 VHB 4 HIV In the workplace 4 Ministry of Health Staff 2 Project HOPE staff 5 days The HIS specialist who was present during the midterm evaluation was trained in Epi Info which enabled him to enter and analyze the data from the KPC conducted mid-way through the project. The project’s technical backstop officer at the time did not have confidence in the results, however, so the data was sent to headquarters for further analysis. Training in IMCI was provided to one of the project’s trainers during the first half of the project, which enabled her to help train many of the District Hospital and estate clinic staff in this approach. Unfortunately, her newly acquired skills made her very desirable to other organizations and she left the project to work elsewhere. Almost all of the staff training sighted above is related to the VCT/PMTCT add-on component, which began in April 2002. This training made it possible for project staff to train others in the techniques and procedures and increased their knowledge and skills enormously. Resources for staff training were clearly inadequate during the first half of the project and only became available toward the end of the project through the add-on component. The Country Director and Project Manager both suggested that the lack of staff development opportunities may have contributed to the high staff turnover. On the other hand, there is at least one incidence where staff training contributed to the departure of a staff member. Given this, one must conclude that there is no guarantee staff will stay with a project even if staff development opportunities are made available. It is clear however, that investments in staff development contribute to the employability of the trained person since trained people are scarce and are in high demand. Furthermore, most people put their training to good use, thereby contributing to the overall development of the country. Staff training is very appreciated by the staff involved and helps to create and maintain high staff morale and dedication to the project. C. Supervision of Program Staff The Country Director supervises the Project Manager and the Project Manager supervises the project staff. Project HOPE has a system whereby an annual performance appraisal is conducted of senior staff. This procedure allows the supervisee to provide feedback on the comments made by the supervisor. Other than this, Project HOPE does not have any other means by which staff 22 performance is assessed against a standard on a more regular basis and put in writing. A more formal and regular supervision system would be beneficial to management and staff alike. D. Human Resources and Staff Management Limbe Leaf and the District Health Office are responsible for continuing operations initiated by the project. Limbe Leaf has committed itself to providing limited curative facility-based care as a part of their social responsibility agenda. They have already hired a Medical Assistant for the second clinic and a third clinic is planned. The Limbe Leaf Medical Officer will backstop these clinics and their staff through periodic visits and by ordering medicines and supplies. During the final evaluation the Medical Officer visited one of the clinics and was advising senior management about infrastructure up-grading work that needs to be done. The District Health Officer has also pledged to support the estate-based health facilities by supervising staff and providing such things as contraceptives, condoms, ORS packets and reagents for HIV/AIDS testing. During the project the morale of the staff varied a lot. At mid term it appeared that the staff all got along well. During the third year of the project however, interpersonal issues arose which in part led to the departure of several staff members. The project suffered unusually high staff turnover. For example, there were 4 HIS specialists, 3 project managers, 2 sets of trainers and 3 secretaries. Of course, when staff depart, they take with them all of the training and preparation they’ve had to do their job, as well as their institutional memory and rapport with project partners and beneficiaries. As could be expected the departure of every staff member caused a delay in project implementation and sometimes a reduction in quality as well. By the end of the project none of the staff had been with the project more than 15 months and several had just joined in the last half-year. No plans had been made to facilitate staff transition to other paying jobs, but staff were being given their compensatory days to attend interviews. The atmosphere was conducive enough such that staff members were free to share their plans including job seeking, and management was supportive and flexible to allow staff to take their compensatory days to attend interviews. It was suggested that the Country Director contact all of the NGOs implementing projects with health components and let them know that qualified staff are available. E. Financial Management The project employed an accountant who sent monthly financial reports to the Country Finance and Administration Manager (FAM) in Blantyre. The FAM then reviewed the reports from all of the country projects and submitted these to the Finance Department at headquarters. Expenses were reimbursed and if additional funds were needed they were sent. While the final evaluation team did not examine the financial management of the project in-depth, anecdotal evidence suggest that financial management was sometimes a problem. For example, funds were sometimes not available or adequate for planned activities, which then had to be postponed or aborted at the last minute. Furthermore, when the no-cost extension was requested, it was thought that there was enough funds to last a full extra year. In the final analysis, it turned out that there was only enough money to prolong the project by 6 months. Such occurrences suggest 23 weaknesses in financial management. Some of these difficulties can surely be attributed to a major restructuring of HOPE Center’s financial reporting procedures, which took place in the second year of the project. Others may be simply a lack of knowledge and experience. It is not clear whether Limbe Leaf has the skills and experience to accurately estimate costs and elaborate budgets for future health care programming. They had difficulty at first recruiting a Medical Assistant for the second clinic because their salary scale was not competitive and there has been a 6 month delay in getting motorcycles for the 4 HSAs who have been sitting idle this past half year for lack of transport. While the management, including the Medical Officer, has some experience running an urban clinic associated with the tobacco processing factory in Lilongwe, it’s not clear to what extent that experience will transfer to an estate-based health delivery system. It is certain however, that senior management needs to keep their eye on the bottom line and progress slowly in the development of the health care system, since the former funds the later. The project did not receive any technical assistance to assist its partners to develop financial plans for sustainability. F. Logistics Project HOPE’s match for the child survival project consists of Gifts in Kind (GIK), which are essentially donations usually of medical supplies and some drugs. Unfortunately the organization has little control over what gifts are donated. Consequently the gifts are not always appropriate or useful to the project. During the first half of the project communication between the field office in Kasungu and the Country office in Blantyre and HQ was limited to telephone. In the third year of the project an email system was installed which greatly facilitated communication thereby contributing to more efficient problem resolution. The vehicle, which was bought in 1999 with project funds, was in seriously bad shape by August 2002. A newer vehicle from a project that had closed subsequently replaced it. National stock outs of such things at Vitamin A and ORS packets hampered progress of these interventions. During the debriefing workshop conducted at the end of the final evaluation, the Limbe Leaf Area and Estate Managers, the Medical Officer and Operations Managers were challenged to address the some of the most pressing logistical issues. This includes ensuring the re-supply of contraceptives, ORS packets and TBA supplies to the community volunteers. Together the Medical Assistant, Medical Officer, and District Health Officer agreed on a feasible system to ensure the re-supply. While the medical team was working on that problem, the Area and Estate managers were working to resolve the HSA’s transport problem. While waiting for the motorcycles to be delivered, the estate managers from each of the 3 zones agreed to provide transport to the HSA from their zone so he can visit each estate at least once in a month. While these are only two problems, the notion of being pro-active in resolving health delivery system problems was planted and will hopefully continue. 24 G. Information Management The system to measure progress toward program objectives was very weak throughout the life of the project. This is due to the lack of experienced MIS people in country, and MIS technical assistance. As is frequently the case, it seems that no project staff clearly understood that monitoring is a task that should help improve project implementation (health worker performance), not just to supply statistics to headquarters. The baseline and midterm KPC surveys were both seriously flawed which again is due to lack of experience and technical assistance. The second project HIS person developed a system of collecting data from the HSAs and community volunteers, but as was stated in the MTE, there was serious inconsistencies in the way the data was collected which put all of the data into question. Project trainers, the HSA supervisors, seemed not to adequately understand the data collection system and therefore could not serve as resources to the HSAs in this regard. The division of tasks according to technical area (ORS, family planning, malaria etc) exacerbated this problem since none of the trainers understood all of the interventions, and yet the data collection forms combine all of the data from each of the interventions into one reporting form. The HIS system was made even weaker when the second HIS specialist (the one who developed most of the tools and systems) quit and was replaced by an even less experienced person. As a result of this weak monitoring system, no one had an accurate understanding of what the project had achieved. Everyone was aware of the inputs (numbers of training courses conducted, shelters built etc) and some of the outcomes (numbers of volunteers trained), but not regarding progress toward specific objectives. Despite this, during the last year of the project the HIS person taught the PAL statistician the data collection and analysis system so that PAL could continue to collect data on the health delivery system. When PAL was bought out, the project HIS person began to work with the District Health Information Systems officer and now the two systems are integrated and the DHMT will continue to collect, analyze and report on the estate-based health delivery system. See Chapter One for discussion of special assessments, and surveys. H. Technical and Administrative Support Aside from technical assistance provided by HOPE Center staff, the project received very little outside technical assistance. Outside consultants were used during the midterm and final evaluations, to develop the Mother’s Reminder Materials, and to conduct a project review during the last year of the project. Project staff would have benefited from TA in setting up information systems, in carrying out and using KPC surveys, in developing a comprehensive behavior change strategy as well as a sustainability strategy. If HOPE Center had a mechanism to generate unrestricted funds, then there would be funds available for such technical assistance. 25 During the first half of the project both technical and administrative backstopping were handled by the same person. While this may have been efficient and cost effective, at times it caused confusion. Just after the midterm evaluation this backstop person moved to Lilongwe and her contact with the project increased whereas the Country Director’s contact decreased. In the middle of the third year, this backstop person resigned and there followed a period where there was essentially no administrative or technical backstopping provided by the headquarters even though costs for this service were being charged to the project. Discussions with HOPE Center staff reveal that the Human Resources Department (on) had one person and therefore recruitment took inordinately long. Finally in 2002 an Africa Regional Director was hired who maintained contact with project staff, visited the project a couple of times and helped to resolve some serious personnel issues. Unfortunately this assistance came too late in the project to rectify many problems. The Technical Support Manager was hired in July 2002 a few days before the PAL restructuring. While this much-needed assistance was useful it also came too late to have a significant impact. I. Management Lessons Learned The evaluation team identified the following lessons with regard to management issues: 1. To better ensure “buy-in” all of the partners should participate in the development of the DIP. 2. While investing in staff development does not guarantee that staff will stay with the project, it is worth the time and effort particularly with regard to technical competence and morale. 3. Annual staff appraisals are not a frequent enough mechanism for providing feedback and guidance to project staff. 4. High staff turnover has had a detrimental effect on the project. Closer supervision and follow up of the project may have identified the problems early enough to avert staff departure. 5. Changes in the NGO’s financial reporting procedures and weak financial management skills sometimes negatively affected cash flow and resulted in the project ending 6 months prior to it’s expected termination date. 6. Technical assistance to help establish and monitor the health information system would have strengthened the capacity of the HIS person and made the HIS system more effective and reliable. 7. High staff turn over in headquarters disrupted administrative and technical assistance to the project. As a result project staff did not receive the assistance they needed. J. Other Issues Identified by the Team Please refer to the case study, which focuses on the NGO/Corporate partnership relationship. K. Conclusions and Recommendations 1. Achievement of Objectives Keeping in mind that the final KPC survey did not follow the standard 30 cluster random selection method and therefore the data is less reliable, it appears that the majority of project beneficiaries acquired key knowledge and adopted many healthful practices related to all but one objective. While the project did not attain all of the levels of achievement it set for itself, 26 improvements across all of the 8 interventions were noted. Most noteworthy is the vaccination coverage, with 81% of all children being completely vaccinated according to their vaccination cards. Use of ORS in the case of diarrhea surpassed the objective, increasing from 66.5% to 87%. Care seeking behaviors in cases of suspected ARI rose from 63.9% to 84.1% and professional attendance at deliveries increased from 54% to 89%. Men’s understanding of STD symptoms increased from 20% to 64.5% and both men and women’s understanding of ways of transmitting HIV/AIDS increased significantly. Given these results, it is evident that even in the face of significant challenges and a very ambitious program, the project was a success. 2(a). Achievements ¾ Although no empirical evidence is available to support this assertion, most estate managers report that worker morale is higher, laborers are more productive and fewer man-days are lost for health-related reasons, because health services are closer and workers have adopted more healthful behaviors. ¾ Access to health education and various health care services has been increased in the project area through the training and support of scores of community health workers. These include 40 Traditional Birth Attendants (TBAs), 29 Community Based Distributor Agents (CBDAs), 54 Oral Rehydration Distribution Point Managers (ORDPs), 37 Breastfeeding Support Group members, 28 T4T volunteers, 20 community counselors, 65 drama group members, 340 Compound Health Committee (CHC) members on 34 estates and 290 Village Health Committee members in 29 villages. ¾ The quality of health on the targeted estates and in the district has improved by building capacity of various health care providers, providing drugs and medical supplies, building/up￾grading infrastructure (1 maternity, 2 multi-purpose shelters and 1 Voluntary Counseling and Testing (VCT) center). ¾ A strong and mutually supportive relationship between the project and the District Management Health Team was established, paving the way for continued support of health services on the estates. 2(b). Constraints ♦ Vastly over ambitious program design: eight intervention areas each with 4 –5 objectives and a total of 86 key messages; ♦ Inexperienced staff and too little technical assistance; ♦ Project HOPE’s lack of understanding of the structure and hierarchy of the tobacco industry; ♦ Press Agriculture’s continual restructuring and financial difficulties, which prevented PAL from providing all of the inputs anticipated; ♦ High staff turnover; ♦ Project HOPE’s lack of unrestricted funds to provide more appropriate inputs to the project including staff development opportunities. 27 3. Lessons Learned ™ When two organizations that have different mandates decide to go into partnership, they first need to agree on the benefits that each partner will gain from the joint effort. In other words, they need to identify the “value-added” to their partnership. In the case of the Kasungu Child Survival partnership the “value added” was that improved health and closer access to health services, would lead to fewer man-days lost which would contribute to higher production. ™ When working with a for-profit business, project activities (training events, meetings, and service provision) need to be planned/scheduled in consultation with the company managers, so that the demands of the work schedule can be taken into consideration (peak season vs. off season availability of workers). ™ It is not accurate to assume that a for-profit partner will have fewer financial/resource constraints than traditional partners such as the Ministry of Health. ™ In the project design stage, an in-depth study of the management structure of the partner needs to be conducted so that the project can be designed to fit seamlessly into that structure. ™ When two organizations go into partnership, the roles and responsibilities of all the stakeholders need to be clearly and specifically defined, assigned and communicated to all the stakeholders in a very transparent manner early on in the project. ATTACHMENTS Attachment 1. Team Members Bonnie Kittle – Team Leader, Independent Consultant Shalote Rudo Chipamaunga, Technical Support Manager, Project HOPE Ellen Thom – Project Manager, Project HOPE Nyuma Manda – Trainer, Project HOPE Edda Mambo – Trainer, Project HOPE John Njunga – Trainer, Project HOPE Lovemore Gondwe – IEC Specialist, Project HOPE Doreen Nyirenda – HIS Officer, Project HOPE Elton Chongwe – District AIDS Coordinator MOH & PMTCT/VCT Coordinator MOH Attachment 2. Assessment Methodology The evaluation was carried out in six stages. The first stage consisted of document review by the outside consultant. Once in country a 3-day team-planning meeting was conducted with the evaluation team members. During the team-planning meeting, amongst other things, the information collection instruments were developed. It was also decided at this time to conduct a modified KPC survey in a sample of project villages. The third stage was the field-based and key informant information gathering. The information was then analyzed and interpreted by the evaluation team and findings, conclusions and recommendations identified. A 4-hour debriefing workshop was held with Limbe Leaf employees and the District Medical Officer to share the preliminary findings and to get additional input. The last stage consisted of writing the evaluation report. Attachment 3. Persons Interviewed and Contacted Mexon Nyirongo – USAID/Malawi, HPN Program Project HOPE Child Survival and Mother Care Project staff (see list of evaluation team members) Dorothy Namate – Country Director, Project HOPE, Malawi Dr. Henry Phiri – District Health Officer Mr. A.W. Mbowe – Deputy District Health Officer B.A. Mkwala- Area Manager Central Zone, Limbe Leaf E. Liwonde – Area Manager, East Zone, Limbe Leaf W. Mtukula, Area Manager, West Zone, Limbe Leaf John Mzembe, Area Manager, PED coffee estates, Press Agriculture Limited Mr. Pete Baker – General Manager, Limbe Leaf Mr. Daka, Assistant General Manager, Limbe Leaf Dr. Jaap Vryburg, Medical Officer, Limbe Leaf Jeremiah JK Mateyo, Medical Assistant, Estate 17 Winston Somanje, Medical Assistant Estate 81 Mr. Issac Kambirinya, Former Human Resource Manager, Press Agriculture Limited 11 Estate Managers, Limbe Leaf 4 Health Surveillance Assistants, General Farming and PED 120 mothers, residents of Limbe Leaf tobacco and coffee estates 120 men, residents of Limbe Leaf tobacco and coffee estates 10 ORDP Managers 8 TBAs 10 CBDAgents Attachment 4. TRAINING TABLE FOR PARTNERS, AND COMMUNITY VOLUNTEERS Oct. 2000 – Feb. 2003 TOPIC/ INTERVENTION TRAINEE TRAINER DATES 1 VHCs Initial Training 340 Community Health Volunteers Project HOPE Staff (CSMC) 1st Quarter of 3rd Year, Oct to Dec 2000 2 TBAs Initial Training 40 TBAs 4 MoH Staff, 1 Project HOPE staff (CSMC) 1 Month, 6th November to 6th December, 2000 3 Reorientation of CHCs and VHCs 21 Community Health Volunteers 13 HSAs Project HOPE Staff (CSMC) 2nd Quarter of 3rd year Jan to Mar 2001 4 Supervisory Skills Training 1 PAL Medical Assistant 1 Nurse Midwife Project HOPE Staff (CSMC) 2nd Quarter of 3rd year Jan to Mar 2001 5 Counseling skills Training for ORDP/GMVs 54 ORDP Managers 1 Project HOPE staff (CSMC), 1 MoHP Staff, 1 PAL staff 2 days, 3rd to 4th February 2001 6 HSA and general supervision 8 HSA supervisors MOH Trainer Dec. 2001 7 Participatory Drama Refresher Training 130 Adult and Adolescent drama Members 1 Project HOPE Staff (CSMC) 8th to 20th December 2001 8 CBDA refresher Training 21 CBDAs; 13 HSAs 1 Project HOPE Staff (CSMC), MoHP Staff 14th to 19th May 2001 9 CBDA refresher Training 21 CBDAs; 13 HSAs 1 Project HOPE Staff (CSMC), MoHP Staff July 22 – 26, 2002 10 HSA Supervisory skills for Nurses and Medical Assistants (PAL) 1 Nurse/Midwife (PAL), 2 Senior HSAs (MoHP) 2 Project HOPE staff (CSMC) 4 days, 18th to 22nd November, 2002 11 Training for Transformation refresher 30 Community Health Volunteers 1 Project HOPE staff (CSMC), 1 MoHP Staff, 2000 TOPIC/ INTERVENTION TRAINEE TRAINER DATES 12 Participatory drama and HIV/AIDS refresher Training 120 Adolescent and Adult drama members 1 Project HOPE staff (CSMC), 2 MoHP Staff 13th to 15th July, 20th to 22nd July, 8th to 10th August 2002 13 Drama Training 65 Adolescent and Adult drama members 3 Project HOPE staff (CSMC), 1 MoHP Staff 3rd to 20th December 2000 14 TBA Refresher Training 29 TBAs 1 Project HOPE Staff (CSMC), 2 MoHP Staff 7th to 20th July 2002 15 CBDA refresher Training 21 CBDAs; 13 HSAs 1 Project HOPE Staff (CSMC), 3 MoHP Staff 1 Week, 21st to 27th July 2002 16 ORDP Managers Refresher Training 44 ORDP Managers 2 MoHP Staff, 1 Project HOPE Staff (CSMC) 3 days, 27th to 29th May, 2002 17 CBDA Initial Training 21 CBDAs; 1 HAS 2 MoHP Staff, 1 Project HOPE Staff (CSMC) 20 days, 11th to 31st March 2002 18 Community Faith Volunteer Counselors Training in PMTCT/VCT 20 Members from various Churches, Organizations and communities MoHP Staff, Project HOPE Staff (CSMC) 13 days, 22nd July to 7th August 19 Integrated BFHI Infant feeding and HIV/AIDS into Health Care and Community Services Training 2 MoHP (KDH) Health Providers Project HOPE Staff (CHAPS) and linkages Project 12 days, 2nd to 14th September 2002 20 Disease Sensitization Programme Adult and Adolescent drama Members 1 Project HOPE Staff (CSMC) 2nd to 8th February 2002 21 VCT/PMTCT Drama Training 32 Adult Drama Members 2 Project HOPE Staff (CSMC), 2 MoHP Staff 28th to 30th May 2002 22 VCT/PMTCT Drama Training 33 Adolescent Drama Members 2 Project HOPE Staff (CSMC), 2 MoHP Staff 25th to 29th June 2002 23 Syndromic Management of STIs for service providers 9 MoHP Staff (KDH Partners) 2 estate staff (MA and Nurse) MoHP (LCH) Staff 3rd to 14 June 2002 24 HSA Training in HSA Methodology course 2 PAL HSAs MoHP Trainers Mponela PHC 2000 TOPIC/ INTERVENTION TRAINEE TRAINER DATES 25 Orientation of DACC Members on PMTCT 22 Members of District AIDS Co￾ordinating Committee Project HOPE Staff (CSMC) 1 day, 15th January 2003 27 Orientation of TBAs on PMTCT 19 TBAs Project HOPE Staff (CSMC), MoHP Staff 2 days, 16th t0 17th January 2003 28 Orientation of Hospital Support Staff on PMTCT 65 Support staff of KDH and its Health Centres Project HOPE Staff (CSMC), MoHP Staff 2 days each session 21st to 22nd and 23rd to 24th January 2003 29 Orientation of CBDAs and TBAs on PMTCT 29 CBDAs and TBAs Project HOPE Staff (CSMC), MoHP Staff 2 days 3rd to 4th February 2003 30 Orientation for Health Providers on PMTCT 19 Nurses and Clinicians from Nkhamenya Hospital and KDH and its Health Centres MoHP Staff (KDH) 2 days 28th to 29th January 2003 31 CBDA supervision 15 HSAs MOH trainers March 2002 32 Training of Breast feeding support groups 27 Community members from Limbe Leaf estates Project HOPE Staff (CSMC), MoHP Staff 5 days, 17th to 21st February 2003