Final Evaluation Report Community Health Partnerships (CHAPS) project Chikwawa District, Malawi Funded by the United States Agency for International Development (USAID), RFA No: 690-97-002. Sponsored by International Eye Foundation 10801 Connecticut Ave. Kensington, MD 20895 Report Written by Richard Crespo, Ph.D. Marshall University School of Medicine Huntington, West Virginia EXECUTIVE SUMMARY……………………………………………………………………...1 PROJECT BACKGROUND ............................................................................................................... 1 MAJOR ACCOMPLISHMENTS......................................................................................................... 1 PRINCIPLE RECOMMENDATIONS................................................................................................... 2 PURPOSE………………………………………………………………………………………...3 METHODOLOGY………………………………………………………………………………3 FINDINGS………………………………………………………………………………………..5 DISTRICT-LEVEL CAPACITY ......................................................................................................... 5 EXTERNAL CONSTRAINTS ............................................................................................................ 6 Discussion……………………………………………………………………………………6 DISTRICT-LEVEL ASSESSMENT OF CAPACITY BUILDING .............................................................. 8 SUPERVISION SYSTEM................................................................................................................ 12 Discussion…………………………………………………………………………………..12 HEALTH CENTER INFRASTRUCTURE........................................................................................... 13 Discussion…………………………………………………………………………………..14 ASSESSMENT OF KNOWLEDGE AND ATTITUDES ABOUT HIV/AIDS........................................... 15 COMMUNITY-LEVEL CAPACITY BUILDING ................................................................................ 16 Discussion…………………………………………………………………………………..17 DRUG REVOLVING FUND ........................................................................................................... 18 Discussion…………………………………………………………………………………..19 GROWTH MONITORING .............................................................................................................. 20 Discussion…………………………………………………………………………………..20 OUTREACH SHELTERS................................................................................................................ 21 REPRODUCTIVE HEALTH............................................................................................................ 21 Discussion…………………………………………………………………………………..23 TRAINING................................................................................................................................... 24 VILLAGERS ................................................................................................................................ 24 Discussion…………………………………………………………………………………..25 RECOMMENDATIONS………………………………………………………………………26 SUMMARY…………………………………………………………………………………….28 List of Abbreviations CBD Community Based Distribution (Family Planning) CHAPS Community Health Partnerships DEHO District Environmental Health Officer DHO District Health Officer GMV Growth Monitoring Volunteer HSA Health Surveillance Assistant IEC Information/Education/Communications IEF International Eye Foundation IMCI Integrated Management of Childhood Illnesses MA Medical Assistants TBA Traditional Birth Attendant TBA Traditional Birth Attendants USAID United States Agency for International Development VHC Village Health Committee Executive Summary The purposes of this evaluation were to: 1. Document the accomplishments of the project in terms of strengthening the district’s infrastructure and community services. 2. Document the benefits of the CHAPS project for the Ministry of Health and Populations. 3. Make recommendations to USAID for future infrastructure development projects. 4. Identify lessons learned and make recommendations for the new Child Survival project in Nsanje District. Project Background The Community Partnership Project (CHAPS) is a component of USAID’s assistance to Malawi. CHAPS is a $15 million, five year initiative designed to utilize Private Voluntary Organizations (PVOs) as partners with the Ministry of Health and Population (MOHP), to extend key health services and enhance institutional capacity. Five PVOs were awarded cooperative agreements in five districts. The IEF was awarded one of the cooperative agreements to work with the MOHP in Chikwawa District in the Lower Shire Valley, where IEF has a history of technical and development assistance. An external evaluator conducted a mid-term evaluation in September of 1999, in conjunction with a Final evaluation of another project. The USAID Mission recently completed its evaluation of the entire program in February 2002. IEF is obligated to conduct an End of Project evaluation to fulfill requirements of its cooperative agreement. Major Accomplishments 1. The accounting system was computerized and is functioning. Posting of accounts were two to three months behind schedule, but all tasks were done electronically and reports were generated from the system. 2. Budgeting was done on the computer, which facilitated planning when problems occurred such as unannounced cuts in monthly budget allocations from the central office. Now that the manual system has been replaced, it can be expected that the accounting will be done electronically in the future. 3. The fleet management system has been credited with increasing the number of functional vehicles from one to eight. 4. The person initially trained as the fleet manager was still in place and producing daily and monthly usage and maintenance reports. 5. The radio system for communication among the health centers and the district hospital was evident at every health center that the evaluation team visited. It worked and was used extensively. 6. CHAPS contributed to sustaining the child survival project that preceded it. Financial and technical assistance from CHAPS that concentrated on capacity 2 building and district-level technical assistance, and provided support for continuing community-level services. This is a model worth emulating. 7. Volunteers who distributed contraceptives (CBDs) were the best trained and most active of all the volunteer groups. It is impressive to note that all of the volunteers were active; they were the most knowledgeable, served multiple villages and had neat and up-to-date records. It should also be noted that the CBDs were women. Principle Recommendations Following are three principle recommendations. The full list of recommendations can be found beginning on page 26. 1. In order for a decentralized system to work effectively, the district should have decision-making power on personnel, material resources and have input into national policies. The decentralization in this CHAPS project did not go far enough, and as a result the project was limited by significant factors over which it had no control. 2. Objectives and interventions for capacity-building projects should be limited to a short list of major interventions and then most of the staff resources should be limited to these interventions. It will mean that some important problems may not be addressed, but given the limitations and constraints in the Malawian health care system it is more realistic to keep the list short. 3. CHAPS projects should provide an incentive for the District Health Officer to stay in the district for the life of the project. A PVO such as IEF could offer to pay for, or contribute towards, an MPH. The DHO could periodically do course work during the life of the project and receive a scholarship to finish at the end of the project. There is a risk in that the DHO could take advantage of this incentive and still leave the district without good leadership. This is a risk worth taking given the alternatives. 3 PURPOSE The International Eye Foundation (IEF) commissioned an end-of-project evaluation of their Community Health Partnerships (CHAPS) project in Chikwawa District, Malawi. The field survey was conducted from August 30, through September 14, 2002. The United States Agency for International Development (USAID) funded the project, RFA No: 690-97-002. A number of stakeholders contributed to defining the purpose of the evaluation. The consensus on the evaluation’s primary purpose was to: 1. Document the accomplishments of the project in terms of strengthening the district’s infrastructure and community services. 2. Document the benefits of the CHAPS project for the Ministry of Health and Populations. 3. Make recommendations to USAID for future infrastructure development projects. 4. Identify lessons learned and make recommendations for the new Child Survival project in Nsanje District. Methodology The IEF contracted this author to be the Lead Evaluator. The IEF Program Director and Child Survival Coordinator had several telephone conversations with the evaluator about IEF’s agenda. As a result the evaluator prepared an agenda of the issues that IEF wanted to address. Once in Malawi the Lead Evaluator, along with the IEF Child Survival Coordinator, IEF Country Director and CHAPS Project Director spent an afternoon to further develop the evaluation agenda. They expanded the purposes of the evaluation and defined what should be evaluated in order to accomplish the stated purposes. This group identified a set of inputs out of the whole list that would be the focus of the evaluation. Subsequently, this group met with the IEF project team and a representative of the District Management Team who provided additional input to the evaluation plan. The whole team spent a day and a half designing the data gathering protocols and instruments. The Lead Evaluator trained the team in the basis of instrument design, and organized the team into small groups to develop tools for the specific groups who were to be interviewed. He then circulated among the groups to give advice and review drafts of their documents. The team used a combination of individual and group interviews for data gathering. The following table lists the research population, data gathering method, and number of people interviewed. 4 Table 1: Interviews and number of participants Groups Interviewed Method Number Representatives of the District Management Team, Program Coordinators, and District health care staff Focus group and nominal group technique 6 District Health Officer Personal interview 1 Health Center staff (medical assistants and nurses) Survey 10 staff from 6 health centers Health Surveillance Assistants Personal interview 16 H.S.A.s from 13 villages Growth Monitoring Volunteers Personal interview 25 from 15 villages Villagers Personal and group interviews 67 from 13 villages Village Health Committees Personal and group interviews 16 committees Community-based Distributors (contraceptives) Personal interview 161 Traditional Birth Attendants Personal interview 151 IEF Project Director Personal interview 1 Drug Revolving Fund volunteers Personal interview and inspection of drug kits 5 villages IEF accountant Personal interview 1 1 CBD’s and TBA’s serve multiple villages each Data gathering at the health centers and in villages occurred over a four-day period. The schedule is listed below. The locations listed below refer to the area where the health centers were located. These health centers were selected because they served the largest number of people in their catchment areas. From these centers the team dispersed to near-by villages to interview villagers and the health care volunteers. Wednesday PM: Ndakwera catchment area Thursday AM: Makhwira catchment area Thursday PM: Makhwira catchment area Friday AM: Ngabu catchment area Friday PM: Chipwaila catchment area Saturday AM: Kakoma catchment area Saturday PM: Chapananga catchment area In addition to conducting interviews, the Lead Evaluator reviewed mid-term and annual reports, training reports, quality assurance reports, and findings from end-of- 5 project assessments done prior to his arrival. The EOP assessments were namely: comparison of mid-term and final IMCI skills, home-based caregivers activity report, reproductive health counseling skills and knowledge, DRF assessment and attitudes about HIV/AIDS. Findings This section is organized according to the outputs in the logical framework of the original proposal. Each subsection begins with a list of the findings followed by an analysis. At certain points crosscutting issues will be addressed where issues pertain to more than one output. A note needs to be made about findings related to community volunteers. Unlike some projects where volunteers have a variety of responsibilities, this project trained different types of volunteers from a village, each with specific responsibilities. Findings and analysis regarding volunteers are thus included in the pertinent output sections of this report. For example, the drug revolving fund volunteers will be included in the subsection on the drug revolving fund, while the community based family planning distribution volunteers are included in the section on reproductive health. District-level Capacity The major findings are listed below. 1. The investment in leadership development has been limited by the constant rotation of district health officers. During the five years of the CHAPS project there have been nine District Health Officers, almost two per year. 2. The accounting system was computerized and functional. Posting of accounts were two to three months behind schedule, but all tasks were done electronically and reports were generated from the system. 3. The fleet management system been credited with increasing the number of functional vehicles from one to eight. 4. The person initially trained as the fleet manager was still in place and was producing daily and monthly vehicle usage and maintenance reports. 5. The district has changed the vehicle maintenance system from contracting with dealerships to hiring their own mechanics. The IEF accountant stated that the mechanics engaged in a scam of selling parts that disrupted the maintenance schedule. 6. Seven of the coordinators who were interviewed rated the interventions of the radio communications system and equipping the Ngabu rural hospital with an operating theater as those that had the greatest success. 7. None of the health centers had functional motorcycles because of a lack of parts. Even though the DHO invested 150,000 Kwacha (approximately $2,000) for motorcycle maintenance in the last month, none were working at the end of the project. 6 8. The program coordinators were trained in management skills, but there was no investment in training the district’s administrative staff in management skills. Consequently, the administrative support for health programs was weak. External Constraints Several factors beyond the District and IEF’s control inhibited CHAPS. These factors hampered IEF’s ability to implement this project. 1. The Ministry of Health was substantially understaffed. The medical providers (physicians and medical assistants) had the pattern of rotating from post to post around the country because they could find an opening almost anywhere. This constant change disrupted capacity building. For example, there were almost two DHOs a year during the life of the project. Each time a new DHO took over priorities and procedures were changed thus disrupting continuity in leadership. Relationships between the IEF Project Director and the DHO would be established during the course of a year, then the DHO would take another position and the Project Director would have to start over. 2. The Nursing Director, who had acquired years of experience in CHAPS left a month before the end of the project. This human resource will thus not contribute to sustaining the capacity that was developed by CHAPS. The new nursing director was a recent graduate of nursing school and did not have training in CHAPS interventions such as IMCI, health volunteers, etc. It was unfortunate that just as CHAPS was ending, an inexperienced person took over. 3. One DHO did not want the pharmacy to be computerized so the system was not used for almost a year. 4. In the last year the pharmacist was trained in using the computerized pharmacy system, but this person left before the end of the project for a higher paying post. There was no backup, thus this human resource was lost. 5. The plans for personnel management were undercut by staffing shortages. For example, the DHO’s job description was of minimal use when the medical officer resigned and he had to function as DHO and medical officer. In another case, the Reproductive Health Coordinator had to cover for nursing shortages and did not have time to perform her duties in the community. 6. Understaffing made it difficult to assess staff members’ performance based on job descriptions. Evaluating a staff member’s performance was complicated by the fact that a supervisor could not always know if the person’s poor performance was due to being pulled away for other duties or because he or she was negligent. Discussion The CHAPS project in Chikwawa District made some important contributions to the district’s capacity to deliver health service. Components that were strengthened the 7 most were accounting, fleet management and the technical skills of the health services staff. In the area of accounting the outcome of CHAPS was that the financial reports were generated at least quarterly and used for program and budget planning. By the end of the project the accounting staff did all their work on the computer. Budgeting was done on the computer, which facilitated planning. For example, when the MOH central office made unannounced cuts in monthly budget allocations, the accounting staff was able to quickly make adjustments and the DHO could make rational decisions about how to allocate the funds. Now that the manual system has been replaced, it can be expected that the accounting will be done electronically in the future. This is an important contribution to capacity building. One potential complication may occur when the central office of the MOH adopts its own electronic accounting system. If it is different from the one being used in Chikwawa (i.e. Quicken), then the District will have to change to the new system. Assuming that the current staff members remain in the District, their task of learning a new system will be easier because of what they learned from CHAPS. The most significant barriers to capacity building in this project were due to factors beyond the control of CHAPS. These barriers are listed above, beginning on page 6. In future capacity building projects, USAID, the MOH and the PVO should agree on policies that address the problems associated with understaffing. The district must be in control of the human and financial resources in order for it to be accountable. No system will be perfect, especially given the dramatic shortage of physicians and nurses, but the current situation must be improved. One possibility is to create a package of incentives for the DHO to stay at his or her post for the life of the project. Part of the incentive package could be support for advanced training such as an MPH degree. One factor that limited the IEF-District counterpart relationship was that some IEF staff did not have professional qualifications commensurate with those of their MOH partners. For example, the IEF counterpart to the DHO (a physician) was a medical assistant. And the IEF counterpart to the Reproductive Health Coordinator (a nurse) had no degree in health care. The consequence was that the IEF staff did not have as much credibility as they could have. In some instances IEF staff were not able to hold the MOH staff to the highest professional standards because they did not have commensurate professional training. For example, staff members did not have the overall knowledge of nursing standards, to give professional counsel to the MOH nurses when their performance was lacking. This observation is not intended to disparage the IEF staff. They were highly skilled, dedicated and had a passion for their work. No amount of professional training can compensate for that. It must also be recognized that the shortage of health care professionals in Malawi is going to affect a PVO’s ability to recruit staff with professional degrees. It is not necessary, however, to have an exact match in professional degrees between counterparts. IEF could consider having a comparable match in professional skills in a few key positions without having to do so in all counterpart relationships. 8 While CHAPS invested a lot in management training for the health program staff, there was no commensurate training for the district’s top administrative staff. Consequently at times the administrate staff thwarted the program staff. For example, every time that a new district administrator took over, this person would change policies and procedures regarding the fleet management. He would disregard the work done by the Fleet Manager who had set up systems based on his training through CHAPS. Similar situations occurred with the pharmacist, accountant and program coordinators who were trained by CHAPS. Another factor that affected capacity building in this project was the large number of interventions. The number of interventions that were in the plan of action diluted the impact of those that were more crucial. CHAPS had nine program interventions – reproductive health, community-based management of common diseases, nutrition/food security, safe water and sanitation, preventive eye care, strengthening village health committees, HIV/AIDS prevention, home based care for persons with AIDS and IMCI. Concomitantly, a major investment was made to strengthen financial management, personnel management, district planning and fleet management. Additionally, the district’s coordinators had major initiatives in Safe Motherhood, Information/Education/Communications (IEC), and in working with traditional birth attendants (TBA). In total the logframe had 20 outputs. Training sponsored by this project covered thirty-three distinct topics. Given the staffing problems and constraints that are discussed above, there simply were not enough people to do all this work and participate in all the sponsored events. With these constraints it was unrealistic to expect that IEF and the district’s staff could do a thorough job in all areas. Evidence of this is that the district-level staff members rated most of their programs as having moderate to low success (refer to Chart 1, page 10), and the data from the field assessment showed deficiencies in some of the programs. (Refer to page 14 and the following section.) CHAPS in Chikwawa could have been even more effective if it had concentrated on capacity building and a core set of program interventions. The effectiveness of this CHAPS project was affected by the volume and wide range of activities. It may be more prudent to concentrate on fewer outputs and interventions and thus have time to follow through more comprehensively. District-Level Assessment of Capacity Building The evaluation team was able to interview six staff members who worked at the district level, namely: • Paul Chunga- DEHO (District Environmental Health Officer) • Fellina Kaliati- Safe Motherhood Coordinator • Frederick Kapinga- IMCI Coordinator • Steven Kanjoloti- Primary Eye Care Coordinator • Patricia Tembo- EN/Midwife who works in the operating theatre • Ephram Duncan- Dermatologist Assistant 9 The District Environmental Health Officer, Paul Chunga, also participated as an evaluation team member. It should be noted that despite the advance notice and efforts of the Project Director, it was not until the next to the last day in the field that he was able to arrange for the evaluation team to interview the District Management Team or the program Coordinators. The Lead Evaluator asked to interview the District Management Team but they always had excuses for not doing so. He also asked to interview the Program Coordinators, but was only able to meet with three of them. Unfortunately, the district￾level staff in general did not demonstrate much interest or sense of obligation to participate in the final evaluation. District Coordinators were critical stakeholders in CHAPS, but they were the ones who were most affected by the constraints inherent in the district’s understaffing. The IMCI and the Safe Motherhood coordinators reported that they were able to only perform a fraction of their coordinator duties because they had to cover for unfilled positions and for colleagues who were on leave or absent. When asked about the barriers to their roles as coordinators they listed the following: • Conflicting priorities with patient care • Staff shortages • Lack of transportation • Poor program scheduling It is important to note that the barrier of transportation was not because vehicles were in disrepair, but because they were used for transporting patients. In regard to program scheduling, the coordinators reported that events such as supervision for IMCI and reproductive health would be scheduled for the same day in different areas. Only one vehicle would be available, so one of the visits would be canceled. While the District Health Management Team did discuss scheduling, decisions were not always communicated with coordinators outside of the DHMT. Consequently, some coordinators’ schedules were not taken into account. The district-level staff members were asked to identify the programs that were supported by CHAPS. It was interesting to observe that none of the seven mentioned HIV/AIDS without prompting. This is an indicator that even though USAID/Malawi and IEF considered this a critical program, that the coordinators did not view it as a priority. IFF staff stated that HIV/AIDS training had been a priority, and they were surprised that it was not mentioned by coordinators. The absence of HIV/AIDS was apparent when listing the training they had received and in their list of programs that CHAPS promoted. For example, no mention was made of the training or programs in Prevention of Mother to Child Transmission, and home based care. When the Lead Evaluator pointed this out, the coordinators and other district-level staff rushed to affirm its importance. In a formal interview setting, however it was not possible to determine if indeed they considered it important but forgot to mention it, considered it unimportant, or if they were in denial about the HIV/AIDS epidemic. 10 The lead evaluator tried to follow-up with personal conversations with some of the coordinators and IEF Malawian staff. Their view was that that HIV did not have a high prevalence in Chikwawa, thus the reason why they did not give it much importance. This point of view was supported by interviews with HSA’s and medical assistants in the health centers. Unfortunately no data were available on the prevalence of HIV in this district. This is a matter of concern, due to the fact that the national prevalence rate of HIV/AIDS indicates that it is a major problem. If some people are in denial, or they simply do not know, then the district could be in a position of not being able to respond to a major epidemic. The district-level staff members were asked to rate CHAPS-supported interventions based on the extent to which they achieved success and on their potential for sustainability. The rating was based on a scale of 1 for high level of achievement, 2 for a moderate level, and 3 for a low level. The findings are presented in the following table. Chart 1: District-level Staff Members’ Rating of CHAPS’ interventions Establishing the radio communication system and equipping the operating theater at the Ngabu rural hospital were the two highest rated interventions. The operation of the radio system for communication among the health centers and the district hospital was evident at every health center that the evaluation team visited. It is interesting to note that the district-level staff rated the operating theater just as high as the communication system, even though it was not functioning at the time of the final evaluation. Their rationale was that when it becomes functional, it will relieve much of the pressure put on the district hospital. The trend in Chart 1 rates tangible interventions (i.e. radios, operating equipment, buildings, and transportation) at the higher end of the scale. The high rating of 0123 Sanitation Facility-based IMCI supervision skills Quality assurance Infection prevention training IMCI management Growth monitoring shelters School eye screening Primary eye care Transportation HIV/AIDS education Operating theater Radio communication High Moderate Low 11 HIV/AIDS is tempered by the fact that the interviewees had to be reminded of this intervention. Primary eye care is also rated high in part because it has its own equipment and mobile unit. The interventions that were given lower ratings were those that were programmatic and skills oriented. A noteworthy observation is that the district-level staff did not rate sanitation very highly, in contrast to the rating given by volunteers and villagers. (Refer to the findings and discussion beginning on page 17.) Supervision also received a low score. Despite the fact that supervision systems development and training were a high priority of CHAPS, the problems associated with understaffing had a negative affect on its success. The district-level staff also rated CHAPS’ interventions in terms their potential for sustainability. Sustainability was defined as the continuation of an intervention for 12 months after the end of the project. The next table presents ratings by intervention. Chart 2: District-level staff members’ rating of the sustainability of CHAPS interventions. 0123 Transportation Quality assurance Facility-based IMCI Growth monitoring shelters School eye screening HIV/AIDS education Primary eye care Interventions that the district-level staff members believed would have a high level of sustainability were primary eye care and the operating theater. It is ironic that they rated sanitation as having low success, but that their confidence in the community members’ ability to sustain the boreholes and latrines was high. One factor that explains this discrepancy is that the district-level staff wished that there had been more villages served with this program. Interventions in which they projected a drop-off are IMCI, transportation, quality assurance and the radio communication system. They had little confidence that the fleet management system would continue to function well without CHAPS. They believed that prospects for sustaining IMCI interventions would decrease because of uncertainty of a consistent drug supply and the lack of continuing education. Overall, the coordinators expected that CHAPS interventions would have moderate to low prospect for sustainability. Factors that affected their assessment included the country’s financial crisis and a shrinking health care budget, and revolving district High Moderate Low 12 leadership. Additionally, they stated that high staff performance would probably drop because of a lack of staff member accountability. Supervision System These are the findings on the supervision system. 1. Staff members who were based in the district center (the district hospital) completed less than half of their supervision visits. 2. The DHO had little time to invest in supervision because of having to perform the functions of district health officer, chief medical officer and physician on call. Consequently none of the district coordinators were supervised on a regular basis and were not given performance reviews based on their job descriptions. 3. The health center staff members stated that they were supervised, but it was done sporadically and by a wide variety of people. 4. Twelve of 16 HSAs stated that they met monthly with their village health committee. Three stated that they met bi-monthly and one stated that he never met with the village health committee. 5. HSAs knew their job responsibilities (according to their job description) and actively completed them. Their most common activities include health talks, under-five clinics, reporting outbreaks, and village inspection. 6. HSAs actively supervised the growth monitoring volunteers. One hundred percent of the volunteers (25/25) stated that they were supervised by HSAs. Village health committees verified this finding. All of the committees that had these volunteers in their villages stated that HSAs were their supervisors. 7. HSAs actively supervised CBD volunteers; 100% of CBD volunteers said they were supervised by HSAs. Eighty percent reported being supervised within the last month. Discussion The supervision system in CHAPS worked best at the community level. HSAs had substantial supervisory responsibilities and did their job. A factor in the success at this level was the positive relationship between IEF’s previous child survival project and CHAPS. Additionally, the external constraints that hindered supervision at the district center did not affect activities in the community. There was a great deal of positive synergy between the child survival project and CHAPS. This is a model worth emulating. At the district and health center level, staff members are not supervised on a regular basis. The system of performance standards and staff reviews did not work. It was difficult to identify specific flaws in the system because the external constraints make any supervision system inoperable. One issue that will need to be addressed in the future is how performance reviews fit in African culture, and in particular the Malawian culture. A performance review system based on direct interaction is difficult to implement in a culture that places a premium on relationships and shies away from 13 direct criticism. There is a need to identify and apply effective ways to express disapproval for poor job performance in African cultures. Health Center Infrastructure The evaluation of the health center infrastructure focused on personnel management, services and drug supply. The findings are as follows: 1. The evaluation team visited six of the eleven health centers in Chikwawa District. Five of the six had a Medical Assistant (MA) present. 2. Four of the six centers had a nurse present. 3. Half of the centers were staffed with both a MA and a nurse. Only one health center had a full staff team of MA, nurse and Senior HSA. 4. All of the MAs and nurses could properly explain their duties. They knew their job descriptions. 5. The one Senior HSA who was interviewed by the evaluation team described half of his official duties. 6. The health center staff members stated that they had training in management skills, quality assurance, IMCI, and family planning. Only one mentioned HIV/AIDS, drug revolving fund and primary eye care training. 7. Two of ten staff members stated that they had received a refresher course. No one had received a second refresher course. 8. All ten of the staff had received supervision. Five had been supervised by the DHO, two by the IMCI supervisor, one by the DEHO, one by the family planning coordinator and one by the District Nursing Officer. 9. The health center services that were ranked as the most important were acute care and antenatal services. Family planning services was ranked in second place. The services that the health center staff identified as having a lot of activity were all related to maternal & child health, reproductive health, growth monitoring and safe motherhood. 10. The services identified as having very little activity in the community were home￾based care for persons with AIDS, and the program of Prevention of Mother To Child Transmission (PMTCT). 11. The chart on the next page presents the status of drug availability at the six health centers. 14 Chart 3: Availability of Drugs at the Health Centers Discussion The evaluation team expected that most of the health centers would not have MAs, and it was thus somewhat of a surprise to find that five of the six had MAs. The reason why it was unexpected is that MAs frequently rotate from post to post. In only half of the health centers, however, did the team find both a MA and a nurse working simultaneously. All the staff knew their duties, thus the emphasis that CHAPS placed on job descriptions and performance based on assigned duties was successful at the health center level. It should be noted, however that the health centers have a simple organizational structure, and thus did not have the same pressures as the more complex district hospital structure. Additionally, while officially the MAs and nurses’ duties are separate, the evaluation team found that when the MA was not present, the nurses functioned as the MAs. They knew the protocols and patient management procedures from having worked side-by-side with the MAs. This is part of the “reality in the field” that cannot be bound by official distinctions in professions. In the judgment of this evaluator, it is better to have the nurse step in than to close the center. A pre and post survey of IMCI skills done by the project staff before the final evaluation provided evidence that health center staff improved their skills. The areas in which they showed the greatest improvements were assessment of danger signs, assessment for malnutrition and assessment of main symptoms. Some of the specific new skills that they acquired were ability to assess ear problems (5% to 70% of health 0123456 Fasidar Aspirin Iron GV Paint Benzoic Benzyl Acid Panadol Tetracycline Whitfield Ointment Abendazole Sulphur Salicylic Acid Ointment 15 workers), counting respiratory rate (5% to 97%) assessment of malnutrition (3% to 65%) and giving the caretaker the child’s diagnosis (31% to 82%). Another part of capacity building that was successful was supervision. All of the staff reported having been supervised. This assessment must be tempered, however, by the problem of the lack of quality control, especially of drugs. Chart 3, above, shows that no health center had a full complement of drugs. And only one drug, fansidar, was available in all centers. Yet the district pharmacy was well stocked most of the time and documentation showed regular shipments to the health centers. Drugs were disappearing somewhere between the district pharmacy and the health center shelves. There was no documentation of shipment dates and no balance sheet of quantities of drugs received, distributed and in stock. With no paper trail it was impossible to document the nature of the problem. In future projects, whether CHAPS or child survival, a system of drug quality control needs to be in place before they are distributed to health centers and communities. Another problem was the use of tetracycline at the health centers. It is surprising that this antibiotic was used without any control. There was no documentation on whether it was administered correctly, nor whether or not patients were taking it correctly. The way that it is used at the health centers may be was creating resistance to what is a very valuable drug. It is surprising that the MOH allowed it to be used without any quality control. The health center staff supported the finding from the villagers and village volunteers that maternal and child health was a major activity in communities. The emphasis on this area was also evident in the staff’s assessment of their own activities. HIV/AIDS is a topic that again is not identified as a priority by health center staff. Although IEF staff placed emphasis on this topic, it was not regarded as a major activity by the health center staff. Assessment of Knowledge and Attitudes about HIV/AIDS The IEF staff conducted an assessment of baseline and end of project knowledge and behaviors about HIV/AIDS prior to the final evaluation. These findings add insights into the discussion about HIV/AIDS. A very high percentage of both men and women had heard about AIDS at the baseline (90% and 88% respectively). The percentages increased by the end of the project (97% and 92%). A similar pattern existed for knowledge about risk factors and behavior change. One especially encouraging finding from this survey is that at baseline, 88% of male respondents said knowledge of AIDS changed their behavior, which increased to 99% by the end of the project. The most frequently practiced preventive behaviors mentioned were, having sex with only one partner (60%), discussing faithfulness with partner (51%) and avoiding sex with prostitutes (39%). It is evident that there is an extensive awareness of HIV/AIDS and awareness that behavior change is needed to prevent it on the part of those who were interviewed. One factor that could have influenced the relatively low important that health care staff gave to this topic is that a high percentage of people know about HIV/AIDS and its risk 16 factors. Health care professionals may have a false sense of security. What is not known is the extent of denial and the prevalence of infection in the adult population. In future projects it will be important to have this information in order to tailor training and education to the full range of knowledge and behaviors necessary to address the problem. Community-Level Capacity Building Capacity building at the community level mainly consisted of volunteer leadership training. The major findings are as follows: 1. Village health committees, HSAs, and village headmen independently stated that that they consulted with each other about health issues. 2. Community leadership was stable throughout the life of the project. The following table presents a summary of this finding. Table 2: Village leaders’ average years of service Leadership Position Avg. Yrs. Village Health Committee 4 Growth Monitoring Volunteers 3.75 Community-based Distribution Volunteers (family planning) 5.5 TBAs 8 3. According to community leaders, the major accomplishment of the CHAPS project consisted of improvements in water and sanitation. The following table presents a summary of this finding. Table3: Village leaders’ estimation of major accomplishments Leadership Position Proportion who Identified Water and Sanitation HSAs 16/16 Village Health Committees 8/16 latrines 12/16 boreholes 4. Leadership in the village health committees was consistent and stable. As mentioned above, they have served an average of four years and the number of members in the 16 committees that were interviewed by the evaluation team ranged from 9 to 13. This was a remarkably consistent range of service. 5. Village health committees met on a regular basis: 13 of 16 met monthly. 6. Nine of 16 committees kept written minutes of their monthly meetings. 7. The committees were actively addressing health issues. The three most common topics were child nutrition (associated with growth monitoring), disease surveillance, and community mobilization. 17 8. Village health committees held an average of two community-wide meetings per year. Nine of 16 discussed child nutrition at their last community meeting and 8 of 16 discussed oral rehydration. It should be noted that some committees discussed both topics. 9. A large majority of the volunteers were elected to their posts. The following table presents a summary of this finding: Table 4: Percent of volunteers who were elected Volunteers % Elected Growth Monitoring Volunteers 80% Community-Based Distribution (family planning) Volunteers 81% DRF volunteers 80% 10. A large majority of the volunteers identified their specific duties. A summary of this finding follows. Table 5: Percent of volunteers who correctly identified their major duties Volunteers % Correctly Identified Growth Monitoring Volunteers 100% Community-Based Distribution (family planning) Volunteers 100% DRF volunteers 100% TBAs 100% Discussion The community capacity building was the strongest component of this CHAPS project. A large majority of volunteers were elected, knew their job responsibilities and had been working for many years. As previously, part of this success is due the fact that IEF worked in Chikwawa prior to CHAPS. IEF staff did a good job preparing community leaders as demonstrated in CHAPS. Villagers and volunteers had a strong interest in water and sanitation. Where communities were given the opportunity, the majority accepted and used boreholes and pit latrines. It is encouraging to see that there is no resistance to these interventions. If there is an opportunity for a follow-up to CHAPS, investment should be made in providing water and sanitation to as many villages as possible, as villagers are eager for improved sanitation facilities. One important factor in this observation is that the CHAPS project had a sub￾contract with Concern Universal for water and sanitation projects. Prior to the CHAPS project, Concern Universal had installed 115 water points. Pit latrine coverage increased from 22% to 38% in the district. A flood disaster, however, destroyed some of this infrastructure. In this CHAPS project, Concern Universal rehabilitated 25 boreholes 18 and drilled 20 new ones. Concern Universal conducted training to strengthen existing village health committees and established new ones where they did not exist. At the end of the project, each village had committee members who were trained in borehole maintenance. The working relationship between IEF and Concern Universal was exemplary and their respective staff collaborate very well together in the field. By sub-contracting the water and sanitation component IEF was able to extend the scope of the project without having to invest in it own additional human resources and technical expertise. This is a good model that should be emulated. Drug Revolving Fund The major findings are as follows: 1. One out of five community drug revolving funds (DRFs) had a full stock of medicines. (See the table below for a description of the drugs on hand from each committee.) 2. Four out of five communities had an active drug revolving fund committee. 3. Two out of five DRFs had a written accounting system and their books were in order. 4. Two out of five DRFs had good records of drugs distributed and payments received. These records were neat and up-to-date. One DRF had records on scattered pieces of paper with illegible notations, while another had a record book, but no entries for the year 2002. One DRF had no records of any kind. 5. All of the volunteers stated that they had no trouble getting payment from villagers. 6. There were two major problems in the re-supply of drugs. a. The price of drugs has gone up, thus when the volunteers went to restock, they are forced to buy less quantity than expected. In one village, the drug revolving fund committee would not let volunteers raise prices to cover the full cost of the new medicines because the price would exceed that of the local village stores. b. There was no quality control for drugs sent to the health centers and then on to the village. The district pharmacy had adequate records of sending drugs to the health centers, but there was no documentation of what happened to them from that point on. As seen in the table on the next page, most of the community DRFs were under stocked. 19 Table 6: Drugs available in the community DRFs Villages Drugs Matengambari Chipwhaila** Mamakhula Manjolo Julio Fancidar* X X X Aspirin* X X X X G.V.* X ORS* X X X Tetracycline* X Calamite Lotion* X Iron tablets* X X Chloram￾phenicol X Benzyl Benzoate X * Drugs that are on the official formulary. ** Evaluation team was not able to see the drug kit and confirm that they had drugs. Discussion Drug supply was a constant problem in all but one of the community DRFs that were assessed by the evaluation committee. The district pharmacy had an adequate supply of drugs and records of filled orders. The problem was the control of drugs at the health center. The health centers did not have a quality control system so there was not way to audit the drugs received. It was not possible to determine which drugs were dispensed at the health centers and which were distributed to community pharmacies. While some people speculated that drugs were sold, the evaluation team could not to document the allegation due to lack of a paper trail. The DHO was supposed to supervise the medical assistants, but it was impossible to expect him to do so with all his other responsibilities. (Refer to the discussion on page 6.) It is ironic that the one village that had a full stock of medicines was the one that did not have a road and was cut off from the health center during the rainy season. Two factors were associated with the success of this village DRF committee. First, the DRF volunteers were high quality people. They were confident, very articulate about their responsibilities, well organized and creative. For example, when this evaluator told them that other committees did not know how to charge for liquids (e.g. GV) one of the men immediately whipped out a measuring spoon. Second, a medical assistant who had been at his post for nine years, and had a strong commitment to people’s welfare, led the health center that served this village. This medical assistant and one other in a neighboring catchment area had refused opportunities to rotate to different posts, having decided to settle in the area. The IEF staff conducted a survey of 41 DRF committee members before the final evaluation. This study had similar findings. Sixty-seven percent of committee members interviewed stated that the main reason for not having a full stock of 20 medicines was that they were not available at the health center. A second was that 44% could not document their cash flow and stock for the last three months. On the other hand, 88% reported that community members never complained about the price of drugs, and they had a good understanding of the roles of DRF committee members. Additional findings from the IEF survey were that community members had a high level of awareness of the DRF in their village (98%) and that they appreciated DRFs as an easy way to get medicines (82%). Growth Monitoring The findings are listed below. 1. Twelve of 16 village health committees stated that they had growth monitoring volunteers. Two committees did not have any volunteers and two did not know if they had volunteers. 2. Where growth monitoring volunteers were active, the village health committee members clearly identified their main responsibilities. All of the committees stated that the volunteers weighed babies. 3. Sixty percent (15/25) of the volunteers stated that they received support from the community in terms of respect and/or teamwork. Forty percent stated that they received no support. 4. The most common problems that the volunteers faced included, a lack of writing materials (40%), a lack of incentives (32%), and a lack of refresher courses (16%). Discussion Growth monitoring volunteers had a visible and consistent presence in their villages. As presented in tables 4 and 5, most them had been elected and had served in their villages for over three years. They have been an important factor in child health, being one of the two most frequently discussed topics in village health meetings. The most frequently barrier to success mentioned by volunteers was a lack of writing materials. It is important to realize, however, that an unnecessary dependence would be created if volunteers expected these materials to be supplied “from the outside.” It is important to recognize that none of these barriers paralyzed the volunteers. In regard to the lack of incentives, the evaluation team was careful to word the questions so that they did not refer to cash or equipment. A lack of incentives referred to items such as clothing and carrying bags. During data analysis one team member made the comment that the volunteers had already been given t-shirts and bags. Incentives such as t-shirts, hats and bags, however, can have a worth beyond their monetary value. The value of an annual gift is a small price to pay for a year’s worth of volunteer labor. Continuing education can also be an important incentive, beyond the ostensive purpose of enhancing knowledge and skills. These events can have a social and emotional value that can compensate for voluntary service. The “fun factor” should be acknowledged as an important incentive. 21 Outreach Shelters As part of community-level infrastructure, development the CHAPS project supported the construction of outreach shelters. These shelters are large enough to hold groups of people for activities such as monthly growth monitoring sessions, health talks and community meetings. Seven shelters were constructed. Most of the shelters were not finished until the end of the project. The evaluation team inspected four of them. Each had recently been finished but had not yet been used. The project staff members wanted to promote community participation in the construction, but they did not provide enough supervision and support for community organization. All of the HSAs reported difficulty in getting community cooperation, low attendance on workdays and shortage of supplies. They also reported that each shelter should have served an average of seven villages, but that on average 4 villages per shelter participated in construction. This finding indicates that a 1 to 4 ratio of outreach shelter to villages might be a more effective ratio. An additional problem reported by the MOH reproductive health coordinator was that some of the shelters were used for storage rather than as a gathering place. Consequently these shelters were of little use during the life of the project. Reproductive Health This section contains findings and discussion related to the Community-Based Distribution (CBD) volunteers and traditional birth attendants. The first list has the findings from the CBD volunteers. 1. All 16 volunteers interviewed were active. 2. All volunteers served more than one village. 3. All stated that the use of family planning methods had increased, however data to verify this was not available. 4. CBD volunteers had an average of more than three refresher courses. 5. When asked, without prompting, for their reasons for making referrals, 15 out of 16 volunteers mentioned that mothers requested family planning methods other than the pill. 6. Nine of 16 mentioned that they referred a mother for reasons other than family planning. The reasons included: sexually transmitted diseases, infertility and general physical examinations. 7. All of the volunteers identified at least one complication in pregnancy that warranted referral. 8. Six of the volunteers were asked to show their record books. They volunteered to do so and each one had neat, complete and up-to-date records. 9. All the volunteers stated that they took the initiative to visit mothers when their cycle of pills was complete to check on complications and inquire if mothers wanted to stop or continue. In other words, they did not wait for the mothers to come to them. 22 10. All volunteers stated they received community support. The two most frequently mentioned means of support were respect and a high level of family planning acceptance. The second list has the findings from interviews with the Traditional Birth Attendants (TBA). 1. Fifteen TBAs were interviewed. They have been functioning as trained TBAs on an average of eight years. They received their most recent refresher course an average of three years ago. 2. All 15 TBAs were active, averaging six deliveries a month. 3. Eight of 15 provided antenatal care. 4. These eight saw an average of nine mothers a month for antenatal care. 5. The following table shows how TBAs reported disposing of placentas. Table 7: Methods TBAs Use to Dispose the Placenta. Method Number of TBAs Dispose in a pit latrine 9 Dispose in a placenta pit 2 Dig a hole and burry it 4 6. All TBAs were able to identify at least three emergency obstetric conditions without prompting. Between the 15 TBAs, all emergency conditions were mentioned. The following chart documents emergency complications that warrant referral to health centers according to TBAs. Chart 4: Emergency Obstetric Conditions Referred by TBAs. 13 9999 8 0 2 4 6 8 10 12 14 APH Grand multipara Deformity of the fetus Previous scar Prolonged labor Primi￾gravida 23 7. A wide variety of methods were used for transporting emergencies to the health center. The table below presents the findings. Table 8: Methods Used for Transporting Obstetric Patients Method Number of TBAs who have used the method Stretcher 10 Bicycle 6 Bicycle ambulance 5 Ambulance 5 Ox cart 5 8. Six of 15 TBAs stated they received feedback from the health center about patients whom they had referred. 9. All of the TBAs stated that the community supported them. The most common form of support was financial remuneration in cash and gifts, followed by respect and trust. Discussion Volunteers who distributed contraceptives were the best trained and most active of all the volunteer groups. It is impressive to note that all of the volunteers were active, knew the basics about reproductive health, served multiple villages and had neat and up-to-date records. They helped to make this the strongest and most productive community component of CHAPS. The task of CBD gives women in the village, especially those who have formal schooling, a productive and important job. It is one of the few openings for women to take leadership positions. In terms of strengthening administrative infrastructure, all of the CBD’s knew their job descriptions and all were supervised by HSAs. All but two had been supervised within the last month. The fact that the CBDs found a broad acceptance of family planning is an important finding. One would expect that in a setting as traditional as Chikwawa District that there would be resistance. None of the CBD volunteers however, reported rejection of contraceptive advice. In subsequent CHAPS projects it would be helpful to create a health information system to monitor certain key variables, such as contraceptive use, to document change. TBAs were also active and well integrated into the reproductive health system. As with other community leaders they were active and supported by the community. They had a good understanding about pregnancy complications and faithfully referred patients to the health center. Counter-referrals from the health center were a weak link. The documentation system at the health centers was practically nonexistent, thus the evaluation team was not able to verify the TBAs’ contention about the lack of referral information. CHAPS 24 could have ameliorated this by having a referral form with tear off sections and standard follow up instructions printed on it. The instructions for the TBAs could be done in picture form for those who have difficulty reading. While the TBAs did have solid information, it is also evident that there are needs for continuing education. According to the TBAs their last continuing education course occurred on average more than three years ago. The list of courses that CHAPS sponsored showed one five-day continuing education course, but did not give dates. Training IEF invested a large amount of time and money in training and refresher (“continuing education”) courses. As mentioned previously, IEF staff sponsored 33 training events with nearly 4,000 participants. (See Appendix A for a list of all the training courses.) One factor that complicated training for health professionals was the high turnover rate, requiring that IEF repeat training for the new staff. Nevertheless, the number of people trained at the community level is high. This is a case where it could be that more could be accomplished with less. In the future, IEF should reconsider its approach to continuing education. It seemed that some project staff had the impression that a training course alone would equip staff to do their job. There was no system for assessing learners’ knowledge and skills and then following up with a targeted training plan. The data from the interviews with TBAs is a good example. While nearly all of them could identify three or more risk factors in pregnancy, none of them knew all of them. Additionally, most of them did not know how to dispose of a placenta, even though they had been working as trained TBAs for over eight years. A systematic approach to training and continuing education would have identified these learning needs, and improved skills over time. IEF should move away from a formal “refresher course” approach and sponsor informal gatherings in the field. The encounter could focus on just one topic, placenta disposal, for example, and encourage learning through hands-on practice and exchanging volunteer experiences. It would be much less costly, especially in per diems, and it would facilitate more effective learning than a formal course. The need to develop curricula and document training courses is also important in order that the district has materials for subsequent training events. Villagers The findings from interviews with villagers are listed below. Sixty-seven villagers were interview in 13 villages. First, without prompting, villagers were asked what health services were provided in their community. The table on the following page presents a summary of these findings. 25 Table 9: Villagers Identification of Health Services in their Community. (N= 67) Health Service % Who Identified the Service Assistance with deliveries (TBAs) 85% Growth monitoring 79% Contraceptive use (CBDs) 61% Drug revolving fund 19% Home-based care 13% 1. Of the 85% who acknowledged the work of the TBAs, all of them stated that they assisted in deliveries, 24% stated that they made referrals, and 15% stated that they provided antenatal care. 2. Of the 79% who acknowledged the work of the GMVs, all of them knew their function was to weigh infants. Thirty one percent stated that they also assisted with immunizations. Villagers were also asked about referral services. 3. Villagers expressed a high level of satisfaction with health center services, with 90% stating that they were treated with respect. 4. Of those who went to the health center because of an illness, 87% stated that the last time that they went they were treated before noon. Only 13% stated that they had to wait the whole day. Discussion In general, the villages were highly aware of antenatal services. This is not surprising because IEF had a child survival project in Chikwawa before the CHAPS project. This finding provides additional evidence of the positive synergy between child survival and CHAPS. An effective child survival project contributed to the success of the community-level infrastructure of CHAPS. The level of villager satisfaction with health services registered by the villagers is difficult to assess because of people’s habit of being polite and avoiding confrontation. Additionally, it may be true that people do not know what to expect, or know anything different, so that they do not have a point of reference to determine service quality. The low number of people who identified home-based care for patients with HIV was partly a result of the fact that none of the villages had HBC volunteers. The HBC intervention was implemented in very few villages. This again raises the question, is this a result of the fact that not enough resources were invested in HBC, or that the prevalence of AIDS is low in Chikwawa? An important step in answering the question is to investigate the prevalence AIDS and invest resources accordingly. 26 Recommendations 1. In order for a decentralized system to work effectively, the district should have decision-making power on personnel and material resources, and have input into national policies. The decentralization in this CHAPS project did not go far enough, and as a result, the project was limited by significant factors over which it had no control. 2. Objectives and interventions for capacity-building projects should be limited to a short list of major topics. The majority of staff resources and time should be limited to these topics. It will mean that some important problems may not be addressed, but given the limitations and constraints of the Malawian health care system, it is more realistic to keep the list short and achieve high quality results. 3. CHAPS projects should provide an incentive for the District Health Officer to stay in the district for the life of the project. A PVO such as IEF could offer to pay for, or contribute towards, an MPH. The DHO could periodically do course work during the life of the project and receive a scholarship to finish at the end of the project. There is a risk in that the DHO could take advantage of this incentive and still leave the district without good leadership. This is a risk worth taking, however, given the alternatives. 4. Incentives to remain in their posts should be given to other key personnel such as those in reproductive health, HIV/AIDS and medical assistants in health centers. The evaluation team noted that the best run health centers were those where the medical assistants were at their posts for nine and ten years. There is the danger of getting stuck with a poor quality worker, but there are ways to deal with this problem. In poor districts such as Chikwawa and Nsanje the greater problem is one of good people leaving after a short tenure of service. 5. IEF should consider hiring project staff members that have professional degrees that are comparable to those of their MOH counterparts. Their effectiveness would be enhanced if they could interact with their counterparts on a professional level as well as on a technical level. This recommendation may be difficult to implement in Malawi due to the shortage of qualified health care professionals, but it should be attempted even if only a few of the counterpart role qualifications are matched exactly. 6. Job descriptions of district-level staff members need to take into account the fact that health care personnel will be pulled away to cover for staff shortages. It is better to expect that less will be done and have it actually accomplished rather than to expect an ideal that will not be achieved. In Chikwawa, job descriptions for some of the district-level staff were considered so unrealistic that they held little credibility. In the future it will be important to take into account external constraints such as staff shortages when writing job descriptions. 7. In the opinion of this evaluator, the district-level staff members who were interviewed expressed pessimistic but realistic assessments of CHAPS sustainability. USAID should consider a scaled down CHAPS II, especially given the external 27 constraints that affected the project in Chikwawa. CHAPS projects should have a phase out period, rather than terminating a fully functioning project abruptly at the end of the funding period. It is a matter of protecting the investment that has been made to date. Much of the benefits of CHAPS will dissipate over the next twelve months if there is no follow-up support. 8. The IEF Country Director should plan to be at the project site in Nsanje on a weekly basis. The distance from Blantyre and the condition of the road will complicate this recommendation, however, the Director’s constant presence is needed for quality control and to avoid management problems. 9. A quality control system for drug management needs to be in place at the health centers and community. An inventory and quality control system needs to be designed and implemented at the time that drugs are first distributed. This should help to ensure that essential drugs are available for people where they live. The control system would include an inventory slip that documents the quantity and date that the package of drugs left central supply and a signature from the MA acknowledging receipt of the package. In turn, when the DRF volunteers pick up their allotment, they should sign the same document signifying that the quantity received is accurate. Checking the DRF ledger for the quantity of drugs distributed and the remaining stock could verify this document. This would leave a paper trail for auditing purposes. 10. If community drug kits are used (DRF), the staff should provide monthly on-site continuing education and supervision, especially focused on those who have trouble with record keeping. 11. In the new project in Nsanje, if the pneumonia case management strategy is used, IEF and the MOH should consider training community volunteers who will only distribute pneumonia antibiotics, while putting the remainder of medicines under the drug revolving fund committee. This way these volunteers can concentrate on case management and they only have to be accountable for one medicine. 12. In future projects the “fun factor” should be recognized as an important incentive for volunteers. Events such as continuing education can have a social and personal value beyond their ostensive purpose. For example, an annual celebration with roasted goat, prizes and token gifts has been a very effective incentive in other projects. 13. In future project IEF should use referral forms with tear-off sections for health center staff and community volunteers. This will enhance communication, provide better patient care and strengthen the relationship between the health system and the community. 14. IEF should use a systematic approach to continuing education. It should include a process of pre-course assessments, implementing learning experiences, assessing its application in the field and following up with targeted learning experiences. It may be that the use of the word “refresher” course should be dropped. It communicates the idea that all the material is presented again. Additionally, the concept of formal workshops with costly stipends should be modified to include informal encounters 28 in the field where volunteers from neighboring areas meet for part of a day. The encounter could focus on just one topic, placenta disposal, for example, and encourage a learning environment of practice and exchange of knowledge and experience among the volunteers. 15. The MOH should reconsider using front-line antibiotics in the health centers. Under present circumstance it is most likely that the current use of tetracycline is creating resistance to this valuable drug. Summary The CHAPS project in Chikwawa District was successful in strengthening the accounting system, fleet management, communications between the district hospital and outlying health centers and the community-level capacity. Because of IEF’s work, Chikwawa District has a large cadre of active and committed community leaders. The major limitations of this project were due to constraints that were beyond the control of IEF and the district’s health care system. The constant staff rotation made it difficult to have continuity of leadership. In too many instances IEF had to start over again in building leadership capacity. The concept of a decentralized health care system embodied by the CHAPS projects needs to include the district’s participation in areas such as personnel management. 29 Evaluation Team Members Richard Crespo: Lead Evaluator George Mekiseni: Training/QA Advisor, IEF Tommy Mthepheya: Project Manager, IEF Frank Chola: Systems Advisor/Administrator, IEF Elton Chiuma: Reproductive Health Coordinator, IEF Paul Chibisa: Assistant Reproductive Health Coordinator, IEF Byson Chidzalo: Assistant Community Health Coordinator, IEF Lovemore Mvula: Food Security Coordinator, IEF Paul Chunga: Chikwawa District Environmental Health Officer Fellina Kaliati: District Nursing Officer, CDH Caster Bondo: Deputy DHO, Nsanje (Clinical Officer) George Chitimbe: District Environmental Health Officer, Nsanje Gwen O’Donnell: Child Survival/Vitamin A Coordinator, IEF HQ APPENDICES Appendix A: Training Status Report Appendix B: Questionnaires Used in Field Evaluation • CBD Volunteer Questionnaire • GMVs that Weigh Babies Questionnaire • GMVs that Give Medications Questionnaire • Outreach Center Questionnaire • Health Center Staff Questionnaire • HSA Questionnaire • VHC Questionnaire • TBA Volunteer Questionnaire • Villagers Questionnaire TRAINING STATUS CONDUCTED UNDER THE COMMUNITY HEALTH PARTNERSHIP PROJECT IN CHIKWAWA DISTRICT MARCH 1998 THROUGH SEPTMBER 2002 NATURE OF TRAINING # TRAINED NUMBER OF DAYS # REFRESHED NUMBER OF DAYS Training of Health Workers on Integrated Management of Childhood Illness (IMCI) case management 86 12 days 0 0 Training of Health Workers on Integrated Management of Childhood Illness (IMCI): - Facilitators - Clinical Instructors - Support staff 4 3 10 12 days 12 days 3 days Community Based Distributors of Contraceptives a) Volunteers b) Supervisors (primary) c) Secondary 48 10 17 10 days 15 days 5 days 1 118 38 17 5 days 5 days 5 days Traditional Birth Attendants (TBA) 2 112 5 days Orientation of Village Health Committee (VHC) on CBD activities 910 2 days Training of Growth Monitoring Volunteers 100 5 days Home Based Care -Volunteers -Supervisors -Traditional Healers) 57 10 100 12 days 12 days 3 days 3 28 0 5 days Core Family Planning Providers (FPP) 4 15 5 days Quality Assurance -Coaches -Teams (HC based) -Community based 13 120 20 10 days 10 days 10 days 7 5 days Community AIDS Committee 39 2 days Capacity Building for District Aids Coordinating Committee (DACC) 50 5 days Health Surveillance Assistants data collection (HAS) 20 5 days Primary Eye Care -Teachers -Traditional Healers -HSAs -Clinical and Nursing 108 195 54 17 2 days 2 days 5 days 5 days 100 2 days 1 Some CBDs were trained under IEF/STAFH Project but were refreshed under CHAPS Project 2 CHAPS Project only refreshed already existing TBAs 3 HBC volunteers refreshed were trained by IEF/STAFH Project 4 Core Family Planning Providers were trained under STAFH Project Management and supervision skills for Health staff 35 5 days DRF (Drug Revolving Fund) -Established -Supervisors 43 28 5 days 5 days 12 12 3 days 3 days Food Security -Biological Pest control -Permaculture -Seed preservation - Small livestock production - Soya utilization - Fruit and vegetable preservation 172 121 172 213 108 60 2 days 2 days 2 days 2 days 2 days 2 days Infection prevention 84 3 days Adult Literacy students 523 5 days Adult Literacy Committees 300 3 days Training of Community Development Assistant on Adult Literacy 10 3 days Training of Adult Literacy Instructors 40 21 days Training of Rural Information Assistants 40 5 days Life Saving Skills for Nurses (Safemotherhood) 13 5 days Information Education and Communication - Drama groups - IEC Coordinators - IEC strategic planning 22 15 13 5 days 5 days 21 days Health Information System (data collection) –HSAs 18 5 days Defensive Driving for the Drivers 10 2 days Training of Clinical Officer and Nurses on Norplant 2 CO and 1 Nurse 21 days Training of Nurses to manage Ngabu Rural Hospital Theatre Training of Trainer on STI 3 10 days Training of Trainers on EBF 2 10 days Prevention of Mother to Child Transmission of HIV/AIDS Health Workers Support staff 110 65 12 days 3 days Psychosocial Counseling 9 21 days Community Based Child Care -Caretakers -Committee 20 20 10 days 5 days APPENDIX B: Questionnaires Used in Field Evaluation CBD VOLUNTEER QUESTIONNAIRE Name of Village:_________________________________ 1. How many CBDs do you have in your community? __________________________________________________________________________________ 2. Are you aware of the roles of the CBDs? Yes No 3. What do you do? a) Provide FP methods b) Provide advice on FP issues c) Refer FP clients to the health facility d) Provide of condoms 4. Who supervise the CBDs? a) HSAs b) H.C. Personnel c) Don’t know d) Other _____________________________________________________________________________ 5. Do CBDs report to you about their activities? Yes No 6. Are any of the CBDs members of your VHC? Yes No 1 GMVs THAT WEIGH BABIES QUESTIONNAIRE Name of Village:_________________________________ 1.a) How did you become a GMV? _____________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ b) When were you trained as GMV? Month: Jan Feb Mar Apr Jun Jul Aug Sep Oct Nov Dec Year: 2.a) How many refresher courses have you attended as GMV? None Three One Four or more Two b) When did you attend the last GMV refresher course? Month: Jan Feb Mar Apr Jun Jul Aug Sep Oct Nov Dec Year: b) What services do you provide in your community as a GMV?______________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ c) What are the most important services provided to the community by GMV? _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 3. Who is your immediate supervisor?_________________________________________________ 4. When was the last time you were supervised? Month: Jan Feb Mar Apr Jun Jul Aug Sep Oct Nov Dec Year: 5. What support do you receive from the community? _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 2 6. What problems do you face when rendering services? District Lvl. Health Center Lvl. Community Lvl. 7. What are the major problems you face (list 3 only) _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 8. How did you solve those problems?___________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ THANK YOU VERY MUCH FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL. 1 GMVs THAT GIVE MEDICATIONS QUESTIONNAIRE Name of Village:_________________________________ 1. Do you have GMVs in your village? Yes No 2. If yes, do you have any members in your VHC? Yes No 3. How many are they? ____________________ 4. How many were trained in the last four years? ___________________________ 5. We have a mother whose child has diarrhoea. What advice would you give to this mother? i. Advice mother on dangers of dehydration ii. Demonstration on how to prepare ORS iii. Always mothers should 6. What materials do you need when you want to prepare ORS? (a) Water (b) ORS (c) Spoon (d) 1 liter container (e) Bucket a. Boiled a. Not boiled b. Not expired b. Expired c. Clean c. Not clean d. Clean d. Not clean e. Clean e. Not clean 7. How do you prepare ORS for diarrhoea management? (a.) Correct preparation of ORS (b.) Incorrect preparation of ORS 8. Did the GMV give any advice to mothers with diarrhoea children in the last month? Yes No 2 9. If the answer is NO, Did he/she advise anybody else before? Yes No 10. What advice should GMVs give to mothers who have children on EBF? a.) Children to be breastfed only for 4 to 6 months b.) Mothers to practice personal hygiene c.) Breastfeeding to continue even if the child has diarrhoea d.) Importance of colostrum. 11. Is it good to give children under 4 months water when it is very hot? Yes No 12. Who supervises the GMVs? a) HSAs b) VHCs c) H/C Staff d) Others ______________________________________________________________ 13. How often is this supervision done? _________________________________________ 14. What are other functions of GMVs in the village? a) Community mobilization for under five clinics b) Insist on growth monitoring c) Give advice on management of diarrhoea d) Promotion of EBF e) Other _________________________________________________________________ THANK YOU VERY MUCH FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL. OUTREACH CENTER QUESTIONNAIRE Name of Village:_________________________________ 1. What do you think is the main reason the outreach shelter in your community was constructed? a) To provide privacy b) To have a proper structure for the services c) To bring health center staff more frequently to the community d) Venue for meetings (e.g. VHC meetings) e) Gives prestige to the village f) Other:_______________________________________________________________ 2. What services are offered at the outreach centers? a) Family planning b) Growth monitoring c) Immunisations d) Other________________________________________________________________ 3. Who offers these services? a) HSAs b) HC staff c) District Hospital staff members d) Don’t know e) Other:_______________________________________________________________ THANK YOU VERY MUCH. THIS HAS BEEN VERY HELPFUL! QUESTIONS FOR HEALTH CENTER STAFF Name: _______________________ Job Title __________________________ Location of Health Center ____________________________________ 1. Please tell us what are your main duties? a. ____________________________________________________ b. ____________________________________________________ c. ____________________________________________________ d. ____________________________________________________ e. ____________________________________________________ 2. What training have you received? Course Mo. & Yr. Refresher 2nd refresher 3. Who supervises you? ______________________________________________ 4. When was the last time you were supervised? a. Month ________________ b. Year _______________ 5. Who do you supervise? 6. What services do you provide at your health center? After you have listed all the services, please rank their importance. 1 = the most important 2 = the second most important 3 = the next most important and so on. . . Services Rank Person Most Recent Date Topic 7. Please tell us whether these programs have been implemented in the communities in your catchment area a lot, some or a little bit. Put a check mark in the correct box. Program A lot Some Little Program A lot Some Little Reproductive Health Malaria IEC Maternal and Child Health (MCH) Safe Motherhood Traditional Birth Assistants (TBA) HIV/AIDS education Preventing Mother to Child Transmission Home Based Care [HBC] Community Based Distribution [CBD] Drug Revolving Fund (DRF) Growth Monitoring 8. We want to ask some more about the Drug Revolving Fund. Drugs that Have Been Most Available Drugs That Are Out Of Stock Right Now How Long Out of Stock 9. About how long does it take to restock your drugs from the District? ___________ 10. What suggestions do you have for improving the Drug Revolving Fund? a. _____________________________________________________________________ b. _____________________________________________________________________ c. _____________________________________________________________________ THANK YOU FOR YOUR TIME. THIS IS VERY HELPFUL! Questions for HSAs Name of H.S.A. __________________________ 1. When did you last meet with the VHC? __________________ 2. What have been the accomplishments of the VHC in the community? a. _______________________________________________________________ b. _______________________________________________________________ c. _______________________________________________________________ d. _______________________________________________________________ 3. When there is a health problem in the community whom do you discuss this with? Ask the question without reading the options. a. Headman/Chief ___ b. VHC ____ c. Medical Assistant ____ d. Political leaders ___ e. Senior HSAs _____ f. Other [specify] ___________________________________________________ 4. Tell us all the things that you do in the village. Duties Check if done Duties Check if done Supervision of VHC Supervision of GMV Supervision of CBDs Supervision of DRF Supervision of shelter construction Reporting of outbreaks Conducting health talk meetings Conducting under-5 clinics Sanplats casting Village inspection 5. What are the things that make your job difficult? List of Problems Big problem Medium Little Problem 6. Tell us about community participation in the construction of the outreach shelters. a. How many villages does your outreach shelter serve? ____ b. How many villages fully participated in the construction of the outreach shelter? ____ 7. What materials did the villages contribute? Materials Check if yes Materials Check if yes Bricks Sand Stones Manpower Construction site Water 8. What duties did you carry out in the construction of the outreach shelter? a. ______________________________________________________________ b. ______________________________________________________________ c. ______________________________________________________________ d. ______________________________________________________________ 9. What problems did you have in the construction of the outreach shelter? List of Problems Big problem Medium Little Problem 10. What are the important things that the health shelter has done for the community? a. Ask the H.S.A. to tell you everything that he can think of. b. After he has finished, repeat all his answers. c. Then ask him to rank the accomplishments – which is the most important of all = 1 – which is the next important = 2 – next important = 3 – and so on Accomplishments Rank 11. What can be done at the community level to improve the health status of the community? a. ________________________________________________________________ b. ________________________________________________________________ c. ________________________________________________________________ d. ________________________________________________________________ THANK YOU FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL. Questions for VHC Name of Village ______________________________ 1. When was your VHC formed? ________________________ 2. How many members are present in your VHC? _____ 3. Do you have monthly meetings? a. Yes _____ b. No _____ 4. If yes, when was the last time that you met? ______________________ 5. What was the topic? ____________________________________________________ 6. When was the last time that the H.S.A. attended the meeting? _________________ 7. Do you keep minutes of your meetings? a. Yes ___ If yes, can we see them? [check if there were minutes ____ ] b. No ___ 8. How many households have pit latrines in your village? [information] a. In most of the homes ___ b. In about half ___ c. In very few, or none __ 9. Is there a borehole in your village? [Information] a. Yes ___ b. No ___ 10. When was the last time that you met with the entire village? ______________ 11. What health issues did you talk about at that meeting? a. ______________________________________________________ b. ______________________________________________________ c. ______________________________________________________ d. ______________________________________________________ 12. About how many people attended the meeting? _____________ 13. What are the main reasons that the outreach shelter was constructed in your village? [judgement] Ask the group to give you all the reasons that they can think of. Make a list of their reasons on the flip chart. Ask each person to vote for the reasons that they think are the most important. Each person has three votes. 14. Do you have GMVs in your village? a. Yes _______ b. No __________ 15. If yes, who supervises the GMVs? [information] a. HSAs ___ b. VHC ___ c. Health Center staff __ d. Others __ 16. When was the last time that the GMVs reported to you? ________________ 17. Do they do this many times or a few times? a. Many times ____ b. A few times ____ 18. What are the important things that the GMVs do in the village? [judgement] Ask the group to give you all the reasons that they can think of. Make a list of their reasons on the flip chart. Ask each person to vote for the reasons that they think are the most important. Each person has three votes. 1 TBA VOLUNTEER QUESTIONNAIRE Name of Village:_________________________________ 1. When were you trained as a TBA? Month: Jan Feb Mar Apr Jun Jul Aug Sep Oct Nov Dec Year: 2. When did you go for a refresher course? Month: Jan Feb Mar Apr Jun Jul Aug Sep Oct Nov Dec Year: 3. What services do you offer to the community? __________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 4. On average, how many deliveries do you conduct in a month? _____________________________ 5. On average, how many mothers per month do you provide antenatal services to? _____________________________________________________________________________________ 6. How do you dispose of placentas? Placenta pit Take it home Pit latrine Other (specify):____________________________________________________________________ 7. What emergency obstetrics conditions do you know? _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 8. What conditions or problems should you refer to the H/C? Previous scar Leg or spine deformity Ante Partum Hemmorage (APH) Primigravida Grandamultipara + four deliveries Prolonged labor 9. When there is an emergency, what is the most common means of transport? Stretcher (Machila) Push bicycle Bicycle ambulance Ambulance Others (specify)____________________________________________________________________ 2 10. Do you get feedback after referral? a) Yes ⇒ b) HOW? ______________________________________________________________ _____________________________________________________________________________________ c.) No ⇒ d) WHY NOT? _________________________________________________________ _____________________________________________________________________________________ 11. Who supervises you? HSAs Nurse Other(specify):________________________________________________________________ 12. How often are you being supervised? Every fortnight Monthly Every four months Other (specify):__________________________________________ 13. What support do you get from the health facility? Supplies Supervision None 14. How best can you be supervised? _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 15. What support do you get from the community? _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 16. What problems do you face when rendering your services to the community? LIST PROBLEM District Lvl. Health Center Lvl. Community Lvl. THANK YOU VERY MUCH FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL! 1 VILLAGERS QUESTIONNAIRE- Individual Questions Name of Village:_________________________________ 1. WHAT SERVICES HAVE YOU RECEIVED FROM THE VOLUNTEER(S) IN THIS VILLAGE? (Note: do NOT read the following options, rather check off responses.) i. GMV Weighing children Advise/Counseling on corrective action Immunization Social time/community involvement Education on child health Referral(s) Other: _________________________________________________________________________ ii. HBC Drug supply Food distribution Patient care House cleaning Counseling Drawing water for the patient Referral(s) Other: _________________________________________________________________________ iii. CBD Distribution of FP Group education on family planning Referral(s) Other: _________________________________________________________________________ iv. DRF Easy access to drugs Advice on medication/illness Referral(s) Other: _________________________________________________________________________ V. TBA Delivery assistance Antenatal care Iron supplementation 2 Referral(s) Other: _________________________________________________________________________ 2. HAVE YOU EVER BEEN REFERRED BY A VOLUNTEER TO A HEALTH FACILITY (CLINIC OR HOSPITAL)? Yes No 3. IF YES, FOR WHAT REASON(S)? Underweight child Physical exam(s) Sickness Infections Investigations Treatment Pregnancy complications Other Complications Antenatal care Other: _________________________________________________________________________ NOW I WOULD LIKE TO LEARN ABOUT YOUR EXPERIENCE AT THE HEALTH FACILITY. 1. WERE YOU GIVEN PRIORITY WHEN YOU ARRIVED AT THE HEALTH FACILITY WITH A REFERRAL? Yes No 2. IF NO, HOW LONG WERE YOU MADE TO WAIT? 1. The whole morning (approximately until 10 or 11am) 2. Till noon (approximately until 12 or 1 pm) 3. The whole day (approximately until 3-6 pm) 3. WERE THE SERVICES THAT YOU RECEIVED AT THE HEALTH FACILITY: 1. Good 2. Okay 3. Poor 3 THANK YOU VERY MUCH FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL! 4 VILLAGES QUESTIONNAIRE- Group Questions Name of Village:_________________________________ 1. WHY DO YOU PERSONALLY FEEL THAT THE VOLUNTEERS’ SERVICES ARE IMPORTANT TO YOUR VILLAGE/COMMUNITY? WHAT BENEFITS DO THEY BRING TO YOUR LIFE? (Listing & voting) 2. a. HOW DOES THIS COMMUNITY SUPPORT THE VOLUNTEER TO CARRY OUT HIS/HER DUTIES? (Listing) b. WHAT RECOMMENDATIONS DO YOU HAVE FOR YOUR NEIGHBORS TO HELP THE VOLUNTEERS TO DO THEIR JOB MORE EFFICIENTLY? (Listing & voting) Note: After the list is done, ask each person to put ONE check after the help they are willing to give. 3. A PART FROM ASSISTING HIM/HER TO CARRY OUT DUTIES, WHAT OTHER INCENTIVES (ZINTHU ZOWALI MBIKISTA) DOES THE COMMUNITY GIVE VOLUNTEERS? (Listing) THANK YOU VERY MUCH FOR YOUR TIME. THIS HAS BEEN VERY HELPFUL!