Ph--flfw-~ -+y ESPERANCA PERU FINAL CHILD SURVIVAL EVALUATION REPORT Cooperative Agreement Number: FAO-A-00-97-00058-00 Written by: Richard Crespo, Ph.D. Marshall University School of Medicine August 2001 TABLE OF CONTENTS A . EXECUTIVE SUMMARY ................................................................................................................. 3 1 . PROGRAM OBJECTIVES ......................................................................................................................... 3 2 . MAIN ACCOMPLISHMENTS .................................................................................................................... 4 3 . BASELINE COMPARED TO FINAL INDICATORS .......; ........................................................................... 5 4 . PRIORITY CONCLUSIONS ....................................................................................................................... 7 B . ASSESSMENT OF RESULTS AND IMPACT ............................................................................ 7 1 . SUMMARY RESULT OF MEASURABLE INDICATORS .............................................................................. 7 2 . RESULTS: TECHNICAL APPROACH ....................................... .............................................................. 10 A) Results in Diarrhea Control ........................................................................................................ I1 B) Results in Breastfeeding ............................................................................................................. 12 C) Results in Nutrition .................................................................................................................. 12 D) Results in Immunization .............................................................................................................. 13 E) Innovative Approaches ............................................................................................................... 13 3 . RESULTS: CROSS-CUTTING APPROACHES ......................................................................................... 14 A) Community Mobilization ............................................................................................................. 14 B) Conzrnunication for Behavior Change ...................................................................................... 15 C) Capacity Building Approach ....................................................................................................... 15 D) Sustainability Strategy ................................................................................................................ 19 C . PROGRAM MANAGEMENT ........................................................................................................ 20 1 . PLANNING ............................................................................................................................................ 20 2 . STAFF TRAINING ................................................................................................................................. 21 3 . SUPERVISION OF PROGRAM STAFF.. ................................................................................................... 21 4. HUMAN RESOURCES AND STAFF MANAGEMENT ............................................................................... -22 5 . FINANCIAL MANAGEMENT .................................................................................................................. 22 6 . LOGISTICS ........................................................................................................................................... 22 7 . INFORMATION MANAGEMENT ............................................................................................................ 22 8 . TECHNICAL ASSISTANCE AND ADMINISTRATIVE SUPPORT ................................................................ 23 9 . MANAGEMENT LESSONS LEARNED .................................................................................................... 23 D . OTHlER ISSUES ............................................................................................................................ 24 E . CONCLUSIONS AND RECOMMENDATIONS .......................................................................... 24 PRIORITY RECOMMENDATIONS .... , ....................................................................................................... 24 APPENDIX 1 : METHODS USED .......................................................................................................... 27 APPENDIX 2: EVALUATION TEAM ................................................................................................... 28 A. Executive Summary CHARACTERIZATION OF THE PROJECT ZONE This project is located in the Southern highland of Peru, in the Department of Apurimac, Cotabambas Province. The communities served by the project are located between 10,000 and 15,000 feet above sea level. They are draped across Andean mountain peaks and require hours of hiking to traverse from one to the other. The people are Quechua speakers; In fact the vast majority of women only speak Quechua. All of the project's field staff are native Quechua speakers. Access to healthcare. The Esperansa project zone in Cotabambas Province is characterized by extremely difficult circumstances for geographical and cultural access to Ministry of Health (MINSA) services. Geography of the zone. The intervention area encompasses 130 Quechua￾speaking communities located 8-14 hours from Cusco by car, plus an additional walk of 0-12 hours beyond the road. During the rainy season, road access is significantly limited due to landslides and high rivers. MINSA health services. Health services in the project zone suffer from infrastructure and personnel shortages due to inaccessibility and a relatively low level of financial investment from MINSA. For example, half of the health establishments in the project zone do not have any means of outside communication, such as a radio or telephone line. Half are staffed only by health technicians (non-professional health personnel), and there is no surgical capacity anywhere in the project zone. Difficult working conditions lead to constant staff turnover, and especially a difficulty in retaining experienced Quechua-speaking health personnel. Culture and health practices. A recent survey in the project communities found that over 70% of births are attended in the home by the husband or other family members. 85% of mothers reported that they are exclusively Quechua speakers, and 80% say that they do not read or write Spanish or Quechua. 1. Proaram Objectives A) PROJECT GOALS AND ACTIVITIES Since 1998, the Esperanra Child Survival project has worked with community health promoters, families and the Ministry of Health (MINSA) on education and technical skills in four areas of intervention: pneumonia case management (PCM), diarrheal disease control, nutrition/growth monitoring, and maternal health/obstetrical emergency management. The project has four child survival interventions: The specific objectives for each intervention are listed in Table 1, page 6. The overall goals for the project were to I) Reduce infant, child, and maternal morbidity/mortality in the project zone; 2) Faciiitate sustainable improvements in the quality of health services in the project zone; and, 3) Strengthen the capacity of the partner organization CADEP for health programming. PROJECT HIGHLIGHTS: 1. Monthly training of 130 health promoters, formation of health committees at each health post, and a local health committee in each community. Support for routine training of traditional birth attendants. 2. Health education of families in 130 communities to reduce child morbidity and mortality. 3. Implementation of a pictorially-based referral system for health promoters and traditional birth attendants. Health promoters manage patients' follow-up care in the community. 4. Development of IEC (Information, Education and Communication) materials: health promoter flip-charts for each program, a health promoter manual, greenhouse manual, radio spots, sociodramas, stretchers, community medicine chests. 2. Main Accomplishments Community-level health knowledge 1. Mothers have a strong sense of change in community health over the last four years. They identified maternal-child health insurance and better maternal-chiid health as the most important change. 2. Mothers can identify the main sign of pneumonia. Data from the final survey found that 73% could state the sign of rapid breathing at rest. Almost all mothers answered immediately with no hesitation. 3. Mothers know the signs of dehydration from diarrhea. Data from the final survey show that 64% of the mothers could identify at least one sign. 4. Mothers giving solids to children six months or older during incidents of diarrhea and mothers knowledge of the sign for pneumonia were two interventions that attained both a substantial change from the baseline and had a high level of adoption at the end of the project. (Charts 1 and 2, pgs. 7 and 8). 5. Mothers know the signs of risk during delivery. Data from the final survey found that 68% of the mothers knew at least one sign. 6. Mothers know the meaning of the infant growth chart, Interviews with 189 mothers showed that 88% knew the meaning of the green zone, 84% knew the meaning of the yellow zone and 79% knew the meaning of the red zone. Community health promotion 1. Health promoters have been selected and trained in all 130 project communities in the areas of pneumonia control, nutrition, diarrhea control and maternal health. According to the final survey, health promoters are active and are recognized for their labors both by their communities and the Ministry of Health (MINSA). Mothers who were interviewed during the field survey for this report gave them a rating of on a 3-point scale for quality of their work. 2. MINSA also gives high marks for the promoters' performance. They gave them a rating of 2.9 on a four-point scale. 2. The community substantiates the promoters' perception of their role. When asked about the most important things that the promoters do, mothers and community leaders all identified the same top three activities of the promoters: home visits, health talks, making referrals to the health center. The health promoters are truly active in their communities. * 3.Cornmunity support is broad-based. All 125 communities have health committees. Most of them have been working on birthing centers and creating plans for emergency transport of patients to the health center. Baseline Compared to Final Indicators Please refer to Table 1, on the next page. A discussion of the indicators will be made in the corresponding sections of this report. Table 1: Baseline Indicators Compared to final Objective 1. Children from 0-6 months will receive exclusive breastfeeding. Baseline 59% 2. Children from 6 to 24 months will eat solid food, complimented by breast milk. 40% 3. Children less than 2 years of age will receive monthly growth monitoring, as documented by their growth charts. 37% 4. Children 0 to 6 months who have an episode of diarrhea, will receive more breast milk than usual. 79% 5. Children 6 to 24 months old who have an episode of diarrhea will receive more liquids, including breast milk and ORS, than usual. 54% 6. Children 6 to 24 months old who have an episode of diarrhea will receive more food than usual for 2 weeks after the episode. 22% 7. Mothers with children less than 2 years of age will be able to recognize two signs of diarrhea. 21 % 1 8. Mothers with children less than 2 years who have diarrhea danger signs will promptly seek appropriate care. 1 91 % 9. Mothers with children under 2 will recognize one sign of pneumonia. 10. Mothers with children under 2, who have signs of pneumonia, will promptly seek appropriate care and comply with SCM. 16% 79% 11. Communities will have an emergency transport plan to promptly take sick children to an appropriate care provider when indicated. NA ! 12. Pregnant mothers will have 2 doses of tetanus toxoid vaccine document by their maternal health card. 13. Pregnant mothers will have at least two prenatal care visits documented on their maternal health card. 62% 63% 14. Women of childbearing age will be able to recognize 3 signs of obstetrical emergencies. N A 15. Communities will have an emergency transport plan to promptly take women with obstetrical emergencies to appropriate care. N A 2. Prioritv Conclusions 1. It is possible for change in knowledge and behavior to occur in traditional, illiterate societies in a relatively short time. 2. Substantial progress has been made since the mid-term. In large part it is because the Esperanza staff trained and worked through the promoters instead of working directly with mothers. 3. The consistent perception of the promoters' role among mothers and community leaders gives evidence of the quality of their work. 4. The level of support: from MINSA appears to be superficial and is not likely to be sustained unless there is strong leadership from the new DIRESA Director. 5. Assessment of Results and Impact 1. Summarv Result of Measurable Indicators The project showed an improvement from the baseline to the end in seven of its measurable indicators. The following table presents the indicators and the amount of change in percentage points. Chart 1: Indicators that achieved improvement from the baseline Pregnant mothers with 2 TT doses Exclusive breastfeeding Prenatal visits Mothers know signs of dehydration Growth monitoring Mothers know sign of pneumonia Children from 6 to 24 mo. eat solids & breast milk. 0 10 20 30 40 50 60 70 Percentage Point Improvement These indicators give evidence of substantial improvement in the health status of mothers and their children. It is especially important to note the amount of change in mother's ability to recognize signs of dehydration and pneumonia, the two greatest causes of infant morbidity and mortality. Another way to obtain a perspective on the achievement of the project is to note that the project attained a level of at least 50% in knowledge or practice of preventive behaviors in all but two of its indicators. This is evidence of a broad foundation for improving the health status of mothers and their children. The areas in which 70% or more of the mothers had the intended knowledge and skills are presented in the following chart. > Chart 1: Indicators that had the highest achievement. Exclusive breastfeeding Mothers know sign of pneumonia 73% i Growth monitoring 82% Emergency transport Children from 6-24 mo. lave solids & breast milk Prenatal visits The fact that 100% of the mothers who had their maternal health card went to the health center for prenatal visits is a major accomplishment. One must note that for over half of the villages it is more than a four-hour walk to the health center. For the most isolated villages it is up to a 12 hour walk. This also shows the Quechua population has grown in its acceptance of the government health centers. Another important achievement is the high percent of mothers who exclusively breastfeed their infants up to six months of age. The tradition has been to supplement breastfeed after a couple of months to make the babies fatter. The fact that so many mothers now accept exclusive breastfeed is a significant change and will contribute to reducing infant morbidity. According to the data in Table 1, page 5, there has only been a 14-percentage point difference between the baseline and final survey in exclusive breastfeeding. It should be noted that at the mid-term evaluation the team questioned the reliability of the baseline data. The field staff believed that mothers did not fully understand the questions because of language difficulties and their lack of exposure to this kind of questioning. Thus the significance of the change from the baseline is possibly understated. Combining data from Charts I and 2, above, it is noteworthy that two interventions attained both a substantial change from the baseline and had a high level of adoption at the end of the project. These were giving solids to children six months or older and knowledge of the sign for pneumonia. Regarding the first intervention, it should be noted that while there had been a tradition of giving teas and infusions to infants, most mothers (60% according to the baseline) introduced solids late during the child's development. The extent of change in nutrition for young children and early treatment for signs of pneumonia should make a significant difference in the morbidity and mortality of children in this province. In some of the interventions there has been a decrease in adoption of some indicators from the baseline to the end. These are presented in the following table. Table 2: Interventions in which there was a decrease between baseline to final survey. Objective Child. 6-24 mo. with diarrhea more liquids Baseline 54% 1 Child. 6-24 mo. with diarrhea more solids 15% Mothers seek care for pneumonia 1 Of the five interventions in the above table, three have to do with treatment in cases of diarrhea. A further analysis of the date indicates that this table does not present the full picture. When the questions are expanded to include both the variables "the same amount of liquids" and 'more liquids" the data are represented as follows. Final 50% 22% -7 79% 64% Child. 0-6 months with diarrhea, more breast milk Change (points) -4 -1 5 Mothers seek help child diarrhea -38 79% 91 % 53% 19% -60 Table 3: Percent of mothers who give the same amount or more liquids and solids during incidences of diarrhea. Objective Final Child. 6-24 mo. with diarrhea the same or more liquids Child. 6-24 mo. with diarrhea the same or more solids Child. 0-6 months with diarrhea, the same or more breast milk 8 6 O/o It is especially important to note that 85% of infants 0 to 6 months old receive the same or more breast milk during an episode of diarrhea. Thus, the data in Tables 1 and 2 misrepresent mothers' care of their children during incidences of diarrhea. Table 3 gives a more accurate picture. The other two areas that showed a decrease between the baseline and final surveys were help-seeking behavior for diarrhea and pneumonia. One factor that could possible affect the decrease in seeking help in cases of childhood diarrhea is the mothers' increased confidence and ability to address diarrheal disease at home, with the presence of the health promoter. Thus the decrease in seeking medical help for diarrhea from a distant health post can be regarded as a positive. 2. Results: Technical A~~roach This project is located in the Southern highland of PerQ, in the Department of Apurimac, Cotabambas Province. The 125 communities served by the project are located between 10,000 and 15,000 feet. They are draped across Andean mountain peaks and require hours of hiking to traverse from one to the other. The people are Quechua speakers; In fact the vast majority of women only speak Quechua. All of the project's field staff are native speakers. There are no paved roads in the province and communication is extremely limited. The town where the project is based has one community telephone. The vast majority of villages are not accessible by road, and most car access is curtailed during the rainy season. Generally, the project vehicles bring the field staff to drop-off points and they walk from there. It is important to note that four years ago Cotabambas had the highest infant mortality rate (118 per 1,000 live births) and maternal mortality rate (451 per 100,000) in Peru. It also had the lowest child development index, and the highest percentage (98%) of children with basic needs unmet, in the nation. (Source: Peru National Institute of Statistics and Information, 1996) Local MOH officials reported that the major causes of under-five mortality in the zone were pneumonia and diarrhea. Causes of maternal mortality were hemorrhage, gestational hypertension, infection, and trauma. The two broad project strategies were: 1) to improve the coverage and quality of health services at all levels by training MOH staff and community volunteer Promoters; and, 2) to improve the home care of children, the recognition of danger signslemergencies, and prompt seeking of appropriate care by training caretakers and communities. These strategies were to be addressed by using two main lines of action: training and communication. The project team used the Behavior Change Communication model as presented in Communication for Health and Behavior Change. The focus was on observable behavior and skills training. The plan was to follow these five steps: (1) assessing behavior, (2) planning for behavior change, (3) ski I Is training, (4) monitoring behavior change, and (5) maintaining health practices. A project director, one field supervisor, and seven field staff led the project. The field staff trained and worked alongside 15-20 volunteer health promoters each. 6) RESULTS INDIARRHEA CONTROL The impotTant findings in this intervention were as follows. The incidence of diarrhea in the KPC survey, in children under 2 years old, was 30%. 75% of mothers gave the same or more liquids when their children had diarrhea. 86% of mothers gave the same or more breast milk when their infants had diarrhea. 50% of mothers gave the same or more solids when their children had diarrhea. 94% of the MINSA staff stated that their coverage of cases of diarrhea increased because of the work of the promoters. (n=16) 89% of the promoters knew all of the signs of diarrhea in children under two. (n=18) The promoters who were observed by the evaluation team giving health talks on diarrhea control did so with complete accuracy. The educational materials on diarrhea prevention and management were based on culturally appropriate drawings which were well suited for this population. The staff team gave a high priority to this intervention because of the extent of the problem. At the baseline the incidence was 38%. Anecdotal evidence from interviews with promoters indicates that while cases of diarrhea in children still occur frequently, the mothers know how to manage it and they rarely need to refer a child to the health center because of dehydration. MINSA staff states that they rarely see cases of dehydration from the communities in the project-and findings from the final survey support this. These statements are supported by the large percentage of mothers who give more liquids in cases of diarrhea. Additionally, there was an increase in the percent of mothers who could give at least two sign of dehydration from 21% to 64%. While the Rnal percent is lower than one would want, it is important to recognize that mothers did achieve a 43-percentage point increase. The relatively low percent of mothers who give the same or more solids indicates a need for continued education in this area. MINSA staff and the promoters should continue to help mothers understand the importance of solids in managing diarrhea. The Key finding for this intervention was as follows. 1. 73% of the mothers practiced exclusive breastfeeding. 2. 99% of mothers stated that they gave introduced solids (in addition to breast milk) to children when they were six months and older. It is especially encouraging to know that mothers understand how to manage the transition from breastfeeding to solids and at what age to do so. Progress begins with the fact that 73% of mothers practice exclusive breastfeeding. While the baseline survey results stated that 59% practice it, the field staff believe that mothers did not understand the question, and that exclusive breastfeeding was overestimated in the baseline survey. In their experience, most mothers gave water or teas to their infants by 3 months of age. Thus the field staff should be very proud of the fact that 73% now practice this behavior. When the project began there was a wide range of practices regarding the transition from breastfeeding to giving solid foods. It was a common practice to give solids to infants in their third month in order to make them fat and beautiful. On the other end of the spectrum there were numerous mothers who delayed giving solids until eight to twelve months of age. They primarily gave breast milk to their infants. The fact that 99% of the mothers knew how to mange this transition has been a major change. In this project problems related to the knowledge and practice of breastfeeding and of when and how to give solids were intrinsically linked. Dl RESULTS IN NUTRITION 1. 88% of the mothers in the field survey (n=189) were able to state the meaning of a child in the green area of the growth chart, 84% the meaning of the yellow area, and 79% of the red area. 3. No growth chart that was reviewed in the field survey showed a child in the danger area. Additionally, no child stayed in the "warning" or yellow area of the growth chart for more than one weighing period. 4. 82% of the children 5 months or older had been weighed at least once in the last four months (up from 37% at baseline). 5. The community leaders who were interviewed (n=17) and the promoters (n=18) stated that one of the important ways that they supported each other was in distribution of the child food supplement to prevent child malnutrition. 6. 89% of the 18 promoters who were interviewed in the field survey could identify the three food groups without any prompting. Teaching on nutrition was one of the last interventions that the project worked on. This makes sense given the extent of the problems in diarrhea and respiratory infections. Good progress was made in this area despite the relatively short time that was dedicated to it. The data on growth monitoring are very encouraging. Given the circumstances of geography and climate it, these data represent: good progress. It is especially note worthy that the mothers understand the dangers of underweight babies and have done a good job in weight management. This finding is linked section on breastfeeding and the transition to with the observations made earlier in this giving solids. The important findings regarding this intervention are as follows. 1. 85% of the mothers in the KPC survey had their child's vaccination card available. 2. 76% of the children 12 to 24 months old were fully immunized. 3. All seven of the mothers who were pregnant during the KPC survey had their prenatal cards. 4. All seven of these mothers had at least one tetanus toxoid vaccination. 5. 38% of all mothers in the KPC survey still had their prenatal cards available to demonstrate to the interviewers. 6. 69% of the mothers who had a prenatal card showed evidence of at least two tetanus toxoid vaccinations. Improvement was clearly seen in maternal immunizations. At the baseline 16% of the mothers had their maternal health card, while 38% did so at the final. Sixty nine percent of these mothers had at least two tetanus toxoid vaccinations. More importantly, all seven of the mothers who were pregnant at the time of the final survey had their cards and had at least one vaccination. While this number is small, this is a better indicator, since many mothers discard their maternal health cards when their children reach one year old. The child immunization rate at the end of the project was satisfactory; however, it is difficult to give a fully accurate assessment of this intervention because of two intervening factors. One factor is that the distance most mothers must walk to get to the health center represents a major barrier to following the prescribed schedute. It is very common that a child will be slightly behind schedule on vaccinations because families try to time visits to the health post with other activities they may need to conduct outside of their community. The second factor is the politics surrounding immunization rates. The project staff were not able to report the low immunization rate from the baseline KPC because the provincial health authorities stated that according to their records the rate was 90%. Thus while the vaccination rate of 76% may be modest in comparison to other CS projects, it is likely that the increase from the baseline was substantial. One indicator of the improvement is that the percent of mothers with a child vaccination card rose from 42% at baseline to 85% at the final KPC survey. F) INNOVATIVE APPROACHES One innovative approach was the project's use of drawings to communicate health messages. Initially health teaching was in lecture format and communicated by the project's field staff. After the mid-term evaluation the field staff created drawings of the health messages on cloth flip charts and taught the health promoters how to use them. These were a big attraction, as almost no graphics existed .in the communities except for consumer items. The flip chart on cloth instead of paper made them easy to carry and were more durable than paper. This gave the promoters a great incentive for doing home visit and community health talks. In many communities the promoters were invited to give talks as part of community meetings. During the field evaluation, promoters were asked to give a demonstration and in every case they communicated the health messages accurately and in an engaging manner. Another innovation for this province was the construction of maternal waiting houses (Casas de Espera). These are Quechua-style houses built next to the local health center where mothers and their family can stay during the last two weeks of the pregnancy. This is a concept that is well suited for an area like this where more than half of the villages are more than a six hour walk from a health center and there are almost no roads. At the time of the final field evaluation the majority of health centers either had a maternal waiting house or one was being planned. A critical feature of these centers is that the community constructs them with support from the health center, on land donated by the municipality. Shared ownership will contribute significantly to the sustainability of this innovation. One of the main goals of the project was to help communities organize emergency evacuation plans in cases of a medical emergency. A creative technique for stimulating community organization was that the project sponsored a stretcher making competition with the Health Promoter Committee at each of the health centers. The committees took the challenge seriously and the competitions had a festive air. As a result most of the villages now have defined their evacuation procedures and have stretchers. 3. Results: Cross-cutting amroaches A) COMMUNITY MOBILIZATION One of the obstacles inherent in this project is the rural, isolated location of the target communities, among the most challenging places to access in Peru. Despite this difficulty, the field staff did a commendable job in maintaining a regular pattern of visiting most villages on a monthly basis, For the villages requiring more than a 10-hour walk, a goal was set that these would be visited at least six times a year. Due to regular contact with each community, and monthly contact with each health promoter, the project has been able to assist in the formation and training of local health communities in all 130 villages. The primary task of the committees has been to recognize the importance of community organization to react immediately to live-threatening emergencies, particularly for during births and the post-partum period. The field staff made substantial progress in this area since the mid-term evaluation, when there were only three health committees. Another achievement in community mobilization has been in involving community leaders in the supporting community-level health promotion activities. During the final field survey, the evaluation team interviewed 17 community leaders. The key findings are as follows. 1. They see themselves as active participants and have a high sense of ownership of the promoter-based health system, in contrast to regarding MINSA as having control. Eighty eight percent of the community leaders stated that either they alone, or in conjunction with MINSA, participate in the naming of new promoters. This is a major change from the beqinninq of the project. 2. They have a high regard for the work of the health promoters, giving them a rating of 2.2 on a 3-point scale (1 being low quality of work, and 3 high). 3. They give strong support for distribution of enriched oatmeal for mothers with children under two years of age and for health talks. The promoters who identified these as the areas in which community leaders provided the greatest support independently confirmed this. Bl COMMUNICATION FOR BEHAVIOR CHANGE Project activities in this area involved extensive baseline qualitative assessments in maternal health, nutrition, diarrheal disease and respiratory illness. This information served as the base for the development of an information, education and communication plan including: A training plan for health promoters in the four primary intervention topics Sociodramas (skits) performed by health promoters, project staff, and MINSA health personnel Radio spots and dramas broadcast by radio (which only reached a percentage of communities) and taped as part of community health education campaigns A health promoters work manual, using culturally appropriate images and health practices A pictorially-based community reference system for health promoters and traditional birth attendants, for reference and counter-reference from the health post. A culturally-appropriate greenhouse manual and training program. A community notification system, with pictorially-based birth and death registry. One of the most important changes in the project from the mid-term the shift from project staff conducting the majority of community health education to training the health promoters to communicate health messages. Once given the responsibility to deliver health messages, the promoters were able to reach many more people and do so more frequently. The results are evident in the high percentage of mothers who know the basics of diarrhea and pneumonia control, family planning methods and growth monitoring. As mentioned in the section on innovative approaches, the use of drawings on cloth flip charts was a communication tool that was very well received. If Esperanca has the opportunity to continue working in this province, this evaluator recommends that they integrate literacy training with the communication of health messages. This strategy was made popular by Paulo Freire and has been used extensively in other parts of Latin America. Esperanca should also increase their use of stories, drama and parables to communicate health messages, instead of a dry recitation of facts. Along with these techniques they should write discussion guides so that the listeners have the opportunity to discuss and analyze the health messages for themselves. Developing critical thinking skills will make an important contribution to development in this region. CI CAPACITY BUILDING APPROACH i. Strenuthenina PVO Esperanca intended to work in partnership with CADEP, a local NGO. They did not have a prior working relationship, and it showed as over the four years of this project the two organizations have struggled to define their roles, with only a little success. Three factors played a predominant role in this struggle. One was the fact that CADEP is primarily an 15 agricultural agency, with little experience in health care or child survival. Esperanca underestimated CADEP1s learning curve, and CADEP assumed that they knew more about maternal-child health than they did. A second factor was the revolving leadership in CADEP until 2000. With an average of one director for each year of the project, it was almost impossible to implement a consistent relationship. A third factor was that the CS project was implemented in an area where CADEP was not originally working. Without a presence in the area it was impossible to work together. Despite the factors that limited the strengthening of CADEP, a positive relationship has developed between the two organizations. CADEP has conducted an extensive evaluation of its institutional capacity, and now has a health division, with short and long-term goals. A very encouraging postscript is that as this CS project was finishing, CADEP had received a grant to implement literacy training in Cotabambas Province. Esperan~a and CADEP finally had a true opportunity to work together and were doing so. Esperanqa field staff were working hard to help CADEP get started. If Esperan~a is able to obtain continuation funding, they could have a strong working relationship in the future. Also, it is important to note that despite the difTiculties, CADEP has consistently provided valuable logistical and administrative support to the project. Staff of both organizations have been open with each other in discussing issues, which has set the stage for a partnership that will continue as Esperanqa implements activities with funding received from the USAID mission in Lima and seeks ways to continue working in Cotabambas. ii. Sfrenathenina Local Partners The local partner in this project was the Ministry of Health (MINSA) at the regional, health network and local level. The project staff had an excellent working relationship at the regional level (Direccidn Regional de Salud de Apurimac-DIRESA), primarily due to the strong support of the regional director. On the other hand, the relationship with the health network level staff was an on-going struggle, impacted by frequent personnel changes at this level. Following are the important findings from the field survey. Sixteen MINSA staff were interviewed and completed a questionnaire. 1. MINSA staff stated that the project increased their coverage in all four major interventions. Table 6, following presents data from the survey, Table 6: Percent of MINSA staff who stated that the project has improved coverage of Child Survival interventions Intervention Diarrhea Control Umer Res~iratorv Infections 2. A major change is MINSA's enthusiastic support for the community-based patient referral system managed by health promoters. 100% of the respondents supported Result 94% 94% Growth Monitoring Prenatal Care Obstetrical Emergencies 75% 63% 56% * referrals from the promoters and they rated the prompters work in making referrals as 2.6 on a 3-point scale. In turn the promoters also stated that the referral system was a primary area of improvement. At the mid-term MINSA did not have a high regard for the promoters and did not give much significance to their referrals. After the mid-term the project staff created a perforated, three-part form for referral and counter referrals that became a standard tool. 3. Health personnel at individual health posts do not see themselves as having the financial or logistical resources to provide incentives for the promoters'contribution. Their support for the health promoters is superficial in this respect as they rate the value of the promoters highly (87% rated their work either good or excellent), but they are not willing to provide much tangible support. When asked what incentives MINSA could provide after USAID funding ended, the only two meaningful contributions were training (25% of the respondents) and free medical care (19% of the respondents). 4. MINSA however, benefits financially from the promoters referrals by billing the maternal-child social security system, but they are not willing to share the reimbursement with the promoters. It will not be long before the promoters catch on, and demand that they be compensated. If this is not resolved, the prornoter￾based community health system will crash. 5. There is a contrast between the high value that MINSA places on the promoters and the low level of support that the promoters say they receive from MINSA. Sixty one percent of the promoters stated that they receive no support from MINSA. Those who did say they receive support were evenly divided between receiving training and supplies. 6. The project staff made a major effort to work alongside their MINSA counterparts. They had offices in the health centers and when they were not making community visits, they worked out of the health center. Whenever possible Esperansa scheduled joint visits to communities with the counterpart at the local health center. Additionally, Esperansa provided funding so that MINSA staff could receive CS training. iii. Strenutheninu Health Facilities This was not an area in which this CS project intended to provide major assistance. Esperanca, however, did provide educational materials, support for obstetrical emergency evaluation transportation, infrastructure for Maternal Waiting houses, and occasional medicines and medical supplies. With its own funds, Esperanca donated an ultrasound machine to the provincial health office, iv. Strenuthenha Health Worker PerFormance After the mid-term Esperanca made a substantial contribution to the health promoters' performance. As stated eariier in this report, after the mid-term evaluation the project staff changed direction and began working through the promoters instead of working directly with mothers. Much of the motherst gains in knowledge and practice came after this change in strategy. The following list of findings provide evidence of the quality of their work and their acceptance in the community. 1. The promoters say they are primarily involved in health talks, home visits and making referrals, which is supported by observations from the community leaders and mothers. The consistency in the comments which were made independently by the mothers and community leaders provides a solid indication that the promoters indeed assumed the educational role of the project staff. 2. Promoters ' level of knowledge in the areas of pneumonia case management, diarrhea control, perinatal care and nutrition is very high. The following table presents the promoters ' responses to open-end answers about health issues and danger signs for the health of women and children. They had to site all relevant factors in order to obtain a correct response. Table 7: Promoters knowledge base. Topic Percent who reported all signs correctly Signs of pneumonia Signs of diarrhea Identify three food groups (with examples) 3. The promoters state that one of the most significant resources that they have been given are the small boxes of non-prescription medicines called "community pharmacies." This tangible resource gives them very high credibility in the community, while attracting patients to share health complaints with the promoter for possible referral to the health post. 4. Promoters also value their educational materials. There is a good balance between their regard for their role in health promotion and medical care. When asked during the KPC what additional materials they require, health promoters requested health materials on new topics be provided for community education. This is an important recommendation for allowing health promoters to refresh their materials, and expand community knowledge of new health issues. 5. In the last year of the project there was a less than 10% turnover in promoters. This is an indication of the value that the promoters place on their role, and of the support that they receive from the community. This makes it all the more important that the ministry of health place a high priority on finding ways to support them. 89% 89% 89% Postpartum danger signs The following table contains the sustainability plan from the detailed implementation plan (DIP) with the end-of-project status. The status of the objectives listed below is very positive; however, MINSA's institutional capability to follow through is weak. Despite the progress that has been made in mothers' knowledge and practice, and in the community￾level health system, the sustainability of this project is dependant on the new government's commitment to supporting community health. 94% Elements of the project which have been institutionalized in the project zone: 1. Role of health promoters as a crucial link between MINSA health services and the community (health education, referral of patients, management of emergencies) 2. Monthly training of health promoters in topics related to maternal and child health 3. Formation of local health committees in each community (although the role of these committies, and training agenda need to be formalized) 4. Health Promoter Committees are active in each health post, under a signed document outlining the roles and responsibilities of promoters and health personnel 5. Community-based patient referral and counter-referral system has been institutionalized as part of MINSA standard procedure 6. Community-based birth and death registry provides health data which in the past has been unrecorded by the Peruvian government. with skills to improve the health of their children 107 communities with skills to improve the health of their families and access health services effectively implement MOH programs and coordinate with their respective health units 13 health units with staff better able to provide quality services and manage the community health system The MOH of the department of Apurimac with a well-managed health system in its most difficult province The partner organization CADEP with a vision for health, programming and the capacity to carry out that vision C. Program Management End of Project Status 73% of mothers know the sign of pneumonia 64% of mothers know the signs of diarrhea 100% of pregnant mothers had documented prenatal visits 100% of the 130 communities have a health committee. 4 100% have a plan for emergency evacuations for a medical crisis. At least one promoter in all 130 villages. Refer to Table 7, above, for a report on their base knowledge. Less than 10% turnover in the last year of the project. A high level of acceptance by mothers, community leaders, and MINSA. A functionha referral svstem with MINSA. Staff from all 14 health centers have received training CS training. A functioning referral system with the health promoters. Recognition by MINSA staff that the project has improved their coverage in CS interventions. A very low commitment on the part of MINSA to support the community health program once USAID funding is ended. r Constantly changing leadership in CADEP has been a major limitation. CADEP has conducted an extensive evaluation of its institutional capacity, and now has a health division, with short and long-term goals. 1. Planninq Throughout the project, the project director faithfully held monthly technical meetings with the entire staff to review the previous months activities, review objectives and make activity plans the coming month. Both of the project directors conducted these sessions in a participatory style, and the staff made substantial contributions. The staff coordinated all activities on a monthly basis with the MINSA staff, but the response from individual health centers was sometimes sporadic and inconsistent. Regardless of what is stated in the Ministry of Health policies and procedures, in this remote area formal systems are not going to be studiously applied. What it comes down to is the quality of the individual and their commitment to community health-unfortunately this is a commitment that no external agent can successfully impose. With rotating responsibility, the project staff and MINSA convened a monthly training meeting with the health promoters pertaining to each health center. The project took primary responsibility for the training agenda every other month, and the health center staff assumed the coordination every other month. After the mid-term there was good attenda,nce and these meeting gave the promoters a change to participate in program planning and in setting priorities. They considered these meetings an important part of their work and felt validated by them. 2. Staff Traininq MINSA did the initial training in CS interventions and the project staff participated with their health center counterparts. Esperan~a did a good job in providing continuing education for project staff in relevant areas of staff interest. They took advantage of training events sponsored by other NGOs in Peru, and when needed, the project sponsored its own training. The effect of the training was evident in the staff's high level of CS skills. Usually training is more effective the closer it is to the area in which it is to be implemented. In this situation, however, the staff had to go outside of their region because it was so difficult to reach the project area. Additionally, the staff benefited by seeing other CS projects. 3. Supervision af Procrram Staff The organizational structure of the project worked well, particularly considering the distances and lack of reliable communication in the project zone (mostly radio communication was used). During the start-up phase of the project and establishment of the field office in Haquira, the project director resided almost exclusively in the project zone. Later, the project director was based out of the local counterpart organization in Cusco, spending roughly a week each month in the project zone, with additional time spent in Abancay at the regional health directorate. The Esperansa team regularly explored other options, but the need to coordinate with the Ministry of Health, other NGOs and the Esperanga home office kept an office in Cusco with the local counterpart. This arrangement worked well and the field staff were well supervised. The two important reasons for this were, one, good training, and two, on-the-job supervision. There was one traveling field supervisor for seven staff. The supervisor worked alongside each field staff at least once a month and did the liaison work with the local health authorities. Esperansa quickly did away with performance check lists because these could not account for all the variability in the field and because of the competency and dedication of the field staff. It is important to note that they were Quechua-Spanish speaking bilingual, which is essential in this area, especially working with women. The field staff merit special commendation for sticking with the work and for their commitment to the Quechua people. They worked in the field for 18 consecutive days at a time, renting a room as their base in one of the communities. From this base they would make two to four-day walking trips to visit their communities, sleeping on the floor of a promoter's home in their sleeping bags. Nighttime temperatures were almost always below zero year around, and torrential rains and landslides make access to the communities extremely difficult during three-four months of the year. Most days they would have a four to six hour walk in the mountains to reach the next village, They worked under the most difficult conditions seen by this evaluator, and they did not quit. 4. Human Resources and Staff Manauement Please refer to the previous section for comments on this section. 5, Financial Manauement Both of the project directors kept good control of finances, consistently staying close to budget. No major problems occurred. 6. Loaistics Logistics were difficult because the drive from Cusco to the field office took ten hours to cover 80 kilometers. The drive was more like a boat trip in rough seas than riding on a road. There were frequent problems in getting supplies to the area, especially when Esperan~a tried to coordinate logistics with CADEP. The difficulties related to the problems between the organizations that have been mentioned earlier, which were differing missions and geographic locations, While logistics were challenging, none of the problems were a significant hindrance to the project's activities. 7. Information Manauement The health information system (HIS) functioned satisfactorily. Field staff kept good records of their perFormance and progress and their reports were studiously used in the monthly planning meetings. After the mid-term, the team conducted local rapid assessments to monitor the status of measurable indicators in diarrhea and pneumonia control. This helped them greatly in assess their performance in the field. The team communicated the finding of their HIS with the local health centers, but these were not always well-received. Sometimes health center directors would be threatened by the information and sometimes it would contradict the data that they reported to the Ministry of Health. Efforts at coordinating health information systems depended on the strength of character and goodwill of health center staff. This is something that the project staff could not control. The project team could have done a better job of sharing health information with the community. While there were barriers of literacy and lack of supplies, the health promoters could have been used more effectively to set outcome goals in their communities and monitor progress. They did keep records but the information was used more for project reporting than for giving feedback to the community. 8. Technical Assistance and Administrative Su~~ort Esperan~a did a good job in providing technical assistance for its staff. When a need arose, the organization responded promptly. The field staff stated that they could not have asked for more responsiveness. Administrative support was difficult because of the geographic and logistic circumstances. The project directors did not use this as an excuse, however, and were dedicated to supporting the field staff. This was one of the reasons for the low staff turnover, and helped to compensate for their difficult working conditions. 9. Manaaement lessons learned from the ~ersaective of the Es~eranca aroiect staff Coordination with community leaders should have been done from the beginning. I. The project staff decided to complete training in all of the interventions before engaging in community organizing. In retrospect they believe that they should have started organizing right after promoters received training in the first intervention. Training and community organizing need to be implemented concurrently. The limitation is that implementation of all of the interventions will be delayed. There is a trade off between doing Child Survival interventions and community organizing, however, the project staff believe that it would have been more effective if these are done in concert, rather than sequentially. 2. Community organizing. The DIP should have anticipated the need for training local leaders in community organizing and included a budget item to hire contractors who are skilled in this area. While the staff have made good progress in this area, they feel that more could have been done. 3. MENSA staff should be invited to contribute to community organizing activities. It is possible that they would have been more supportive because of doing so. 4. More emphasis should have been given to organizing women's groups and training them in their rights. The level of spouse abuse and degradation of women has been much higher than anticipated. Use of Health Information System and project data 5. The health information system should have had a stronger focus on monitoring changes in knowledge and behavior rather than document activities. Tracking knowledge and behavior would have also provided opportunities for research during the life of the project. 6. The field staff and the promoters would have gotten off to a faster start if they had had a manual with procedures and instruments from the beginning. 7. A map and census of each community should be done at the beginning. This would be a good activity for giving health committees a concrete activity, thus giving a solid impetus to community organization. 8. The data from the census should be used to assign the field staffs territory so that there is not an imbalance numbers of communities and population. Activities and Training 9. Health promoters had difTiculty establishing their credibility for the first two years. The project should have provided them with teaching materials and a community pharmacy at the beginning. 1O.Family gardens should have been one of the planned interventions in the DIP. This is another example of a tangible activity that can spark interest in health promotion. (Although this activity was not in the DIP, the project has successfully implemented greenhouses, training activities, and educational materials on their management) 11. A package of incentives for health promoters should be established from the beginning. The project staff did not do this until the mid-term and consequently lost a lot of momentum because of high promoter turnover early in the project. 12. Training field staff and health promoters in adult education techniques should be integrated with training in the planned interventions. At the end of the project the only technique that the promoters have is flip charts. Very soon people will become bored with this technique and promoters will not have alternatives. 13. The next time Esperan~a should consider hiring at least some of the field staff with specialized skills in areas such as nutrition, community organizing, adult education, etc. D. Other Issues As mentioned earlier in this report, the lack of support from MINSA has the potential to erode much of what has been accomplished by the Esperan~a project unless the existing project (now USAID mission-funded) makes a strong commitment to the sustainability of the project activities. Cotabambas is one of the most isolated and dif'Ficult places to live, thus it is understandable that the Ministry of Health will face significant challenges in serving the health needs of the province. On the other hand, the difficult conditions only outline the extent of need quality health workers and services. Even though it is out of the purview of this evaluator, it seems obvious that the Ministry of Health should offer adequate incentives for health care workers who serve in the most difficult conditions. E. Conclusions and Recommendations 1. It is possible for change in knowledge and behavior to occur in traditional, illiterate societies in a relatively short time. 2. Substantial progress has been made since the mid-term. In large part it is because the Esperanza staff has emphasized the training and supervision of local health promoters instead of working directly with mothers. 3. The consistent perception among mothers and community leaders of the promoters' role gives evidence of the quality of their work. Priority Recommendations I. The level of support from MINSA is superficial and is not likely to be sustained unless there is strong leadership from the new DIRESA Director. The project must devote itself to strengthening the systems which would guarantee continuation of project activities, such as health promoter training) in the project zone. 2. The relatively low percent of mothers who give the same or more solids during episodes of diarrhea indicates a need for continued education in this area. MINSA staff and the promoters should continue to help mothers understand the importance of solids in managing diarrhea. 3. If Esperanp has the opportunity to continue working in this province, this evaluator recommends that they integrate literacy training with the communication of health messages. This strategy was made popular by Paulo Freire and has been used extensively in other parts of Latin America. 4. Esperanqa should increase the use of stories, drama and parables to communicate health messages, instead of a dry recitation of facts. Along with these techniques they should write discussion guides so that the listeners have the opportunity to discuss and analyze the health messages for themselves. Developing critical thinking skills will make an important contribution to development in this region. 5. MINSA benefits financially from the promoters' referrals by billing the maternal-child social security system, but they are not willing to share the reimbursement with the promoters. It will not be long before the promoters catch on, and demand that they be compensated. If this is not resolved, the promoter-based community health system will crash. Esperanqa should advocate for incentive policy for health promoters. - 6. More emphasis should be given to organizing women's groups and training them in their rights. The level of spouse abuse and degradation of women has been much higher than anticipated. Results Hiuhliuhts Child Health Child mortality rates have decreased significantly, especially in the key intervention areas of respiratory infections and diarrheal disease. The infant mortality rate in 1996 for the province was 118 per 1,000 births. For 2000, the rate is calculated as 72.5 per 1,000 births in Apurimac. It is noteworthy that two interventions attained both a substantial change from the baseline and had a high level of adoption at the end of the project. These were: 1) giving solids to children six months or older, and 2) knowledge of the sign for pneumonia. Regarding the first intervention, it should be noted that while there had been a tradition of giving teas and infusions to infants, most mothers (60% according to the baseline) introduced solids late during the child's development. The extent of change in nutrition for young children and early treatment for signs of pneumonia should make a significant difference in the morbidity and mortality of children in this province. Mothers can identify and react to the primary danger signs of pneumonia. Data from the final survey found that 73% could state the sign of rapid breathing at rest. Almost all mothers answered this question immediately with no hesitation. Health Promoters The health promoters are active and are recognized for their labors. Mothers who were interviewed during the field survey for this report gave them a rating of 2.4 on a 3-point scale for quality of their work. Additionally, the community substantiates the promoters' perception of their role. When asked about the most important things that the Promoters do, mothers and community leaders all identified the same top three activities of the promoters: home visits, health talks, making referrals to the health center. The promoters are truly active in their communities. Mothers ' Awareness Mothers have a strong awareness of change over the last four years. Of same group of mothers, 70 of them identified maternal-child health insurance and 51 identified better maternal-child health as the most important change. A factor that gives added significance to the above findings is that at least 65% of the women in the province cannot read. Their knowledge based has increased substantially despite this limitation. Appendix 1: Methods Used Qualitative portion of final survev: The evaluation team of 10 divided into pairs and visited 24 communities spread out evenly among the four health districts. The evaluation team interviewed 189 mothers in 24 communities using an interview format designed by the team with the evaluator. Sixteen MINSA staff were interviewed by the team from 10 health centers. They were given a questionnaire to fill out instead of being interviewed so that the interviewer would not influence their answers. The team interviewed 17 community leaders and 18 promoters using one on one interviews. This evaluator likes to use more interactive techniques but in this case it was not possible to bring groups of mothers and promoters together. Quantitative portion (KPC): For the quantitative portion of the final survey, the project used a version of the KPC 2000-k survey, modified for applicability as comparison with the baseline survey. The survey was translated into Quechua, and also reverse-translated to improve the accuracy of the translation. The sample size was 312 mothers of children under the age of two, randomly selected by clusters (56 communities, in this case). The project contracted an all-female team of ten outside interviewers who had participated in a three-week duration course on interviewer training with another institution. The project trained the group for use of the Esperansa project questionnaire, and the detailed questionnaire guide. All surveys were conducted as verbal interviews of the mothers in Quechua, in the vocabulary that the project, the community and the Ministry of Health use to describe maternal and child health issues. A field trip was conducted to validate the questions in a Quechua-speaking community, and revisions were made after the validation. The survey guide was detailed so that the interviewers knew how not to prompt the mothers' responses, and exactly what language to use. The project staff were interviewer supervisors (each assigned outside of their own project territory), and they traveled with the independent interviewers to orient them in the remote communities, and to observe and correct their styles. None of the project staff filled out any part of the questionnaires in order to retain impartiality of the responses. Appendix 2: Evaluation Team Richard Crespo, Ph.D. - team leader Kali Erickson, MS, MPH -project director Arturo Palomino, nurse-midwife Daniel Mendoza, nurse Mancy Aragh, nurse Apolinar Coyla, nurse Juvenal Bautista, nurse Zaida Quispe, nurse Salorne Ccasani, nurse Health Personnel from throughout the Tambobamba Health Network, Cotabambas Interview team (independently contracted): Gloria Arredondo Raguel Zegarra Maria Elena Acurio Justina Nljfiez Rozana Auca Luz Marina Huaman Ana Guzmh Ahyluz Tupayachi Leoncio Tupayachi