LIGHT FOR LIFE CHILD SURVIVAL PROJECT World Relief Corporation in cooperation with the Ministry of Health CAMBODIA FINAL EVALUATION SEPTEMBER 2002 Evaluation Consultant: Donna Sillan, MPH Submitted by: World Relief Corporation World Relief Contacts: Meredith Long, DrPH, Director Int’l Health Programs Melanie Morrow, MPH, Child Survival Specialist WR Cambodia Contacts: Kay Hansen, RN, Child Survival Project Director Tim Amstutz, WR Cambodia Country Director Project Dates: September 30, 1998 – September 30, 2002 Cooperative Agreement #: FAO-A-00-98-00051-00 Results Highlight: The Formation of Care Groups World Relief Corporation experienced tremendous success through the formation of Care Groups in the Vurhonga (“Dawn”) Child Survival program in Mozambique so they decided to use the same model in the Light for Life Child Survival program in Cambodia. There was initial skepticism, as it was believed that volunteerism and group formation would not work well in Cambodia’s cultural climate. This proved to be a false assumption as the project now includes 126 active care groups made up of 940 volunteers, called Women Health Educators (WHEs). Each WHE volunteer is responsible for visiting 10-15 families throughout each month in their immediate “neighborhood,” comprised of her own friends and neighbors. The number of monthly home visits totals 15,000! During these home visits, they teach health messages related to CSP interventions, using Information Education and Communication (IEC) materials and demonstrations, and dialogue with the mothers about their children’s health, and collect basic data. There are three vital events tracked: pregnancies, births and deaths. Most of the volunteers are illiterate, so the reporting is done verbally. The Care Group leader, also a WHE from the village, is literate and records the vital events on a simple form. The events are reported by name, so that individuals can be identified and located. Discussions of these events and collected data form the basis for problem solving and follow up. Once a month, all the WHE’s gather in a Care Group meeting, were they discuss the past month’s activities, report vital event data, and train for the next month’s health message with the WR Health Field Staff (HFS). World Relief staff consists of two area coordinators, who each supervise eight HFS. Each HFS in turn supervises between 6-8 care groups each having 6-10 WHEs. The health field staff visit each Care Group (once a month) to teach or reinforce a health message and to provide trainer of trainer (TOT) skills, so that WHEs can successfully transfer behavior change practices and health messages to the mothers, and to provide support and encouragement. What keeps 940 WHEs going? WHEs are truly volunteers without any monetary awards. The drop out rate has remained at a low 13%. There are several salient features of the care groups in Cambodia that are highlighted below: Sheer Numbers: • There are a large number of volunteers so that home visits are manageable and do not involve an unrealistic time commitment. By keeping the ratio of volunteer: family very low, (1:10 Households) it is logistically feasible and allows for a personalized relationship of trust to build between WHEs and the households for which they are responsible. • Engendering illiterate women to become WHEs broadens the scope of community participation. Training: • Intervention training along with TOT skills training has satisfied the WHEs unquenchable thirst for knowledge. • Most WHEs said that their husbands encouraged them to get as much training as possible from WR. This supportive attitude from families greatly enhances participation. • Training by CG members is self-sustaining: new WHEs are mentored by experienced WHEs. • WHEs have gained respect in the community by demonstrating positive changes in health that have occurred due to their knowledge. HIS: Health Information System • It is a census-based HIS that includes every household, so that no one is left out. • It is based on three simple indicators that illiterate WHEs are able to collect and report on verbally for the households they visit. • The data the WHEs collected is used to guide the discussion of individual and community problems, not just to pass on to the project leadership. It is useful to them. Power of the Group: Internal Support System • Performance is based on the Care Group, not individual volunteers. This encourages WHEs to support each other through self-correcting mentoring. The group is non-competitive. • It is community-managed, as the care group leaders run the meetings. • It provides a support system for women and a group identity. • It is a fun and friendly, safe place to grow and share. Care Groups provide social interaction and friendship, another factor that contributes to the high retention rate. Empowerment of Women • Community women began to demand health services and speak out. This is a major shift in women’s outlook and attitude. The changes seen in the majority of WHEs- their awareness, self-confidence, their knowledge - particularly so among illiterate women shows an emergence of self-respect and empowerment. The Care Group has worked itself into the fabric of the community, forming a new movement, a new component of community culture. The community members now demand WHEs to visit them every month. The name Care Group is most appropriate as it shows CARING by volunteers for their neighbors (of women and children), and a mutual caring by the community of the Care Groups. Care Group Meeting ACRONYMS BS Birth Spacing CDD Control of Diarrheal Disease CSP Child Survival Project DIP Detailed Implementation Plan EOP End of project EPI Expanded Program of Immunization HFS Health Field Staff IEC Information, Education, Communication KPC Knowledge, Practice and Coverage Survey LRA Local Rapid Assessment MOH Ministry of Health MTE Midterm Evaluation OD Operating District OPV Oral Polio Vaccine ORT Oral Rehydration Therapy PVO Private Voluntary Organization RACHA Reproductive and Child Health Alliance TBA Traditional Birth Attendant TOT Training of Trainers TT Tetanus Toxoid Vit. A Vitamin A capsule WHE Women’s Health Educator (volunteer) Acknowledgement Fantastic, fantastic, fantastic! I must state this up front, as although I have conducted many evaluations, this report may sound like “biased” reporting. But in all honesty, it was difficult to find the cracks. So although I tried very hard to make this report more balanced in terms of strengths and weaknesses, it became a very difficult task to find the weaknesses. The results achieved clearly speak to its strength. I found this program to be seamless and a stellar example of a “positive deviant” Child Survival program. I would like to thank the communities, which our evaluation team visited, for their open doors and open hearts. And I would like to congratulate the staff of WR in Ponhea Kriek, who worked tirelessly to create a program that produced tremendous results. My hats off to Kay Hansen and Oun Sivan, Co-Project Directors, who guided the team down a straight and wide path. I would also like to pay tribute to the strong headquarters team who provided the necessary support the to the field in a steady and consistent manner. The evaluation team was also excellent. I so appreciated the insights and sharing from all the members. I thank Dr. Sotia (for his wonderful humor) and Ms. Sukeng for their invaluable participation, both Save the Children, Australia representatives who generously contributed their time, and to Meredith Long and Katie Norgang, who came from World Relief Headquarters. I steadfastly embrace with excitement the new round of Child Survival funding that has been awarded to World Relief to expand this program. World Relief most certainly deserves it as do the families that will greatly benefit as they expand the fine work they accomplished in this program. It was an honor to bear witness to such a program. I feel uplifted and inspired to continue to support the cause of Child Survival. The Light for Life is the most appropos name for the project as it shone brightly in many people’s hearts, this one included. Thanks for this very enriching opportunity, Donna Sillan, MPH This is the team that made it happen: World Relief’s Light for Life staff TABLE of CONTENTS A. Summary……………………………………………………………………1 B. Assessment of Progress towards Objective 1. Results: Summary Chart………………………………….……………….4 2. Results: Technical Approach a. Brief overview of program……………………………………………5 b. Progress by intervention………………………………………………7 c. Special Studies……………………………………………………….17 3. Results: Cross-cutting approaches a. Community Mobilization……………………………………………..18 b. Communication for Behavioral Change………………………………20 c. Capacity Building Approach i) Strengthening the PVO Organization…… ……………...22 ii) Strengthening the local partners………… ……………...23 iii) Health Facility Strengthening……………………………23 iv) Strengthening health worker performance………………23 v) Training………………………………………………….23 d. Sustainability Strategy………………………………………………..25 C. Program Management 1. Planning…………………………………………………………………..27 2. Staff Training……………………………………………………………..27 3. Supervision of Program Staff……………………………………………..27 4. Human Resource and Staff Management…………………………………29 5. Financial Management……………………………………………………30 6. Logistics…………………………………………………………………..31 7. Information Management…………………………………………………31 8. Technical and Administrative Support……………………………………33 9. Management Lessons Learned…………………………………………….33 D. Conclusions and Recommendations…………………………………………34 E. Results Highlights…………………………………………………………….40 III. Attachments A. Evaluation Team Members B. Assessment methodology C. List of people interviewed and contacted D. Evaluation Tools created E. Evaluation of Evaluation World Relief Cambodia CSP Final Evaluation Report 1 A. Summary The Light for Life Child Survival Program is a collaborative partnership between World Relief Corporation and the Ministry of Health of Cambodia, and communities of the five communes (the northern section) of Ponhea Kriek District in Kompong Cham Province. It is located about 5 hours drive north and east of Phnom Penh, on the Ho Chi Minh trail, close to the border of Vietnam. The project location is primarily rural with several small urban and peri-urban areas. Total population in the project area of 84 villages is 74,873 (1998 census). The majority of the people are ethnic Khmer and Buddhist, while 20% are ethnic Cham (Muslim) minority. Female literacy in the 84 project villages is 26% compared to the national average of 55%. There are approximately 33,700 beneficiaries: 14,600 children under 6 and 18,700 women of reproductive age. Reproductive-aged women constitute about 25% of the population, and children under 6 represent 19.5%. (0-4 year olds are 13.4% of population). Goals of Project: • Improve the preventive and care-seeking behaviors of mothers in the project area • Reduce mortality and morbidity • Build the capacity of Ponhea Kriek District MOH Four Key Interventions: A. EPI: immunization of children and women B. CDD: control of diarrheal disease C. BS: Birth Spacing D. Micronutrients: Vitamin A, maternal iron, iodine and exclusive breastfeeding The program aims to provide an innovative and holistic approach to the improvement of maternal and child health in the most poverty stricken, rural communities in the five communes. Using a participatory approach, the project initiated women's groups where they did not exist and strengthened those already in place to address critical health issues among their own children and within their own communities. In order to ensure sustainability, the project worked with and through existing structures such as the Health Centers, and district medical institutions. The program also created Care Groups, an organically grown group of volunteer women mobilized to improve health within their own village through changing behavior. Volunteers for each village are organized into care groups of 7-10 women, providing structure for training and supervision in addition to social support. The care group units enable the field staff to effectively train and supervise a large number (940) of volunteers that learn health messages and practices in order to train their neighbors through regular home visiting. Sustainability Objectives: a. Volunteers in the care groups will develop a sense of efficacy and pride in promoting changed behavior. b. Key community leaders will publicly support behavioral change targeted by the program. The program approach focused on Behavioral Change, primarily at the household level, through the training of: MOH Health Center staff, World Relief management staff, World Relief Health Field staff, Women Health Educators (WHE) volunteers, mothers/caregivers during home visits, children through BCC activities, TBAs and drug sellers. World Relief Cambodia CSP Final Evaluation Report 2 The program strategy was to increase family level protective behaviors through organizing village volunteers (WHEs) into Care Groups, promotion and assistance of EPI outreach services, and the BCC drama and puppetry activities. The project’s main accomplishments are summarized below. • The program achieved all of its objectives, and in fact far surpassed them in spite of the extremely low baseline. • The immunization rates for children have risen from 5% to 81% and for toxoid tetanus for women from 2% to 71%. This implies a strengthening of the MOH’s capacity, as well as mobilization of the community to seek and utilize EPI outreach services. • The mobilization of illiterate women to become Women Health Educators (WHEs). • The formation of 126 active Care Groups proved to be an effective venue for disseminating health messages and empowering communities to take charge of their health. • The desertion rate of volunteers is 13%. This is outstanding, especially considering that there are over 900 volunteers. The care group system for organizing the volunteers is very effective. • The HIS was streamlined, manageable, owned by the community and the data were actually USED! • The intervention of iodized salt was added and achieved great success, from 0% to 55% use from the midterm to the final. Priority Conclusions: • Focussing on four major interventions allows concentration of efforts and leads to high quality programming. • Objectives that clearly delineated changes in behavior lead the program to work towards achieving those changes. • Faith-based organizations provide an added dimension to the work, which enhances team spirit and commitment. Building a strong, cohesive team is worth the effort. • An exceptionally good management system does make a big difference! • Better financial management at the headquarters trickles down. • Depending on other NGOs or MOH policies, which are outside of the control of WR’s domain requires contingency plans (the formation of VHCs by AusAid did not happen, nor did the supply of birth spacing methods at EPI sessions by the MOH). IMPACT Decreased mortality: Comparison of deaths and births from first & last quarters of project U-5 immuno-preventable deaths dropped from 19.9% to 3% Number of births dropped from 155 to 110 OUTCOMES 81% EPI coverage of 12-23 month olds 71% TT2 coverage of pregnant women 80% ORT Use among mothers of U-2s 50% mothers give extra foods after diarrhea World Relief Cambodia CSP Final Evaluation Report 3 99% HH have a hand washing station 78% mothers wash hands before cooking and after defecation 67% of under-twos have received Vit A 38% of pregnant women take Iron 37% of lactating women exclusively breastfeed for at least 4 months 55% of households use iodized salt 56% of couples follow birth spacing methods OUTPUTS 940 trained volunteers teaching 15,000 mothers monthly through 15,000 monthly home visits 126 active care groups meeting monthly 84 monthly EPI Sessions INPUTS Trained: 21 World Relief staff 22 Health center staff (2 health centers) 940 volunteers 80 TBAs 40 drug sellers 168 village leaders B. Assessment of Results and Impact of the Program Results: Summary Chart Table 1: Indicators (illustrated fully in Chart 1) Goal ** Diff.* Goal B->F BL MT Final SS* Immunization 12-23 mos. +11 70 63 5 28 68 81 TT 2 immunization +1 70 48 2 31 50 71 ORT Use +5 75 62 18 18 80 NA Know 2 danger signs DD +26 30 52 4 7 56 NA Diarrhea: extra food +30 20 46 4 14 50 NA Wash hand before cook +28 50 56 22 34 78 NA Wash hand after defecate -12 90 72 6 6 78 NA Hand washing station NA NA NA NA 57 99 NA Vit. A child +7 60 46 0 10 46 67 Iron for pregnancy +18 20 38 0 10 38 NA Birth spacing +26 30 39 17 21 56 NA World Relief Cambodia CSP Final Evaluation Report 4 Exclusive breastfeeding +21 16 29 8 7 37 NA Iodized salt use NA NA 55 0 7 55 NA * Expected achievement. Point difference from goal to achievement using the formula to calculate the probable final levels presented in “Performance of Private Voluntary Organizations in Increasing Population Levels of Child Survival Behaviors and Knowledge in Developing Countries,” The Johns Hopkins University. PVO Child Survival Project Database. Baltimore, MD 1998. ** Percentage point difference between baseline and final ***special survey TABLE 2: COMPARISON OF FIRST AND LAST LRA 11/99 2/02 PROGRAM TECHNICAL OBJECTIVE LRA #1 LRA #6 Women with 2+ TT (all women, not just pregnant women) 10.6% 74% Modern contraceptive prevalence 23.4% 49.4% Children 11-23 months with complete immunizations 12.8% 61.3% OPV drop out rate 59.4% 18.1% DPT drop out rate 53.6% 16.9% Child Imm. TT 2 ORT Use 2 signs DD Diar.: x-tra food WH cook WH defecate HW station Vit. A childFe BS Exclu.BF Baseline 81 71 80 56 50 78 78 99 67 38 56 37 28 31 18 7 14 34 6 57 10 10 21 5 7 2 18 4 4 22 6 0 0 0 17 8 0 10 20 30 40 50 60 70 80 90 100 Chart 1: Measureable Objectives Baseline MTE Final World Relief Cambodia CSP Final Evaluation Report 5 Please refer to Chart 1: Measurable Objectives. In summary, there has been an improvement in all the measurable objectives from baseline to final. The objectives in which there has been the greatest increase in percentage points from the baseline (first to final) KPC to the final KPC are, hand washing after defecation (72%), child immunizations (63%) and ORT use (62%). The hand washing after defecation objective had the highest percentage point difference from baseline to final, yet fell just short of the end of project goal, reaching 78% instead of 90%. a. Brief Overview The project serves five communes of Ponhea Kriek District in Kampong Cham Province. It is located about 5 hours drive north and east of Phnom Penh, on the Ho Chi Minh trail, close to the border of Vietnam. The project location is primarily rural with several small urban and peri-urban areas. Total population in the project area (84 villages in the 5 communes of Ponhea Kriek District) is 74,873 (1998 census). The majority of the people are ethnic Khmer and Buddhist, while 20% are ethnic Cham (Muslim) minority. Female literacy in the 84 project villages is 26%; the national average is 55%. There are approximately 33,700 beneficiaries: 14,600 children under 6 and 18,700 women of reproductive age. Reproductive-aged women constitute about 25% of the population, and children under 6 are 19.5%. The main objectives were immunization of children and women, control of diarrheal disease, birth spacing and micronutrients (Vitamin A, Iron, Iodine, and exclusive breastfeeding). The main strategy was to increase family level protective behaviors through the establishment of Care Groups of village volunteers (Women Health Educators, or WHEs), promotion and assisting of EPI outreach services and BCC activities including drama and puppetry. The Light for Life project was initiated in 1997. However, Cambodia’s tenuous political situation, culminating in a coup d’etat during the ‘97 elections, caused USAID funding to be put on hold. During that time, World Relief remained committed to launching a Child Survival Project in Ponhea Kriek, and funded a year of preparation and project set up. Expatriate Project Director Kay Hansen began recruiting and training the managerial level staff, so that when the decision was made to continue USAID funding to projects in Cambodia, the Light for Life Project was ready to forge ahead with intervention activities. A brief history of the main events of the programs follows: 1998 1999 2000 2001 2002 Sept. Sept. Sept. Sept. Sept. |========================|============================|============| BL KPC MTE FE KPC KPC/+EPI survey 10/99 Quarterly LRAs|--------|-------|-------|-------|------|-------|-------|-----| ¾ 9/98 Hire Health Field Staff (HFS) ¾ 4/99 Select WHEs; form Care Groups ¾ 6/99 Train WHEs to begin Home Visits ¾ 6/99 1st Training (BS) first technical intervention ¾ 11/99 Monthly reporting of vital events (pregnancies, births, deaths) World Relief Cambodia CSP Final Evaluation Report 6 ¾ 12/99 2nd Training (EPI) 2nd technical intervention ¾ 3/00 3rd Training (CDD) 3rd technical intervention 6/00 4th Training (Micronutrients) 4th technical intervention b. Progress by Intervention Area: 1. Immunization: Objectives: • Increase from 5%-70% complete immunization of children under-one. • Increase from 2%-70% pregnant women who receive at least 2 doses of TT Results of the program: The baseline KPC for immunization of children 12-23 months old was 5%. The final KPC was 68%. When the staff saw that the goal of 70% was not being achieved, they created a “campaign style” reward system for those villages that reached the goal. After the competition, a special survey was conducted and the final immunization coverage rate increased to 81%, 11 points over the goal. Coverage of pregnant women with TT2 increased 2 percentage points to 71%, just over the goal for of 70%. The OPV drop out rate fell from 59% at the baseline to 18% by the final KPC. The DPT drop out rate fell from 54% to 17%. World Relief Cambodia CSP Final Evaluation Report 7 Upon interviewing volunteers, the evaluators found them very well versed in the benefits of immunization, the diseases immunization prevents and in the schedule. TBAs were well aware of the need for pregnant women to have tetanus toxoid immunizations. Mothers were very eager to have their children immunized and the evaluators observed many newborns being immunized at the EPI sessions. The cold chain was well maintained. Sterilizable and disposable needles and syringes were available. A concrete covered well at the health center was specifically for the disposal of needles and syringes. Within the next few months the MOH will be receiving self-destructible disposable needles and syringes to distribute to the districts. EPI Coverage 12-23 mos. 70 5 28 81 0 20 40 60 80 100 1 Percentage Final Midterm Baseline Goal 70 2 31 71 0 10 20 30 40 50 60 70 80 1 KPC Survey results TT2 Immunization Goal Baseline MTE Final World Relief Cambodia CSP Final Evaluation Report 8 In order to verify the effectiveness of the EPI intervention, a comparison was made of the percentage of deaths from immuno-preventable diseases over the course of the project. Deaths from such causes dropped from 19.9% in the first quarter to 3% in the last quarter. Percentage of reported deaths from Immunizable Preventable Diseases: 2000 2001 2002 Quarter 1 2 3 4 5 6 7 8 9 10 11 # EPI Deaths 19.9 21.3 15.3 11.7 8.6 9.5 8.5 1.8 2.1 4.2 3 Factors Positively Affecting Achievement A. The community mobilization efforts that the volunteer WHEs so adeptly implemented greatly accelerated immunization coverage. Prior to having volunteers, EPI teams used to arrive at a village and find only 2-3 children present for immunization. This low turn out was de-motivating for the teams. Now with the volunteers making monthly home visits to motivate women and children to come to the sessions there are usually between 10-30 children present for immunization, depending on the size of the village. The EPI team provides a list of mothers and children due for immunizations to the volunteers who then make sure that they show up at the next session. One mother stated during a home visit made by the evaluation team, “Before we were afraid of immunizations. Now we understand why we need to have them and we bring our children. The volunteers explain and encouraged us to get our children immunized.” B. The linkage between MOH staff and volunteers was firm and cooperative. Since the EPI teams realize that the WHEs help facilitate their work, there is a strong sense of mutual respect and appreciation. Each party encourages the other. When more children and EPI preventable deaths 19.9 21.3 15.3 11.7 8.6 9.5 8.5 1.8 2.1 4.2 3 0 5 10 15 20 25 1 2 3 4 5 6 7 8 9 10 11 Quarters 2000-2002 World Relief Cambodia CSP Final Evaluation Report 9 women show up, an expectation is created and the EPI teams are more encouraged to come. If the EPI team does not show up, the Care Group leader will contact the village leader and they will go to the health center together for explanation and to re-schedule the session. This was a very important relationship to build for sustainability. It was clear from interviews with the Care Group leaders and village leaders that immunization was a right and that they were willing to demand the services if need be. C. In order to boost immunization coverage, a healthy competition was held by the CSP to encourage achievement of project targets in the last months of the project. The contest was between communes. Each EPI team that reached the goal received a prize (a briefcase) and each village that met the goal received a prize for the WHEs . A special EPI survey was conducted after the contest to determine if the target of 70% had been reached—and found that they had achieved 81% coverage project-wide. Factors Negatively affecting Achievement A. EPI is primarily the responsibility of the MOH. They maintain the cold chain, procure vaccines, and conduct monthly vaccinations at outreach immunization sessions. Given the dearth of government services in Ponhea Kriek District, sustainability of such extensive outreach continues to depend on external funding in order to pay for the gasoline for the EPI teams to reach the villages by motorcycle. If WR were not providing the gas money, the EPI staff would have to pay for gas out of their own pocket, which is unreasonable (particularly considering their meager salaries). Luckily, the district MOH may receive funding to provide for outreach activities, as the province will be “contracted” out to an NGO to aid in its running. Without external help, there simply are not funds to meet these low yet recurrent costs. WR will continue to supply gas EPI teams until a contractor assumes responsibility for the Operational District (OD). B. Aseptic technique of sterilizable equipment was questionable. The EPI team used their fingers rather than prongs to lift needles out of the sterilizer tray, compromising sterilization. Main Lessons Learned: • Advocate at the policy level to expand EPI and to integrate other services at the EPI sessions such as family planning, growth monitoring and promotion, and PNC. The audience is available. The Indonesian Pos Yandu system is a good one to study for replication. • Sometimes an NGO cannot avoid providing a non-renewable resource for MOH workers, since there simply are not any funds from the health center or the community fund and the MOH policy is not to charge user fees for EPI services. • A little healthy competition helps to reach targets. Continue to conduct EPI contest once a year. • Motivate the EPI teams to properly dispose of needles (in an incinerator or well) and to keep antiseptic. Provide tools to pick up the sterilized goods. If practices are not World Relief Cambodia CSP Final Evaluation Report 10 followed properly, report to Health center director, not directly to team. (Autodisposal syringes are coming in 2003 from the MOH). Lessons Learned Applied to future activities: The above lessons will be followed in the expansion area and old area, given the second round of funding will begin this year. 2. Diarrhea Case Management: Objectives: • Increase from 42% to 75% the proportion of children with diarrhea treated with ORT • Increase from 4% to 20% the proportion of mother who give extra food to children for 2 weeks following diarrhea • Increase from 4% to 30% mothers who know at least two symptoms of diarrhea that require medical treatment. • Increase from 57% to 90% the number of women who wash their hands after defecating. • Increase from 22% to 50% the number of women who wash their hands before cooking. The WHEs were trained to promote the ORT messages above. The WHEs also learned appropriate mixing and feeding of ORS packets, available at all health posts. Results of the final KPC indicate a significant increase in the proportion of children who receive ORT during a diarrheal episode from 18% to 80%. The proportion of mothers giving an increased amount of food to a child recovering from diarrhea increased from 4% to 50 %. The percentage of mothers who know at least two symptoms of diarrhea that require medical treatment increased from 4% to 56%. The number of women who wash their hands after defecating increased from 6% to 78% and those who wash their hands before cooking increased from 22% to 78%. Although the presence of a washing station was not measured during the baseline, the midterm revealed that 57% of households had one. By the final, 78% of households had a hand washing station. Cross checking with the death reports, there were only two deaths from dehydration during the entire project (June 2001 and February 2002). World Relief Cambodia CSP Final Evaluation Report 11 ORT USE Baseline Midterm LRA #3 LRA# 4 LRA #6 Final KPC Goal 42% 17% 70% 89% 98% 80% 75% Factors Positively Affecting Achievement 1. Monthly home visits by volunteer WHEs to households, providing individual ORT training in the home and using available utensils when demonstrating how to mix ORT. Home fluids, such as readily available coconut water, were also promoted. During the evaluation, mothers repeatedly referred to the fact that WHEs continued to visit them at their homes, even though initially the mothers had not been receptive to the new messages. 2. The availability of ORT is another important factor contributing to the success of this intervention. External Evaluators verified that supplies of ORT packets were abundant at all levels of the health system visited. 3. BCC efforts trained children in ORT messages so that the children could be supportive of home case management of diarrhea and further reinforce the practice at home. 4. After the midterm it was found that ORT use was not improving at the speed necessary to achieve the project target. The teams made an extra effort through home visit demonstrations and increased education, improving the level of ORT use from 17% to 70% in 6 months. Factors Negatively affecting Achievement 1. The prevalence of diarrhea in the two weeks prior to the survey remained the same at baseline and final. The extension area results of the KPC also matched the old area final KPC data, at 47%. So although there is better case management of diarrhea and prevention of dehydration, the prevention of diarrhea still needs attention. One Care Group commented Diarrheal Disease 18 18 80 4 7 56 14 50 22 34 78 0 10 20 30 40 50 60 70 80 90 123 BL MTE FE ORT Use Knows 2 signs Extra food HW before cook World Relief Cambodia CSP Final Evaluation Report 12 that they needed latrines for their village; latrine construction and use should be among the early issues addressed with VHCs as they are formed. 2. Although the project trained drug sellers in case management of diarrhea, Adam Smith’s invisible hand of capitalism still operates according to supply and demand. The drug sellers continue to stock and sell pills, which is understandable given it is their livelihood. Parents are still in demand of anti-diarrheals and look to buy pills, which indicates that the WHE’s must continue to be persistent in educating communities about effective diarrheal disease prevention and treatment. 3. The drug sellers who were trained were from the rural areas only. However, many of the community members shop at the larger marketplaces in the towns. The project should devise a strategy to extend training to vendors in urban marketplaces patronized by the project population. Main Lessons Learned: 1. Carefully target the training of drug sellers to those that people buy from. Conduct a focus group discussion among mothers first to find out where they shop. 2. While working on the prevention of dehydration, it is important to also work at preventing diarrhea. 3. Involvement of village leaders and men in the community is necessary to combat the prevalence of diarrhea through more concerted efforts in community-wide preventive methods such as safe water supply and sanitation. Lessons Learned Applied to future activities: 1. Drug sellers and other community members (village leaders and parents) in the new project will be trained in ORT and the detrimental effects of anti-diarrheals. 2. Mothers will be interviewed to find out where they shop. 3. Men should be involved in diarrhea prevention activities. 3. Birth Spacing: Objectives: • 17%-30% the percentage of women seeking contraceptive services from MOH • By the end of the project MOH will be selling birth spacing methods at 20 EPI sites Birth Spacing 17 21 56 0 10 20 30 40 50 60 123 BL MTE FE Series1 World Relief Cambodia CSP Final Evaluation Report 13 The project far surpassed the target of 30% contraceptive use from 17% at the baseline. The most important indicator in this area is the demonstrated use of modern family planning methods by 56% of women. This increased dramatically after the midterm KPC, more than doubling the percentage from 21%. The trend in reported births also decreased due to the acceptance of modern methods of birth spacing. 2000 2001 2003 Quarter 1 2 3 4 5 6 7 8 9 10 # births 155 126 158 168 133 112 130 137 117 112 Factors Positively Affecting Achievement 1. Birth spacing was not readily accepted at the time of the baseline KPC, however with the continual promotion of the benefits of spacing births by the WHEs, women were much more accepting of birth spacing methods. Home visiting provided a safe and private venue to discuss various methods and suggest an appropriate method couple by couple. 2. The WHEs and TBAs were trained well in the methods of birth spacing, side effects of each method, and knowledge of where to obtain the services. 3. The Provincial MOH trained MOH Midwives (3) and the WR Health Field in birth spacing methods for one week. This was followed up by a two week training of trainers workshop in Birth Spacing Counseling by RACHA for one MOH staff member and two WR managers. They in turn trained 32 MOH midwives and 16 WR staff in birth spacing counseling for another week. All WHEs received Birth Spacing training for 2.5 days. Forty drug sellers also received a one-day training in birth spacing. Number of Births 155 126 158 168 133 112 130 137 117 112 0 20 40 60 80 100 120 140 160 180 1 2 3 4 5 6 7 8 9 10 Quarters in Project World Relief Cambodia CSP Final Evaluation Report 14 4. Birth spacing was chosen as the first technical intervention to introduce given the felt need expressed by the community. 5. TBAs were also trained in birth spacing, so that women received the messages from birth attendants as well. 6. Most of the WHEs were good role models, as they themselves used modern birth spacing methods. 7. The village leaders were highly supportive of birth spacing and encourage men of their village to practice birth spacing. Factors Negatively affecting Achievement 1. The Chinese “once a month pill” was included among modern methods in the baseline KPC. Subsequently that method was calculated separately as the MOH determined it to be unsafe. There were still 1.6% of women using this pill at project end and drug sellers continued to sell the pill in the open market. 2. Village leaders were not adept at describing various methods of birth spacing and requested further technical training to understand the different methods available. 3. Family planning methods were not available at EPI sessions, an objective outside of World Relief’s control that was not met. Although the CSP advocated for the availability of pills, condoms and injectibles at the EPI sessions, the MOH did not agree to provide them at this point in time. Main Lessons Learned: 1. Training volunteers in birth spacing counseling is a necessary component of training in a birth spacing intervention. Many times simply imparting the technical knowledge of birth spacing methods is considered enough. How to transfer the knowledge and encourage/motivate mothers to accept birth spacing is equally important. 2. Consider starting “men’s groups” similar to the WHE’s Care Groups to discuss health issues, as men need to be more involved in birth spacing interventions. (Mothers interviewed preferred the daily pill over condoms, which they communicated were sometimes difficult to convince men to use). 3. The village leaders require more technical knowledge of birth spacing methods to discuss with village men. 4. If an intervention does not reach its expectation and is dependent on MOH policy, it should not be considered a weakness of the program if adequate advocacy was provided. Some issues are not within the control of an NGO. 5. Contingency plans should be made in the event that a party does not follow through with expectations. If the MOH does not provide birth spacing methods at EPI sessions, other plans need to be devised. World Relief Cambodia CSP Final Evaluation Report 15 Micronutrients Objectives: • Increase from 0%- 60% the percentage of children 6-59 months who receive at least one recorded dose of Vitamin A per year. • Increase to 20% the percentage of pregnant women taking iron at least 60 days during their most recent pregnancy and to 10% the percentage taking iron for 90 days. • Increase from 8%-16% the percentage of mothers who exclusively breastfeed for at least 4 months. • EXTRA: Increase the use of iodized salt among households. Factors Positively Affecting Achievement Micronutrients 0 10 38 7 55 10 67 123 BL MTE FE percent Vita A Iodized salt Ferrous Exclusive BF 8 7 37 0 5 10 15 20 25 30 35 40 123 BL MTE FE Exclusive BF World Relief Cambodia CSP Final Evaluation Report 16 1. Mothers are aware of the benefits of Iron and Vitamin A, and could name food rich in both these nutrients. 2. During interviews with the health staff and WHEs, it was clear that the successful increase in iron tablet consumption during pregnancy resulted primarily from women sharing their own positive experiences with their neighbors. Women reported that iron pills gave them strength during their pregnancy. As a result, other women actively sought to buy iron tablets from the drug sellers. 3. Iodized salt became available in the market after the communities were educated about its benefits. The market responded and began to stock salt packets labeled “iodized.” The HFS tested salt to ensure that it was in fact iodized. Factors Negatively affecting Achievement 1. The side effects of taking iron during pregnancy negatively affect compliance. 2. Exclusive breastfeeding is not always truly exclusive. 3. An operations research study on iron distribution mechanisms did not happen. Main Lessons Learned: 1. Raising the awareness of iodine deficiency in the community created an unexpected “phenomenon”. Salt sellers were extremely responsive to community demand and use of iodized salt in the households rose from 0% at the midterm to 55% by the final KPC. Once again the invisible market forces in action! Lessons Learned Applied to future activities: 1. Iron deficiency anemia is a serious problem in Cambodia. The project staff will use its experience in this area to help increase demand for iron supplements and to aid drug sellers in increasing their supply of tablets in the expansion area. c. Special Studies: Neonatal Tetanus Reduction: When the project initially began gathering community data, several cases of neonatal tetanus were reported each month. At the mid-term questions were added to the KPC survey to help assess contributing factors and monitor improvement. From focus groups, the project realized that wasp nest poultices (which are made from dirt and have been shown to sometimes grow tetanus when cultured) were being used as part of traditional cord care. The cord care question was added to help monitor this problem. Cord care using a wasp nest poultice dropped in the original area from 41% to 17%. The use of these poultices in the extension area baseline KPC is World Relief Cambodia CSP Final Evaluation Report 17 54%. The number of neonatal deaths fell from 27 in the period between January-September 2000 to just 9 between October-September 2001. Lessons Learned Applied to Future Activites: • Given that TBAs receive training regarding cord care (one of the 5 Cleans), consider providing TBA kits to all TBAs. The Care Groups could put them together and distribute them at cost. The TBAs could then sell them to their clients. Or, WR could collaborate with RACHA to obtain and distribute TBA kits to volunteers and/or care group leaders and drug sellers to sell. EPI survey: A special survey was conducted in August 2002, following the final KPC, to measure child immunization, maternal TT and vitamin A capsule coverage. The final KPC had shown immunization coverage to be just shy of the 70% target (68%) and in response the staff challenged the MOH and villages to a competition to increase coverage. This provided tremendous impetus to increase coverage and was very effective! The coverage rate for children 12-23 months increased from 68% to 81%. The coverage of TT2 increased from 50% to 71%. Vitamin A distribution increased from 46% to 67%. Simultaneous final and baseline KPCs: (The final for the first CSP and the baseline for the cost extension area). Since the cost extension project was slated to begin as the original project was ending, the team conducted its baseline KPC for the new villages at the same time that it conducted the final KPC in the original project area. This double KPC provided a rough case-control study. Though there will be some natural selection bias and environmental causes in play, it is interesting to compare the extension area results with the “old area” results. These comparisons verified that many of the dramatic increases observed in the original project area were not simply due to trends occurring throughout the province but rather unique to the area that had received the CS interventions. Institutional Strengths Assessment (HQ): An Institutional Strengths Assessment was conducted in February 2002 with inputs from Child Survival Technical Support. The study provided WR with feedback to set practical goals for performance improvement. One need identified was to increase the frequency of financial reports to the field for management purposes. RECOMMENDED STUDIES to BE DONE: a) The Anemia Study that never happened: Iron deficiency among children is an important issue in the project area that WR would wish to investigate further. When the project was beginning, the Cambodian government did not include iron for infants/children on its essential drug list. Upon further inquiry, they said it was because there had not been studies in Cambodia to prove that iron supplementation would help Cambodian children. In response, Melanie Morrow and Kay Hansen worked with Dr. Rebecca World Relief Cambodia CSP Final Evaluation Report 18 Stolzfus of the Johns Hopkins School of Public Health (who graciously donated her time) on a research design for an iron supplementation study. The WR HQ staff was also in dialogue with the WHO, Johns Hopkins University, and MOST to develop an iron supplementation strategy. Unfortunately, due to budgetary concerns and time constraints, the study was not conducted during the original grant period. b) Stillborn Study: Given the high number of still born deaths, (Jan-Sept. 2000: 24 stillborns and between Oct-Sept. 2001:39 stillborns) a study of the reasons for so many stillborns, which are on the rise, should be conducted to see if they could be prevented. SECTION 2: CROSS-CUTTING ISSUES a. Community Mobilization Tremendous progress was made in mobilizing the community and raising awareness of the four health interventions. They included the following activities: A. Recruiting and mobilizing WHEs (940) as volunteers: *Maintained purity of volunteerism without any cash incentives *Volunteer drop out rate was astonishingly low at 13 %. B. Formation of 126 Care groups with leaders from the community C. EPI teams mobilized to visit each village monthly D. Village leaders mobilized to support and advocate for health activities. Constraints: Although AusAID intended to form Village Development Committees in the same villages as this project, they did not. Hence, unfortunately, development committees or health committees with representatives from community groups were not formed. The EPI teams are providing immunization services only. Although one of the project objectives was to have 20 EPI sites provide birth spacing, WR was unable to control this. The MOH has their own policies, which although may be influenced by NGOs, are dictated by national policy. Unfortunately, this did not occur. Also, given the effort being made to bring vaccines to the people through the community outreach, it seems that other activities could be integrated into the session, since the audience is already mobilized and ready. For instance, adding growth monitoring, PNC and birth spacing seems most logical, as in Indonesia’s Pos Yandu system. This is an integrated health post that has five tables: EPI, Growth Monitoring, Nutrition Counseling (and Vitamin A), Birth Spacing and Prenatal Care. MAIN GROUPS MOBILIZED: World Relief Cambodia CSP Final Evaluation Report 19 Village Leaders: The village leadership is truly behind this project. They feel “ownership” of it and have been involved from the start. The initial mobilization efforts started with the village leaders. They also recruited the WHEs from their community and were supportive of the WHEs. When the WHEs had a problem, they felt comfortable requesting help from their leader. Community Volunteers (WHEs) Prior to the project, the community did not have volunteers (WHEs). There was a missing link between the village and the health system. With the introduction of a new cadre of front-line workers, the WHEs, the link was formed and proved to enhance and expand the interface between community members and health services. WHEs in turn mobilized the target population to attend immunization sessions, seek prenatal and birth spacing services at the health centers, and refer children with danger signs of dehydration. They coordinated with the EPI teams to ensure appropriate services, assisted in mobilizing attendance monthly, especially those specified by the EPI team after the last session. The WHEs conducted home visits to counsel the target beneficiaries (mothers and <5 children) in preventive health behaviors. The main role of the WHEs was to provide preventive health education to the community. Care Groups Rather than have community volunteers work independently, the WHEs were organized into support groups. Care groups are community-based and volunteer-driven. Each village has at least one Care Group, made up of 6-10 WHEs. The Care Groups provide a venue for volunteers to work within a group structure, which is supportive and sustainable. The volunteers help their community “together” rather than doing WR’s job without pay. They are functioning well and have strong leaders from within the village itself. EPI Teams The MOH greatly increased its capacity in administering vaccinations, essentially from zero to 4/5 of the child and maternal population. It is able to do this because WR supplies the fuel for transport to the villages and ensures the cold chain. Also, the WHEs created a demand for EPI services, by mobilizing women and children to seek immunizations. Before the teams may have immunized a few children each month, which discouraged them. After the WHEs motivated the families, the sessions had between 20-30 people per month. This mobilized the teams into action. Lessons learned for future community mobilization efforts are: • Rather than wait for another NGO to develop village health committees (VHCs), start working with other non-formal leaders to garner support and community involvement in village level health directions and decisions. In this project, the formation of village development committees was supposed to be implemented by AusAid, yet they were unable to fulfil their commitment. A similar structure could have been formed to make up for this fall out by WR. World Relief Cambodia CSP Final Evaluation Report 20 • Caution should be taken in adding credit activities or a savings/loan component to the care groups as it may change the nature of the group. Currently they maintain purity in their focus on primary health care. When groups add enterprise activities, monetary issues may start to take priority and health issues become less important. Also, the sense of volunteerism becomes diluted, not only within the volunteers themselves, but also in the way communities view the volunteers. It may be better to keep them separate. Volunteers • Select volunteers who are a bit better off so they do not lose too much in terms of opportunity cost and so as to not impact on family income. • Select volunteers that are older (over 25 years old) rather than young girls (18 years old), so that mothers can respect them. They would naturally have more life skills and experience. • Consider functional literacy for the volunteers. • Have each care group select a “name” for themselves to create a unique group identity. • Arrange to have Care Groups cross-visit other Care Groups to create a network of Care Groups and a feeling that they are part of a larger movement. • Care Group leaders who are the star performers should be promoted to become trainers for the new project area. This will provide a sense of upward mobility for WHEs and develop their leadership skills further. The emergence of women leaders that organically grew from the opportunities presented by this project is remarkable. • Advocate at the policy level the expanded EPI to integrate other services at the EPI sessions, such as birth spacing, growth monitoring and promotion and PNC. Overall, the community has been quite receptive to the various mobilization efforts. There is a high degree of community acceptance, recognition and a welcoming spirit towards the WHEs and project staff. Health has become one of the top priorities in the community. The community wants program activities to continue, as measured by demand created among women for the continuation of WHEs and a desire to learn more and participate in the activities. The village leaders have responded to this new demand with a willingness to continue supporting the activities even with less support from WR. The project staff is currently preparing to expand into the new area, and the current area is prepared to continue the health messages with less attention from WR heath staff. This next project will provide a certain degree of weaning, so that eventually WR will be phased out of the project area completely. World Relief Cambodia CSP Final Evaluation Report 21 b. Communication for Behavioral Change (BCC): Puppetry One of the main features of the BCC component was the puppetry provided by two staff members during EPI sessions. This was a major draw in rallying attendance and providing health messages in an entertaining manner to a youthful audience. By targeting children who came along with their mothers and younger siblings to the EPI session, the health education messages were expanded to a larger audience. Ranging from about 5 to 10 years old, the children learned many preventive health practices through the shows. The puppet shows were participatory. The puppeteers also engaged the audience in questions and answers, songs and games. They directly observed the children and compelled action (“Who has their nails clipped? If you do not, run home and get them done and come back!”). By targeting young children, who often take care of their younger siblings, practical knowledge was imparted. Not only does targeting children help reinforce health messages that their mothers were starting to practice at home, it initiates preventive health education at an early age. This is the beginning of shifting community norms among the children who are at an influential age, so that when they grow up the new behaviors will not be “new” but rather internalized and the norm. The BCC staff was well trained, quite talented and commanded a large audience. Hopefully, they can find an agent who will send them on the road to spread the good news throughout Cambodia! IEC Materials The WHEs were the behavioral change agents who conducted home visits and encouraged mothers to attend health sessions. They used an array of IEC materials to facilitate this process. AusAid and the MOH staff produced pamphlets and posters. There were flip charts obtained from the Center of Health Education (MOH). There were some illustrated single sheets, which World Relief Cambodia CSP Final Evaluation Report 22 the WHEs colored in, portraying the health messages. But overall, the materials were rather old￾fashioned and not very innovative. Problems that arose were: • The female literacy rate is only 26% in the 84 village so the written word is not appropriate (most of the materials had text). • Flipcharts can become a barrier/shield between the health educator and the mother. They block eye contact, and are easy to hide behind, which discourages dialogue. • There seemed to be a lot of materials (pamphlets) floating around the WHEs homes, with lots of “poster” wall paper: the same poster in multitude next to each other, but not necessarily distributed to each family’s home. • Some materials were single sheets of photocopied text with illustrations. These were too flimsy and very temporary. • During the EPI session, the EPI teams distributed fliers on various messages. They were handed out randomly and not discussed. • When interviewing village leaders, most were not well aware of the BCC activities, which are also part of IEC. Suggestions: A. Develop more appropriate materials to be used by the WHEs during home visits. • Use a set of key health messages that are illustrated, and kept in a binder, so that they can be removed individually and changed when need be. These could be laminated for durability. • Have the illustrated health messages printed on fabric; join several pieces of fabric and roll them up like a scroll. • Take photographs of local people to use in posters or fliers. This makes it more real. B. If the EPI teams bring fliers to the sessions, the WHEs could ask them how they could help to explain the messages and reinforce them during home visits. C. Encourage the village leadership to attend the BCC activities and generate support among community members, so that attendance rises. D. The project could have benefited from some technical IEC inputs to strengthen its BCC efforts. Focus more on developing a creative targeting plan and identifying a set of core messages for each intervention that are clear, concise and easy to recall. E. For each intervention, channels of communication for these key messages should be expanded and the maximum number utilized. Radio spots, photos of community members and drama troupes are other creative avenues to explore. F. Whenever possible, use “actuals” to demonstrate rather than relying on flipcharts or written materials. For example, using a packet of ORT to describe how to use it, or show actual mangos and papayas, if available, in a nutrition counseling session on Vitamin A rather then showing pictures of them. World Relief Cambodia CSP Final Evaluation Report 23 G. Involve the local community in drama and puppetry to transfer the skill for sustainability and also to engage teens in the health education cause. H. Great to have BCC for children during EPI sessions to create a fun day. Every moment is a health education opportunity. Train mothers AND children. I. Give a copy of all IEC materials to the village leaders. J. All materials should be field-tested in the community and approved by WHEs and selected mothers before replication. K. Involve the WHEs and mothers in the development and design of BCC materials so that they provide their understandings and ideas that work. L. The Muslim Cham may need to have materials appropriate to them. M. Work more closely with WR’s in country program “Hope for Cambodia’s Children” in BCC. During this evaluation, interviews and discussion carried out in communities assessed awareness, understanding and recall of the messages being disseminated by the project. This qualitative evaluation process involved talking to as many women as possible. It is clear that the messages were received and that behavior change occurred. New illustrations for BCC materials arrived from an artist during the evaluation. The drawing style was beautiful. They will have to be tested in the community and reviewed by WHEs and mothers before being replicated. In terms of sustaining BCC efforts, the two BCC staff will continue to be important vehicles for message dissemination, and more BCC staff will be hired as the two experienced staff members move to the new area. Recruiting local community members to conduct puppet shows would enhance sustainability. c. Capacity-building Approach a. Strengthening of WR This project played a key role in enhancing WR’s child survival programming capacity by allowing the organization to: • Extend lessons learned and experience gained to other community-based programs in other regions (particularly WR’s “Vurhonga” Project in Mozambique). • Contribute professionally through dialogue and discussions with other international organizations and participation in workshops and conferences. b. Strengthening of Cambodia Field Office: World Relief Cambodia CSP Final Evaluation Report 24 This office existed for a year before the actual grant period, which was delayed due to the political situation. When the project was given the green light, the Project Director and Assistant Director were ready to hire and train the Health Field Staff. Today there is a large, competent team of 16 Health Field Staff, 2 BCC staff and 2 Area Coordinators and a Khmer Assistant Director. The Project Director has actively mentored and developed the skills of her team. c. Strengthening of local partners: The capacity-building objective for the program was to build the capacity of Ponhea Kriek District’s MOH in EPI and Birth Spacing. Since both of these interventions are dependent upon the local MOH, it is evident from the dramatic increase in these two indicators, that the strength of the local partners was greatly increased. d. Strengthening Health Facilities: The Health facilities were strengthened basically through the improvement of the cold chain and fuel for EPI outreach. AusAID invested in strengthening two health centers and AOG invested in the district hospital. e. Strengthening health worker performance The capacity of health workers has increased through training. This has been a significant contribution of the project. Not only have the MOH staff’s capacity increased, there is a new cadre of worker who are front-line workers. WHE’s are health educators with an enhanced capacity for developmental activities. They have been active in using PRA for action planning and monitoring. When interviewed it became very apparent that the WHEs are well trained and knowledgeable about child survival. Lesson Learned: Monitoring health worker performance: Performance monitoring tools were designed to measure the work of Care group performance, rather than individual WHEs. This worked as a motivational tool to the groups to support their sister WHEs to perform well, as it reflected upon the entire group, and to emphasize group success rather than focus on criticizing individual performance. This approach was particularly successfully within Khmer culture, where shame and “loosing face” can deter individuals from identifying mistakes or weaknesses. The HFS created and maintained an environment where learning was safe and data could be used without being perceived as threatening. f. Training The training program was designed to enable the project staff and WHEs to learn one intervention at a time and practice it, before moving on to a new intervention training. This method known as “drip training” is training over time, with time to master each intervention before moving onto a new intervention. This prevents people from being overwhelmed with all the training at once in the beginning. Drip training proved to be highly effective, as indicated by the high level of information retention among the volunteers and mothers. The training schedule was to add one intervention every three months. The training for each was for 2.5 days, which this evaluator and the project staff have identified as being too long. The HFS training coordinator is currently reviewing and revising the training schedule for the next World Relief Cambodia CSP Final Evaluation Report 25 project in response to lessons learned. The training lesson plans were compiled, however the training methods used were not well documented. Another component to the training process that added to its success was that project members who attended any outside workshops or conferences were obligated to share the information by training the rest of the team. This provided the opportunity to practice more training and to reinforce successful methods of transferring information, so that these experiences would be fresh reference points when training the Care Groups. A strong emphasis was placed on TOT throughout the project. The staff did not just provide technical knowledge; they were skilled and able to impart their knowledge to mothers. Although there were few dropouts, if a WHE needed to be replaced due to an opening, the other experienced WHEs were able to train the new ones. The following training topics by group are listed below. World Relief Staff: a) Assistant Director 1) Birth Spacing TOT b) WR Managers 1) Birth spacing counseling TOT 2) KPC surveys 3) Management of Project Implementation 4) Staff Management 5) Project Implementation, Monitoring and Evaluation 6) Management of Change c) Health Field Staff: 1) Orientation 2) Nutrition Vitamin A 3) Birth Spacing 4) Puppetry and drama 5) TOT 6) Community Health Promotion and Management 7) Nutrition and Micronutrients 8) Diarrhea 9) Immunizations 10) Teaching technique for Health Educators 11) KPC and LQA 12) Community health education techniques 13) Lay HIV counseling 14) Professional counseling for voluntary HIV testing d) BCC staff 1) Puppetry and drama World Relief Cambodia CSP Final Evaluation Report 26 e) All Staff: 1) HIV/AIDs TOT 2) Drama and Adult Participatory Education MOH Staff 1) Birth spacing counseling TOT 2) KPC Surveys and Management 3) EPI 4) TOT 5) Professional counseling for voluntary HIV testing 6) Management of Change Midwives: 1) Birth spacing counseling Village leaders 1) Summary of WHE health messages 2) Community Health Promotion 3) Leadership Training 4) Community Development WHE 1) Orientation 2) Birth Spacing 3) EPI 4) Diarrhea 5) Micronutrients 6) TOT 7) Community Health Promotion 8) Leadership Training 9) Community Development 10) Writing health lessons Drug sellers 1) Birth spacing 2) Diarrhea, ARI, micronutrients (iodine) Lessons Learned: • Original area Care Groups can serve as demonstration sites for training new care group members from the extension area, creating a “living university”. • Use the top care group leaders as “trainers” in the new area. Experienced peer trainers can be extremely effective. • Drip training is effective in providing training over time with practice. • Reducing the drip training from 2.5 days straight days to 2 hours with 2 hours field practice once a week for three weeks would be better. Just design the training in smaller drops! World Relief Cambodia CSP Final Evaluation Report 27 • Those mothers who want to become volunteers should be encouraged and mentored by the CG leader or another volunteer, to spread the news. • Involve village leaders in a “special training” to learn more about the technical training and give copy of lesson plans and IEC materials to him for reference. Train them how to train men. d. Sustainability Strategy The sustainability objectives specified in the project are: 1) Volunteers in the care groups will develop a sense of efficacy and pride in promoting changed behavior. • The indicator is the attrition rate will be less than 35%. This objective was met, and in fact, far surpassed the objective more than two-fold (there was only a 13% drop out rate). Care Groups held celebrations in honor of their accomplishments. • Results of the Baseline and MTE will be communicated in the care groups: Done. • Program staff will regularly praise volunteers before community leaders: Done. 2) Key community leaders will publicly support behavioral change targeted by the program. • Before the midterm, key Muslim leaders among the Cham community will make public statements supportive of birth spacing: Accomplished. • By the Midterm 100% of commune leaders and 75% of village chiefs will publicly display results of the program in their area and correctly explain what they mean. This did not occur, as the data was difficult for the leaders to understand and they did not post it. • By the EOP, influential community members other than volunteers will know the children in their villages who are not immunized and will encourage immunization. This occurs in that if an individual family remains un-immunized, the Care group leader seeks the support of the village leader to visit the family and encourage them to get immunized. The phase-over plan is on schedule. The second round of funding has been awarded and the same staff will continue to work in the new project. A few new team members will be hired for the expansion area. This will force a controlled weaning process in the existing project area. The staff will continue to monitor and be available to Care Groups in the present area, avoiding “abrupt weaning” leading to malnourishment. The Project Director’s position will be phased over to the current Assistant Director (Sivan Oun). As Project Director, Sivan will be supported by Kay (Former Project Director), who will move to the role of Project Advisor. Kay will be based in Phnom Penh and travel to the project site on a scheduled basis. This transition will take place in continued efforts to build capacity and sustainability. The Care Groups have been linked with the EPI teams and health center staff. The continuation of this relationship is a critical link for the project’s sustainability. Behavioral sustainability was achieved as evidenced by changes in health outcomes. Also, Care Groups have been trained to sustain most of the project activities. The project staff have been World Relief Cambodia CSP Final Evaluation Report 28 equipped with new skills and knowledge which has been internalized as have the WHEs and care group members. A demand for services has been created within the community and is internalized. Links between the health centers and the community are strong, so that services will most likely continue at a high level. The Light for Life CSP was designed with the goal of seeing communities sustain behavior change and good health practices after the funded project ends. Everything from the training methods used, to the creation of groups, to building the capacity of local people and health institutions, contributes towards this objective. Evidence that new skills and knowledge has been internalized at many different levels, but most importantly with the caregivers directly responsible for their families’ health, indicates that this goal is being met. By creating a demand for health services, the Care Groups have helped to ensure that these services will continue at a high level. Just as with other WR CSPs, WR will monitor how the objective of sustainable behavior change is being met well into the next project. Financial sustainability The program is an expensive program, mainly in terms of personnel costs. But, the team has so effectively transferred their skills at the community-level, that it is not a recurrent cost. It is based on the participation of community members and tapping the existing government personnel. The WHEs are true volunteers and therefore are sustainable from a financial standpoint. C. PROGRAM MANAGEMENT 1. Planning: The planning process was as inclusive as it possibly could be, given the constraints of time and designing a field project without raising too many expectations before funding was assured. Once the funding was secured, the MOH at all levels were involved in the planning as well as the village leaders. The DIP was an important opportunity to garner partner involvement and “ownership” in the DIP development process. There were not any gaps, in fact, there were too few gaps. Given the level of high detail required by DIP guidelines, it is difficult to allow for flexibility in responding problems that come up during the life of a project and unforeseen circumstances. For example, there were a few interventions that arose due to immediate need: iodize salt and cord care. These problems were revealed by the HIS but were not in the DIP. The project staff was able to incorporate them without deviating from the scheduled activities, however, they indicated a desire to be able to respond more adequately to felt needs that became evident throughout the 4 year project. • An inordinate amount of time goes into developing the DIP, yet it becomes too burdensome to be used as a regular “road map”. It is only reviewed bi-annually or at annual staff retreats. If it were truncated, and less detailed, it would be more user￾friendly and allow for unexpected contingencies in the field reality. World Relief Cambodia CSP Final Evaluation Report 29 Kay Hansen and Oun Sivan both attended the DIP Review in USAID/Washington. This was an excellent learning opportunity for them both. 2. Staff Training: Also see the section above “Training”. It is clear that the each staff member found the training by the project to be practical, relevant and easily applicable. There is a very strong cadre of Child Survival practitioners on board. The Health Field Staff are a 4Cs team, exhibiting Commitment, Competency, Community and Compassion. All of the Health Field Staff have a health background. The Assistant Director is a medical assistant, as are the two area coordinators. The Health Field Staff is comprised of 5 RNs and 9 midwives. By hiring staff with strong clinical backgrounds, the level of competence was already high. WR added training in development, primary health care, Child Survival, bio-statistics, surveying and public health. This rounded out their knowledge and experience. The program moved the staff members from being part of a national or local health system, into the global world of child survival. Although, they are not presently working for MOH, which is desperately in need of more staff, they are receiving training that will eventually drop back into the system, not dropout. The tremendous investment in staff training is an excellent contribution to Cambodia in the long term. The field staff is well trained and highly motivated. They are a resource that the MOH should find a way to incorporate into its service. Given the deficit in human resources in Cambodia, they are needed. 3. Supervision of Program staff: This program stands out as one of the finest in supervision. The Project Director so adeptly supervised and nurtured the team that it was cohesive and strong. The Health Field Staff can be considered a Care Group in and of itself. It is a good example of the qualities and benefits that a Care Group offers. Each staff member exclaimed with heartfelt emotion and conviction that they had a most supportive and strong leader in Kay Hansen. It is a tribute to her outstanding rapport with the team, natural public health instincts, knowledge and a lot of finesse and grace that this program has been so successful. It is rare to find a person who has both the quantitative skills, as well as the qualitative skills in one package. There is usually a preference towards one, a weakness in the other. Kay possesses both in a balanced and complemetary way. The Project Director mentored the Khmer staff from the start to take over her role, which worked out beautifully. Oun Sivan is similar in personality to Kay and has learned many of her skills. As Kay moves to an advisory role, Sivan will be an excellent replacement and team leader. She is highly respected by the staff and the community (both project area and within the MOH). She has a firm foundation and has learned many competencies to continue the job. World Relief Cambodia CSP Final Evaluation Report 30 The supervision system is watertight. It is carefully designed and works on the principles of: • Dialogue between the supervisor and staff member • Learning from mistakes leading to corrective actions, not punishment • Problem-solving: practical • Performance-based according to health outputs There is no sense of a “big brother” breathing down one’s neck and “checking up”, trying to catch errors. Rather it is simply based on performance from measurable outputs. Staff members not performing well are reflected in the data. This is then discussed and analyzed by the supervisor along with the staff member. When a staff member requires more support, another HFS will be assigned to mentor the person who needs more direction. Supervisory checklists are used for the Area coordinators, Health Field Staff and WHEs by their respective supervisors: a) The Project Director and her Assistant will supervise the two Area coordinators using a checklist. b) The area coordinators regularly supervise HFS members also using a checklist. They conduct their supervisory visits by first giving positive feedback, then two things that can be corrected and then giving encouragement. Areas for improvement are based on the data from the LRAs and Monthly Meetings. c) WHEs are supervised by the HFS, who use a checklist to assess knowledge and the process of her work. Contests are held periodically (3 per year) using a small sampling of WHEs from each Care Group. The Care Group as a whole is tested on knowledge and those that answer 70% of the questions correctly receive a prize, either a sarong, mosquito net, blanket or bag of iodized salt). Mini-surveys are conducted by randomly selecting WHEs and visiting the households she is responsible for. Three care groups are randomly selected and three WHEs from those groups are randomly selected. Seven households are visited per WHE, bringing the total to 21 households per Care Group selected. The survey is a truncated version of the Midterm KPC. Lessons Learned • A good supervisor is personality dependent and takes a special type of person with an innate sense of respect and honouring of her staff. • Performance-based supervision keeps the project staff focused on outcomes and is based both on quantitative data and on qualitative processes. • A supervisory checklist is an excellent job aid, which makes the supervision more pointed and consistent. • Contests that focus on the group level encourage peer support and mentoring. 4. Human Resources and staff management Essential personnel policies and procedures are in place for those activities, which will continue through the next grant period. World Relief Cambodia CSP Final Evaluation Report 31 The morale among the staff was superb. When asked to describe their feelings about working with the team, each and every HFS exclaimed that they were grateful to the Director and their fellow colleagues. The working relationships are supportive of each other, not competitive, and the cohesion is incredibly strong. There is a synergy among this team that can move mountains. This is a clear example of how the whole is greater than the sum of the parts it’s made of. The team spirit that was created and led by such a skillful manager certainly was a factor in the project results. The team was built through many events, such as retreats and celebrations and field trips. But underlying these intentional activities is a set of common values, shared by all and forming a steady force sealing the group. There is a strong identity with the organizational culture, which is faith-based. The degree of devotion to their work of service is extraordinary. Quotes from HFS staff: “ I am pleased to work with the Health Field Staff team. I learn from my co-workers. I like to see women improving the health of their families”. – Bohungry, HFS team member. “ I surprise myself by being able to change. During the war I was hard; I looked for violence. Now I feel compassion toward people. I look to help them through my work” – Virak, Behavior Change and Communication educator) “ I see the love that people in the communities have for each other and I love helping them. Now I am equipped to help. The most important thing is to see them change their habits. We have great leaders here! (Referring to the project director and managerial level staff). –Soterie, HFS team member. “ I am so happy to work for World Relief. Now I know about life in the villages, about their struggle and poverty. Before, when I was a nurse in the clinic, I didn’t see the community, I didn’t understand them. Now, I have learned how to help them, how to help them have good health. I love to extend love to people. Kay (Project Director) has a good heart, and much commitment. She is kind to us. I have patience now, and have learned kindness.” –Gumsat, HFS team member. The level of staff turnover was minimal (2 out of the 16 HFS). One of the HFS members moved with her husband to another town and one died of AIDS. The woman who died of AIDS was greatly supported until she died. The experience was painful for the staff but also a true learning experience. They become examples to the community of how to care for and support someone living with, and then dying from HIV/AIDS. There is a surviving child whom the staff members support on their own volition and take responsibility for, as their own child. Support from the WR Country Office contributed to the program’s success. In 2000 the Country Director position in Phnom Penh became vacant. Tim Amstutz, the present Country Director, arrived in June 2001 and has been very supportive of the program. He traveled more frequently to Kriek, advised the project director, encouraged the staff and became an advocate for project needs. Luckily with the second round of funding, the staff will transition directly into the new project, bringing their experience to the new program and up in scale (tripling the project area). It is at the end of that project that jobs will be needed for such talented Child Survivalists. World Relief Cambodia CSP Final Evaluation Report 32 5. Financial management: The institutional strengths assessment conducted by CSTS in February 2002 identified financial management as a priority for World Relief. This project demonstrated the accuracy of their self￾assessment. Project expenditures were recorded by the office assistant in Kriek, then submitted to the accountant in Phnom Penh, who compiled the country office books to be submitted to headquarters on a scheduled monthly basis. Until fairly recently, communication and inter￾departmental understanding of the project’s financial tracking needs were not consistent between headquarters and the country office. As a result, feedback to the project director was not timely. The delay was due to several factors. World Relief headquarters moved from Wheaton, Illinois to Baltimore, Maryland in August of 2000. During that move, there was a 100% turnover in the finance staff at World Relief headquarters. This coincided with the same period of time that the WR Cambodia Country Director’s position was vacant. Finally, there was also a turnover in the health program assistant position at HQ, which assists with financial tracking. Conseuqently, many new staff members were becoming oriented to the CSP’s financial needs all at the same time. This lead to periodic miscommunication with the field regarding availability of project funds. World Relief is aware of its weaknesses in the area of financial management and is taking steps to make sure that this situation does not repeat itself. Responsibilities for coding expenses, monitoring and reporting are being clarified between the field and headquarters. The finance team in Baltimore is now fully staffed and better integrated in their roles, and is able to work more closely with the country accountant. As an additional assurance, the health program assistant, Katie Norgang, has completed one USAID Grants Management Training Workshop and will attend additional ones. She will more actively monitor and facilitate the communication between finance and the field. She was a member of this evaluation team and stayed after the end of the evaluation to assist in financial planning and reporting. 6. Logistics: There were not any major logistical problems, however the most formidable logistical challenge in the project was caused by geography. Since WR is committed to serving the most marginalized people, they chose a remote, under-served area. Basically, the villages are far from each other and under-developed in terms of roads and electricity. The roads are bumpy and rough on vehicles, so maintenance is essential and expensive. Many villages became flooded during the rainy season. The motorcycles required frequent repair and sometimes took a few weeks to get parts. Fortunately, the road from Phnom Penh to Kriek was improved during the last year and the five-hour journey was reduced to three hours. Communication was difficult due to the project area’s remoteness. Besides not having telephone lines, there was not an Internet connection established at the Kriek office. Last year a two-way radio system was installed. Cell phones currently do work and a large radio communication system is being installed for the new project, in order to reach the expanded area. World Relief Cambodia CSP Final Evaluation Report 33 Safety was also a concern. The management team had to constantly be aware and make provisions for staff safety, including ensuring that vehicles were maintained, radio communication was possible, that at least two staff members were together at all times, that no one traveled after dark, and various other safe guards. 7. Information Management: The HIS is an example of the finest of community-based monitoring systems. It should serve as a “model” for all others in its excellence. It is amazing how it is based on illiterate data collectors, who report simply on vital events: pregnancies, births and deaths. This forms the basis of the system. It is population-based, in that each and every household is accounted for through a sweeping home visiting scheme on a monthly basis. Service providers keep service statistics, without a duplication or creation of a parallel system. The urge to write down minute pieces of information on individual children (such as dates of specific antigens) was non-existent since the team knew that the information was being recorded by the MOH. The other tool that complements the vital events reporting are the LRAs, which measure behavioral change and change in knowledge. 1. The Monthly Meeting statistics: The Care Group leader who is literate collects the following data from the other WHEs on a monthly basis. Each and every death was discussed the WHEs in their Care Groups and in fact, formed the major basis for discussion (most apropos for a Child Survival project whose goal is to lower under-five and maternal mortality). The monthly meeting health statistics are formatted below: Monthly Care Group Health Stats Deaths immuniz￾able diseases Month # still births # child < 3 mo # child 3-11 mo # child 1-2 yrs # child 2-5 yrs total < 5 yrs # mater￾nal # neo. tetan￾us N % Other Important Information * = planting or harvest seasons Month % Gp attend. # Gp < 70% WHE drop out % vill. with immun. out-reach Public Health Concerns (# villages) 2. LRAs Change of behaviors among the beneficiary population is measured by a modified lot quality assessment system. Quarterly LRAs (Local Rapid Assessment) were conducted through a series World Relief Cambodia CSP Final Evaluation Report 34 of mini-surveys, selecting a few of the KPC questions each time. Results are consistent with those obtained from the 30-cluster KPC surveys, but unlike the 30-cluster approach, LRA’s allow the team to monitor the progress of individual promoters. The flexible design allows the staff to choose different questions for each survey. Below is a list of the LRAs, which were conducted. These complemented the Baseline, MTE and Final KPC and were used as a monitoring tool. Period n= 1. October 1999 890 2. April 2000 179 3. October 2000 294 4. March 2001 343 5. July 2001 376 6. February 2002 334 Results were displayed graphically on the Project Office walls so that everyone could see them. Elaborate pie charts and bar graphs were prepared with the data both from the 6 LRAs, 3 KPCs and 30 monthly meetings, which happened in 84 villages (2,520 monthly meetings). The most salient feature of the system for this project is that the data was USED! It was used to inform the supervisors of staff performance. It was used to measure progress towards the program objectives. The project staff made program decisions based on data. For example, it was found that a large percentage of infant deaths were due to tetanus. This alerted the program staff to hold focus group discussions in the community to determine the cause of these deaths. It was found that the cord care practices of the TBAs (application of wasp nest poultice on the cord) were causing a high level of tetanus. The team that responded to the data trained the TBAs in proper cord care. This was not planned for in the DIP, yet the program management knew that it greatly affected child survival, so they included it in the project. The same occurred when the HFS saw that the incidence of goiter was disturbingly high. After receiving education on this issue from the HFS, the community began demanding iodized salt at the markets, which was supplied by the salt sellers in no time. The HIS greatly benefited the Health Center staff and the EPI teams. A midwife at the heath center exclaimed how she is able to send her reports to the Provincial level, which are completely done by the WHEs and that she knows the data set is complete and accurate. Prior to this reporting system, her statistics only reported those that attended the health center--a subset of the population. She is thrilled to have good data to report. The EPI teams know that each woman and child eligible to be immunized will be motivated by the WHEs, and sometimes even accompanied by them, to attend the EPI sessions. Because every household is tracked on a census basis, to make sure no mother or child is left out, the data on deaths and births is very accurate. This allows the staff to measure the impact of the program and see trends such as a dramatic decrease in the percentage of deaths due to vaccine-preventable diseases. World Relief Cambodia CSP Final Evaluation Report 35 The major weakness was in feedback to the community. The information was collected, and was delivered very consistently back to the community, but its presentation was too complicated and so was not understood by village leaders, Care Group leaders and WHEs. Data charts were provided to the leaders (village and Care Group), however they were unable to explain their meaning to the evaluators. They were neatly kept and un-used. The data were therefore not shared with the larger community. This lessened the degree to which the community “owned” the data. While they were aware of the overall project results (less sickness, fat babies!) detailed information was not presented in a manner that was useful in the villages. Recommendations: • Engage the Care Group leaders and WHEs in designing the presentation of data. Simple illustrated versions of the pie charts and bar graphs would be more understandable. The level of sophistication was beyond the educational level of the community members who received the information. For example, rather than a bar graph, use a piece of bamboo and paint the percentage coverage of an intervention. Or draw ten stick figures on a poster and color or circle the number of figures that would represent a percentage. • Post the data, in its simplest form, in a central location in the village, so that the entire community can see the measurable changes. Set up a scoreboard or bulletin board to post updated data. It is important for field staff to understand the “road map” of their efforts through a monitoring and evaluating system. The implementers themselves had information necessary for them to keep on track. The community had the community-level information as well. And MOH and all stakeholders were greatly informed and provided with the compiled and analyzed data. By the end of the project, the four major health outcome indicators were readily accessible and available. If a good monitoring system is set up from the start, an evaluation team can easily find the necessary data without having to look for it, drawing directly from the monitoring tools and not doing any extra work to compile data. 8. Technical and administrative support: Technical assistance to the project that was provided included: Meredith Long, Dr PH, WR Director of International Health (Baltimore, Maryland) visited the project three times and was a member of this evaluation team. Charles Moon, WR Asia Regional Director (Baltimore) came once in August 2002. Ambrasi Edward Raj,Ph.D, MPH WR Director of MCH (Baltimore) visited once in 2001. Olga Wollink, MSHSE and Melanie Morrow, MPH, WR Child Survival Specialists (Wheaton and Baltimore, respectively) came a combined 4 times. Katie Norgang, BA, WR International Health Program Assistant (Baltimore) was a member of the team and stayed after the evaluation to provide assistance in grants management for a few days. World Relief Cambodia CSP Final Evaluation Report 36 9. Management Lessons Learned • A rich cross-fertilization between child survival projects intra-agency is an excellent way to implement what works! This project was modeled after WR’s Vurhonga (“Dawn”) CSP in Mozambique. Many of the lessons learned there were directly applied in Cambodia. • A Christian organization such as WR is able to work without the bias of religion/caste or creed, which is often associated with a faith-based group and in fact, can deepen the commitment of the staff. • Community-based HIS with only a few key pieces of information, avoided information overload. Each piece of data collected was used. If a piece of data is not used, it wasn’t collected. • A community-based monitoring system, utilizing simple illustrations is needed for the community level workers and leaders. Community based monitoring and evaluation information empowers the community to understand the impact of the project, which also provides motivation to sustain activities. • No individual can take the responsibility of the community. Only through collective action and the concerted efforts of a group can high achievements be attained. This enhances participation and gives the opportunity for leaders to emerge and develop. E. Conclusions and Recommendations Overall, this project reached its objectives with flying colors and had a tremendous spillover effect in terms of female empowerment. Much of the project effort was sparked by the inspirational motivation by the staff. This was an excellent example of a faith-based service organization. This project served as a “model” in that a front-line worker was created who served as a linkage between the community and the government system. An overall summary of the many exciting conclusions and recommendations made by the FE team are listed below: STRENGTHS 4 Cs Team of Health Field Staff: Commitment Competency Community Compassion • Lots of team spirit led by a skillful manager: a steady force who built a very cohesive and strong team (Kay Hansen), which became a Care Group itself (a role model). World Relief Cambodia CSP Final Evaluation Report 37 • The team supported and sealed through a set of common values • Health Field Staff all with health competencies and learned through TOT to transfer their knowledge Clear, measurable Goals and Objectives: • Surpassed all Objectives (except for one sub-objective regarding handwashing) starting with extraordinarily low baseline indicators. Community Mobilization efforts excellent • 940 volunteers in 126 care groups are highly motivated. • Maintained purity of volunteerism with only 13% dropout rate. • Care groups provided a cohesive body of women who became empowered. HIS Community-based information system: • 100% population-based HIS (census) which counted everyone: Equity • Data collected by illiterate volunteers and focused on vital events only. • Did not create a parallel information system of service statistics, but rather quarterly LRAs to tack behavioral changes. • Data collectors feel “ownership” of the data. • HFS used the data to evaluate performance of program and their volunteers. • WR managers used the data for program and performance-based supervisory decisions (iodine and TT deaths). The team was RESPONSIVE to the DATA. Training focused on behavioral change. • Drip training (training over time) was an excellent way to allow time for the new practice to be internalized by the trainee. • Training in training (TOT) was provided to WHEs so that the technical skills could be transferred. • Investing in staff members is an excellent strategy to prepare the staff for the challenges, but also provides a new cadre of trained individuals within the country. • Use the care group leaders as peer trainers in the new project and for new recruits in the present program. Supervision • Supportive supervision system is in place as a strong follow-up to training • The supervision system was based on performance measured through the HIS. This allowed for pin pointing low performers and providing adequate support to remedy the situation. • Project Director very effective at mentored a Khmer staff to take over her role from the start. World Relief Cambodia CSP Final Evaluation Report 38 Sustainability Strategy well-thought out • Volunteers are truly volunteers. BCC using puppetry for children • Effective during EPI sessions to draw a crowd. • Multi-pronged approach to health messages: through children. Good collaboration with OD and Provincial MOH by WR • Important linkages between the volunteers/village leaders and EPI teams and Health Center staff were created. • Healthy competitions worked well (EPI teams increased coverage quickly). World Relief’s Organizational Culture conducive to CS • Faith based organizations have a value-added dimension which seems to have deepened the commitment to the work. • World Relief effectively cross-germinated from its successful Child Survival projects and shared lessons (Mozambique) Spill-over Effects: • Empowerment of women: both volunteers and HFS. • Communities are demanding health care and working for change. • Demand for Iodized salt was created. WEAKNESSES 1. In the control of the program: • IEC materials were not well developed and are old-fashioned. • HIS tabulated community-wide data feedback to the community was not well understood (too academic and sophisticated) and thus was not posted by village leaders nor understood by the volunteers. The community did not “own” the data yet, although the volunteers did to some degree. • Although ORT usage improved, the prevention of diarrheal disease efforts through hand washing education was not as effective. The prevalence of diarrhea stayed the same (about 47% of under￾two children in the preceding two weeks). • Drug sellers that sold in the market were overlooked as a target for training. The village drug sellers were trained, but many villagers shopped at the market. World Relief Cambodia CSP Final Evaluation Report 39 2. Out of project’s control: • The partner NGO changed plans and did not follow-through with the development of the VDC, something the project was depending on. • Unable to have birth spacing methods available at EPI sessions as planned in spite of the advocacy from the project. It is a MOH policy decision. • Drug sellers did not discontinue selling of anti-diarrheals, though they did sell more ORS. Market forces rule (supply and demand’s invisible hand) and so increased efforts must be made when they are a factor influencing behavior change. • Sustainability needs more time to organically develop and grow. It is given a chance now with the second round of funding. RECOMMENDATIONS TRAINING • Use Original area Care Groups as demonstration sites for training new care group members from the extension area (a “living university”) and have the care group leaders become “trainers”. • Reduce the drip training from 2.5 days straight days to 2 hours with 2 hours field practice once a week for three weeks. Smaller drops. • Those mothers who want to become a volunteer should be encouraged and mentored by a CG leader or another volunteer, to spread the news. • Involve village leaders in a “special training” to learn more about the technical training and give copy of lesson plans and IEC materials for reference for him. Train them how to train men. HIS • Devise ways to present the data to the village leaders and care group leaders. Involve them in the process to have their input. A possible example: for percentages use 10 stick figures or a group of houses. PLANNING • DIP process should be truncated, as it is too detailed and time-consuming and does not account for unexpected contingencies in the field reality. COMMUNITY MOBIILIZATION • Continue to keep credit out of the care group. This will change the focus away from health and lead to other problems. World Relief Cambodia CSP Final Evaluation Report 40 • Rather than wait for another NGO to develop the VDC, start working with other non-formal leaders to garner support and community involvement in village level health directions and decisions. • Advocate at the policy level the expanded EPI to integrate other services at the EPI sessions such as family planning, growth monitoring and promotion and PNC. HEALTH CENTER • Continue EPI contests once a year • OD hospital staff should be trained in methods to motivate the health center staff to do their work. • Request MOH (PP and provincial) to staff more of the health center. • Provide workshop for HC staff (assistant volunteers) to train them like the WHEs. • Provide BS and HIV/VCT training for health center staff • WR should continue to support gas to EPI teams until contractor wins bid for OD. • Autodisposal syringes are coming in 2003 from the MOH. Motivate the EPI teams to properly dispose of needles (in incinerator or well) and to keep antiseptic. Provide tools to pick up the sterilized goods. If practices are not followed properly, report to Health center director, not directly to team. LESSONS LEARNED: • Care groups are a good organization to reach each and every household. • Drip training is effective in providing training over time with practice. • A HIS that focuses on Vital Events and leaves service statistics to the service providers, monitoring only behavior change periodically is manageable and useful. • Very good to start a new contiguous project area by doing two KPCs at the same time (one at the end of one project and one at the beginning of another. This also serves as a rough “case-control” comparison. • Sometimes an organization cannot get around providing a non-renewable resource for a MOH worker, since there simply are not any funds from the health center and there are not any funds in the community and the MOH policy is to not charge user fees for EPI services. • Hiring a staff of well-qualified health practitioners helps to keep the quality of the program (Medical assistants, midwives and RNs). • Being part of a Child Survival program links the staff to the global health community and creates a network of worldwide child survivalists. This group cohesion is highly motivating and invigorating. • Where women are more liberated and free, greater chance of success. World Relief Cambodia CSP Final Evaluation Report 41 • Literacy needn’t be considered a block in recruiting village volunteers. • Healthy competition works well to increase coverage (EPI team contest). • Great to have BCC for children during session. Creates a fun day. Every moment is a health education opportunity. Multi-pronged approach to health education. Train mothers, train children. In conclusion, I will let the WR team and Light For Life community speak for themselves by presenting some quotes which recorded during the course of the evaluation. Quotes from Interviews: Mothers • “Before we were afraid of immunizations. Now we understand why we need to have them and we bring our children. The volunteers explained and encouraged us to get our children immunized.” • “The volunteers came to visit us and with interest and concern for our babies. It made us feel that we, as mothers, have to pay even more attention to our babies now”. • “I expect the volunteer to come every month.” • “I want to become a volunteer.” Volunteers (WHE/Care Group Members) • “The mothers didn’t know how to take care of their children. Now they keep themselves and their children clean and take better care. They didn’t know how to keep the house clean, or wash their hands. They didn’t know about birth spacing. Now they know a lot. There are less sick children and less children dying.” • “We are motivated by training. We want more training. Our husbands encourage us to get as much training as possible”. • “We were ordinary housewives before, now we are able to help our community and serve others”. TBAs “People always thought that TBAs could not learn or change. But, I learned new skills and changed my practices. I want to teach other TBAs my new skills.” EPI Team • “Before we used to come to a village and only immunize 1-2 children. It was not worth going all that distance. Now that the volunteers motivate the community, we immunize between 15-30 each time. We can give them a list of children and women to come to the next session and they will show up the next time. “ World Relief Cambodia CSP Final Evaluation Report 42 Operating District MOH • “We are not able to reach the community. With WR they are able to motivate the community to come for our services. We are very grateful that they have closed the gap.” Midwife at the Health Center • “With the reports I get from the volunteers, I can send them directly to the district and provincial MOH. The data is complete now. Before it was sporadic data taken by those that showed up at the EPI sessions. Now it is population-based data which counts everyone.” • “I have more clients now that the volunteers have motivated women to come for PNC.” Village leaders: • “I will continue to support the volunteers because they have made our village healthy. Before we had a lot of illness and death. Now most deaths are just due to old age.” • “The mothers had a lack of knowledge.” World Relief Project Director Assistant: • “I feel so fortunate to be part of a global community and tapped into Child Survival. It is so much wider than my job at the hospital as a medical assistant. I can do so much more”. World Relief Health Field Staff: • “ I feel excited, more patient and more loving to myself, my team and the community.” • “ Before I was shy, now I can be open and train others.” Sharing Light for Life’s Lessons Learned with the Development Community at Large As World Relief develops increasing expertise in implementing successful Child Survival Programs, it remains committed to sharing experiences with the global Child Survival and development communities. WR seeks to build upon strengths of past and existing projects, to apply lessons learned from one project the next, and to collect and utilize data, not only inside the organization but with its partner and colleague institutions through a variety of means. Within Cambodia, WR Child Survival participates in training workshops and conferences with other organizations doing similar work near to the project, (particularly World Vision, AusAid, and AOG). Most recently Light for Life has sent representatives to Child Survival meetings in Phnom Penh which provides yet another opportunity for organizations in attendance to see WR’s approach and lessons to contribute. Light for Life hosted representatives of Save the Children Australia as team members for their final evaluation. This was an excellent experience of sharing lessons learned and getting to know a key partner organization, as SAVE Australia will likely bid and win the OD contract were WR’s CSP will continue and expand within the next five years. Light for Life has worked closely with AOG and their program to build capacity within Phonea Kriek’s only hospital. WR has received much support and cooperation from AOG in navigating the often difficult bureaucratic structures within district and national institutions, but has also World Relief Cambodia CSP Final Evaluation Report 43 been able to contribute positively to AOG’s mission by sharing results and lessons learned from the CSP activities. In addition, Light for Life has collaborated with WR Cambodia’s CREDIT program to assess potential for incorporating savings programs in community health programs. They have worked with HOPE for Cambodia’s children to learn from its successful use of drama as a means of education. These programs can in turn share Light for Life’s approach when relevant with others, making the distribution of project story and lessons learned spread exponentially. World Relief’s headquarters largely facilitates communication of lessons learned. WR’s Child Survival Specialist Melanie Morrow participates in CORE, a consortium of organizations involved in Child Survival Project implementation and grants. Through annual meetings, workshops and presentations, Melanie shares lessons about Light for Life and all of WR’s Child Survival Projects. She reports both concrete data and qualitative observations, as well as shares stories and dialogues informally with others, all as part of ongoing collaboration between PVO’s who share similar goals, and/or work within common institutional structures and geographical locations. Melanie’s involvement in CORE is key in providing WR the opportunity to communicate its activities and to benefit from others’ experiences. Results and experiences from Light for Life have been submitted in the form of an abstract for presentation at the annual meeting of the Global Health Council in 2003. Annex A. Evaluation Team Members 1. Dr. Sotia, Save the Children 2. Ms. Sukeng, Save the Children 3. Dr. Meredith Long, Director of Int’l Health Programs, WR headquarters 4. Ms. Katie Norgang, Program Assistant, WR headquarters 5. Donna Sillan, team leader, independent consultant From left to right: Oun Sivan, Katie Norgang, Ms.Sukeng, Kay Hansen, Meredith Long (back), Dr. Sotia and Donna Sillan. World Relief Cambodia CSP Final Evaluation Report 44 Annex B: Assessment Methodology Evaluation Objectives: a. assess if program met the stated goals and objectives b. the effectiveness of the technical approach c. development of overarching lessons learned from the project d. a strategy for use of communication of these lessons both within the organization and to partners. Evaluation Process: The evaluation team evaluated the project using a participatory approach. The team was comprised of experienced and expert primary health care specialists. Upon initial meetings in the Project office in Kriek to orient the staff and evaluation team to the evaluative process, the team, using the program goals and objectives as a guide, devised interview tools to be used in the projects. The set of evaluation tools for the persons to be interviewed, and focus groups to be facilitated follows below. The team was then divided into four sub-teams to cover eight villages in two days. Upon return from the villages, the full team consolidated the field findings and reached consensus on the conclusions. The team discussed each of the crosscutting issues. A feedback meeting was held in Phnom Penh for staff and stakeholders at the Royal Phnom Penh Hotel. Photo: Health Field Staff working in small groups. World Relief Cambodia CSP Final Evaluation Report 45 a) Quantitative Analysis: 1. Comparison of baseline and final data (KPC surveys) 2. Review progress through the health information and monitoring system: LRAs 3. Monthly meeting reports 4. Home-based records b) Qualitative Analysis: Focus group discussions and key informant interviews: 1. Home visits: mothers/fathers 2. WHEs 3. CARE Group members 4. Village leaders 5. TBAs 6. Drug sellers 7. Health facility staff 8. EPI teams Evaluation Schedule DAY 1: 1. Team meets: orientation to evaluation 2. Determine process: sub-teams and villages to be visited 3. Development of evaluation tools (8 guided interview sheets) DAY 2: 1. Review Quantitative data by intervention (small groups) 2. Refine tools and practice using tools DAY 3: FIELD VISIT (4 villages) Qualitative Four sub-teams visit one village: Interview village leaders, volunteers, tbas, mothers, make home visits, attend EPI session/observe BCC performance, visit Health Center, interview health center staff. DAY 4: FIELD VISIT (4 villages) Four sub-teams each visit one village: same as above DAY 5: Analysis of field data (qualitative data), presentation by groups DAY 6: Discussion of lessons learned DAY 7: FIELD VISIT ( 2 villages) 1. Reach consensus on strengths, weaknesses, recommendations and lessons learned. 2. Visit 2 Care Groups ( two sub-teams) DAY 8 & 9: 1. Travel to Phnom Penh: Interview Country Representative World Relief Cambodia CSP Final Evaluation Report 46 2. Preparation of presentation DAY 10: Presentation and Feedback session with stakeholders Annex C. List of People Interviewed: (259) Village level: (232 across 10 villages) Village leaders 8 Volunteers 85 Mothers 115 Tbas 8 Health center staff 8 Drug sellers 8 WR Staff: (24) Director of Int’l Health 1 HQ Country Director 1 Phnom Penh Project Director 2 Kriek Area Coordinator 1 Training Officer 1 Health Field Staff 16 BCC staff 2 Partners: AOG 1 Deputy District Chief 1 OD Director of Health 1 Facilities/activities: (9) EPI sessions 2 BCC sessions 2 Care groups 2 Health centers 2 Hospital 1 People Interviewed: World Relief Staff: Tim Amstutz, Cambodia Country Director Kay Hansen, Light for Life Project Director Oun Sivan, Light for Life Assistant Director Phan Buntheng, Light for Life Area Coordinator/Trainer Partners Ly Leng, Deputy District Chief Mr. Hok Hean, Operational District Ministry of Health Manager Carol Feigleson, The Assemblies of God, Primary Health Care Advisor World Relief Cambodia CSP Final Evaluation Report 47 Annex D. Discussion Guide TOOLBOX: 1. Interview with Village Leaders 2. Focus Group Discussion with Volunteers 3. Interview (FGD) with TBAs 4. Home Visit Guide 5. Health Center Staff (EPI and MWs) 6. Interviews with drug sellers 7. Immunization Session Observation 8. Care Group Observation/Questions Tool #1: Village Leaders 1. What is your understanding of the program? Goals and objectives? 2. What are the main causes of death in your village? 3. What achievements have been made since WR started? 4. What obstacles did you still face? 5. What is your relationship with the volunteers? 6. How did you select the volunteers? 7. What is the benefit of being in a group if any? 8. What if there weren’t any Volunteers? 9. How do you support the volunteers? 10. Should they continue once the program ends? 11. Do you know the immunization coverage of your village? Who is not immunized? 12. What do you do if the EPI team doesn’t arrive as scheduled? 13. Do you receive reports and post the program’s progress? 14. Has the data been explained to you? If so, by whom? 15. How do you support birth spacing? 16. What are you benefits of Vit A and ferrous? 17. What is to be done about diarrhea in children? 18. Were the BCC activities effective? Did you attend any? Any suggestions? 19. Do you have a VDC or Health committee in your village? 20. What are the dreams for your village? World Relief Cambodia CSP Final Evaluation Report 48 Tool # 2. FGD Guide for Community Volunteers 1. How long have you been a Volunteer? 2. What is your understanding of the program? Goals and objectives? 3. What are the main causes of death in your village? 4. What achievements have you made? 5. What obstacles did you face? 6. What is your relationship with the HFS of WR, your supervisor? 7. What is the benefit of being in a group if any? 8. What if there weren’t any Volunteers? 9. What have you done since you started? 10. What behavioral changes did you concentrate on? 11. Where you able to change behaviors? Explain. 12. What changes have you experienced personally? 13. What kinds of support did you receive? 14. How is your group perceived within the community? HIS 1. What monitoring tools did you use? How did you target beneficiaries most in need? How did you use the data? 2. Do you report diseases and deaths? To whom? 3. Did the data help or hinder you? 4. Did you discuss the data in a group? 5. What was the most important piece of information you collected or needed to know? TRAINING 1. What type of training did you receive? 2. What training gaps exist? What topics would you like to know more about? 3. How did you train mothers? SUPERVISION 1. How were you supervised? Was it sufficient? 2. What changes do you feel? 3. Why do you volunteer? What motivates you? 4. How often do you meet with the Care group? 5. How many home visits do you regularly make? 6. Do you celebrate your successes? 7. Does your supervisor recognize you in front of the leaders? 8. Do the leaders honor your contribution to the community? How? BCC: 1. What types of materials did you use? 2. Did you feel they were enough? 3. Were they effective? World Relief Cambodia CSP Final Evaluation Report 49 4. How would you change them? Technical: EPI 1. Describe the immunization schedule for 12-23 months and pregnant women 2. What are the signs and symptoms of vaccine-preventable diseases? 3. When do you refer? 4. What is needed for EPI? Do the teams visit regularly? 5. How often does the EPI come? 6. Do you attend the EPI sessions? What do you do? 7. How do you motivate mothers to come to the sessions? DD 1. What do you do about diarrhea in children? 2. How do you mix ORT? 3. How do you teach mothers about Diarrhea Case Management? 4. What are the signs and symptoms when its time to refer? Share some examples. 5. How do you teach about hand-washing? Messages? 6. Have you taught anything about feces? BS 1. Explain birth spacing. 2. Where are these methods available? 3. How do you counsel someone in accepting BS? Micro Nutrients 1. What do you teach mothers about the benefits of Vit A? 2. What age group? # of doses? 3. What foods are rich in Vit. A? 4. Where do you tell mothers to get the VAC? 5. What do you teach mothers about the benefits of FE? 6. Where do you tell mothers to get the Fe? 7. What foods are rich in iron? 8. What do you teach mothers about the benefits of exclusive BF? Sustainability: 1. What components of the program are sustainable? 2. What parts of the program are most important to continue in your opinion? 3. If the FS stops supervising, what will happen? 4. What will you do now that this phase of the program has ended? 5. What is the difference between your area and one without a Volunteer? 6. What motivates you? 7. What kinds of activities have you planned? 8. How do you replace drop-outs? 9. What if a mother wants to become a volunteer? What you do you? 10. What are your future aspirations? 11. Any special stories you would like to share? World Relief Cambodia CSP Final Evaluation Report 50 12. Any questions? Any concerns? Tool # 3 Interview for TBAs 1. Are you trained and if so, what topics and where, by whom, when? 2. After training, what practices did you change? 3. Are there any obstacles to practicing your new skills? 4. How many deliveries do you assist in a month? 5. Within the last year, how many stillborns? What is the cause of these? 6. How many maternal deaths? 7. How many neonatal deaths? 8. Do you report births or deaths? 9. Do you provide ANC care? 10. What kind of PNC is necessary? Do you provide it? 11. How do you cut the cord? With what? 12. What cord care do you do? 13. When do you refer? 14. When should a woman initiate breast-feeding? 15. What else should a newborn be given besides breastmilk? 16. Any special stories you would like to share? Tool # 4: MOTHERS: Home Visits Questions 1. What is your understanding of the program? 2. Have you heard of Volunteers? What is the name of your volunteer? 3. What does a CV do for you? 4. How many times have you been pregnant? 5. How many children do you have and by gender? 6. Did you plan your children in terms of spacing? Do you use CS? 7. Who motivated you to use birth spacing methods? 8. Do you have immunization services in your area? Do you get your children immunized? Can we see the cards? 9. Who encouraged you to get immunizations? 10. When you were pregnant last how many doses of TT? Can we see the card? 11. Did you have any supplements when you were pregnant (iron?) how many? 12. Did your children receive Vit A? What is it for? 13. Where and who delivered your children? Was she trained? 14. What did the TBA do to the cord when you delivered? 15. Did you breastfeed? When did you start? What else is given to the newborn besides breastmilk? 16. Have any children been ill in the last 2 months? What illness? What did you do? 17. If your child had diarrhea in the last 2 weeks? What did you do? 18. Who taught you what to do? 19. What do you do for recovery from diarrhea? World Relief Cambodia CSP Final Evaluation Report 51 20. What signs/symptoms cause you to go to the health center? 21. When do you wash your hands? If so, where is the soap? 22. Do you have a special place for handwashing? 23. Where do you defecate? 24. Did you go to any BCC events? 25. What did you enjoy most? 26. What did you learn at these events? 27. How would you feel if the volunteer stopped visiting you? 28. What are your aspirations for your child? What about for your sons? Daughters? Tool; # 5. Health Professionals at health center 1. How many years have you worked in this area? 2. What is your role? What services do you provide? What is the response in the community to your services? 3. What is your understanding of the project? 4. What were your problems 4 years ago? Have they changed? 5. What is your impression of the Volunteer? Should the position be continued? What is the value-added if any? 6. How often do the EPI teams visit each village? Is the schedule consistent? 7. How does a volunteer assist during an EPI session? 8. What is the situation of birth spacing? Do you have enough supply? 9. Has birth spacing been available at the EPI sessions? If not, why not? 10. Did you receive any training from the project? BS counseling? 11. Do you have adequate supplies of Vit A? 12. Do you have adequate supplies of Ferrous? Is it available at the EPI sessions? 13. Have many diarrhea cases been referred by the volunteers? Has that increased? Tool # 6 Drug Seller Interviews 1. What is your understanding of the Child Survival program? 2. Have you received training from the project? 3. What are the danger signs of diarrhea? 4. What do you sell to mothers who come with a child with diarrhea? 5. When do you refer cases? 6. Do you sell ferrous? Have sales increased since the volunteers started? 7. How much iron does a pregnant woman need? Tool # 7 EPI sessions 1. What is the average number of children getting immunized each session? 2. When was the last session held in this village? 3. How often does the team come to the village? 4. What prevents the team from coming? 5. Is the cold chain maintained? 6. Is there adequate supply of vaccine? 7. Are proper sterilization techniques followed? World Relief Cambodia CSP Final Evaluation Report 52 8. What kind of needles? How disposed? 9. Are the side effects of immunizations explained to the mother? 10. Is the immunization recorded properly on the yellow card? 11. What do the volunteers do at the session? 12. What data does the volunteer use? 13. How is the information shared? 14. How do the volunteers know who is a no-show who needs to be immunized? 15. What is the follow-up for left-outs? Drop-outs? Tool # 8 Interview for CARE GROUPS 1. How long have you been meeting? 2. What is the normal attendance? 3. What is the purpose of your group? 4. Does your group have a name, identity? 5. What do you like best about the group? 6. What do you like least about the group? 7. How does the group benefit you? 8. What do you discuss at the meetings? 9. What kind of support do you get from the HFS? 10. Who is your leader? 11. Who chose her? 12. Who facilitates the meeting? 13. Does a WR HFS join in at each meeting? What does she do? 14. How do you replace volunteer drop-outs? 15. What information do you report? 16. How do you analyze this information? 17. If WR stopped coming, would you continue to collect this data? 18. Is the data useful to you? How is the data useful to you? 19. What skills have you learned? 20. What new skills would you like to learn? 21. Do you have sufficient BCC materials? 22. How do you support the EPI teams? 23. What if they don’t show up? 24. Will you continue the group once WR leaves? 25. Any special story or anecdote you would like to share about your group? 26. Any suggestions? World Relief Cambodia CSP Final Evaluation Report 53 Annex E. Evaluation of the Evaluation Each team member and the entire staff were asked to do an evaluation of the evaluation process. They answered three questions, which are compiled below: What helped the process? • To find out the strengths and weakness of the project to help us to build on our strengths and fill in the weaknesses to improve the process. • High quality of the evaluation, questionnaires were prepared, groups were assigned, materials were prepared and the schedule carefully planned. • Clear management and implemented smoothly and in order. • Highly participatory evaluation involving all stakeholders: staff, community, and MOH • Perfect evaluation with check up with many places to make sure the data was realistic and true. • It helpful to the staff to learn more about evaluation and know how to do correctly • Encouraging and motivating for the staff to see the results they have achieved through hard work. • Each step of the process was clear. • Increased good relationships and trust between the evaluator team and the project staff. • Good chance to know more people. • You are a very good evaluator, thank you very much. What hindered the process? • Time is too short for the process • Language barrier • Some discussions questions were difficult to understand because most of them were written in English • Work too much, not enough breaks Recommendations • Please come again for the next evaluation • You should study Khmer language and speak it next time. • Write all flip charts in both English and Khmer. Time should be on schedule.