World Relief Mozambique Vurhonga II CSXV Child Survival Project FINAL EVALUATION REPORT Submitted to USAID Evaluator Carl E Taylor, MD, MPH Professor Emeritus, Johns Hopkins Bloomberg School of Public Health Baltimore, MD Project Location: Chokwe District of Gaza Province, Mozambique Submission Date: December 31, 2003 Project Dates: September 30, 1999 – September 29, 2003 Cooperative Agreement #: FAO-A-00-99-00042-00 2 ACRONYMS AIDS Acquired Immune Deficiency Syndrome BCC Behavior Change Communication CG Care Group C-HIS Community-based Health Information System CS Child Survival CSTS Child Survival Technical Support DIP Detailed Implementation Plan EPI Expanded Program of Immunization HIS Health Information System HIV Human Immuno-Deficiency Virus ITN Insecticide Treated Nets KPC Knowledge, Practices and Coverage survey LRA Local Rapid Assessment MCH Maternal Child Health MOH Ministry of Health MTE Midterm Evaluation NGO Non-Governmental Organization ORT Oral Rehydration Therapy PVO Private Voluntary Organization STIs Sexually Transmitted Infections TBA Traditional Birth Attendant USAID United States Agency for International Development VHC Village Health Committee WR World Relief Chef de saude Village Health Representative Socorrista Trained first aid health worker at health post 3 Table of Contents 1. Summary.............................................................................................................4 2. Results: Summary Chart .....................................................................................5 3. Results: Technical Approach...............................................................................8 4. Cross Cutting Approaches................................................................................10 A. Community Mobilization..............................................................................10 B. Communication for Behavior Change ..........................................................12 C. Capacity Building Approach.........................................................................14 C1. Strengthening the PVO ...........................................................................14 C2. Strengthening Partner Organizations: Health Facilities and Health Worker Performance......................................................................................14 C3.Training....................................................................................................15 5. Sustainability Strategy.......................................................................................16 6. Program Management........................................................................................17 A. Planning ........................................................................................................17 B. Staff Training ................................................................................................17 C. Supervision of Program Staff........................................................................18 D. Human Resource Management.....................................................................18 E. Financial Management ..................................................................................19 F. Logistics........................................................................................................19 G. Information Management .............................................................................19 H. Technical and Administrative Support.........................................................20 I. Management Lessons Learned .......................................................................20 7. Meeting with Prime Minister Dr. Pascoal Mucumbi.........................................20 8. Summary Statement of Conclusions..................................................................21 9. Recommendations..............................................................................................22 10. Results Highlight .............................................................................................23 11. Annexes ...........................................................................................................25 A. Summary Information from Project Detailed Implementation Plan.............25 B. Evaluation Team Members...........................................................................43 C. Assessment Methodology.............................................................................44 D. List of Persons Interviewed and Contacted ..................................................45 E. Workshop on Sustaining and Scaling Up Community Development Programs ............................................................................................................46 F. Final KPC Survey Report..............................................................................47 World Relief Mozambique CSXV Final Evaluation Report 2003 4 1. Summary World Relief’s Vurhonga II Child Survival Project in rural Chokwe District (population 140,000) of Gaza Province, Mozambique, has unique potential for becoming a world￾class demonstration of success in child survival. The CSP achieved or exceeded all of its stated goals and objectives, including substantial reductions (over 60%) in infant and child mortality. The goals achieved by the project were to: 1) reduce morbidity and mortality in children under five and women of fertile age; 2) strengthen the capacity of the Ministry of Health to implement child survival interventions; and 3) empower communities to improve their health. Vurhonga II helped 48 communities create 220 Care Groups to train 2,800 volunteer mothers and church leaders as behavior change agents to all households for nutrition; malaria, pneumonia, and diarrhea case management; maternal and newborn care; child spacing; and STI/HIV/AIDS prevention. The project assisted the MOH to expand its network of health posts so that 95% of the population lives within 5k of a trained health care provider (nurse or first aide-trained socorrista). The project also established communication and coordination links between the communities and health posts. In the first three years of the project, 0-5 child mortality declined by 62.2 percent (119/1000 in 2000, 65/1000 in 2001, 45/1000 in 2002). Likewise, infant mortality declined by 65.8 percent (69.8/1000 in 2000, 37.1/1000 in 2001, 23.9/1000 in 2002). These declines in mortality can be attributed in part to changes in behavior from baseline to final, summarized below: Indicator Baseline Final ITN use among under-fives 0% 85% Treatment seeking for Malaria within 24 hours 28% 90% Treatment seeking for suspected pneumonia within 24hours 2% 60% ORT use to treat diarrhea 53% 94% Child given additional food for two weeks following diarrhea 19% 87% Malnourished children who received nutritious weaning foods after counseling 50% 97% Complete immunization coverage in children 12-23 months-old 74% 91% Delivery with trained provider 65% 88% World Relief Mozambique CSXV Final Evaluation Report 2003 5 The evaluator strongly recommends three actions in light of the termination of present funding at the end of September 2003 and the project’s potential for scaling up if momentum can be maintained: 1) Emergency funding is needed for FY 2004 to maintain momentum. 2) The process of Scaling Up should start with building capacity for a learning center in Chokwe District, also during the next year. 3) Phased expansion should then start in northern districts of Gaza Province where commitments have been made earlier. Then progressive expansion should follow systematically. 2. Results: Summary Chart Objectives and Target Indicator Baseline Final Nutrition: 1a. Increase from 75% to 85% the number of children 0-35 months weighed at least once during last 3 months. 1a. Children 0-23 months with documented weighing at least once in last 3 months (by card). 75% 90% 1b. Increase from 22% to 80% the number of malnourished children's mothers who received nutrition counseling 1b. Mothers of malnourished children 0-23 months who received nutritional counseling by volunteers, animators, or MOH personnel. 22% 92% 1c. Increase from 50% to 70% the number of malnourished children who received nutritious foods/enriched porridge 1c. Mothers of malnourished children 0-23 months who receive nutrition counseling and stated they give nutritious weaning foods/ enriched porridge to their child at least once per day. 50% 97% 2. At least 70% of children who completed the HEARTH program achieve and sustain adequate or catch-up growth per month during at least 2 months after HEARTH. 2. Malnourished children 0- 3 years old who enter 12 day Hearth program and complete 9-12 days. 2b. Children who complete Hearth program and achieve adequate or catch-up growth. 2c. Children who continue to gain weight at a rate equal to the international standard for their age at 6 months and 1 year following the Hearth. NA 73% World Relief Mozambique CSXV Final Evaluation Report 2003 6 Maternal Care: 3. Increase from 45% to 70% the number of mothers who eat the same amount or more food during pregnancy. 3. Pregnant mothers who say they ate the same amount or more food now as they did before their pregnancy. (For KPC only: the mothers of children 0-23 months who report that they ate the same amount or more food during their last pregnancy than they did when they were not pregnant.) 45% 82% Malaria and Pneumonia: 4a. Increase from 28% to 75% the number of children treated within 24h for fever (suspected malaria) at any health facility. 4. Children 0-23 months old who had fever (suspected malaria) in the past 2 weeks that were treated within 24 hours at appropriate health facility. 28% 90% 4b. Increase from 0.3% tot 70% the proportion of children under 5 who use insecticide treated nets year round. Percentage of children <2y who slept under an insecticide treated net the previous night. 0% 85% 4c. Increase from 2% to 50% the number of children treated within 24h for rapid, difficult breathing at appropriate health facility 4b. Children 0-23 months old who had cough/ rapid, difficult breathing, with or without fever (suspected pneumonia) in the past 2 weeks that were treated within 24 hours at appropriate health facility. 2% 60% 4d. Increase to 100% the health facility-based providers who have received continuing education in malaria and pneumonia protocols during the project. (MOH Capacity Building) 4c. Number of Health facility-based providers who received continuing education in malaria and pneumonia protocols by the Midterm Evaluation of the project. 0% 100% Diarrhea Case Management 5a. Increase from 53% to 80% the proportion of children with diarrhea treated with ORT by mothers/volunteers. 5a. Children 0-23 months with diarrhea in the past 2 weeks who received ORT. 53% 94% 5b. Increase from 19% to 65% the proportion of mothers who give extra food to children for 2 weeks following diarrhea. 5b. Children 0-23 months who received extra food after their last diarrheal episode. 19% 87% Reproductive Health: 6a. Increase from 0.3% to 50% the number of mothers who know 3 ways to prevent transmission of STI including HIV/AIDS. 6a. Mothers of children 0- 23 months who know 3 ways to prevent transmission of STI including HIV/AIDS. 0% 53% 6b. Increase from 1.6% to 50% the number of women and partners who recognize at least 3 common symptoms of STI. 6b. Mothers of children 0- 23 months who know at least three common symptoms of STI other than HIV/AIDS. 2% 59% World Relief Mozambique CSXV Final Evaluation Report 2003 7 6c. Increase from 0.3% to 50% the number of women and partners who recognize that STI increases risk of HIV/infertility. 6c. Mothers of children 0- 23 months who know that STI increases the risk of HIV transmission and infertility. 0% 52% 7. Increase from 65% to 70% the number of mothers who deliver child by trained provider. 7. Mothers of children 0-23 months who had their youngest child delivered by a trained health worker (includes trained TBA.) 65% 88% 8. Increase from 7% to 20% the number of women who are using a modern method of birth spacing. 8. Mothers of children 0-23 months who are not pregnant and who are using a modern method of birth spacing. 7% 29% Immunization Immunization was not a project objective but was promoted and tracked. Children between 1 – 2y that were completely immunized. 74% 91% SUSTAINABILITY & CAPACITY BUILDING: 1. MOH / District Level 1a. All beneficiaries will have access to primary health care centers. At least 95% of beneficiaries in the project area will live within 5 km of a health facility staffed with trained personnel and equipped with essential supplies. 65% 99% 1b. MOH personnel at district level will creatively address shortages of materials to provide continuity of services. Chloroquine will not be out of stock for more than three days per month in at least 90% of health posts. Percent of health posts with chloroquine stockouts less than three days per month. 80% 98% 1c. All Socorristas will receive regular supervision. 75% of Socorristas will have received a supervisory visit from Chokwe District staff or Vurhonga project director within the previous quarter. 0% 85% 1d. Community groups will maintain linkages with the MOH. 70% of VHCs will have met at once during the last two months. Health Post staff will attend at least 80% of the meetings of VHCs 0% 0% 91% 95% 1e. Community members will expect and demand provision of basic services. 1. 50% of beneficiaries will report that they are satisfied with their last visit to an MOH facility as measured by a satisfaction index NA 99% World Relief Mozambique CSXV Final Evaluation Report 2003 8 2. Community–Level 2a. VHCs will monitor, plan and evaluate maternal and child health for their areas. 1. 95% of villages will have established VHCs by EOP. 2. 70% of VHCs will meet at least once in the last two months. NA NA 1. 100% 2. 91% 2b. Care Groups are thoroughly trained in CS intervention. 100% of Care Groups will “graduate” during the fourth year of project. NA 100% 2c. Churches in the project area will actively promote behavioral changes specified in the interventions. 1. The monthly attendance rate at church leaders Care Group meetings will be at least 60%. 2. 50% of mothers who attended church during the past month report that they heard a health message. NA NA 1. 70% 2. 72% Volunteer and Family Level 3a. Volunteers in the Care Groups will develop a sense of efficacy and pride in promoting changed behavior. 1. At least 60% of families have been visited by their volunteer during the past two weeks. 2. The attrition rate of volunteers for reasons other than death, disability, or movement out of the project area will be less than 10%. NA NA 1. 92% 2. 3% 3b. Families will collectively demonstrate and sustain positive health behaviors Use indicators for specific CS interventions, e.g., participation in growth monitoring, EPI coverage, etc. 74% 78% EPI=91% GM=90% 3. Results: Technical Approach Strategy The Vurhonga II Child Survival XV Project (1999-2003) covers the population of 130,000 living in the rural area of the Chokwe district in Gaza province of Southern Mozambique. The goals of the project are to (1) reduce morbidity and mortality in children under five and women of fertile age; (2) strengthen the capacity of the Ministry of Health (MOH) to implement child survival interventions; and (3) empower communities to improve their health. The local partners are the district-level management team and the health centers of the MOH and the churches and communities of rural Chokwe. The main strategy of the Vurhonga II project is to help all the communities in the impact area to create Care Groups (CG) of volunteer mothers, and then train these volunteer mothers and the church leaders in their communities as behavior change agents. The child survival behaviors that the project promotes are World Relief Mozambique CSXV Final Evaluation Report 2003 9 related to general nutrition (30%), malaria (20%), pneumonia (10%), diarrhea (10%), maternal and newborn care (10%), child spacing (10%), and STI/HIV/AIDS prevention (10%). The project also helps the MOH to expand its network of health centers and health posts. Specific project objectives are summarized in the table above and a more extensive program description, excerpted from the project’s DIP, can be found in Annex A. Progress report by intervention area Vurhonga 2 achieved or surpassed all intervention objectives, as measured by baseline and final KPC surveys. The complete KPC reports can be reviewed in Annex E. The outstanding accomplishments resulted from a confluence of factors. Project staff and subsequently volunteers hold tremendous credibility in the community. They have fostered a sense of ownership and responsibility for health in the community that has taken root in village health committees, village leadership and care groups of volunteers and church leaders, increasing dialogue within individual communities and with the government health services. Points relevant to particular interventions follow: Nutrition: The dramatic results of the Hearth community-based nutrition rehabilitation intervention helped people to overcome traditional beliefs that poor growth is of spiritual rather than nutritional origin; when they saw how the children responded positively to the enriched porridge (maize meal enriched with groundnuts, oil, greens and oil) they were consuming as part of Hearth, they were able to see how they could positively impact the health of their children by feeding them locally available foods. One grandmother explained that feeding the porridge “even helped those children whose spirit had been sat upon.” Once caretakers understood the relationship between food and growth (and believed it), they were motivated to improve their child feeding practices. Malaria and Pneumonia: The malaria intervention received a significant boost when ITNs were distributed to every household as part of relief activities following massive floods in 2000. Prior to this, ITNs were not available and use was nonexistent. With education, usage rates were as high as 95%. Usage had seasonal variation based on perceived need, decreasing each year during the period from June through August— though the final KPC conducted in July still had rates as high as 90%--nearly four years after their initial distribution. Improved treatment seeking for malaria and pneumonia was facilitated by the training and placement of first-aid trained socorristas in villages such that all villages were within 5K of a health post with access to chloroquine. The socorristas were trained by the MOH with assistance from WR and are managed by village health committees. As caretakers learned the causes of the symptoms of malaria and pneumonia and saw that they could be treated effectively at the health post/center, they stopped choosing to seek treatment with traditional healers, including Zionist pastors who previously would admit patients to their homes, delaying treatment with drugs. Many Zionist pastors were among those trained in the care groups for church leaders. They learned to refer patients with World Relief Mozambique CSXV Final Evaluation Report 2003 10 suspected malaria and pneumonia—though as one Zionist pastor acknowledged during the final evaluation, he does not really have to refer people because the caretakers are already choosing the health post over admission to their house. The pastor interviewed was not bitter about the decrease in business, rather he remarked how effective the treatment at the health post was for such ailments—and that it was helping to save children in the village. Diarrhea: The project promoted use of ORT rather than ORS because of its improved accessibility. The approach was effective. Child spacing: The CSP found it effective to introduce the idea of child spacing in the context of teaching about nutrition. As people came to understand the relationship between how they were feeding their children and children’s growth, they saw the importance of spacing births so that each child would have access to adequate resources for good nutrition. As the husbands of a large percent of women work in South African mines and are not present for the majority of the year, child spacing methods are only seasonally relevant for much of the population. However, messages on HIV prevention and recognition and treatment of STIs are especially important for the same reason. STI/HIV: The CSP initially experienced difficulty in raising knowledge with regard to STIs using verbal communication; it was reluctant to use pictures that were overly graphic as it was thought that they would not be accepted in the village, particular as used by church leaders. However, when it was clear that verbal communication was insufficient for teaching about signs and symptoms, the project staff worked in partnership with church leaders to develop materials that would both communicate the necessary information without being offensive to the point that people would be reluctant to use them as a teaching aid. This joint development of materials was effective both in ensuring buy-in from church leaders as well as creating visual aids that were locally acceptable and effective, as demonstrated by subsequent improvements in knowledge related to STIs and HIV. 4. Cross Cutting Approaches A. Community Mobilization The Vurhonga 2 approach to community mobilization based on care groups of mother￾volunteers, grandmothers and church leaders as well as village health committees has been extremely effective at community mobilization. The project recruited and trained 2350 volunteers with a 97% retention rate over the four￾year period. The volunteers truly are active in conducting home visits as evidenced by household survey results reporting that 92% of households had received a visit from their volunteer in the prior two weeks and as seen in changes in the community. Corroborating these results of volunteer activity, one team during the final evaluation interviewed 21 women who were waiting with their children to be attended at Manjangue Health Center. Nineteen of the 21 caretakers had been visited by their volunteer within the last week and World Relief Mozambique CSXV Final Evaluation Report 2003 11 six said that the volunteer had visited daily. The two who had not received visits had inquired about the volunteer. One had found that the volunteer had been sick and so went to visit the volunteer. The other, who had not received a visit in about a month, inquired and found that the volunteer was visiting her husband in South Africa. Special care groups for grandmothers were an innovation in this project that later was deemed unnecessary. Involving grandmothers and soliciting their endorsement of new behaviors was pivotal to behavior change—but in the end project staff felt that the grandmothers’ need for involvement could just as well be met by including them during the regular home visits of volunteers and an occasional special meeting, rather than through the monthly meeting of specialized care groups. Church leader care groups received the same teaching as care groups of mother￾volunteers, with the expectation that church leaders would share consistent messages with the members of their congregations during their meetings. Additionally, when summoned for help or prayer, they could also council families on the importance of seeking physical treatment from the health post. The church leader care groups were very effective in promoting community dialogue and unity across denominations. During the final evaluation, a group of church leaders representing Catholics, Seventh Day Adventists, Pentecostals, Presbyterians and Zionists from the same large village stated that they hadn’t even known each other existed before, but now they saw how much they had in common and were able to support and encourage one another in their work. They lamented the challenges of reaching youth in a rural culture that despite its isolation, is being penetrated by western media and differing values that challenge tradition. The pastors asserted intent to continue to meet together for mutual support and to discuss common issues. Village health committees are an essential community structure for mobilization and sustainability of essential program activities. VHC membership includes the Chef de Saude, the village Socorrista, a trained TBA, leaders representing each neighborhood, one or more care group leaders, and a representative of the church leaders and traditional healers. The VHC is a forum for discussion of the data collected as part of the C-HIS, as compiled by the socorrista for all care groups in the village. Extending the discussion of local data from the all female care groups to the VHCs, made of a cross section of village leadership, enables more community wide commitment and action in response to health information. The CSP has provided leadership training to the VHCs, helping the members to understand their respective roles, how to interpret data from the C-HIS, and recourses for action. Members of VHCs met with during the final evaluation, in commenting on the impact of Vurhonga in their village, commented that they had “learned how to talk to one another” with the guidance of their animator. In the same discussion, members stated that they intended to continue meeting and expressed that it would further encourage them if they were to be part of a larger association of VHCs. VHCs have created self-imposed regulation of latrines in their villages, set fee schedules for their respective socorrista, and even initiated Hearth nutrition rehabilitation sessions in response to growth monitoring data showing that a some children in the village were not growing well. World Relief Mozambique CSXV Final Evaluation Report 2003 12 One particular example that underscores the effectiveness of the project’s community mobilization, in contrast to other districts, were the high retreatment rates of ITNs. At the final evaluation dissemination meeting in Maputo, a Unicef representative remarked off the record that he knew that Vurhonga was different from other programs. Unicef/Roll Back Malaria had contracted with a number of NGOs to promote the sale of retreatment kits in various districts and compensated their organizations to do so—with coverage rates well under 20% in most districts. In contrast, World Relief, with no extra compensation, was able to tap into its community networks such that 80-95% of ITN were retreated per village. B. Communication for Behavior Change From a technical perspective, Vurhonga 2’s behavior change communication (BCC) strategy uses cascade training to transmit health messages using picture codes, stories songs, dance and interpersonal communication, from the community to individual household level. The printed materials have all been developed locally using the local language and are produced on durable, water resistant materials. Clearly the BCC has been effective—as evidenced by achievement of behavior change indicators and by the transformation evident in the villages. Of the 21 mothers who were interviewed at Manjangue health center, all those with feverish children cited the instruction of the volunteer as the reason for her coming for care. It was clear during focus group discussions in the villages that everyone knew about things like oral rehydration and how to prevent and seek appropriate care. In fact, focus groups almost seemed to think questions about prevention and treatment of illness were nonsensical—there was an attitude that “of course everyone knows this is what we do now.” Participants were much more enthused to talk about the transformations they have experienced as a result of this knowledge: “Babies aren’t dying now like they used to.” Another participant explained, “Now that we know what causes malaria (mosquitoes, not bewitching) we don’t have to go to the witch doctor to find out who cursed the child. There is no need for revenge, only treatment (at the health post).” More important than the cardstock of any illustrated picture in facilitating effective BCC is the relationship of trust, care and mutual respect that the Vurhonga staff has established in the villages. Community members repeatedly referred to the positive relationship they have with “their” animator, and mentioned that she lived right there in the village with them—a tangible sign of commitment and common perspective. The BCC approach used multiple channels to transmit consistent health messages, via the volunteers, grandmothers and church leaders. Care groups were verbally quizzed on their knowledge and had to achieve a collective passing score in order to “graduate” as a group. Community pressure for compliance has not only contributed to high rates of adoption of new behaviors but also improves the probability of their sustainability. The Location Secretary for Chati explained during the final evaluation that some of the volunteers who World Relief Mozambique CSXV Final Evaluation Report 2003 13 he had appointed were initially reluctant to take up their responsibilities. He said that he would counsel them and tell them of their responsibility to the community. The next step would be the convening of a community assembly to discuss the problem, though it was not clear of this was ever necessary. At the household level, the Location Secretary said it was always the husbands who did not want to receive the visits or follow the advice of the volunteers. In such instances, the volunteers would report it to the head of their care group who would report it to the Chef de Saude. The Chef would visit the man to discuss it with him and pressure him to comply. If the man was still stubborn, the Chef would report it to the Secretary who would also go to counsel (and pressure) the man. Again, a meeting with the village assembly was a last resort. As Secretary Albino explained, “We could not leave him (the reluctant husband) behind. He would poison the entire village.” It was interesting that the secretary specifically excluded the rough speech of a drunken husband to a volunteer from this list of offenses. The husband gets away with this unless he does it all of the time or continues to do it when he is sober. The Secretary said that most men were happy with Vurhonga, saying that so many would not have constructed latrines had they not been. In the case of reluctant women, volunteers would first seek out another volunteer to go with her for support when addressing the household and if necessary the care group leader before invoking male participation from the VHC. As part of the final evaluation, the Superivisors were asked to prepare stories, some of which contained errors about the messages that had been included in the interventions and some of which were accurate. The mothers keenly picked out the errors and right answers. The only statement that caused some debate was the interjection of the statement “And another reason that we use mosquito nets is to prevent the spread of AIDS through mosquitoes,” by a member of the evaluation team. There was some discussion because mosquitoes do spread malaria from one person to another and they also puncture the skin, perhaps bringing some of the blood from the HIV+ person. The issue was settled by saying that malaria can live in the mosquito and so can spread from one person to another but that AIDS cannot live in the mosquito so cannot spread that way. The forcefulness and nature of the mothers’ responses to false information, though in jest, nevertheless argued well for the sustainability of the changes. “We must teach you!” If that’s the way it is in your place, go back to it.” Do not give us poison!” In response to a question, the mothers said that the practices and knowledge would remain because they would teach it to their children from when they are young. At the time of the final KPC, the staff returned to the Vurhonga 1 project area to do a follow-up survey on the districts of Guija and Mabalane and found that nearly all tracked behaviors from the end of Vurhonga I had been maintained and some had even improved. Treatment seeking within 24hours for suspected malaria increased from 11% to 85% over the life of the project, and in the 4 years hence has varied between 80-93% in subsequent measurements. ORT use during diarrhea increased from 46% to 82% at the end of the project, and since has ranged from 84-100%. Family Planning use increased from 3% at baseline to 23% at end of project, and subsequently has ranged from 22% to 27%. World Relief Mozambique CSXV Final Evaluation Report 2003 14 Complete child immunization coverage increased from 37% to 93% during the life of Vurhonga 1 and in the four years since has ranged from 89-91%. These results reflect not only sustained behavior change at the household level, but also continued service provision on the part of the MOH. Everything in Vurhonga 2 points to similarly sustained results, further abetted by the earlier creation of VHCs that are stronger in the Vurhonga 2 areas than they were in Vurhonga 1, where they got a later start relative to the duration of the project. C. Capacity Building Approach C1. Strengthening the PVO Vurhonga has become WR’s ‘gold standard’ with regard to implementation of community based child survival programs, influencing every program that WR has implemented since. WR has adapted the care group model for training and supervising volunteers to CSPs in Cambodia, Malawi and Rwanda and is proposing further adaptations for use in an HIV/AIDS program in Haiti. Likewise, improvements in the C￾HIS and sampling methodology for local rapid assessments (randomly selecting one care group per Animator every 3-4 months) have been successfully applied to WR’s other CSPs. In addition to HQ staff disseminating lessons learned, staff members from three of WR’s CSPs have visited Vurhonga 2 for hands-on learning and exchanges of ideas. WR has shared lessons learned from the Vurhonga 2 CSP in presentations at meetings of Global Health Council, American Public Health Association, CORE and in classes at Johns Hopkins Bloomberg School of Public Health. C2. Strengthening Partner Organizations: Health Facilities and Health Worker Performance WR worked closely with the MOH and the 48 communities in Chokwe district in partnership with Vurhonga. Specific areas of capacity building with the MOH included C-HIS, monitoring and supervision, training of health workers in IMCI using videos in Portuguese from the Pan American Health Organization, and collaborative training of TBAs and socorristas. In the communities, WR provided extensive capacity building via the care groups of volunteers and care groups of church leaders (including those who function as traditional healers) and formation and training of village health committees. Though there was not a formal organizational capacity assessment, there is much evidence of improvements in the health system, the communities and how they work together. Improvement in objectives like rapid care seeking for malaria to a level of 90% did not happen because of the independent work of 30 WR Animators and Supervisors. Rather, such results reflect widespread changes in knowledge, community commitment to seeking care, improved access to quality care and mechanisms for problem solving. Access to care within 5 K increased to 99% with the training and placement of socorristas—whose successful functioning reflects community demand for services, World Relief Mozambique CSXV Final Evaluation Report 2003 15 technical training and supervision by the MOH including provision of drugs (stockouts of chlorquin less than 2%) and effective management by VHCs that select the socorristas, established their terms of service including fee schedule and address any community grievances regarding their performance. Ninety-nine percent of people who sought care from a health facility reported satisfaction with their most recent visit. There is a symbiotic relationship between demand for services and motivation of health care workers. At Mapapa, the nurse explained that prior to Vurhonga, he saw perhaps 400 patients a month—now he sees over 1000. To his satisfaction, people are coming in early enough for treatment to be effective. He stated that the increased demand has made him feel like he has an important job and his increased ability to provide effective treatment good about his abilities as a nurse. Furthermore, he said it is a lot more gratifying to work with people who have a basic understanding of hygiene and health— and with volunteers whom he can rely on to mobilize the community as needed. At Majangue Health Center, Sister Dalva Trinadade, a Brazilian nurse who was working at the center quickly recognized the work of the volunteers. She explained that the volunteers bring mothers and children to the health center. On Fridays, when the government is distributing milk and conducting supplementary feeding for children from at-risk families, the volunteers come to give nutritional instruction to the women and also assist with food preparation. The volunteers use that opportunity to teach about the importance of breastfeeding and the use of enriched porridge during weaning and afterwards. The volunteers also regularly help to clean the health center and the grounds—something that was never envisioned by WR as part of the volunteers’ responsibilities. Over the course of the program, rather than volunteer interest waning, it has actually increased as they have taken increasing pride in their work and the impact they can see it is having in the community. C3.Training The training strategy used by the CSP was highly effective, as demonstrated by the measurable difference in behavior change and child mortality, in addition to the commitment and testimonies given to the program activities during the final evaluation site visits. As each intervention was phased-in, the animators trained the care group volunteers in a new lesson each meeting; the volunteers then replicated the lesson during home visits to their neighbors. When interventions were later reviewed, the animators trained just the volunteer care group leader, and then supportively supervised her as she trained her care group. The animators presence during care group training ensured accuracy of message delivery and enabled her to coach the care group leader as needed, while building greater independence within the care groups to take responsibility for their own training. In the future, if the MOH has BCC to disseminate in the villages, they can share it with the Village Health Committee, including care group leaders who can then train their care groups with confidence. Each volunteer and pastor has a complete set of durable picture codes further protected by plastic sleeves and organized in a three ring binder. World Relief Mozambique CSXV Final Evaluation Report 2003 16 Additionally, the care group leaders have published training guides for maternal and child health. These materials are seen as village property; if a volunteer dies or moves from the area, her materials are passed on to a replacement volunteer selected by the village health committee. This self-sustaining system has been successful in the Vurhonga 1 project area and has every reason to continue in Chokwe as well. At the time of the midterm evaluation, it was noted that some members of the village health committee lacked sufficient technical knowledge of the child survival interventions and that ability to interpret the C-HIS data was mixed. In response, the animators placed more emphasis on introducing interventions to VHCs before they trained the care groups of volunteers, pastors and grandmothers in them. Additionally, the CSP continued to build the leadership capacity of the VHCs, with training in how to run meetings and on topics such as vision to enable them to set their own agenda and priorities for the future. The training of pastors was a unique feature of this CSP that appears to have tremendous impact on the transformation that took place in the villages. As mentioned, many of the pastors functioned as quasi-traditional healers; by all accounts in the village, it appears that people have stopped going to pastors for treatment before seeking care at the health post. Additionally, pastors are providing another conduit for sending consistent messages to reinforce those that are delivered by care group volunteers during home visits. Unfortunately, the project made no formal assessments of changes in pastor’s knowledge—though the personal testimonies in focus groups abound. 5. Sustainability Strategy WR’s sustainability strategy and exit plan was on-track for Vurhonga 2 at the time of the final evaluation. The program was designed so that paid WR staff could be removed, leaving stable community structures in place to continue with community monitoring and mobilization around health and other issues prioritized by the VHC. As indicated in the Objectives Table, Socorristas were trained and functioning in village health posts with adequate stocks and regular supervision by the MOH. The socorristas interviewed were satisfied with the fee for service they receive in return for their care and felt their services were both needed and appreciated in the villages. There is clearly strong community commitment to continuing as care group volunteers. Village health committees will provide the link between MOH and care groups. There is a high degree of community ownership of activities and evidence that the communities have a voice with the MOH. The MOH at district and province levels are highly appreciative of the work of the care groups and the data they generate through the C-HIS. Demand for clinical services has skyrocketed over the course of the program, to the delight of health center staff who report that they feel much more validated in their work than before, when the few patients who did come for care waited until it was too late to be of assistance. The communities have already demonstrated an ability to communicate World Relief Mozambique CSXV Final Evaluation Report 2003 17 their rights and needs for quality care to the MOH via the VHCs. In one example, C-HIS data from the project showing an increase in diarrhea was able to help the district anticipate and respond to what turned out to be the beginning of a cholera epidemic. While there is no doubt that the impact and key activities of Vurhonga 2 will continue in Chokwe District, of considerable concern is the loss of opportunity to use this highly equipped team and effective program as a training center for scaling up activities to the rest of the Province, if not the country. As of the final weeks of the project, it was feared that most paid staff would have to be let go, as USAID had not funded WR’s application for an expanded impact grant to leverage the existing strategies for implementing C-IMCI to additional districts. However, Unicef took interest in the dramatic decrease in under￾five mortality in Vurhonga 2 and evidence of sustained behavior change from Vurhonga 1 and has since funded activities for an additional year, expanding to a new district and revisiting the communities from Vurhonga 1 to introduce new education materials and interventions developed in Vurhonga 2. The funding from Unicef is essential to keep the momentum going that WR has generated in Chokwe, until it can be used as a training center for scaling up this approach. 6. Program Management A. Planning The planning process was lead by the program manager in consultation with the other staff, partners and headquarters. Planning with the community and partners was of course restricted to focusing on the breadth of possible activities covered by the grant. In the case of this program, DIP writing was initiated in 2000 and then interrupted by massive flooding and repeated the following year, just months before the midterm evaluation. After adjustments were made in the DIP to take into account relief activities following the flood, plans largely went according to schedule and as anticipated. As the staff had all worked together on Vurhonga 1, they were highly experienced and had realistic expectations of what they could accomplish according to timeline. In fact, the DIP for this project was highly accurate in its description of how the project was in fact executed. B. Staff Training As a testament to effective staff training and capacity building, it is noteworthy that none of the Vurhonga staff aside from the physician program manager have completed secondary school. They are local women who had limited opportunities for formal education but have flourished with the training they have received on the job, beginning with Vurhonga 1. They have learned to become effective agents of change in the communities where they are working as well as to manage the data the project collects for monitoring and evaluation. The final KPC survey (the staff’s fifth, counting surveys at baseline, midterm and final of Vurhonga 1) was lead nearly independently by the program supervisors who consulted with the Program Manager only when they had questions to resolve and for quality control. During the majority of the final evaluation, World Relief Mozambique CSXV Final Evaluation Report 2003 18 the program manager was absent from the country due to health reasons; it was impressive to watch the supervisors and animators organize themselves in his absence and facilitate all of the logistics for the field visits. The supervisors also helped present program results at the final dissemination meeting in Maputo, as did Mr. Inacio Xithlangu from the district MOH. C. Supervision of Program Staff The supervisory system for Vurhonga staff was highly effective: regular meetings for joint problem solving, supportive supervision and annual 360-degree staff evaluations were all a regular part of the CSP. The VHCs have demonstrated capability for handling management issues related to the socorrista from their village and as demonstrated by the sustainability objectives, the MOH has provided regular technical supervision of the socorristas. To date, the MOH has benefited from transportation provided by the project to the various villages. D. Human Resource Management The morale, cohesion and working relationships of program staff are a remarkable aspect of this CSP. Their commitment to one another, to their volunteers and to the program beneficiaries extends far beyond that of a job. The staff members take pride in their work and share in a spiritual as well as professional dedication to what they do. As they gather for meetings, they sing one or more of the numerous antiphonal songs they have written on child survival themes and seek resolution of occasional disagreements in prayer, diffusing tensions in doing so. They are an empowered group of women whose sincerity and enthusiasm have been very effective in empowering the people in the villages where they work. During site visits, project staff regularly were spoken about with admiration and appreciation for their commitment. Staff commitment and high morale is also reflected in the very low turnover rate during the life of the project. In fact, most staff members have been employed for eight years, since the beginning of Vurhonga 1. The most capable animators from Vurhonga 1 were promoted to fill additional supervisory positions created at the beginning of Vurhonga 2, rewarding those staff members for their good record and leadership skills. Regrettably, a few staff members have been lost to death during Vurhonga 2. Obviously the project was successful in maintaining quality implementation as all activities were carried out as planned. The majority of staff who desire to continue with World Relief are being retained at the conclusion of funding from USAID, as Unicef has agreed to fund work in other neighboring districts. For those people who will have continued duties as volunteers and village health committee members, their roles are clearly defined. The evaluator recommended that they plan for rotating replacement of VHC membership in the future. Clearly posted on the wall of health posts are the responsibilities of the Socorrista, as upheld by the VHCs. World Relief Mozambique CSXV Final Evaluation Report 2003 19 E. Financial Management The Vurhonga 2 project in Chokwe relies on the timely financial feedback it gets from the WR financial department in Maputo in order to effectively plan activities. Timely feedback was a problem at the beginning of the project but has significantly improved over the last two years of the project. F. Logistics The CSP has successfully overcome any obstacles to smooth logistics due to careful planning, as is evidenced by the program’s successes. The only possible weak link that may surface after project end involves that of transport for MOH employees who were able to coordinate rides with the project staff during the life of the program. However, the project did turn over a motorbike to the MOH to facilitate continued supervision. G. Information Management WR has a very effective system for information management developed and improved during the Vurhonga 1 & 2 CSPs. The system captures information at regular intervals that is shared at the community level and used for decision making within the community, the MOH and the project as a whole. A description of the health monitoring and information system (HMIS) can be found in Annex A. WR presented a paper at the CORE Data for Action Workshop in September 2002 on the Vurhonga HMIS. In addition to the description of the system, the paper also gives examples of how the community, project and MOH have all taken action in response to data captured as part of the HMIS. The person in charge of statistics for the District MOH was invited by the Province to present the HIS and how he is using it at a Provincial meeting of the MOH. The project conducted a census-based survey to retrospectively ask mothers about births and deaths that took place in one year intervals from March 2000 – February 2001 and March 2001-February 2002. March was selected as the starting month so as not to cloud results with deaths that may have occurred as a direct results of the massive flooding in February 2000. A third year of data was generated from the project HIS and followed up on by the Animators in each village to verify births and deaths. During this three year period, under-five mortality decreased by 62%, (119/1000 in 2000, 65/1000 in 2001, 45/1000 in 2002). As part of the final evaluation, the staff interviewed 250 women about their pregnancy histories, using a convenience sample from 10 randomly selected villages. The results corroborated those data already collected by the project. The project staff hand-tabulated the results of the survey, and expressed great jubilation at the significance of the results. Staff Supervisors participated in the final evaluation dissemination meeting in Maputo. World Relief Mozambique CSXV Final Evaluation Report 2003 20 The monitoring system used by Vurhonga has been adopted by all of WR’s ongoing USAID-funded child survival programs. H. Technical and Administrative Support The project management for Vurhonga 1 & 2 has very effectively incorporated the technical assistance provided to the program from WR HQ health unit staff, visiting consultants, and advice from USAID and technical reviewers as part of the DIP review process. The project’s commitment to ongoing learning and improvement are evidenced by programs that have adapted to an environment beset by natural disaster and developed innovations adopted by other CSPs. I. Management Lessons Learned The Vurhonga 2 CSP is a model for good management of community-based health programs. The management has struck a balance between efficiency and responsiveness to community needs that is commendable. All staff members understand the overarching goals of the program and the role that they play to contribute to program success. Group consensus building both among staff members and among community members has created a cohesive movement for social change that is evident in staff morale and community sentiment. 7. Meeting with Prime Minister Dr. Pascoal Mucumbi Prior to undertaking the trip to Mozambique the Evaluator was informed about the Prime Minister, who was the former Minister of Health and Chairman of the WHO Global Health Care Forum. When he realized the magnitude of child mortality decline after starting to review the evaluation data it was recommended that the information should be communicated to the Prime Minister. Appointments were made to meet with him prior to the Workshop in Maputo for Mechanisms for sustaining and scaling up community based programs and to demonstrate the results of the CSP in Mozambique. (Annex E). The Prime Minister was most gracious and deeply interested in learning about the program achievements. It was obviously exciting to him that he had in Mozambique a project that had produced the remarkable changes that have become apparent in Chokwe. It was especially evident that he was intrigued by the fact that he and the President are natives of this province and this had happened without either he or anyone else being aware of what was going on. He knew that this kind of child survival success has been very rare in the world in spite of all the investments. It was natural that he was skeptical and said he would not believe it until he had a chance to see it in the villages for himself. He immediately understood the description of the reality that what is most unique about the Chokwe project is what the evaluator called the Chokwe Spirit, a sense of excitement and enthusiasm that permeates all activities, the fervor of “true believers” who support each other, awareness that their “we can do it” dedication has produced successes that World Relief Mozambique CSXV Final Evaluation Report 2003 21 they themselves feel. They know intuitively that this results from behavior change and new social norms much more than just implementing technical interventions and scientifically designed models. We need to change our traditional perspective even in activities like this “evaluation” and look more carefully at outcomes that are not overtly measured by the usual process indicators of Child Survival Projects. The fact is that these intermediate indicators are used mostly because the managers responsible know that the important outcomes of real child survival usually take time. But Chokwe has shown that 62 % decline in 0-5 child mortality can occur in just two years of field work. As the evaluator described the Scaling Up process described in his book Just and Lasting Change: When Communities Own Their Futures, he immediately got the point that this is not something that can be put into a blueprint and managed. He agreed strongly that just trying to replicate Chokwe in a government run mass program would destroy what was special and essential for producing this kind of dramatic benefit. We must not let those who are from outside the community take ownership away from the people. He immediately recognized the relevance of the analogy of riding a bicycle, which is that forward momentum must be maintained; otherwise the whole movement collapses when a gap in action occurs. Many other projects around the world have shown that a community based movement is inherently fragile and the empowerment process can be stopped abruptly when professionals and officials take over and take credit. 8. Summary Statement of Conclusions Anyone who has visited the Vurhonga child survival project in Chokwe District and taken the time, with all appropriate skepticism, to rationally judge these results has little problem accepting the quantitative findings of the remarkable achievements. Particularly to get almost 50 % increase in child survival the ultimate outcome in just one year and 62 % in two years is such a dramatically unexpected outcome that it was not even specified in the expected list of outcomes. As in all USAID child survival projects a great deal of effort has gone into careful quantification of process indicators. Most of these have dramatically increased to 90-97 percent coverage and almost all have increased to over 50 percent, as shown in the body of this report. The main technical feature of this success is that by integrating interventions they have achieved the great benefits of synergisms when services are brought together by communities, illustrating the reality that a whole functioning system can be greater than the sum of the component parts. A more important reality is that no numbers can quantify the sense of enthusiasm and joy that now permeates village life. A palpable sense of self-reliant social competence has reached a tipping point in reflecting high-level community empowerment. People, and this includes fathers as well as mothers, say they don’t need data to know that babies are no longer dying as they used to. All those participating in the project do, however, enjoy World Relief Mozambique CSXV Final Evaluation Report 2003 22 getting good information in order to find out what can still be improved. There is no question that it is the communities who have produced and therefore own the process of empowerment under the guidance of World Relief staff. This project can provide a foundation and framework for rapid expansion. For the future, the great challenge now is Scaling Up this dramatic success. Rather than the usual process of developing a management blueprint to implement a mass government approach of extension, a new alternative is available. A phased effort can implement a new methodology for Going to SCALE. The unique success in Chokwe can be transformed into a Self-help Center for Action Learning and Experimentation (SCALE center). Rather than moving staff to try to start over in new districts, the SCALE Center provides practical training for other communities and districts. The reason for this is that community empowerment depends more on what the village people feel and do as on staff. Continuing the momentum for improving child and maternal health is like riding a bicycle, once momentum stops, people fall off. Once several SCALE centers have been established, the process then shifts to exponential expansion since a network of centers can provide multiple nodes for growth. At the same time at national level, an enabling environment of systems to guide the process of extension is established to provide participatory training, experimentation, funding and supportive regulations and legislation. 9. Recommendations 1) Funding for the project was scheduled to terminate at the end of September 2003. Emergency funding was needed to maintain momentum. Discussions were started during our meeting in Maputo reporting to international agencies on the remarkable outcomes from this project and UNICEF spontaneously agreed to talk about immediate and interim support. (Outcome: Unicef has committed to one year of funding for work in new areas and introduction of new health education modules in the Vurhonga 1 communities.) 2) The process of Scaling Up should start with building capacity for a learning center in Chokwe District, during the next year. Chokwe can be the first demonstration of a SCALE Squared Center for Action Learning and Experimentation. This can be the model for developing a network of such centers that could rapidly be extended to national coverage in a process that has been defined as going to SCALE Cubed. 3) Phased expansion should start in northern districts of Gaza Province where commitments were made last year as part of World Relief’s Expanded Impact Proposal to USAID. Then progressive expansion should follow systematically within a framework with very unusual potential for national scaling up, especially because of the great enthusiasm expressed by the prime minister. For USAID this could demonstrate not only that child survival works but even more urgent is the unique demonstration of community based success through community action of World Relief Mozambique CSXV Final Evaluation Report 2003 23 amazing results in prevention and care for HIV/AIDS, Malaria and TB. The mobilizing of communities includes remarkable organization of women’s associations who have gotten village pastors into cooperation in pastors’ committees that are promoting the behavior and value changes that are so evident. Similarly, leaders and men’s groups have been mobilized by the women so the whole community is truly involved. 4) If action is delayed, a tremendous opportunity will have been missed at this, the most fragile, stage in Scaling Up. Empowerment will collapse if government and public health officials move in with the usual Blueprint Approaches of making vertical program impose the regulations of experts. If the communities can retain ownership with current demonstrations of self-reliance, self financing and sustainability a model will be established that can be a beacon of hope that is badly needed in Africa at this time. 10. Results Highlight World Relief’s Vurhonga II Child Survival Project in rural Chokwe District (population 140,000) of Gaza Province, Mozambique, has unique potential for becoming a world￾class demonstration of success in child survival. In the first three years of the project, 0-5 child mortality declined by 62.2 percent (119/1000 in 2000, 65/1000 in 2001, 45/1000 in 2002). Likewise, infant mortality declined by 65.8 percent (69.8/1000 in 2000, 37.1/1000 in 2001, 23.9/1000 in 2002). Rapid results were facilitated by team experience in neighboring districts during Vurhonga I. Birth and mortality data from 2000-2001 were collected retrospectively in a survey that included every household in the project area. The addition of data from 2002 was based on monthly reporting by project volunteers and verified by paid health animators. To confirm these dramatic trends, the external evaluator lead a special survey using a new methodology based on the pregnancy history module from the Demographic and Health Survey (DHS). Careful quality control of data gathering permits calculation of birth and death rates for up to 10 years retrospectively. Twenty-five animators conducted a total of 250 pregnancy histories over 2 days in 10 of 47 project villages, yielding data on all births and deaths of children under age five. None of the interviewers collected data from their own work areas. Results were analyzed by separating data from before and after 2000, a year marked by the worst flood in living memory and thought to have unmeasured influence on both births and deaths. The analysis therefore covers the two￾year period of 1998 and 1999, the single year 2000 and the two-year period 2001 and 2002. The 0-5 mortality for 1998-99 was 128.2/1000 then spiked in 2000 to 189.7/ 1000, and dramatically decreased to 30.2/1000 in 2001-02. This represents a decline of 76.4 percent over the five-year period. The infant mortality rates were as follows: 59.8/1000 in 1998-99 and 59/1000 in 2000. In 2001-02 there was only one infant death out of 132 births. The small numbers make it inappropriate to estimate a percentage reduction figure in this small a population, however by adding the cluster of deaths in the first eight World Relief Mozambique CSXV Final Evaluation Report 2003 24 months of 2003 with a total of 179 live births, the infant mortality equals 11.2 /1000, an 81.3 percent reduction in 5+ years. Though the results of the special survey are based on only 250 interviews, they do confirm the trends in mortality reduction observed above and invite formal measurement on a larger scale. These quantitative estimates or mortality decline are supported by achievement of project objectives. Most project indicators have increased to 90-97 percent coverage, including complete immunization, use of insecticide treated nets, and treatment seeking for malaria and pneumonia within 24 hours, for example. By integrating interventions and increasing access to quality services, the project has achieved synergistic effects. Most important, however, is the reality that no numbers can quantify: the sense of enthusiasm and joy that now permeates village life. A palpable sense of self-reliant social competence has reached a tipping point in reflecting high-level community empowerment. Fathers as well as mothers say they don’t need data to know that babies are no longer dying the way they used to. They do, however, enjoy participating in getting good information in order to find out what can be improved. There is no question that it is the communities who own the process of empowerment. World Relief Mozambique CSXV Final Evaluation Report 2003 25 11. Annexes A. Summary Information from Project Detailed Implementation Plan 1. Field Program Summary Potential Beneficiaries There are 53,418 potential beneficiaries for Vurhonga II, including 31,338 women age 15-49 years old and 22,080 children less than five years of age. Table 1: Vurhonga II Intervention Mix Component Child Survival Intervention: Level of Effort Nutrition General Nutrition: 30% CDD Control of Diarrheal Disease 10% Malaria and Malaria Control 20% Pneumonia Pneumonia Case Management 10% Maternal and Newborn Care 10% Child Spacing 10% Reproductive Health STI/HIV/AIDS Prevention 10% 2. Program Goals and Objectives Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities Nutrition: 1a. Increase from 75% to 85% the number of children 0-35 months weighed at least once during last 3 months. 1a. Children 0-23 months with documented weighing at least once in last 3 months (by card). 1. Project quarterly Monitoring and Evaluation (M&E) Plan. (See description in monitoring and evaluation section.) 1. Volunteers trained to assist MOH staff at monthly EPI/GMC sessions, mobilize community for attendance. World Relief Mozambique CSXV Final Evaluation Report 2003 26 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities 1b. Increase from 22% to 80% the number of malnourished children's mothers who received nutr. counseling 1b. Mothers of malnourished children 0-23 months who received nutritional counseling by volunteers, animators, or MOH personnel. 1c. Increase from 50% to 70% the number of malnourished children who received nutritious foods/enriched porridge 1c. Mothers of malnourished children 0-23 months who receive nutrition counseling and stated they give nutritious weaning foods/ enriched porridge to their child at least once per day. evaluation section.) 2. KPC surveys (conducted three times during project￾at the baseline, and midterm & final evaluations.) 2. Train MOH health center and health post staff to do nutritional counseling. 3. Volunteers trained to give nutritional counseling on home visits and at GMC sessions, especially to malnourished children. 4. Mothers trained by volunteers in how to make and why to give enriched porridge. 2. At least 70% of children who completed the HEARTH program achieve and sustain adequate or catch-up growth per month during at least 2 months after HEARTH. 2. Malnourished children 0-3 years old who enter 12 day Hearth program and complete 9-12 days. 2b. Children who complete Hearth program and achieve adequate or catch-up growth. 2c. Children who continue to gain weight at a rate equal to the international standard for their age at 6 months and 1 year following the Hearth. 2a. Hearth program registers data, constantly updated. 2b. Hearth program children follow-up weights taken 6 and 12 months after completing program. 1. Volunteers trained in Hearth methodology conduct 2 Hearth cycles per year in the first 2 years, repeat in Years 3 and 4 as needed. 2. Maintain high coverage in bimonthly GMC sessions. 3. Animators maintain Hearth registers until child turns 5. World Relief Mozambique CSXV Final Evaluation Report 2003 27 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities Maternal Care: 3. Increase from 45% to 70% the number of mothers who eat the same amount or more food during pregnancy. 3. Pregnant mothers who say they ate the same amount or more food now as they did before their pregnancy. (For KPC only: the mothers of children 0-23 months who report that they ate the same amount or more food during their last pregnancy than they did when they were not pregnant.) 1. Project quarterly M&E. 2. KPC surveys 1. Train volunteers and mothers about the importance of eating more during pregnancy. Malaria and Pneumonia: 4a. Increase from 28% to 75% the number of children treated within 24h for fever (suspected malaria) at any health facility. 4. Children 0-23 months old who had fever (suspected malaria) in the past 2 weeks that were treated within 24 hours at appropriate health facility. 1. Project quarterly M&E. 2. KPC surveys . 1. Volunteers and mothers trained in: a) recognition of fever (and other signs) of malaria and b) the importance of rapid (within 24 hours) treatment. 4b. Increase from 0.3% tot 70% the proportion of children under 5 who use insecticide treated nets year round. Percentage of children under two who slept under an insecticide treated net the previous night. 1. Project quarterly M&E. 2. KPC surveys 1. Distribution of ITN bednets 2. Retreatment of bednets 4c. Increase from 2% to 50% the number of children treated within 24h for rapid, difficult breathing at appropriate health facility 4b. Children 0-23 months old who had cough/ rapid, difficult breathing, with or without fever (suspected pneumonia) in the past 2 weeks that were treated within 24 hours at appropriate health facility. 1. Project quarterly M&E. 2. KPC surveys 1. Volunteers and mothers trained in: a) recognition of cough and fast, difficult breathing as signs of pneumonia and b) the importance of rapid (within 24 hours) treatment. World Relief Mozambique CSXV Final Evaluation Report 2003 28 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities 4d. Increase to 100% the health facility-based providers who have received continuing education in malaria and pneumonia protocols during the project. (MOH Capacity Building) 4c. Number of Health facility-based providers who received continuing education in malaria and pneumonia protocols by the MTE of the project. 1. Project training records. 2. MTE interviews with MOH staff. 1. Project Director (PD) trained in IMCI or equivalent. 2. PD gives in￾service training to Chokwe District MOH health providers in IMCI / malaria and pneumonia protocols. (assuming MOH Chokwe participates in IMCI protocols. Diarrhea Case Management 5a. Increase from 53% to 80% the proportion of children with diarrhea treated with ORT by mothers/volunteers. 5a. Children 0-23 months with diarrhea in the past 2 weeks who received ORT. 5b. Increase from 19% to 65% the proportion of mothers who give extra food to children for 2 weeks following diarrhea. 5b. Children 0-23 months who received extra food after their last diarrheal episode. 1. Project quarterly M&E. 2. KPC surveys 1. Volunteers train mothers to give ORT to children during diarrhea. 2. Volunteers train mothers to give extra food to a child recovering from diarrhea. 3. Assure ORS packet supply. Reproductive Health: 6a. Increase from 0.3% to 50% the number of mothers who know 3 ways to prevent transmission of STI's including HIV/AIDS. 6a. Mothers of children 0-23 months who know 3 ways to prevent transmission of STIs including HIV/AIDS. 6b. Increase from 1.6% to 50% the number of women and partners who recognize at least 3 common symptoms of STI's. 6b. Mothers of children 0-23 months who know at least three common symptoms of STIs other than HIV/AIDS. 1. Project quarterly M&E. 2. KPC surveys 3. District MOH statistics. 1. Couples trained in STI transmission and prevention. 2. Couples trained in a) how to recognize at least 3 common STI symptoms, b) importance of rapid treatment. 3. Couples trained in how STIs increase risk of HIV World Relief Mozambique CSXV Final Evaluation Report 2003 29 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities 6c. Increase from 0.3% to 50% the number of women and partners who recognize that STI's increase risk of HIV/infertility. 6c. Mothers of children 0-23 months who know that STIs increase the risk of HIV transmission and infertility. risk of HIV transmission and infertility. 7. Increase from 65% to 70% the number of mothers who deliver child by trained provider. 7. Mothers of children 0-23 months who had their youngest child delivered by a trained health worker (includes trained TBA.) 1. Project quarterly M&E. 2. KPC surveys 3. District MOH statistics. 4. Yearly client satisfaction surveys. 1. Workshops with MOH staff on QA (with TA from SEATS) to improve client perception of treatment at health centers. 8. Increase from 7% to 20% the number of women who are using a modern method of birth spacing. 8. Mothers of children 0-23 months who are not pregnant and who are using a modern method of birth spacing. 1. Project quarterly M&E. 2. KPC surveys 3. District MOH statistics. 1. Couples trained in benefits of spacing births and methods available. 2. Train Socorristas as CBD to improve access 3. Village leaders and pastors address men in community about Child Spacing. World Relief Mozambique CSXV Final Evaluation Report 2003 30 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities SUSTAINIABILITY & CAPACITY BUILDING: 1. MOH / District Level 1a. All beneficiaries will have access to primary health care centers. At least 95% of beneficiaries in the project area will live within 5 km of a health facility staffed with trained personnel and equipped with essential supplies. Project, VHC and MOH records. 1. Select and train Socorristas for un￾staffed health posts. 2. Monitor health posts through VHCs. 3. Assist community in the construction and equipping of health posts with Oxfam funding 1b. MOH personnel at district level will creatively address shortages of materials to provide continuity of services. Chloroquine will not be out of stock for more than three days per month in at least 90% of health posts. Project, VHC and MOH records and meetings. 1. As shortages arise, catalyze a joint response between MOH personnel and community to solve the immediate problem without WR resources. 2. Establish contingency plans for repeated problems. 1c. All Socorristas will receive regular supervision. 75% of Socorristas will have received a supervisory visit from Chokwe District staff or Vurhonga project director within the previous quarter. Supervision reports Mtgs with Chokwe DHMT 1. A motorbike will be made available to the MOH Socorrista supervisor. 2. A monthly report on health posts visited and their current status will be completed. 1d. Community groups will maintain linkages with the MOH. 70% of VHCs will have met at once during the last two months. Health Post staff will attend at least 80% of the meetings of VHCs M&E, MTE, Project records 1. Socorrista supervisor monitors VHC meetings and participation of health post staff. World Relief Mozambique CSXV Final Evaluation Report 2003 31 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities 1e. Community members will expect and demand provision of basic services. 1. 50% of beneficiaries will report that they are satisfied with their last visit to an MOH facility as measured by a satisfaction index M&E, MTE, Project records 1. Churches, community groups, or VHCs will learn to approach the MOH concerning disrupted or poor quality of services. 2. Community–Level 2a. VHCs will monitor, plan and evaluate maternal and child health for their areas. 1. 95% of villages will have established VHCs by EOP. 2. 70% of VHCs will meet at least once in the last two months. Staff reports, VHC meeting records, M&E. 1. Animators facilitate formation of the VHCs/ 2. Animators orient and train the VHCs in their responsibilities and use of meeting agenda guide (MAG) 3. Pair strong CGs and VHCs with weaker ones for instruction. 2b. Care Groups are thoroughly trained in CS intervention. 100% of Care Groups will “graduate” during the fourth year of project. Project records See description of Care Group “graduation” in project design section. World Relief Mozambique CSXV Final Evaluation Report 2003 32 Table 19: Vurhonga II Objectives, Indicators and Major Activities Objectives Indicators Measurement Methods Major Planned Activities 2c. Churches in the project area will actively promote behavioral changes specified in the interventions. 1. The monthly attendance rate at church leaders Care Group meetings will be at least 60%. 2. 50% of mothers who attended church during the past month report that they heard a health message. Church Care Group meeting records, M&E, 1. Conduct census of congregations within project area. 2. Approach pastors and church leaders to participate in church Care Groups. 3. Adapt health education for selected interventions to church use, e.g., for sermon outlines. 4. Through animator and supervisor visits to church leaders, promote cooperation across denominations. ii. Volunteer and Family Level 3a. Volunteers in the Care Groups will develop a sense of efficacy and pride in promoting changed behavior. 1. At least 60% of families have been visited by their volunteer during the past two weeks. 2. The attrition rate of volunteers for reasons other than death, disability, or movement out of the project area will be less than 10%. Project records, M&E plan 1. Care Groups will celebrate successes. 2. Results of the base line, MTE and M&E will be communicated to the volunteers in the Care Groups to chart their success. 3. Vurhonga II staff of at least supervisor level and “important visitors” will regularly praise volunteers before community leaders. 3b. Families will collectively demonstrate and sustain positive health behaviors Use indicators for specific CS interventions, e.g., participation in growth monitoring, EPI coverage, etc. Ministry of Health HIS Quarterly HIS 1. BCC by volunteers emphasizes the importance of sustaining positive health behaviors World Relief Mozambique CSXV Final Evaluation Report 2003 33 3. PROGRAM SITE INFORMATION 1. Program location A map of Chokwe district and the project impact area is included in the front of this report. The map includes the names of all towns and villages, and indicates the location of existing hospitals, health centers, and health posts, including 27 health posts constructed since the beginning of Vurhonga II. The project map also indicates the five geographic areas of responsibility for the five project supervisors. Large poster-size versions of this map are used for planning purposes by the project offices and the MOH district office. Situated on the southern border of the Limpopo River about three hours (200 km) north from Maputo (capital of Mozambique), Chokwe is one of the most densely populated districts in Gaza province. The project site for Vurhonga I was in the districts of Guija and Mabalane just on the other side of the Limpopo River. The project impact area for Vurhonga II includes all towns and villages in Chokwe district except for the urban center of Chokwe town and the more rural town of Hokwe. Hokwe has been excluded because it has been assisted, since February 1999, by a project of the Nazarene Church. The Hokwe project was designed using the same development approach as the Vurhonga project, and even uses the same health education materials. 2. Population and beneficiaries The estimated population for Chokwe is 202,000. The project area, however, excludes Chokwe town (60,000 inhabitants) and the village of Hokwe (12,000 inhabitants). The total population for the project area is therefore estimated to be 130,500. A project census of the impact areas was completed in early 2000, and calculated that the total number of potential beneficiaries would be 53,418, i.e., 31,338 women age 15-49 years old and 22,080 children under five years of age. The estimated number of births to occur during the life of the project is 19,836 (130.500 x 3.8 births/100 x 4 years). 3. Health status Rural Mozambique is a dangerous place for women and children. The national infant mortality rate is 133, the under 5 mortality rate is 214 and the total fertility rate is 6.2.1 Weakened by malnutrition, children die most frequently of malaria, diarrhea and pneumonia. According to national DHS data, malaria and diarrhea are the top two killers of children under 5. The 1997 Demographic Health Survey (DHS) found high incidence of diarrhea (1 in 5 prevalence within the past 2 weeks) and low levels ORT knowledge (35%). Among children who died after admission to Chokwe Hospital in the last six months, hospital records attributed 50% of the deaths to malnutrition, 16% to malaria, and 15% each to pneumonia and diarrhea.2 1 State of the World’s Children 1998. UNICEF. Oxford University Press, 1998. 2 MOH Chokwe District health office, 1998. World Relief Mozambique CSXV Final Evaluation Report 2003 34 Maternal mortality for Mozambique (1990 data) is estimated as 1,500/100,000 live births.3 USAID estimates a maternal mortality rate of 631/100,000 live births in their focus areas.4 About one-third of the project effort will address reproductive health issues (birth spacing, maternal care, and HIV/STI education) highly relevant to child survival in Chokwe. 4. Other factors influence health status a. Economic characteristics of the population The rural economy of the project area lags behind the increased commerce of the towns. Most of the families in the project area raise only enough food from their farms to sustain themselves. The land is flat and susceptible to seasonal droughts and floods. Even with the financial assistance from the many men who work in South African mines and industry, six in ten people live below the level of absolute poverty. b. Social characteristics Shangan is the mother tongue and ethnic identity of 97% of the people of Chokwe. Most (60%) identify themselves as Christians of various denominations. The remaining 40% follow traditional African beliefs. Because the Shangan people are both patriarchal and patrilocal, women are among the most powerless. The freedom of young women of child￾bearing age to adopt new health behaviors is constrained by long hours of hard work each day and allegiance to their husbands and the older women of their husband’s household. Fewer than four in ten women are literate. Further, the disproportionate ratio of women (58%) to men (42%) because of war and male migration to South Africa encourages polygamy and multiple sexual partners. Because women’s sense of value and future security depends upon childbearing, the incidence of sexually transmitted infections increases even as their power relative to men decreases. c. Practices regarding the care of infants A mother provides complete care of her child during the first several months. She keeps the child with her at all time, even when going to the fields, so that she can breastfeed the child on demand. Weaning is often done suddenly. When the child’s grandmother feels that the right time has come, or when the child’s mother again becomes pregnant, the child is removed from the breast and often sent to the grandmother’s home to completely separate the child from the mother for a time. Children who are able to put food in their mouths are placed on the floor with the food dish and expected to eat with their fingers. Usually during mealtimes small children are only supervised by older siblings. The mother also frequently leaves a child under the 3 State of the World’s Children 1998. UNICEF. Oxford University Press, 1998. 4 USAID Mozambique Mission Strategic Objectives, 1998. World Relief Mozambique CSXV Final Evaluation Report 2003 35 care of siblings when she goes to the field for the day. The Grandmother, if available, may supervise all the children, but not necessarily provide direct care for the youngest child. A child will usually be given food left over from the previous day, and competes poorly with older siblings for the available food. Rarely do young children receive protein or vitamin-rich foods in their porridge, and they do not eat enough meals per day. No differences exist between the manner that young boys and girls are fed, and they generally eat out of the same dish. Typically, men receive the first choice of food, with the rest left for the other family members to eat. d. Potential geographic, economic, political, educational, and cultural constraints Most villages in Chokwe district are situated along roads that are accessible year round. Floods in Chokwe town and neighboring villages are a rare but an all too real and potential constraint for the project. The last floods of February 2000 left the town of Chokwe under two meters of water, and interrupted the project activities of Vurhonga II for more than six months. There are a number of traditional beliefs that represent cultural constraints for the promotion of improved health practices. For example, traditionally mothers attribute the visible signs and symptoms of malnutrition not to poor nutritional practices but to oppression (being ‘sat upon’) by malevolent spirits and therefore cannot see the relationship between malnutrition and feeding behaviors. 5. Existing status of health services The existing health infrastructure for Chokwe district is summarized in Table 2. At the beginning of Vurhonga II the MOH infrastructure included the hospital of Chokwe, three health centers with nurses and 14 health posts with either a nurse or Socorrista. There were 34 villages without health posts. Population access to health services within 5 km of the village was 65%. Table 2: Health Infrastructure of Chokwe District (1999-2001) Health Facilities Chokwe District (construction materials + type of staff) 1999 2001 Health Center (durable + nurses + midwife) 3 3 Health Post (durable + nurse and/or midwife) 10 10 Health Post (durable + Socorrista) 2 3 Health Post (local materials + Socorrista) 2 28 Total 17 44 Population within 5 kms of a Health Post 84,630 132,084 Total Population in the Project Area 130,492 132,480 % of population within 5 km of a health post 64.8% 99.7% World Relief Mozambique CSXV Final Evaluation Report 2003 36 In addition the MOH health facilities, other health providers in the Chokwe District include: o a Catholic hospital in Chokwe that tests and treats AIDS patients; o the health program of the Nazarene Church in the community of Hokwe; o Med-Air assistance to a number of health posts; o Socorristas managing health posts under the supervision of the MOH o Traditional Healers (Inyanga spiritists) The population pays a nominal fee for services at health posts and health centers. At MOH health posts and health centers there is a 1000 mt. (.06 USD) charge for service and an additional charge for medicines of approximately 500 mt. (.03 USD). The total payment per episode is therefore around 1500 mt. or nine cents in U.S. currency. At health posts with a Socorrista the level of cost recovery is somewhat higher, i.e., around 2000 to 3000 mt. per episode treated depending on the agreement at the village level. An average workload of a Socorrista is five to ten patients per day. These funds are kept entirely at the health post level and constitute the salary of the Socorrista. The only other significant cost for Socorrista is transportation cost to travel to the district office or to the nearest health post with a nurse at least once a month. The Socorrista cover this cost out of their salary. The morale of MOH health workers has traditionally been low, with a correspondingly low utilization of services by the population, who did not feel that they were well received. However, the general economic improvements throughout the country and salary increases for health personnel make the current situation more positive than it was during Vurhonga I. 6. Identify any groups in the program site that you consider disadvantaged One disadvantaged group within Chokwe district consists of villages that lack good access to EPI services. While the mobile team theoretically serves these villages from Chokwe, the frequent unavailability of a vehicle has resulted in a relatively low EPI coverage for Chokwe district. 4. PROGRAM DESIGN The overall goals of the Vurhonga II project are to: 1) Reduce morbidity and mortality in children under 5 and women of fertile age, 2) Strengthen the capacity of the MOH to implement CS interventions; and 3) Empower communities to improve their health. The specific purposes and strategies within the above goals are to: · Reduce malnutrition by 50% by growth monitoring, counseling and weaning foods; · Improve maternal nutrition by increasing food consumption during pregnancy and lactation; World Relief Mozambique CSXV Final Evaluation Report 2003 37 · Reduce malaria death rate in children by 50% by early recognition and treatment; · Reduce pneumonia deaths by early recognition and treatment; · Reduce deaths and speed recovery from diarrhea by ORT and improved nutrition; · Reduce transmission of HIV/AIDS through improved recognition and treatment of STIs; · Double the number of couples spacing their children with a modern method; · Increase the number of mothers delivering children by a trained provider; · Increase access to health care to 100% of the population within 10 km of a trained provider; · Improve the management of childhood malaria and pneumonia at health posts; · Establish sustainable village health committees to monitor health activities; and · Mobilize and train over 2,000 volunteers, churches and community leaders. Program Design and Stakeholders World Relief has designed Vurhonga II with USAID Mission Strategic Objectives (SO) in mind. All proposed program interventions share Mission SO3 goals of improving access, quality and management of MOH services while also mobilizing community demand for basic health services. National government policies for child survival are basically sound and still developing. Vurhonga II strategies are consistent with MOH policies. Where there are differences the project will train them in new protocols, e.g., malaria and pneumonia. Vurhonga II will also assist the MOH in improving their capacity to plan, implement and supervise child survival interventions and community-based activities. Vurhonga II will help 47 communities create 220 Care Groups to train 2,800 volunteer mothers and church leaders as behavior change agents to all households for nutrition; malaria, pneumonia, and diarrhea case management; maternal and newborn care; child spacing; and STI/HIV/AIDS prevention. The project will help the MOH expand its network of health posts so that 95% of the population lives within 5 kms of a trained health care provider (nurse or Socorrista). The project will also establish communication and coordination links between the communities and health posts. 5. PARTNERSHIPS The local partners for Vurhonga II are the Ministry of Health and 47 local communities. The working relationship with these partners will build on an important lesson learned from Vurhonga I related to working in partnership with the MOH. Lesson Learned: Working in collaboration with the MOH from the beginning and with a discipline to respect each others’ roles creates a synergy that both improves health care and is also sustainable. Project interventions for Vurhonga I showed dramatic improvement for all interventions. Project goals were not only met, but greatly surpassed. This success was attributed to a combination of 1) enhanced awareness of health issues at the household level; 2) improved access to health services; 3) increased availability of supplies; and 4) cooperation and communication with the MOH. World Relief Mozambique CSXV Final Evaluation Report 2003 38 The Vurhonga I project increased the demand for services by focusing on communication for behavior change at the household level. The MOH increased access to and quality of health services with the creation of new health posts and the training of Socorristas. These approaches, and the links between them, had a synergistic impact that was greater than either could have achieved alone. All child survival interventions were carried out in collaboration. There was a demonstrated willingness and capacity by both the MOH and community to continue and sustain these services. In addition to working successfully with the MOH as a partner, the Vurhonga I project also worked in partnership at the community level. The final evaluation of Vurhonga I found that Knowledge of child survival has significantly increased at the household level. What is more important, changes to positive health behaviors are clearly sustainable. The project’s strategy to use a participative methodology, “mentored” training, home visits and demonstrations have resulted in very effective and sustainable educational. The Care Group structure created a dynamic support structure for CS interventions and behavior change. “People are doing things differently now” and “It is now part of our daily lives” were heard repeatedly during evaluation visits. Therefore, the two major partners for Vurhonga II are the MOH and the communities, especially the Care Group structures. The respective roles of these partners and the project staff are outlined in the Table 6. See also Table 5 regarding the relationship with other health providers. Table 6: Roles of Major Partners by Child Survival Intervention Intervention MOH Community Project Staff Nutrition Growth Monitoring - Growth Monitoring -Vit A distribution -Assist with GM and HEARTH -Training in GM Malaria/Pneumonia - Diagnosis and treatment - Stock of Essential Meds -BCC -Distribute Bednets - Bednet retreatment -Train Socorrista & nurses -Procure bednets -Plan Bednet Retreating Reproductive Health Maternal Health Child Spacing STI/HIV/AIDS -Stock of Contraceptives Train communities CDD -Stock of ORS packets -Training in treatment - Immunizations Maintain Cold Chain Mobilize for EPI Support as needed 6. HIS World Relief Mozambique CSXV Final Evaluation Report 2003 39 There are four processes – formal evaluations, intervention specific research, cycles of organizational learning, and the promotion of personalized feedback loops – which WRC will combine to provide a richly detailed picture of the Vurhonga II program. a. Formal Evaluative and Monitoring Cycles. These include the proposal and DIP reviews, the midterm and final evaluations led by external evaluators and the three KPC surveys that punctuate the progress of the project. By choosing external evaluators who are not only technically competent but also highly participatory and carefully examining and responding to the recommendations, the Vurhonga II team will incorporate these inputs into program formation and change. The Vurhonga II project will use the quarterly HIS M&E developed in Vurhonga I. See detailed description later in this section. b. Intervention Specific Research. Each of the project interventions will require specific qualitative approaches to inform program and BCC strategy. The Hearth program, for instance, will use 24 hour diet recalls in each village to identify locally sustainable nutritious diets, that are continually assessed to take into account regional and seasonal changes. The quality of care programs in reproductive health will involve exit interviews, focus groups and other procedures. The feedback loop for intervention specific research is short and extremely practical. Knowledge about health services and the needs and practices of people will be used to revise objectives and plans at several key junctures: 1) during objective-setting at the DIP, after the KPC survey, 2) during development of curriculum, (TOT with animators, and when they train Care Groups 3) after midterm evaluation, monitoring visits, and to a lesser extent after each quarterly M&E. c. Organizational Learning Cycles. The formal evaluation and monitoring system is only one of the organizational learning cycles built into Vurhonga II. Each workweek will be sandwiched between meetings every Monday morning and Friday afternoon when animators, supervisors and technical staff discuss progress, results and problems. These meetings function as quality assurance (QA) meetings where project staff identify opportunities for improvement, define problems, establish desired outcomes and plan steps to achieve them. During these meetings animators will get their questions answered, review progress toward goals and develop weekly action plans that will be left up on the board for discussion the next week. In monthly cycles, the Project Director will attend district MOH staff meetings to share feedback. The M&E and training camps mark quarterly cycles of information gathering and organizational reflection. The Project Director is responsible to collect and analyze data on limiting factors and will plan corrective action and monitor results. The animator uses a weekly monitoring and reporting form that includes columns for each intervention. She writes down how many groups she met with and what World Relief Mozambique CSXV Final Evaluation Report 2003 40 interventions she discussed. This includes meeting with and individual or group, MOH, NGO’s, or church leaders. On back of the form the animator may comment on any significant issue the supervisor needs to know. She fills in this form daily, and reports the results to supervisor at the end of each week, but keeps the form. The supervisor in turn reports results to the project director. d. Personalized Feedback Loops. A unique strength of the Vurhonga I program was the highly personalized feedback loops. Unlike many CSP’s, the project language from top management to the beneficiaries is Shangan, greatly enhancing communication and cultural understanding. Personal feedback is essential to the cultural fit and flexibility of program interventions as well as to continuous cycles of quality assessment. In Vurhonga II, technical staff and supervisors will live with animators in the field and assess progress through interviews and observations, guided by supervisory forms and project indicators. The Project Director (PD) will continue to work in the district hospital in Chokwe one day a week, allowing him to personally assess coverage, quality and needs of the health system and access MOH statistics (malaria and pneumonia cases treated, FP acceptors, EPI coverage, mortalities and morbidities). The project director and supervisors can assess quality of MOH services in visits to clinics and Socorristas. Needs of existing health services are discussed in routine local visits and in monthly district meetings with MOH, which also provide an ideal opportunity for the Director to share M&E results, report on progress, issue invitations to TOT camps, etc. Quality of care can also be assessed through exit interviews, focus groups and other methods learned at the Quality of Care workshop sponsored by SEATS as part of Vurhonga I. Animators and volunteers will discover needs in the community and monitor client satisfaction by their constant contact with Care Groups representing cross-sections of the population. Qualitative personal feedback from the animators will be essential to the program. The animators, who share the culture of the people, encode key intervention messages in song, using communal learning and support to facilitate change and develop accountability. They will address behavior change in a shared context of community norms, with culturally appropriate communication models. Each animator will track volunteer attendance at Care Group meetings as an indicator of collective volunteer performance. Care Groups must have 60% average attendance to be considered viable. World Relief Mozambique CSXV Final Evaluation Report 2003 41 Quarterly HIS Monitoring and Evaluation The quality, effectiveness and coverage of the interventions are monitored through these quarterly rapid assessment surveys (the M&E plan), which use KPC survey questions to monitor most knowledge, practices and coverage indicators. Sample survey forms are shown in Figures 4 below and Figure 5 on the following page. Figure 4: HIS Survey Form: Questions for Mothers Every three or four months, one Care Group from each of the 26 animators is randomly chosen and all the families, approximately 150 households, that they are responsible for, are surveyed. To preserve objectivity, the animators trade areas to conduct the home surveys. The resulting data set of 2,500+ interviews is hand-tabulated and summarized by the animators, then analyzed and reported project-wide to chart progress on objectives. Animators and supervisors will also use this time as an educational opportunity; the volunteers benefit from immediate feedback, and the results are left with each animator for them to review with their Care Group. Thus in addition to routine informal visits from animators and supervisors, every home will be surveyed every two years. The collected information is used to track progress towards objectives, and the animator gives feedback to village health committees. World Relief Mozambique CSXV Final Evaluation Report 2003 42 Figure 5: HIS Survey Form: Questions about Children The quarterly HIS system used by Vurhonga I proved to me quite accurate in matching the results of the formal KPC process. World Relief Mozambique CSXV Final Evaluation Report 2003 43 B. Evaluation Team Members Name Position Affiliation Project Staff Dr. Pieter Ernst Program Director World Relief Margarida Massingwe Secretary/Administrator World Relief Crestina Cossa Area 1 Supervisor World Relief Veronicao Cima Area 2 Supervisor World Relief Clara Javana Area 3 Supervisor World Relief Selina Hlonga Area 4 Supervisor World Relief Delfina Maluleque Area 5 Supervisor World Relief Paucilda Chambal Animator Area 1 World Relief Victoria Bule Animator Area 1 World Relief Santissima Quinica Animator Area 1 World Relief Filista Sondo Animator Area 1 World Relief Rute Hlongo Animator Area 1 World Relief Esperanca Muhlanga Animator Area 1 World Relief Ligia Langa Animator Area 2 World Relief Lurdes Cubai Animator Area 2 World Relief Melita Mapsanganhe Animator Area 2 World Relief Rafa Massingue Animator Area 2 World Relief Ilda Timane Animator Area 2 World Relief Maria Marcela Animator Area 3 World Relief Maria Odete Animator Area 3 World Relief Rosita Chambal Animator Area 3 World Relief Clementina Bila Animator Area 3 World Relief Celina Mapsanganhe Animator Area 3 World Relief Antonieta Hlongo Animator Area 4 World Relief Filismina Mapsanganhe Animator Area 4 World Relief Rute Chauke Animator Area 4 World Relief Dulce Matusse Animator Area 4 World Relief Maria Louisa Animator Area 5 World Relief Cacilda Mapilele Animator Area 5 World Relief Maria Rosaria Animator Area 5 World Relief Roda Manhique Animator Area 5 World Relief Carmelia Machava Animator Area 5 World Relief Viegas Ngovene Driver Area 1 World Relief Fenias Muhlui Driver Area 2 World Relief Isaias Muiambo Driver Area 3 World Relief Pastrelo Mapengo Driver Area 4 World Relief Agostinho Calanga Driver Area 5 World Relief World Relief Mozambique CSXV Final Evaluation Report 2003 44 Dora Tcheco Granny Trainer World Relief Partner Representatives Inacio Xithlangu District Training Coordinator, Socorrista Supervisor and In-charge of Statistics District MOH Translators Sybil Baloyi Child Development Director World Relief Joa Pedro World Relief External evaluators Dr. Carl Taylor Professor Emeritus Johns Hopkins University Meredith Long Director of Health Programs WorldRelief HQ Melanie Morrow Child Survival Specialist World Relief HQ Other Persons Contacted Dr. Pascual Mucumbi Prime Minister of Mozambique Govt. of Mozambique Chris Barratt PHN Officer USAID/ Mozambique Ilka Esquivel Health Sector Support Leader USAID/Mozambique Titus Angi HPN Specialist USAID/Mozambique C. Assessment Methodology The final evaluation took place from September 8-19, 2003 and was based on a review of survey results (from the final KPC report as well as project data from regular monitoring surveys and monthly statistics reported by care groups), site visits to villages and discussions with staff, volunteers, village health committees, pastors and church leaders and health facility staff including health post socorristas and health center nurses. The district MOH actively contributed to the assessments via the participation of Mr. Inacio Xithalangu, the staff member most closely involved with implementation of the program. Because of striking project data showing a 62% decrease in under-five mortality from 2000 through 2002, the evaluator chose to check these results by conducting 250 pregnancy histories of women living in the project area. Not only were the results corroborated, but the exercise afforded opportunity to observe the staff (in the absence of the project director, who was absent from the first phase of the evaluation) work together in developing the necessary instrument and coordinating logistics to collect the data in just two days from 10 different villages. Results were hand-tabulated, as the staff are accustomed to doing for KPC and other survey results. Overall program results, in particular those showing the dramatic decline in mortality were highlighted at a dissemination meeting in Maputo sponsored by Unicef and with attendees from USAID, MOH, WHO and numerous NGOs. The evaluator also made recommendations for scaling up this program in Mozambique. World Relief Mozambique CSXV Final Evaluation Report 2003 45 The evaluator also shared results and recommendations in a private meeting with Prime Minister of Mozambique, Dr. Pascoal Mucumbi. D. List of Persons Interviewed and Contacted Sept 9 Mothers (14) from Xiaquelane Village Sept 9 Religious leaders (6 women, 7 men) from Cotsuane Village Sept 9 Community Authority and Nurse at Mapapa Village Health Center Sept 9 Conhane Village Health Committee (33 members) Sept 9 Matuba Village Health Committee (22 members including 1 Socorrista and 2 TBAs) Sept 9 N’Waxicolwane (27 volunteers, 2 village leaders, 5 members of national women’s committee) Sept 10 VHC (13 women, 5 men present, 22 total) from Matuba Village Sept 10 Church Leaders (7 women and 5 men) from Muzumia Village Sept 10 Volunteers (9 women) from Machel Village (Malaria drama) Sept 10 Caretakers (21 mothers and nurse) from Manjangue Health Center Sept 12?? Prime Minister Dr. Pascoal Mucumbi World Relief Mozambique CSXV Final Evaluation Report 2003 46 E. Workshop on Sustaining and Scaling Up Community Development Programs “Mechanisms for Sustaining and Scaling up Community Development Programs” Carl Taylor Prof Emeritus, Johns Hopkins University September 15,2003 Maputo, Mozambique Cosponsored By UNICEF, USAID, and WR 9.00 AM Welcome and Introductions 9.30 AM Introduction of the SEED/SCALE methodology- Taylor 10.30 AM Tea 11.00 AM Selected Achievements of the Child Survival Programs in Gaza Province – World Relief 12.30 PM Lunch 1.30 Pm Sustainability and Future Directions for Scaling up Community Based Programs to build Community and Health System Capacity – Taylor 3.00 Pm Tea and Open Discussion 4.00 Pm Closing remarks – Ministry of Health World Relief Mozambique CSXV Final Evaluation Report 2003 47 F. Final KPC Survey Report