Achieving Equity, Coverage, and Impact through a Care Group Network Project Location: Mozambique, Sofala Province In the districts of: Caia, Chemba, Maringue, Marromeu, Beira, Dondo, Gorongosa, & Nhamatanda October 1, 2005 – September 30, 2010 Cooperative Agreement No. GHS-A-00-05-0014-00 Report Submitted: October 24 2008 Mid-term Evaluation July 28-August 7, 2008 Donald T. Whitson, MD, MPH Evaluator FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 2 This report was prepared by: Donald T. Whitson, MD, MPH 24/41 Sukhumvit Soi 18 Domus Condominium 12D1 Khlong Toey, Bangkok 10110 Thailand Tel/Fax: 66-2-259-7628 E-mail: dtwhitson@gmail.com FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 3 ACRONYMS ACS Community health agent, a community level health worker within the MOH system. These have been paid during some periods. BCC Behavior change communication CDC Community Development Committees CG Care Group C-IMCI Community-based integrated management of childhood illness. One LM in each Care Group was trained in C-IMCI and is referred to as a C-IMCI LM.. CDC Community development committee CSP Child Survival Project DIP Detailed implementation plan DPS Provincial Health Department EBF Exclusive breastfeeding EOP End of project FGD Focus group discussions FH Food for the Hungry GM Growth monitoring (not genetically modified) HAI Health Alliance International HH-IMCI Home health IMCI—similar to C-IMCI HMIS Health management information system IMCI Integrated management of childhood illness. ITN Insecticide treated mosquito net KPC Knowledge, practice and coverage survey LM Leader Mother. C-IMCI LM Leader Mother trained in community￾based IMCI. One for every twelve mothers. 14 per Care Group LOE Level of effort. LQAS Lot quality assurance sampling M&E, M and E Monitoring and evaluation MOU Memorandum of understanding MUAC Mid upper arm circumference (a rapid nutrition screening technique) MOH Ministry of Health MOU Memorandum of understanding MPH Masters in Public Health MTE Mid-term evaluation OR Operations research ORS Oral rehydration salts ORT Oral rehydration therapy POU Point of use PSI Population Services International, an international NGO specializing in social marketing QIVC Quality improvement and verification checklist USAID United States Agency for International Development FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 4 TABLE OF CONTENTS A. Executive Summary…………………………………………………. 5 B. Assessment of Results and Impact of the Project………………….. 8 1. Results Technical Approach…………………………………. 8 2. Results: Cross-cutting approaches………………………….. 26 a. Community Mobilization…………………………….. 26 b. Communication for Behavior Change……………… 29 c. Capacity Building Approach……………………….... 32 i. Local Partner Training……………………….. 32 ii. Training……………………………………….. 33 d. Health Systems Strengthening……………………….. 35 e. Policy and Advocacy………………………………….. 36 f. Contribution to Scale/Scaling Up…………………….. 37 g. Equity………………………………………………….. 37 h. Sustainability………………………………………….. 37 D. Changes in Grantee Organization Capacity (NA)…………………. 42 E. Mission Collaboration………………………………………………... 42 F. Contextual Factors that Have Influenced Progress to Date……...... 42 G. Conclusions and Recommendations………………………………… 44 H. Action Plan…………………………………………………………….46 I. Annexes 1. Results Highlight……………………………………………… 48 2. Publications and Presentations………………………………. 49 3. Project Management Evaluation……………………………...49 a. Planning………………………………………………...49 b. Supervision of Project Staff…………………………...50 c. Human Resources Management………………………51 d. Financial Management………………………………...53 e. Logistics………………………………………………... 54 f. Information Management……………………………. 55 g. Technical and Administrative Support………….….. 59 h. Management Lessons Learned…………………….… 60 i. Other Issues……………………………………………. 60 4. Full M&E Table………………………………………………. 62 5. Rapid CATCH table………………………………………….. 67 6. Midterm KPC Report………………………………………… 67 7. Evaluation Team Members……………………………..……. 74 8. Evaluation Assessment Methodology……………………….. 74 9. List of persons interviewed and contacted………………....... 75 10. Project Data Form……………………………………………76 11. Revised Beneficiary Table & Staffing Plan .……………......77 FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 5 A. Executive Summary Food for the Hungry’s Child Survival Project is an expanded impact project that aims to scale up the Care Group model that successfully reduced the under-five mortality rate (U5MR) by 62% and the Child Mortality Rate (CMR, 12-59m) by 94% between 1999 and 2004 in parts of four districts in the province of Sofala Province, Mozambique.1 These results were achieved through FH’s Title II Food Security program. The CS project has four strategic objectives: improve child nutritional status; assure appropriate diarrhea case management (including proper feeding) and prevention of diarrhea; increase the proportion of mothers of young children who have access to a C-IMCI-trained provider within one hour of their home; and assure the sustainability, quality and expansion of the Care Group Model in Mozambique. The CS project is being implemented in a phased manner, beginning first in five districts and then expanding to three more in Sofala Province, Mozambique. The CS project has achieved or exceeded its ambitious targets for almost all technical indicators in the phase I areas within two years of community-level implementation. The few targets that were not reached were almost reached, and the targets were high (e.g. vitamin A supplementation coverage achieved: 91%, target 95%). Targets for the sustainability indicators from the fourth objective would not be expected to be achieved until the second half of the project. FH has successfully and systematically implemented the Care Group model, achieving nearly one hundred percent coverage of intensive BCC messages through trained volunteer “Leader Mothers”. This approach depends almost exclusively on community-based activities for behavior change. There are preliminary indications that the new behaviors are being adopted as community “norms”, and therefore may be highly sustainable. The success of the behavior change strategy is attributed to the meticulously planned execution of the project. Messages are well-defined and based on formative research, and materials are well-tested and of high quality. FH has made extensive use of intensive systematic supervision using Quality Improvement and Verification Checklists (QIVCs) and mini-KPCs using lot quality assurance sampling (LQAS) to monitor and improve the quality of the implementation at every stage. Other innovative strategies include the use of verbal autopsies and Barrier Analysis2 to refine messages. Two important issues have appeared. FH has had staff turnover that is far higher than expected, though this problem has been largely overcome, and has not measurably affected the project outcomes. In addition, partnering for sustainability and scale-up has proven difficult, as NGO partners have largely been unable to alter their programming to participate. Likewise, FH has had difficulty engaging local and national MOH partners to encourage them to adopt, replicate and scale-up the Care Group model as originally expected (although many of the scale-up efforts were expected to occur during the second half of the project). The principal conclusions of this evaluation are that the project strategy has succeeded in sustainably changing the behaviors of mothers and other community members in the phase I 1 From results of a Johns Hopkins University study presented at the APHA Annual Meeting, “Care Groups Significantly Reduce Child Mortality in Mozambique,” Davis, T, Hernandez, E, and Wetzel, C, Nov 6, 2007 2 See http://barrieranalysis.fhi.net for more details. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 6 areas, leading to a decrease in malnutrition, and higher health service utilization. Some specific conclusions and recommendations in technical areas relate to the need to further concentrate on reducing neonatal mortality (suggestions are given in the report), seeking strategies to guarantee the participation of younger mothers (especially second wives) in Care Groups, doing advocacy for scale-up, and improving the monitoring of maternal health indicators using the MOH HMIS. As phase II begins, some aspects of implementation can be simplified, including reducing the number of indicators being routinely monitored, simplifying the curricula in use by Facilitators, and simplification of C-IMCI materials for C-IMCI LMs. FH may wish to approach the Provincial Health Training Institute in order to institutionalize training curricula, including C-IMCI. Sustainability and scaling-up will require more attention and a more explicit strategy if they are to be successful. This should focus on the MOH, as partner NGOs cannot be counted on to sustain the effort – they generally have their own agendas and funding constraints. Some strategies to engage the MOH include: incorporating HMIS indicators into monitoring, directly engaging health facility staff in the urban areas, engaging the Provincial Health Director more directly in project planning and evaluation and expending more effort articulating with the MOH in Maputo. FH may wish to submit its BCC and training materials to the MOH in Maputo for formal approval at this stage. Finally, as overhead costs are expected to grow significantly with phase II (due in part to the unexpected closure of the Title II and HIV/AIDS projects in some districts), FH may wish to request another alteration in project geographical areas in order to consolidate the target population. This could include the abandonment of some low-density (and expensive-to￾reach) rural areas and substitution with some peri-urban areas. Summary Impact Model Elements for Project Inputs Phase I Activities Phase I Outputs Phase I Desired Outcome Actual Outcome Goal ¾BCC materials (flip-charts) ¾Training curricula for flip-charts ¾C-IMCI training curriculum ¾Regular (monthly) meetings with district health offices for coordination ¾Baseline census ¾KPC & retro￾spective mortality study ¾Barrier Analysis and development and reproduc￾tion of flip￾charts ¾Train staff and LMs in nutrition modules, ¾Train Facilitators and C-IMCI LMs ¾About 2,000 LMs trained in six modules ¾Mothers of children 12- 23m visited regularly by trained LMs ¾Children receiving 6- monthly vitamin A and deworming in the community. ¾ Reduce malnutrition (WAZ<-2) from 27% to 18% ¾ Increase EBF from 17% to 60% ¾ Increase children who eat foods with oil added from 35% to 80% ¾ Increase vit A supplement coverage every ¾ Reduced from 27% to 18.6% ¾ Increased from 17% to 95%. ¾ Increased from 35% to 84%. ¾ Increased from 82% to 91%. Improve the nutritional status of children 6-59m of age. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 7 Inputs Phase I Activities Phase I Outputs Phase I Desired Outcome Actual Goal Outcome ¾Vit A and mebendazole from District offices ¾Positive deviance materials (nutritious weaning foods) in Vit A and mebendazole distribution ¾Systematic supervision of each step. ¾Follow-up mini-KPC surveys for monitoring. ¾Anthropometry survey ¾Pos. Dev. study used to guide message development ¾Train 30 Facilitators in PD/Hearth Methodology. ¾Children weighed and counseled regularly. ¾Malnourished children recovered through PD/Hearth program (scheduled for Phase II) 6 months from 82% to 95% ¾ Increase deparasitization every 6 months from 24% to 75% ¾ Increased from 24% to 86% ¾BCC materials (flip-charts) for diarrhea and hygiene module ¾Training curricula for flip-charts ¾C-IMCI training curriculum ¾ORS for community distribution (from districts) ¾Regular meetings for coordination with DPS and districts. ¾Barrier Analysis for hand washing with soap/ash. ¾Train staff and LMs on diarrhea and hygiene modules ¾Train staff, partners, facilitators and C-IMCI LMs in C-IMCI ¾Follow-up mini-KPC surveys for monitoring after modules for monitoring ¾About 2,000 LMs trained in diarrhea and hygiene modules ¾Mothers of children 12- 23m visited regularly by trained LMs ¾LMs distributing ORS in communities ¾C-IMCI LMs referring children to health facilities ¾ Increase % of children with diarrhea receiving ORT from 71% to 90% ¾ Increase feeding during diarrhea from 31% to 60% ¾ Increase proportion of mothers who know when to seek care for sick children from 29% to 75% ¾ Increased from 71% to 78%, ¾ Increased from 31% to 70%. ¾ Increased from 29% to 85%. Assure appropriate diarrhea case management ¾Support for DPS nurses to train C￾IMCI ¾Curriculum located and updated ¾Training for 150 LMs, ¾Supervision checklists ¾Health facility assessment (IMCI) ¾Training of staff and 150 LMs in C￾IMCI ¾Regular monthly supervision of C-IMCI LMs ¾150 LMs and partner staff trained in C￾IMCI ¾Monthly supervision of C-IMCI using checklists ¾ Same as above (no outcome set for reduced mortality) Increase proportion of mothers with access to C￾IMCI provider FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 8 Inputs Phase I Activities Phase I Outputs Phase I Desired Outcome Actual Goal Outcome by Facilitators ¾CDC curriculum and training materials ¾Regular coordination meetings with DPS ¾Reports on mini-KPCs, results translated into Portuguese ¾Summary reports and power-point presentations in Portuguese ¾Zinc OR protocol designed ¾Train CDC members in 30 communities ¾Monthly supervision of LMs ¾Meeting with MOH in Maputo to share Care Group model. ¾Training for partner NGOs in Care Group model ¾Staff trained in protocol, OR study conducted ¾30 CDCs established and meeting ¾MOH in at least one province provided with technical assistance to implement CGs ¾OR report ¾ First-phase Care Groups that continue to meet and do health promotion following reduction of health Facilitator staff in Year 2.5. (no target set) ¾ The MOH in at least one other Mozambican province requests assistance (during the life of the program) from FH to expand the Care Group model into their geographical area. ¾ OR is conducted on the reasons for Care Group effectiveness. MOH in Cabo Delgado has requested FH’s use of the Care Group model in CD (with use of Title II funding) Scheduled for Phase 2 Assure sustainability B. Assessment of Results and Impact of the Project B.1. Results: Technical Approach B.1.a. Brief overview of the project The FH Expanded Impact CS Program’s three goals as stated in the DIP are: (1) significantly reduce morbidity and mortality – especially among children 0-23 months of age and pregnant women – in Sofala province; (2) increase access to community/HH IMCI-trained health providers in the program areas; and (3) to transfer the knowledge, skills, tools, and passion needed for effective and sustainable community health development through the Care Group approach to project partners, including Leader Mothers, in order to continue child survival activities once this project has ended. To this end, the project has four strategic objectives: (1) improve child nutritional status; FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 9 (2) assure appropriate diarrhea case management (including proper feeding) and prevention of diarrhea; (3) increase the proportion of mothers of young children who have access to an IMCI-trained provider within one hour of their home; (4) assure the sustainability, quality and expansion of the Care Group Model in Mozambique. The DIP describes that the project will be carried out in ten of the thirteen districts of the province of Sofala, in central Mozambique, with varying degrees of population coverage (from 34% to 100%). The reason for the original district and coverage selection is that FH’s Title II nutrition project included about half the population of Marromeu, Caia, Nhamatanda and Gorongosa. Since the elaboration of the DIP, an Austrian NGO, Comussanas has become active in Chibabava and Buzi with a community-based project that will apply a model similar to FH’s CSP (with TA from FH). At the request of the DPS, FH submitted a letter to USAID requesting permission to modify the target zone: not entering these two districts, and substituting larger populations in Nhamatanda and Gorongosa that have been left without CS services after the FH Title II project closed in April 2008. This request was approved by USAID on July 8th 2008 and the table below summarizes the changes from the original district coverage to the current coverage. See section “F: Contextual Factors” for suggestions on how the second phase may be made more cost￾effective given current logistical and budgetary constraints faced by FH/Mozambique. Annex 11 provides a complete, revised beneficiary table. The principal beneficiary population is children under two, followed by pregnant and lactating women and then less intensively working with children 24-59 months. The intervention mix was chosen to provide maximum impact on child mortality, malnutrition and sustainability: infant and maternal nutrition 80% LOE and prevention and management of diarrhea 20% LOE. Execution has followed this estimate fairly closely. Project Area: District Original DIP Beneficiaries Change Approved by USAID July 2008 Revised Beneficiaries Caia (Sofala), 50% 10,566 No change 10,566 Chemba 9,053 No change 9,053 Maringue 13,818 No change 13,818 Marromeu (Sofala), 34% 6,788 No change 6,788 Beira (Portion of pop. only) 22,654 No change 22,654 Dondo, 50% 16,835 Increase to 80% 32,111 Gorongosa, 50% 8,975 Increase to 80% 17,120 Nhamatanda, 50% 19,161 Increase to 80% 36,549 Chibabava 13,422 Not covered directly 0 Buzi 28,845 Not covered directly 0 TOTAL 150,117 148,659 The maternal and child nutrition intervention includes exclusive breastfeeding to six months, immediate breastfeeding, continued breastfeeding to 24 months (except for HIV positive mothers), appropriate complimentary feeding (adding oil, vitamin A, quantity, density, frequency), vitamin A supplementation, deworming, Hearth nutritional rehabilitation, post￾partum iron and vitamin A, encouragement of antenatal and post-natal care and institutional delivery, appropriate nutrition messages for pregnant women and danger signs in pregnancy and post-partum. The diarrhea intervention includes appropriate home management of diarrhea, community ORS distribution, recognition of danger signs and care-seeking (C- FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 10 IMCI), C-IMCI for staff, some LMs, partners and some MOH staff, and prevention measures (hand washing with soap/ash, disposal of children’s feces, latrine use, covering food and POU water treatment). The Care Group model is implemented in a tiered approach as follows: Level Responsibility Numbers Notes Beneficiary mothers (about 25,063 June 2008) Care for families. All mothers of children 0-23 months are enrolled. 12 per Leader Mother (LM) Leader Mothers (LMs) total of 2,102 LMs. (June 2008) Educate beneficiary mothers through home visits and group activities using flip-charts by themes. Reporting on vital events. 14 LM in each Care Group Volunteers. Receive a full set of flip-charts. C-IMCI LMs (total 150) One for every 14 LMs. Trained in C-IMCI by DPS. Must visit each LM every week plus visit their own 12 beneficiaries each week. Referral of sick children. 14 LMs per C￾IMCI LM, or one per Care Group. Volunteers. Provided with a bicycle and flip￾charts. Use pictorial reporting forms. Provide ORS. Facilitators / Facilitators (or ACS) ( 26 total as of MTE) Train LMs, provide LMs with materials, and conduct regular structured supervision to LMs in their activities through visits every two weeks. Collect monthly reports (including vital events) through recording of verbal or pictorial written reports from LMs. Will be responsible for Hearth in phase 2. Currently there are a total of 26: Manga (8), Caia (3), Chemba (5), Maringue (7), Marromeu (3). One for every five to six Care Groups: 804 – 1,008 beneficiary mothers. There are now 26 Facilitators. Paid by FH CSP. May be absorbed as paid ACS by MOH. Also trained in C-IMCI. Have supplies of vitamin A Mebendazole, and ORS. Supervisors (3), Manga, Caia/Chemba and Maringue/Marromeu. The project will soon contract three more for Phase II areas. Supervise Facilitators and C￾IMCI LMs every two weeks using checklists, collect and collate monthly reports. Meet regularly with district health officers. One each for two districts. Each supervises about 5-10 Facilitators. Total 3 in phase one. Project staff. Based in Caia, Maringue and Beira during phase one. Phase 2 to add one each in Dondo, Gorongosa and Nhamatanda. CSP management technical staff:. Project manager, M&E technical officer, Training officer Train and supervise field staff, develop curricula, liaise with DPS, prepare reports for HQ. One training specialist, one M&E officer, and national project manager. Based in Beira office. Much time in field (about 50% or more). FH HQ support staff Help with curricula and M&E system development, training for some field staff, feedback on reports, reporting to USAID. 30% of FH Director of Health Programs and 35% of FH CS and Health Programs Manager’s time FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 11 As mentioned earlier, the CS project is being introduced in a phased approach. Phase one includes five districts. During the second phase, beginning just after the MTE, two changes will occur: 1) the CSP will expand the geographic area to include three more districts plus, possibly, another peri-urban neighborhood of Beira. This expansion will require that managers reduce the frequency of supervision of Supervisors in phase one districts. Secondly, Facilitators in phase one districts will begin to implement Hearth nutritional rehabilitation, which implies a reduction in the intensity of visits to Care Groups from twice to once each month. Hearth will also be introduced into phase two districts after training is complete. During phase one, topics were phased in as well in the following order: responsibility and community mobilization; water and sanitation; management of diarrhea; breastfeeding , complementary feeding, and micronutrients; nutrition during pregnancy and maternal and newborn care. Training for each module is done as a series of lessons over a period of about three months, first by training Facilitators in the entire module. These, in turn, train LMs lesson by lesson every two weeks. LMs then visit their beneficiary mothers with the new messages every two weeks. It has taken over a year to phase in all six lessons. A new development in the CSP that was not present in the Title II program is the introduction of C-IMCI. Facilitators plus one LM per Care Group received training from DPS nurses in the C-IMCI curriculum. This improves their ability to assess and refer sick children in their respective catchment areas. Additional tools (HFA, verbal autopsies, MUAC, LQAS for mini-KPCs) were also added. Finally, after all modules are in place, the CSP will undertake a large-scale nutrition screening using MUAC (originally described as WFA in the DIP, but was altered for logistical reasons). C-IMCI LMs have been trained in MUAC. Facilitators will weigh children identified as possibly malnourished to confirm their status. Those with uncomplicated malnutrition will be enrolled in the Hearth nutritional rehabilitation program. Severely malnourished children and those with complications will be referred for treatment. Facilitators have been trained in Hearth, and will supervise its implementation in each Care Group area. Monitoring is performed by regular mini-KPC surveys using LQAS, quality checklists (QIVCs), a baseline health facility assessment (focusing on IMCI), and a routine reporting system that tracks vital statistics and LM outputs. There is no reliance on MOH data for monitoring. The FH CSP is generally following the plan in the DIP, although the introduction of phase two was delayed several months until after the mid-term evaluation was completed. One of the more prominent changes has been the mix of potential partners. The primary partners are still the communities themselves and the DPS and district health offices. The CSP has been less successful in engaging NGO partners, in spite of initial commitments. Reasons are discussed in the section C.4.c.1. on Partner Strengthening. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 12 B.1.b. Summary Indicator Table Baseli ne Mini KPC Objective Indicator Feb 2006 Dec 2007 EOP Targ et Comments 1. To decrease malnutrition (underweight) in children 0-23m % of children 0–23m months who are underweight 27% 18.6 % 18% WAZ < 2.0. Weighted for population. 2. To increase exclusive breastfeeding of children 0-5m % of infants aged 0-5m months who were fed breast milk only in the last 24h 17% 95% 60% 4. To increase the proportion of young children fed nutrient-dense foods % of children 6-23m with oil added to their weaning food 35% 84% 80% 6. To decrease VAD by increasing the proportion of young children in Sofala who are regularly receiving vitamin A supplements % of children 12-23m who received one Vit A cap in the past 6 mos. 82% 91% 95% Questions differed: Baseline is asked mother and show capsule. Mini-KPC is card￾verified only. 7. To decrease helminthiasis and improve nutrition % of children 12-23m who received deworming medication in the last 6 months 24% 86% 75% Questions differed—baseline asked mother. Mini-KPC is card-verified only. 9. To increase the proportion of young children with diarrhea who are given ORT in order to decrease dehydration and death % of children 0-23m with diarrhea in the last 2 weeks who received ORS and/or recommended home fluids 71% 78% 90% Achieved the target in Sep 2007 survey and then later did not. Subsequent surveys will verify the results. 10. To increase feeding of young children during diarrhea % of children 0-23m with diarrhea in the last 2 weeks who were offered the same amount or more food during the illness 31% 70% (Sep 2007) 60% Questions differed---baseline: among those with diarrhea last 2 weeks, mini-KPC last time had diarrhea. 12. To increase the proportion of mothers of young children who know when to seek care for sick children % of mothers of children 0– 23m who know at least 3 signs of childhood illness that indicate the need for treatment 29% 85% 75% 14. To increase the proportion of mothers able to report at least 2 known maternal danger signs during the postpartum period % of mothers able to report at least 2 known maternal danger signs during the postpartum period NM 72% 80% In mini-KPC, mothers of children 6-23 months. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 13 B.1.c. Workplan Status Table Objective / Activity Objective Met? Activity Status Improve Child Nutritional Status • BCC and nutrition promotion via CGs • Home visits every two weeks • Technical training of partner staff in nutrition • Community IMCI training of LMs and Facilitators • PD/Hearth nutritional rehabilitation • Barrier Analysis to improve nutritional messages • Integration of Motivational Interviewing into CG curricula • Promotion of the production of Vit A rich foods • Vit A supplementation of children > 6m • Deworming of children >12m • Malnutrition Screening as part of Hearth Yes for phase I areas Phase I complete (except Hearth) • Materials development and training completed • Supervision ongoing • Home visits ongoing • PD study done. Hearth, anthropometry screening (MUAC) and latest mini-KPC planned for immediately after MTE. (Three conducted so far.) • Barrier Analysis complete for EBF and used in messages • Motivational interviewing training done • Promotion of Vit A rich foods ongoing • Community-based Vitamin A supplementation and deworming ongoing • Nutrition screening planned. Training completed for MUAC screening Assure Appropriate Diarrheal Case Management • BCC on diarrhea management including ORT in CGs • Technical training of partner staff in diarrhea and C￾IMCI training • ORS stocks provided to LM’s • Demonstration of preparation and use ort ORS through CGs and home visits. Yes for phase I areas Phase I complete • Diarrhea and hygiene materials development complete. BCC underway in CGs. • Community-based ORS distribution underway. Stocks adequate. • MOH and available partners trained in C-IMCI Increase the proportion of mothers of young children who have access to an IMCI-Trained Provider within one hour of their Home • Technical training of partner staff in C-IMCI training • Development of C-IMCI education modules for use by CGs Training of selected LMs in C-IMCI Yes for phase I areas Phase I complete • Training in C-IMCI complete for LMs, staff and MOH staff. No NGO partners trained. • C-IMCI curriculum complete and available for repetition. • C-IMCI materials for community use in place, but require modification. Assure the sustainability, quality & expansion of the CG model in Mozambique • Operations Research on the reasons for CG effectiveness • Presentations of FH model with MOH leaders in two adjacent provinces • Advocate for CG model in international, national, and provincial meetings • Training of selected LMs in C-IMCI • Regular supervision of LM and the use of QIVCs to improve LM health promotion • Training of CS supervisors and partner representatives in high quality health promotion Partially (replication and scaling up nationally somewhat behind schedule) • OR scheduled for final year of project • Presentation to DPS during regular meetings. USAID interested and aware, with successful presentation to MOH shortly after the MTE. • USAID funding OR in zinc-supplementation using CG model • Cabo Delgado Province MOH approved CG model use there (using Title II funds). • C-IMCI training complete in phase I areas • QIVCs in place and in regular use. • Training in verbal autopsy, health facility assessment, LQAS for partners and staff complete FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 14 B.1.d. Progress by Intervention Area B.1.d.1. Breastfeeding and child mortality Nutrition accounts for 80% of the level of effort of the CSP, and breastfeeding is a critical component of the nutrition intervention. The focus of the intervention is on household behavior change. The principal strategy is individual behavior change communication with mothers during home visits and group sessions led by trained LMs using flip-charts. Barrier Analysis was conducted on exclusive breastfeeding prior to finalizing the design of the training module and materials using a structured questionnaire. (Results of this were presented at the 2008 Global Health Council annual meeting.) The results were incorporated into the materials by focusing on messages related to the determinants found (e.g., perceived self￾efficacy, perceived social acceptability, and perceived divine will, and the lower cost of EBF). The results of the Barrier Analysis are found in the 2007 annual report. The breastfeeding and complementary feeding modules and materials were finalized and training of staff and Facilitators began in September 2007, followed by training for LMs over the following two months. Key breastfeeding messages include early initiation and colostrum, exclusive breastfeeding to six months, proper technique (e.g. giving both breasts, and emptying one before proceeding to the other) and managing common BF problems. Analysis during the positive deviance study found that emptying both breasts was positively associated (OR=7.09, p=0.006) with good nutritional status, and therefore this message was reinforced. FH implemented a series of strategies to assess child mortality including: collection of vital statistics (Facilitators collect from LMs) and implementation of verbal autopsies by Facilitators. Training for staff in verbal autopsies was done by FH headquarters staff in early 2007, followed by training for Facilitators. Vital statistics are analyzed to discern patterns of mortality among children under two, though the sample is not large and representative enough to provide mortality trends. Verbal autopsies were originally to be done on all deaths, but it proved too time-consuming to do so. Each Facilitator now performs verbal autopsies on deaths in only one Care Group (as sentinel sites). Results are used primarily to determine factors that contribute to mortality. Analysis has shown that delays in detecting illness, deciding to seek care, reaching a facility and follow-up care are important factors. Delays in receiving adequate care, once children reached a health facility were less common. i). Results baseline to mid-term The results for indicators for the breastfeeding intervention can be found in the table in Annex 4. The indicator for exclusive breastfeeding has been exceeded, reaching 95% in the most recent mini-KPC survey. In addition, all breastfeeding mothers now report feeding from both breasts and emptying both breasts. 86% of mothers now believe that immediate breastfeeding is best and 73% believe that it is not harmful to breastfeed while pregnant. (At baseline only 43% of mothers breastfed their child within one hour of birth. The question on breastfeeding and pregnancy was not included in the baseline survey, but staff feel that this was problematic before the beginning of the project based on previous KPC surveys in other Title II districts.) The most recent mini-KPC survey also found that 94% of the mothers could correctly describe how a child should be breastfed when the mother was HIV+. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 15 Analysis of mortality from October 2006 to September 2007 found that 47% of deaths were associated with malaria or fever, and another 26% with diarrhea. These two diseases are discussed in sections that follow. It is too early to analyze mortality trends from the vital statistics data, however. Focus group discussions with beneficiary mothers and other community members confirmed that the importance of exclusive breastfeeding and using proper technique are widely known in the community, even among CDC members who did not receive specific training. In addition, no group cited significant resistance or barriers to the practice. Mothers and CDC members spontaneously mentioned the benefits of colostrum and early breastfeeding, for example. Even more impressive, one CDC spontaneously described how HIV positive breastfeeding mothers should exclusively breastfeed for six months and then abruptly wean. ii). Factors affecting achievement of program objectives One factor which may have affected the measurement of the achievement of objectives is differences in wording of survey questions between the baseline survey and mini-KPCs. The baseline requires multiple denials, once to each type of food or liquid to establish exclusive breastfeeding, whereas the mini-KPC asks only if the mother is giving “anything other than breast milk, including water, other liquids or other foods”. The latter question is more likely to lead to an interpretation of “exclusive breastfeeding” than the former. In addition, the evaluation team speculated that, as mothers now know the “correct” answers to the survey questions, they may be tempted to provide the answers that are expected. (This, however, is true of all child survival projects, including those with lesser gains in EBF.) It is very possible that the final KPC survey will find a somewhat lower rate of exclusive breastfeeding than the mini-KPCs, but it will almost certainly achieve meet or exceed the project target. Although CSP staff and Facilitators felt strongly that coverage for vital statistics is complete in their areas, the number of births and deaths reported is too low to be credible for the size of the population. The number of reported births average 260 per month from December 2007 to April 2008, equal to about one percent of the total estimated target population per year. This is about one third of the minimum estimate for the birth rate in this population. Nevertheless, trends are useful, as the underreporting is likely to be similar month to month. Graphing mortality for 0-11 months, 0-23 months or the entire age range from October 2007 to June 2008 shows no clear trend (contrary to the suggestion made in the 2007 annual report using a shorter data series). A longer series may produce a clearer result. Mortality data patterns have been useful in identifying causes of death, however. For example, it was the mortality data that identified the high proportion of neonatal deaths and deaths associated with fever. iii). Contributing factors for objectives not fully achieved (constraints) The evaluation team could not identify any significant constraints to exclusive and early breastfeeding that were not overcome by the project. FH identified resistance to continuing breastfeeding during subsequent pregnancy early-on as a potential barrier to breastfeeding, as well as low perceived self-efficacy, perceived social norms, and perceived divine will. Mothers who EBF were also much more likely to mention low cost as a benefit of EBF (OR=4.5, p<0.05). Specific messages were included in training materials and flip-charts to FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 16 overcome this, and this approach seems to have been effective, as confirmed by mini-KPCs and focus groups. Vital statistics registration data from October 2007 to June 2008 show that 28% of deaths under two occurred among children under one month of age. Reduction in neonatal mortality requires different interventions than for older children. FH has begun to address this potential obstacle by encouraging C-IMCI trained LMs to visit households of newborns more frequently during the first week of life (e.g. daily) at which point LMs will begin to visit them more frequently (twice a week for the first month). This may help reduce delays in care￾seeking, which is important as symptoms in newborns can be subtle, and progression to death can be very rapid. Focus group discussions with LMs and C-IMCI LMs during the MTE confirmed that this would be a feasible strategy and that LMs were willing to implement it. Initially during the evaluation, there was some indication of cultural resistance to viewing newborns before the cord falls off. However, this was brought up by only one focus group (Manga). In all other FGDs, mothers, LMs and others said that this was an old belief that was no longer practiced and that there should be no resistance to LMs visiting and inspecting newborns in the first days of life. In addition to this strategy, FH has intensified training and messages regarding danger signs in the newborn period by including these in the most recent flip-chart accompanied by a song (to help make the message memorable) listing newborn danger signs. Finally, FH continues to encourage institutional deliveries for all mothers. All of these, together with encouraging regular antenatal care, are reasonable and likely to be effective in reducing neonatal mortality. Many facilitators of FGDs during the evaluation heard complaints that young mothers and single mothers do not listen to project messages and do not take proper care of their infants. One young mother (and third wife) who was interviewed stated that she was “too young to understand those things” so she was not enrolled in the CSP and didn’t receive visits from LMs. Cross-tabulations done at baseline using KPC data found that: ¾ Younger mothers (< 28 years, the median age of respondents) were more than three times as likely to have a severely malnourished child than older mothers (≥ 28 years, OR=3.56, CI: 1.58-8.11, p = 0.0005). ¾ Interestingly, younger mothers were also more than 2.5 times more likely to feed their child three times per day (non-liquid feeds) than older mothers (OR = 2.47, CI: 1.1-5.6, p = 0.02). Given this, FH should develop a strategy for giving these younger mothers additional support. One possible strategy would be to have C-IMCI mothers visit these mothers more frequently and help reinforce messages heard from their usual LM. It would also be useful to return to the baseline KPC data set and look for trends in key child survival practices by age of the mother that may be behind this trend in malnutrition. The verbal autopsy form should also be modified so as to collect the mother’s age. An unexpected obstacle that has been identified during the MTE FGDs is that second and third wives may be excluded from attending Care Group meetings. The team indicates that this may be a common phenomenon, and they will investigate this further. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 17 Recommendation Given the continued high proportion of neonatal mortality, FH may wish to consider three additional interventions: 1) a more in-depth assessment of the circumstances surrounding newborn mortality (prematurity, prolonged labor, facility-based delivery, other factors) using modified and specific verbal autopsies. 2) provide refresher training for obstetrical nurses in improved newborn assessment and discuss with districts a policy of keeping high-risk mothers and newborns in facilities longer before discharge. 3) refresher training for nurses, Facilitators, C-IMCI LMs and LMs in management of premature infants (kangaroo method) if prematurity is identified as a significant contributing factor in newborn mortality. If exclusion of second and third wives in Care Group meetings is a problem in a particular community, the Facilitator can raise this issue with the Community Development Committee and enlist their support in reasoning with those involved to change this practice. Alternatively, LMs should be encouraged to provide one-on-one education to these mothers whenever possible. iv). Special outcomes and unexpected successes No special outcomes or unexpected successes were identified as yet for breastfeeding and mortality except the extraordinarily low level of cultural resistance to the orientation, and the focus group evidence that exclusive breastfeeding has been adopted as a social norm and will therefore likely be sustained after the end of the project. B.1.d.2. Complementary feeding, micronutrients, child malnutrition and maternal care and nutrition Together with breastfeeding, these interventions comprise 80% of the level of effort of the CSP, and have received considerable attention. The primary strategies for addressing nutrition have been much the same as with breastfeeding, including intensive individual education of mothers using flip-charts, encouraging mothers to attend regular growth-monitoring sessions, encouraging antenatal care, institutional deliveries and post-natal care. In addition, the CSP is supporting community-based vitamin A supplementation for children and lactating women and deworming for children. Key messages follow international norms, including those for pregnant women (rest and eat more food and nutritious food). Interpretation of growth monitoring charts was included in C-IMCI training for Facilitators and C-IMCI LMs. Health facility nurses refer children identified as malnourished during facility-based growth￾monitoring for outpatient supplementary feeding at a district health facility (moderately malnourished) or in-patient therapeutic feeding in Beira (severely malnourished or with complications). In September of 2005, FH carried out a Local Determinants of Malnutrition (Expanded Positive Deviance) Study as part of the Title II program in Nhamatanda, Gorongosa, Caia, and Marromeu. The analysis was useful in several ways. First, key behaviors were identified that are associated with good nutrition: adding oil, giving three or more meals per day, breast feeding on each breast long enough to allow the high protein and fat hind milk to flow, and, curiously, feeding children cabbage (this last observation requires a longer technical discussion that is beyond the scope of this evaluation). Other PD foods including maize, FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 18 whole grains, fish, beans and cassava were also associated with good nutrition, as was water purification. This led to greater emphasis on these messages. In addition, FH has adapted weaning food recipes specific to each district based on the most easily-available foods. While these recipes will be formally introduced when the Hearth program begins, they are being used informally by Facilitators and C-IMCI LMs in their meetings currently. The DPS have supplied C-IMCI LMs with Vitamin A capsules, Mebendazole and ORS for community-based distribution during the last year. The MOH/DPS has had a long-standing policy of regular deworming for children but lacked sufficient quantities of de-worming medication to implement this policy. FH has been able to provide additional Mebendazole/Albendazole through a donation from the US. FH has been supporting mass campaigns with logistics and personnel, but has notified the DPS that it will no longer be able to continue as the project enters its second phase, as this was not included in the original workplan and budget and is not sustainable long-term. Maternal nutrition is included in the CSP, but has received less attention than child nutrition in the CSP. The training materials do include information on danger signs during pregnancy and post-partum, and the importance of iron supplementation during pregnancy and post￾partum period. Training in micronutrients and maternal care and nutrition was done as the final module in 2008. v). Results baseline to mid-term Results for key indicators are summarized in Appendix 4. All indicators for children were achieved or surpassed with the exception of vitamin A coverage in children (91% coverage, 95% target), which has an unusually high target. Indicators for maternal nutrition are not being measured by surveys due to the need for a separate sample. There are neither baseline nor interim values for post-partum vitamin A and iron. According to the CSP M&E system, the number of vitamin A capsules given post-partum showed a dramatic increase between March and April 2008 (21 to 242), corresponding to Facilitators receiving vitamin A capsules for community distribution. This was only slightly fewer than the 311 births reported for that month, and the results for May and June appear to be similar. However, prior to this, the numbers vary so much from month due to campaigns, only time will tell whether coverage will remain high. The proportion of mothers who know two or more post-partum danger signs has reached 72% against a target of 80%. Weight-for-age was measured at baseline and again in September 2007. The proportion of children with a weight-for-age Z-score < 2.0 was 27.1% at baseline and 18.6% at follow-up3 , showing a significant reduction in malnutrition (a reduction of about 32%), and nearly achieving the target of 18%. In FGDs with beneficiaries, LMs, CDCs and Facilitators, all were aware of key nutrition messages and the importance of GM. Few communities cited distance to health facilities as a significant barrier to GM, even though they must walk up to several hours. 3 These are population-weighted estimates. FH did not provide confidence intervals for the weighted estimates, but these are about ±4 percentage points for the unweighted estimates. The baseline and follow-up estimates are almost certainly significantly different. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 19 DPS data were analyzed for selected relevant indicators from January – June for three consecutive years. In each case, data were collected from three project districts (Caia, Chemba and Maringue) and from three control districts (Buzi, Chibabava and Dondo) where there have been neither Title II nor CS interventions. In each case, an attempt was made to convert raw outputs into rates in order to correct for the much larger population size in the control districts. Controls were used in order to attempt to correct for any improvements attributed to MOH efforts in the province. The results are summarized in the graphs in this section. Coverage of Vit A per Child 0% 20% 40% 60% 80% 100% 120% 140% 160% 1s 2006 1s 2007 1s 2008 Period % coverage Caia, Chemba, Mar Buzi, Chib, Dondo Weighings per child 0% 50% 100% 150% 1s 2006 1s 2007 1s 2008 Period Caia, Chemba, Mar. Buzi, Cjhib, Dondo Subsequent ANC Consults 0 1 2 3 4 1s 2006 1s 2007 1s 2008 Period Caia, Chemba, Maring Buzi, Chib, Dondo Initial ANC Consults 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1s 2006 1s 2007 1s 2008 Period Caia, Chemba, Maring Buzi, Chib, Dondo Est. Coverage for Institutional Deliveries 0 0.2 0.4 0.6 0.8 1 1s 2006 1s 2007 1s 2008 Period Caia, Chemba, Maring Buzi, Chib, Dondo The values in the graphs are not as important as the trends, which clearly demonstrate greater improvement in CSP target zones than control zones for each of the indicators. This demonstrates the utility of using the MOH HMIS data to monitor outputs for the project and provides independent supportive evidence for the project’s impact. vi). Factors affecting achievement of program objectives As with breastfeeding, there have been no significant cultural barriers which have not been overcome by the project to the adoption of the desired nutrition behaviors. The evaluation team confirmed this during FGDs with beneficiaries, LMs and CDC members. The fact that the DPS has agreed to allow community-based distribution of vitamin A and Mebendazole is FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 20 positive, as it is likely to improve coverage as well as boosting the credibility of the C-IMCI LMs. vii). Contributing factors for objectives not fully achieved (constraints) The DIP already describes how the MOH policy prohibiting community-based weighing would negatively affect the project, and how this would be ameliorated by having LMs accompany mothers to GM sessions and provide nutrition counseling at that time. This has not proven feasible. Likewise, recording nutritional status in a “behavior box” on the GM card as described in the DIP has not been possible, and FH has not printed separate cards. Instead, mothers are referred for GM at health facilities, and counseling at that time continues to be superficial and largely inadequate, as GM sessions are usually performed en masse together with immunization. Little time or attention is given to individualized nutrition counseling. In many facilities, nurses do not consistently graph the child’s weight, but rather, the weight is simply recorded. Although FH has not formally evaluated the quality of GM, the health facility assessment found that the “proportion of children who were underweight who were correctly evaluated (checking on breastfeeding, other foods given, and feeding during illness) by the health worker” during a sick-child visit was 0%. This has not affected project objectives directly, but may hinder the effectiveness of GM as a strategy to reduce malnutrition. GM is a notoriously complex activity with many components: care seeking, proper weighing, graphing, interpretation and nutrition counseling. If any one of these is not performed properly, the entire effort is wasted. Although Facilitators and C-IMCI LMs are trained to read GM charts, these are not routinely evaluated during visits unless a child is ill (and then, not consistently), as C-IMCI LMs do not routinely visit all 168 of the mothers in their Care Group. The CSP will soon screen all participating children for malnutrition using MUAC in preparation for Hearth. However, there is as yet no consistent mechanism to identify children before they become malnourished (that is, growth failure). FH’s strategy to deal with this is to provide intensive nutrition education to mothers of all children, regardless of their nutritional status. This is appropriate under the circumstances, as it is beyond the scope of the project to improve the quality of facility-based GM. Encouraging mothers to have their children weighed regularly still has value, however, as it reinforces the message that children should grow, provides an opportunity to review immunization, supplementation and deworming status, and provides an opportunity for a public health nurse to examine the child and talk to the mother, however briefly. It may not be feasible (or appropriate) to implement the Hearth nutrition model in some sparsely populated areas due to the low prevalence of malnutrition (below 20%). It may be necessary to use Hearth only in areas with sufficient population density to make it feasible to gather over ten malnourished children together. In other areas, FH may consider simply redoubling efforts in nutrition messages for all mothers, and to include the messages on nutritious weaning foods for all mothers. Analysis of the survey questionnaires found some differences in some questions between the KPC and the mini-KPCs for deworming and vitamin A coverage that may affect the results. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 21 The baseline KPC recorded mothers’ yes/no responses after the interviewer showed her a pill of Mebendazole or a capsule of vitamin A. The mini-KPCs rely on examination of the GM card. Surprisingly, the latter yielded higher coverage than the former, showing that mothers do not have a clear understanding of the purpose of what their children are given. Recommendation Post-partum maternal supplementation with iron and vitamin A is being monitored using the routine information system only (data from Facilitators). This system probably captures community-administered post-partum vitamin A quite well, but is less useful in registering facility-administered vitamin A. FH should supplement their community-based data with facility-based data for this indicator. FH may wish to monitor other health facility-specific outputs from the HMIS rather than attempting to incorporate further indicators into the CSP information system at this stage. The fact that FH will no longer support campaigns may reduce the high coverage for vitamin A supplementation and deparasitization. Community-based distribution of vitamin A and Mebendazole may help reduce this potential problem. Good monitoring will be essential to measure any trends and evaluate community-based distribution. Some districts have significant numbers of adherents to a religious sect that shuns health care, including ORS, supplements, and vaccines. Many of these families have either been excluded as they tend to reside in their own communities, or they have excluded themselves. Some mothers from this sect have sought counseling from Care Groups and services from the MOH. This may be appropriate at this time, as the project is not yet covering all eligible families and communities in the target zone. If these groups are to be included at a later time, the home￾care and nutrition messages would be appropriate. viii). Special outcomes and unexpected successes The evaluation team noted that mothers now continue to feed their children during illness, and they value local products more than before. This was confirmed by mini-KPC data and in FGDs. B.1.d.3. Prevention and management of diarrhea The diarrhea and hygiene intervention comprises 20% of the LOE of the CSP. The strategies adopted are similar to those for the nutrition intervention, including individual and group education of mothers by LMs using flip-charts, and community-based distribution of ORS. The diarrhea intervention also relies on training and supervision of Facilitators and C-IMCI LMs in C-IMCI, and implementation of a patient referral form for sick children. Messages include danger signs for sick children (including songs with the danger signs), home management of diarrhea (home fluids, proper mixing of ORS, continued BF and feeding), proper hand washing with soap/ash (time and technique---including a song that should be sung while washing the hands that lasts 30 seconds), promotion of hand washing stations that use little water (i.e., using the “tippy tap”) and proper latrine placement and use (including diagrams), proper disposal of children’s feces, handling and covering of food, and water FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 22 purification (using hypochlorite or other commercial products). Messages conform to international best practices. Finally, FH performs health facility assessments in an attempt to identify weaknesses in the quality of care for sick children and shares the results with district health officials and local NGOs so that they can address these issues. ix). Results baseline to mid-term Progress toward targets for CSP indicators for the diarrhea and hygiene intervention are summarized in the results table in Appendix 4. Targets were exceeded for feeding during diarrhea, knowledge of how to prepare ORS and danger signs for ill children. Use of ORT during diarrhea has been both above and below target in mini-KPCs, most recently 78% against a target of 90%. Although targets were not established for water, sanitation, and hygiene indicators, the mini-KPC surveys indicate that coverage is now almost universal (over 95%) for covering food after preparation, hand washing at proper times, having hand washing supplies at the hand washing station, and use of soap/water or ash for hand washing. Coverage is around 80% for water purification and proper disposal of children’s feces. During FGDs, it was learned that many families had either built latrines or moved them farther from the house after orientation by the project. In addition, the team heard repeated appeals from mothers and LMs for soap as an “incentive”. There is strong demand for soap in the communities, though it is not known whether it is for hand washing or washing clothes. The evaluation team was able to observe new hand washing stands (with tippy-taps) and dish￾drying racks in homes during visits to the communities. In several focus groups, older women mentioned that their daughters-in-law had cleaner houses than before. Beneficiaries, LMs and CDC members were knowledgeable about the home management of diarrhea and danger signs. An attempt was made to extract data from the HMIS to confirm increased care-seeking for childhood illness. However, the HMIS combines all pediatric consults regardless of age, and does not specify the type of illness, making it difficult to discern clear trends. During the evaluation, health workers and district health officials repeatedly described how the project has improved care-seeking and increased demand at facilities. One district health director described how several months earlier, she noticed an abrupt fall in pediatric consults. This led to the discovery that the CSP Facilitator had abandoned his post. Recommendation Although routine HMIS summary reports are not disaggregated, it may be useful to periodically extract usage data from registries in health facilities in the project zone as part of routine data collection. The registries contain the necessary information on age and presenting complaint, and could be used to assess care-seeking behavior. Sample data could be extracted regularly (quarterly?) with little effort by the project Supervisors in order to triangulate with the results of the mini-KPC surveys. In February 2007, FH performed a health facility assessment using the health worker observation and mother exit interview modules of the BASICS health facility assessment tool. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 23 This tool assesses the quality of sick child services and implementation of IMCI protocols by health providers. Results showed good results for courteousness, weighing and checking immunizations and medicine dosing and explanation. Assessment of the presenting complaint was fair, but there were deficiencies in assessment for respiratory problems (respiratory rate, chest indrawing, no timing device), thermometer use, assessment of feeding, assessment for all diseases (not just presenting complaint) and testing for malaria. The results were shared with district and provincial health officials during their meetings with district-level health providers, although it is not known whether actions have been taken at the health facility level to remedy the weaknesses that were detected. A follow-on survey may be done later in the project. This would be advisable only if there is strong interest from the provincial and district officials to make changes based on Health Facility Assessment findings. x). Factors affecting achievement of program objectives Once again, there appeared to be little or no cultural resistance to adopting the desired behaviors in the community that the project was unable to overcome. Furthermore, in most communities the distance to the nearest health facility was not raised as a serious obstacle to care-seeking if a child was ill. Beneficiary mothers stated that they are generally well￾received in health facilities, and those arriving with referral forms are frequently given priority treatment (though not in all facilities). Health workers in most facilities who were interviewed were aware of the project referral forms and reported receiving frequent referrals from communities either with a form or without. These same workers reported that the project was responsible for an increase in care-seeking from their catchment area, and that mothers were generally well-informed about childhood illness danger signs. Mothers of children were reportedly seeking care at an earlier stage of illness than they did before the project began. The MOH distributes liquid chlorine for water purification free of charge at all health facilities, and stocks appeared to be good (no stockouts were reported to the evaluation team when they asked). In addition, the Red Cross supports “activists” stationed at water sources in many communities. Activists put chlorine in water containers at the time of collection as part of the cholera prevention program. It is not known how long this activity will continue, however. PSI also sells a commercial water purification product at very low cost through many channels (Certeza). However, in FGDs, most rural mothers reported using chlorine bleach, as it is free and easily available. As with other interventions, there appears to be little resistance to water chlorination in the communities in spite of the disagreeable smell. xi). Contributing factors for objectives not fully achieved (constraints) Unlike breastfeeding and nutrition, the diarrhea intervention (and care-seeking for other illnesses) requires access to quality health care at an affordable cost. The health facility assessment identified some weaknesses in the quality of care. In addition, the distance to health facilities poses a significant barrier to care-seeking in some communities. Some health districts have implemented mobile brigades that provide basic services, and some communities reported taking advantage of these, although most report that they are too infrequent and sometimes they are suspended. According to DPS and district officials, the major obstacle to expanding these brigades is a lack of sufficient funds for fuel and per diems. This problem is beyond the scope of the CSP to resolve. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 24 Almost all rural communities deny that cost is an important obstacle to care-seeking, as consults are free and medicines are very inexpensive (0.5 meticais, about $0.02 per prescription), and stockouts of essential medicines (antimalarials, ORS and essential antibiotics) are reportedly uncommon. This is not the case in the urban area, however, where mothers and LMs complained about frequent stockouts and the high cost of medicines at local pharmacies. Although antimalarials are distributed free of charge, the prescriptions are often “bundled together” with other drugs (such as antipyretics), thus incurring a charge, which in urban hospitals is about US$2.00 per prescription. Community-based ORS distribution will help overcome this obstacle for ORS, but will not be useful for malaria. Recommendation The project should discuss with the DPS the option of allowing printing of pre-printed prescriptions for malaria medicines, and supplying these to health providers in urban facilities. This may inhibit them from “bundling” prescriptions and may help improve access to free malaria medicine. The cost to the CSP would be negligible (though admittedly not sustainable). This recommendation is only for consideration, as it is recognized that malaria is not an intervention in this CSP and access to ORS was not identified as a serious problem. The evaluation team suggested that the occasional stockouts of ORS could be a cause of the slight underperformance on the ORT indicator. This does not seem like a plausible explanation, however, as “appropriate home fluids” also are considered as ORT. One curious observation made by the evaluation team is that – in some communities – there is resistance to sharing a latrine among certain family members (e.g. the daughter cannot use the same latrine as the father, and during the post-partum period mothers cannot use the same latrine as any man). Further discussion with community members may be needed in order to find an acceptable solution to this problem. While discussing plans for the evaluation, care-seeking from traditional healers was cited by the team as a topic for investigation, as CSP staff felt that a significant proportion of mothers were still seeking care from traditional healers (based on baseline KPC and verbal autopsy results). In the baseline KPC, of the 50% of mothers who sought care for their ill child, about 10% reported having sought care from a traditional healer as first line care, and 12% for second line care. Project staff found through verbal autopsies that – of the 87% who sought care for their child – 32% sought care from traditional healers and 18% from religious leaders. One strategy suggested for dealing with this was to train traditional healers to manage diarrhea and to recognize and refer serious illnesses in children. Questions about care-seeking from traditional healers were included in the FGD guides for the evaluation. Although there were some differences in opinion, the most common response was that traditional healers should not be trained, and that mothers should be discouraged from seeking care from them. Community members pointed out that traditional healers are often more expensive than health care facilities, and that they are more interested in making money than curing disease. There were no important differences in the frequency of responses between mothers, LMs and CDC members. Project staff should continue to promote usage of the C-IMCI trained LMs when children are sick in lieu of traditional healers and religious leaders. The project could also FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 25 explore if communities would be in favor of religious leaders receiving some training on assessment and referral. xii). Special outcomes and unexpected successes The improvements in water and hygiene behaviors to high (>80) and near universal (>95%) levels are remarkable. No other special outcomes or unexpected successes were identified by the evaluation team. Given that mothers’ hand washing with soap has recently been found to lead to dramatic decreases in both pneumonia and diarrhea prevalence, this intervention may have also had an impact on pneumonia (which is not one of the main project interventions). B.1.d.4. Other interventions (malaria, zinc, other) Although the project was designed with only nutrition and diarrhea/hygiene interventions, the CSP has directly and indirectly supported some other interventions. Some have already been indirectly discussed above, including encouragement of antenatal and post-partum care and institutional deliveries. C-IMCI training for Facilitators and C-IMCI LMs included recognition and referral to health facilities of suspected cases of malaria and pneumonia. Fever and pneumonia are included as a symptom on referral forms. In addition, CSP staff distributed 2,100 long-lasting ITNs in the project zone, primarily to staff, Facilitators and LMs. During focus groups, demand for ITNs was found to be very high in all communities, and every focus group where it was discussed complained loudly about the lack of access to affordable and accessible nets. Clearly, widespread demand has already been created, and if nets were available, there are strong indications that they would be utilized. Recommendation CSP staff may be able to secure prioritized supply of ITNs by coordinating with the provincial malaria control officer. Together, they should contact the National Malaria Control Program to request nets,. In justifying a prioritized consignment of ITNs, they should emphasize that the province has an NGO that has created demand and would provide logistical support for storage and distribution. This would be an excellent opportunity for FH/Mozambique to take advantage of its underutilized transportation and warehousing facilities in Beira. The CSP has received additional funding to undertake an operations research study on zinc supplementation for management of diarrhea. The pilot will be done in one rural and one urban district, where zinc tablets will be distributed for treatment of diarrhea. The study will investigate various issues including ways CHW (e.g., LMs and C-IMCI trained LMs) can increase the correct usage of ORT and Zinc in the treatment of diarrhea and the development of effective messaging to overcome barriers to use of ORT and Zinc. B.1.b.5. New and Innovative Tools or Approaches The design of the CSP stresses BCC to change household behaviors and community mobilization in order to maximize impact and sustainability. The project design deliberately included little institutional strengthening for the MOH due to the fact that lasting improvement is difficult to achieve in the highly-centralized structure and resource-poor environment of the MOH in Mozambique. In addition, HAI is implementing a health systems strengthening project in Sofala Province. Finally, the CSP was designed as an expanded impact project, to replicate and extend the design of the Title II nutrition project using Care FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 26 Groups. This approach is quite reasonable, and FH is achieving rapid and positive results, as has already been demonstrated. Using LQAS, the CSP has identified some geographic heterogeneities in coverage and performance. Most notably, Marromeu consistently appears to have lower performance for several indicators, including vitamin A, deworming and growth monitoring, but is not lower in most nutrition and hygiene indicators (except adding oil and knowledge of how to mix ORS). In addition, fully 34% of all childhood deaths reported through the vital statistics system are from Marromeu, which represents only 10% of the project target population. This last observation remains unexplained (over-reporting?). It was noted by the evaluation team that Marromeu has several unique characteristics: it is governed by the opposition party, and therefore health staff assigned to the district are often not the best-qualified and resources are often short. Interventions that require interaction with the district health office are therefore more precarious. The project is active in only part of the district (the other half was covered by the Title II project), an area which is very sparsely populated and difficult to reach. The CSP Supervisor lives in Marromeu, but the need to split his time between Marromeu and Maringue (six hours distant) means that he is often absent. Finally, the communities nearest to the district capital are the ones that are agitating most strongly for the project to pay CDC members, LMs and C-IMCI LMs, possibly with encouragement from the district administration. A discussion on scaling up for phase two of the project is found in the section B.2.f. on Contribution to scaling up. B.2. Results: Cross-cutting approaches B.2.a. Community mobilization The CSP’s community mobilization strategy is intimately linked with its behavior change strategy. It involves two parallel strategies: the formation of thirty Community Development Committees and 150 Care Groups during the first phase of the project, and repetition of the process during phase two. As outlined above, mothers of children 0-23 months are organized into groups of twelve who elect a Leader Mother. She, in turn, receives training and teaches mothers key messages every two weeks. Each group of fourteen LMs chooses one LM for further training in C-IMCI. This volunteer C-IMCI LM is able to provide more advanced assessment and referral for sick children. This grouping of thirteen LMs and one C-IMCI LM forms a Care Group. One Facilitator (or Promoter in some reports) is trained and supervises five to six Care Groups, and in this way, can cover between 1,050 and 1,260 children 0-23 months of age4 . Community Development Committees CDCs form part of the Government of Mozambique’s formal administrative structure under the national plan for decentralization. Members serve voluntarily, and they coordinate directly with the District Administration. CDCs received initial training by FH Supervisors and 4 Assuming an average of 1.25 children 0-23 months for each mother. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 27 Facilitators regarding their role and responsibilities, as well as a brief introduction to key CSP messages and behaviors. All CDCs have continued to meet regularly and fulfill their responsibilities. This simple strategy has proven effective. In focus group discussions, CDCs described their roles as leaders and problem solvers. LMs seek them out when there are families that are resistant to adopting desired CS behaviors or who refuse to take their sick children for care. (C-IMCI LMs are often called, as well). CDCs, Administrators and district health officers cited the CDC’s role in communication and coordination for campaigns and other community efforts. According to District Administrators, this role extends to political and development activities in other sectors, as well. CDCs are closely linked with the District Administration. In one community, the CDC showed the evaluation team a new grain mill they had helped obtain for the village. That same CDC provided the evaluation team with a written report of their accomplishments as well as a list of demands (one of which was for payment of salaries). CDCs have remained active in spite of the fact that they have received little formal support from the CSP after their establishment and training. The strategy of forming CDCs was successful because it reinforced a structure that was already in existence (at least on paper) rather than forming a new structure, such as a health committee. This gives the CDC more sustainability, as it continues to receive support through the existing political and administrative structure. Care Groups This strategy also seems to have been successful. Communities are aware of the LMs and support their activities. The evaluation team could identify no resistance to home visits or mothers’ participation in educational activities. Older women stated that their lives are easier now, as their daughters and daughters-in-law can take care of their own children better and require less help. CDCs and LMs stated that in some cases, husbands encourage their wives to attend education sessions and sometimes sit in themselves. Health workers in facilities are aware of the community work, and almost all stated that they notice an increase in demand for services, which is supported by data from the HMIS (see above). Some health workers also noticed that mothers from CSP communities are better￾informed and ask more questions. Both mothers and health workers stated that patients who are referred are often given priority attention in facilities, though there were some exceptions. LMs and C-IMCI LMs have noticed that beneficiary mothers are now more aware of the meaning and importance of preventive health interventions, such as immunization, regular growth monitoring, vitamin A supplementation and deworming. They also state that interest in and demand for family planning services has increased, though this was not objectively verified by the evaluation team. (Some basic family planning messages were included in the most recent flip-chart and lesson about maternal nutrition). The Care Group structure appears to be sustainable long-term, as LMs and C-IMCI LMs are not paid. During the evaluation, all groups were asked what would happen once Facilitators FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 28 reduce the frequency of their visits or stop altogether. In all cases, LMs and C-IMCI LMs stated that their activities would continue, although they would not be able to report them to anyone. The Care Group model has drawn the attention of government, NGO, and community groups recently. During the evaluation some district health officers mentioned the potential for the Care Group structure to be used with other programs, including family planning and HIV/AIDS education to easily disseminate information amongst a large network of individuals at low cost. The power of Care Groups to sway public opinion and change behavior was so clearly recognized in Manga that the district administration briefly prohibited project supervisors from continuing their activities as the administrator thought the project was canvassing for votes in the upcoming elections. One potential constraint is the high demand for services of the C-IMCI LMs. The C-IMCI LMs are responsible for visiting their own 12 mothers, in addition to being responsible for regularly visiting the 13 other LMs in their Care Group and triaging sick children. This last activity reportedly takes a great deal of time, as the C-IMCI LMs have become the de facto first line of care. Some even reported being called “nurse” by community members. Some C￾IMCI LMs complained of being overworked, spending almost all of their time on their project-related activities and having too little time to care for their families. All of those who were asked stated that they assumed this responsibility in hopes of eventually obtaining a salary in the future, perhaps being selected as an ACS by the MOH. Recommendation: The evaluation team suggested that C-IMCI LMs be relieved of the responsibility to train their 12 mothers. Unexpected Results: One unexpected outcome was the effect on youth that was cited in some communities. One group of LMs said that youth in their community are now more interested in continuing their studies as they see the example of the Facilitators and how knowledge can lead to advancement (and paid work). The evaluation team suggested that youth be included in Care Groups in the future. Constraints---demands for payment of salaries One constraint on sustainability is the issue of salaries (or “incentives”, as they are euphemistically called). The CSP was wise in not offering incentives to LMs and C-IMCI LMs. Likewise, it was wise not to offer to pay CDCs as there is no provision for the Government of Mozambique to pay them in the future. The CSP is facing some challenges arising from demands from LMs, C-IMCI LMs and CDC members for payment of salaries. This trend is especially strong in the urban area of Marromeu where it appears that the district administration is encouraging this, and in Caia, where NGOs involved in the recent flood relief effort paid community members in money or in-kind for their efforts. Some ill-feeling results from misunderstandings, as community members assume that LMs and C-IMCI LMs are receiving payment. In addition, CSP staff and community members cited instances where MOH health facility nurses were jealous of their colleagues who FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 29 participated in project activities as they incorrectly assumed that the project was paying them “incentives”. Sustainability (Recommendation) The evaluation team suggested that the CSP can present the names of all LMs, C-IMCI LMs and CDC members to the district health offices and District Administrations together with the profile of their training and responsibilities their job descriptions in order to allow them to support, sustain and take further advantage of these structures. B.2.b. Communication for behavior change The BCC strategy used by the CSP is at once very well planned and executed, and yet very simple. It has been one of the most successful aspects of the project. The BCC strategy was based on that used in the Title II nutrition project (as well as World Relief’s use of Care Groups earlier in Gaza Province) and refined for the CS project. Message content The headquarters management team together with field staff chose the key messages based on 1) project priorities (nutrition and diarrhea) 2) international best-practices and 3) MOH policy. For breastfeeding and hand washing, FH carried out formal Barrier Analysis surveys on exclusive breastfeeding and hand washing with soap or ash to determine factors associated with successful and non-adoption of behaviors. Messages were then modified based on the results of these analyses. For example, for exclusive breastfeeding, perceived self-efficacy, perceived social acceptability (especially by relatives), and perceived divine will were discovered to be important factors in successful adoption, so messages on these determinants were reinforced. Specific lessons were introduced on overcoming challenges to EBF, and LMs were instructed to work with family members when promoting EBF. For hand washing, availability of soap and cues to remembering to wash were found to be important, so messages about substituting ash for soap (mentioned seven times in the flipchart), detailed guidance in the flipchart on how to construct “tippy taps” (which are also visual reminders for hand washing), and songs on how to remember were developed. Unfortunately, as there has been considerable staff turnover, some of the contribution of Barrier Analysis to message development was unknown to the evaluation team. Comments here were reconstructed from project documents. Additional messages were added as new priorities emerged. For example, danger signs in the newborn period were added after analysis of vital statistics uncovered the high proportion of infant deaths occurring in newborns. A review of the training materials, flipcharts and messages showed them to be well-developed and targeted. The clear and concise message development led to little wasted effort transmitting extraneous messages. Target audience Given the pattern of growth faltering as well as the mortality pattern discovered in the retrospective mortality study at baseline, the CS project targeted mothers of children 0-23 months as the primary audience for all messages, with the goal being increasing knowledge FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 30 and the adoption of key behaviors by these mothers. The only other community members specifically targeted were CDC members, who received abbreviated training in key messages as part of their initial training. Other community members received messages indirectly, either through LMs (informally) or by “diffusion” from the beneficiary mothers. However, as the project progresses, mothers will rotate in and out of the target audience as children pass two years of age and thus “graduate”, and new children are born. Over the life of the project, nearly all women of childbearing age will have been exposed to the CS messages. Although CS DIP reviewers asked FH for strategies for targeting men, FH responded that they are confident that targeting women directly, and men indirectly via CDCs is sufficient. It would appear that FH is correct: There are indirect indications through FGDs and key informant interviews that men are being exposed to messages. When asked who makes decisions in the household about care-seeking, mothers’ responses were about evenly split between men and women. Mini-KPCs and FGDs indicate that CS behaviors are being widely adopted in the community without specifically targeting men. Strategies, channels of communication and materials used Individual and small-group communication using flipcharts is by far the most important communication strategy in the CSP. A total of five flip-charts comprising seven modules have been produced: i) reaching the community, ii) water and sanitation, iii) diarrhea, iv and v) breastfeeding and introduction to complementary feeding, vi and vii) micronutrients feeding and maternal and newborn care (includes family planning). Each flip-chart is rather large (A3 paper) and printed on two-sides, the front with color illustrations and the back (black and white) with instructions on how to present the messages. All materials were field￾tested before being reproduced. Each module is accompanied by a detailed written lesson plan that is to be used by Facilitators as they train LMs. A total of 2,135 sets of the six flip-charts have been reproduced and distributed. Introduction of modules and messages was staged and cascaded. For each module, Supervisors in Beira were trained, and then repeated this training for Facilitators in each district. The training includes not only the content of the messages, but how to follow the detailed lesson plan. Facilitators, in turn, train LMs over the following six to eight weeks during supervision and training meetings held every two weeks. Each stage of training is supervised by CS Supervisors using quality checklists (QIVCs). LMs, in turn, disseminate the messages that they have just learned (and those they have previously learned) to beneficiary mothers during home visits and group education sessions. These sessions, in turn, are supervised by Facilitators using quality checklists. Songs were designed by staff and introduced during the training in order to help mothers memorize “lists”, such as danger signs or the steps in preparing ORS. Minor communication channels included messages printed on capulanas (traditional wrap￾around dresses) distributed to LMs and on uniforms given to Facilitators. A total of 2,100 capulanas and 33 t-shirts have been printed with CS messages and distributed yearly since the project began. Although radio is mentioned in the DIP, the project has not yet explored this means of communication. Church leaders are also mentioned, but have not yet been engaged. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 31 Health workers give educational talks in health facilities, and flip-charts were distributed to all facilities for use. These were visible in all facilities visited, but it was unclear how much they are being used. Only a total of thirteen MCH nurses received C-IMCI training, and most health facility staff have not been formally trained in the LM curriculum and use of the flip￾charts. The “extensive health worker training and joint health worker supervision using quality checklists” described in the DIP have not been possible because the MOH has not been able to provide the per diems and other associated training costs as originally stated in the MOU between the MOH and FH. The CSP budget was based on the assumption that the MOH would be able to abide by the MOU agreement. The inability of the MOH to secure funds for training of MOH staff by the CSP project has meant that the project has had to cover MOH per diems, a problem magnified by the fact that MOH per diem rate doubled in 2006. Time period required for effectiveness The previous FH Title II program that the CSP project was built upon (which in turn was built on World Relief’s early use of the model in Gaza Province), targeted only one cohort of women and worked with them for five years without replacement. FH’s experience revealed that the key behavior changes mothers needed to adopt in order to significantly reduce childhood malnutrition and mortality could be shared with mothers via Care Groups in a shorter period of time. The previous Title II programs repeated messages over the five year period and found that individuals willing to adopt new behaviors usually did so within the first six to twelve months of being introduced to the new information and supported in their behavior change attempts. Considering that CS targets have essentially been met within the first 2.5 years of the project (and less than two years of community-level health promotion), the CSP project experience would appear to confirm that significant behavior change can be achieved in the 2.5 year time period. It will be interesting to see if the final survey reveals whether Phase I beneficiaries can sustain the behavior changes documented to date without the same intensity of follow-up provided during Phase I. Evidence for effectiveness The best evidence for the effectiveness of approach is described in the technical sections above for the various interventions. Almost all targets have been met or exceeded, and behaviors have been widely adopted and accepted in the community. Lessons learned As with studies and surveys, MOH policy requires that all BCC materials be pre-approved by the MOH before deployment, a lengthy and bureaucratic process. It was not possible to obtain prior approval for materials prior to reproduction. The CS team simply made certain that all messages conformed to MOH guidelines. Recommendation: It may be useful to submit the flipcharts and training materials together with a description of the development process (Barrier Analysis, BEHAVE framework, FG testing) and outcomes (mini-KPC) for formal approval at this stage. It could be explained that phase one was done as a sort of field test prior to scaling up. This would provide the CSP with a means to introduce the Care Group model to the MOH through the MOH’s own channels. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 32 Sustainability The sustainability of behaviors using the Care Group model is described in the DIP based on a study done by World Relief in Gaza Province. In addition, evidence from the mini-KPCs and FGDs performed for the MTE indicate that communities have adopted most of the CS behaviors as the community norm. This process is essential for the sustainability of the behaviors, as once societal norms are established, they need little reinforcement to propagate and sustain themselves. Recommendations No significant changes in approach are required at this stage, as the current methodology has proven itself to be effective. The evaluation team suggested that the project may wish to explore community radio as an option for reinforcing messages, especially in Caia, Marromeu, Dondo and Beira. There is no community radio in Maringue. In addition, training for pastors and priests together with distribution of “sermon packs” with key messages substantiated by scriptural references may be an effective means to reinforce messages. FH may be in a unique position to implement the latter since, as a Christian NGO, some of the training materials are already supplemented and reinforced by scriptural references. The lack of cohesion among the many fragmented Protestant churches in the project zone precludes a more systematic or intensive approach to this strategy, however. B.2.c. Capacity-building approach B.2.c.1. Strengthening local partner organizations Capacity building of partner institutions is not a major strategy of the CSP except as it relates to scaling up and replication of the Care Group model. Efforts to strengthen the District and Provincial MOH offices will be discussed in the specific section on Health Systems Strengthening below. A number of potential partnerships with other NGOs were described in the DIP, but few of these have materialized as hoped. The following table describes the current situation with partner organizations: Organization Current relationship CUAMM (Manga only) Active case finding of malnutrition in communities, CUAMM takes advantage of the Care Groups for case-finding. CUAMM supports ambulatory supplementary nutrition programs health facilities. CUAMM activists provide oral reports to Facilitators about malnourished children who are referred. CUAMM has participated in most of the CS trainings. Health Alliance International (all districts) Provides support to health facilities and health system strengthening, especially for HIV/AIDS. This complements the FH CSP. HAI and FH meet weekly to share monitoring results and studies. This began only three months prior to the MTE. AISPO (Maringue) Is only now returning to Maringue with new funding. It is not yet clear what kind of partnership will be possible. COMUSSANAS (Buzi, Chibabava) Newly-arrived. Will act in Buzi and Chibabava and has expressed interest in adopting the Care Group model. Attended the first day session of the MTE. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 33 There is further discussion about partners, scaling up and replication in Section H : Sustainability. The overall lesson learned in attempting to partner with NGOs is that the partnership and especially funding should be planned from the beginning. NGOs have their own limited funding and each has its own agenda. It is unrealistic to expect most NGOs to change their strategies unless supplemental funding is available. The intensive focus on communities and household behaviors has led to the inadvertent strengthening of local government through the establishment and support of CDCs. The District Administrators and CDCs both described their roles as much broader than only in health. The CDCs serve as the primary contact between District Administration and District health offices and the communities, including development activities and political campaigning. B.2.c.2 Training Training and systematic supervision form the cornerstones of the CSP implementation strategy. Most of the training effort has been directed toward improving BCC and nutrition efforts. For example, much training done by FH headquarters relates to training in mini-KPC and LQAS, verbal autopsy, Barrier Analysis and Hearth. These aim to monitor and improve the quality and targeting of BCC and nutrition efforts. Training to improve health facility performance was limited to the project orientation, C-IMCI training and the health facility assessment. Both annual reports have a list of training activities carried out each year. The following table outlines training performed since the last annual report: No. trained Subject Trainer Who was trained Male Female Pre test Postte st Breastfeeding and introduction of complementary feeding (5 days) Luciano, Emma Supervisors, facilitators, district health officers 23 10 33 % 94 % Micronutrients, feeding and maternal care (5 days) Luciano, Emma Supervisors, facilitators and district health officers 23 11 65 % 89.5 % Conducting Mini￾KPCs, verbal autopsies and health facility assessments (7 days) Tom Davis, Lauren Erickson HNP Manager, CSP Manager, M&E Spec, M&E Tech, 6 Facilitators, MOH Rep. 8 7 54% 87% Hearth – and Nutritious weaning foods (5 days ) Carolyn Wetzel Supervisors, facilitators, CS training officer, district health officers, CUAMM and COMISSANAS 25 10 36 % 77 % Review of verbal autopsy results and health facility assessment May 5-9, 2008 Tom Davis Project officers, CUAMM, HAI. 6 6 None None FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 34 After Facilitators were trained on each of the last two modules, they proceeded to train LMs in the communities during the following weeks, as with previous modules. Training has closely followed the training plan as laid out in the DIP in both order, timing and approximate number of people trained. Some modules have been combined in order to reduce costs related to contracting artists, printing the albums, and training staff. There was a slight delay of several months in the C-IMCI training for LMs due to difficulties in locating and reproducing C-IMCI materials. There has also been a slight delay of about three months in moving into the planned phase II expansion. Module one of phase II was to have begun in July 2008, but was delayed until after the MTE. This is a fairly minor delay given the challenges imposed by changing the phase II geographic area. It should not adversely affect the overall project outcome significantly. Also, phase I Facilitators and LMs were to have begun training in supplementary modules, including malaria and HIV/AIDS in June 2008, but have only now finished the last two modules: maternal and newborn care. These supplementary modules have not yet been designed, and FH is currently evaluating the additional cost of reproduction of flipcharts against available funds. Training for BCC Project data, comments by staff and observations during the MTE indicate that the training cascade for BCC has been effective, as demonstrated by the final result: behavior change and reduction in malnutrition. Training materials are complete, including content, methods and exercises, pre-and post-tests, and supervision quality checklists for observation of training sessions in cascade. The 2007 annual report presents a chart of the results of these Quality Improvement and Verification Checklists as FH Supervisors observed Facilitators training LMs. This chart demonstrates steady improvement in average checklist scores for group education sessions from 73% to 83% and for individual education sessions from 72% to 93% over a seven-month period. This demonstrates the utility of FH’s quality checklists. Two observations were made by the evaluation team. The verbal post-test methodology that is to be used during LM training is not being systematically used by Facilitators. This is due to reluctance of Facilitators to put LMs “on the spot” and lose face by showing that they may not know something. This shortcoming is at least partially compensated for during supervision visits by the Facilitator to observe LMs teaching mothers. This is a reasonable concern considering the cultural norms of the region, and results have been good to date. It is probably not essential to continue to insist on rigorously adhering to post-testing of individual LMs, but rather, allowing ample space for questions and clarification, and relying on LMs to demonstrate skills rather than answering questions. Project managers also noted that the written curricula for Facilitators may be too complicated for them to use consistently, as many Facilitators do not read proficiently. This was observed directly during the MTE during observation of a Facilitator training LMs. The Facilitator did not have the curriculum with him and did not follow it closely, but rather, he followed the flip-chart itself, using the messages on the back to orient himself. Excellent initial “training by doing” followed by a set of simplified written and pictorial guidelines may be more effective than detailed written lesson plans. This will become more important as the project progresses FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 35 and Facilitators are expected to train replacement LMs. FH may wish to consider preparing and piloting a streamlined curriculum guide for Facilitators during Phase II. Training in C-IMCI According to FH staff, the C-IMCI training posed significant challenges. It was more difficult to locate an approved curriculum than originally thought, and in the end it had to be procured from Maputo. During year two, FH helped the MOH train 13 DPS trainers who, in turn, trained three FH Supervisors and 150 C-IMCI LMs. FH staff pointed out some deficiencies related to the quality of the training methods used by the DPS trainers, who were not professional teachers and were reportedly somewhat didactic in their approach. The results were satisfactory, however, although FH was required to assume the unexpected cost of paying per diems to trainers. There are concerns that the training may not have been well￾institutionalized and that organizing the Phase II round may be difficult. The evaluation team suggested that FH may consider approaching the Provincial Health Training Institute about Phase II training, as it may improve the quality of training and is more likely to be able to institutionalize the training. This approach may also be useful for the BCC module training as well. The evaluation team noted that the C-IMCI materials, including flow-charts, are too complicated for Facilitators and LMs to use. They suggested developing another flip-chart with pictorial representation of C-IMCI protocols. Likewise, the C-IMCI observation checklist should be simplified if it is to be useful (it is currently not being used consistently as it is too complex). Training of partners Training of partners, especially DPS and district officers, has been less successful than training for FH staff. The evaluation team noted that district health offices and the DPS often send officials who are not directly responsible for implementing the activities in the training course. It was noted that although districts and the DPS have been included in training in almost every aspect of the CSP, they could independently implement very few of the project strategies. That is, they are not prepared to train Facilitators, use supervision checklists, perform verbal autopsies, carry out a mini-KPC using LQAS or perform a health facility assessment. The Austrian NGO Comussanas has shown interest in adopting the Care Group model in Buzi and Chibabava during Phase II. This may prove to be an opportunity for partner strengthening that was not present in Phase I. Training by HQ for the FH/Mozambique project. In contrast to the poor institutionalization of training by partners, the FH/Mozambique staff have proven that they are able to successfully apply the knowledge gained through training by FH headquarters. They successfully carried out a baseline census and KPC survey, three mini￾KPCs using LQAS, Barrier Analysis, verbal autopsies, and health facility assessment. The team is also confident in their ability to launch Hearth after the MTE. B.2.d. Health systems strengthening The FH CSP contribution to health system strengthening has been discussed piecemeal in other sections. The CSP was not designed with a significant component of health system FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 36 strengthening, choosing instead to emphasize household behaviors. Health system strengthening component as described in the DIP was limited to training in the Care Group model, involvement in C-IMCI training and supervision, joint supervision of health facility personnel and community workers and joint planning and monitoring and evaluation. It was hoped that the DPS would assume responsibility for the Care Groups at the end of the CSP. The health facility assessment is discussed more fully in the section on diarrhea above. The evaluation team suggested that the observation checklist in the health facility assessment be simplified, as it is too complicated to be used easily. The instrument was not examined by the evaluation team, so detailed suggestions were not made. The FH CSP collaborates with the DPS and district health offices through the following mechanisms: monthly district coordination and informational meetings, quarterly provincial planning and informational meetings, monthly and quarterly CSP program reports are shared, coordination of activities such as vitamin A supplementation, deparasitization and ORS supplies, FH CSP logistical support for national campaigns (to be halted after the MTE), and community mobilization and education to support campaigns (to continue). The CS team was unable to give concrete examples of improvements in health system performance that could be attributed to the project, other than improved coverage for programs that resulted from CSP interventions. After the CSP ends, the evaluation team believes that the CDCs will continue to have regular coordination meetings with District Administrators and Health officials. In addition, if the MOH decided to pay ACS and the CSP Facilitators are given preference in hiring, they will likely continue to supervise Care Groups. B.2.e. Policy and advocacy This section overlaps somewhat with the discussion in the sustainability section, as the FH CSP sustainability strategy involved scaling up of the Care Group model. Locally FH is demonstrating the feasibility of community-based distribution of vitamin A, deparasitization and ORS using supplies from District health offices and according to policy assent from the DPS. FH is attempting to demonstrate the impact and cost-effectiveness of the Care Group model, as well as demonstrating the essential role that Facilitators (or ACS) can play in making this model work effectively, specifically in training, supervision and monitoring. As FH has little presence in Maputo where policy is formulated, the project is working through the representation of the USAID mission and other large NGOs, including World Relief, which has also used the Care Group model. USAID has expressed interest in the Care Group model and is closely following the implementation and outcomes of the FH CSP. Finally, USAID has awarded FH supplemental funding for an operations research project to demonstrate the feasibility of community based zinc supplementation for the management of diarrhea. The results of this study will serve to guide MOH policy on national implementation, including using community health workers to disseminate messages and supply zinc at the community level. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 37 B.2.f. Contribution to scaling up The issues relevant to this section are discussed in greater detail in the section on sustainability that follows, as the scale-up strategy was included in the sustainability objectives for the CSP. B.2.g. Equity The initial selection of the project zone was based on need, as described in the DIP. FH CSP addresses equity in the simplest manner by including the entire population of the target areas (and almost the entire district) with either CS or Title II Care Group interventions. As Care Group membership was determined by a baseline census, almost no families were excluded, regardless of their socio-economic status or other factors. As participation is passive, that is, mothers receive home visits and do not have to “participate” in any activities, full coverage is guaranteed. The most recent mini-KPC survey found that 95% of mothers of children 0-23m of age had received a visit from a LM in the past two weeks. The most notable exceptions have occurred in the urban area, Manga, where coverage is high, but does not reach one hundred percent, as initial population estimates were low. FH staff admit that there were instances of deliberate exclusion of some community adherents of a religious sect that rejects all medical care, including ORT. The evaluation team suggested that if these are brought into the project, perhaps messages could be limited to messages regarding breastfeeding, complementary feeding, home management of diarrhea and hygiene. With regard to gender equality, FH notes that all LMs and beneficiary mothers are women. FH CSP staff are 34% female and 66% male. Facilitators were chosen by the communities where they work and the majority were community residents prior to becoming FH staff. Supervisors are mostly men due to self-selection and the need to travel long distances by motorcycle and the long time away from home (over 50% of their time). B.2.h. Sustainability Project sustainability is a high priority and is one of the four strategic objectives: “assure the sustainability, quality and expansion of the Care Group Model in Mozambique”. The following indicator was included: “Continue to expand usage and improve the Care Group model in Mozambique”, and the measurements are: “1) The MOH, in at least one other Mozambican province, requests assistance (during the life of the program) from FH to expand the Care Group model into their geographical area. 2) Operations Research is conducted on the reasons for Care Group effectiveness.” In the section on the expanded impact program, the DIP states that the third goal is “to transfer the knowledge, skills, tools, and passion needed for effective and sustainable community health development through the Care Group model to project partners – including Leader Mothers – in order to continue child survival activities once this project has ended.” From the above, and discussions with the evaluation team, it is clear that FH’s sustainability strategy occurs on three levels: sustainability of behaviors at the household and community level, engagement of key partners (especially the MOH) in assuming responsibility for continuing supervision and management, and replication and scaling-up by key partners in other parts of Mozambique. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 38 Sustainability of household behaviors: there is substantial evidence for the sustainability of household behaviors and the continued activity of LMs described in the DIP. This comes from evidence of a study in Gaza province done by World Relief twenty months after the end of a similar project using the Care Group Model which found high levels of continued participation of LMs (and replacement of many who left). This study also found that household behaviors were sustained for all project indicators. As it is using a very similar methodology, there is no reason to believe that the CSP will have greatly different results. Qualitative results in the focus groups indicate a high degree of knowledge of and support for messages among both participating mothers and CDC members who were interviewed. This is remarkable, as most CDC members are men, and they did not receive specific training in the messages. This indicates that the messages are probably diffusing through the communities, and that, over the five-year span of the project, the behaviors will likely be adopted as the social norm, ensuring their sustainability. One curious aspect of the project is the effect of beneficiary coverage and turnover. It was clear from discussions with project staff and Facilitators that coverage is not complete in most areas of the project zone, especially in Manga (urban area). This was explained by the fact that the baseline population estimates were inaccurate, as they were taken from extrapolated 1997 census data. According to staff, in at least some areas, the actual population was larger than expected, and once a quota of twelve mothers was reached for each LM, no others were sought, leaving some mothers of children under two uncovered. Staff stated that even though all newborns are automatically enrolled, when a child graduates (over two years old) or dies, it is not difficult to locate a mother of another child under two to take her place. In addition, as the project continues, with graduations, deaths and births, the participating mothers change. However, the training cycle has not been repeated (until now). Yet, both mini-KPC surveys and focus group discussions with participant mothers indicate widespread adoption of key behaviors by all current participants, who have been participating for varying periods of time, some having been only recently enrolled. This provides supportive evidence for a high degree of knowledge, acceptance and adoption of key behaviors by the entire community, and not only those mothers who were enrolled from the outset. Recommendations: If possible (and funds permit), conduct a rapid assessment in a sample of Title II districts (Nhamatanda, Gorongosa, Caia and Marromeu) to assess sustainability of the household behaviors by conducting a mini-KPC in one or more of the Title II districts during phase II of the CS project. It would be even more interesting if it were possible to include an anthropometric assessment as well, though this would increase the cost significantly. MOH and other partner support to assume ongoing support and management after the end of the project The DIP describes how district MOH staff will participate in joint supervision of Facilitators and LMs on a quarterly basis, with the eventual goal of the districts taking over the management of the Facilitators and Care Groups after the end of the CSP. Other joint project / MOH activities include regular planning and data analysis meetings, sharing of data and FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 39 reports, and the DPS providing the C-IMCI training for FH staff, Supervisors, Facilitators and C-IMCI LMs. The provincial health office signed an MOU with FH acknowledging this arrangement. However, according to the DPS, provincial and district offices have insufficient funds for travel and per diem even to meet minimal requirements, and the CSP budget did not include payment of travel and per diem to DPS and district personnel. As a result, the C-IMCI training was more expensive than originally planned, as FH was obligated to assume the cost of travel and per diems for provincial nurses. In addition, the planned joint DPS/FH supervisions have not been possible, neither for field personnel (Facilitators and LMs) nor for health facility staff. The evaluation team interviewed various district and health facility personnel about the project. In almost all cases, they expressed their satisfaction with the project, but with a focus on improvement of coverage of vitamin A and deworming, increased care-seeking for illness, uptake of antenatal care and institutional deliveries and support for campaigns. Improved household behaviors, health, breastfeeding and nutrition, hygiene, and mortality reduction were not mentioned. This provides useful insight on the point of view of health facility, district and provincial MOH officials. The situation in Manga, the peri-urban area within the urban district of Beira, is different from that in other districts with respect to engagement with the District health officers. The health facilities in the target zone in Manga are large and heavily utilized, with large catchment areas. During interviews, staff were aware of the CSP in that they knew the Facilitators and knew the FH Supervisors, and they knew they received referrals from the community and LMs. However, FH coordination has emphasized the District Health Office, which manages the entire urban area of Beira, with many health facilities and a large population. Discussions regarding the possibility of joint supervision of Facilitators and LMs have been held at the District level, but not at the facility level. Health facility staff had only superficial knowledge about project strategies and achievements. Public health nurses and directors at two facilities expressed interest in coordinating community-based activities with the CSP. Recommendation In urban areas, CSP management should arrange coordination meetings with both District and health facility staff, either together or separately. The opportunities for collaboration in supervision and community outreach are greater in the urban area than in rural areas, as travel and per diem costs are not an issue. Urban health facilities may be able to assume more responsibility for supervision of Facilitators and LMs in a way that rural districts cannot. The MOH in Mozambique has recently decided to pay a salary to Facilitators (by another name), although it is uncertain when this will begin. According to preliminary information, the MOH plans a ratio of about one Facilitator per 500 families, well within the ratio of 840 families per Facilitator in the CSP. Given the amount of training that CSP Facilitators have received, it is highly likely that these same people would be prioritized during MOH recruitment. The evaluation team asked District and Provincial health officers whether, in the event that Facilitators were paid by the MOH, would continuation of the Care Group model be a priority FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 40 activity. Answers were generally positive but non-committal. It appears that District and Provincial health officers have yet to be fully convinced that the Care Group model is the best means to achieve their goals and objectives. Another important lesson learned by the project concerns the presentation of project data. According to national regulations, all scientific studies and surveys performed in Mozambique must seek prior approval from the ethics committee of the National Institute of Health in Maputo, a lengthy and bureaucratic process. The baseline and mini-KPC surveys as well as all other assessments (health facility assessment, positive deviance assessment, Barrier Analysis, etc.) were done without this approval. Because of this, these data cannot be presented to the DPS and MOH as “survey” results, but rather, as project-specific “assessments”. According to project staff, this has had the effect of reducing somewhat the credibility of the results. Nevertheless, the DPS receive quarterly reports and the District monthly reports and monthly meetings that include the results of all studies and activities. Conclusion Although Provincial and District health officers understand the goals and objectives of the project and state that communication has been good and they are aware of many of the achievements, they still not fully engaged in the project. This may be due to the fact that the FH has been describing the project’s achievements in terms of the project’s own health indicators using project-based data (e.g. mini-KPCs, quality checklists, and project monitoring forms). It must be kept in mind that the MOH does not operate a health system, but rather a health care system, and that District and Provincial managers are evaluated on the basis of service and coverage indicators, not health. These indicators are derived from their own information systems. It is easy to dismiss project-derived data as less relevant and perhaps biased or less trustworthy than their own. Even if the DPS adopts the Care Group model as a priority and agrees to prioritize CSP Facilitators in hiring, it is unlikely that the information system used by the CSP (including registers and mini-KPCs) would be formally adopted due to their relatively high recurrent costs and the probability that a new harmonized national HIS would be adopted. Recommendation As was noted in the technical sections above, it is possible to use some indicators derived from the MOH’s HMIS to monitor coverage. In this way, the CSP will “speak the same language” as the MOH. Some of the most obvious indicators include: • number and coverage of institutional deliveries • number and coverage of new and return antenatal care consults, • number and coverage of outpatient pediatric consults for illness • number and coverage of new and return growth monitoring visits • number of doses and coverage of vitamin A supplementation • number of doses and coverage of deworming In addition, the following HMIS indicators would be useful for project monitoring, as they are not being measured in the KPC surveys: • number of visits and coverage for post-partum care FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 41 • number of doses and coverage of post-partum vitamin A The marginal cost of monitoring these is minimal, as the information is already being collected by health facilities, district and the province. By demonstrating improvements in key MOH service coverage indicators, District and Provincial health officers are more likely to pay attention to the CSP’s results. Replication and scaling up by partners The difficulties with partnering with other NGOs are described in the partnering section above. Active engagement of other NGOs for scale-up and replication has been relatively unsuccessful thus far, and the Care Group has yet to capture the attention of the MOH in Maputo. USAID has shown strong interest in the Care Group model, and Tom Davis traveled to Maputo to present FH’s results during the mid-term evaluation. USAID Mozambique is planning a review of their community health programs. The prospective partnership with Comussanas presents an unexpected opportunity. This Italian NGO is beginning a community-based health program in Buzi and Chibabava, and has expressed interest in applying the Care Group model as part of its strategy. FH is still in the process of negotiation with Comussanas as to how this will be done. At the request of the DPS, FH has requested and received approval from USAID to alter the project target zone so as to increase population coverage in some districts and to leave these two districts to Comussanas. If Comussanas applies the Care Group model in these districts, FH will count the populations as indirect beneficiaries. FH’s Title II Nutrition and Food Security project has left Sofala Province, but will be replicated in Cabo Delgado. It will replicate the Care Group model in that Province, providing yet another setting in which to demonstrate the effectiveness of this strategy. Internationally, in part due to extensive presentations on the Care Group model by FH staff at APHA, Global Health Council, the CORE Group annual meetings, USAID/ Washington meetings and other settings, the Care Group model is now being used by 12 or more PVOs in seven African countries (Angola, Burundi, Kenya, Malawi, Mozambique, Rwanda, & Zambia) as well as two other regions (Cambodia, Philippines, & Indonesia in Asia, and Bolivia and Guatemala in LAC). The model was also featured in the 2008 State of the World’s Children Report. Recommendation Encouraging the MOH to change policy and adopt the Care Group model will not be easy. First, the MOH is most concerned with management of its health care system, human resource limitations, and managing the complex donor environment in the context of decentralized management and the common fund. Policy changes are slow. If FH wishes to attract the attention of the MOH to replicate the Care Group model as policy, two strategies would likely be the most effective (not mutually exclusive): 1. Find a champion within the national MOH who understands the Care Group model and can lobby for its adoption (or trial). This strategy may be difficult for FH as the organization does not maintain a permanent office in Maputo. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 42 2. Attract the attention of the Provincial Director of Health. This is different from the DPS as a department. The DPS himself can have substantial influence in provincial and even national policy, as MOH management is decentralized in Mozambique. However, it would not be sufficient for FH to work only with department heads (MCH, for example). D. Changes in Grantee Organization Capacity (new grantees only) Not applicable. E. Mission collaboration Little can be added beyond the statements in the previous annual report. FH field management staff have worked closely with the USAID mission as evidenced by the mission’s invitation to FH/M to pilot the Zinc operational research project. During the mid-term evaluation, FH was asked by the USAID Mission to send a representative to Maputo to present the Care Group model to other NGO representatives and representatives from the MOH. As Tom Davis was in-country for the MTE, he took advantage and traveled to Maputo to make the presentation. FH was also informed that a joint delegation from the Mission and the MOH would be visiting the project shortly after the MTE to discuss the zinc pilot and the Care Group model. F. Contextual factors that have affected the program to Date The most important contextual factor that has unexpectedly affected the program is the unexpected closure of two programs, raising overhead costs for the CSP considerably. As FH is preparing to scale up for Phase II, some unexpected events have conspired to make this more difficult. The Title II project was expected to continue in the same areas as before, but has been obligated by requirements of the funding agency to relocate to another province beginning in April 2008. In addition, two offices of FH’s HIV/AIDS project will close unexpectedly in November 2008. These are distributed according to the following table: Location Child Survival (% of population) Nutrition (% of population) HIV/AIDS (by project title) Marromeu 50% 50% ABY Chemba All Caia 50% 50% CTD & ABY Maringue All Manga All Nhamatanda 50% → 80% 50% ABY Gorongosa 50% →80% 50% ABY Dondo All →80% Shaded boxes are unexpectedly closing: Nutrition in April 2008 and HIV/AIDS in November 2008. This unexpectedly leaves field offices in Marromeu and Nhamatanda with no cost￾sharing for administrative costs, and reduces cost-sharing for Caia and Gorongosa offices. There was never a source for cost sharing for Chemba, Maringue, Dondo and Manga. This leaves the FH CSP alone to support one hundred percent of the cost of eight field offices and FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 43 half of two more, rather than the planned 100% of four and 30% of four. At the same time, the cost of fuel has risen unexpectedly. Heavy flooding of the Zambezi River last year caused problems with access to Marromeu and some other regions but didn’t seriously delay project implementation. However, the influx of relief organizations has caused many communities in Caia and Marromeu to become dependant on handouts and less willing to contribute to their own well-being. This has led to unexpected management problems for FH as LMs and CDCs demand payment for their work. This has not yet seriously affected quality nor led to delays in project implementation, but it has caused difficulties for FH field staff. Recommendation FH has already sent a request to USAID for permission to modify the project’s geographic target area based on a request by the DPS to not overlap with Comussanas. At this time, FH may wish to consider amending this request to consolidate the target zone somewhat in order to reduce ballooning overhead costs. It would be justifiable for FH to consider leaving some or all of the more distant and smaller districts at the end of Phase I and adding new populations in closer geographic areas, including peri-urban areas. Marromeu comprises only 3.3% of total project beneficiaries, but is very distant and results have been the least satisfactory. Caia (5.3%) and Chemba (4.5%) are other candidates. This could even be seen as an opportunity for operations research, as to whether a continued presence or early departure produce very different long-term outcomes. It would also provide an expanded opportunity to test the Care Group model in peri-urban areas. Sofala province has an HIV prevalence rate of 24% (INS, 2007). This has affected the project in two ways. The first, is that CS staff have a high turn over and substantial amount of staff time is lost due to chronic illness, health care needs of family members, and attendance at funerals. Secondly, the high HIV rate has created numerous jobs as donors and NGO’s have increased programming in the area. As new programs start, CSP employees frequently move to other organizations where salaries or benefits are more attractive. This means that the CSP management must spend a lot of time and energy hiring and training new staff. Other factors that have adversely affected the project include the following: • Reduction of per diem for mobile brigades and outreach to Mt150 (about $6 per day) has reduced the incentive for public health nurses to go to the field. This has lowered the cost to the districts, but outreach has been significantly reduced as a result, as nurses are unwilling to undergo the hardship for so little extra money. • The unexpected need to pay per diems to MOH staff when they provided C-IMCI training raised the cost of C-IMCI training considerably. • Some prospective NGO partners have either left the area, or been unable to participate as expected. The following have positively affected the project (or potentially may affect it): • The MOH has decided to pay ACS in the future, although it is unknown when this will begin. This may positively affect project sustainability. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 44 • The fact that FH has been running HIV/AIDS care and prevention, Food Security, Civil Strengthening, and Child Development programs in some of the Phase I CS communities has reduced overhead costs and allowed a more rapid entry into these areas. G. Conclusions and recommendations In general, the project is on track and achieving its goals and objectives. The following observations and recommendations are taken from the sections above. Further detail is included in the body of the report. Breastfeeding and mortality: Neonatal mortality comprises nearly one-third of all mortality under two. This will require special interventions and messages in order to reduce it. FH has already addressed this by including neonatal danger signs in messages and encouraging C￾IMCI-trained and regular LMs to visit post-partum mothers more frequently. Some other specific suggestions are made in the text. Complementary feeding, supplementation and deworming, GM and maternal nutrition Maternal and post-partum nutrition is not being monitored well. FH may consider combining indicators from community-based monitoring with facility-based indicators to track maternal vitamin A and iron supplementation. In addition, it may not be feasible to implement Hearth in sparsely populated areas. FH may wish to concentrate on more densely populated areas and intensify nutrition messages in the other areas. Diarrhea, hygiene, malaria and other interventions: FH may wish to consider adding some indicators from the MOH HMIS to its monitoring system, especially demand for pediatric consults. Although the HMIS does not disaggregate by age, the health facility registries contain all the necessary information, and Supervisors could easily extract this information on a regular basis for monitoring purposes. In addition, the cost of medicines on “bundled” prescriptions was identified as a barrier to treatment. FH may consider implementing pre￾printed prescriptions for anti-malarials in health facilities to facilitate access to free anti￾malarials. Community mobilization: The evaluation team suggested that youth be reached through the care group structure in the future, even those without children. Due to the high workload of C-IMCI LMs, who are volunteers, the evaluation team suggested that C-IMCI LMs may be relieved of the responsibility to train their 12 mothers. Behavior change communication: No strong recommendations were made, as the BCC component is doing well. The evaluation team suggested a trial of training for pastors and priests together with distribution of “sermon packs” with key messages substantiated by scriptural references may be an effective means to reinforce messages. FH may be in a unique position to implement the latter since, as a Christian NGO, some of the training materials are already supplemented and reinforced by scriptural references Training: Written curricula in use by Facilitators may be too complicated for them to use. FH may wish to consider simplifying them for phase II. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 45 FH may consider approaching the Provincial Health Training Institute about Phase II C-IMCI training, as it may improve the quality of training and is more likely to be able to institutionalize the training. This approach may also be useful for the BCC module training for Facilitators and districts as well. FH would supply the curriculum and pay the cost of the training, but the Institute would implement it. The evaluation team commented noted that the C-IMCI materials, including flow-charts, are too complicated for Facilitators and LMs to use. They suggested developing another flip-chart with pictorial representation of C-IMCI protocols. Likewise, the C-IMCI observation checklist should be simplified if it is to be useful (it is currently not being used consistently as it is too complex). Scaling up and engagement of MOH and sustainability: It may be useful to submit the flipcharts and training materials together with a description of the development process (Barrier Analysis, BEHAVE framework, FG testing) and outcomes (mini-KPC) to the MOH for formal approval at this stage. As was noted in the technical sections above, it is possible to use some indicators derived from the MOH’s HMIS to monitor coverage. In this way, the CSP will “speak the same language” as the MOH. The text includes suggestions for specific indicators. In urban areas, CSP management should arrange coordination meetings with both District and health facility staff, either together or separately. The opportunities for collaboration in supervision and community outreach are greater in the urban area than in rural areas, as travel and per diem costs are not an issue. Encouraging the MOH to change policy and adopt the Care Group model will not be easy. First, the MOH is most concerned with management of its health care system, human resource limitations, and managing the complex donor environment in the context of decentralized management and the common fund. Policy changes are slow. If FH wishes to attract the attention of the MOH to replicate the Care Group model as policy, two strategies would likely be the most effective (not mutually exclusive): 3. Find a champion within the national MOH who understands the Care Group model and can lobby for its adoption (or trial). This strategy may be difficult for FH as the organization does not maintain a permanent office in Maputo. 4. Attract the attention of the Provincial Director of Health. This is different from the DPS as a department. The DPS himself can have substantial influence in provincial and even national policy, as MOH management is decentralized in Mozambique. However, it would not be sufficient for FH to work only with department heads (MCH, for example). Contextual issues: Due to the unexpected closing of the Title II and HIV/AIDS projects in several districts, FH overhead expenses will soon grow. FH may wish to consider consolidation of the project area for Phase II, including abandoning some very low-population and inaccessible rural areas and substituting more accessible and peri-urban areas. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 46 Human resources management: High staff turnover, especially at the lower levels has presented a challenge to project management. FH may wish to implement some staff￾incentives to encourage staff to stay on. Suggestions are included in the text. Information management: The current routine information system is unnecessarily complicated and presents a burden for Facilitators. The number of routine indicators should be reduced significantly, and some HMIS indicators may be used in their place. Specific suggestions are made in the text. H. Action plan Year 4 Year 5 MTE Action Plan 1 2 3 4 1 2 3 4 1 Present Care Group Strategy Formally to the MOH in Cabo Delgado 2 Present Care Group Strategy Formally to the MOH leader in adjacent province 3 Organize a formal presentation of the results of the MTE study with the Sofala MOH and other interested stakeholders that highlights the effectiveness of Care Groups 4 Find a champion within the national MOH who understands the Care Group model and can lobby for its adoption (or trial), for example the Sofala DPS. 5 Formalize OR on reasons for Care Group effectiveness, include information about age and wife status of mothers involved in the program 6 Create Lesson Plan Outlines (instead of the Lesson Plan manual) for Facilitator use with Flip￾Charts for Phase I & II communities 7 Simplify Project Reporting Systems by decreasing the number of indicators being monitored and including information form the MOH HMIS 8 Create simplified C-IMCI training materials 9 Submit Phase I Flip-Charts and training materials to the Provincial Health Training Institute in order to institutionalize training curricula (including C-IMCI simplified flow-chart) by seeking national MOH approval. 10 Finalize decision about Phase II geographical coverage (consolidating target population) 11 Decide which additional activities to include to reduce neonatal mortality 12 Activities to reduce neonatal mortality 13 Supply pre-printed prescriptions for malaria medicines to health providers in urban facilities to increase access to free malaria medicine. 14 Together with the Provincial Malaria control officer petition the National Malaria Control Program to request ITNs, offering to provide logistical support for storage and distribution. 15 Consider making Phase II Care Groups consist of 15 ML (instead of 14) with one LM to be responsible solely for C-IMCI visits and triage. 16 Present the names of LM, C-IMCI LM, and CDC members to the district health offices and District Administrators along with a profile of their training and job descriptions on a yearly basis to encourage the MOH and district administration to make use of the human resources trained and available in their communities. 17 Explore using radio in Caia, Marromeu, Dondo, and Beira to reinforce CG messages. 18 Explore creating sermon packs with key messages sustained by scriptural references to pastors and priests 19 Simplify the Observation checklist used in the health facility assessment FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 47 Year 4 Year 5 MTE Action Plan 1 2 3 4 1 2 3 4 20 Consider conducting a rapid assessment in a sample of Title II districts to assess sustainability of the household behaviors. 21 In urban areas, arrange coordination meetings with District and health facility staff to discuss supervision of Facilitators and LM. 22 Analyze the MUAC results from the Aug 2008 mini-KPC and determine if it's feasible to run in Hearth in all areas. 23 Present need to implement some staff incentives to encourage staff to stay on to the FH HR Manager. 24 Balance out the collection of quantitative data with the collection of more qualitative data 25 Organize discussions with logistics and the CS staff to strategize how systems can be improved to decrease staff time spent coordinating the logistical aspects of a program. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 48 I. Annexes 1. Results highlight Although it has been described in other reports, the Care Group model has proven to be such a successful strategy for achieving sustainable behavior change that it deserves another look. The primary challenge in attempting to change key household behaviors is how to achieve a level of adoption throughout the community that is high enough that the behaviors are adopted as the community norm. This must also be achieved in a short amount of time, usually less than two to three years. Only in this way can it be expected that behaviors will be sustained after the end of the project interventions. The essential elements of the Care Group model begin with inclusion of all target mothers in a group of about twelve. In this case, these were mothers of children under 24 months of age. These mothers elect a Leader Mother, a volunteer who receives training and materials for child survival messages. She then transmits these messages to the beneficiaries through regular home visits and group sessions. She, in turn, belongs to a group of about twelve Leader Mothers, who form a Care Group, the basic unit of training and supervision by project staff. There are several elements that appear essential to the success of the model. These include: 1) a very clear definition of the exact messages to be transmitted and behaviors to be adopted, 2) excellent, durable and well-tested materials, usually flip-charts, for each Leader Mother to use during her sessions 3) methodical, slow roll-out of very well-designed lessons over time and 4) systematic regular supervision using quality improvement and verification checklists (QIVCs). In the case of the FH CSP, each module was introduced in a series of lessons every two weeks over two to three months each. Leader Mothers would then take each lesson to their beneficiaries during the two weeks following each training session. In this way it was possible to introduce six full modules in about one year, comprising only two CS interventions: 1) child breastfeeding and nutrition and 2) prevention and management of diarrhea. Over a period of about two years, almost all women of childbearing age will likely participate in a Care Group, as almost all will have become pregnant at least once during this period. This leads to a high degree of saturation of the messages. The strategy is interesting as it flies in the face of some of the principles of BCC, that is, it uses only a single communication channel to communicate messages, and directly targets only a single audience. Nevertheless, the strategy has achieved impressively large and rapid behavior change that appears to be sustained after the end of the intervention. Up-front costs are considerable, especially for reproduction of materials (one full set of high￾quality color flip-charts for every twelve mothers), as well as training and supervision (one paid supervisor for every 800 mothers). However, FH has evidence that the behaviors are sustained at least two years after the end of the intervention, and that behavior change can occur within a two year time period. Therefore, the payment of supervisors during this period appears to be justified, as it does not have to be sustained long-term. One of the most important challenges to the Care Group model will be its application in urbanized settings, FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 49 where mothers work outside the home and neighbors do not generally know and trust their neighbors. 2. Publications and presentations July 2008: Community Health Programming with Impact: The Care Group Model and its Role in Mortality Reduction in Mozambique was presented to the USAID mission in Mozambique and other CS stakeholders in Maputo, Mozambique by FH CS HQ staff. June 2008: Barrier Analysis methodology, the results of the exclusive breast-feeding Barrier Analysis study, how results were used to create project messages, and the impact and affect this had in program areas was presented at the Global Health Council by FH CS HQ staff. January 2008: Community Health Programming with Impact: The Care Group Model and its Role in Mortality Reduction in Mozambique was presented to USAID and other CS stakeholders in Washington, DC by FH CS HQ staff. April 2008: Experience gained from CS project design, survey implementation, and establishing monitoring and evaluation systems was presented to YWAM community health students at the University of the Nations by FH CS HQ staff. November 2007: Care Groups significantly reduce child mortality in Mozambique was presented at the APHA annual meeting by FH CS HQ staff. October 2007, Barrier Analysis: A Tool for Successful Behavior Change Interventions in International Health Programs presented at UNC-Chapel Hill School of Public Health’s Health Behavior/Health Education Global Health Roundtable by FH CS HQ staff. June 2007: Barrier Analysis as a tool to improve health messaging and improve behavior change was shared at the CSHGP Mini-University by FH CS HQ staff. April 2007: Community Transformation in Bolivia & Mozambique through a Behavior￾change Focus presented at the International Food Aid Conference by FH CS HQ staff. May 2006: Breaking Down Barriers to Behavioral Change: Barrier Analysis was presented at a USAID brown-bag meeting in Washington, DC by FH CS HQ staff 3. Program Management I.3.a. Planning A great deal of effort went into the initial planning of this CS Project. The initial DIP planning workshop involved all potential partners, including staff from three District MOH offices, DPS, potential and potential NGO partners. During early implementation, staff held meetings with communities to discuss the project and the Care Group model, plan the baseline census and retrospective mortality study. Models and strategies were taken from the Title II Nutrition project and refined for the CS project. Many tools were designed in advance, FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 50 including detailed curricula and training guides, quality improvement checklists and routine information collection tools, among others. Overall, project implementation has closely followed the original plan in the DIP. However, two significant difficulties have arisen. First, partner NGOs were unable to participate as actively as originally expected. Although no aspect of project implementation depended on partner NGOs, it was originally expected that they would replicate and expand the reach of the Care Group model. This has not happened, due to funding and programmatic constraints of the partners. Second, local MOH participation has been disappointing. It was originally expected that the MOH would participate in training in C-IMCI and joint supervision as counterpart (and MOUs were signed to this effect). However, it became evident later that this would occur only if FH and the CS project would pay per diems and transportation costs for MOH staff. As this was not included in the CS budget, full collaboration has not been possible as was originally planned. This has not directly hindered project implementation, but may affect sustainability and ownership at the end of the project. There are no significant changes in the DIP planning or review process that could have avoided these problems, other than to be more cautious about partner participation. It may be important to include budgetary provision to guarantee MOH participation in future projects. I. 3.b. Supervision of Program Staff Supervision of program staff was remarkably regular, systematic and effective given the geographic extent of the project and the number of levels involved. The following table outlines the levels and frequency of supervision: Level Supervisor and frequency Beneficiary mother Visit at least twice each month by LM Leader Mothers and LM￾IMCI Visited twice each month by Facilitator. Twice each week by LM-C-IMCI Facilitators (Promoters) Visited twice each month by the supervisor (Manga) once each month in other districts FH Supervisors Visited once every 2 months by Luciano (Trainer) and Cecilia (M&E Specialist), every three months by Emma (CS PM) FH/Moz staff 2-3 times a year by FH/HQ Each level (except for HQ to field) uses standardized quality improvement checklists for supervision. The results of these checklists are tabulated and monitored for aggregate improvement. In addition to regular supervision, all Supervisors are able to coordinate from the field by mobile phone (except to Chemba and Maringue). The CS project has also helped Facilitators purchase mobile phones and provides them with a time quota each month. During focus groups, all staff at all levels felt that supervision was adequate and helpful. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 51 As has been described in preceding sections, joint MOH/CS supervision was planned into the project, but is not taking place as the DPS and District health offices are requesting that the project pay cover per diems for MOH staff to undertake joint supervision. In addition, in the urban area (Manga), the CS has coordinated with the Beira city health department, rather than working directly with health facilities. It may be possible to engage individual health facility staff in contact (and possibly supervision) of Facilitators simply through better coordination, as there is no travel involved. If the MOH assumes responsibility for the Facilitators/Promoters as paid ACS in the future, they will be responsible for theirs regular supervision. In any case, whether the district and provincial MOH absorb the ACS or not, the LMs will likely continue their voluntary role of teaching mothers in their respective communities. I. 3.c Human resources management FH field staff is generally satisfied with their participation in the project and with their positions in general. They all state that personnel policies are clear and clearly communicated to them, with some exceptions about the coverage details of accident insurance. It should be noted that, among FH CSP technical staff (officers and supervisors), all are seconded from the Ministry of Health and none have any prior experience working anywhere but the public sector. The exceptions are the CS Program Coordinator, who is Colombian and has international experience, and the M&E officer, who came from the private sector (accounting). The current 26 Facilitators have been recruited from the areas where they live and have a ranged of different prior experience, though many have worked with the MOH as ACSs in the past. The FH CSP has suffered from a considerable amount of staff turnover during its first half. At headquarters, the CS and Nutrition Programs Coordinator, Lauren Erickson Mamane was replaced by Carolyn Wetzel in September 2007. Carolyn had been the FH/Mozambique Health Programs Manager and did initial project start-up until Emma Hernandez arrived in country and took responsibility for the project as the FH/Moz Child Survival Project Manager. Emma Hernandez initially reported to the Health Programs Manager and currently reports to the Sofala Programs Director. The position of M&E officer was originally occupied by one part-time regional employee and another full-time local. The latter assumed the position full time with the departure of the regional person. He, in turn, left for training and was replaced by Cecília Lopes, who currently occupies the position. She was recruited from the private sector about six months prior to the evaluation. A new position, FH/Moz Training Officer was created in 2006, and was initially filled by a high-level professor. He left three months later, and the position was filled by a field Supervisor, Luciano Menete, who continues to occupy this position. The FH/Moz financial officer has been in her position for about one year. Among FH field Supervisors, the following changes have occurred: Districts Progression Current at mid-term Caia / Chemba 1 (died) → 1(fired) → 3 (rescinded after probationary period) → 4 (current) Ginto Mungiane: transferred from nutrition program 1 month ago Marromeu / 1 (promoted to training officer) → 2 Fernando Lampeão: FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 52 Maringue (quit) → 3 (current) transferred from nutrition program 2 months ago Manga 1 (left to return to MOH) → 2 (from AIDS program, maternity leave and now returned) Miranda Luis (1.5 years) Phase 2 One newly recruited and in training, one recently transferred from nutrition program. One more yet to be recruited. Source: interviews with program staff There has been considerable turnover in Facilitators as well. The following table illustrates changes among field Facilitators: District Originally trained Changes Current at mid￾term (no more will be added) Original Facilitators remaining Manga 10 4 of 10 replaced, 2 recently left 8 4/10 Caia 5 1 replaced, 2 left 3 2/5 Chemba 5 2 replaced 5 3/5 Maringue 7 3 replaced 7 4/7 Marromeu 3 Unchanged 3 3/3 TOTAL 30 10 changed, 4 left 26 16/30 Source: interviews with program staff Reasons for changes among the Facilitators are many. Some were either not performing adequately and were fired and some were hired by other organizations. There was at least one example each of theft and abandonment of post. In contrast to the above, there has been almost no turnover among volunteers: LMs or C-IMCI LMs. This turnover in program staff has created serious challenges to continuity. It has also required a significant investment in retraining for Facilitators. Most of the Supervisors had previously worked in the nutrition program, which had a strategy similar to the CS project, so training was relatively rapid. Turnover of Facilitators is particularly problematic, as it is impossible to provide formal “makeup” C-IMCI training. The highly autonomous nature of their work and their critical responsibilities in the project make this problematic. Facilitators train LMs, supervise LMs (using checklists), collect all project routine data, and will also assume responsibility for Hearth. They work with only minimal supervision (one visit every two weeks in Manga, every month in the other areas). Their responsibilities are demanding, often requiring them to bicycle long distances every day to carry out their duties. Once they are fully trained, they are in great demand by other organizations in the Province, especially among relief NGOs along the river and in the urban area. Staff describe CSP salaries as higher than in the public sector but somewhat lower than similar positions in other NGOs in the area. Likewise, most of the CSP Supervisor positions demand both a high ability to function autonomously without continuous supervision, relocation to Districts with few amenities, and physically demanding daily travel by motorcycle that is uncomfortable and somewhat dangerous. Most of the departures of Project FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 53 Officers and Supervisors have been to accept higher-paying positions with other NGOs, or positions that are less personally demanding than those with FH. All Supervisors and Facilitators said that supervision and support were very good. When asked why they continue to work for FH, they cite job satisfaction, learning opportunities and the opportunity to work directly with communities (many Supervisors come from hospital backgrounds). They say things like “my CV has improved a lot”, or “the work is rewarding”. All stated that working relationships among colleagues among the team are good. The CS Project Manager and the Project Officers (training and M&E) are satisfied as support from the headquarters office is strong and easily accessible. Backstopping has been very good. When asked about the differences working for a Christian organization rather than a secular one, most welcomed the opportunity to put their faith in practice and to use it to reinforce what they teach in communities. Some stated that, they had expectations that people may “behave better” in a Christian organization, but they had not found this. In the communities, Facilitators do not work especially well with religious leaders, which surprised some of the staff. In general, personnel issues are good, though the turnover of Officers and Facilitators places a great strain on the project. That said, the FH CSP has achieved its goals in spite of the difficulties. If it is possible within the budget, FH may wish to consider offering incentives to keep staff in their positions, e.g. “annual bonuses” for Facilitators that remain in their positions for twelve full months (not paid partially if they leave even after eleven months). These types of arrangements are less costly than salary increases, and are more likely to encourage staff to remain in their positions. Likewise, offering benefits (like health insurance coverage) to lower level staff may be another less expensive mechanism to encourage them to stay. Other creative mechanisms such as lotteries or a mutual savings club to give away a motorcycle (or other highly valued item) to a Facilitator each quarter are other possible mechanisms that would not be expensive and would encourage them to stay longer with the project. The effect of the HIV epidemic on employee absenteeism must be taken into account in future projects. In African societies, extended families are very important, and therefore, each employee is enmeshed in a large network of extended family members. All organizations in high-prevalence HIV countries are experiencing high levels of absenteeism due to illness and family funerals. The CSP is attempting to reduce the impact of this problem by hiring a “floating” Supervisor who can take the place of those who are absent. Future projects must take this unexpected effect of the HIV epidemic into account when planning human resource needs. I.3.d. Financial management Issues relating to headquarters financial management and systems are discussed at length in the 2007 annual report. The current FH/Moz finance officer, who was transferred from Latin America, has been in her position in Mozambique for about one year. According to project managers, financial reporting has improved significantly in the past year, both with the implementation of the new computer system and the abilities of the new finance officer. Management staff are able to know how their budgets are executed and how much remains. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 54 The new system allows real-time tracking of the budget, as both headquarters and field can access the accounts in real-time. Field staff comment that there have been no significant problems with financial management, and that all payments and salaries have been timely. Local logistics are formidable, as there are no banks in any of the districts. All transactions and salaries are paid in cash, which requires project staff to regularly transport large amounts of cash to field offices. According to the field financial officer, project funds are about 40% expended, and about 77% of the third year budget has been expended. This is approximately where the project should be, given the slight delay in implementation of phase II, with its significant increase in the number of Facilitators (with salaries, transportation and per diem costs and field offices). One specific difficulty cited by field staff is the absence of any kind of small cash fund, which overly complicates financial management. For all expenses, checks are emitted which must be cashed in banks in Beira before cash can then be transported to sites in the districts to be paid. Headquarters may wish to discuss this detail with field staff to work out an acceptable alternative. 1.3.e. Logistics Logistics challenges have challenged the project from the outset. Most have been overcome, and implementation has proceeded according to schedule. Three issues have proven most troublesome. The first are somewhat inflexible procurement regulations that complicated and delayed the production of flip-charts. A solution was negotiated whereby a single large procurement was done locally (in Beira) for printing. The second problem identified by staff comes from a new internal FH rule that all procurement be preferentially done in Maputo. This is impractical in a country like Mozambique, with very poor transportation North-South infrastructure, and FH having few staff in Maputo. In many cases, both national and imported items are cheaper and more easily available in Beira than in Maputo due to the excellent port facilities in Beira. Again, this issue should be negotiated between CSP staff and the Country Director, and a reasonable solution found. The third logistics problem relates to transportation. Vehicles are aging, parts are difficult to find, and the logistics department appears to be overwhelmed. Staff complained of frequent logistics and transportation delays, and these were noted many times during the evaluation. Field management spent an inordinate amount of time working to overcome difficulties with logistics arrangements during the evaluation. Once again, a workable solution should be discussed with the Country Director. I.3.f. Information management The FH CSP has a formidable information system that has provided excellent data for monitoring of project implementation, quality, outputs and outcomes. Most of the instruments and how they were used have been discussed in the various preceding sections. The following information systems and strategies were used: Instrument Explanation (what, who, when, how) Example of how utilized Baseline census with retrospective mortality study Once by FH at the outset of the project. Used to determine the number of Care Groups and assign mothers of children 0-23 months to LMs and pregnant women. Retrospective mortality study used to determine the pattern of child deaths by age and symptoms. Baseline KPC survey using LQAS sampling Once at baseline in the 5 phase I districts. Will be repeated at final Used as baseline outcome and coverage indicators to set project targets and to assess progress. Mini-KPC surveys using LQAS sampling Once after each module is implemented. Three have been done to date. Measures many of the same indicators in the baseline KPC survey. Used to track progress toward reaching targets, and to identify underperformance in districts and on specific indicators for intensification of activities and encouragement. Barrier Analysis Survey carried out prior to finalizing BCC and training materials for interventions. Done once for hand washing with soap/ash and once for exclusive breastfeeding shortly before those modules were produced. Identified determinants that are positively and negatively associated with the adoption of desired behaviors. Used to improve the quality of hand washing and breastfeeding messages. Not washing hands was associated with lack of soap, and with lack of cues for action, so specific messages were designed for these. Likewise for EBF, where support from family and the belief that breastfeeding saves money, is approved of by God, and can be done easily were determinants associated with the behavior. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 56 Instrument Explanation (what, who, when, how) Example of how utilized Quality Improvement and Verification Checklists (QIVCs) Used for supervision of group training (Supervisors, Facilitators and LMs), LM home visits, C-IMCI by C-IMCI LMs (not successful), and by management staff to supervise Supervisors. They are also used during surveys (mini-KPCs and others) to monitor the quality of interviews and data recording. Used to improve quality of training and performance (see training section for example of improvement with successive supervision) as well as being compiled to evaluate training quality in the project overall, and performance of LMs and C-IMCI LMs overall (see the CSP objectives regarding LM and C-IMCI performance). Pre- and posttests for training Applied in almost every training by trainers, including verbal testing after LM training (though this has not been as successful). Standardized testing forms are used for BCC modules. Ad hoc instruments were developed for C-IMCI training and staff training. Utilized to determine the quality of training methods and trainers, whether refresher training is needed, and whether training materials and methods need modification. Local Determinants of Malnutrition Study (an expanded Positive Deviance study) Survey performed once by FH staff (prior to project start-up) to determine foods and behaviors that are associated with good and poor nutrition in children. Used to develop materials and messages for the breastfeeding and complementary feeding modules, water and sanitation messages, as well as for Hearth. Identified emptying both breasts, feeding 3+ times per day, water purification, for example. Designed nutritional food recipes that are specific for each district depending on locally available foods. This information was used to reinforce nutrition messages for Module 5. Anthropometry (weight for age and MUAC) Project staff conducted at baseline, February 2007, and scheduled for Sept 2008. Weight for age was measured during KPC and mini-KPC surveys. In WFA survey used to monitor baseline and interim nutritional status. MUAC + WFA will be used to screen the beneficiaries for referral to Hearth. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 57 Instrument Explanation (what, who, when, how) Example of how utilized 2008, Facilitators and C-IMCI LMs were trained in the use of MUAC. All beneficiary children will be screened with MUAC by C-IMCI LMs, with confirmation by Facilitators using WFA for those who are yellow or red on MUAC. Verbal autopsies Gathered by Facilitators from one of their Care Groups (selected at random at the beginning of the process) for all deaths in that Care Group each month using a standard form. Facilitators must visit the home of the mother and perform an interview two weeks after death. Used to identify morbidity and mortality patterns and associated factors, including delays in seeking care and reasons. Verbal autopsies were originally to be conducted on all deaths, but this proved too time-consuming. Now they are conducted in one Care Group per Facilitator. They have been used, for example, to determine that the principal delays in receiving care occur in the home. As this is only a sample, it cannot be used to track the mortality rate. Vital statistics Births and deaths are registered as part of routine data gathering. Facilitators take verbal reports from LMs during monthly meetings for this purpose. Cause and age of death is also estimated, classified as diarrhea, respiratory infection, fever or other. Birth is classified as institutional or home. This is used to track the number births and deaths in children by approximate cause. It can be used to roughly track trends as long as reporting remains approximately the same (there is significant underreporting---see discussion in the section on mortality above). Routine output monitoring This comes from data collected from LMs by Facilitators during monthly meetings with Care Groups. It is collated monthly by Supervisors and forwarded to This system is used to track the number of mothers visited each month in order to estimate ongoing coverage of the program. If coverage falls, project staff can investigate. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 58 Instrument Explanation (what, who, when, how) Example of how utilized the M&E officer. This system tracks many indicators, as discussed in the following section. The routine information system tracks a large number of output indicators as well as vital statistics. These include (list is not exhaustive): • Training and visits (number of mothers visited and number attending training sessions) • Number of supervision checklists completed and results (by type) • Mothers and children seen for C-IMCI (by diagnosis), number referred by diagnosis, number who actually sought care, and who were followed up afterward, number improved, number receiving ORS, home fluids • Doses of vitamin A, deparasitization and ORS given by type (campaign or during visit). Children evaluated but not receiving doses because they are up-to-date • Malnourished children seen • Pregnant women who went for prenatal care visits this month, post-partum vitamin A doses, births, stillbirths, institutional deliveries, A data collection session was observed during the MTE where FH Supervisor collected data from the Facilitators in the area. The process took most of a day. When questioned about how much time Facilitators spend collecting the data from the LMs in their five Care Groups, most responded that it takes them about ten days (in the urban area). In the rural area, they responded that it takes about half their time: two weeks every month gathering data, as they must interview each LM individually. In addition, there was significant confusion as to the meaning of some of the indicators being collected, for example “number of children evaluated for vitamin A but not given a dose because they were up-to-date”. Clearly, this will not be feasible once Hearth begins. Many indicators are being collected, but are not being utilized either because the information isn’t terribly useful, or there are doubts about the completeness or validity. The number indicators should be reduced to the minimum necessary to monitor the project implementation. Indicators that should continue to be monitored include: number of visits to beneficiary mothers, number of people trained and supervised (all levels), and vital statistics. As the project is piloting community-based distribution of vitamin A, deparasitization and ORS, the number of doses administered in the community should also be monitored. Indicators that can be eliminated include those relating to C-IMCI “consults” or assessments, all morbidity indicators, referral for services of any kind (antenatal care, GM, sick child, malnutrition, because the LM doesn’t always know whether the mother actually sought services or not), administration of any medicines or supplements in health facilities (e.g. number of pregnant women who received iron in a health facility), and any services provided by a health facility (e.g. number of children weighed in a health facility). Most of the morbidity and treatment-related indicators can be substituted by mini-KPCs for coverage, and by MOH HMIS data for treatment or service provision. This will eliminate at least half of the indicators being gathered. The CSP did not utilize MOH data to monitor project outputs or outcomes. However, as was noted above, some MOH indicators may be prove to be informative. Incorporating a review of these indicators during coordination meetings with the DPS and districts will both improve dialogue between the CSP and the DPS and districts as well as improving their ability to utilize data for decision-making. Finally, the project is collecting vast amounts of quantitative information, but little qualitative information. The utility of this was seen during the MTE when a relatively small number of semi-structured focus group discussions revealed a wealth of information that had not been previously identified. Recommendation Compliment the vast amount of quantitative information by regularly collecting some qualitative information. This would be especially helpful with Barrier Analysis and identification of factors causing poor coverage after LQAS mini-KPC surveys. I.3.g. Technical and administrative support FH headquarters has provided substantial technical support to the program since its outset, including strong involvement in program planning and design, revisions, and training and oversight for the introduction of new tools and strategies. The FH field team had strong technical support through the fact that Carolyn Wetzel, MPH/TM served as the Mozambique Health Programs Manager and worked closely with the CS Project Manager during the first two years of the project, with strong backstopping from HQ. This allowed for smooth introduction of mini-KPCs, LQA sampling and tabulation, verbal autopsies, and the HMIS. During the first year, the CS and Nutrition Programs Coordinator and the Director of Health Programs traveled twice to Mozambique to facilitate a start up workshop, train program staff on baseline KPC methodology and data entry (Epi Info), and to conduct a DIP Workshop. During the second year, they traveled to Mozambique twice again to conduct a training on Mini-KPC Surveys using LQAS, Verbal Autopsy, and Health Facility Assessments. The CS Project Manager and Mozambique Health Programs Manager were also able to travel to the US to attend the CS Mini-University. Finally, in year three, both Carolyn Wetzel and Tom Davis traveled to Mozambique, the former to provide training in Hearth, and the latter to assist with the analysis of the verbal autopsy and HFA results, and to participate in the MTE. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 60 At a distance, HQ technical staff provided ongoing input into the design of survey forms and questions, analysis, forms for Barrier Analysis, positive deviance studies, training curricula, verbal autopsy forms, quality improvement checklists and the routine information system. Regular use of telephone and Skype were cited as strong points by the field CS management team. One said, “I feel like I can call them any time to clarify any point I need to.” It is noteworthy that it has not been necessary to contract external consultants. All technical assistance has been provided in-house by FH staff. No unmet technical assistance needs were identified by field staff during the evaluation. The fact that Carolyn Wetzel began as the Mozambique Health Programs Manager and then moved to headquarters to assume the CS and Nutrition Programs Coordinator position has led to a strong link between headquarters and the field. It is also helpful that Carolyn is fluent in Portuguese. The FH headquarters staff is not large, and staff take a personal interest in their field projects. I.3.h. Management lessons learned (1) The most important lesson learned is to plan for staff turnover, especially in settings with high HIV prevalence as well as in areas with severe human resource shortages. (2) Constant communication between staff and regular systematic supportive supervision are also key to achieving excellent staff performance. (3) The fact that the CS Project Manager has extended her contract beyond the initial 2 year commitment has been a key factor in the projects success, in addition to her ability to communicate well with the staff and willingness to manage all aspects of the program, including logistical and procurement problems that have plagued the project. FH has frequently prioritized English over Portuguese in their hiring of project managers, but fluency in Portuguese and ability to adapt to Mozambican culture has been critical to the successful management of the CS project. The skills the CS Project Manger brings (e.g., experience working with MOH, managing programs in isolated areas where resources and education are limited, staff development, and flexibility) should be prioritized in the hiring of additional management staff in Mozambique. (4) The project did not initially plan to hire a coordinator or trainer to assist the Project Manager. It was later found that because of the repeated training needs of field staff, high levels of supervision required, high staff turnover, and time required to deal with financial, logistical, and procurement issues that these positions were necessary to allow the Project Manager time to focus on the higher level objectives of the project (such as the sustainability and expansion of Care Groups). Other management lessons learned have been thoroughly discussed in each relevant section. I.3.i. Other issues Whereas the overall project design is quite good, the fact that there was only a minimal “MOH strengthening” component has caused difficulties for the project. There is an expectation by the MOH, USAID and other important development partners that all NGOs that work in the health sector will provide logistical and human resources support to the MOH for all national initiatives. This is especially true for national campaigns, FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 61 such as national immunization days. These can be costly and time-consuming, and there was no provision for these activities in the project workplan and budget. Even when achieving major results in behavior and decreases in malnutrition, refusal to participate is simply not an option, as non-cooperating organizations are seen as “not supporting national priorities”, which can have negative consequences when the organization asks for MOH cooperation or collaboration in project activities. In the future, FH should take this into account in its planning. During the planning phase, FH was reassured that the DPS would provide training in C￾IMCI and that the DPS would participate in joint supervision with the CSP. However, the CSP design and budget did not take into account the need to subsidize these activities, including the need to pay training allowances, per diems and transportation in order for the DPS to participate. Although the DPS had implied that this would not be necessary, if these activities are critical to the success of the project, the cost must be provided for in the budget. This lesson should be incorporated into future project planning. 4. M&E Table Targets in bold are those that have been met or exceeded. Baseline Mini KPC Mini KPC Mini Objective Indicator KPC Feb 2006 May 2006 Sep 2007 Dec 2007 EOP Target Comments 1. To decrease malnutrition (underweight) in children 0-23m Percentage of children age 0–23 months who are underweight (WAZ<-2.0) 27.1% (weighted) NM 18.6% (weighted) NM 18% This is somewhat different that the 2007 annual report (baseline: 26%, 2007: 21%). Those represented preliminary unweighted estimates. The values here are definitive. 2. To increase exclusive breastfeeding of children 0-5m Percentage of infants aged 0- 5 months who were fed breastmilk only in the last 24 hours 17% 67% 75% 95% 60% Questions differed NOTE: I believe there is a copying error in all of the report summary tables, which say “92%”. 92% is the next indicator down in the table (BF both breasts) 3. To increase feeding frequency of children 9- 23m who are fed solid or semi-solids food at least three times a day Percentage of children 9- 23m who receive food other than breastmilk at least three times per day [Nationally accepted indicator (Title II)] 33% 99% 65% 67% 65% 4. To increase the proportion of young children fed nutrient-dense foods Percentage of children 6-23 months of age with oil added to their weaning food [Nationally accepted indicator] 35% 76% 87% 84% 80% 5. To decrease VAD by increasing the proportion of young children who regularly consume vitamin A rich foods. Percentage of children 6- 23m who have consumed at least one vitamin A rich food in the previous day 29% 83% 87% 95% 80% Questions differed 6. To decrease VAD by Percentage of children 12-23 82% 77% 89% 91% 95% Questions differed: Baseline FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 63 Baseline Mini KPC Mini KPC Mini Objective Indicator KPC Feb 2006 May 2006 Sep 2007 Dec 2007 EOP Comments Target increasing the proportion of young children in Sofala who are regularly receiving vitamin A supplements months of age who have received one vitamin A capsule in the past six months asked mother and showed capsule. Mini-KPC is card￾verified only. 7. To decrease helminthiasis and improve nutritional status by increasing the % of young children who are regularly dewormed Percentage of children 12-23 months who received deworming medication in the last six months 24% 68% 83% 86% 75% Questions differed— baseline asked mother. Mini-KPC is card-verified only.. 8. To increase the proportion of children 0- 23m of age who participate regularly in growth monitoring/promotion activities Percentage of children aged 0-23 months who were weighed in the last four months (card-confirmed) 70% 86% 89% 94% 90% Mini-KPC is 6-23 months 9. To increase the proportion of young children with diarrhea who are given ORT in order to decrease dehydration and death Percentage of children aged 0-23 months with diarrhea in the last two weeks who received oral rehydration solution (ORS) and/or recommended home fluids (RHF) 71% 83% 96% 78% 90% Achieved the target, and then later, it fell below target. Subsequent surveys will verify the results. Questions differed--- baseline: among those with diarrhea last 2 weeks, mini￾KPC last time had diarrhea. 10. To increase feeding of young children during diarrhea Percent of children aged 0- 23 months with diarrhea in the last two weeks who were offered the same amount or more food during the illness 31% NM 70% NM 60% Questions differed--- baseline: among those with diarrhea last 2 weeks, mini￾KPC last time had diarrhea. 11. To increase the proportion of mothers of young children who are competent in preparation Percentage of mothers of children 0-23m who can correctly prepare ORS 44% 79% 91% NM 80% FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 64 Baseline Mini KPC Mini KPC Mini Objective Indicator KPC Feb 2006 May 2006 Sep 2007 Dec 2007 EOP Comments Target of ORS 12. To increase the proportion of mothers of young children who know when to seek care for sick children Percentage of mothers of children age 0–23 months who know at least three signs of childhood illness that indicate the need for treatment 29% 94% 86% 85% 75% 13. To increase the proportion of mothers who receive a vitamin A dose during the first two months after delivery Percentage of mothers who receive a vitamin A dose during the first two months after delivery NM 80% 14. To increase the proportion of mothers able to report at least two known maternal danger signs during the postpartum period Percentage of mothers able to report at least two known maternal danger signs during the postpartum period NM 80% 55% 72% 80% In mini-KPC mothers of children 6-23 months. 15. To increase the proportion of mothers receiving post partum iron supplements Percentage of women who received iron during the post partum period following the birth of their youngest child NM 70% 16. Continue to expand usage and improve the Care Group model in Mozambique The MOH in at least one other Mozambican province requests assistance (during the life of the program) from FH to expand the Care Group model into their geographical area. OR is conducted on the reasons for Care Group effectiveness. N/A None requested OR not yet done None established 17. To increase to 80% the proportion of LMs trained % of Leader Mothers trained in community-IMCI N/A Mean score 79% 80% Mean posttest score of LMs after training was 79% FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 65 Baseline Mini KPC Mini KPC Mini Objective Indicator KPC Feb 2006 May 2006 Sep 2007 Dec 2007 EOP Comments Target in IMCI who can properly use the IMCI protocols for children 2-59m of age modules who score 80% or higher on an IMCI QI checklist. (approx 50% scored 80% or higher). Supervision checklists were not successfully introduced due to their complexity and difficulty in observing LMs performing IMCI. 18. To increase to 80% the proportion of LMs who are able to do high-quality health promotion % of Leader Mothers who score 80% or higher on the Health Promotion checklist (QIVC). Mean score on QIVC grp. 91%, indiv. 84% 80% of LMs can provide quality health promotion Measured mean score rather than % scoring above 80%. Will report on % scoring above 80% for remainder of project. 19. Increase the capacity of local partners and 90% of project communities to effectively address local health needs. % of first-phase Care Groups that continue to meet and do health promotion following reduction of health Facilitator staff in Year 2.5. N/A N/A N/A N/A N/A This will only become apparent during the second half of the project. Other indicative indicators being measured but not included in the DIP Nutrition % of mothers of children 0-5 months who are breastfeeding who report breastfeeding from both breasts NM 98% 92% 100% N/A Denominator is those who are breastfeeding % of mothers of children 0-5 months who are breastfeeding who empty both breasts NM 62% 59% 99% N/A Denominator is those who are breastfeeding % of children 0-5m with a growth monitoring card 84% Diarrhea and hygiene FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 66 Baseline Mini KPC Mini KPC Mini Objective Indicator KPC Feb 2006 May 2006 Sep 2007 Dec 2007 EOP Target Comments % mothers of children 6-23 months who report purifying water for the child NM 83% 94% 83% N/A Includes boiling or using chlorine product % mothers of children 6-23 months who report the child defecated in a proper place NM 71% 79% 78% N/A % mothers of children 6-23 months who report having a hand-washing station NM 84% 79% NM N/A % mothers of children 6-23 months who report having a hand-washing station with soap/water or ash NM 99% NM NM N/A % mothers of children 6-23 months who report handwashing at proper times NM 52% 79% 98% N/A Before preparing food, feeding children; after defecation or cleaning child who defecated % mothers of children 6-23 months who report covering food after preparation NM 98% NM NM N/A % mothers of children 6-23 months who report covering food after preparation Other indicators % of mothers of children 6- 23 months visited by a LM in the previous 2 weeks 93% 94% 95% FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 67 5. Rapid CATCH table No specific mid-term Rapid CATCH survey was performed. Some Rapid CATCH indicators appear in the M&E table in the previous section. 6. Mid-term KPC report As the CS project is performing regular mini-KPC surveys, no special mid-term KPC report was done. Results cited in this report refer to the most recent mini-KPC, December 2007, unless otherwise specified. The narrative report for the December 2007 mini-KPC are found below. Results of Round#4/5 of Mini-KPC for the FH/Mozambique Expanded Impact Child Survival Project Cooperative Agreement N◦ GHS-A-00-05-0014-00 Updated January 2008 Water and Hygiene Behaviors (covered in Module #4/5) a. Water purification: 85% average coverage, above project target of 75%. Chemba is below both average coverage and project target. Conclusion: Surpassed target, however more work needed in Chemba to improve the results. b. Defecated proper place: 80% average coverage, surpassed project target of 70%. All districts above both average coverage and project target. As it was in previous round, Marromeu this time did particularly well. Conclusion: Overall target surpassed, maintain performance. c. Hand washing proper times: 98% average coverage, surpassed project target of 40%. No districts below average coverage or project target. Manga and Caia were below coverage in round before but achieved coverage this time. Conclusion: Surpassed target, so continues teachings related to hygienic behaviors. d. No diarrhea last two weeks: 70% average coverage, equal project target of 70% without diarrhea. All districts above both Average coverage and Project target. The indicator was improved this time; project target was not reached in the previous round but was reached this time. Chemba was below target and coverage (previous results) but this time is above. Caia was below target and this time is below. Conclusion: On project target, achieved overall target but more work is necessary to surpass overall project target. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 68 Overall Results, Water and Hygiene Behaviors: Good results on almost all water and hygiene indicators used in this survey. Project targets were met or surpassed for all indicators measured. Although the indicator for no diarrhea last two weeks is above target for all districts combined, efforts should be devoted in order to overtake the project target in all districts (e.g., including Chemba and Caia). Possible reasons for this in Caia and Chemba should be explored. Coverage Indicators: a. Infants visited by LM: 95% average coverage, surpassed project target of 90%. No districts below average coverage or below project target. Conclusion: Surpassed overall project target, and maintain the service. b. Children 6-23m, visited by LM: 95% Average coverage, above project target of 90%. All districts above target and coverage. Conclusion: Above project target; continue stepping up the exercise. c. Weighed at least once during last 4 months: 95% average coverage, above project target of 90%. All districts above coverage and target. Previous round it was on project target but this time is above. Marromeu was below target and coverage when compared to previous results but this time is above. Conclusion: Above project target, continue to work to maintain the indicator performance. d. Child dewormed: 90% average coverage, equal project target of 90%. Above coverage and target in Marromeu. Maringue, Chemba are on target and coverage. Below coverage and target in Manga and Caia. Conclusion: On project target. Overall project target achieved but more work is needed, particularly in Manga and Caia to achieve expected performance on both, target and coverage. e. Vitamin A supplementation: 95% average coverage, equal project target of 95%. Caia is below target and average this time while in round before was above. Marromeu this time is above target and coverage while previous round was below. Conclusion: On project target. More work need to be done especially in Caia. Overall Results, coverage indicators: Good results, achieving or surpassing project targets for all coverage indicators. Targets and equity achieved in almost all districts. However Vitamin A supplementation and deworming was expected to be better in all districts. Explore reasons for below-target performance in Manga (deworming) and Caia (Vit A and deworming). Strengthen FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 69 campaigns on Vit. A supplements. Strengthen teaching related to importance of deworming in all districts, but particularly in Caia. More work is needed in Manga to achieve expected coverage and target (deworming). Overall project target achieved but efforts should be devoted to surpass the overall project targets and to assure equity. Other Knowledge and Behaviors Measured a. Exclusive BF: 95%, surpassed project target of 60%. No districts below project target or average coverage. b. BF both breasts: 99% average coverage, surpassed project target of 75%. All districts above project target and good equity. Manga was below average coverage in comparison to other districts in last round, but now at or above average coverage. c. Completely empties both breasts: 99% average coverage, surpassed project target of 80%. Very good achievements and surprising results. All districts above project target and good equity. Indicator was well improved (see previous results). Overall project target was not achieved last time. Manga and Maringué were below target and coverage earlier, but this time they have improved. Caia was below target and now is above. d. Consumption of vitamin A foods: 95% average coverage, surpassed project target of 80%. No districts below average coverage or project target. Maringue was below average coverage before, but this time coverage was achieved. e. Child ate solid or semi-solid foods: 70% average coverage, below project target of 95%. Project target has not been met. (In previous round, the overall project target was not achieved as well). Only Marromeu is above both average coverage and project target. Caia and Maringue are still below coverage and target when compared to previous results. Manga and Chemba are above coverage but still below project target. f. Mother adds oil to child's food: 85% average coverage, above project target of 80%. Overall project target achieved but Marromeu is still the one below target and average. g. Gave ORT to prevent dehydration: 80% average coverage, below project target of 90%. Indicator had very good performance in Round#2 (see previous results) but strangely, failed this time. Marromeu is the one below target and average coverage. Marromeu was weaker in performance which significantly affected the overall result. Maringue did very well. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 70 h. Mother knows 3 child danger signs: 85% average coverage, surpassed project target of 75%. Only Chemba is below average coverage. Caia this time is above average coverage while last round was below. i. Mother knows 2 maternal danger signs: 75% average coverage, below project target of 80%. All districts except Chemba are above average coverage and project target. In the previous Round, Marromeu and Chemba were below coverage and target. Marromeu this time is above target and coverage while Chemba is still below. Overall project target was not achieved but even so, indicator is improved this time when compared to previous results. j. Know proper Breastfeeding when HIV+: 95% average coverage, surpassed project target of 80%. No districts below project target and good equity. Overall Results, other indicators: Surpassed the overall project target in more than half of indicators measured (7 of 10) but some indicators still need particular attention. The indicator, “Child ate solid or semi-solid foods 3+ times” has been failing for two consecutive rounds. Unsuccessful districts (see summary table below) should explore reasons why change is not happening. Efforts need to be devoted to surpass project targets. Contrary to the previous round results, the “gave ORT to prevent dehydration” indicator failed this time. Results were expected to be different, since mothers had already got knowledge on it, moreover previous results were shown to be very good. Marromeu in particular was weak in performance this time affecting significantly the overall results. The main reasons for failure in Marromeu need to be explored and an action plan to change may be developed. Mother knows 2+ maternal danger signs indicator, also failed in two consecutive rounds, but teaching on this is not scheduled until the next module. Marromeu has improved performance this time, while Chemba has kept as before (see previous results). More work need to be done in Chemba to boost results. Other Beliefs measured a. Belief that child deaths will end: 90% average coverage, above project target of 80%. All districts above average coverage and project target b. Belief that women are as valuable as men: 55% average coverage, below project target of 80%. Marromeu and Maringue above average coverage and project target while Caia and Chemba is below. Manga is not below average coverage, but is below target. c. Believes immediate BF is best: 90% average coverage, surpassed project target of 80%. Only Marromeu is below both, coverage and target. FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 71 d. Believes ok to BF if pregnant: 75% average coverage, slightly below project target of 80%. Marromeu is the one below average coverage and project target. Overall Results, other Beliefs measured:: Good results in half (2 of 4) of indicators measured. We have not achieved the overall project target on Belief that women are as valuable as men indicator, but this indicator is new, and we have not done much teaching on this. More work is needed to change the people’s minds related to the value of women. Some of these indicators were added to survey this time therefore results cannot be compared yet. Main Focus Areas by District (Below Project Target) for Project Indicators District Project Indicator Manga • Child ate 3+ solid/semi-solid foods in last 24h • Deworming Caia • Deworming • Vitamin A supplements • Belief that women are as valuable as men • Child ate 3+ solid/semi-solid foods in last 24h Marromeu • Believes immediate BF is best • Believes okay to BF if pregnant • Ads oil to food • Gave ORT to prevent dehydration Chemba • Water purification • Belief that women are as valuable as men • Child ate 3+ solid/semi-solid foods in last 24h • Mother knows 2 + maternal danger signs Marringué • Child ate 3+ solid/semi-solid foods in last 24h Cecilia Lopes/ FH Moz - CS/M&E FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 72 Mini-KPC Results Feb-06 May-06 Sep-07 Dec-07 # Project Indicators Percenta ge CI Percentage CI Percentage CI (%) CI EOP Target EOP Target Already Achieved (Phase I Communities) Percent Change since baseline Sept 07 Percent Change since baseline Dec 07 1 % of children 0-23 months who are underweight (WAZ<-2.0) 26% 22.6- 30.1% NM 21% 16.6- 24.7% NA 18% x 19.2% improvemen t (decrease) NM 2 Percentage of infants aged 0-5 months who were fed breastmilk only in the last 24 hours 17% 8.2- 30.3% 67% 57.9- 76.8% 75% 66.0- 83.5% 95% 92.2- 99.2% 60% a 412% increase 457% increase 3 Percentage of children 9-23m who receive food other than liquids at least three times per day 33% 24.4- 41.6% 99% 96.8- 101.0% 65% 55.7- 74.8% 67% 57.5- 76.5% 65% a 97.0% increase 103% increase 4 Percentage of children 6-23 months of age with oil added to their weaning food [1] 35% 27- 43% 76% 66.8- 84.2% 87% 80.7- 94.0% 84% 76.6- 91.4% 80% a 149% increase 140% increase (decreas e since last KPC) 5 Percentage of children 12-23 months of age who have received one Vitamin A capsule in the past six months (card or mother’s report)[2] 82% 73.3- 89.1% 77% 67.7- 86.0% 89% 82.0- 95.3% 91% 85.8- 97.1% 95% x 8.5% increase 11.6% increase 6 Percentage of children 6-23m who have consumed at least one Vitamin A rich food in the previous day 29% 21.4- 36.6% 83% 74.9- 90.4% 87% 80.7- 94.0% 95% 90.2- 99.2% 80% a 200% increase 227% increase FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 73 Mini-KPC Results Feb-06 May-06 Sep-07 Dec-07 # Project Indicators Percenta ge CI Percentage CI Percentage CI (%) CI EOP Target EOP Target Already Achieved (Phase I Communities) Percent Change since baseline Sept 07 Percent Change since baseline Dec 07 7 Percentage of children 12-23 months who received deworming medication in the last six months 24% 16.2- 33.9% 68% 57.4- 78.6 83% 73.2- 91.9% 86% 78.1- 94.1% 75% a 246% increase 259% increase 8 Percentage of children aged 0- 23 months who were weighed in the last four months (card￾confirmed) 70% 63- 77% 86% 78.3- 92.8% 89% 82.8- 95.5% 94% 88.6- 98.5% 90% x 27.1% increase 34% increase 9 Percentage of children aged 0- 23 months with diarrhea in the last two weeks who received oral rehydration solution (ORS) and/or recommended home fluids (RHF)[3] 71% 60- 81%- 83% 74.9- 90.4% 96% 91.7- 99.8% 78% 69.2- 86.4% 90% a 35.2% increase 9.5% increase (decreas e since last KPC) 10 Percent of children aged 0-23 months with diarrhea in the last two weeks who were offered the same amount or more food during the illness 31% 21- 43% NM 70% 60.2- 79.0% NM 60% a 125% increase NM 11 Percentage of mothers of children 0-23m who can correctly prepare ORS 44% 37.2- 51.4% 79% 70.7- 87.1% 91% 84.6- 96.4% NM 80% a 107% increase NM 12 Percentage of mothers of children age 0–23 months who know at least three signs of childhood illness that indicate the need for treatment 29% 22.8- 35.5% 94% 88.8- 98.6% 86% 79.4- 93.2% 85% 78.1- 92.4% 75% a 197% increase 194% increase (decreas e since last KPC) FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 74 7. Team members and their roles Name Position Emma Hernandez CSP Project Manager Cecília Lopes CSP M&E Technical Officer Luciano Menete CSP Training Officer Ginto Munguiane CSP Project Officer (Supervisor) Fernando Lampeão CSP Project Officer (Supervisor) António Fache CSP Project Officer (Supervisor) Miranda Luis CSP Project Officer (Supervisor) Amélia Azevedo CSP Project Officer (Supervisor) Joaquim Ernesto CSP Project Officer (Supervisor) Justice Hudon FH Project coordinator, Zinc Operations Research Coordinator (candidate) João Guio Head of the Nutrition, City of Beira Health Department Carolyn Wetzel CS and Health Programs Coordinator, Food for the Hungry, Int. Tom Davis Dir. of Health Programs, Food for the Hungry Donald Whitson Mid-term Evaluation Consultant / facilitator 8. Assessment methodology The assessment was done during a period of ten days in the field from July 28-August 6, 2008. The evaluator had reviewed key project documents in advance, including the DIP, annual reports, training curricula and the reports from the baseline KPC, mini-KPCs, verbal autopsies, Barrier Analyses, etc. In addition, a teleconference between Tom Davis and the evaluator was done to orient the evaluator to background issues and the approach to the evaluation that was desired. The first day was taken up with a team meeting for background presentations, review, finalization of the evaluation work plan, finalization and reproduction of the focus group and key informant interview instruments, and logistics planning. The team was divided into two evaluation teams. Day two, both teams traveled to Manga and performed focus group discussions and key informant interviews. Both teams traveled to Gorongosa, and at the end of the day, the group met to discuss results, modify the questions and clarify issues. On day three, both teams proceeded to Maringue and performed a series of interviews and focus groups. Again, both teams met in the evening to share and review results. Day four continued in Maringue, and one team proceeded to Caia while team one finished in Maringue and then proceeded to Caia to begin interviews. Day five, team one traveled to Marromeu and completed the interviews there, returning to Caia. Team two proceeded to Chemba and returned to Caia. The two teams finished Caia day six in the morning FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 75 (Saturday) and returned to Beira. Day seven (Sunday) was a day to rest, review results, and prepare for the group sessions. Tom Davis arrived and joined the team. On Monday (August 4, day eight) further interviews were done in Manga, while several members of the team went to the DPS to collect data from pre-arranged indicators. Tuesday and Wednesday, August 5-6, the team met together in the office. Three groups were formed. Discussion guides were distributed corresponding to the principal sections of the evaluation report. The teams were instructed to present quantitative and qualitative evidence and draw conclusions. The groups then met in plenary each day to present their findings, conclusions and recommendations. Finally, on Thursday morning, August 7, key informant interviews were held with the DPS (nurse in charge of maternal and child health), as well as FH finance, and logistics. The number and distribution of focus groups and key informant interviews performed is summarized in the following table: Focus groups District Manga Maringue Marromeu Caia Chemba Total Administration 1 1 1 3 Community Leaders (CDC) 5 1 1 1 8 C-IMCI LM 5 1 2 2 10 LMs 2 4 2 4 1 13 Mother beneficiaries 3 4 3 3 1 14 Health workers 2 2 2 1 7 9. List of persons interviewed and contacted during the MTE This list does not include the names of health facility workers, FH staff, CDC members or LMs interviewed. It also excludes members of the evaluation team listed above. District Name Position Marromeu Enriques Bonguesse Administrator Ricardo Molinho District Director of Health (DDS) Maringue Absalao Chabela District Administrator Lopes Mangate DDS Caia Jose Coel Antonio District Administrator Ana Paula Matiquite DDS Chemba Antonio Januario District Administrator Luciano Carismo Estupe DDS Manga (Beira Cidade) Joao Manuel Meque Community Secretary Graciana de Pita DDS Sofala Marina Karagianis Provincal Director of Health (DPS) FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 76 10. Child Survival and Health Grants Program Project Summary Oct-15-2007 Food For The Hungry, International (Mozambique) General Project Information: Cooperative Agreement Number: Project Grant Cycle: Project Dates: Project Type: GHS-A-00-05-00014 21 (9/30/2005 - 9/30/2010) Expanded Impact FH Headquarters Technical Backstop: Carolyn Wetzel Field Program Manager: Emma Hernandez Midterm Evaluator: Donald Whitson Final Evaluator: USAID Mission Contact: Jeri Dible Field Program Manager Information: Name: Emma Hernandez Address: Rue Martires de Massangano, No. 557 Beira Phone: 258 23 320 595 Fax: 258 23 320 587 E-mail: ehernandez@fhi.net Alternate Field Contact: Name: Israel Keys Address: Rue Martires de Massangano, No. 557 Beira Phone: 258 23 320 595 Fax: 258 23 320 587 E-mail: ikeys@fhi.net Funding Information: USAID Funding:(US $): $2,699,910 PVO match:(US $) $847,653 FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 77 11. Revised Beneficiary Table, Staffing Plan, and Activity Table Beneficiaries during the Life of the Project (revised, approved plan) Sofala Province Children 0-59m Children 0-23m Children 24-59m Pregnant Women Mothers of Children 0-23m District: District Population targeted by CS Program 16.80% 6.72% 10.08% 2.00% 5.38% Years of Program Operation Caia 50% 56,202 9,442 3,777 5,665 1,124 3,021 2005-2010 Chemba 100% 48,154 8,090 3,236 4,854 963 2,588 2005-2010 Maringue 100% 73,498 12,348 4,939 7,409 1,470 3,951 2005-2010 Marromeu 34% 36,105 6,066 2,426 3,640 722 1,941 2005-2008 Beira (Portion) 120,500 20,244 8,098 12,146 2,410 6,477 2005-2010 Dondo 80% 179,094 22,825 9,130 13,695 2,517 7,303 2008-2010 Gorongosa 80% 95,478 12,169 4,868 7,301 1,342 3,894 2008-2010 Nhamatanda 80% 203,843 25,980 10,392 15,588 2,864 8,312 2008-2010 Chibabava 71,393 0 0 0 0 0 none Buzi 191,790 0 0 0 0 0 none TOTAL 1,076,057 117,164 46,866 70,298 13,412 37,487 2005-2010 Target Population is exploited from the 1997 census 2009-2010 Staffing Plan Sofala Province District: Officials Facilitators Mother Leaders Mother Beneficiaries Care Groups Caia 50% 5 348 4200 25 Chemba 100% 5 350 4200 25 Maringue 100% 1 7 504 6029 35 Beira (Portion of pop. only) 1 8 700 8398 50 Total Phase I Districts 2 25 1902 22827 135 Dondo 80% 1 12 840 10080 60 Gorongosa 80% 1 12 840 10080 60 Nhamatanda 80% 1 12 840 10080 60 Total Phase II Districts 3 36 2520 30240 180 TOTAL 5 61 4422 53067 315 Marromeu District closed in Aug of 2008 CS program operations in Marromeu were unable to continue as of August of 2008 because Mother Leader’s refused to continue as volunteers for the CS program in this district. Another FH program that operated in some CS communities, was mandated by policies of the government of Mozambique to pay Home-Based Care volunteers subsidies. Mother Leaders, with encouragement from the Marromeu District FH/Mozambique Expanded Impact Child Survival: Mid-term evaluation Page 78 Government, demanded to receive subsidies of equal value to continue their work. As this was not possible, nor sustainable and taking into consideration the recommendation of the MTE consultant to reduce Phase I expenses during the final years of the project by limiting work in Marromeu, where very few of the project beneficiaries resided, project resources and staff were withdrawn from Marromeu. Phase I Changes during 2009 and 2010 The CS project was designed to operate in two phases. The first phase, included the districts of Caia, Chemba, Maringue, Marromeu, and barrios in Manga, a region of Beira District. After August of 2008, Phase I activities were reduced in the Phase I districts (with the exception of Marromeu, which closed completely) and Phase II activities were started in the districts of Dondo, Gorongosa, and Nhamatanda. Activity Plans Phase I Phase II Location Beira (Manga), Caia, Chemba, Maringue, Marromeu Beira (Manga), Caia, Chemba, Maringue Dondo, Nhamatanda, Gorongosa Dates District Activities from October 2005 to August 2008. District Activities from September 2008 - September 2010. District Activities from September 2008 - September 2010. 1 Care Group Education Modules I￾VII Care Education Review and one new module each year Care Group Education Modules I-VII 2 Care Group supervision and teaching every 2 weeks by Facilitators Care Group supervision and teaching every 4 weeks by Facilitators Care Group supervision and teaching every 2 weeks by Facilitators 3 C-IMCI training of CHW C-IMCI training of CHW 4 C-IMCI provision of services C-IMCI provision of services C-IMCI provision of services 5 Vitamin A & Mebendazole Campaigns with MOH 6 Vitamin A, Mebendazole, and SRO provision via Care Groups and C￾IMCI Mother Leaders Vitamin A, Mebendazole, and SRO provision via Care Groups and C￾IMCI Mother Leaders Vitamin A, Mebendazole, and SRO provision via Care Groups and C-IMCI Mother Leaders 7 Mini-KPC studies every 6 mths Mini-KPC Studies 1X year Mini-KPC studies every 6 mths 8 Barrier Analysis to develop Education Modules Barrier Analysis as needed for BCC development 9 Verbal Autopsy Tracking Verbal Autopsy Tracking 10 Mortality Tracking Mortality Tracking Mortality Tracking 11 Health Facility Assessments Health Facility Assessments 12 Hearth nutritional rehabilitation Hearth nutritional rehabilitation 13 Zinc OR Start-up & Coordination Zinc OR in Manga & Caia only 14 Monthly District Staff Meetings Monthly District Staff Meetings Monthly District Staff Meetings 15 Quarterly Provincial Coordination Meetings Quarterly Provincial Coordination Meetings Quarterly Provincial Coordination Meetings