1 Mid-Term Evaluation Report Kibungo Child Survival Project Implemented by International Rescue Committee (IRC) in partnership with The Ministry of Health, Kibungo Province, Rwanda Submitted by: Bonnie L. Kittle October 2003 i Acronyms BLD Baseline Data CBNP Community based nutrition program CHW Community Health Workers DIP Detailed Implementation Plan DHT District Health Team (or DMT = District Medical Team) DHO District Health Officer (Office) IR Intermediate Results IRC International Rescue Committee KPC Knowledge, Practices and Coverage (survey) LQAS Lot Quality Assurance Survey NGO Non-governmental Organization NPC National Program Coordinator MTE Mid-term Evaluation PA Program Advisor PLA Participatory Learning and Action PNLP National Malaria Control Program PVO Private Voluntary Organization TBA Traditional Birth Attendant ii Table of Contents Acronyms i Chapter One – Summary 3 Chapter Two - Assessment of Progress toward Objectives 6 A. Results Chart 6 B. Technical Approach 8 - Overview - Progress by Intervention C. New Tools, Operations Research 23 D. Cross-Cutting Approaches 23 1. Community Mobilization 23 2. Communication for Behavior Change 25 3. Capacity Building Approaches 26 (i) PVO 26 (ii) Local Partner 27 (iii) Health Facility Strengthening 29 (iv) Health Worker Performance 30 (v) Training 4. Sustainability 33 Chapter Three - Program Management 34 A. Planning 34 B. Staff Training 38 C. Supervision of Staff 40 D. Human Resources and Staff Management 43 E. Financial Management 43 F. Logistics 43 G. Information Management 43 H. Technical and Administrative Support 43 Chapter Four - Principle Conclusions and Recommendations 46 Attachments 1. Baseline Information from the DIP 2. Evaluation Team Members 3. Evaluation Methodology 4. List of People Interviewed 5. Example CBNP Checklist 6. Example – Individual Work Plan 3 Chapter One - Summary ______________________________________________________________________ Brief Description of the Project The Kibungo Child Survival Project is being implemented by the International Rescue Committee (IRC) in partnership with the Ministry of Health (MOH) in the eastern Province of Rwanda. This four-year project began in October 2001 following a two-year Entry Level grant (1999-2001). The project seeks to reduce maternal and child morbidity and mortality and is focusing on three intervention areas: Nutrition, Maternal and Newborn Health and Malaria control. The specific objectives and indicators can be found in the Results Table in Chapter One. Main Accomplishments 1. Increased and improved capacity of more than 1,780 Community Health Workers (CHW – traditional birth attendants, nutrition animators and health animators) by providing training in collaboration with the Ministry of Health. CHWs are now able to refer pregnant women to the health center for prenatal consultations, identify at risk signs during pregnancy, delivery and the post-partum period, weigh children, and collect demographic data. 2. Strengthened links between the CHW and the Health Center Staff by organizing monthly meetings at the health center and through participation of TBAs at monthly pre-natal consultations. Improved knowledge of CHWs among health center staff and consequently a better appreciation of the roles each provider plays. These linked have also contributed to increase use of health center services, particularly prenatal consultations and obstetrical services. 3. Increased access of the rural population in Kibungo Province to nutrition services by organizing 114 Community Based Nutrition Program sites in 4 districts 4. Development of a community-based data collection system which enables, both the project and the MOH to follow the work of the CHWs and to collect such demographic data as births, deaths and causes of deaths. 5. In two districts, Kibungo and Rwamagana, the percentage of women who gave birth in a health center increased from 16% at the beginning of the project to 25% and 36.2% respectively, according to the LQAS survey conducted in July 2003. 6. Sustainability planning for the activities initiated/supported by the project by organizing the CHWs into associations. These associations have income generating activities that motivate CHWs to meet regularly at the health center to which they are affiliated. Among the 25 TBA associations all of them make and sell their own birthing kits as one of their income generation activities. Thirteen of these 4 associations are self-sufficient; buying their own kit materials. All of the associations have bank accounts in which they have deposited their profits. 7. Fifty-two percent (52%) of women giving birth at home have purchased and used home birthing kits, which are sold by project-trained TBAs. This practice not only improves the quality of the home birth, but is also an important practice in preventing the potential spread of HIV/AIDS. The Overall Progress Toward Objectives The areas where the project has seen the most progress include: increased attendance at pre-natal consultations, increased deliveries at health centers, increased post-natal follow-up visits by TBAs, and increased consumption of Vitamin A among children. In two districts increases in the consumption of iron during pregnancy have been noted. Main constraints, Problems The size of the project intervention area (4 districts) is too big for the number of field staff and the financial resources of the project. The ratio of field staff to community health workers is in excess of 1:200 and the number of activities is too great for project staff to supervise effectively. Although Health Center staff are supposed to assist with supervision, due to staff shortages, transportation constraints and differing priorities, their contact with CHWs is limited to monthly meetings and occasional contact during outreach activities. The Community-based Nutrition Program, the strategy chosen at the behest of the MOH, is probably not the most effective means to reducing the rate of malnutrition. Studies have shown that while monthly growth monitoring is useful for targeting more intensive nutrition activities and for monitoring the impact of other nutrition activities, as a single intervention (even when accompanied by individual counseling and monthly cooking demonstrations) it has not been shown to be effective.1Furthermore, after further scrutiny, it appears that the UNICEF pilot projects, conducted in Rwanda and which prompted the use of this strategy, were implemented under different circumstances than the CBNP in Kibungo District. Summary of the capacity-building effects of the program As a result of the capacity-building efforts of the project, 652 trained Traditional Birth Attendants are referring pregnant women throughout the Province to health centers for pre-natal consultations and deliveries. They are making, selling and using individual birthing kits, which not only promote attendance at pre-natal consultations, but increase safety and reduce the incidence of infection during home deliveries. Nutrition animators, trained by the project, are now conducting 114 community-based growth monitoring sessions per month, effectively increasing access to nutrition education and to micronutrient supplementation. Health animators have been trained in social marketing of impregnated bed nets and are promoting their use among pregnant women and 1 Thin on the Ground, Save the Children Fund, pg. 47, 48 and citation 123 5 children. All of these community health workers collect data regarding their activities as well as demographic information which is fed into the district health information system. District level MOH staff have been trained in Lot Quality Assurance Surveying techniques and, with assistance from the Project, have used these skills to evaluate the progress of the project. A summary of the prospects for sustainability Many of the activities initiated by the project have an excellent chance of being sustained for two reasons: community health workers receive an incentive through their membership in associations and because monthly meetings with the health center staff have been incorporated into the health center and the CHW’s schedules. The fabrication and sale of birthing kits has enabled many TBA associations to become self￾sufficient. Priority recommendations 1. Pilot-test the Hearth method in each of the four districts. If results are positive, adopt this strategy as the primary means to address moderate malnutrition. 2. Begin immediately to shift responsibility for supervising growth promotion sites to the health center and district health teams. 3. Formally assess and classify growth promotion sites, so that project supervisors can focus their efforts on strengthening the weakest sites. 4. Review current intermediate (process indicators) results (IR) and add IRs where necessary. Establish IR monitoring systems and track these on a quarterly basis. Monitor impact indicators only on a bi-annual basis (twice per year- this does not mean doing an LQAS twice per year.) 5. In collaboration with the Quality Assurance Project, assess the quality of obstetric care at all health centers in the Province. Then, in collaboration with District Health Teams develop in-service training opportunities geared toward up-grading the skills of health center staff responsible for conducting deliveries. 6 Chapter Two - Assessment of Progress toward Objectives ______________________________________________________________________ A. Results Chart Specific Objective Indicators BLD % Mid- Term % by District (LQAS results) Life of Project Target % KIB KIR RWI RWA NUTRITION % of mothers who take Vit. A within a month of delivery 31.8* 11* 16.2* 12* 21.5* 60 1.1 To increase the % of mothers who take Vit. A within a month of delivery 1.2 To increase % of children taking Vitamin A every 4 months % of children 6 - 23 months who have taken Vit. A in the last 4 mths** 3.3 75 38.5 68 53.6 50 To decrease the % of children <80% of weight for age+ 40 21 21 26 24 20 1.3 To decrease the % of children <80% and <65% of weight for age decrease the % of children <65% of weight for age+ 20 3 1 1 0 5 1.4. To increase proportion of children breast￾feeding within an hour of delivery % of children <2 who were breast-fed within an hour of birth 34.3 28 31.4 33 33.6 50 1.5. To Increase proportion of mothers who take Iron for more than a month during pregnancy % of mothers who took iron for at least a month during her last pregnancy 3.1 4 6.3 30 26.3 20 MATERNAL AND NEWBORN CARE % of women of children 0 – 11 months with at least 2 TT during last pregnancy 28.7 mothers with child < 23 m) 39 (2002) 45 % of obstetric transfers from community to health facility, as measured by TBAs 8.8 22.4 20.9 24.1 51.4 15 # of C-sections done at hospitals per reports 477 601 (2002) 2.1. To increase coverage and quality of prenatal care Improved quality of care at health centers 7 Specific Objective Indicators BLD % Mid- Term % by District (LQAS results) Life of Project Target % KIB KIR RWI RWA 2.2. To improve safety of delivery % of women who deliver at health centers or hospitals 16 25 14 19 35.7 20 % of women whose home delivery is attended by a (trained) TBA 30.7*+ 25 27.3 29 32.8 50 2.3 To improve coverage and quality of postnatal services % of women who consult a health facility within a month of delivery 5.4 ? woman c = child 7? 11c 8.7? 21c 9.2? 13c 10.4? 16.8c 20 MALARIA 3.1 To decrease incidence of malaria # of cases of death from febrile illness for children < 5 per year per 1,0000 25.7 12 18 15 3.2 Proportion of children under 2 who slept under a correctly impregnated mosquito net last night 3(0-11) 8(12-23) 17 15 16.6 12.3 22 16 17.8 7.3 20 3.3. Improved quality of malaria case management at the health centers * The baseline figure is inflated because the KPC was conducted just after a National Vaccination Day, when Vit. A was distributed widely. IRC believes that the baseline rate was about “0”. The Vit. A data collected for the LQAS survey stipulated within the last 2 months, not within the last one month which was the indicator for the baseline survey. Hence these figures are representative only. ** The Vit. A indicator for children was changed to every 6 months to conform to the MOH policy; and for the mid term LQAS the age range of the sample children was 12 – 23 months. + Baseline data for children’s weight was taken from a 1998 UNICEF study. The MOE children’s weight data was taken from the LQAS survey conducted in July 2003. +++ The LQAS survey conducted in July 2003 verified for 3 PNC since that is the current MOH policy *+ This is combined trained and untrained but is taken for the districts where over 300 TBAs had been trained by IRC as part of the entry level grant. 8 B. Technical Approach Overview of the Project The Kibungo Child Survival Project, currently being evaluated, was initiated in October 2001 following a 2-year (1999 – 2001) Entry-level grant. This four-year project seeks to reduce infant, child and maternal mortality by implementing activities in the areas of Nutrition, Reproductive Health and Malaria Prevention. The International Rescue Committee, in partnership with the Ministry of Health (primarily District and Health Center levels), is implementing the project in the Province of Kibungo, located in the Eastern part of Rwanda. The Province is divided into four health districts and the project is operational, albeit at different levels, in each of the districts. The specific objectives and indicators for the project are shown in the Results Table above and in the intervention-specific sections below. Progress Report by intervention Area Nutrition Indicators: ß Increase from 31.8% to 60% the percentage of mothers who take Vit. A within a month of delivery; ß Increase from 3.3 % to 50% the percentage of children 6 – 23 months who have taken Vitamin A within the last 4 months; ß Decrease the percentage of children <80% of weight for age to 20%; ß Decrease the percentage of children < 65% of weight for age to 5%; ß Increase from 34.4% to 50% the percentage of children who were breastfed within an hour of delivery; ß Increase from 3.1% to 20% the percentage of women who took iron during at least one month of their last pregnancy; Comments on Objectives: With regard to the Vitamin A indicator as it relates to women, the LQAS, which was conducted in July 2003, assessed Vitamin A consumption within the last 2, not one, month of delivery since this is the current MOH policy. Since it is the official policy it is recommended that the indicator be changed accordingly. Furthermore, given the mid-term results and the difficulties related to accessing Vitamin A, the project should reduce the target objective from 60% to 40%. The MOH has also changed the policy regarding Vitamin A consumption by children. It is now recommended that Vitamin A be taken at 6-month intervals. Hence, that indicator should be revised as well and the target objective increased to 75%. Neither of the indicators relating to malnutrition stipulates which age child the project is targeting. Although the MOH targets all children under 5 years of age, in reality the children most at risk and those who attend the growth promotion sessions are under 2 years of age. Hence for expediency sake, it is recommended that the target of the malnutrition indicators be set at < 2 years of age. (This age is also appropriate since this indicator is going to be evaluated through a KPC survey among mothers with children less than two 9 years of age.) In the Detailed Implementation Plan some of the objectives mention increasing/decreasing the proportion rather than the percentage. These should all be changed to percentages since the project measures in percentages of the whole target audience. Activities: The nutrition component is centered around community-based nutrition program (CBNP), an activity relatively new to Rwanda and currently the accepted national strategy to combat and prevent malnutrition. To implement the CBNP the project organizes, finances and helps implement refresher courses for the community animators, a MOH-recognized form of community health worker. It then equips and supports them in their efforts to monitor the growth of children under age 5 and to provide appropriate nutritional counseling. Related to the growth monitoring activity are follow-up cooking demonstrations for all mothers/children, which are conducted in some CBNP sites following the growth monitoring activities. During the monthly growth monitoring activity, Vitamin A and mebendazole are distributed at 6-month intervals (beginning at age 6 months for Vitamin A and at age 12 months for mebendazole). ORS is also available for children suffering from diarrhea. Fifty Rwandan francs are charged for the services, but no mothers are turned away if they can not pay. Three community (nutrition) animators were trained by the project for each CBNP site and each district has between 25 – 37 CBNP sites, for a total of about 488 trained nutrition animators. Generally speaking one of the three animators weighs the child (with help from the mother) and collects the fee. This person makes certain the mother sees the weight on the scale and s/he records the weight on the mother’s road to health chart. The second animator looks up the mother’s chart in a large register (one is kept for each cell, which roughly consists of 500 families) and records the date and whether the child gained or lost weight or remained the same. S/he charts the weight on the Road to Health graph and marks the target weight for the following CBNP session. At this point a conversation between the mother and the animator should take place and the appropriate advice given to the mother. If the child needs to be referred to the health center for additional follow-up, this animator fills out a referral form. The third animator dispenses the Vitamin A and mebendazole as needed and marks the chart accordingly. To facilitate the work of the animators, the project is using two tools: a target weight chart and a counseling card. The target weight chart shows the present weight of the child in one column and then projects what the child, if growing normally, should weigh the following month. The animators are trained to mark the targeted weight on the Road to Health Card and tell the mother the projected weight. This is supposed to serve as an incentive to the mother. The Counseling card is a laminated sheet that contains general messages appropriate for mothers of children of specific ages and with specific issues. It is supposed to serve as a reminder to the animators regarding the key messages. Nutrition animators attended a 3-day training course implemented by the district health teams and project personnel. Training was done in phases and CBPG sites initiated accordingly. IRC Supervisors, two per district, who are seconded to the District Health 10 Team, are responsible for supervising the CBPGs and helping the animators master the skills necessary to implement the CBPG effectively. A supervisory checklist was developed to help monitor the animator’s abilities to execute the CBNP. At the end of each month the nutrition animators compile the data collected during the CBNP sessions and delivery this information to the health center to which they are affiliated. This is one of the activities that takes place during monthly nutrition animator meetings held at the health center. The Health Center staff then compile the data from all of the CBNP sites in their catchment area and send this to the District Health Office. In an effort to sustain interest and motivation among the animators, the project has helped the animators form themselves into officially recognized associations. Each association has a bank account and pools the proceeds (30 Rwandan francs of the 50 francs collected from each family each month) from the CBNP into the account. These funds are then used by the association to fund other income generation activities implemented by the association. Some associations have gardens, while others raise small livestock for resale. Discussions during the monthly meetings often center around association issues. In addition to helping to support the association’s income generating activities, 40% (20 of the 50 Rwanda francs paid per family per month) of the proceeds from the CBNP fee is used to replenish the supply of mebendazole. In addition to the CBNP strategy, the DIP also mentions the Hearth method as a second possible strategy to address malnutrition. The IRC child survival advisor reports that the MOH partners were not particularly interested in this approach initially, and as result thus far the project has focused on the CBNP. At the time of the mid-term evaluation, however, a team from the project was attending a training on Hearth organized by World Relief (another CS grantee). Training, supporting and supervising traditional birth attendants (TBAs) is the strategy designed to promote achievement of the nutrition objectives related to micronutrient consumption and breastfeeding. Six hundred and fifty-two TBAs were trained by the project and these women (and a handful of men) promote attendance at pre and post￾natal consultations where iron is supposed to be distributed and immediate breastfeeding promoted, and where an initial dose of Vitamin A is administered post￾natally. Progress: The project has trained 488 nutrition animators who implement 114 CBNP sites throughout the Province each month. Not all health sites have CBNP activities or trained nutrition animators. The project weighs approximately 10,000 children per month, or a little less than a quarter of all of the children in the province under 24 months of age (46,349). The supervision checklist has been developed and is being used by project supervisors though not consistently or for the purposes of concretely assessing the quality of CBNP program implementation. Over 650 TBAs have been trained leaving about 150 left to 11 be trained to meet the objectives of the project. The data collection system is in place and nutrition animators are collecting data monthly and submitting it to the health center staff who are compiling it and submitting it to the District Health Office for analysis. Rather than collect anthropometric data itself as part of the baseline KPC, IRC and its partner decided to take this data from a 1998 UNICEF report. The UNICEF study conducted in the Province in 1998 stated that moderate malnutrition among children 6 – 35 months averages around 26% and severe malnutrition averaged around 21.9 %. The LQAS conducted by the project in July 2003 revealed that 37.4 % of the children ages 12 – 23 months are moderately malnourished. This suggests that a significant challenge lies ahead to reduce malnutrition to the project target of 20% (children < 36 months). To achieve this, the project will have to take steps to significantly increase attendance at the CBNP sites, which currently fluctuates between 20% - 30%, according to the project’s data. While post-natal consumption of Vitamin A among women appears to have decreased since the KPC survey was conducted, this is probably due to the fact the KPC survey was conducted just after a National Immunization Day when Vitamin A was widely distributed. Therefore the baseline data shown (31.8 %) is artificially high. Regardless of the reason, the life-of-project target is too ambitious and should be reduced. Unlike Vitamin A consumption among women, the LQAS results suggest that Vitamin A consumption among children has increased significantly (from 3.3% to as high as 75%) and that this project target needs to be increased. The breastfeeding behavior appears not to have changed at all. This can be attributed to the fact that the breastfeeding message during the TBA training was not emphasized enough, as evidenced by the training guide, and consequently TBAs do not actively promote this behavior among new mothers. TBAs interviewed for this evaluation report focusing on other messages, such as the need for pre-natal consultations and the dangers signs during pregnancy and delivery. Iron consumption among pregnant women varies dramatically from district to district; showing hardly any change in Kibungo and Kirehe Districts (from 3.1% to 4% and 6.3% respectively) and impressive changes in Rwinkwavu and Rwamagana Districts (from 3.1% to 30% and 26.3% respectively). It is not clear why such differences should occur, but the evaluation findings suggest that not all (in fact, none) of the districts adhere to the MOH policy of free distribution of iron to pregnant women. Furthermore some districts may have experienced stock-outs due to lack of justification (of distribution), which is required by UNICEF and the MOH Nutrition Division to replenish supplies. Effectiveness: The Community based nutrition program (CBNP) strategy has successfully increased mother’s access to health and nutrition information and to micronutrients, worm medication and ORS packets. It has enabled nutrition animators to identify some malnourished children and to refer those with immediate health/nutrition needs to the health center. To the extent that health center staff offer immunization 12 services at the CBNP sites, vaccination coverage may also have increased as a result of the CBNP strategy. As a strategy to impact malnutrition, however, the CBNP is not likely to have the desired impact. This conclusion is based on several project evaluation findings, and on studies from around the world.2 The project-specific issues that compromise the effectiveness of the strategy include the following: poor attendance at CBNPs; inconsistent and inadequate individual counseling; inconsistent availability of micronutrients and lack of follow-up between sessions and infrequent supervision. Even if CBNP implementation was perfect, however, it is unlikely that a significant reduction in malnutrition would result. Mothers typically do not change their infant and child feeding habits based on one or two (assuming there might be a home visit conducted between growth monitoring sessions) contacts with a nutrition animator. Studies have shown that nutrition workers from around the world have tried to reduce malnutrition through community-based nutrition program, implemented under similar circumstances as those in Kibungo, without significant impact. As mentioned above, attendance at the CBNP sessions is usually well below 30% and it is common knowledge that the most malnourished children do not attend growth monitoring events because they might prove embarrassing. Supervisor checklists and evaluation observations (though only two were done) confirm the presence of a typical and chronic problem: weighing the baby takes precedence over analyzing the health/nutrition situation of the child and giving appropriate and complete advice. With many women waiting for their turn and anxious to return to their daily chores, animators cut back on counseling, which is the intervention’s primary avenue to behavior change. Even with the aid of the key messages card, it appears that for lack of time, and perhaps confidence, animators often fail to provide individual counseling. And the target weight card only serves to focus more attention on recording the desired weight and not on advising the mother what to do to achieve the weight gain. Evaluation team members observed several missed opportunities for reinforcing different key messages and promoting important behaviors. For example, if any individual counseling takes place, it is directed primarily at malnourished or ill children. By focusing on these children alone opportunities to promote such behaviors as exclusive breastfeeding and sleeping under a mosquito net are missed. Furthermore, even though many nutrition animators are at the same time health animators (who are responsible for selling mosquito nets), no nets are for sale at the CBNP sites. In recognition of these opportunities to create synergies between the different components of the project, the evaluation team recommends that project staff deliberately identify 2 Thin on the Ground, Save the Children Fund, 2003, pg. 47, 48; Conclusion from the review of three growth monitoring and promotion projects, one of which was implemented in Uganda. “Nutrition projects like the three reviewed here that rely on growth monitoring and promotion as the primary tool with which to achieve lasting impact on nutrition, are likely to face similar design and implementation problems in other country contexts. Many of the problems highlighted here have been raised elsewhere by others (citation 123 that includes 4 references). Save the Children UK believes these problems cast sufficient doubt on whether this project represents the right model for poor country governments to adopt to improve malnutrition on a national scale”. 13 such opportunities and begin to promote the key messages and behaviors at every possible moment. Micronutrients are sometimes not available at the CBNP. This happens when families refuse to pay the 50 francs and initial supplies can not be replenished for lack of funds, or because there is a stock-out of the item in the district (such was the case with Vitamin A in Kirehe District during the evaluation). And finally, IRC Supervisors can not possibly visit all of the CBNPs that take place in their district each month simply because there are too many (25 – 37), the distances are two vast, and the terrain is difficult. To compound these supervision challenges, the checklist that was developed by the project to help supervisors assess the performance of the nutrition animators contains some design flaws that hampers its effectiveness and has caused it to be used inconsistently by IRC supervisors. More specifically, the checklist contains the following flaws: v There are no instructions regarding how to complete the form; v The levels of performance ( 0,1, 2, 3,) are not defined (some supervisors think “0’ means good while others think “3” means good); v The indicators for performance are not clear enough to be understood and interpreted consistently among Supervisors; v There are no indicators that relate to the provision, quality or appropriateness of individual counseling; v There are no indicators regarding the use of the Counseling card or target-weight chart; v It is not clear how the score should be calculated or how the score should be compared to anticipated outcomes (such as, excellent performance, acceptable performance; average performance; unsatisfactory performance); v The form doesn’t say how the results of the completed form should be used. In addition to these design flaws, the way the checklists are classified (by district and not by CBNP site), it is clear that prior supervision forms are not being compared to completed supervision checklists in order to assess progress toward a desired performance level. And finally CBNP supervision and performance are not reviewed at the weekly staff meetings leading one to believe that this assessment is not valued. For all of these reasons the CBNP, while quite beneficial for other reasons, is perhaps not the most effective means to reduce malnutrition. Fortunately the project had planned on testing the Hearth Method, and project staff took advantage of a training being offered by another PVO in Rwanda to learn how the method works. Given the success rate of the Hearth Method, and particularly its intense focus on the moderately malnourished child, it is recommended that project staff train district health teams in the method as part of pilot tests in each district. The pilot test should be monitored and evaluated and the results disseminated and discussed to determine if the approach should be implemented more widely in the Province. 14 The regular supply and provision of Vitamin A and iron is a problem. According to project staff and the Nutrition Division Chief, District Health Teams typically do not track the distribution of free supplies (such as micronutrients) as carefully as they do drugs that are sold. As a result, when the micronutrients run out and the distribution can not be justified, the supply is not quickly replenished and a stock-outs occur. Consequently stock-outs of Vitamin A have been reported in the province and none of the districts was adhering to the MOH policy of providing iron tablets to pregnant women free of charge. To address this issue, the MOH has decided to provide micronutrients to women and children through bi-annual mass distribution, similar to annual vaccination days. This campaign, which will be on-going, is set to begin in November 2003 according to the Nutrition Division Chief. In this case, the CBNP will serve only to catch those who missed the mass administration campaign. Unfortunately the Ministry’ s plan will not adequately meet the need since pregnant women need to get iron throughout the duration of their pregnancy and Vitamin A particularly during the post-partum period (within 4 weeks of delivery). The Ministry would do well to strengthen the provision of these micronutrients by health centers. Once the mass distribution of micronutrients becomes effective, the TBA’s role in promoting Vitamin A and iron consumption will be less critical. However, TBA training should continue to cover the need for these micronutrients so that pregnant and lactating women are sure to attend the micronutrient distribution days. So far the role of TBAs in promoting immediate and exclusive breastfeeding has been under-exploited. This is in part due to the fact that the MOH requires the project to use an outdated and partially unsuitable training curriculum for training the TBAs. The training curriculum does not focus on immediate or exclusive breastfeeding, but does include such things as family planning, which, while pertinent, does not help to meet the objectives of the project. The MOH/Kigali and District Health Teams need to be more assertive in assessing the relevance of each of the training modules in the standard TBA training course, and chose those that contribute to the achievement of the training objectives. Furthermore, the modules need to be up-dated so they accent the targeted behaviors, such as immediate and exclusive breastfeeding. This is especially important since all training events are limited to three days, making is very difficult to cover anything but the most essential themes. Changes in technical approach and rationale: Recognizing the inherent limits of the CBNP as a means to combat malnutrition and make permanent changes in child feeding habits, the project should focus on implementing the Hearth approach during the second half of the project. As mentioned above, once the MOH begins mass distribution of micronutrients, according to the Nutrition Department head, the role of the CBNP sites in micronutrient distribution will change to that of catching missed opportunities, rather than primary provider. 15 The major activities mentioned in the DIP related to securing a constant supply of Vitamin A are not being undertaken and should be dropped from the program because they are not within the purview of the project. Special Outcomes: As was mentioned previously, the establishment of CBNP sites has facilitated the health centers’ vaccination outreach activities by providing another reason for mothers with small children to congregate. The synergy of these two activities epitomizes the MOH’s idea of integrated service delivery. Follow up/Next Steps: 1. Revise the CBNP supervision checklist to address the flaws identified. Train project and health center staff in its use and divide up supervision tasks. 2. Following training, pilot test the Hearth Method in each of the four districts, monitor and disseminate results; decide whether or not to continue using this approach to reduce malnutrition 3. Continue work on organizing associations Reproductive Health Indicators: ß Increase from 33% to 45% the percentage of women reporting at least 2 TT immunizations for last pregnancy; ß Increase from 6% to 15% the percentage of obstetric transfers from community to health facility, as measured by TBAs; ß Increase the number of cesarean section done in hospitals ß Improve quality of care at health centers Comments on Objectives: The above sighted indicators were presented for the first time in the Annual Project Report for the period Oct. 2001 – Sept. 2002. The objectives/indicators were changed following a review of the DIP, presumably in an effort to strengthen them. The first indicator is stronger in that it defines the quality of the pre-natal consultation, by tracking the delivery of a specific service (TT vaccinations). The second indicator is less strong because it measures a TBA’s behavior (transfers) rather than the impact (increases in facility deliveries and presumably improved birth outcomes). For this reason, the original indicator, increase the percentage of deliveries conducted at health centers and hospitals is preferable. The third indicator, as explained in the annual report, seeks to reflect “improved birth outcomes”. Unfortunately the incidence of C-sections is not an effective indicator of improved birth outcomes. In fact the incidence of C-sections, especially in a developing country, is more likely an indication of increased risk of maternal mortality, given the gravity of this major surgery and the typical quality of care available. Furthermore, the project does not intend to assess the quality of obstetric care at the 2 hospitals or to improve it, if this is warranted. And finally, the project can only monitor this indicator by using data from 2 sources where C-sections are performed. All of this makes this 16 indicator highly questionable and a weak indicator of improved birth outcomes. Since too many factors influence birth outcomes, most of which are beyond the scope of a child survival project, it is not recommended that this project commit itself to directly improving birth outcomes. Several proxy indicators, that the project does influence, can be used, however. For example, the percentage of women reporting having been attended by a trained TBA for their home birth, and the percentage of women who report using a birthing kit during their home birth. The last indicator, which focuses on improving the quality of health care delivery at the health center level, remains undefined and therefore unmeasurable. Because this indicator reflects a behavior on the part of MOH health care providers and not an impact, it is more appropriate as an intermediate result and not an impact indicator. As an intermediate result, it is also more consistent with the fact that IRC can not easily impact the quality of health care delivery because the MOH (central and district) maintain strict control over quality of care issues, and have proved hesitant to take the steps necessary to assess quality of care and to make the necessary improvements.3 In light of this, it is recommended that the improved quality of care at health centers indicator be moved to the intermediate result position and removed as an impact indicator in the project’s matrix. Activities: The Reproductive Health component is centered on the Traditional Birth Attendant (TBA). The project trains and provides refresher courses to TBAs. Trained TBAs promote attendance at pre-natal consultations (PNC) at the local health centers and often attend PNC themselves to provide health education to the waiting women and to sell birthing kits. TBAs fabricate and sell individual birthing kits4 to pregnant women, attend normal home deliveries and weigh the newborn. They refer pregnant women at￾risk to the health center for delivery and visit new mothers at home and check for post￾natal danger signs in both the mother and the baby. They refer mothers to the health center should any danger signs be apparent. The TBAs collect and submit monthly reports on their work (including birth outcomes) and the work of untrained TBAs. The TBAs are supposed to promote immediate and exclusive breastfeeding, vaccination and other healthful behaviors. The project also helped the TBAs to form themselves into officially recognized associations. This has enabled them to open bank accounts in the community-based Banque Populaire into which they deposit the proceeds from the sale of the birthing kits. The small kits sell for RWR. 250 and the large kit (with small blanket) sells for RWF. 500. Progress: Six hundred and fifty-two (652) TBAs have been trained/re-trained in the four districts thus far, which represents a little over 2/3 of the total number (800) of TBAs to be trained and 52 more than the original target, of 600. In order to promote attendance at pre-natal consultations, TBAs fabricate birthing kits for individual use, and sell these to women who have completed 3 pre-natal consultations. While the TBAs were already 3 First Annual Report, pg. 4 4 The birth kits include gloves, cord, razor blade, soap, antiseptic liquid, and in some cases a small blanket. 17 assembling the kits themselves, during the first year of the project, financial assistance from an American PVO called Netaid was received, which provided funds to purchase the birthing kit materials. Assistance from Netaid supported the purchase of kit materials in one district and when this assistance ends in December 2003, the TBAs will revert to making the kits themselves, without financial assistance. Netaid also paid the salary of a project staff member, however, who supported the work of TBAs in all four districts. The project seeks to improve the safety of deliveries by encouraging pregnant women to give birth at the local health center. Unfortunately, under the present circumstances it is not clear that giving birth at the health center is that much safer than a home delivery attended by a trained, experienced TBA who uses a birthing kit. This is because very few deliveries in health centers in the Kibungo Province are attended by health care providers who have professional midwifery or obstetrical training. In recognition of this, according to MOH policy, health center staff are only allowed to handle “normal” deliveries; all others must be transferred to the nearest hospital. The only very clear advantage to delivering in a health center is that the health center staff is authorized to transfer patients to a hospital whereas TBAs are not. In 2002, a reproductive health specialist visited the project to ascertain the level of competence of the health center staff with regards to reproductive health services. The consultant focused on pre- and post-natal services, however, and not on obstetrical care. Her stay of one-week precluded a more in-depth study. The consultant made several recommendations for improving pre and post-natal consultation services and these were referred to the MOH’s Reproductive Health Division who is currently (still) developing a training curriculum to address these issues. While waiting for the MOH curriculum to be completed and approved, and for funds to conduct in-depth training for all of the health center staff responsible for obstetrical care, it is recommended that the project and the district health teams of each district assess the quality of obstetrical care at each health center with assistance from the Quality Assurance Project (located in Kigali). Following the assessment, approaches to address the weaknesses should be identified and implemented within the shortest timeframe possible. Possible solutions could include short competency-based and very specific “refresher” courses and the provision of algorithms to help caregivers follow standard operating procedures. Effectiveness: The project’s work with the TBAs has proven to be quite effective in a number of ways. For example, the training of the TBAs has reinforced their skills and knowledge, particularly as they relate to the use of the birthing kit (clean delivery and prevention of the spread of HIV infection), the importance of pre- and post-natal consultations and the need for some women to give birth at the health center; and the danger signs during pregnancy, labor and the postnatal period. The project’s work has effectively improved relations between the TBAs and the health center staff by bring the two parties into contact regularly during the monthly meetings. 18 This regular contact means the health center staff have become more familiar with the TBAs and the TBAs have gained confidence in the health center staff, thereby increasing the likelihood of transfers to the health center. The fabrication and use of the birthing kit has helped increase attendance at pre-natal consultations and has improved safety and cleanliness during home births. Follow-up (post-natal) visits by trained TBAs have increased as a result of the training as well. An additional achievement worth noting is the data collection system that appears to be providing credible data regarding births, infant and maternal mortality, transfers and the outcomes of births attended by untrained TBAs. This data, coupled with that of the health animators, provides insight into the maternal mortality situation in the province. As with the nutrition animators, the project has facilitated the formation of 25 associations of TBAs. These associations deposit the proceeds from the sale of birthing kits into a local bank account and use these funds to finance other income generation activities. Of the 25 associations, 13 are financially self sufficient – that is, they are able to purchase the birthing kit material themselves from the proceeds of the sales. This approach is an extremely effective means to sustaining the work of the TBAs. At the end of each month the TBAs (and other community health workers) meet at the health center to submit their reports, discuss issues of mutual concern and often times to review technical themes related to the work of the TBAs. The evaluation team assessed the technical knowledge of the TBAs and found that in general their understanding is complete. However, there are some key messages, such as immediate breastfeeding, that the TBAs do not promote actively. To address this gap, health center staff should organize their technical review themes to include the key messages and behaviors of the project and regularly assess the knowledge of the TBAs (especially new association members) in a formal and structured way. The project has successfully created a demand for the birthing kits. According to the LQAS conducted at midterm, kits were utilized during 52% (average of four districts) of last births. Changes in technical approach and rationale: There are no suggestions regarding changes to technical approach; however changes related to training are addressed in the Training section. Special Outcomes: The formation of the TBAs into associations and the assistance provided by the project in support of the birthing kits, has set the stage for all TBA associations to become self-sustaining entities. This will help ensure the continued provision of TBA services in the province. Follow up/Next Steps: · Review the TBA curriculum and modify the content so that all of the modules directly support the training objectives (see training section for discussion of this); put more emphasis on immediate and exclusive breastfeeding and on postnatal visits; 19 · Continue to identify the remaining untrained TBAs and train them; · Formalize and expand the participation of TBAs in pre-natal consultations and monitor and assess this participation; · Assist remaining TBAs to form or join associations; · Assist associations in their birthing kit production; · Assist associations with their income generation plans. Malaria Prevention Indicators: · Decrease the number of deaths from febrile illness among children < 5, from 25.7 per 1,000 to 15 per 1,000. · Increase the proportion of children , 2 who slept under a correctly impregnated mosquito net last night · Improved quality of malaria case management at the health centers Comments on Objectives: The above sighted indicators were presented for the first time in the Annual Project Report for the period Oct. 2001 – Sept. 2002 and represent a change from those objectives/ indicators presented in the DIP. While the evaluation team agrees that the original objectives and indicators needed to be changed, there now appears to be a lack of congruence between the first and last (new) indicators (stated above) and the activities of the project. There is only one activity that could potentially result in reduced febrile illness in children (promotion of bed net use). However because so many other illnesses and conditions also cause fever, promoting impregnated bed net use alone is not likely to achieve the objective as it is so broadly stated. There are currently no activities to improve quality of malaria case management at health centers. Hence these indicators are unrealistic and need to be changed. The project staff has been working with other PVOs and the National Malaria Control Program (PNLP) and has developed some alternative objectives/indicators that it intends to use. These, and others suggested by the evaluation team, are stated below. Proposed New Objectives and Indicators for Malaria Prevention Component: Objective Indicators Intermediate Results Major Activities Increase from ___ % to 50% the percent of mothers of children < 2 who can name 3 ways to avoid getting malaria Numbers of community health workers trained in malaria prevention and BCC techniques Training of CHWs in malaria prevention and BCC techniques To increase the use of malaria prevention measures being taken by targeted community members Increase from 7% to 25 % the percentage of children < 2 years of age who slept under an impregnated bed net Number of impregnated bed nets sold -Train health animators in social marketing of impregnated bed nets; - Secure supply of long lasting bed nets; 20 Objective Indicators Intermediate Results Major Activities Increase from ___ to 65% the percent of mothers of children < 2 who can name 2 symptoms of malaria. Numbers of community health workers trained in malaria prevention and BCC techniques Training of CHWs in malaria prevention and BCC techniques To increase targeted caregivers’ abilities to manage children with fever (presumptive malaria) Increase from ___ to 60% the percent of mothers who sought treatment for their febrile child < 2 years of age (presumptive malaria) within 24 hours - Numbers of community health workers trained in malaria prevention and BCC techniques; - Numbers of CHW trained in community￾based treatment of malaria; - numbers of CHWs equipped to provide community-based treatment; - numbers of CHWs providing community￾based curative services for presumptive malaria. Training of CHWs in malaria prevention and BCC techniques; Training of selected CHWs in presumptive community based treatment of malaria; Establishment of supply system; Establishment of monitoring system; Activities: The Detailed Implementation Plan proposes several activities to achieve the (original) stated objectives. These include: enforcement of current MOH protocols for malaria diagnosis and treatment; formal training and regular supervision of health center staff; development and use of supervision checklist related to malaria case management; work with PSI to purchase bed nets and impregnation supplies from the national pharmacy and supply them to nutrition program centers; sale of bed nets at CBNP sites5 . In lieu of these activities, the project developed malaria health messages; supported the training of community health animators by PSI in social marketing of impregnated bed nets, and has established a data gathering system, which, among other things, tracks the number of bed nets sold by health animators and the causes of death, including febrile illness. The sale of bed nets takes place as follows. The district health office supplies the bed nets to the various health centers. The health animators educate the community regarding the importance of using a bed net, especially for pregnant women and children < 2. The health animators then take bed net orders from community members, collect the money, get the bed net from the health center and deliver it to the family in question. A Swiss PVO originally supplied bed nets and the proceeds from the sale of the nets were supposed to be used to purchase subsequent supplies. Originally health animators maintained a supply of bed nets which they sold from their homes. However 5 Kibungo Child Survival Project, DIP pg. 127 -130 21 this practice was discontinued when too many bed nets were distributed without collecting the payment. Recognizing the need to bolster the malaria component, the Project Advisor, who was hired in December 2002, has been working with the National Malaria Program (PNLP) to redefine the activities and to link up with other PVOs to develop creative means to address the serious issues of malaria-caused morbidity and mortality. Two pilot projects have been designed, one of which is on the verge of implementation and the other awaits World Bank approval and funding. Progress: To date, progress related to the malaria control component has included training 440 health animators in three of the four health districts to promote the purchase and use of impregnated bed nets, and to organize community-wide clean up of mosquito breeding areas. The LQAS results show that the percentage of children sleeping under bed nets in districts where health animators have been trained, has increased from 3.6% to between 12.3% and 16% for children between the ages of 12 – 23 months and between 16.6% and 22% for children ages 0 – 11 months of age. Despite this progress, the project has recorded flat bed net sales following an initial burst of interest. The project has also established a community-based data collection system, which tracks births, deaths and causes of death. This enables the project to firmly establish the prevalence of febrile (presumptive malaria) illness among young children. As mentioned above, two new strategies have been identified to address the malaria situation. One entails community-based presumptive treatment of malaria by trained community health workers (health animators). Although the MOH policy does not allow community health workers to treat malaria, it has agreed to permit three international PVOs, IRC, Concern and World Relief, to pilot test this approach. The pilot test is expected to begin before the end of the year. The second strategy, which will be implemented if World Bank funding is secured, entails the purchase and distribution (free of charge) of long lasting mosquito nets to especially vulnerable populations in the district of Kirehe. The premise behind this approach is that the cost of the bed nets and regular re-impregnation is too high for the poorest segments of society. Therefore, the project seeks to target distribution of these bed nets to those populations using its data collection system. Effectiveness: Compared to all that the project intended to do in the field of malaria, the effectiveness of the project’s approach has been minimal. This is because the input, thus far, has been limited to training health animators. On the other hand, the data collection system managed by the health animators has attracted quite a lot of positive attention. Unfortunately the importance of the data collection system will only be felt once project activities begin to have an impact on febrile illness in children. Should the pilot test of community-based presumptive treatment of malaria prove successful, 22 however, and should it lead to a change in the MOH policy regarding community-based treatment, this will be a major achievement of the project. The potential effectiveness of the long-lasting mosquito net distribution is great but will need to be assessed. This assessment should include not only the impact of the bed net use on the target population, but also the impact of the new strategy ( “free” distribution) on the original and on-going strategy, sale of bed nets. When two different strategies are being applied in neighboring communities, and one is perceived as being more advantageous, the chance for a negative outcome (significantly depressed sales of bed nets) is high. Changes in technical approach and rationale: This report recommends changing the indicators and intermediate results of the malaria component for the reasons discussed above. Other changes include the two alternative strategies, community-based presumptive treatment of malaria and free distribution of long-lasting bed nets to vulnerable communities, discussed in this section. Special Outcomes: The MOH’s willingness to pilot test the community-based presumptive treatment of malaria is a particularly important outcome thus far, and will take on more significance if the pilot activity leads to an official change of policy in this regard. Follow up/Next Steps: · Consider the proposed alternative objectives and indicators and take steps to “officially” incorporate them into the project by notifying the donor; · Finalize pilot activity implementation plan; · Implement steps in the implementation plan; · Secure funding from World Bank; · Purchase long-lasting bed nets, and implement other activities related to the program. Intervention-Related Recommendations Recommendation 1. Change the Vitamin A indicator to match the current MOH policy 2. The project should reduce the target objective from 60% to 40%. 3. Increase target objective for Vitamin A consumption among children from 50% to 70%. 4. Change malnutrition indicators to target children less than 2 years of age. 5. Increase target objective for malnutrition (< 80% weight for age) from 20% to 25%. 6. Standardize all indicators to read “percentage” rather than “proportion”. 7. Modify the CBNP supervision checklist and correct the flaws identified in this report. Train staff and health center personnel to use it correctly; Use the new checklist to assess the performance of nutrition animators and to organize on-the-job training and support. 8. Train district health teams in the Hearth method as part of pilot tests in each district. Monitor and evaluate the pilot hearth activities and disseminate and discuss the results; determine if the approach should be implemented more widely in the Province. 9. Up-date TBA training module so they accent the targeted behaviors, such as immediate and exclusive breastfeeding and need to post-natal consultations 10. Refrain from measuring improved birth outcomes by promoting C-sections. Rather, use the 23 percentage of facility births, percentage of women reporting having been attended by a trained TBA for their home birth, and the percentage of women who report using a birthing kit during their home birth as proxy indicators for improved birth outcomes. 11. Move the “improved quality of care at health centers” indicator to the intermediate result position and remove altogether it as an impact indicator in the project’s matrix. 12. Project staff and the district health teams of each district assess the quality of obstetrical care at each health center with assistance from the Quality Assurance Project (located in Kigali). Following the assessment, approaches to address the weaknesses revealed should be identified and implemented within the shortest timeframe possible. 13. Project staff deliberately identify missed opportunities, and begin to promote the key messages and behaviors at every possible moment 14. The alternative objectives and indicators for the malaria prevention component sited in the evaluation report should be reviewed and considered by the project staff; if approved they should officially replace those in the DIP . C. New Tools or Approaches, Operations Research or Special Studies The project is using a number of new tools and approaches to facilitate activity implementation and achievement of project objectives. These include the counseling card, the targeted approach to growth promotion, the community-based treatment of malaria and distribution of long lasting bed nets. All of these are discussed in the intervention-related sections above. The formation of CHWs into officially recognized Associations to create an incentive for continued work is also a novel approach to sustainability that merits attention. And finally the use of individual birthing kits to promote attendance at pre-natal consultations, to improve the safety of home deliveries and as a means to generate income for TBAs is an approach worth studying for potential replication in other parts of the country. D. Cross-Cutting Approaches 1. Community Mobilization The project works primarily through Community Health Workers (TBAs, nutrition and health animators) who were already designated prior to the project, therefore no (or minimal) community mobilization efforts were required in the selection of CHWs. (Nutrition animators are a “new” type of CHW, but in reality most of them are health animators who have been given additional tasks.) Because no mention of community mobilization efforts is made in the annual reports, it’s not clear what community mobilization activities were undertaken. Project staff mention having had dialogues with community leaders to initially introduce the project and a few Participatory Learning and Action (PLA) activities have been undertaken. It appears that the project relies heavily on the various CHW’s (many of whom are respected community leaders) individual efforts to introduce specific activities and to promote participation. Since most of the behaviors promoted by the project are individual behaviors and the strategies employed by the project do not require collective decision-making or action, the strategy of using individual CHWs as change agents may be sufficient. On the other hand, when the ratio of CHW to community members is too low (3 nutrition animators 24 for 500 families) this many result in insufficient community mobilization and inadequate participation in such activities as CBNP. Despite the challenges, project staff (supervisors, in particular) need to make a greater and more consistent effort to remain in contact with community leaders and to more assertively promote participation in the project’s activities. 2. Community-based Health Information System The project should be commended for having developed a community-based health information system that is managed by CHWs and provides very valuable information not only about the activities of the CHWs, but also about the health status of the target population. All of the CHWs collect information on a monthly basis and submit their reports to the health center during monthly meetings. The health center staff then compile the data and submit it to the district health office who then enters it into their data base and uses some of it in their reports. Traditional Birth Attendants collect information on the number of births they and untrained TBAs conduct, the nutritional status of the newborn (green = well nourished; yellow = moderately malnourished and red = severely malnourished), numbers of pregnant women transferred to a health center and the causes, and any maternal or infant deaths. TBAs also records whether or not a birthing kit was used. Because most of the TBAs are illiterate, a literate family member usually completes their monthly reports. Accuracy is not a particular concern because most TBAs only conduct 1 – 3 births per month, making is easy to recall the information even if it isn’t recorded immediately. Health Animators complete a report each month that includes the following information: births, deaths, deaths in children < 5 years old, causes of death, number of mosquito nets sold, number of home visits conducted, topics covered during health education talks. Since TBAs and health animators collect similar data, his permits some cross confirmation of data. Nutrition animators compile the data they collect during CBNP sessions each month and submit this to the health center. This data includes: the number of children weighed by age group (<3 and < 5); the nutritional status of all children (numbers of children in the green, yellow, red), the number of Vitamin A and mebendezole tablets distributed, and the number of packets of ORS distributed. When this information reaches the IRC office, the numbers are converted to percentages by comparing them with the most recent census data (this is being up-dated in the months ahead). It is by this means that the coverage and overall nutritional status is known. Although this health information system is impressive, project staff spend an inordinate amount of time collecting what is essentially impact-focused data, and much less time monitoring inputs and outcomes. As a result, it is not clear to what activities progress 25 (or the lack of progress) can be attributed. The over-emphasis on data collection is evidenced by the responses of project supervisors to the question: What behaviors is the project promoting among women? Many lost sight of the actual behaviors but cited such things as reduced infant and maternal mortality and reduced malnutrition. As the project’s primary change agents, if they are too focused on the end results and not the means to getting there, then achievement of the objectives is in question. To remedy this, instead of collecting demographic and nutritional data on a monthly basis, the project should reduce the frequency to bi-annually. (Health centers and District Health Offices can continue to collect the data on a monthly basis, if necessary.) In its place the project should establish systems to collect information related to the intermediate results (probably additional intermediate results – outcomes – could be established for each objective) and monitor the provision of inputs and the corresponding outcomes on a more frequent and consistent basis (quarterly). In other words the project should focus on monitoring the process of project implementation with the assurance that, when there is a rational rapport between the inputs, outcomes, intermediate results and the end-of- projective objectives and indicators, then effective provision of inputs should result in achievement of the stated objectives. 3. Communication for Behavior Change The project’s behavior change strategy focuses almost exclusively on the three types of Community Health Workers. The TBAs promote the reproductive health behaviors and the nutrition behaviors directed at mothers. The nutrition animators promote improved feeding habits, especially among mothers of children who fail to grow adequately and care-seeking habits among mothers with severely malnourished or sick children. And the health animators are responsible for promoting the purchase and use of impregnated bed nets and to increase access to bed nets by making them available at the community level. If pregnant women come to the health center for prenatal consultations or to give birth, then some of the key messages may be reinforced by health center staff. The DIP describes other BCC strategies: radio programs, community plays, messages by political and/or religious leaders, but in fact none of these channels were being used at the time of the mid-term evaluation. The DIP also sets forth three principles for a successful behavior change program: communicate through multiple channels; tangible, immediate benefits and simple, clear and consistent messages. Related to the first principle is the understanding that people need to hear a message many many times (some specialists even suggest 8 times) before they will even consider changing their behavior. Missing from these principles is the removal of any barriers to change (although the project does address some barriers). Despite having identified these guidelines the BCC strategy does not adhere to the guidelines very closely. The channels are limited to oral messages by CHWs, and to a 26 much lesser extent to health center staff, and the benefits of the desired behaviors, especially the nutrition and reproductive health behaviors, are not tangible or immediate. The evaluation was not able to assess the consistency of the key messages, though CHW’s knowledge seems to be satisfactory. During CBNP activity the only consistent means of communication is the group health education talk and the identification of the weight of the child. Individual counseling is rarely given and without any close supervision, it is doubtful whether home visits are conducted frequently (certainly not weekly, as the DIP proposes). The nutrition animators observed by the evaluation team were not using the counseling cards and there is no mention of the cards on the CBNP supervision checklist. Because the number of CBNPs is so great compared to the number of project supervisors, supervisory visits to CBNP sites are not frequent. TBAs do attend pre-natal consultations at health centers each week, but the project does not collect data regarding any health talks they might give to the pregnant women. The health animators are supposed to give health talks at the community level and to carry out home visits. They report the number of home visits and the topics covered on their monthly reports. A review of these reports, however, suggests that at least some, if not most, of the health animators are exaggerating the figures (a conclusion with which at least one District Health Officer and all of the project supervisors concur). Since the project does not have a system in place to verify this information, there is no way to concretely confirm the level of effort of these CHWs. The project uses its community-based health information system to track behavior change and LQAS studies, the last of which was conducted from July to September 2003. The health information system is useful in that it shows the number of pregnant women being transferred to a health facility each month and the number of mosquito nets sold. It also shows attendance at CBNP sites and the percentage of malnourished children. Despite the availability of this data, however, it is not evident that this data is reviewed and analyzed by project staff or MOH authorities and used to make strategic decisions. For example, the District Health Teams interviewed for this evaluation did not know the CNGP coverage in their districts (nor did some of the IRC supervisors) or the percentage of malnutrition. The LQAS results for the entire target zone were just being completed during the mid￾term evaluation and plans were being made to discuss the results with each of the district health teams. In order for the results to be useful, however, both parties need to examine them critically with an eye to improving or modifying the strategies. 4. Capacity Building Approach i. Strengthening the PVO Organization In the original project start-up proposal the following PVO strengthening objectives were cited: ÿ Establish institutional child survival expertise and resources at IRC’s headquarters; 27 ÿ Strengthen the child survival expertise of field staff implementing, managing and backstopping the CSP Related to the first objective, the IRC’s current technical advisor reported that the grant enabled IRC to hire a Senior Child Survival Technical Advisor in December 1999. The grant has enabled IRC to build up its library of child survival resource materials, which would be more useful to field staff if it was indexed. The resource library in the Kibungo child survival office is impressive with many books and reports in both English and French. IRC has actively used its experience on this child survival project to inform its other IRC health programs. For example, the CS technical advisor and the project technical advisor recently made a presentation about the LQAS at a IRC Health Coordinator’s conference and one of the projects District Medical Officers made a presentation at a Roll Back Malaria NGO forum hosted by IRC in New York. The CS Technical Advisor reports drawing on his CS experience to backstop and develop other CS programs for IRC. Associated with the second objective, Project staff have been trained in PLA techniques and KPC and LQAS organization and implementation. Project staff went for training in Hearth just following the mid-term evaluation (but not as a result of this evaluation). Project staff have not gained skill in health facility assessments, as originally planned. This is in part because a European Union (EU) project was focusing on up-grading health facility infrastructure and staff capacity and the DMOs at the time preferred the terms of collaboration that the EU partnership could provide over those of IRC. Now that the EU project has ended, and the need for capacity building remains, IRC is looking into a more participatory and focused approach to assessing the provision of quality services. IRC does not use any particular tools or approaches to assess it’s own capacity. Rather the CS Technical Advisor tracks the indicators associated with the two capacity building objectives cited above and discusses these with his supervisor during thrice annual meetings. The activities IRC (particularly the Technical Advisor) has undertaken to build his own capacity with regard to child survival include: reading articles, training others in HIS development and analysis and LQAS, giving presentations, attending workshops, exchanges with other CS grantees, such as Concern, CARE, World Relief etc and field visits. ii. Strengthening Local Partner Organizations The Ministry of Health, as represented by the four District Health Teams (DHT), is the primary partner of the Kibungo Child Survival Project. The staff of the 27 health centers in the Province are also important partners. In recent years responsibility and authority for health care provision has shifted from the national level to the Provincial level (administrative). The Administrative Provincial level allocates funds to the District Medical Office The DHT is responsible for allocating funds to the health facilities in the district and for supervising all health activities including service provision at the health center. The DHT is responsible for training health care 28 providers in the district, and making sure national health policies are followed. They are also responsible for reporting on their activities each month. Although no capacity assessment was conducted at the outset of the project, the DIP clearly states the areas in which the project would help to increase their partner’s capacities. Since all of the district health officers participated in the development of the DIP, presumably the partner capacity building section reflects their desires and understanding of their needs. The areas and the progress to date are shown in the following table. Topics – Training Provided Topic – Training not Provided thus far District Health Teams Manage and analyze information using excel spreadsheets Design and use of supervision checklists Plan and implement LQAS monitoring Regional and District Leaders Manage and analyze information using excel spreadsheet Plan for continuation of child survival interventions after the programs end Plan and use LQAS monitoring English lessons As the above table shows, capacity building at the district level has focused on monitoring and evaluation. Having developed and established a community-based information system, the project trained a district health team member to enter and analyze this data. The extent to which this data is reviewed, evaluated and used to make strategic decisions, however, is not evident. As mentioned previously district and health center staff were trained in LQAS implementation. Following the 2003 LQAS, project staff will discuss the results of the survey with the District Health Teams. Although not a formal training, such discussions do serve to build capacity. A supervision checklist related to the CBNP was developed by the project but neither district health teams, nor health center staff have been trained to use it. No other checklists (prenatal, postnatal or malaria case management) have been developed thus far. These outcomes reflect the project’s own bias and strengths toward monitoring quantity (HIS system and LQAS) rather than quality (facility assessments). The following table shows all of the training provided by the project to MOH staff. 29 Training for Ministry of Health Staff Period # Days Participants # Topic Trainers 6 Regional Administrator 1 Planning for Sustainability, Bamako, Mali Nov. 01 CSTS 6 Regional Medical Officer 1 Roll back Malaria, Kenya WHO Oct. 02 12 HC Chiefs of Kirehe, MOH agent and Dist. LQAS - Muhasi IRC, MOH, Concern Jan. 03 1 District Health Officer -Kirehe 1 Roll back Malaria – NYC, USA WHO/IRC NY March 03 5 Dist. Health Agents & Supervisors 8 Nutrition – Adult Education Robb Davis – Consortium CSP April 03 1 District Health Teams 25 Malaria treatment MOH June 03 10 HC Heads, Dist Health Teams and Dist. Agents 13 LQAS MOH/IRC 11 Same as above dif. Dist. July 03 15 LQAS MOH/IRC 1 HC Heads, Dist Health Teams and Dist. Agent 30 Malaria case management by Health Animators PNLP Aug. 03 9 HC Heads, Dist. Health Teams & Dist. Agents 12 LQAS MON/IRC Sept. 03 10 Same as above diff. Dist. 11 LQAS MOH/IRC TOTAL 77 124 iii. Health Facility Strengthening Although the project DIP proposes to improve the quality of prenatal and postnatal care and malaria case management, no health facility assessment was conducted. This was in part because a European Union project, operational in the Province until 2001, also planned health center strengthening activities. IRC therefore decided to focus its efforts on community level activities, as the table below indicates. Health Center Staff Topics – Training Provided Topics – Training not Provided thus far Compiling, evaluating and forwarding CHW reports Using the (CBNP) supervision checklist Conduct LQAS surveys under district supervision Following basic indicators such as growth monitoring coverage and percentage of underweight Holding monthly meetings with CHWs Provide quality prenatal care 30 Provide quality postnatal care Provide quality care for patients with presumed or confirmed malaria Despite the EU’s efforts, the quality of service delivery at the health center level “was deemed low by the MOH, due to the shortage of trained staff nationwide.”6 In as much as the project is successfully promoting prenatal consultations, facility￾based deliveries and postnatal consultations, it is incumbent upon the project to take action to help improve the quality of obstetric care and malaria case management. iv. Strengthening health worker performance At the time of the mid-term evaluation, the project had focused almost all of its attention on strengthening the capacity of the three types of community health workers: TBAs, nutrition animators and health animators. Two approaches were taken in this endeavor: 1) formal training, and 2) follow-up/supervision by IRC staff and health center staff. The evaluation team sought to assess the effectiveness of the training provided to the CHWs and found that in general their knowledge levels were satisfactory with a few exceptions. TBAs, for example, need to focus more on immediate and exclusive breastfeeding, the importance of TT vaccinations and iron supplementation during pregnancy and taking Vitamin A during the post natal period The project has developed a checklist to assess the performance of the nutrition animators as they conduct monthly community- based growth monitoring sessions. Unfortunately the checklist contains many serious flaws which prevents it from being used to address the gaps in health worker performance. (see Chapter I, Nutrition Intervention for specific details) Other checklists were supposed to have been developed to assess the performance of health center staff with regards to obstetric care and malaria case management, but checklists can not be developed or used according to project and MOH staff until the MOH has developed training curricula and staff trained. v. Training According to the DIP, training activities are at the heart of this child survival project and would be based on an in-depth assessment among project staff, partners and beneficiaries of training needs. The evaluation team could not find evidence of a training needs assessment having been done. The training approach consists of formal training events and then on-the-job training and some refresher courses if necessary. While eight groups were targeted for training: mothers, nutrition animators, health animators, TBAs, health center staff, district staff, regional and district leaders and IRC field Staff, the DIP does not specify how many of each group would be trained. It is therefore difficult to assess progress as it relates to numbers of people trained. 6 Kibungo Child Survival Project Detailed Implementation Plan, pg. 116, Quality of Services 31 The DIP mentions training approximately 600 TBAs and at time of the MTE 652 TBAs had been trained. Project staff indicated that a total of about 800 TBAs should be trained to achieve full coverage, which leaves approximately 148 left to be trained. Four hundred and forty-four (440) health animators and approximately 488 nutrition animators had been trained by October 2003 as shown in the Training Table on the following page. District health team members and IRC project staff members are responsible for training community health workers. This was done in 3-day sessions regardless of the course content. Three days was mentioned in the DIP as the time limit within in which an average trainee could maintain his/her attention. This is probably true if didactic training methods are used and trainees are taught in large numbers. But it is not the case when participatory training methods are employed, communication is enhanced by audio-visual aids and training groups are limited in size. It was very difficult to evaluate the quality of the training provided because complete curricula for the courses offered do not exist. Rather key objectives were determined jointly among the trainers (based on standard MOH-approved courses), training modules assigned to specific trainers and each trainer designed and delivered their own module. Not only does this approach make it difficult to control for training quality (course content and methodology), but it also means that a given training course can not easily be replicated for subsequent groups. It is not clear what expertise as trainers the trainers had prior to being asked to train CHWs, but at least one of the district health team trainers admitted that he had never trained TBAs prior to the project’s experience. By looking at the training objectives, the number of days of training and the number of participants and trainers, it is clear that neither could all of the training content be covered adequately nor was it possible to use participatory methods given the numbers of participants. It is also evident that some of the course content was not directly related (or relevant) to the course objectives. While written pre- and post tests were not administered it is reasonable to conclude from discussions with project staff and district health team members, through questionnaires and observations, that CHWs skills and knowledge have improved as a result of the project’s training and follow-up efforts. Community-based growth monitoring is now conducted monthly in over 100 sites throughout the province, and TBAs are referring more and more pregnant women for pre-natal consultations and facility deliveries. Health animators are selling mosquito nets. All three types of CHWs are collecting demographic and nutritional status statistics and submitting reports about their work in the community. Since none of this was taking place prior to the project, these activities provide clear evidence that the project’s capacity building work among the CHWs has resulted in improved access and improved service delivery. 32 Training Table Date of event # Days Types of Participants # Part. Topic 3 TBAs. 164 Reproductive Health 1 TBA 305 Refresher course RH Pilot Phase 10/99 – 9/01 3 Nutrition Animators 102 Nutrition Totals 7 572 3 TBA HC Kibungo 20 Rep. Health 3 TBA Dist. Rwinkwavu 94 Rep. Health Nov. 2001 3 Nutr. Animators HC Kirehe & Musaza; Health Center Chiefs, Nutr. Service heads 53 Nutrition 1 Health Animators – Rwinkwavu 138 1 Health Animators - Kirehe 61 1 Health Animators -Rwamaganda 38 Use and sale of mosquito nets 3 Nutrition Animators HC Gahara & Mutenderi; Health Center Chiefs, HC Nutrition Services Head 51 Nutrition Dec. 2001 3 Nutrition Animators – HC Munyaga 42 Refresher Nutr. 3 TBA & Health Center staff in charge of Prenatal Consulta. HC - Kirehe 66 Feb. 2002 Rep. Health 1 TBA Assoc. Committee Members Kirehe 35 Management systems Mar. 2002 1 TBAs - Kirehe 189 Refresher course Rep. Health May 2002 3 TBAs - HC Kirehe, Musaza, Nyarubuye 25 Rep. Health June 2002 3 Nutrition Animators – HC Rwi & Cyarubare 50 Nutrition 1 TBAs – HC Nasho 11 Refresher Rep. Health Aug. 2002 3 Nutrition Animators – HC Ruhunda 44 Nutrition Sept. 2002 1 Nutrition Animators – HC Remera & Kibungo 68 Nutrition Nov. 2002 3 TBA – HC Munyaga 27 Rep. Health Dec. 2002 1 Health Animators - Kirehe 178 Malaria Prevention 1 Health Animators - Kirehe 93 Malaria Prevention Feb. 2003 3 Nutrition Animators – HC Rutare, Ndego & Kibungo 90 Nutrition March 2003 1 Nutrition Animators – HC Gahara 51 Refresher Nutrition 1 TBAs – HC Rwinkwavu 25 HIV/AIDS April 2003 3 Nutrition Animators - Kibungo 38 Nutrition 3 Nutrition Animators – HC Kirehe 17 Nutrition May 2003 3 Nutrition Animators – HC Bukora 18 Nutrition June 2003 3 TBA – HC Munyaga 18 Refresher Rep. Health Totals 63 1596 Refresher course participants -311 Total New Individuals Trained 1285 33 5. Sustainability Strategy The project has focused substantially on the sustainability of the project’s activities and each intervention has an objective and corresponding indicators. Forming associations among the different types of CHWs is at the heart of the sustainability plan, as is the fee associated with the CBNP activity and the sale of birthing kits and mosquito nets. Each of these activities produces a profit, which is supposed to be used to replenish supplies (Vitamin A, ORS, Road to Health Card, birthing kit materials, and bed nets) and to be deposited in association bank accounts for use as capital in starting income generation activities. Thus far 13 out of the 25 TBA associations are self sufficient, being able to purchase birthing kit materials from the proceeds of the sales, thereby sustaining the activity. All of the TBA associations, however, have bank accounts, as do many of the animator associations. Although it is one of the sustainability indicators the project does not have a system to track the availability of supplies at CBNP and how these were procured. (The checklist has a space for this, but this data is not collected regularly or entered into a data system or analyzed.) Unfortunately none of the sustainability indicators had targets assigned to them at the time of the mid-term evaluation which makes it impossible to assess progress concretely. Also sustainability activities to date have emphasized the creation of the associations and not on the participation of MOH staff in supervising community health worker activities, such as the CBNP. Health Center staff do attend monthly meetings of CHWs which is another effective way of sustaining interest, but the project should track the number of meetings attended by health facility staff as well as the number of CHWs in attendance. Starting in the second half of the project, more emphasis should be placed on district and health center staff taking responsibility for supervising CHW activities, outside the health facility, as per the stated the indicators. Cross-Cutting Issues – related Recommendations Recommendation 1. Project staff make a greater and more consistent effort to involve community leaders in the project and to more assertively promote participation in the project’s activities through these leaders. 2. Focus more on monitoring the process of project implementation and less on measuring the impact of the project 4. Identify how many of each type of health worker/animator remains to be trained and design a training plan to complete the training; 5. Develop a standard format for module design (or borrow one) and have each trainer follow the format when s/he is designing a module; compile all of the modules into one document, review the document together (all of the trainers) to make sure the course content addresses all of the course objectives (and nothing else). Keep the training curriculum as a record of the course. 6. Collect, design, visual aids to be used during the training; reference training documents to discover more varied participatory training methods, especially ones for illiterate participants. 7. Assign targets for each of the sustainability indicators and set up systems to monitor progress toward them 8. Starting in the second half of the project, more emphasis should be placed on district and health center staff taking responsibility for supervising CHW activities, outside the health facility. 34 Chapter Three - Program Management ____________________________________________________ A. Planning The following people are listed in the DIP as having participated in its development: Dr. Abderrhamane Ba, European Union Representative Mr. BYABAGABO Benjamin, Kibungo Regional Administrator Mr. DUKUZEYEZU Diogene, Rwinkwavu District Administrator Mr. DUKUZUMUREMYI Narcisse, Kirehe District Medical Officer Dr. Emmanuel d’Harcourt, Program Manager, IRC Ms. KAMPIRWA Rachel, Micronutrient Agent, Nutrition Division,MOH– Kigali Mr. MUNANA Charles, Rwamagana District Nutrition Supervisor Dr. Ndeye Fatou Ndiaye, Consultant, Facilitator Mr. NKURIKIYINKA Dominique, Kibungo District Supervisor Dr. RUGUMIRE Guido, Kibungo Regional Medical Officer Mr. Marc Saalfeld, Deputy Director, IRC Rwanda Dr. SAHAHA Jean-Baptiste, Kibungo District Medical Officer Mr. TWAHIRWA Eugene, Assistant Program Manager, IRC Of these, four people are currently directly associated with the project. At present the four District Medical Teams (DMT) and the senior staff of the project are in regular contact and meet periodically to make major decisions about the project. The eight IRC supervisors are seconded to the DMT (2 supervisors per district) and these supervisors attend monthly planning meetings to decide what activities will be undertaken in the district. The DIP provided to the independent evaluator did not contain a life-of-project work plan. However, the fourth Annual Report (actually, the first annual report of the extended project - Oct. 2002 – 2003) did contain a work plan and the table on the following page reflects that work plan. The project objectives have been revised twice since the DIP was written and approved. The objectives and indicators provided in the Third Annual Report (actually the first year of the extended project - Oct. 2001 – Sept. 2002) differ from those provided to the evaluation consultant as the revised DIP indicators. During the evaluation, the project staff did not seem to realize that some changes had been made and yet they are the ones designing and implementing the surveys (LQAS) and collecting the data to track progress toward the indicators. Although it was not confirmed during the in-country part of the evaluation, if the senior staff of the project are confused about the objectives and indicators it would be natural for the partners (District Medical Officers) to be confused as well. Discrepancies between the set of objectives include the omission of the indicators related to malnutrition, which figure in the original DIP and Third Annual report but not in the most recent revised set of indicators. 35 October 2002 – September 2003 Annual Work Plan and Status Activity Status Nutrition 1.1 To increase the proportion of mothers who take vitamin A within a month of delivery To make vitamin A available at community growth-promotion sites Available in some districts, while other districts report having stock outs Refresher training of community nutrition agents regarding new mother vitamin A supplementation No refresher training of nutrition animators was conducted this year Training of new community nutrition agents regarding new mother vitamin A supplementation Several new groups of nutrition animators were trained this year, but it is not clear that their training focused on Vit. A supplementation To continue dissemination of key messages regarding new mother vitamin A supplementation through the use of counseling cards and other means Done - To the extent that individual counseling is done and the counseling cards are used To reinforce health center staff awareness of this activity through regular health center meetings and trainings No training of Health Center staff was done this year To monitor coverage of this activity through health center information and LQAS surveys Done – LQAS conducted in July 2003 1.2 To increase the proportion of children taking vitamin A every 4 months Please see activities for previous objective See above 1.3 To decrease the proportion of underweight children To continue to expand geographically community nutrition program activities Done – several new CBNP site were open this year To continue to supervise community growth promotion activities to improve quality While supervision is done, it is not done systematically, nor frequently enough, nor are the checklists used effectively To continue to expand the scope of growth promotion activities to include cooking demonstrations and home visits Cook demonstrations began this year at several CBNP sites; not sure about home visits To train IRC staff, district staff, and community agents in pilot Hearth zone Not Done - IRC staff went for training in Hearth from Oct. 6 – 10 attending a course offered by another PVO in Rwanda To expand use of Hearth techniques to rehabilitate malnourished children within the community and to promote community best-practices Not Done Distribution of vitamin A, mebendazole, and ORS When available these are distributed at CBNP sites Implement a tracking system to evaluate the effectiveness of transfers from community growth-promotion clinics to nutritional rehabilitation centers Not Done 1.4 To increase the proportion of children breastfeeding within an hour of delivery To integrate this key message into monthly TBA meetings Not Done To integrate this key message into training for new TBAs This message is in the TBA curriculum but is not emphasized enough To introduce appropriate IEC messages into health animator activities through regular meetings Not clear that this is being done consistently 36 Activity Status To reinforce health center staff awareness of this activity through regular health center meetings and post-natal training No post natal training for Health Center staff has taken place, because the project understood that it had to wait for the MOH (central) to finalize the training curriculum To monitor coverage through health center information and LQAS surveys Done – LQAS conducted in July 2003 1.5 To increase the proportion of mothers who take iron for at least a month during pregnancy Training of Trainers in new prenatal consultation methodology Not Done – project is waiting for MOH (central) to finalize training curriculum Health center training on prenatal consultations See above To organize community-based distribution of iron by TBAs Not done - not allowed by MOH To integrate this key message into monthly TBA meetings Not applicable To increase availability of iron at the health center level through creating a regular ordering process Not done To monitor coverage and reinforce health center staff awareness of this activity through LQAS surveys Done – LQAS completed in July 2003 Maternal and newborn health 2.1 The proportion of mothers with at least 2 recorded TT immunizations for last pregnancy Health center training on prenatal consultations NO Done – waiting for MOH (central) to finalize curriculum To integrate this key message into monthly TBA meetings Not Done To revise the system currently used to record TT immunizations and other important prenatal information Not Done – waiting on MOH approval To integrate message into health animator activities through regular meetings Not Done Feedback on coverage from health center manager meetings Monthly meetings are held; staff report that TT coverage was discussed 2.2 To increase obstetric transfers from community to health facility, as measured by TBAs To monitor progress and identify problem areas through the TBA health information system IRC monitors progress through monthly reports; but some maternal deaths went uninvestigated To discuss problems and solutions through monthly TBA meetings Presumed done, but no notes are taken regarding the discussions held during these meetings. To discuss problems and solutions through monthly health center manager meetings Presumed done, but no notes are taken regarding the discussions held during these meetings Promote key messages regarding appropriate transfers at monthly meetings and TBA trainings Presumed done, but no notes are taken regarding the discussions held during these meetings Extension of transfer tracking system to other health centers Not sure Disseminate IEC messages on danger signs during delivery through TBAs, community nutrition activities, and health animators These messages were contained in the initial training of TBAs; don’t know if any follow-up was done 2.3 To increase the number of cesarean sections done in hospitals Activities for 2.2, plus 37 Activity Status To expand the transfer tracking system to insure that transferred women reach health facilities In progress 2.4 Improved quality of reproductive health care at health centers Training of Trainers in new prenatal and post natal consultation methodology Not done Training health center staff in goal-oriented prenatal care Not done Training health center staff in post-natal care Not done Malaria control 3.1 Number of deaths from febrile illness, per year, per 1,000 children under 5 To track indicator per health center zone, and identify problem areas with health animators, health center staff, and district staff Done in one district, Kireche Train health animators in recognition of malaria, pneumonia, and other danger signs Done Community disseminated IEC on recognition of signs of the seriously ill child Animators report conducting health talks but project staff doubt frequency of this activity Develop transfer tracking system from the community to health center level Not Done Activities under 3.2 below 3.2 Proportion of children who slept under a correctly impregnated mosquito net last night Sale of mosquito nets by health animators On-going – but sales are slow Training of health animators on retreatment of bednets Not done Partner with the National Malaria Program to increase the number of pregnant women sleeping under correctly treated mosquito nets Partnering with PNLP but concerning community￾based treatment of malaria Support districts to develop systems to insure a regular supply of bednets Not done 3.3 Improved quality of malaria case management at the health centers Development of a monitoring system to track quality indicators at health facilities Not Done Other activities Development of a list of emergency signs to trigger outbreak alert for meningitis or cholera Not Done Training for health animators regarding the list Not Done LQAS survey in Rwinkwavu Distrct Done – July 2003 Follow-up LQAS survey in Kirehe District Done – July 2003 LQAS survey in Kibungo District Done – July 2003 38 Activity Status LQAS survey in Rwamagana District Done – July 2003 To develop a proposal for a family planning intervention in Kirehe District Conduct research to evaluate the effectiveness of the “target￾weight” method Not Done Data from the community-based information system is collected and compiled by community health workers and health center staff respectively. It is then provided to the District Health Officer for inclusion into their database. It is not clear to what extent the DMTs use this data on their own to make planning and/or program implementation decisions. The DMT and health center staff helped to conduct the most recent LQAS and the results have been discussed with the district medical teams. These results will enable the project and the partners to make programmatic decisions. B. Staff Training The project took an unconventional approach to staff training. Rather than providing in￾depth training to all of its field staff at once, the 8 supervisors were initially sent to their respective districts and spent time at health centers learning on-the-job. This was necessary since the MOH did not allow the project to hire supervisors with medical backgrounds (nurses, for example) and hence the supervisors needed to learn from scratch how the health service system worked. Later on in the project, as different interventions began to be implemented, brief formal training events were organized for the field staff or training opportunities were identified outside the project. In addition to this, from time to time senior staff organized 1 or 2-day training courses on the weekend for supervisors. Further to this, each week the entire project staff meets together to plan the following week’s activities and at times part of this 1-day staff meeting is taken to cover areas of weaknesses or to reinforce certain technical themes. It is not clear whether this approach to training is better or worse than the conventional method. Certainly the periodical refresher courses, or opportunities to correct or reinforce knowledge and/or skills is beneficial assuming the topic correspond to real needs or weaknesses. Senior staff explain that in-service training topics are chosen according to needs observed or by problems brought up by the staff themselves. This approach may have it’s advantages but it must have some flaws since the evaluation uncovered many problems, including checklists incorrectly completed and inadequate supervision of CBNP, that have gone uncorrected. Project senior staff have expressed concern about the level of competence of some of the field staff. As a result, a competence questionnaire was designed for the evaluation and completed by the 8 supervisors. Although the questionnaire was quite elementary, it shows areas of weakness among some of the field staff especially in the areas of reproductive health and individual counseling during CBNP. At this point senior staff should decide whether it is worth investing more time in training/supervising these 39 weaker staff members, or whether it would be more expedient at this time to hire field staff with a health profile. The following table shows the training provided to staff thus far. Project Staff Training Date # Days Participants # Topic Trainers 6 National Health Coordinator 2 Planning for sustainability, Bamak0 Nov. CSTS 01 6 Program Coordinator and Regional Medical Officer 2 Roll back Malaria, Kenya WHO 5 IRC Supervisors 1 Behavior Change Planning - RSA Feb. CORE 02 1 IRC supervisors 8 Nutrition National Coord. CSP Coord. 2 National Coordinator 1 IMCI MOH/WHO/UNI CEG 5 Program Coord. 1 Annual Health Coord. Conference IRC NY Mar 02 1 IRC Supervisors 9 Rep. Health IRC/CSP 1 IRC Supervisors 9 PLA IRC youth deve. Prog. Agent Apr 02 1 IRC Supervisors Marketing and treatment of mosquito nets PSI Rwanda 1 IRC Supervisors 11 Family Planning Prog. Coord 1 IRC supervisors 8 Behavior change planning Supervisor of CHWs May 02 4 IRC Supervisors 5 TOT in IEC MOH 4 IRC Supervisors 4 Community based Nutrition MOH/UNICEF/I RC June 02 5 Reproductive Health Manager - IRC 1 Peer Educators in HIV/AIDS PNLP July 02 0.5 Project Staff 3 GPS Training IRC Consultant 15 IRC Supervisors 4 Driving Lessons – driver’s permit Driving School Kigali Aug. 02 1 National Coord. 1 finance IRC Sept. 02 2 IRC Supervisors 1 Lessons Learned and best practices - Kenya Consortium – Urgent care and protection of children Oct. 02 15 IRC Supervisors 7 Drivers license training Kigali Feb. 03 6 Reproductive Health Manager - IRC 1 Health, sexual and repro. Rights of African women and girls Mar. 03 1 Project Staff 2 DELTA Method Dist Health Officer and IRC 40 Date # Days Participants # Topic Trainers 4 Health Coordinator 1 IRC orientation/ Training NY IRC NY 1 Program Assistant 1 Logistics Planning IRC Log. Manager 1 Program Advisor 1 Malaria Treatment MOH Ap. 03 2 Project Staff 10 Protection Training IRC NY & Tanzania May 3 Program Advisor 1 Quality Assurance MOH 03 6 Program Advisor 1 Qualitative Data Collection and Analysis Methods IRC Health Unit NY 3 Program Coordinator 1 Strategy for Vit. A deficiencies Jun MOH/UNICEF 03 1 Project Staff 10 Partner Collaboration Kirehe District Health Center Chief July 03 1 Reproductive Health Manager 1 Establishing health cooperatives MOH/MINALOC Total 90.5 108 C. Supervision of Program Staff The supervision of staff is as follows: The Social Programs Coordinator located in Kigali supervises the National Program Coordinator (NPC) and the Project Advisor (PA). The NPC and the PA supervise the Reproductive Health Coordinator and the Community Health Coordinator (located in Kibungo) and the 8 supervisors (located in the four districts). While formal performance appraisals are conducted annually, aside from the semi annual performance appraisals, the system for supervising staff is quite unstructured and informal. There are no standard procedures for supervision or forms/checklists to be completed. Rather the two senior staff (NPC and PA) observe staff members on an ad hoc basis and provide feedback to each informally. If and when problems arise or weaknesses are identified they are brought up and dealt with at weekly staff meetings. During the eight months, the Project Advisor has taken over primary responsibility for staff supervision, as the NPC has been chronically ill and unable to do field work. The Reproductive Health and Community Health Coordinators provide support to the 8 supervisors, but they are not the official supervisors of the field-based staff. Supervision at all levels needs to be more formalized so that staff get regular feedback regarding their work and official records can be kept. This is especially important given the high turn over in the PA position. Checklists should be developed that correspond to the specific tasks of each position and official supervisory visits should be made to field-based staff every 3-4 months. When writing performance appraisals reference should be made to the completed supervisory forms. At present the numbers, roles and workload of the personnel are adequate to ensure proper supervision (if the system itself is formalized). However when the present Project 41 Advisor’s contract expires in December, and if she is replaced by a Kigali-based Project Advisor, then another supervisory scheme will need to be developed. This will be particularly important if the present NPC continues to focus on office-based planning and administrative tasks. With a little training, the present Reproductive Health and Community Health Coordinators could supervise the field staff 2 – 3 times per year and the new Project Advisor could directly supervise each field worker once per year. D. Human Resources and Staff Management One major challenge was designed into this project which makes it very difficult to ensure the quality of the project’s interventions. Unlike the other child survival projects subsequently funded and being implemented in Rwanda, the Kibungo Child Survival Project is being implemented not in one district but in an entire Province composed of 4 districts. The decision was taken because the Regional Health Officer at the time did not want to favor one district over another and presumably the IRC staff at the time did not fully understand that they would be spreading themselves too thin by targeting the entire Province. To compound the problem, there are only 2 project field agents (supervisors) per district making a ratio of about 200 +/- CHWs per project supervisor and from 27 – 32 growth monitoring sessions per district per month. Without a doubt, the ratio of supervisor to CHW is far too great for the project supervisors to provide adequate supervision and control for quality performance, let alone to be involved in other activities such as training, conducting hearth sessions, attending monthly meetings, checking on data collection and reporting. The problem is a design flaw, which at this point has no easy solution. Since the number of supervisors can not be increased due to budget constraints nor the intervention area reduced, the only other option is to reduce the workload of the IRC supervisors and create more work time. The former can be achieved by shifting some of the supervisor’s responsibilities on to the health center staff and district health teams (where it needs to be for sustainability purposes anyway), and by identifying and focusing attention on the weakest health centers and CHWs. The IRC supervisors’ tasks can be reduced by training Health Center and District Health Team members to use the (revised) CBNP checklist and by dividing up the monthly supervision responsibilities. To assess the level of competence of the CBNP sites/nutrition animators, an initial round of supervision should be conducted by project, health center and district medical team staff of all of the CBNP sites, and a baseline score determined. Through this effort, all of the CBNPs in a given district would be categorized according to their performance (excellent, satisfactory, inadequate). Initially health center staff and district medical teams can be designated the stronger CBNP sites which would need less frequent supervisory visits. This will enable the IRC supervisors to focus their attention on the weaker nutrition animators and to reinforce their skills through more focused supervision. Gradually all of the supervisory responsibilities would be shifted to the health center staff and district medical teams as part of the sustainability plan. 42 Another mechanism to address this problem would be to create more time in the field for the IRC supervisors. Presently all of the supervisors come to Kibungo each Monday to attend a staff meeting (and, on occasion, in-service training). This effectively takes them away from their fieldwork four days per month. By reducing the staff meetings (that include supervisors) to one day per month, all of the supervisors would have 3 extra days in the field, for a total of 24 man-days per month. Not only is this an effective way to increase the supervisor’s time in the field, but it would make the supervisors more efficient by having to plan by the month (as the DMT does) instead of by the week. At the weekly staff meetings, the supervisors describe their program for the week and each of the four senior staff members takes notes in their personal agendas. It is based on this program that senior staff decide when to visit supervisors in the field. Although notes of the meetings are recorded in a “public” notebook, this mechanism does not allow senior staff to readily and handily check the planned activities of a given supervisor with the completed activities of the same supervisor, or to easily compare what the supervisor did the week before, with what s/he plans to do this week. A system that requires each supervisor to write out and submit his/her schedule for the month and which allows that same supervisor to compare what they actually did accomplish the month before, and to explain any discrepancies, would be a more effective and efficient way to monitor and supervise the work of the supervisors Unlike most Child Survival projects, in the Kibungo Child Survival Project staff turn over has not been particularly high among the field-based supervisors. Rather the project has had two different Project Coordinators and three different Project Advisors. The current Project Coordinator is a Rwanda National who was promoted from the Assistant Project Coordinator position. He has been with the project since 2001. He is assisted technically by a Project Advisor, a Uganda physician, who was hired on a one-year contract, in Dec. 2002. The turn over in these two staff positions has caused a lack of continuity and an inability at times to interface effectively with project partners. Communication among staff and between staff and partners was also severely hampered by the fact that the second Project Advisor did not speak French. The current Project Advisor does not speak French either, but her ability to communicate in Kinyarwanda, the national language, allows her to communicate effectively with most everyone related to the project, including CHWs, a real asset. Needless to say, any international IRC staff hired to work in Rwanda should be fluent in French and/or Kinyarwanda and English. Other staff changes are due to take place in the months ahead. The contracts of three project employees, the Social Programs Coordinator, the Project Advisor and the Community Health Coordinator, will come to an end in December 2003 and due to financial constraints, not all can be replaced. The role of the Social Programs Coordinator vis-à-vis the Child Survival Project, and part of the role of the Project Advisor could be assumed by the present HIV/AIDS Advisor, who is a medical doctor working for IRC and based in Kigali. 43 In a region of the world where HIV infection can be as high as 40% ( it is currently 11% in Rwanda), it is a reality of PVOs that some staff members will become infected with the HIV virus. The Kibungo Child Survival Project is no exception. PVOs need to have a HIV/AIDS in the workplace policy in place and it should be discussed openly with project staff. Staff should be encouraged to be tested and receive counseling and to share their status with senior management so that appropriate support can be provided. E. Financial Management Until recently the system necessary to provide the Social Programs Coordinator and Project Manager with the required financial management information was not in place, making it extremely difficult to effectively track project expenditures. The Social Programs Coordinator and Project Coordinator were not privy to the information they needed to effectively track project expenditures. Some unilateral decisions were made, such as increasing staff salaries to bring them in line with a newly established IRC/Rwanda standard and charging some costs to the project which were not initially anticipated, that were detrimental to the project’s financial well being. Fortunately the new Finance Controller of IRC/Rwanda has introduced more stringent, but open and collaborative financial management processes and is working with the project staff to improve systems and plan for the future. Despite these improvements, as of August 31, 2003, 62% of the project funds have been spent, and at the monthly rate of expenditure of about $30,000 there appears to be only enough funds to sustain the project through the next fiscal year. To address this issue, it is strongly recommended that the IRC/Rwanda Finance Controller participate in the development of the action plan (that will be developed as a follow on to this evaluation process). As activities are being planned she can be associating the costs tied to each activity, so that programming and budgeting can be done simultaneously. It is also recommended that IRC headquarters seek out additional matching funds that can be used to support the operational costs of the project. F. Logistics The Project Manager reports that some of the motorcycles purchased during the first phase of the project are in bad condition and often in need of repair. He feels that new motorcycles will need to be purchased during the second half of the project. G. Information Management Discussion of the Health Information System can be found in Chapter Two, Cross￾Cutting Issues, point # 2. H. Technical and Administrative Support To date the project has received outside technical assistance on four different occasions. Technical assistance was provided in the development and writing of the 44 DIP, assistance was provided to learn about the LQAS process, a reproductive health specialist providing recommendations regarding the reproductive health component of the project and an outside consultant was engaged to facilitate the mid-term evaluation. The project anticipated having to hire a consultant to help initiate the Hearth Approach, but had difficulty finding the right person, and generating interest among the DMTs in the approach. The need for a consultant may not be necessary at this point as the staff recently attended a workshop on Hearth organized by another child survival PVO in Rwanda. The DIP also mentions engaging a consultant to help develop tools and a methodology to assess the quality of care at the health centers. If the District Health Teams favor such an activity, which is highly recommended, then the services of a QA specialist may be necessary. Alternatively, the project could work with the Quality Assurance Project, which is operational in Rwanda. An outside consultant will be required to facilitate the final evaluation of the project. Unlike most child survival projects, this project has an unusually high number of senior advisors. A Child Survival Technical Advisor, based in New York, spends 33% of his time backstopping this project and who visits the project twice per year. As the first Project Manager and co-author of the project proposal, he is very knowledgeable about the project and close to many of the project staff. This level of attachment to a project has its advantages and disadvantages. On the one his historical memory and understanding of the project makes it possible for him to present the project to outsiders comprehensively. On the other hand, this closeness may prevent him from taking a wider more objective perspective of the project. The Social Programs Coordinator who is based in Kigali and maintains weekly contact with the Project Coordinator and Project Advisor spend 15 % of her time supporting the project. Until recently she visited the project in-person once every 2 – 3 months, attending monthly staff meetings and occasionally making field visits. The Social Programs Coordinator’s support is more administrative and programmatic while the headquarters Advisor provides more technical input. In addition to these two support positions, the project’s staff includes a Project Advisor, based in Kibungo. As mentioned previously this person is an expatriate medical doctor whose job it is to ensure the technical quality of the project. Since her employment however, she has often served as the acting Project Coordinator assuming the administrative, logistical and programmatic responsibilities of the project. Management Related Recommendations Recommendation 1. Decide whether it is worth investing more time in training/supervising the weaker staff members, or whether it would be more expedient at this time to hire field staff with a health profile 45 Recommendation 2. Staff Supervision at all levels needs to be more formalized so that staff get regular feedback regarding their work and official records can be kept. Checklists should be developed that correspond to the given position and official supervisory visits should be made to field-based staff every 3-4 months. When writing performance appraisals reference should be made to the completed supervisory forms. 3. Shift responsibility for CBNP supervision to health center and DMT staff 4. Reduce staff meetings (that include supervisors) to once per month 5. Fluency in French or Kinyarwanda, in addition to fluent English should be criteria for hiring international staff to work on the project. 6. PVOs operating in high-prevalence countries need to have a HIV/AIDS in the workplace policy in place and it should be discussed openly with project staff. Staff should be encouraged to be tested and receive counseling and to share their status with senior management so that appropriate support can be provided. 7. The IRC/Rwanda finance controller participates in the development of the action plan (that will be developed as a follow on to this evaluation process). As activities are being planned she can be associating the costs tied to each activity, so that programming and budgeting can be done simultaneously. 8. IRC headquarters seek out matching funds that can be used to support the operational costs of the project 46 Chapter Four – Principal Conclusions and Recommendations ______________________________________________________________________ This chapter presents the more important conclusions and recommendations of the project. Other recommendations are presented in Chapter Two and Chapter Three as they relate specially to interventions, cross-cutting Issues and management. 1. Conclusion: The Project has made a lot of progress in establishing the community￾based nutrition program and there are many real and potential benefits of the strategy. However, it is not as yet having the desired effect on reducing malnutrition in the project zone, and by some accounts does not appear likely to result in reduced levels of malnutrition. Recommendation: a) train project staff in the Hearth Method; b) train ECD in each district in the Hearth Method; c) implement the Hearth Method in 1 pilot site in each of the 4 districts; d) monitor results/ study and disseminate results; e) based on results, (f) decide whether or not to continue with the Hearth Method. 2. Conclusion: The ratio of supervisors to CHW and to the size of the intervention area is too great for the Supervisors to adequately monitor the current community-based activities. The CBNP supervision checklist does not enable the project to classify the CBNP sites by performance, and therefore justify concentrating their capacity-building efforts on fewer sites, thereby reducing their workload. Recommendation: a) IRC and DMT review and revise the CBNP checklist according to MTE report suggestions; b) IRC/DMT train health center and district staff in it’s use; c) together IRC/DMT/HC make a supervision schedule that will permit them to supervise (assess performance) all of the sites twice; d) staff supervise all of the CBGM sites and evaluate their performance using the checklist; d) using the checklist, classify all of the CBGM sites into categories according to performance; e) IRC/ DMT/ HC divide up supervisory responsibilities that enables the IRC supervisors to concentrate their efforts on those CBGM site that need the most assistance. 3. Conclusion: Knowledge among some CHWs on certain subjects is not adequate. Opportunities to strengthen their knowledge exist in the monthly meetings. Recommendation: The health center staff presents health topics during all monthly meetings – TBA, nutrition animators and Health Animators. The activity will not only increase knowledge, but will model appropriate IEC approaches. Health Center staff should use visual aids during these times to help communicate the key messages. 4. Conclusion: Opportunities are being missed to strengthen the skills of the TBAs in pre- post natal consultations, while waiting for the MOH to finalize the training curriculum and forms related to pre- and post natal services. 47 Recommendation: While waiting for the MOH to finalize the curriculum and the forms, the project will finish it’s TBA supervision checklist and will begin supervising the TBA’s participation in pre-natal consultations, beginning with the IEC component. 5. Conclusion: Obstetrical services at the health centers in Kibungo Province are not up to standard. Giving birth at a health center does not guarantee quality care. 6. Recommendation: In collaboration with the Quality Assurance Project, project staff and the District Health Team should implement a facility assessment focusing on obstetrical care. Using these results, the project and DHT should organize in-service training opportunities for the weakest health centers. Checklists for pre- and postnatal service should be developed (following the design of the revised CBNP checklist) by the project and DMT and pilot tested. 7. Conclusion: Even though the relationship between the MOH and the project is very strong and even exemplary, this close rapport has its advantages and disadvantages. The MOH does not appear to appreciate that one of the disadvantages of working with an NGO is that the NGO’s resources and time are limited. One risks not achieving the project objectives, for example, if one needs to wait for the Ministry’s permission to develop or make any changes to a curriculum. One also risks not taking advantage of the benefits of working with an NGO, that is, their ability to test new strategies in a limited intervention area. Recommendation: Share MTE results with MOH authorities pointing out where collaboration with the MOH has facilitated or improved achievements and where it has delayed or compromised the quality of certain activities. Discuss with authorities how the delays can be minimized during the second half of the project so that the project objectives can be met; such as allowing the Project to pilot test certain activities. For example, TBA distribution of iron; prenatal and post natal data collection forms, supervisory checklists, and the Hearth Model. 48 Attachments A. Baseline Information from the DIP Please See Chapter Two, Results Table B. Team members and their titles IRC Employees Bonnie Kittle Independent Consultant Eugene Twahirwa National Program Coordinator Doreen Gihango Project Advisor Annick Uwajeneza Reproductive Health Coordinator Consolé Uwibambe Community Health Coordinator Isreal Seminega Supervisor – Rwamagunda - District There Mujwamaliya Supervisor- Rwamagunda District Camarade Ruganza Supervisor – Kibungo Berthe Mukanyiligira Supervisor – Kibungo Samuel Migisha Supervisor – Kirehe Beata Numupfasoni Supervisor – Kirehe Ildephonse Nizeyimana Supervisor – Rwinkwavu Leonidas Nshogozabahizi Supervisor - Rwinkwavu 49 C. Assessment Methodologies The mid-term evaluation was implemented in three stages between September 15, 2003 and November 12, 2003. During this time 3 days were spent reviewing project documents and planning the in-country segment, 2.5 weeks were spent in-country at the project site and 7 days were spent writing the report and then finalizing it. Between writing the first draft and finalizing the report, three weeks passed so that the draft could be translated and reviewed by project staff and their comments considered in the final version of the report. The in-country segment of the evaluation was spent interviewing each project staff member and as many partner representatives as time and availability would allow. In￾country activities such as site visits were hampered by two holidays and national elections. The evaluation team, made up of 7 staff members, one MOH representative (who participated for two days) and an external consultant, interviewed beneficiaries, project participants and partners and also observed some of the project activities. Structured individual questionnaires were used for these interviews. In addition the external consultant reviewed many project documents, tools, supervision checklists, training curricula, trip reports, TA reports, quarterly and annual reports and various other written documentation about the project. Following the field work, the evaluation team tabulated the questionnaires and analyzed the findings. Together the team developed preliminary findings, conclusions and recommendations, which were shared with the project staff, partners and other stakeholders prior to the departure of the external consultant. A second debriefing was held in Kigali, which was attended by IRC/Kigali staff, USAID and MOH representatives. The final report was written by the external consultant with input from the evaluation team, IRC headquarters staff and project partners. 50 D. List of People Interviewed and contacted IRC 12 Child Survival Project Staff – Names and Titles listed in Attachment B Delphine Perault – IRC/Rwanda/Social Programs Coordinator/Acting Country Director Aime Cruz - IRC/Rwanda, Finance Controller Emmanual d’Harcourt – Child Survival Program Advisor, IRC/NY Ministry of Health Nutrition Department Head - MOH Rachel KAMPIRWA - Rwinkwavu District Health Team and 6 Health Center Managers Placide Musafiri – District Health Officer Edouard Muhima – Administrator Jo Muslennyuana – HC Manager Silas Nuyongenzi – HC Manager Abulkarim Rukunado – HC Manager Justine Rose - HC Staff Eugenie Mukamusoni - HC Staff Etienne Niyonsenga – HC Manager Kibungo District Health Team Edison Twagiranungu – District Health Officer Jeanine Uwera - Administrator Joseph Uwihanganye Supervisor Dominique Nkurikijinku Supervisor HC - Staff Kireche District Health Officer and one Supervisor Narcisse Dukuzumuremyi – District Health Officer - 7 HC Staff 47 Traditional Birth Attendants 38 Nutrition Animators 31 Health Animators 51 E. Example revised CBNP Checklist Kibungo Child Survival Program Community-Based Nutrition Program Supervision Checklist CBNP Site: _____________________Date of Observation: ___________ Quality Rating: ____________ Name of Supervisor: ___________________________________________________________________ Name of Health District _______________________Name of Health Center ___________________ Names of Nutrition Animators present: _____________________________________________________ ____________________________________________________________________________________ Instructions: Study the observation questions so you know them quite well. Observe an entire CBNP from beginning to end giving time to observe each animator individually. Observation includes examining a selection of Road to Health Cards and the register to see if they were completed correctly. Listen and look without comment. Put an X under the “yes” column if the answer to the question is yes. Put an X in the “no” column if the answer to the question is no. Remember that some of the questions say “most” or “always”. When the session is over, count up the number of Xs in the yes column and put that number in the space marked Total Score. Do not count up the number of “No”. Compare the total score with the Qualitative Rating and circle the rating that applies. Discuss your findings with the animators and then write your comments in the Comments section. Ask the animators to sign the form and sign it yourself. Classify this checklist with others from the same site, so that it can easily be compared with subsequent forms. SCORE Observation Questions Yes (1) No (0) Welcome 1. Do the animators greet the mothers in a friendly way when the mothers arrive? 2. Have the animators arranged for the mothers to sit comfortably (out of the sun, adequate space) Collective Health Education 3. Is the theme of the talk appropriate for all the mothers present, not just a limited number? 4. Does the collective health talk take less than 15 minutes? 5. Does the animator ask probing questions to get mothers to participate? 6. Does the animator ask the mothers to summarize the main points of the talk? Baby Weighing 7. Does the animator set the scale on “0” with the culottes attached before beginning the weighing? 8. Does the animator ask most of the mothers how much the child should weigh this month? (according to target weight on the card) 9. Does the animator ask most of the mothers to read the weight of the child out loud? 10. Does the animator always read the weight to an accuracy of 100 grams? 11. Does the animator always accurately record the weight of the child on the Road to Health Card? Nutritional Status/Individual Counseling 12. Does the animator always chart the weight on the graph and show it to the mother? 52 SCORE Observation Questions Yes (1) No (0) 13. Does the animator always show the chart to the mother and tell her /ask her what it means? 14. Does the animator always calculate the target weight and tell the mother what it is? 15. For children who have lost weight or failed to gain weight, does the animator always ask open-ended, probing questions to discover the reason? (Why didn’t this child gain weight this month?) 16. For children who have lost weight or failed to gain weight, does the animator always give appropriate advice in a friendly manner? 17. For children who have gained weight does the animator ask about what the child is eating (or confirm exclusive breastfeeding), stages of development according to the child’s age, immunization status, Vit. A status? 18. Does the animator ask every mother if the child slept under an impregnated bed net? 19. Does the animator always accurately record the weight in the register? 20. Does the animator tell the mother when to return for the next weighing? Supplements etc 21. Does an animator collect the CBFP fee from each mother? 22. Is Vitamin A available? 23. Is Vitamin A always given to children who need it? 24. Is Mebendazole available? 25. Is Mebendazole always given to children who need it? 26. Are ORS packets available? 27. Is ORS always given to children who need it? TOTAL SCORE Comments: Qualitative Rating Circle the Quality Rating that corresponds to the Score Above 22 – 27 points Excellent 15 – 21 points Good 9 – 14 points Needs improvement < 9 points Unsatisfactory _________________________________ _________________________________ Signature of Supervisor __________________________________ _________________________________ Signature of Animators 53 F. Example: Individual Work Plan Form Instructions: Write the dates in the spaces provided starting with the first day of the month. At the beginning of the month, fill in the activities planned for the month including where the activity will take place and a timeframe (for example: CBNP supervision – 9:00 –14:00 - Muguzu). Post this Work plan in the office during the current month. At the end of the month put a plus sign (“+”) if the activity was completed satisfactorily and a minus sign (“ – “) if it wasn’t. Work plans should be filed under the supervisors name and district and referred to when subsequent work plans are being completed. Supervisor: _______________________________ Month/Year: _______________ DAY DATE PLANNED ACTIVITY/LOCALE/TIME DONE? Sun. Mon. Tues. Wed. Thurs Fri. Sat. Sun Mon. Tues. Wed. Thurs Fri. Sat. Sun Mon. Tues. Wed. Thurs Fri. Sat. Sun Mon. Tues. Wed. Thurs Fri. Sat. Sun Mon Comments: