Midterm Evaluation of the Kean Svay Extension Child Survival (Follow On) Project August 29 – September 6, 2002 Report Prepared by Marc Debay On behalf of the MTE Team FAO-A-00-00-00038-00 World Vision i Kean Svay Extension Midterm Evaluation Table of contents Executive summary ................................................................................................................... iii I. Assessment of progress towards objectives ........................................................................ 5 A. Technical approaches .................................................................................................. 8 1. Immunization .............................................................................................................. 8 2. Nutrition ...................................................................................................................... 9 3. IMCI .......................................................................................................................... 11 4. Birth spacing ............................................................................................................. 12 B. Cross-cutting approaches .......................................................................................... 14 1. Community mobilization .......................................................................................... 14 2. Behavior change communication .............................................................................. 15 3. Capacity building ...................................................................................................... 16 4. Sustainability ............................................................................................................. 20 C. Program management................................................................................................ 22 1. Planning, monitoring and evaluation ........................................................................ 22 2. Human resources development and management ..................................................... 23 3. Information management .......................................................................................... 25 II. Conclusions and recommendations ....................................................................... 27 1. Conclusions ............................................................................................................... 27 2. Recommendations ..................................................................................................... 27 III. Results highlights .................................................................................................... 29 IV. Action Plan ............................................................................................................... 30 ATTACHMENTS ....................................................................................................................... 45 Attachment A Baseline Information from the DIP ............................................................... 46 Attachment B Terms of Reference for the Midterm Evaluation .......................................... 67 Attachment C List of key project documents ....................................................................... 69 Attachment D MTE team composition ................................................................................. 70 Attachment E Project implementation timeline: October 2000 – August 2002 ................... 73 Attachment F MTE Assessment Methodology .................................................................... 74 Attachment G MTE Results from Field Visits ..................................................................... 76 Attachment H Key Indicators for Leuk Dek and Kampong Thom ...................................... 93 Attachment I Quality of care indicators for health centers, Kean Svay OD ....................... 94 Attachment J Key findings and recommendations from FAR and progress to date ........... 95 Attachment K Capacity Building Plan from FAR and progress to date ............................. 100 Attachment L Sustainability Plan from DIP and progress to date ..................................... 107 World Vision ii Kean Svay Extension Midterm Evaluation Acronyms AD Administrative District ADP Area Development Program ARI Acute Respiratory Infections BCC Behavior Change Communication CBDDS Community-Based Disease and Death Surveillance System CS Child Survival CSTS Child Survival Technical Support DIP Detailed Implementation Plan EPI Expanded Program of Immunization FBC Feedback Committee HCMC Health Center Management Committee HIS Health Information System IMCI Integrated Management of Childhood Illness KPC Knowledge, Practices and Coverage KSCSP Kean Svay Child Survival (Follow On) Project LQAS Lot Quality Assurance Sampling MCH Maternal Child Health MOH Ministry of Health MTE Midterm Evaluation NGO Non-Governmental Organization OCA Organizational Capacity Assessment OD Operational District ORT Oral Rehydration Therapy PVO Private Voluntary Organization RH Reproductive Health TBA Traditional Birth Attendant USAID United States Agency for International Development VHV Village Health Volunteers WV World Vision WVC World Vision Cambodia World Vision iii Kean Svay Extension Midterm Evaluation Executive summary The Kean Svay Child Survival (Follow On) Project (KSCSP; 2000-2003) aims at applying lessons learned from the original Kean Svay Child Survival Project (1996-2000) to other areas in Cambodia, and at building the related capacity of the MOH and WVC. A particular interest is the development a model of integration of child survival into the Area Development Programs (ADP) in Cambodia. The KSCSP is implemented in the three Administrative Districts (AD) of the Kean Svay Operational District, and in three ADs of the Kampong Thom Operational District . The focus is on EPI, nutrition, case management of diarrhea, ARI and dengue hemorrhagic fever, and birth spacing. In each area, the KSCSP provides various types of support at three levels of the health system: training, supervision, and equipment at the health center level; training and organization at the community level; and logistical support for outreach services. In addition, the KSCSP provides this support through the structure of an ADP in two of these areas. The support provided in the four project areas characterizes various models of implementation of child survival. The more intensive model is implemented in the Leuk Dek AD, with support provided at the three levels of the health system and in collaboration with the newly created ADP. In Kean Svay AD, no support is provided to outreach activities but these are being implemented by the MOH. In the Lovea Em AD, the KSCSP provides support to the health centers but not to the outreach activities or to the community. Finally, the KSCSP only provides support to the health centers of the 3 ADs in Kampong Thom, but the ADPs of these 3 ADs have strong health components. This last situation therefore represents a model where minimum external technical inputs are provided to an ADP that otherwise implement health programs. The fieldwork of the Midterm Evaluation (MTE) ran from August 30 to September 6, 2002. The MTE Team adopt the KSCSP methodology for participatory review that involved an intense participation of partners, and consisted primarily of Focus Group Discussions with various project partners and beneficiaries. The MTE Team included 9 WVC and KSCSP staff members and 5 external evaluators. Two larger groups including also various partner representatives prepared and conducted the fieldwork in Leuk Dek (26 participants) and in Kampong Thom (20 participants). The major activities of the KSCSP have been implemented as planned, building on the experience and achievements of the original project. The baseline assessment and the DIP were conducted successfully. All the positions have been filled by qualified and committed staff who is very able to implement and keep good records of all planned activities. Two important consultancies have been successfully completed and led to valuable improvement in the project plans of work. The FAR and the MTE were completed on schedule in a participative manner that brought to light important issues and responses for the success of the project. All the targeted health center staff is being trained regularly in the 3 AD of the KS OD, and the training of health center staff in Kampong Thom has started. The OD staff in Kean Svay and Kampong Thom conducts regularly supervisory visits of the health centers in Kean Svay OD World Vision iv Kean Svay Extension Midterm Evaluation and the 3 ADs of Kampong Thom. All the targeted Village Health Volunteers in Leuk Dek and Kean Svay AD are being trained every month, and regular meetings are held with the health centers. Outreach activities are conducted in Leuk Dek and Kean Svay. In Leuk Dek, the January 2002 KPC survey showed that after about 8 months of implementation at the community level, the project objectives had already been achieved or surpassed for the EPI, case-management of ARI and diarrhea, Vitamin A, and birth spacing components. Results may be less advanced in Kampong Thom. The nutritional status of children under the age of 2 is of concern, with 43% moderately or severely undernourished in the 12-23 age group in the January 2002 Leuk Dek KPC survey. This finding may be related to the low level of exclusive breastfeeding and timely introduction of complementary food. The MTE developed the following main recommendations to WVC and its partners:  Let the KSCSP staff lead the transfer of activities to the MOH and ADP, the design and implementation of the cost effectiveness study, and the preparation of the final evaluation and lessons learned workshop. WVC should provide technical and managerial assistance as needed.  Assess the capacity of health centers staff and VHVs to identify malnourished children and provide nutritional education, and assess the knowledge and practices of mothers of children under 2 regarding breastfeeding and complementary feeding.  Support IMCI orientation and training in the Kean Svay OD, and begin adapting the related training of the HC staff and VHVs.  Prepare formal Exit Plans with the ODs and each HC partners.  Organize a “study design” workshop with selected partners to define specific objectives, data needs, and analysis plan to estimate the cost-effectiveness of the different models of child survival implementation.  Consider conducting the following other studies: evaluation of the performance of the VHVs and its determinants; completeness and coverage of the CBDDS at the village level, and maybe mortality and birth rates; costing of the inputs provided at the health center, outreach and community levels; analysis of the supervision data.  Prepare the end-of-project lessons learned workshop, which may include some of the studies above but also the findings of the final evaluation, the CBDDS, and the issues and plans for integration of child survival into ADPs.  Ensure that all routine or special data is fully used for programmatic purposes or for development of tools and methods.  Develop the participatory review methodology by defining the specific problems to be solved, analyzing the potential sources of biases in the results, and allocating more time to the preparation and analysis stages The MTE Team recommends that USAID consider the KSCSP Follow On as completed as planned, and provides technical assistance for the proposed cost effectiveness study of the models of implementation of child survival. World Vision 5 Kean Svay Extension Midterm Evaluation I. Assessment of progress towards objectives The background information, the project objectives, and the main features of the design of the Kean Svay Child Survival (Follow On) Project described in the Detailed Implementation Plan (DIP) are reproduced in Attachment A. The Kean Svay Child Survival Project (KSCSP)1 aims at applying lessons learned from the original Kean Svay Child Survival Project (1996-2000) to other areas in Cambodia, and at building the related capacity of the MOH and WVC. Of particular interest is the development of a model of integration of child survival into ADPs in Cambodia. The original KSCSP successfully achieved and surpassed its objectives, presumably because of the intensive direct service delivery through mobiles teams of project and health center staff, intensive training of health workers from the community to the level of the Operational District (OD), and good project management. Among the main recommendations of the Final Evaluation was the extension of the project to further build the capacity of the Kean Svay OD, and possibly other ODs, and to sustain and replicate the project interventions in other areas. Project design The Kean Svay OD has a total population of 235,157 inhabitants and comprises three Administrative Districts (AD): Kean Svay, Leuk Dek, and Lovea Em. The KSCSP focuses on the Leuk Dek AD as Direct Impact Area because of its relative isolation and poor health services, and because of the presence of a new ADP and a Microentreprise Development (MED) program. The two other Administrative Districts of Kean Svay and Lovea Em only benefit from the project indirectly, through capacity building of the KS Operational District and training of the Health Centers staff. In addition, three Administrative Districts2 of the Kampong Thom Operational District in Kampong Thom Province also receive indirect support from the project, primarily training of the Health Centers’ staff, because of the presence of more established ADPs with a strong health component. The KSCSP consists in the provision of various types of support at four levels of the health system in the project areas, as summarized below. Level of health system Type of support Operational District Training, technical assistance Health Centers Training, supervision, equipment, referral system Outreach Services Logistics, per diem Community Training, committees, support groups, VHVs, CBDD The support provided in the four project areas characterizes various models of implementation of child survival that the KSCSP proposes to compare. The more intensive model is implemented in the Direct Impact Area of Leuk Dek, with support provided at the four levels of the health 1 In this report, the acronym KSCSP is used to refer to the Kean Svay Child Survival (Follow On) Project, except if specified otherwise. 2 These are Prasath Sambo, Prasath Balang, and Sandan. World Vision 6 Kean Svay Extension Midterm Evaluation system and in collaboration with the newly created Leuk Dek ADP. This model is similar yet less intensive than that implemented in the Kean Svay AD during the original KSCSP. In the Follow On KSCSP, no support is provided to outreach activities in Kean Svay AD, although these are now being implemented by the MOH. The KSCSP keeps providing training to the VHVs in the Kean Svay AD. In the Lovea Em AD, the KSCSP provides support through the OD like in the two other ADs, and support to the health centers but not to the outreach or the community level activities. Finally, the KSCSP provides supports to the health centers of the 3 ADs on the Kampong Thom Province, but no other type of health system support. The ADPs in these 3 ADs have strong health components, however, including one Health Coordinator in each ADP and one Senior Health Operation Manager covering the 3 ADPs. This last situation therefore represents a model where minimum external technical inputs are provided to ADPs that otherwise already implement health programs. The table below summarizes the levels of health system support and other characteristics of the four implementation areas of the KCSP. This presentation of the project or “study design3 ” is used as much as possible in this report to suggest ways to begin collecting relevant data to answer the fundamental question behind the KSCSP of what is the most cost-effective model of integration of child survival into ADPs. The total population, number of villages and number of health centers are provided in the table below as example of information needed to make such comparisons. Characteristics of the four KSCSP implementation areas Kean Svay Operational District Kampong Thom Province Characteristics Leuk Dek AD Kean Svay AD Lovea Em AD 3 ADs Health system support 1. Health Centers Yes Yes Yes Yes 2. Community Yes Yes - - 3. Outreach services Yes - - - ADP presence Yes - - Yes Other characteristics Total population 55,688 123,113 69,081 45,000 Number of villages 24 36 43 78 Number of health centers 4 7 6 5 Timeline The implementation timeline of the KSCSP from October 2000 to August 2002, when the midterm evaluation (MTE) was conducted, is presented in Attachment E. The baseline assessment (KPC surveys and qualitative studies) was conducted in November 2000 and the DIP 3 See Attachment A, Program Design. World Vision 7 Kean Svay Extension Midterm Evaluation workshop held in February 2001. The DIP was reviewed in June and approved in August, 2001, with the following requests: develop a behavior change strategy, a capacity building plan, and a sustainability plan; monitor the nutritional situation of the children in the project; provide an annual update of the status of IMCI implementation at the national level; and include a review of costs, benefits, and losses of the different implementation models at midterm. The First Annual Review (FAR) of the KSCSP was conducted in September 2002 with representatives from the MOH national, provincial, and district levels and from World Vision US, Asia Pacific Regional Office, and Cambodia. All the KSCSP staff was also involved. This extensive review provided many worthwhile recommendations for the implementation of the project, and was an opportunity to respond to the recommendations from the DIP review such as developing a capacity building and a sustainability plan. Two other significant events on the project implementation timeline are the consultancy on Behavioral Change strategy in December 2001, and that on the Community-Based Disease and Death Surveillance System (CBDDS) in April 2002. The training of Health Center staff began in February 2001 for the 3 AD of the Kean Svay OD, and continued on a regular basis until August 2002. Training of the staff of the 7 Health Centers in Kampong Thom was conducted in March and May 2002. Training of the community members began in June 2001 in the Leuk Dek AD. Other community members like TBAs, Drug Sellers, and Traditional Healers attended training events between September 2001 and March 2002. Training of VHVs in the Kean Svay started a little bit later than in Leuk Dek in September 2001 with a one-week training event followed by monthly one-day events in parallel with the training of VHVs in Leuk Dek. The MTE report The methodology of the MTE assessment is described in Attachment E. The following sections present the results of the analyses conducted by the MTE team according to the general outline proposed in the USAID guidelines: technical interventions, cross-cutting approaches, and program management. For each topic within these main categories, the findings are presented as accomplishments, challenges, and recommendations. Section III of the report simply summarizes these findings. The Results Highlights section provides a summary of the three models of integration of child survival into ADPs, and calls for a formal study of their respective cost-effectiveness. World Vision 8 Kean Svay Extension Midterm Evaluation A. Technical approaches The KSCSP DIP proposed about ten technical interventions, detailed in Section III as: immunization, nutrition and micronutrients, breastfeeding promotion, control of diarrheal diseases, pneumonia case management, maternal and newborn care, control of malaria, child spacing, STI/HIV-AIDS prevention and care, and integrated child survival programs and IMCI. This list of interventions is different from that presented in the Executive Summary of the DIP or that in the CGSP Data form of Section B. These various lists of interventions may not be a useful way to categorize program activities and may not represent current activities anymore. In this section, the following categorization of interventions is used: (1) Immunization (2) Nutrition (3) IMCI (4) Birth spacing Attachment H presents selected project indicators at baseline (November 2001) and midterm (January 2002) for the Leuk Dek AD, the Direct Impact Area. Both surveys sampled the entire Administrative District. Overall, the January 2002 KPC survey showed that most of the key project objectives had already been achieved or surpassed after about 8 months of implementation of activities at the community level, that is, when the health center staff and village health volunteers had been trained. Attachment H also presents data for the 3 ADs in Kampong Thom available from the baseline household survey conducted by the ADP in February 2000 and from LQAS surveys conducted in August 2002. It is unclear from the available report how the indicators from the baseline ADP survey conducted in February 2000 were calculated. LQAS results are based on two samples of 19 children (38 total) and therefore carry wide confidence intervals. Planning aspects of these various KPC and LQAS surveys are further discussed overall section C (1). 1. Immunization Achievements In Leuk Dek, the percent of completely immunized (card documented)4 children by 12 months of age increased significantly between the baseline and midterm KPC surveys, and actually surpassed the end-of-project objectives (Indicator 1 in Attachment H). The percent of pregnant women who received two doses of tetanus toxoid also increased dramatically and surpassed the objectives (Indicator 2 in Attachment H). The KSCSP contributed to this improvement through the support provided to the health centers and to the outreach activities in particular, through community mobilization and training of VHVs, and through the use of the EPI Tracking form.5 4 All mothers in the sample have immunization cards for their children. 5 See Attachment A, Health Information System. World Vision 9 Kean Svay Extension Midterm Evaluation In Kampong Thom, LQAS results suggest that similar levels of immunization coverage were reached while the baseline levels were much lower than those in Leuk Dek. The contribution of the KSCSP was through the training of the staff in the health centers. Recommendation  The KSCSP should review the available data in the ADP household survey to assess their adequacy as baseline to measure the impact of the project. Secondary analyses should be conducted if necessary for this purpose. This information will be critical to compare the effectiveness of the three models of integration of child survival into ADPs. 2. Nutrition Achievements In Leuk Dek, the coverage of children 6-23 months of age and of mothers of children 0-11 months 2 months after delivery with Vitamin A supplements has increased dramatically (Indicators 3 and 4 in Attachment H). The KSCSP contributed to this improvement through the support provided to the health centers, through community mobilization with VHV and TBA, and through the use of the Vitamin A registers. The knowledge of iodized salt by mothers of children under 2 remained high in Leuk Dek between baseline and midterm KPC surveys (83.8% and 88.9%, respectively), but is lower in Kampong Thom (LQAS data: 19/38=50%). In Leuk Dek, the percent of newborn breastfed within an hour of delivery and the percent exclusively breastfed at age 0-5 months increased between baseline and midterm KPC surveys to reach 30.6% and 26.5%, respectively. Similarly, in Kampong Thom, the percent of newborn breastfed within an hour of delivery may have increased significantly (LQAS data: 30/38=79%). Challenges The LQAS coverage data for Vitamin A supplementation of children 6-23 months for Kampong Thom do not show as high a coverage as in Leuk Dek. This may very well be because cards are not as ready available. Indeed, it is well known by the MOH staff that record of the supplementation is rarely done systematically during the mass distribution events. In fact, information from the MOH suggests that coverage is indeed higher.6 Given the periodic nature of the Vitamin A supplementation of children 6-59 months through national bi-annual days, the coverage among children 6-11 months depends on how close to the 6 WVC implements a Nigh Blindness Prevention Project in these 3 ADs. World Vision 10 Kean Svay Extension Midterm Evaluation last distribution day the survey is conducted, and this affects any indicator including that age group. The percent of mothers of children under 2 reporting using iodized salt decreased significantly between the baseline and midterm KPC surveys on Leuk Dek (Indicators 5 in Attachment H). However, LQAS results in Leuk Dek suggest higher levels of reported domestic use of iodized salt (44/76=57.8%). In Kampong Thom, the ADP household survey and the LQAS data suggest lower level of use than in Leuk Dek. This indicator is more difficult to interpret than one reporting the percentage of households in which salt used in the kitchen tested positive for iodine. In Leuk Dek, the percent of children 6-11 months who receive complementary food appears to have remained stable or decreased. Similarly, in Kampong Thom, the percent of children 6-11 months who receive complementary food seems to have remained stable. Following up on a recommendation from the DIP review, the KSCSP conducted an anthropometric survey of children age 0 to 23 months in Leuk Dek as part of the January 2002 KPC survey. The table below shows the percent of children below 2 and 3 standard deviations from the median weight-for-age for boys and girls in three age groups. These data show that boys are more likely to be malnourished at age 6-11 months, but then girls are more likely to be malnourished at age 12 to 23 months. The data also show the high rates of malnutrition after 12 months of age, presumably a consequence of poor complementary feeding practices. While these rates of malnutrition are of concern, they are comparable to those found in the CDHS 2000 for children under five, as shown in the table below. Percent of undernourished children by age group in Leuk Dek (2002) and Cambodia (2000) Age group Indicator1 Leuk Dek KPC 2002 CDHS Boys Girls Both 2000 Sexes 6 – 11 <-2SD 28.6% 11.5% 18.4% 26.9 (95%CI) (15.2-46.5) (4.8-24.1) (11.2-28.4) <-3SD 2.9% 0 1.1% 11.5 (95%CI) (0.1-16.6) (0.1-7.1) 12-23 <-2SD 40.3% 45.3% 42.9% 48.6 (95%CI) (28.3-53.5) (33-58.2) (34.2-52) <-3SD 8.1% 4.7% 6.3% 15.7 (95%CI) (3-18.5) (1.2-14) (3-12.5) 0-23 <-2SD 30% 24.1% 26.8% 40.0 (95%CI) (22.7-38.4) (17.8-31.6) (22-32.3) <-3SD 5.7% 1.9% 3.7% 12.1 (95%CI) (2.7-11.3) (0.5-5.9) (2-6.7) Source: Leuk Dek KPC survey report, January 2002; CDHS 2000 1 Percent below 2 or 3 standard deviations from the median weight-for-age World Vision 11 Kean Svay Extension Midterm Evaluation Recommendations  Given the high levels of undernutrition demonstrated by the Leuk Dek KPC survey, the KSCSP should assess the capacity of health centers staff and the VHVs to identify malnourished children (growth monitoring) and provide adequate nutritional education and counseling.  Given the low levels of timely introduction of complementary food, the KSCSP should assess the related knowledge and practices of mothers of children under 2, and those of the health care providers.  The KSCSP should measure length in future anthropometric surveys to distinguish between acute and chronic malnutrition.  The validity of the indicator of self-reported use of iodized salts should be assessed before using it as a project monitoring and evaluation indicator. 3. IMCI The September 2001 FAR report provides an update on the national IMCI program that indicates that the IMCI adaptation phase was completed and that a core group of trainers were available at that time. The national IMCI training coverage target is at least 2 staff per HC and selected staff at the district, provincial, and national levels. The IMCI home care card was finalized and ready to be reproduced and distributed. Achievements In Leuk Dek, the indicator of case management of diarrhea has increased significantly between the baseline and midterm KPC surveys, and has surpassed the end-of-project objective (Indicators 6 in Attachment H). In Leuk Dek, the percent of mothers of children under 2 who can name at least one danger sign of DHF (cold extremities, bleeding, abdominal pain, and weakness) had increased significantly and surpassed the end-of-project objective of 60% Although the DIP presented interventions ARI and DD as separate interventions, and this because at the time of its preparation the national IMCI was only beginning, the KSCSP is now progressively adopting the IMCI strategy for case management of childhood illnesses. Three WVC staff members have been trained in IMCI. Challenges In Leuk Dek, the indicator of case management of acute respiratory infection may have remained stable between the baseline and midterm KPC surveys (Indicators 7 in Attachment H). The results from observations of clinical practices of the health centers staff during supervision visit by the Kean Svay OD and KSCSP project staff show obvious improvements in compliance World Vision 12 Kean Svay Extension Midterm Evaluation with standard case management procedures at the beginning of the project in Leuk Dek, just after the first training in ARI/DD/DHF, but not Kampong Thom, where no training had taken place.7 The results of observations in March and August in Leuk Dek showed high levels of compliance with the standards but no improvement between these two periods (see Attachment I for these results and section B (3) for comments on the tool and methodology). Recommendations  The KSCSP should support the IMCI orientation and training of the Kean Svay OD staff, and begin adapting the related training of the HC staff and VHVs. This should be done in coordination with the National IMCI unit of the MOH and the OD staff, and by using the 3 WVC staff already trained in IMCI. 4. Birth spacing Achievements In Leuk Dek, modern contraceptive use among mothers of children under 2 who are not pregnant and desire no more children in the next two years (or who are not sure) has increased dramatically between the baseline and the midterm KPC survey, reaching 63.6% and by far surpassing the end-of-project objective of 30% (Indicators 9 in Attachment H). This success is at least partially attributable to the outreach activities and the Women’s Groups supported by the KSCSP. Challenges The sharp increase in modern contraceptive prevalence rate (CPR) among mothers of children under 2 in such a short period is plausible, but raises questions. In a birth spacing perspective, however, it would make more sense to include in the denominator of this indicator8 the number of mothers of a child under 2 who are pregnant or who want another child in the next 2 years since most of them miss the recommended time between births of at least 2 years. Also, some mothers may want another child before 2 years but not now, and therefore they may want to use contraception. Even though the number of these mothers is relatively small (12 and 26 among 287 mothers of child under 2 in the January 2002 sample), their inclusion in the denominator of the CPR indicator decreases its value from 63.6% to 55.5% in the KPC 2002, and from 6.4 to 5.6 in the KPC 2000. Another explanation for the sharp increase in CPR is that the current question of “what is the main method that the couple is using?” may measure “ever use” more that “current use” if the questions is not well understood. 7 See summary table of results in the FAR report pages 37-39. 8 The denominator currently used by the KSCSP follows the KPC 2000 Guidelines World Vision 13 Kean Svay Extension Midterm Evaluation In the Leuk Dek 2002 KPC, the method mix is mainly injection (61%) and pill (33%). This may be the result of informed choice by women, but may also be due to a lack of availability of other methods, a lack of knowledge or skills among providers, or a lack of knowledge among the mothers of children under 2. The KSCSP staff already decided to develop behavioral change strategy to promote men’s involvement in birth spacing practices, and increase acceptance and use of condoms. Recommendations  The KSCSP should introduce a question in the next KPC survey like “What method are you currently using?” and another question to probe the actual protection status (when was the last injection? Are you taking the pill now? Etc). The question used in the KPC 2000 and 2002 should be maintained to make is possible to compare results from one survey to the other, and help understand what each indicator actually measures.  The CPR should include mothers of children under 2 who are pregnant or who want children within 2 years since most of them would in fact benefit from birth spacing.  The KSCSP should explore the opportunities to diversify the contraceptive methods mix available to men and women in the project area.  The KSCSP should identify and document the key determinants of the success of this intervention (high demand, mode of service delivery, community support, etc) before the end of the project. World Vision 14 Kean Svay Extension Midterm Evaluation B. Cross-cutting approaches Given the obvious overlaps between the various sections below, cross-references are made when necessary to minimize repetitions yet provide a complete review of each area. 1. Community mobilization Achievements The KSCSP was able to select, train and maintain a total of 50 Village Health Volunteers in 24 villages in Leuk Dek. Only one selected VHV dropped out of the program in 12 months of project implementation. The KSCSP has also continued training about 70 VHV in 36 villages in the Kean Svay AD. The OD and HC staff has been systematically trained as trainers and then has provided training to VHVs about one day a month since June 2001 in Leuk Dek and since September 2001 in Kean Svay. No training of VHV was provided in Kampong Thom, as planned. The VHV training covers a wide range of topics including EPI, Birth Spacing, ARI, CDD, DHF, IDD, Vitamin A, BF, HIS, and STD. Some of these topics have already been provided a second time as refresher. The training materials were developed by WVC. The KSCSP provided two 1- or 2-day training events other community members like Traditional Birth Attendants, Drug Sellers, and Traditional Healers between September 2001 and March 2002. In Leuk Dek, the KSCSP has been able to strengthen linkages between HCs and the community through assisting in the establishment of 4 Health Center Management Committees (HCMC) and Feedback Committees (FBC), and encouraging VHVs to attend monthly meetings with the HC staff. The VHVs receive a non-financial incentive package that seems appropriate for the nature and amount of the tasks expected from them. The KSCSP created and helped establish 9 Village Development Committees in 3 ADP communes in Kampong Thom. The KSCSP created and helped establish Mothers Groups who are now functioning, and also a few Father’s Groups and Children (school age) Groups. The MTE Focus Group Discussions indicate that community members (mother and fathers) know the VHVs and the various services they provide. The progress in establishing CBDDS is discussed in section C (3). World Vision 15 Kean Svay Extension Midterm Evaluation Challenges The newly formed HCMC and FC are not yet functioning as per the MOH Guidelines,9 and their members still need training. There is no system to monitor the performance of the VHV outside the training environment. Recommendations  The KSCSP should continue supporting the HCMC and FBC to ensure they reach the capacity to operate as per the MOH Guidelines after the end of the project.  The KSCSP should continue supporting the Mother’s Group, and Children Groups.  The KSCSP should consider creating and supporting Men’s/Father’s Groups and Family Groups.  WVC should conduct a formal evaluation of the performance of the VHVs before the end of project to draw lessons about the support environment (HC, HCMC, FBC), training received (methodology, contents, effectiveness), and likely impact on the behavior and health status. This evaluation should be conceived as a component of the cost effectiveness study of the various models of integration of child survival into ADPs. 2. Behavior change communication Achievements The KSCSP supports behavior change communication activities through the trained VHVs and TBA (health education sessions and demonstrations, one-to-one education during home visits, etc), the various women’s and other support groups, and the HC staff. In response to a finding and recommendation of the FAR that there was a lack of IEC materials at the HC and village levels, the KSCSP obtained materials from HKI, UNICEF, and MOH/NCHP. The KSCSP also disseminated the 16 community practices and the IMCI Mother Card. In December 2001, following up on a recommendation from the DIP review, the KSCSP organized a consultancy to develop the behavioral change strategy for the project.10 The consultant guided the project staff and partners through a Behavioral Approach to Program Planning focusing on the 16 key family practices of community IMCI. They applied the various steps of the approach to 5 specific behaviors (birth spacing, immediate and exclusive breastfeeding, weaning and child feeding, home care of diarrhea, and antenatal care). The KSCSP then decided to further develop and implement two specific behavioral strategies: promotion of male involvement in birth spacing practices, and promotion of colostrum feeding. 9 See these Guidelines as attachment to the FAR. World Vision 16 Kean Svay Extension Midterm Evaluation At the time of the MTE, they had conducted formative research, and identified specific activities and developed related communication materials to support behavior change. Challenges The new behavioral change approach developed with the consultant will require formative research and training before each strategy can be applied on a large scale in the project area. The current project staff may not have the time and qualification to proceed with the plan before the end of the project. Recommendations  WVC should ensure that the whole process of developing and implementing at least the two selected behavior change strategies is completed and documented by the end of the project. This will serve to build the capacity of WVC and partners to apply this valuable approach in other projects.  The KSCSP should carefully assess the need for IEC materials for VHVs and HC, presumably, on the basis of well-defined behavior change strategies.  The KSCSP should quantify the behavioral change activities at the community level. 3. Capacity building Following up on a recommendation of the DIP review, the KSCSP developed a capacity building plan during the FAR. Attachment K presents the objectives, indicators and activities of this plan, and the progress to August 2002. This plan makes the distinction between activities to build the capacity of WV Cambodia and the MOH, discussed here, and that of Community-based organizations, discussed in section B (1) above. Achievements WV Cambodia The Leuk Dek ADP and KSCSP staff holds regular “Integration Meetings” to discuss current and plan for future integration of child survival activities into the ADP. Minutes of the meetings are available. Several WVC staff members benefited from training in IMCI, LQAS, CBDDS, BCC, training of trainers and grant management. Overall, this training is well appreciated and increases the capacity of WVC staff to integrate CS into ADPs. 10 See consultancy report as an attachment of the FAR report World Vision 17 Kean Svay Extension Midterm Evaluation At the time of the MTE, 8 of the 20 ADPs supported by WV in Cambodia are using child survival tools developed by the KSCSP, and all 20 agreed upon common health status indicators. Ministry of Health The KSCSP has been able to keep up with its training targets in terms of number of trainees, topics, and schedule. Good records of the training activities (nominative lists of target trainees, dates and topics of all training received) are available.11 The quality of these records makes it possible to conduct a formal evaluation of the effectiveness of this training. The KSCSP is training all the HC staff in the 3 ADs of the KS OD, which represents a target group of about 100 health workers. The topics include ARI/CDD/DHF (5 days), Birth Spacing (clinical-3 days, and management-2 days), Vitamin A (1 day), EPI (3 days), Breastfeeding (3 days), IDD (1 day) and HIS (1 day). Some of these topics have already been provided several times as refresher training. The training the HC staff in the Kean Svay OD began in January 2002 and has been provided on a regular basis since then (see Attachment E). The training of the HC staff in Kampong Thom was provided to about 25 participants in March (ARI) and August (Birth Spacing) 2002. Senior KS OD staff participated in advanced training in LQAS, PLA, and Adult Education. A sound supervision system is in place at the KS OD level, whereby OD and project staff visits all health centers on a quarterly basis and use a standardized guideline including a clinical observation checklist, an exit interview questionnaires, and a record review checklist. These supervision data for ARI and diarrhea in all health centers of the KS OD in March and August 2002 show in general relatively high proportion of observations that meet the standards, but no obvious improvement of the individual item indicators between these two assessment periods (see Attachment I; data prepared during the MTE). The supervision data for ARI, diarrhea, and DHF using a three-level score (Good; Fair; Weak) for each observed clinician showed clear improvement between February and April 2001. The same data and presentation did not show clear improvement in health centers of the Kampong Thom OD.12 WVC identified and adapted an Organization Capacity Assessment13 (OCA) tool that comprises a limited set of questions on areas of governance, management practices, human resources, financial resources, service delivery, and sustainability. Each is rated by the organization staff on a 6-level scale between “needs urgent attention” and “no need for immediate attention.” The Kean Svay OD used this tool in January and August 2002, and the Kampong Thom OD in September 2002. No assessment identified any area needing urgent attention or needing 11 See Training Report 2001 and Training Report 2002. 12 See summary table of results in the FAR report pages 37-39. 13 The original tool was developed in 1996 by the Pact Ethiopian NGO Sector Enhancement Initiative and described in a Handbook “Assessing Organizational Capacity Through Participatory Monitoring and Evaluation.” World Vision 18 Kean Svay Extension Midterm Evaluation attention. Results of the two assessments in the KS OD showed improved capacity in nearly all areas. The KSCSP has been able to support intensive outreach activities in the Leuk Dek AD with less direct inputs than in the first phase of the Kean Svay project. At the time of the MTE, almost all outreach activities were managed by HC’s staff. Outreach activities continue in the Kean Svay AD. The KSCSP facilitated changes in the organization and management of HCs to improve hours of operations, teamwork, and staff loyalty. The MTE focus group discussions consistently showed that respondents recognized improvement in important elements of quality of health services (such as being open 24 hours a day, having welcoming and polite staff, availability of drugs, affordability of services, capacity to refer patients when needed) have improved since the beginning of the project, and that as a consequence the utilization of health services has increased. The HCs hold monthly meetings with VHVs. These meetings are critical to maintain the links with the community and ensure motivation and quality of work by the VHVs. The KSCSP provided motorbikes and generators to all four HCs in the Leuk Dek district. Challenges --WV Cambodia Although activities are progressing well in the Leuk Dek AD, the fundamental question behind the KSCSP of how to integrate CS into the ADP program may remain unanswered. It is not clear yet how the ADP staff will actually continue the current activities at the end of the project. Among the various issues is the availability of qualified health staff in the Leuk Dek ADP, and the need for continued funding of the regular training and other support activities. --Ministry of Health The KS OD staff has made good progress in adopting and using systematic approaches to supervision of the HC’s staff, but they do not seem to have the ability to analyze, interpret and develop solutions to the problems identified. The tools used for supervision visits may still need improvement to ensure feasibility, effectiveness, and consistency with MOH case management guidelines. The role of the HC staff in supervision of VHVs is not clearly defined yet. The planned training of HC staff in interpersonal skills has not been done yet. The original KSCSP site in Kean Svay district does not serve as a demonstration site. World Vision 19 Kean Svay Extension Midterm Evaluation The OD and HC staff recognizes the need to provide quality health services meeting demand of the users, but there is no clear standard of quality of care beyond those for clinical and technical services (case management, EPI, etc). Health Center staff and community leaders recognize the improvement resulting from the new three-wheel motorbike ambulances but also the remaining difficulties to refer patients when needed because of transportation issues and because of the poor quality and accessibility of the hospital. Several KS OD senior staff members recognize the need to undertake a formal Health Services Management Training course at the National Institute of Public Health. The application for a team from KS OD was not accepted by NIPH last year but has been accepted for the course starting in December 2002. Recommendations --WV Cambodia  The KSCSP should use one of the Health Officers in Leuk Dek as an ADP Health Coordinator for the remaining of the project to facilitate the transfer of the current health activities, expertise, and responsibilities of the KSCSP. WVC should encourage the ADP and KSCSP staff to plan and conduct activities, and ensure that the ADP staff has the opportunity to benefit from the KSCSP training in health and child survival, as appropriate.  WVC should ensure that at least some continued funding will be available to the Leuk Dek ADP for 1 or 2 years after the end of the project to cover critical activities like training of VHVs and HC staff.  WVC and KSCSP should plan for a complete analysis of the data collected during the supervision visits by the end of the project. This requires ensuring quality and consistency of the data collected, and recording any changes in the procedure that may affect the results. The data should be entered in a structured electronic database that allows statistical analyses. The emphasis of such analyses should be on assessing improvement in quality of care in the various models of implementation, and on assessing the value of the current supervision tools and data for improving and monitoring quality of care.  The KSCSP should consider using the OCA tool with the HCs, HCMCs, FBCs, and VDCs. These assessments should be done now and before the end of the project to assess progress in the capacity of these organizations and learn about the use of these tools. These tools may complement or overlap those recently proposed by the Ministry of Rural Development. --Ministry of Health  The KSCSP and the KS and KPT OD staff should continue regular supervision using the observation, exit interview, and record review tools with the primary objective of providing direct feedback and support to improve the quality of care in health centers.  WVC should introduce Problem Solving methods and skills among KSCSP and partners staff. World Vision 20 Kean Svay Extension Midterm Evaluation  WVC and KSCSP should assist the KS OD in the definition of a few indicators of quality of care and client satisfaction with the health services.  The KSCSP should ensure that the appropriate information and reports about VHV activities reach the KS OD office.  The KSCSP and KS OD should continue to improve the referral system, for instance starting with the establishment of a referral letter.  The KSCSP should implement the recommendation of the First Annual Review that a team of three persons from the OD undertake the Health Services Management training for (a six￾month on-the-job practical training conducted by the National Institute of Public Health and highly rated). The USAID Mission in Cambodia is supportive of this training course. 4. Sustainability Attachment L includes the sustainability plan of the DIP with comments on the progress to August 2002. Achievements The progress in integrating CS into all WVC ADPs is discussed in the previous section C (3). The KSCSP continues to provide training to all HC and OD staff and in-country cross visits and study tours to selected MOH staff. The KSCSP also systematically involves the PHD, OD, and HC staff in project planning, implementation, and evaluation activities. The VHVs regularly attend training and meetings with HC staff, and are active in the community. During the MTE Focus Group Discussion with VHVs, the respondents indicated that they would be able to continue work when the project ends. The community members are actively involved in selection of VHVs and HSMC/FC members, and in the development of HC action plans. Mother’s Group members are active. The current HC fees for services, along with exemption for the poor, seem reasonable to community members who find them cheaper than the cost of drugs on the market, appreciate that the health workers are trained to provide the adequate treatment, and recognize that the fees support the health center (as per MTE Focus Group Discussion with mothers, fathers, and village leaders) The establishment of the CBDDS is discussed in section C (4). World Vision 21 Kean Svay Extension Midterm Evaluation Challenges There is no formal Plan of Action with SMART objectives for handing over management responsibilities of project activities to the OD and HC staff. The current cost-recovery practice is not fully in-line with the MOH Guidelines. HCs do not have other sources of funding than the MOH budget and the cost-recovery system. Recommendations:  The KSCSP staff should prepare formal Exit Plans with the ODs and each HC partners.  The KSCSP should assist the new HCMC and FC in improving the cost-recovery mechanisms in the HC.  The KSCSP and the OD and HC staff should explore possible use of the new MED program in Leuk Dek to provide alternative sources of funding for health.  The KSCSP should assist the OD and HC staff in creating partnership with private providers and increase their involvement in provision of quality CS services. World Vision 22 Kean Svay Extension Midterm Evaluation C. Program management 1. Planning, monitoring and evaluation Achievements Regular standard monthly reports from October 2000 up to the time of the MTE are available. These report show clearly the routine activities of the months, with numbers when appropriate (number of trainees, etc); outputs (immunization or VAC given; cases treated); and key events such as meetings, consultancy, procurement, etc. Quarterly reports on selected outcome indicators provide useful monitoring information. The source of data and methods of calculation is not provided, making it difficult for an outside reader to understand or assess the validity of the data reported. The estimated populations for outcome coverage indicators are consistent with those of the Leuk Dek District, the Direct Impact Area. Overall, the percent achievement of the majority of the indicators reported show good progress toward the specific project targets. With the study of the models of integration of child survival in ADPs in mind, the KCSP conducted baseline KPC surveys in the Leuk Dek and Lovea Em ADs, and in Kasach Kandal AD as a comparison area. These three surveys were conducted between November 2000 and January 2001, using the same questionnaire and 30-cluster methodology.14 The methodology and results are available in the survey report as appendix of the DIP. Most child survival indicators in these surveys are significantly lower than those in the June 2000 Final Evaluation KPC survey of the original KSCSP, considered as a baseline for the KSCSP Follow On in the Kean Svay AD. In February 2000, the Kampong Thom ADPs conducted surveys that included child survival questions, and that are considered as baseline for the KSCSP in Kampong Thom ADs. In January 2002, the KSCSP conducted a KPC survey at midterm of project implementation. The table below presents the realized and planned KPC surveys in the four KSCSP implementation areas. Leuk Dek AD Kean Svay AD Lovea Em AD 3 ADs in Kampong Thom Baseline Nov 2000 Jun 2000 Nov 2000 Jan 20021 Midterm Jan 2002 No No - Final (plan) Yes - Yes - 1 ADP survey 14 Thirty clusters of 7 mothers of children 0-23 months and of 7 additional mothers of children 12-23 months to estimate immunization coverage. World Vision 23 Kean Svay Extension Midterm Evaluation In addition to the KPC surveys, the KSCSP conducted various qualitative assessments and LQAS surveys, and collect services statistics regularly from the health centers and from the VHVs. Challenges There is no definite plan to conduct KPC or other assessment surveys in the various KSCSP implementation areas by the end of the project. Although the design of the KSCSP and the quality of the available data may allow estimates of cost effectiveness to support the study of the various models of implementation of child survival interventions in the KSCSP, there is no specification yet of what (measure of effectiveness) is to be compared to answer the question of what is the most cost effective model of implementation of child survival interventions, and of what is the gain from integration into ADPs. Recommendations:  The KSCSP should carefully document the nature, quantity and cost of the inputs provided to the outreach and community levels activities because these are probably an important determinant of the effectiveness of the four KSCSP models of implementation.  WVC should begin planning for drawing lessons learned, including the cost-effectiveness study and the final evaluation  WVC should organize a study design workshop with the MOH, USAID, and selected interested partners to define specific objectives, data needs, and analysis plan to estimate the cost-effectiveness of the different models of implementation of child survival. This protocol should be developed soon enough to allow primary data collection before the end of the project if needed.  WVC should plan the various KPC and other assessments in the various project areas on the basis of the cost effectiveness study protocol.  WVC should continue developing its participatory review methodology by defining the specific problems to be solved, analyzing the potential sources of biases in the results, and allocating more time to the preparation and analysis stages 2. Human resources development and management Achievements The KSCSP professional positions have been filled since the beginning of the project, with the Project Manager and the 3 Health Officers having continued working on the project since the original KSCSP. The current team is working well and smoothly, and demonstrates commitment and good understanding of the project issues and objectives. World Vision 24 Kean Svay Extension Midterm Evaluation During the MTE Focus Group Discussion with the project staff, they all recognized the success of the project as demonstrated by the KPC survey and the various outputs and achievements of the projects. They attribute this success to the geographic location of the project site (easy access through the main road), the early start of the community mobilization activities, the good collaboration with and support from the MOH partners (the KS OD in particular), the good quality of the technical assistance received, the adequate funding of the program, and the experience accumulated since the beginning of the Kean Svay project in 1996. Internally, other factors of success are the experience and qualification of the staff, the teamwork, and the commitment to respond to the needs of the communities. One major change in the organization of the teamwork has been the assignment of full responsibility for all activities related to each health center to one Health Officer, as opposed to the previous division of tasks across the 3 Health Officers according to technical lines (EPI, Birth Spacing, and IMCI/Nutrition, for instance). The change was introduced during the last year of the Original KSCSP. The other significant change for the project staff is the emphasis on monitoring and capacity building instead of implementation. Overall, the project staff feels these changes are positive for their professional experience and for the HC staff who will be more likely to be able to sustain activities at the end of the project. All the KSCSP staff had several opportunities to receive training and other professional development opportunities in the last 12 months, and this is recognized as very beneficial for their work with the communities. They welcome refresher training in the same topics, but also feel the need for more training in BCC, and in training of trainers, facilitation, and communication skills. In general, the KSCSP staff would like to remain employed with World Vision at the end of the project. They appreciate the supervision system based on the Performance Development Management process. The workload has improved and become manageable, as compared with the period prior to the FAR when this problem was identified. Challenges The KSCSP staff recognizes serious constraints to the progress towards the project objectives, such as the size of the catchment area, the poor quality of the roads, the frequent floods, poor communication with villages and HC, and the low qualification of the HC staff. The KSCSP staff is well aware of the WV priority for integration of child survival into the ADP but feels that the ADP has just started in Leuk Dek and that it is still unclear how to proceed. Recommendations:  The KSCSP professional staff should take the lead in the completion of the project as planned, and in particular in the transfer of activities to the MOH and ADP, in the design and implementation of the cost effectiveness study, and the preparation of the final evaluation and lessons learned workshop in 12 months. WVC should provide technical and managerial assistance as needed. World Vision 25 Kean Svay Extension Midterm Evaluation  WVC should discuss with the KSCSP staff the options of continued employment with WV at the end of the project. 3. Information management Achievements In April and May 2002, the KSCSP organized a consultancy to review the strength and weakness of the CBDDS in Leuk Dek and Kean Svay AD, develop a social autopsy questionnaire, conduct a one-day workshop to discuss findings and recommendations, and develop an action plan to implement the recommendations.15 The consultant acknowledged the value of the CBDDS implemented in Kean Svay since 1996, and the successful transfer of its operations to the HC staff when the project moved from Kean Svay to Leuk Dek. He identified weaknesses in the implementation of the system in Leuk Dek, such as double counting of events with the heath centers reports; limited data feedback and use at the local level; and inadequate verbal autopsy forms. He recommended various measures to improve the response of the health system to the reports from the VHVs; to increase community participation; and to increase the involvement of the HC, OD and PHD staff in the management of the CBDDS. He also addressed the issues of transfer of the CBDDS to the HC and OD at the end of the project, and that of the timeliness and completeness of the system. The Action Plan to further improve the current operations of the system through training and supervision, to develop an outbreak investigation guideline, and to revise the verbal and social autopsy questionnaires is being implemented. During the MTE, a rapid review of the data available at the KSCSP office indicates that the forms from each HC are carefully and systematically filed and retrievable. Reports from Health Centers in Leuk Dek are available from June 2001 to July 2002. Monthly summary tables report the number of births and deaths per HC, and annual summary tables report the deaths or births per months and per HC. Another annual summary table is the number of deaths by cause and by months, from which the distribution of the causes of 50 deaths reported during FY 2002 is presented below. The predominance of ARI as a cause of mortality under five is of prime programmatic importance, and the recommendation of the CBBDS consultant to improve the identification and reporting of this cause of death is very judicious. The large percentage of unknown causes of death also raises the questions about the quality of the verbal autopsies. ARI Meningitis Premature Unknown Total Number 23 3 6 18 50 % 46 6 12 36 100 The KSCSP successfully developed and uses various other sources of information discussed in this report such the KPC surveys, the LQAS surveys, the Quality of Care monitoring indicators from supervision visits, and the Organizational Capacity Assessments. Project staff carefully 15 Community-based Disease and Death Surveillance: System Operation, Verbal and Social Autopsies. Consultancy report, Sophal Oum. May 2002. World Vision 26 Kean Svay Extension Midterm Evaluation collects, compiles, and reports monitoring data such as those on training and on services statistics, as discussed in section C (1). Challenges In addition to the deaths under five, the VHV reports include data on assistance at delivery, age at death, maternal deaths, cases of measles, DHF or cholera, health education sessions, distribution of condoms and ORS packets, and ORS dilution demonstrations. These data do not seem to be aggregated at the district level. The MTE Focus Group Discussions indicate that the reporting of deaths under five and cases of epidemic-prone diseases do result in useful actions at the local level, but the use of these other data at that level is not clear either. The reports prepared by VHVs do not reach regularly the KS OD office. Overall, the KSCSP collects a wealth of good and useful information, and the data is analyzed and presented in well-conceived reports or summary tables, but there seems to be limited interpretations, conclusions, and recommendations that leads to explicit decision making based on this information. Recommendations  The KSCSP should ensure that the CBDDS system is fully operational in the Leuk Dek district by the end of the project.  WVC and the KSCSP should undertake a full analysis of the CBDDS data available at the KSCSP office before the end of the project to first calculate coverage and completeness rates at the village level,16 and then mortality and birth rates, and maybe other rates and trends. This analytical effort should aim at drawing conclusions about the health status and health services of the Kean Svay and Leuk Dek ADs, if possible, but also at conclusions about the feasibility and usefulness of using CBDDS data at the HC or district levels. The potential use of aggregate data at the HC and district levels does not diminish the importance of its use at the HC and community levels.  WVC and the KSCSP should ensure that the data collected on a routine basis or during special studies is fully used for programmatic purposes or for development of tools or methods. This may require planning more time for data analysis, interpretation, and dissemination. It may also require to further develop the internal capacity to do so, or to procure external assistance when needed.  WVC and KSCSP should consider the various changes or additions to the KPC questionnaires as suggested above (age group for vitamin A; denominator for CPR; measures of length) 16 A completeness table by month and HC is available in the CBDDS consultancy report. World Vision 27 Kean Svay Extension Midterm Evaluation II. Conclusions and recommendations 1. Conclusions The major activities of the KSCSP have been implemented as planned, building on the experience and achievements of the original project. The baseline assessment and the DIP workshop were conducted successfully. All the professional positions have been filled by qualified and committed staff who is very able to implement and keep good records of all planned activities. Two important consultancies have been successfully completed and led to valuable improvement in the project plans of work. The FAR and the MTE were completed on schedule in a participative manner that brought to light important issues and responses for the success of the project. All the targeted health center staff is being trained regularly in the 3 AD of the Kean Svay OD, and the training of health center staff in Kampong Thom has started too. The OD staff in Kean Svay and Kampong Thom conducts regularly supervisory visits of the health centers in Leuk Dek and the 3 ADs of Kampong Thom. All the targeted VHVs in Leuk Dek and Kean Svay ADs are being trained every month, and regular meetings are held with the health centers. Outreach activities are conducted in Leuk Dek and Kean Svay. In Leuk Dek, the January 2002 KPC survey showed that after about 8 months of implementation at the community level, the project objectives had already been achieved or surpassed for the EPI, case-management of ARI and diarrhea, Vitamin A, and birth spacing components. Results may be less advanced in Kampong Thom. Eight of the 20 ADPs supported by WV in Cambodia are using child survival tools developed by the KSCSP, and agreed upon common health status indicators. Although activities are progressing well in the Leuk Dek AD, it is still not clear how the ADP will actually continue the project activities at the end of the project. Among the issues are the availability of qualified health staff in the Leuk Dek ADP and the need for continued funding of the regular training and other support activities. The nutritional status of children under the age of 2 is of concern, with 43% moderately or severely undernourished in the 12-23 age group in the January 2002 Leuk Dek KPC survey. This finding may be related to the low level of exclusive breastfeeding and timely introduction of complementary food. 2. Recommendations The MTE developed the following main recommendations To the KSCSP and partners  Take the lead in the transfer of activities to the MOH and ADP, in the design and implementation of the cost effectiveness study, and the preparation of the final evaluation World Vision 28 Kean Svay Extension Midterm Evaluation and lessons learned workshop in 12 months. WVC should provide technical and managerial assistance as needed.  Assess the capacity of health centers staff and VHVs to identify malnourished children and provide adequate nutritional education and counseling.  Assess the related knowledge and practices of mothers of children under 2, and those of the health care providers.  Support IMCI orientation and training in the Kean Svay OD, and begin adapting the related training of the HC staff and VHVs.  Explore the opportunities to diversify the contraceptive methods mix available to men and women in the project area.  Prepare formal Exit Plans with the ODs and each HC partners. To WVC  Assign one of the KSCSP Health Officers as an ADP Health Coordinator for the remaining of the project to facilitate the transfer of the current activities, expertise, and responsibilities.  Ensure some funding for child survival activities in the Leuk Dek ADP for 1 or 2 years after the end of the project  Organize a “study design” workshop with selected partners to define specific objectives, data needs, and analysis plan to estimate the cost-effectiveness of the different models of child survival implementation.  Consider conducting the following other studies for their own value and as part the cost effectiveness study above:  Evaluation of the performance of the VHVs and its determinants  Assessment of the completeness and coverage of the CBDDS at the village level, and maybe estimate of mortality and birth rates and trends.  Costing the various inputs provided at the health center, outreach and community levels  Evaluation of the data collected during the supervision of the health centers, and maybe assessment of the improvement in the quality of care in the various models of implementation  Prepare the end-of-project lessons learned workshop, which may include some of the studies above but also the findings of the final evaluation, the CBDDS, and the issues and plans for integration of child survival into ADPs.  Ensure that all routine or special data is fully used for programmatic purposes or for development of tools or methods.  Develop the participatory review methodology by defining the specific problems to be solved, analyzing the potential sources of biases in the results, and allocating more time to the preparation and analysis stages To USAID  Consider the KSCSP completed as planned.  Consider providing technical assistance for the cost effectiveness study World Vision 29 Kean Svay Extension Midterm Evaluation III. Results highlights Integrating Child Survival into Area Development Programs World Vision and the Ministry of Health in Cambodia are implementing a Child Survival Project in the Kean Svay Operational District (OD) and the Kampong Thom Province in Cambodia. One of the objectives of this project is to identify cost-effective models of implementation of child survival interventions, and within the context of Area Development Programs (ADP) in particular. ADPs are World Vision’s programmatic unit of integrated development based on a 10-15 year commitment to improve child well being and reduce poverty in defined geographical areas. In each ADP, World Vision works in partnership with community leaders to empower and build the capacity of community members. Typically the size of a district, ADPs can include integrated initiatives in health, education, agriculture, infrastructure, landmines, micro-enterprise or sustainable livelihoods depending on the needs and priorities of the communities. The Kean Svay Child Survival Project (KSCSP) is implemented in the three Administrative Districts (AD) of the Kean Svay OD, and in three ADs of the Kampong Thom Operational District in Kampong Thom Province. The technical focus is on EPI, nutrition, IMCI and birth spacing. In each area, the KSCSP provides various types of support at three levels of the health system: training, supervision, and equipment at the health center level; training and organization at the community level; and logistical support for outreach services. In addition, the KSCSP provides this support through the structure of an ADP in two of these areas. These implementation features are summarized below. Leuk Dek Kean Svay Lovea Em Kampong Thom Health system support 1. Health Centers Yes Yes Yes Yes 2. Community Yes Yes No No 3. Outreach services Yes No No No ADP presence Yes No No Yes The KSCSP conducted KPC baseline surveys in the 3 ADs of the Kean Svay OD, and the 3 ADPs in Kampong Thom conducted similar surveys that include key child survival indicators. The KSCSP also collects detailed information on the services provided by the health centers and the village health volunteers, and on the inputs and activities conducted in the four project areas. In addition, the KSCSP plans to conduct various KPC surveys in the four project areas. Given the design of the KSCSP, the availability of reliable data, and the importance for WVC and its partners to know the most cost-effective models of implementation of child survival interventions, and whether their implementation is more cost-effective when integrated into ADPs, the MTE team recommended to develop a detailed study protocol to define the specific objectives, data needs, and analysis plan that best answer that question. World Vision 30 Kean Svay Extension Midterm Evaluation IV. Action Plan World Vision 31 Kean Svay Extension Midterm Evaluation KEAN SVAY CHILD SURVIVAL PROJECT: PLAN OF ACTION IN RESPONSE TO MTE RECOMMENDATIONS On Thursday 28 November 2002 the following persons met to develop this POA. Due to the short time period between receipt of the draft MTE Report and the date for submission, some important stakeholders were not able to participate in this POA discussion. This POA will therefore serve as a guide but may be subject to some changes once input from these other stakeholders in obtained. List of participants: Mr Tuy Saroeun (Chief Kean Svay OD), Dr Lim Somaly (Project Manager), Keo Sereivuth (M&E Officer), Dr Pen Sophea (Training Coordinator), Ouk Piseth (Admin/Finance Officer), Huot Sokda, Kan Chamrouen and Huy Dany (Public Health Nurses) . Technical Approaches Comments Person(s) Responsible Due Date Resources Technical Approaches 1. Immunisation  1.1 The KSCSP should review the available data in the ADP household survey to assess their adequacy as baseline to measure the impact of the project. Secondary analyses should be conducted if necessary for this purpose. This information will be critical to compare the effectiveness of the three models of integration of child survival into ADPs. Will review KPC and ADP Household Survey questions against expected impact and outcomes – adjust as necessary but allow comparison with previous surveys KPC Survey Team Q2 CSTS Technical Manual, KPC 2000+ Manual World Vision 32 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources 2. Nutrition  2.1 Given the high levels of undernutrition demonstrated by the Leuk Dek KPC survey, the KSCSP should assess the capacity of health centers staff and the VHVs to identify malnourished children (growth monitoring) and provide adequate nutritional education and counseling. Opportunistic, but not routine, growth monitoring, is being done. Quarterly GM possible with VHVs supporting HC staff. Will review National policies/ guidelines on growth monitoring. Age group 0-23 months. Will address nutrition education and counseling through BCC approach. OD, HC, VHVs with support from KSCSP Q2 Q3 Q4 National Policies/ Guidelines, BCC approaches, other CS experiences (World Relief, Partners for Development)  2.2 Given the low levels of timely introduction of complementary food, the KSCSP should assess the related knowledge and practices of mothers of children under 2, and those of the health care providers. Will review available Cambodian literature through cIMCI Working Group to decide whether more formative research is needed (this has already been recommended by a WHO Consultant for cIMCI). Link findings to BCC approach. Dr Ly Vanthy / Dr Sour Kim An through cIMCI WG Q2 cIMCI Working Group  2. 3 The KSCSP should measure length in future anthropometric surveys to distinguish between acute and chronic malnutrition. No consistent National Guidelines – WHO-cIMCI recommend weight-for-age for logistical reasons. Will review with WHO and any MoH Guidelines. Could use data from other sources (eg HKI) but may not be representative for Leuk Dek. Dr Douglas Q2 WHO, MoH Guidelines/ Policies World Vision 33 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  2.4 The validity of the indicator of self￾reported use of iodized salts should be assessed before using it as a project monitoring and evaluation indicator. Will continue to use self￾reported use questions (for comparison reasons) but will test salt in ALL households in survey sample. Will also ask schoolteachers/ students and VHVs to test salt on an ongoing basis. KPC Survey Team Q4 3. IMCI  3.1 The KSCSP should support the IMCI orientation and training of the Kean Svay OD staff, and begin adapting the related training of the HC staff and VHVs. This should be done in coordination with the National IMCI unit of the MOH and the OD staff, and by using the 3 WVC staff already trained in IMCI. In principle approval for this recommendation obtained from MoH and WHO – will revise current case-management training plan and budget for OD and HC staff to combine into IMCI approach. Dr Lim Somaly & Dr Ly Vanthy Q1-Q4 World Vision 34 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources 4. Birth Spacing  4.1 The KSCSP should introduce a question in the next KPC survey like “What method are you currently using?” and another question to probe the actual protection status (when was the last injection? Are you taking the pill now? Etc). The question used in the KPC 2000 and 2002 should be maintained to make is possible to compare results from one survey to the other, and help understand what each indicator actually measures. Khmer language questionnaire clearly asks about current use but adding question on last use would strengthen validity. For OC pill will consider asking to see current pill packet. KPC Survey Team Q4  4.2 The CPR should include mothers of children under 2 who are pregnant or who want children within 2 years since most of them would in fact benefit from birth spacing. It should be possible to report data for different denominators – also following KPC recommendations and other best-practice KPC Survey Team Q4  4.3 The KSCSP should explore the opportunities to diversify the contraceptive methods mix available to men and women in the project area. Will link this recommendation to plans to increase men’s involvement in birth spacing. Will consider qualitative studies in women (?and men) and health providers to assess availability, knowledge and skills. OD, HC, VHVs and KSCSP Q2-Q4 Review available relevant literature World Vision 35 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  4.4 The KSCSP should identify and document the key determinants of the success of this intervention (high demand, mode of service delivery, community support, etc) before the end of the project. This will be done in advance of the Lessons Learned Workshop / Final Evaluation OD, HC, VHVs, mothers and fathers, KSCSP, WVC Q3 Q4 May use local consultant to document this success Cross-Cutting Approaches 1. Community Mobilisation  1.1 The KSCSP should continue supporting the HCMC and FBC to ensure they reach the capacity to operate as per the MOH Guidelines after the end of the project. MoH have released final Guidelines (with name changes! Feedback Committee now Village Health Support Group) – we will review our plans to match these PHD, OD, HC, VHV, community, KSCSP Q1-Q4 MoH Policy and Guidelines  1.2 The KSCSP should continue supporting the Mother’s Group and Children Groups. Ongoing project activity VHV & community Q1-Q4  1.3 The KSCSP should consider creating and supporting Men’s/Father’s Groups and Family Groups. Current experience is that it is very difficult to form father’s group for regular meetings due to their absence from the village for work – but for intermittent activities (eg. birth spacing discussions) informal groups can gather. Similar difficulty with Family Groups. ? Experience from other CS projects in Cambodia World Vision 36 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  1.4 WVC should conduct a formal evaluation of the performance of the VHVs before the end of project to draw lessons about the support environment (HC, HCMC, FBC), training received (methodology, contents, effectiveness), and likely impact on the behavior and health status. This evaluation should be conceived as a component of the cost effectiveness study of the various models of integration of child survival into ADPs. This study should link to the Perceptions of VHV studies conducted by Barbara Main in 2000 (see summary in First Annual Review). This evaluation would be presented as part of the Lessons Learned Workshop/ Final Evaluation. OD, HC, VHV, community, KSCSP Q3 Local Consultant would be needed to conduct this evaluation. 2. Behaviour change communication  2.1 WVC should ensure that the whole process of developing and implementing at least the two selected behavior change strategies is completed and documented by the end of the project. This will serve to build the capacity of WVC and partners to apply this valuable approach in other projects. Ongoing activity for men’s involvement in birth spacing and promotion of colostrum feeding. This will also be presented at Lessons Learned Workshop/ Final Evaluation. Also present document to cIMCI WG OD, KSCSP, WVC Q1-Q4 Will ask local BCC specialist to review/ revise documentation  2.2 The KSCSP should carefully assess the need for IEC materials for VHVs and HC, presumably, on the basis of well-defined behavior change strategies. IEC material development will be coordinated with National Centre for Health Promotion IEC WG, cIMCI WG and CS partners KSCSP, OD and WVC Q1-Q4 NCHP IEC WG CIMCI WG World Vision 37 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  2.3 The KSCSP should quantify the behavioral change activities at the community level. Will follow cIMCI indicators recommended by MoH including quantitative and qualitative methods. KPC survey will measure colostrums feeding. Separate survey for men needed to assess changes in behaviour related to birth spacing. Dr Ly Vanthy/ Dr Sour Kim An, OD and KSCSP Q3 Q4 CIMCI WG 3. Capacity Building  3.1 The KSCSP should use one of the Health Officers in Leuk Dek as an ADP Health Coordinator for the remaining of the project to facilitate the transfer of the current health activities, expertise, and responsibilities of the KSCSP. WVC should encourage the ADP and KSCSP staff to plan and conduct activities, and ensure that the ADP staff has the opportunity to benefit from the KSCSP training in health and child survival, as appropriate. One Public Health Nurse has already been selected as interim ADP Health Coordinator to work alongside ADP staff commencing from June 2002. Regular meetings with ADP and KSCSP staff will continue with a focus on developing a phase￾out/ handover plans for the 3-4 communes where the ADP is working. OD, KSCSP, Leuk Dek ADP, WVC Q1-Q4  3.2 WVC should ensure that at least some continued funding will be available to the Leuk Dek ADP for 1 or 2 years after the end of the project to cover critical activities like training of VHVs and HC staff. Draft project proposal/ budget for the 3-4 communes not covered by the ADP will be submitted to donors (WVUS as first choice). Phase-out/ handover plan will be closely linked to this proposal. Dr Douglas, Dr Ly Vanthy. OD, KSCSP, ADP, WVC, WVUS Q2 World Vision 38 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  3.3 WVC and KSCSP should plan for a complete analysis of the data collected during the supervision visits by the end of the project. This requires ensuring quality and consistency of the data collected, and recording any changes in the procedure that may affect the results. The data should be entered in a structured electronic database that allows statistical analyses. The emphasis of such analyses should be on assessing improvement in quality of care in the various models of implementation, and on assessing the value of the current supervision tools and data for improving and monitoring quality of care. A few key IMCI quality of care indicators have already been selected at MTE to replace the “good” “fair” “poor” categories. Raw data is available from the start of Phase 2 and can be entered into this database. Dr Ly Vanthy and KSCSP M&E Officer with at least one OD representative to continue this process after project phase￾out. Q2 Additional IT expertise will be needed to develop the database for easy statistical analysis.  3.4 The KSCSP should consider using the OCA tool with the HCs, HCMCs, FBCs, and VDCs. These assessments should be done now and before the end of the project to assess progress in the capacity of these organizations and learn about the use of these tools. These tools may complement or overlap those recently proposed by the Ministry of Rural Development. Current OCA tool is not suitable for community-based organizations (HCMC, FBC & VDC). Alternative tools, including MRD tool to assess VDCs will be reviewed. The project will aim to conduct at least baseline studies during the year but for continuity this process should be done by ADP staff / community (with help from KSCSP) Dr Douglas/ Dr Ly Vanthy – identify tool. ADP, OD, HC and KSCSP- to apply tool Q2-Q4 MRD Evaluation Tool and other Evaluation Tools World Vision 39 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  3.5 The KSCSP and the KS and KPT OD staff should continue regular supervision using the observation, exit interview, and record review tools with the primary objective of providing direct feedback and support to improve the quality of care in health centers. Ongoing activity but will focus on feedback to improve quality of care KSCSP, OD in KS and KPT Q1-Q4 MoH Supervision Forms, KSCSP/ OD Quality of Care Supervision and Exit Interview Forms  3.6 WVC should introduce Problem Solving methods and skills among KSCSP and partners staff. PHD (1), OD (3) and KS Project Manager has attended COPE Facilitator training in September 2002 and developed COPE training plan for KS OD Health Centres. Three HC in Leuk Dek received training already. OD, KSCSP Q1-Q4  3.7 WVC and KSCSP should assist the KS OD in the definition of a few indicators of quality of care and client satisfaction with the health services. A few key indicators selected during MTE – see 3.3 this section Will add client satisfaction indicators from COPE training Q1-Q4  3.8 KSCSP should ensure that the appropriate information and reports about VHV activities reach the KS OD office. For sustainability purposes KSCSP should not pass on information from VHV to OD. KSCSP will support flow of information from VHV-HC-OD and back. This has been strengthened since MTE in September. OD, HC, VHV and KSCSP Q1-Q4 World Vision 40 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  3.9 The KSCSP and KS OD should continue to improve the referral system, for instance starting with the establishment of a referral letter. Other CS partners have some experience in this area but not much success – will review their experiences and any MoH Guidelines/ Policies (MoH have a referral letter for referral from HC to Referral Hospital). KSCSP and OD will explore ways to strengthen referral from VHV to HC to Referral Hospital. Dr Douglas/ Dr Vanthy Q2-Q4 CS Partners, MoH Policies/ Guidelines 4. Sustainability  4.1 The KSCSP staff should prepare formal Exit Plans with the OD and each HC partners. High priority – progress made but needs to be formally documented and linked to ADP handover plan. OD, HC, KSCSP, ADP, WVC Q1-Q4  4.2 The KSCSP should assist the new HCMC and FC in improving the cost￾recovery mechanisms in the HC. Ongoing activity linked to Community Mobilization 1.1 OD, HC, VHV, VDC, ADP, KSCSP Q1-Q4  4.3 The KSCSP and the OD and HC staff should explore possible use of the new MED program in Leuk Dek to provide alternative sources of funding for health. This will be noted in the Exit Plan/ Handover Plan with primary responsibility taken by ADP/MED staff with support from OD, HC and KSCSP MED, ADP with OD, HC, KSCSP Q3-4 World Vision 41 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  4.4 The KSCSP should assist the OD and HC staff in creating partnership with private providers and increase their involvement in provision of quality CS services. Training of drug-sellers already part of project. Village “corner￾stores” identified as an important source for some medications but in project plan. One key area to address is the private practice of public health providers. This issue faced by all CS partners –plan to review of available Cambodian literature to help direct action. Dr Douglas/ Dr Vanthy to review literature and make recommendations Q2 Studies on private providers in Cambodia Program Management 1. Planning, monitoring and evaluation  1.1 The KSCSP should carefully document the nature, quantity and cost of the inputs provided to the outreach and community levels activities because these are probably an important determinant of the effectiveness of the four KSCSP models of implementation. This process will depend on the outcome of the workshop detailed in 1.3 this section. Q3 External Technical Assistance will be required with Scope-of￾Work based on outcomes from Workshop in 1.3  1.2 WVC should begin planning for drawing lessons learned, including the cost￾effectiveness study and the final evaluation A formal “cost-effective” study, with an emphasis on maximum use of available data complemented by a manageable level of additional data, will be presented at Lessons Learned Workshop/ Final Evaluation. World Vision 42 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  1.3 WVC should organize a study design workshop with the MOH, USAID, and selected interested partners to define specific objectives, data needs, and analysis plan to estimate the cost-effectiveness of the different models of implementation of child survival. This protocol should be developed soon enough to allow primary data collection before the end of the project if needed. Need to identify Consultant to facilitate this workshop – then develop study protocol and identify Consultant to lead the actual study Dr Douglas /Dr Vanthy to identify Consultant to facilitate this workshop Q2  1.4 WVC should plan the various KPC and other assessments in the various project areas on the basis of the cost effectiveness study protocol. This recommendation not clear – but once initial Workshop conducted and protocol developed, studies can be designed to fit this protocol  1.5 WVC should continue developing its participatory review methodology by defining the specific problems to be solved, analyzing the potential sources of biases in the results, and allocating more time to the preparation and analysis stages Will work with Staff Development and M&E Units of World Vision to enhance methodology for PLA with a focus on improved analysis – share outcomes with project partners SD Unit and M&E Unit of WVC Q2-Q4 World Vision 43 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources 2. Human resource development and management  2.1 The KSCSP professional staff should take the lead in the completion of the project as planned, and in particular in the transfer of activities to the MOH and ADP, in the design and implementation of the cost effectiveness study, and the preparation of the final evaluation and lessons learned workshop in 12 months. WVC should provide technical and managerial assistance as needed. See comments above KSCSP with PHD, OD and support from WVC Q1-Q4  2.2 WVC should discuss with the KSCSP staff the options of continued employment with WV at the end of the project. This is an important part of the Exit Plan – discussions have already started and will continue WVC, KSCSP Q1-Q4 3. Information management  3.1 The KSCSP should ensure that the CBDDS system is fully operational in the Leuk Dek district by the end of the project. Revised Verbal Autopsy Form – concerns about length of form – will discuss with Dr Oum Sophal. Social Autopsy Form – still to be finalized (after Verbal Autopsy form revised) Dr Ly Vanthy, KSCSP M&E Officer Q1-Q2 Dr Oum Sophal World Vision 44 Kean Svay Extension Midterm Evaluation Technical Approaches Comments Person(s) Responsible Due Date Resources  3.2 WVC and the KSCSP should undertake a full analysis of the CBDDS data available at the KSCSP office before the end of the project to first calculate coverage and completeness rates at the village level, and then mortality and birth rates, and maybe other rates and trends. This analytical effort should aim at drawing conclusions about the health status and health services of the Kean Svay and Leuk Dek ADs, if possible, but also at conclusions about the feasibility and usefulness of using CBDDS data at the HC or district levels. The potential use of aggregate data at the HC and district levels does not diminish the importance of its use at the HC and community levels. Will need Technical Assistance. This report will form part of the Lessons Learned Workshop and Final Evaluation. Q3 Local Consultant needed to lead this review  3.3 WVC and the KSCSP should ensure that the data collected on a routine basis or during special studies is fully used for programmatic purposes or for development of tools or methods. This may require planning more time for data analysis, interpretation, and dissemination. It may also require to further develop the internal capacity to do so, or to procure external assistance when needed. Agreed – see comments in relation to Planning, M&E 1.5 (PLA). Will need to develop a way to measure how data is used for programming. Dr Douglas and Dr Ly Vanthy to develop tool to measure use of data. Q2-Q4  3.4 WVC and KSCSP should consider the various changes or additions to the KPC questionnaires as suggested above (age group for vitamin A; denominator for CPR; measures of length) Agreed – see comments already made above KPC Survey team World Vision 45 Kean Svay Extension Midterm Evaluation ATTACHMENTS World Vision 46 Kean Svay Extension Midterm Evaluation Attachment A Baseline Information from the DIP 1. FIELD PROGRAM SUMMARY Estimated Program Effort and USAID funding by intervention LEVEL OF EFFORT BY INTERVENTION AREA PERCENT CUMULATIVE % Immunization 15 15 Micronutrients 10 25 Prenatal-Postnatal 10 35 Family Planning 10 45 Promotive – Health Behaviors 25 70 Case Management 25 95 Other as needed 5 100 Program Site Population: Children and Women The children under five and women of childbearing age in the Kean Svay Operational District are the direct beneficiaries of project interventions. Type Number 0-59 month old children 39,982 Women 15-49 46,730 Estimated number of births *8,432 MTE Comment: * Number of births as reported in the Demographic Profile Table on page 9 of the DIP. MTE Comment: The number of “direct beneficiary” defined in the statement and the table above (from the Executive Summary, page 5) corresponds to that of the Kean Svay Operational District and not the “Direct Impact Area” (Leuk Dek) as defined in the Project Design section, page 22. 2. PROGRAM GOALS AND OBJECTIVES Strategic Objective: The KSCSP extension project’s strategic objective is to assist the Kean Svay OD, private sector and community partners to accomplish, sustain, document and replicate promising practices to reduce infant and child mortality and morbidity through an innovative mainstreaming child survival and reproductive health improvement project in the primary impact area (namely, Leuk Dek District, where WV Cambodia’s ADP is located) of the Kean Svay Operational District (OD) in Kandal Province in Cambodia over a three year period. World Vision 47 Kean Svay Extension Midterm Evaluation Accomplishment of this strategic objective will be achieved through the accomplishment of the following Intermediate Results and Sub-results: Intermediate Result #1: Increased use of high-impact child survival and reproductive health services by the target population (in the primary impact area-- Leuk Dek District) Intermediate Result #2 Increased Quality of Case Management of Childhood Illness In Health Centers (in the primary impact area-- Leuk Dek District; and in the four indirect impact areas: Lovea Em District in Kean Svay OD ; and in three other Districts in Kompong Thom province where 3 WVC’s ADPs are located.) Intermediate Result #3: Enhanced capabilities of individuals, families and communities to protect and provide for their own health (in the primary impact area-- Leuk Dek District) Intermediate Result #4: Enhanced sustainability of child survival and reproductive health services and support systems (in the primary impact area-- Leuk Dek District) Intermediate Result # Sub-results (S.R.) IR # 1 Focus: Coverage S.R # 1.1: Increased Coverage of Essential Care of the Sick Child S.R # 1.2: Increased Coverage of Immunization S.R # 1.3: Prevention of Malnutrition/Vitamin A Deficiency/ Iodine Deficiency S.R # 1.4: Improved Reproductive Health Coverage IR # 2 Focus: Quality S.R.# 2.1 Strengthened Health Center Case Management of Childhood Illnesses S.R.#.2.2 Improved Private Practitioners’s Case Management of Childhood Illnesses (depending on results of PLA survey) IR # 3 Focus: Capacity Building S.R # 3.1: Improved Caretaker and Family Practices S.R # 3.2: Capacity building of community institutions S.R # 3.3: Capacity of WV ADP staff S.R # 3.4: Capacity of WV HQ staff IR #4 Focus: Sustainability S.R # 4.1: Sustainability initiatives in place S.R # 4.2: Increased community participation S.R.# 4.3: Increased capacity of community organizations to sustain long-term viability of health development processes/impact World Vision 48 Kean Svay Extension Midterm Evaluation Matrix of Objectives for Sub-results S.R #.1.1 Increased Coverage of Essential Care of the Sick Child Matrix for Pneumonia Intervention 80 % of children 0-23 months with cough and difficult or rapid breathing in the past 2 weeks were brought to a trained health provider. (FY 01: 70%; FY 02: 75% ; FY 03: 80%) Indicator: % of children 0-23 months with cough and difficult or rapid breathing in the past 2 weeks were brought to a trained health provider. Measurement Method: KPC Survey 90% of mothers of children 0-23 months would be able to recognize 2 or more dangers signs/symptoms of /pneumonia that call for immediate referral and treatment (FY 01:86%; FY 02: 86% ; FY 03: 90%) Indicator: % of mothers of children 0-23 months would be able to recognize 2 or more dangers signs/symptoms of /pneumonia that call for immediate referral and treatment Measurement Method: KPC survey Matrix for control of diarrheal diseases section 80% of children 0-23 months with diarrhea in the past two weeks who were treated with ORT (FY 01: 60%; FY 02: 70% ; FY 03: 80%) Indicator: % of children 0-23 months with diarrhea in the past two weeks who were treated with ORT Measurement Method: KPC Survey 80% of children 0-23 months with diarrhea in the past 2 weeks were given the same amount or more food during their diarrheal episode. (FY 01: 50%; FY 02: 65% ; FY 03: 80%) Indicator: % of children 0-23 months with diarrhea in the past 2 weeks were given the same amount or more food during their diarrheal episode Measurement Method: KPC survey 80% of children 0-23 months with diarrhea in the past 2 weeks were given the same amount or more fluids other than breast milk during their diarrheal episode. (FY 01: 60%; FY 02: 75% ; FY 03: 80%) Indicator: % of children 0-23 months with diarrhea in the past 2 weeks were given the same amount or more fluids other than breast milk during their diarrheal episode Measurement Method: KPC survey Less than 10% of mothers with child less than 2 years who had diarrhea in the past two weeks report their child received an anti-diarrheal medicine. (FY 01: 40%; FY 02:20% ; FY 03: 10%) Indicator: % of mothers with child less than 2 years who had diarrhea in the past two weeks report their child received an anti-diarrheal medicine Measurement Method: KPC survey 90% of infants 0-6 months were exclusively breastfed in the last 24 hours (FY 01: 20%; FY 02: 60 % ; FY 03: 90%) Indicator: % of infants 0-6 months were exclusively breastfed in the last 24 hours Measurement Method: KPC Survey World Vision 49 Kean Svay Extension Midterm Evaluation S.R #.1.1 Increased Coverage of Essential Care of the Sick Child Matrix for Dengue Hemorrhagic Fever Intervention 60% of caretakers of children 0-23 months can name at least one danger sign of DHF (cold extremities, bleeding, abdominal pain, weakness) (FY 01: 30%; FY 02: 40% ; FY 03: 60%) Indicator: % of caretakers of children 0-23 months can name at least one danger sign of DHF (cold extremities, bleeding, abdominal pain, weakness) Measurement Method: KPC Survey S.R #.1.2 Matrix for Increased Immunization Coverage 85% of children (12-23 months) will have been fully immunized before age 12 months ( have received card-documented doses of BCG, OPV3, DTP3 and measles vaccines before age 12 months); (FY 01: 75%; FY 02: 80%; FY 02::85%;) (Indicator: Proportion of children (12-23 months) who had been fully immunized before age 12 months ( have received card-documented doses of BCG, OPV3, DTP3 and measles vaccines before age 12 months) Measurement Method: KPC survey 75% of mothers with childre 0-23 months will have received two doses of tetanus toxoid (TT2) before the birth of their youngest child; (FY 01:40%; FY 02:55%; FY 03:75%; (Indicator: Proportion of mothers would have received two card-documented doses of tetanus toxoid (TT2) before the birth of her youngest child less than 24 months) S.R #1.3 Matrix for Prevention of Malnutrition/Vitamin A Deficiency/Iodine Deficiency 90% of infants 0-6 months were exclusively breastfed in the last 24 hours (FY 01: 20%; FY 02: 60 % ; FY 03: 90%) Indicator: % of infants 0-6 months were exclusively breastfed in the last 24 hours Measurement Method: KPC Survey 80% of children 6-23 months will have received a card-documented vitamin A within the last six months doses semi-annually; (FY 01: 40%; FY 02: 60 %; FY 02: 80 %; ); (Indicator: Proportion of children 6-23 months who received a card-documented vitamin A within the last six months doses semi- annually; Measurement Method: KPC survey, community registers of vitamin A distribution 80% of mothers of infants 0-11 months will have received a card-documented dose of vitamin A supplement within 2 months of delivery (FY 01: 30%; FY 02: 60 %; FY 03: 80 %; ); (Indicator: Proportion of mothers of infants 0-11 months will have received a card-documented dose of vitamin A supplement within 2 months of delivery) Measurement Method: KPC survey; maternal cards; TBA records 45 % of families with children 0-23 months eill have reported use of salt that is iodised. (FY 01: 250%; FY 02: 35 %; FY 03: 45 %; ); (Indicator: % of families with children 0-23 months eill have reported use of salt that is iodised Measurement Method: KPC survey S.R #1.4 Matrix for Improved Reproductive Health Coverage 30% of mothers with children under 2 years who are not pregnant and desire no more children in the next two years, (or are not sure), will be using a modern contraceptive method. (FY 901: 10%; FY 02: 20%; FY 02:30%;); (Indicator: Proportion of women with children under 2 years who are not pregnant and desire no more children in the next two years, (or are not sure), who are using a modern contraceptive method.) Measurement Methods: KPC survey; ; CHC records; World Vision 50 Kean Svay Extension Midterm Evaluation 3. PROGRAM LOCATION The health status of the Cambodian people is one of the lowest in the Western Pacific Region. Recent national surveys have yielded infant mortality rates ranging from 89 (1998 National Health Survey) to 115 per 1000 live births (Royal Government of Cambodia official figure) and child mortality rates from 115 (1998 National Health Survey) to 181 per 1000 (RGC official figure). In 1996, 49.3% of children aged 0-59 months were moderately and severely underweight (Health Situation Analysis 1998 and Future Direction for Health Development, 1999-2003). Almost 40 000 under-five Cambodian child deaths each year result from a high burden of diseases, widespread malnutrition, poor maternal health, limited availability of basic health services and poor household health knowledge. Over half of all child deaths are caused by diarrheal disease, acute respiratory infections and vaccine preventable infections, especially measles (National Plan of Action for Early Implementation Phase of IMCI, 1999-2000). There are several factors which influence the health of Cambodians including a low Ministry of Health budget allocation which results in inadequate medical supplies and poor salaries for health workers. There is an expanding and unregulated private health sector, some sections of the media advertise medical services and pharmaceutical products using misleading information and tobacco and alcohol advertising is increasingly aggressive. World Vision 51 Kean Svay Extension Midterm Evaluation DEMOGRAPHIC PROFILE Kean Svay Operational District Kampong Thom Operational District Focus Original Project Site Direct Impact Area Indirect Impact Area Indirect Impact Area District Name Kean Svay* Leuk Dek Lovea Em Sandan Prasath Ballang Prasath Sambo WV-ADP presence No Yes No Yes Yes Yes Total population 123,113 55,688 69,081 9,246 17,098 18,514 # communes 9 7 15 2 4 4 # villages 36 24 43 18 27 33 Total # households 22,517 9,790 12,735 N/A N/A N/A # children 0-11 months 4,190 1,891 2,351 317 615 669 # children 12-59 months 16,005 7,250 8,981 2,000 3,866 2,568 Total # children 0-59 months 20,195 9,141 11,332 2,317 4,481 3,237 # women 15-45 years 24,624 11,140 13,813 4,438 4,445 5,104 Est. # new births/year 4,190 1,891 2,351 351 649 703 Est. # new births over 3 years 12,570 5,673 7,053 1,053 1947 2,109 *data from 9 communes where project operating World Vision 52 Kean Svay Extension Midterm Evaluation HEALTH PROVIDERS AND FACILITIES Kean Svay Operational District Kampong Thom Operational District Focus Original Project Site Direct Impact Area Indirect Impact Area Indirect Impact Area District Name Kean Svay (1) Leuk Dek Lovea Em Sandan Prasath Ballang Prasath Sambo WV-ADP presence No Yes No Yes Yes Yes # District health staff 91(2) 22 (3) # commune health centers 7 4 6 1 1 3 # commune health staff 57 27 46 14 10 41 # doctors 22 0 0 0 1 0 # medical assistants 20 0 2 0 0 0 # primary nurses 27 13 18 7 5 20 # secondary nurses 21 2 7 1 1 7 # primary midwives 5 9 3 0 0 0 # secondary midwives 13 3 7 0 1 2 Others 40 15 18 3 1 6 # private providers (4) 6 14 75 238 165 270 # TBAs 29 27 55 24 46 66 # traditional practitioners (5) 3 3 4 24 15 17 # drug sellers 59 82 26 5 5 27 # VDCs members 0 9 (4) 0 9 3 4 # VHV 72 0 65 23 27 28 (1) data for 9 communes in Kean Svay District (2) based in Kean Svay but serving all three districts (3) based in Kg Thom OD serving 21 health centers in 5 districts including the 3 with WV ADPs The lack of a clear and consistent definition of private health provider probably accounts for the greater number of private practitioners in Kg Thom. The greater number of TBAs and traditional practitioners may reflect the reduced access and use of public health services in the rural and remote areas of Kg Thom OD. (4) The actual number of traditional practitioners in Kean Svay OD is not known. The OD Chief was aware of 10 who were the most popular 4. PROGRAM DESIGN The Kean Svay Child Survival (KSCSP) follow-on project has been designed to continue to achieve the goal of reduced infant and child morbidity and mortality, both within and beyond the original project area by assisting the Kean Svay health Operational District, private sector, and community partners to achieve, sustain and replicate “essential” elements of KSCSP. The original KSCSP has been successful in reaching and surpassing its major child health indicators. The key components of this success have included direct service delivery through mobile teams comprising project staff and health center staff, training at all levels (community VHVs, health center staff, OD staff, Provincial Health Department staff and WV project staff) and WV project management. Anticipated results for the KSCSP are stated as both health outcome objectives and objectives for individual and institutional capacity building. World Vision 53 Kean Svay Extension Midterm Evaluation 1. Health Outcome Objectives The project will implement four major interventions in order to further the goal of reducing infant and child morbidity and mortality. The KSCSP will:  Prevent childhood diseases through immunization,  Improve case management of childhood illnesses,  Improve nutrition including breastfeeding and eliminate micronutrient deficiency, and  Increase the use of birth spacing technologies. 2. Capacity Building Objectives Capacity building is the central strategy for ensuring the sustainability and replication of project benefits and intervention delivery systems. Capacity building will take place at both the institutional and individual levels. Individual capacity building is targeted to achieve positive changes in behavior and relies on training, ongoing technical assistance, tools and reference documents, performance evaluation, and supportive supervision. Institutional capacity building is targeted to achieve positive changes in systems, and relies on all the same strategies, but with inputs targeted to develop and sustain management systems in support of service delivery, including those for the human resource development needed for a healthy and functional system. KSCSP investments in capacity building will strengthen systems and change individual behaviors in order to:  Sustain CS services in Kean Svay OD for at least 10 years by developing the project area as a World Vision “Area Development Project” (ADP) with a CS component;  Replicate the essential elements of CS in three other World Vision ADPs by developing the Kean Svay OD as a training and demonstration site for ADP managers and their MOH partners from other districts in Cambodia; and,  Promote the broader adoption of CS best practices by inviting national, provincial and district MOH personnel from outside WV ADPs to use the KSCSP training and demonstration site as a benchmark for development of child health program strategies for the national MPA. Details about the capacity building objectives and strategies are described in Section G. Capacity Building. With the proposed extension of the project, some significant changes in program design will be required, but these will draw on the experiences gained to date in Kean Svay. This new program design will address the following key issues:  Increasing the number of beneficiaries while maintaining quality of service;  Increasing the focus on capacity building of project partners at all levels with a view towards sustainability of health benefits; and  World Vision Project staff will move from a service delivery mode to a technical assistance and support role while at the same time health center staff will assume greater responsibilities for delivery of quality services. World Vision 54 Kean Svay Extension Midterm Evaluation The Approach World Vision’s primary vehicle for sustainable community development, including health, is the Area Development Program (ADP) approach. From 1995-98 all of World Vision’s community based development programs have been in a gradual transition to becoming ADPs which is World Vision’s primary model and vehicle worldwide for ‘transformational development that is community based and sustainable, and focused on the well being of children’ (WVI mission). By the end of 2000, WV Cambodia had 18 ADPs in 6 provinces, managed from five decentralised provincial ‘operations’ offices. By 2004 expansion is planned to 23 ADPs. This is a long-term commitment (10-15 years) by World Vision to work alongside a given community and is being increasingly used by World Vision on an international scale with considerable experience already gained in many different settings. One of the most important strategies for sustainability of child health will therefore be the integration of child survival interventions into the ADP planning, design and activity cycle. All ADPs already have a health component, with a focus on child health. The Kean Svay Extension project will serve as a model to explore different strategies to support sustainable health interventions, drawing on child survival experience. Lessons learned from these experiences will be shared with all project partners, with other NGOs in Cambodia, with maternal and child health projects and within World Vision internationally, Impact Areas The Kean Svay CS extension project application (December 1999) alluded to direct and indirect impact areas. The Kean Svay Health Operational District (OD) comprises three administrative districts; Kean Svay, Leuk Dek and Lovea Em. Based on baseline information, the administrative district of Leuk Dek in the Kean Svay OD has been selected, in consultation with project partners, as the site for direct extension of the child survival project (direct impact area). The major reasons for this choice are the documented poorer health and socioeconomic health of the population and the fact that the district is in the same health OD as Kean Svay. The Kean Svay OD staff have already received training in many aspects of project management and child survival interventions and will remain a focus for increased capacity building during the proposed extension. This is another critical strategy in the direction of sustainability of child health benefits. In FY00, World Vision commenced an ADP in Leuk Dek district. Consistent with the community development approach of “start small, grow slowly”, this “seed” ADP proposes to commence its community development approach in three of the seven communes of Leuk Dek. The Child Survival project extension therefore has an ideal opportunity in these three communes to work closely with ADP project staff and the local community with the focus on community capacity building. In the remaining four communes of Leuk Dek, the Child Survival project extension will implement child survival interventions in a way which will facilitate expansion of the ADP to these communes in the next two to three years. This will be achieved through a focus on capacity building of the community, health center staff and OD staff. This strategy will draw on the successful Training for Transformation approach (Hope A and Timmel S [1984], Mambo Press) which brings together five key methodologies: community World Vision 55 Kean Svay Extension Midterm Evaluation awareness raising (Freire P 1972, Pedagogy of the Oppressed, Penguin Books), human relations training for participatory education, organizational development to build community structures for self-reliance, social analysis to identify the root causes of community-identifed problems and the Christian concept of transformation. The indirect impact areas include the original project location of Kean Svay district and Lovea Em district, both also within the Kean Svay Health OD. In these areas, World Vision project staff will continue to provide support, technical assistance and training to health center and OD staff. An agreed phased plan for handing over increasing responsibility for management of child health activities and provision of quality health services to the OD has been developed as another critical component of sustainability. The second indirect impact area will be three districts in Kompong Thom province where World Vision has WV-US funded ADPs. There is good ADP project infrastructure in these areas with a competent and well-trained WV health coordinator (Dr Yim Yaren) and local ADP health coordinators. The strategy for these ADPs will be to facilitate the training of health center staff at the commune, district and provincial level in child survival interventions using the resources developed during the original Kean Svay CS project. This will specifically include training in case-management and follow-up monitoring and evaluation of the effectiveness of this training. Other components of a comprehensive approach to child health, including community capacity building, training in planning and management skills, training in quantitative and participatory qualitative methodologies will be the primary responsibility of the ADP managers and health coordinators, funded from within their own budgets, but with technical assistance and support from World Vision’s National Health Program staff (Dr Douglas Shaw and Dr Ly Vanthy) and the Kean Svay CS Project Manager (Dr Lim Somaly). Integration of KSCSP with other health-related activities World Vision Cambodia has a four-year (FY00-04) strategic plan to integrate its sectoral programs into ADPs as a means to ensuring sustainability of benefits. These sectors include health, HIV/AIDS, Children in Exceptionally Difficulty Circumstances (CEDC), Mines Awareness, Gender Equality and Microenterprise Development. Of direct relevance to the Kean Svay CSP extension is the presence of a recently established World Vision Cambodia (Japanese Government funded) HIV-AIDS project targeting at-risk groups and the general population along Highway 1 which runs between Phnom Penh and Ho Chi Minh City in Vietnam. The main components of this project are HIV-AIDS prevention, counseling and testing and care (including home care teams). Expertise from this project will be shared and implemented in addressing the maternal and child health impacts of HIV￾AIDS in the KS CSP extension. The Kean Svay CS project extension therefore presents an ideal opportunity to implement a number of possible models of integration in order to determine which have the greatest potential for maintaining and improving health in the long-term. Both the Mid-Term Evaluation and Final Evaluation will assess these models and make recommendations. World Vision 56 Kean Svay Extension Midterm Evaluation 3. Specific Aspects of Program Design There are three main components in the Program Design for the KSCSP Extension. 1. Strengthening Health Center Case Management of Childhood Illness 2. Outreach from Health Centers to communities to provide preventive services, including immunization, Vitamin A, antenatal care and birth spacing. 3. Establishing a health presence in all 24 villages in Leuk Dek district to improve health behaviors. 1 Strengthening Health Center Case Management of Childhood Illness During the first phase of the KSCSP, verbal autopsy studies were carried out on 310 under five deaths, as displayed below. Verbal Autopsy Identified Causes of 433 Child Deaths Cause Number Percent ARI/Pneumonia 147 34 Neonatal 43 10 Meningitis 39 9 Malnutrition 22 5 Dengue 13 3 Diarrhea 9 2 Other including drowning/injuries 74 17 Unknown 86 20 Total 433 100 Given that most childhood deaths represent the synergistic interaction of undernutrition, infection and delayed/inappropriate care, the above figures need to be interpreted with caution. The fact that half the deaths were acute infections, most of which could have been prevented by early and immediate treatment, emphasizes the importance of early recognition and treatment of acute childhood illnesses. Given that neonatal mortality accounts for an estimated 25% of infant mortality and 10-15% of under-five mortality, it is clear that many neonatal deaths (25-50) went unreported in the above enumeration. Under case management, priority will be given to the following activities  Identification of government health facilities in the areas carrying out case management (Leuk Dek, Lovea Em and Kompong Thom ADPs).  An assessment of these facilities regarding the availability of equipment, e.g., watches/timers, equipment, and drugs needed for use of the algorithm. Assessments will be carried out by cross-visiting of facilities by government staff using a standard checklist and supported by World Vision personnel.  Provision of case management training to health facility staff in all three areas using these algorithms. The first phase of this training was carried out from February 26 through March 2, 2001. World Vision 57 Kean Svay Extension Midterm Evaluation  Monthly visits will be made by World Vision staff to each health facility to support and encourage program operations and upgrade clinical skills. As part of these monthly visits, a continuing education session will be carried out on a topic as epidemiologically appropriate, e.g., pneumonia, diarrhea, birth spacing. Results of these support visits will be provided to the facility and to the OD Medical Officer. Checklists will include availability of staff, essential drugs, number of patients at facility, quality of care observed, results of exit interviews and results of most recent outreach.  Given that many areas of Leuk Dek have no accessible affordable referral sites, health centers will need to develop contingency referral practices for treatment of severe conditions such as dengue, dehydration, meningitis and severe pneumonia.  Given the low non-subsistence salaries of health personnel and the usual practice of providing private care from their homes, the project will encourage health workers to use the same case management alogrithm in their private practices. 2. Outreach from Health Centers to communities to provide preventive services, including immunization, Vitamin A, antenatal care and birth spacing. Government strategy calls for facility outreach to communities in their catchment areas for the delivery of preventive services including:  Immunization of children under 1 with using the government schedule of BCG at birth or first contact, OPV and DPT at 4, 6, and 10 weeks; and measles at 9 months.  Examination of pregnant women, including administration of tetanus toxoid (1-2 doses as required to ensure protection), provision of iron, and identification of high risk pregnancies requiring referral, e.g., headaches, edema, bleeding, abnormal lie).  In coordination with National Immunization Days, which provide oral polio vaccine and vitamin A, to ensure that all children 6-60 months received vitamin A every 6 months.  Postpartum women will be given vitamin A provided contact occurs in the first two months.  Postpartum women and women of fertile age not desiring pregnancy at this time will be advised on birth spacing options and provided with the contraceptive of their choice. 3. Establishing a health presence in all 24 villages in Leuk Dek district to improve health behaviors.  Given that among the three areas of the OD, Leuk Dek is the most needy, has the least access to health facilities including the capital Phnom Penh, and is the location of the long term ADP program, Leuk Dek will be the focus for community empowerment as detailed in the overview above.  Where villages identify the need for a health person or group in the village, a village health resource will be identified and training provided.  As these groups identify areas of concern, World Vision CS and ADP project staff will work with them to identify and implement solutions. Specific groups will be World Vision 58 Kean Svay Extension Midterm Evaluation identified in the village for messages on promotion, prevention, and treatments such as: a) Pregnant women: prenatal care, tetanus immunization, importance of ingestion of iron tablets, and “cleans” (cloth, hands, knife, tie) for delivery. b) For each village, traditional providers of delivery care will be identified and encouraged to participate in prenatal care and delivery skill updating. c) For newborns: importance of stimulation, drying, warming, and immediate breastfeeding. d) Pregnant women and new mothers: postpartum vitamin A, exclusive breastfeeding for 6 months, appropriate foods at six months. e) Mothers of children under two: recognition of illness, home treatment, e.g., ORS, importance of food and fluids with all illnesses. f) Recognition of severe illness and importance of immediate referral. g) Identification for each village of currently used referral sources, e.g., health center or other provider. If the village has no ready access to a facility with a provider trained in case management, community and WV will work together to identify a care source and provide training in case management. World Vision 59 Kean Svay Extension Midterm Evaluation INTERVENTIONS BY IMPLEMENTATION AREA Kean Svay Phase 2 Leuk Dek Lovea Em Kompong Thom PREVENTIVE Immunization of infants and pregnant women Once weekly and outreach Once weekly and outreach Once weekly and outreach ADP support to HC Vitamin A 6-71 months National campaigns + National campaigns + National campaigns + National campaigns + Prenatal assessment and iron HC daily and outreach HC daily and outreach HC daily and outreach ADP support to HC Postpartum Vitamin A Track pregnancies and births Track pregnancies and births Track pregnancies and births ADP support to HC Postpartum & Fertile Age Counseling on Birth Spacing HC daily and outreach HC Daily and Outreach HC Daily and Outreach ADP support to HC Iodized Salt Local Sale Local Sale Local Sale PROMOTIVE Clean Delivery Community & TBA Community & TBA Community & TBA ADP support to HC & TBAs Newborn Care Newborn Care Manual 6/2001 CDC/CARE Newborn Care Manual 6/2001 CDC/CARE Newborn Care Manual 6/2001 CDC/CARE ADP support to HC & TBAs Exclusive BF for 6 months HC and outreach Community HC and outreach ADP support to HC Protein-Calorie Dense Foods at 6 months HC and outreach Community HC and outreach ADP support to HC Recognition of illness and home care Ongoing Community HC and outreach ADP support to HC Hygiene Ongoing Community HC and outreach ADP support to HC CASE MANAGEMENT Assess Health Facilities Peer assessment using checklist Peer assessment using checklist Peer assessment using checklist Peer assessment using checklist Training HW in Case Management Initial and quarterly update Initial and quarterly update Initial and quarterly update Initial and quarterly update Identification of referral sites HC Community HC ADP support to HC Caretaker recognition of severe illness Ongoing Community HC ADP support to HC Support-a-vision to Health Center staff Monthly Twice Monthly Monthly ADP support to HC ORT Corners Maintain Develop and maintain Develop and maintain ADP support to HC Continuing education Monthly Monthly Monthly ADP support to HC Hospital “Hand’s on” Refresher Training Every 6 months Every 6 months Every 6 months Every 6 months 5. PARTNERSHIPS World Vision 60 Kean Svay Extension Midterm Evaluation Selection and Roles of Partners The DIP workshop was held on 5th and 6th February 2001 at the Kean Svay project center. Among the outcomes of the workshop were discussion and provisional agreement on the roles and contributions of various partners. The list of partners invited is found in Annex B. Small group discussions at the workshop focused on roles and contributions of all project partners. The outline for this discussion followed an assessment of human, financial and material resources that each group felt they may be able to contribute. Four groups were formed comprising central Ministry of Health and Provincial Health Department representatives, Operational District and commune health representatives, community representatives and World Vision KSCSP staff and ADP staff. Small groups then fed back their conclusions to the whole group. Provisional agreement on roles and contributions was obtained and incorporated in Annex B. Written agreement was not possible at the DIP workshop as many participants needed to further discuss roles and contributions with their managers or communities. These commitments are still being persued, and the results will be included in the first progress report. Collaboration with Country Partners World Vision Cambodia is committed to working with other NGOs and IOs to achieve the project objectives and to more widely share lessons learned from the project. In Kean Svay OD there are no other NGOs working specifically in maternal and child health. World Vision Cambodia is an active member of Medicam, an NGO body with a mandate to coordinate health activities of over 60 International and Cambodian NGOs. Netweorks have been established involving major in-country partners (National Institute of Public Health, National Pediatric Hospital) and UNICEF and WHO. Others WV and the KSCSP staff will seek close collaboration with BHR/PVC and the CSTS Program in support of the proposed program. Technical assistance will be sought to assure that the training strategy and plan reflect the state of the art. The training expert will be asked to assess the capacity of KSCSP partners and other local institutions to support the training activities, and to assist in the development of the plans for curriculum development and training. Issues identified for potential CSTS assistance are - revised KPC survey methodology, DIP preparation, client-centered project evaluation, consultant identification and networks, IMCI initiatives and lessons learnt from other CSP activities. 6. HEALTH INFORMATION SYSTEM Organizational Approach to Monitoring and Evaluation World Vision 61 Kean Svay Extension Midterm Evaluation This extension project will assist the Kean Svay OD Health Chief to develop a strong Health Management Information System (HMIS) guided by the following principles:(1) avoid duplication of data collected; (2) collect data that can be use for immediate decision-making by the health worker who collects it; and (3) assist in prompt analysis and feedback to communities; (4) strengthen the MOH’s existing data collection system by promoting use of tools that are based on MOH protocols; (5) make better use of data for decreasing barriers to adequate health seeking; (6) streamline that data collection procedure to make the data more useful and utilized; (7) make better use of existing service statistics; (8) increase the the HMIS’s effectiveness in tracking the important work of handing over responsibility for project-initiated activities to partners such as the MOH and partners, and finally (9) is the juice worth the squeeze---does the amount of work done to collect data worth the effort? In phase I KSCSP, community health information was collected by VHVs and passed to the Kien Svay OD staff through WV project staff. The plan now is to facilitate the VHVs to report such data directly to CHC/OD staff. The project is committed to the use of special tools that have been contextualized and are based on MOH protocols as much as possible. Monitoring and Evaluation (M&E) Plan M&E will entail these three activities: 1) Program monitoring through the use of supervisory tools, health management information system, and self-assessments and 2) performance evaluation using KPC surveys and health facility assessment. The M & E plan focuses on tracking the following: a) Monitoring the Four Intermediate Results (IR):  IR #1: Monitoring Coverage/Use of high Impact Child Survival/Reproductive Services  IR #2: Monitoring Quality of training, supervision, information and service  IR #3: Monitoring Capacity Building Efforts  IR #4: Monitoring Progress towards achieving Sustainability b) Monitoring clinical and social causes of infant and child mortality. Community-Based Disease and Death Surveillance: (only for the primary impact area￾Luek Dek District) Information on the social and clinical causes of infant and child mortality for planning will be introduced in Leuk Dek. The project will continue with the Kien Svay OD the community-based surveillance of : (a) childhood diseases (polio, measles and neonatal tetanus). Following discussions with National Immunization Program and Kandal Provincial Health Department a decision will be made regarding the projects role in AFP World Vision 62 Kean Svay Extension Midterm Evaluation surveillance, and (b) deaths in neonates (o-28 days), children 1-59 months, and women 15-45 years. The project will not be creating a whole new HMIS, but will complement the national , state and district and commune level surveillance, which is primarily institutional surveillance. The project will undertake with its partners the following steps in developing the community-based disease and death surveillance (CBDDS) (Annex L￾Steps in Community Surveillance). A revised Data Flow Diagram for Community Surveillance will be initiated, drawing upon the lessons learned in CBDDS from WV’s USAID/BHR/PVC-funded CHAMPs project in the Philippines which reveived TA from Dr Henry Kalter of Johns Hopkins University (please Annex M) . Death Reporting: The first contact worker (VHG, VHV or CHC staff will report immediately any deaths to the CHC/district health chief, who will conduct a verbal autopsy/death investigation to determine the cause of deathProject staff will sensitively explore the possibility of obtaining notification from achars (associate monks) or monks in the project area. Any deaths will be reported immediately to the CHC. The project will liase with the OD Hospital to obtain death notification among under-5 children and women 15-45 years from Luek Dek district. It will also arrange for maintenance of line listing of all neonatal, child and maternal deaths to facilitate summary and to prevent double-counting . The project will discuss with community partners on possible strategies to raise the proportion of neonatal, child and maternal deaths captured by the death surveillance system. Community Surveillance of 3 vaccine-preventable disease cases (polio, measles and neonatal tetanus) will be carried out. To facilitate Case Identification/Reporting, the project will:: 1) Develop simplified definition of a “suspect” case for popular use in the project area in the detection of each target disease: (i) polio: new paralysis in a child (ii) measles: fever and rash in a child; (iii) neonatal tetanus: stiffness or inability to suck in a newborn; 2) Field test/adapt all CBDDS forms to assure that design is optimal. 3) Train CHC/OD staff, VHVs, TBAs and CHDMTs in the above-mentioned simplified case definitions that should detect most of the cases of polio, measles and neonatal tetanus. Consideration will be given to extending this training to youth groups, private practitioners family members of WV sponsored children, WV sponsored children and school teachers 4) Conduct community education through community meetings, CHCs, schools and home visits to ensure that health workers and community members will be able to detect suspect cases. 5) Adapt and distribute promotion materials including ‘memory aids’ and posters within project area; World Vision 63 Kean Svay Extension Midterm Evaluation 6) Instruct community members to immediate report suspected cases of the 3 EPI target diseases to their first contact health worker (VHVs, TBAs, CHC staff or to their closest contact in the CHC). 7) If a decision is made for the project to have a role in AFP surveillance - Create a climate where everyone needs to report every case of AFP 8) As the community definitions are very general, this will result in more reported cases (e.g. reported AFP cases) than actual cases (e.g. case-based AFP with stool positive for polio virus 8)--- the first contact health worker will be instructed to report every suspected case immediately to the CHCstaff 9) Train the CHC staff to promptly (within hours) investigate any suspected cases by filling out a “Suspected Case Identification Form”, which is submitted to the Provincial EPI Officer . 10) Conduct a full investigation in collaboration with the PHC doctor or the PHO using the national case report form for suspected polio and neonatal tetanus cases. 11) If a decision is made for the project to have a role in AFP surveillance- arrange with the OD for stool specimens from suspected polio cases to be taken and sent on ice or in specially designated “reverse cold chain” to the designated polio laboratory at the National Institute of Communicable Diseases. 12) Assist the OD in regular monitoring of timeliness and completeness of reports. Annex N provides copies of CBDDS/verbal autopsy forms and the EPI case investigation forms.. The project will also facilitate more effective use of surveillance data at: (i) the community level--e.g if there is a diarrheal or pneumonia death or NNT death or maternal death , target all pregnant women in the cluster/compartment for intensive health education regarding ORT, Pneumonia recognition and referral; TT or family planning/spacing; (ii) program level--map the geographical distribution of all identified neonatal, child and maternal deaths; (iii) strategic/planning level--recommend family planning as the single most important intervention to reduce neonatal mortality. Current Information in the Community: Village-owned EPI/Pregnancy Registers in the primary impact area (Luek Dek District). The project will facilitate community empowerment by introducing village-owned health registers which will enrol eligible women, children, and newborns to enter and participate in the program. Together with OD/CHC staff, the project staff will facilitate and train the women’s groups, VHVs and TBAs to adapt and adopt two innovative village-owned registers from the USAID/BHR/PVC-funded Phase I KSCSP. They have been very useable in the community and very effective in increasing CS/RH coverage of eligibles. Dr Stan Foster of Emory University has declared the village EPI register as the “World's Best Buy" for immunization recording systems. The two village-owned registers are the: World Vision 64 Kean Svay Extension Midterm Evaluation i) Community EPI (Immunization) Register; (Annex O”) ii) Community Pregnancy Register (Annex P”). The Community EPI Register: This is the same as the CHC EPI register - the major (and crucial) difference is that children are entered by month/year birth - regardless of age at time first presenting for immunizations. Hence there is one page for each month of birth. This way, VHVs, women’s groups and TBAs know that all children on a given page are due for their immunizations at the next session, and that all empty spaces for children preceding that are children who are late. They know that all children in pages for the preceding 9 months should have had their measles. In addition, newly born children will be listed by the VHV each month even if they do not present for EPI (immunization)- and are tracked down if they don't come. In contrast, the usual OD/CHC routine is to list the children sequentially as they present for immunizations - in order to know how many defaulters you have, you must look at each entry, calculate the age, and then determine whether or not they are behind. Community Pregnancy Register: The OD/CHC have a TT register that lists all women (pregnant or not) receiving TT. The project will keep this, but will add a pregnancy register in order to pick up neonatal and maternal deaths, and to make sure all live children got listed in the Community EPI Register. From WV Cambodia’ experience, these two community-owned registers are effective and low cost maternal and child health monitoring system that is maintained and used by villagers to set priorities for health care. These two village monitoring registers are simple, effective tools that enable communities to make decisions regarding their need for CS/RH services. The tools have been be so designed that materials are available locally and at low cost, able to be maintained by users with elementary reading level, in a format compatible with government monitoring systems, and all information collected be useful for those collecting it. VHVs/women’s groups can easily identify individuals in need of key preventive services. By linking pregnancy registers with infants EPI registers, maternal and neonatal deaths can be identified. The development of the two registers will undergo an ongoing process of periodic review and evaluation. The MTE in September 2002 will include a review of the village monitoring system. Findings from sample surveys (such as the KPC survey) will be compared with village registers, and the project/ADP staff and their CHC/OD partners World Vision 65 Kean Svay Extension Midterm Evaluation will conduct focus group discussions with VHVs and women’s groups discuss their understanding and use of the information. Reviews of the Phase I KSCSP have revealed that VHVs using these two tools are able to maintain it with minimal assistance, and are able to use the information for setting health priorities. Communities have ongoing access to information such as current census, names of individuals in need key services, number of maternal and infant deaths, and migration patterns. The communities using this system have made impressive gains in major health indicators. The BRICS project hopes to replicate this success and community empowerment in Ballia and beyond in India. Village-based HIS: In addition, the project will also assist the CHDMTs to choose 5-6 critical indicators that can be (i) visually represented, (ii) tracked regularly by themselves, (iii) result in action and (iv) lead to improved health and decreased mortality of women and children. e.g. (a) Women’s age at first birth--indication of pre-birth interval (women’s age at first birth minus women’s age at marriage)--to promote community/social norms for delayed age of marriage, longer pre-birth and birth intervals, and need for family planning/ spacing; (b) Percentage of active users of modern contraceptive methods (temporary/permanent); (c) Pregnant women--identification for TT, prenatal care, and tracking of pregnancy outcome and post-partum Vit A supplementation; (d) Measles coverage of children aged 9-35 months--to identify leftouts and refer them for measles immunization; (e) Percent of newborns who are of low birth weight (<2500 gm at birth)--to target their mothers for breastfeeding and intensive education in best practices in infant feeding and weaning; (f) Percent of girls finishing primary school; For program management at CHC, commune or village level , the project will assist OD/CHC staff to be coverage-oriented in micro-levels by focusing on ‘spot maps’ and graphs for visual representation of aggregate data on coverage, (rather than just on listings and registers of children immunized or given vitamin A or of women given TT or prenatal care). Monitoring Tools include: Monitoring Tools For Coverage of Interventions Tool Who will develop Who will test/produce KPC Survey USAID/BHR/PVC & CSTS Done LQAS PVO CORE/CSTS PVO CORE Community EPI Register WV/OD WV/OD Community Pregnancy Register WV/OD WV/OD Community FP/Birth spacing register WV/OD WV/OD Monitoring Tools For Quality PLA/FGDs, In-depth interviews, Exit Interviews Done Done World Vision 66 Kean Svay Extension Midterm Evaluation for Customer Appraisal Social Autopsy Form Done WV/OD Health Facility Assessment Checklist Done WV/OD Referral Completion Form Done WV/OD/CHC Supervisory Checklist for CHC WV/OD/CHC WV/OD/CHC IMCI Guidelines/ Protocol/Algorithms to assess, classify, treat, counsel, and refer childhood illness MOH/WHO WV/OD/CHC Missed Opportunity survey form Done WV/OD/CHC Monitoring Practice Ibdicators of CHC Staff WV/OD/CHC WV/OD/CHC Monitoring CHC Facility Performance Indicators WV/OD/CHC WV/OD/CHC Monitoring Indicators for CHC Supersvision by OD Staff OD/CHC/WV OD/CHC/WV Monitoring Indicators for VHV Supervision by CHC Staff OD/CHC/WV OD/CHC/WV Monitoring indicators on caretaker’s ability to recognize danger signs OD/CHC/WV OD/CHC/WV Monitoring indicators on training/referral coverage OD/CHC/WV OD/CHC/WV Instrument to study quality of home and private practitioner/drug seller case management of childhood illnesses BASICS II OD/CHC/WV/NIPH Checklist to monitor twice a day temperature recording of refrigerator Done Done Evaluation tools for community-based monitoring PVO CORE M & E Working Group OD/WV/NIPH Supervisory checklist to monitor interpersonal communication/counseling skills WV/OD/CHC WV/OD/CHC Monitoring Tools for Progress towards Capacity Building Appreciative Inquiry Done OD/WV Discussion-Oriented Self- Assessment (DOSA) Done WV/OD Negotiated Competency-based assessment with checklists WV WV 360 degree performance assessment Done WV Team Climate Assessment Done WV Monitoring tools for Progress towards Sustainability Ratings on Focus Group Discussion Responses re: Community Participation Done WV Post –Grant Sustainability Assessment/Review Tool (2 years after BHR/PVC funding stopped) OD/WV/CSTS/CORE WV/ADP/OD/CHC FGD responses using scorecard; document review Done WV World Vision 67 Kean Svay Extension Midterm Evaluation Attachment B Terms of Reference for the Midterm Evaluation Introduction: World Vision Cambodia, in collaboration with World Vision, Inc. (WVUS), seeks to use the services of the Independent Consultant named above to be Team Leader for the Mid￾Term Evaluation of the Kean Svay Child Survival (Follow-On) Project as detailed below. The Kean Svay Child Survival Project is a USAID funded project implemented by World Vision Cambodia. The project completed a first phase of four years in October 2000 and is now at the mid-term of a three-year follow-on project. The principal project activities are facilitating the implementation of key child survival interventions (immunization, control of diarrheal disease, pneumonia case management, child spacing and nutrition and micronutrients) and capacity building of local partners. Key Objective: The Consultant will supervise the process of the MTE and prepare a written MTE Report. Specific Tasks: Specific objectives of the mid-term evaluation are as follows: 1. Assess the progress towards achieving program results and impact, specifically the technical, cross-cutting, capacity building and sustainability components of the project. 2. Assess the strengths and weakness of the program management support systems at HQ, Regional, field level, with partners and with the community. 3. Address other issues identified by the MTE Team and other project partners. 4. Document lessons learned and promising practices. 5. Develop conclusions and recommendations. 6. Write up a one page “Results Highlight” on selected elements of the project with supporting data that would make a good communication piece for World Vision or USAID. 7. Assist project staff and partners in developing an Action Plan to implement the MTE recommendations. 8. Write up and submit the MTE Report within 30 days of the completion of the MTE . World Vision 68 Kean Svay Extension Midterm Evaluation Expected Output: 1. MTE report as per the USAID MTE guidelines (CS-15, 1998-2002) 2. One-day workshop for feedback and lessons learned 3. Debriefing with key stakeholders 4. Other recommendations for the future of KSCSP Process: The Final Report will be prepared in Microsoft Word (spreadsheets in MS Excel) and an electronic copy will be submitted to all contact persons named below. Three hard copies will be submitted to Mr. David Grosz. Timeline: 30 August to 6 September 2002 + 2 days for preparation and 10 days for in country travel, and 3 days for postwork and report writing time. A draft timetable has been prepared and is summarized here: Day 1: orientation and expectations, task assignments, review/ develop field instruments Day 2: preparation for qualitative methodologies, records review, staff interviews Day 3: Field visit Day 4: Field visit – debrief, analysis Day 5: Analysis, key findings and recommendations, feedback to staff Day 6: Preparation for feedback workshop Day 7: Feedback to partners workshop Day 8: Feedback to senior World Vision Cambodia management and to USAID. Team Composition: Team Leader: Dr. Marc Debay. Team Membership: Representatives of Ministry of Health at Operational District and Provincial levels, District Local Government, Commune Council leaders, Village Development Club members, mothers, fathers, mothers-in-law, private practitioners (drug sellers, traditional midwives, traditional healers), World Vision Area Development Program staff from Leuk Dek and Kompong Thom, USAID, other Child Survival NGOs, World Vision Cambodia senior management, health technical support staff, Regional Health Advisor and WV United States. Contact Persons World Vision Cambodia: Dr Douglas Shaw, National Health Advisor, World Vision Cambodia Dr Lim Somaly, Project Manager, Kean Svay Child Survival Project Dr Ly Vanthy, National Health Program Technical Support Contact Person, World Vision United States: David Grosz, Program Officer, World Vision United States World Vision 69 Kean Svay Extension Midterm Evaluation Attachment C List of key project documents Kean Svay Child Survival (Follow-On) Project Detailed Implementation Plan, World Vision Cambodia, April 2001 and Reviewer’s Comments Annexes to Kean Svay Child Survival (Follow-On) Project Detailed Implementation Plan, World Vision Cambodia, April 2001 (includes results of KPC Survey) Kean Svay Child Survival Project Final Evaluation (Phase 1), World Vision Cambodia, September 2000 Kean Svay Child Survival (Follow-On) Project First Annual Review, World Vision Cambodia, October 2001 Monthly and Quarterly Project Reports Quantitative and Qualitative Survey reports/ results USAID PVO Child Survival Grants Program: Guidelines for Mid-Term and Final Evaluations CS-15, 1998-2002. Community-based Disease and Death Surveillance: System Operation, Verbal and Social Autopsies. Consultancy report, Sophal Oum. May 2002. World Vision 70 Kean Svay Extension Midterm Evaluation Attachment D MTE team composition WVC and External MTE Team Members NAME POSITION AFFILIATION WVC staff Douglas Shaw National Health Advisor WV-Cambodia Ly Vanthy Technical Support Coordinator, NHP, WV-Cambodia Lim Somaly KSCSP Manager WV-Kean Svay Hout Sok Da Health Officer WV-Kean Svay Kan Chamroeun Health Officer WV-Kean Svay Huy Dany - Health Officer WV-Kean Svay Pen Sophea Training Coordinator WV-Kean Svay Keo Sereivuth Monitoring Coordinator WV-Kean Svay Mr. Ngoun Pheap Leuk Dek ADP Leuk Dek District External evaluators David Grosz Program Officer WVUS Shri Chander Regional Health Advisor WVAPRO Lady Walakandou Medical Officer MOH Indonesia Sidupa Untung ADP Manager WV Indonesia Marc Debay MTE Team Leader Johns Hopkins University World Vision 71 Kean Svay Extension Midterm Evaluation MTE Kean Svay/Leuk Dek Core Group On 30 and 31 August and on 2 and 5 September 2002 No. Name Sex Function Location 1 Dr. Douglas Shaw M National Health Advisor WV-Cambodia 2 Dr. Marc Debay - MTE Team Leader Johns Hogkins 3 Dr. David Grosz - WVUS USA 4 Dr. Lady Walakandou F Indonesia Indonesia 5 Dr. Sri Chander M WVAPRO/Singapore Singapore 6 Mr. Untung - ADP Manager WV Indonesia 7 Dr. Hong Rath Mony - CDC/MoH Ministry of Health 8 Dr. Prak Phan - Vice Chief PHD Kandal PHD 9 Mr. Tuy Saroeun - Chief of OD Kean Svay Kean Svay District 10 Mr. Yi Sophoanarith - Vice Chief OD K. Svay Lovea Em District 11 Mr. Bin Samrith - Vice Chief OD K. Svay Leuk Dek District 12 Dr. Ou Chandaramony - Chief of Technical Kean Svay District 13 Mr. Yun Sa Eng - Chief of HC Kg. Phnom Leuk Dek District 14 Mr. Mak Sakhoeun - Chief of HC Prek Tunloap Leuk Dek District 15 Mr. Teang Sarath - Chief of HC Sandar Leuk Dek District 16 Ms. Hem Samorn F VHV Kg. Phnom Leuk Dek District 17 Mr. Hean Soun M VDC Sandar Leuk Dek District 18 Ms. Sok Samnang F Commune Council Leuk Dek District 19 Mr. Sin Samai M ADRA Kg. Thom Provin. 20 Mr. Ngoun Pheap - Leuk Dek ADP staff Leuk Dek District 21 Dr. Lim Somaly F KSCSP Manager WV-Kean Svay 22 Ms. Hout Sok Da - Health Officer WV-Kean Svay 23 Ms. Kan Chamroeun - Health Officer WV-Kean Svay 24 Ms. Huy Dany - Health Officer WV-Kean Svay 25 Dr. Pen Sophea M Training Coordinator WV-Kean Svay 26 Mr. Keo Sereivuth - Monitoring Coordinator WV-Kean Svay World Vision 72 Kean Svay Extension Midterm Evaluation MTE Kompong Thom Core Group On 2 and 3 September 2002 No. Name Sex Function Location 1 Dr. Ly Vanthy M Technical Support Coordinator, NHP WVC 2 Dr. Yim Yaren M Operation Manager WVC-KPT 3 Mot Sana M P. Sambo ADP Manager KPT 4 Leng Virak M P. Balang ADP Manager KPT 5 Khieng Kunthamith F Sandan ADP Manager KPT 6 Mey Socheat M Health Coordinator, P.Sambo ADP KPT 7 Yim Pivatho. M Health Coordinator, P.Balang ADP KPT 8 Pen Sophea M Training Coordinator, KSCSP Kien Svay 9 Keo Sereyvuth M Monitoring Coordinator,KSCSP Kien Svay 10 Huy Dany F Health Officer, KSCSP Kien Svay 11 In Torn M Commune Development Committee Mean Chey 12 Tep Sarkin M HC Chief Taing Krasao 13 Ty Torn M HC Staff Taing Krasao 14 Kock Phat M Commune Development Committee Tul Kroeul 15 Sin Samay M ADRA Staff KPT 16 Chan Samneang F HC Staff Mean Chey 17 Leng Thy M HC Staff Chouk 18 Po Virath M HC Staff Sambo 19 Nou Puthy M HC Chief Mean Chey 20 Prom Vay M Village Development Committee Mean Chey World Vision 73 Kean Svay Extension Midterm Evaluation Attachment E Project implementation timeline: October 2000 – August 2002 2000 2001 2002 Selected Project Activities O N D J F M A M J J A S O N D J F M A M J J A Project management Baseline assessment x x Midterm KPC Leuk Dek x DIP workshop / review / approval x x x FAR / MTE x x Consultancy BCC / CBDDS x x Capacity building Training Community members- Leuk Dek AD --Village Health Volunteers x x x x x x x x x x x x --Traditional Birth Attendants x x --Drug Sellers x x --Traditional Healers x x Community members-Kean Svay AD --Village Health Volunteers x x x x x x x x x Health Center staff—KS OD x x x x x x x x x x x x Health Center staff—3 KPT AD x x World Vision 74 Kean Svay Extension Midterm Evaluation Attachment F MTE Assessment Methodology World Vision Cambodia and the Kean Svay Child Survival Project (KSCSP) staff had prior experience with midterm and final evaluations, and had conducted a First Annual Review (FAR) of the project in September 2001, one year before the Midterm Evaluation (MTE). During the FAR, they developed a methodology and tools that involved an intense participation of partners, and consisting primarily of Focus Group Discussions (FGD) with various project partners and beneficiaries, and of standardized observations of clinical practices, exit interviews, and record reviews in health centers. The KSCSP adopted this methodology and tools for the MTE. Prior to the MTE, the KSCSP staff prepared all the relevant data on project activities and a briefing package for the external MTE team members including summary tables on progress on recommendations from the FAR, on the capacity building plan and on the sustainability plan (see Appendices in this report). The KSCSP staff also prepared a detailed program and organized all the related logistics. The fieldwork of the MTE ran from August 30 to September 6, 2002. The composition of the MTE team is presented in Attachment D. There were 9 WVC and KSCSP staff members and 5 external evaluators. Two larger groups including various partner representatives prepared and conducted the fieldwork in Leuk Dek (the Leuk Dek Core group, 26 participants) and in Kampong Thom (the Kampong Thom Core group, 20 participants). After orientation of the external MTE team members on the day of their arrival (11/29), the Leuk Dek Core group revised the questions used for the FAR during a one and a half day workshop in the KSCSP Leuk Dek office. For the site visits, the MTE external evaluators divided into two groups (one remained with the Leuk Dek Core group and the other one joined the Kampong Thom Core group) to conduct the MTE data collection and analysis in this area of the project. The field visits and data collection took two days. The list of questions was used as guidelines with representatives of 9 different groups of project partners and beneficiaries. Sites visits were conducted in 3 health centers in Leuk Dek and 3 health centers in Kampong Thom. A total of 16 interviews/discussions were conducted in KS and 12 in KPT. Attachment G presents the revised list of questions for the various FGD and the responses from Leuk Dek, the revised FGD questions and responses from the health centers in Kampong Thom, a summary of all the FGD prepared by the Kampong Thom Core group, and the results from the observations, exit interviews and record reviews in the health centers in Leuk Dek and Kampong Thom. After the two days of data collection, the key members of the two groups from Kean Svay and Kampong Thom met in Leuk Dek to present and discuss the results and findings by type of interviewees. A smaller team of senior WVC and KSCSP staff and external evaluators met the next day to draw preliminary conclusions of the MTE, using the findings from the field visits as well as other sources of information such as the January 2002 KPC, the LQAS, quarterly HC staff checklists, and the Organizational World Vision 75 Kean Svay Extension Midterm Evaluation Capacity Assessments. These discussions were organized by main topic of the MTE as needed for the report: technical interventions, cross-cutting approaches, and program management. Various other tasks (small group discussions, documents review; compilation of data; preparation of the feedback presentation the next day) were performed throughout these two days of analyses. Preliminary conclusions of the MTE were presented on November 6 during a half-day meeting in the Leuk Dek office attended by about 50 participants, including representatives from the other ADs of the KS OD and from various organizations at the national level in addition to MTE members. Selected senior MTE members from WV and the external consultant then debriefed with USAID health officers and with senior management staff from WVC. The external consultant prepared the MTE final report. The MTE methodology based on a series of FGDs and observations, and involving a large participation from various partner representatives, allows for a broad review of issues relevant to the participants and implementers of the project. The numerous discussions and intense exchange of information that this process generates certainly makes it very worthwhile to all. Such a large group of participants with different background, perspectives and interests in the project makes it difficult to analyze and interpret the findings of the various FGD and observations, or even to interpret the conclusions and recommendations of the various groups. Below is an indicative list of difficulties identified and that should be kept in mind in future evaluation of this type.  Analysis and interpretation of findings is part of the participatory process, and takes time.  The sample of sites and interviewees may not be representative.  The ideas, responses, or even consensus expressed in a group may not be representative of the group.  The interviewers and interviewees may have their own bias (interviewers see or report what they want to see or be reported, interviewees say or do what they think the interviewer want to hear or see). Some of these difficulties may be alleviated as follows:  Clearly define the issues or problems to be solved before designing the data collection scheme, such as the population to be studied, the selection of the sample, or the questionnaire.  Consider the potential sources of biases during the design of the data collection scheme, and document the decisions made and what really happened.  Plan the time needed for participatory analysis and interpretation, and document the process. World Vision 76 Kean Svay Extension Midterm Evaluation Attachment G MTE Results from Field Visits The following results (reorganized and slightly edited for this report) were available at the end of the field visits and before the smaller team of senior WVC and KSCSP staff and external evaluators met to draw preliminary conclusions of the MTE. 1. Results from individual Focus Group Discussions in Leuk Dek (Mothers; Fathers; Village and Group Leaders; Village Health Volunteers; Traditional Birth Attendants; Commune Council Leaders; Operation District staff; Leuk Dek ADP staff) 2. Results from Focus Group Discussions with Health Center staff in Kampong Thom 3. Summative results from Focus Group Discussions in Kampong Thom prepared by the Kampong Thom Core group 4. Health Center Assessments in Leuk Dek and Kampong Thom _________________________________________________________________ 1. Results from individual Focus Group Discussions in Leuk Dek Mothers 1. Do you know what services are provided at the HC?  General consultation: DHF, Fever, Pneumonia  Antenatal visit  Delivery  Birth spacing, EPI on the 6 diseases for children and women  Health education 2. Nowadays, there are many patients go to seek consultation?  Yes, there are many and more than before 3. Since the project has been started (mid 2001), are there any changes the services provided by HC?  Yes there are changing 4. If yes, what are they?  Service is available 24 hours  Staff are changing their attitude in communication with patients  Staff work regularly  More good medicines  Treat effectively compared to the previous time 5. Nowadays, how is the quality service of the HC? Is there better before the project has started? World Vision 77 Kean Svay Extension Midterm Evaluation  Treat effectively compared to the previous time  There are good materials  Staff are more capable and knowledgeable  Less death cases  Less DHF cases  Less referral cases to further health facilities 6. How the fees for services of HC? Is it cheap or expensive ?  Not expensive  Appropriate  Exemption for poor patient  HC has tariff of the payment 7. What do you think about the fees for the services?  Satisfy to pay, because it is cheaper than buy from market  With instruction on using medicines from HC staff  Staff have been trained and give good medicines (not harm to our health)  Fees for delivery is cheap as well 8. Did you ever see HC staff come to your village?  Yes every month  Give immunization  Give birth spacing services 9. Do you know your village health volunteers (VHV)?  Yes, Mr. Sok Samnang and uncle Rin. What work does he/she do?  Gather mother for immunization sessions  Educate mother about birth spacing, first milk, disease prevention (DHF, pneumonia)  Visit women who are using birth spacing  Advice caretakers, take sick children to hospital  Give Vit. A post-partum women  Put Abat in water container  Sell iodized salts 10. Do you have any idea or opinion in order to improve HC services?  Give more medicines  HC come regularly to village: 4 times per month  HC staff respond quickly when patient comes  Precisely examine the patient  Use disposable syringe in order to avoid contamination  Continuously putting Abat World Vision 78 Kean Svay Extension Midterm Evaluation  Supply cover container  Continuously experiment the water well for Arsenic and provide safe water Recommendations  Strengthening duty hour service (24 hour per day)  HC staff should respect the working hour  Strengthening the sterilization according to national protocol for immunization  Strengthening the standard case management of HC staff Fathers 1. Do you know what services are provided at the HC ?  Delivery.  ANC and PNC  Treatment of ARI,CDD and DHF.  Birth spacing. 2. Are more people using the health services at the HC nowadays?  Yes. More patients in the morning 3. Has the HC services changed since WV began supporting the HC?  Better than previous. 4. If yes. What are the changes?  There is staff at the HC all the time.  Good relation with patients.  Enough drug and equipment.  Many services available at the health centers.  Means of referrals 5. Is the quality of the HC service nowadays (examination of patient/case management) the same or better since WV began supporting the HC?  Better quality in patient examination.  Timely refer of severely sick patients.  Decrease child mortality rate. 6. Is the cost of the service at the HC expensive or not?  Cheap. 7. What do you think about the fee at the HC?  It is consistency to the level of income of villagers.  Help the villagers to save money.  There is exemption for the poor. World Vision 79 Kean Svay Extension Midterm Evaluation 8. Have you ever seen the HC team in your villages?  Yes. ( EPI, BS, Tetanus vaccination, Vit A ) 9. Do you know the VHVs in your village? What work does he/she do?  Yes. What work does he/she do?  Do health education in the villages.  Gather children for vaccination.  Distribution of abate.  Provide Vit A for post partum mothers.  Advise on use of ORS.  Keep record of birth and death. 10. Do you have any suggestion or ideas to improve the center services?  Bed for patient admission to the health center.  Care of severely ill patients (snake bite, drug poison)  HC to remain in good quality service, especially when the project terminates.  Provide training for HC to strengthen their capacity. Recommendation: In order to improve the quality of HC service and sustain, the HC must:  Support from villagers, OD and PHD  Strengthen the role and responsibility of HC chief and vice chief especially in the management field.  OD management teams to get management training from national level so that OD can back up and strengthen the HC.  Superior level should fulfill the request from inferior (HC)  HC staff must have willingness and ethnic of health personnel.  Continue to have regular meeting.  Improve relationship between OD+HC+FBC. Village and Group Leaders 1. Do you know what services are provided at the HC?  Treatment services: OPD, TB, Delivery, Ante-natal care, Admission, Malaria,.STD  Preventive Services: Birth spacing, Health Education, Immunization against the 6 diseases, Distribution of abate. 2. Are more people using the health services at the health center nowadays?  Yes. There are more people coming to health center. 3. Have the health center services changed since WV began supporting the HC?  Yes, there are changes. World Vision 80 Kean Svay Extension Midterm Evaluation 4. If Yes. What are the changes?  New building  Efficacy of treatment.  There is motor for patient referrals.  There are VHVs  There is IEC, TV show in the villages.  Formation of mother’s group and children’s group.  Good equipment.  Meeting between HC staff and community about health issue.  There is staff at the health center for 24 hours.  Villagers know more about health issue.  Decrease in morbidity and mortality (especially disease outbreak)  There is a reporting system. Community to HC and HC to community (via VHVs) 5. Is the quality of the HC services nowadays (examination of patients/case management) the same or better since WV began supporting the HC?  Better (there are more patients)  Can be admitted at any time. 6. Is the cost of the service at the health center expensive or not?  Cheap (some people are in doubt about the charge for health services.) 7. What do you think about the fees at the HC?  It is good because it can help support the activities of the health center 8. Have you ever seen the HC team in your village?  Monthly (Good activities) 9. Have you ever collaborated with the HC so far?  Participate in many activities: collection of mothers, measles campaign, Vit A, encourage mothers to seek service at the health center. 10. Do you have any requests or ideas regarding the services delivered at the HC?  Provide training to HC chief in management.  Project to provide enough equipment for the health center.  Project to help strengthen the capacity of VHVs.  HC staff to pay more attention to severe and emergency cases.  Request for exemption for the poor, orphans, handicapped and unaccompanied minors.  Make sure there is staff at the health center for 24 hours.  Disseminate information about the fee for services to the villagers. 11. After the project finishes in 12 months time, how will you be able to support the HC to sustain its function? World Vision 81 Kean Svay Extension Midterm Evaluation  Will collaborate and continue support every HC activities.  Continue good relationship with health center.  Help to protect the achievement of the health center.  Help to encourage villagers to use the HC services. Recommendation:  Training of HC chief and vice chief in Management, leadership and planning.  Strengthen capacity of FBC.  To make the cost recovery scheme going well.  Training of the health center staff to fulfil the work dimension.  OD support to HC (Drug, Budget, MIS) Village Health Volunteers (Combined with VHVs who are also VDC members) 1. Describe your relationship/communication with the HC staff?  Monthly meetings  EPI, BS, Abate distribution, Condom distribution, Health education using IEC material, report of any outbreak and death. 2. Describe the health activities you are participating in nowadays?  Explain about ARI, CDD, DHF.  Explain about advantages of Birth Spacing, Iodine, and colostrum feeding.  EPI activities, mixing ORS demonstration, showing how to use condom.  Record of newborn and death.  Provision of Vit A to postpartum mothers. 3. Is the work too much or not?  Moderate. 4. Will you be able to continue to improve in this work in the future?  Be able to 5. TO VDC member only : Describe the relationship between your VDC and HC? 6. What problems have you encountered in doing the work?  Difficulty in referring sick patient from HC to RH (poor people)  Villagers have poor knowledge of health.  Vaccination is irregular (resettlement, sick) 7. How have you used/applied the health knowledge you have received in your training in your work in the community?  One to one education.  Meeting or group.  IEC materials. World Vision 82 Kean Svay Extension Midterm Evaluation 8. What additional training do you need in the next 12 months?  First aid: drowning, snake bite, drug poison.  Knowledge of hypertension  Refreshment training 9. When the Mother and Child Health Project of WV stop in 12 months, what do you think about your continuing relationship with HC and Community?  Remain the same. 10. What do you think about the use of the reports that you complete?  Useful because it can tell the information of the community.  Request for death tables.  Feedback given by the health center. ADDED DURING ONE FOCUS GROUP : What tasks that VHVs currently do are best done by male VHVs? What are the best done by female VHVs?  Male VHVs: Convenience to work. Report on time and more talkative.  Female VHVs: Convenience to advise women in term of birth spacing, know better in term of maternal and child health. Recommendation:  There should be more VHVs in a large village.  Should provide first aid training: - Drowning - Snake bite. - Drug poison and - Syncope.  Refreshment training  Should provide material for health education (loud speaker)  Increase the transport allowance. Traditional Birth Attendants 1. After being trained. Have you applied these newly acquired skills in your community? If Yes. What have you done? If No, for what reasons?  When there is newborn report to VHVs  Breastfeed within one hour after birth.  Advise mothers to take children for vaccination against 6 diseases.  Advise mothers on how to care for children.  Pregnant women to go to HC for ANC.  Women with too many children to accept birth spacing.  Advise mothers to eat enough food during pregnancy.  If dystocia refer to health center.  After delivery mothers have to eat enough food (not to fast)  Provision of Vit a to < 2 months newly delivery mothers. World Vision 83 Kean Svay Extension Midterm Evaluation 2. How do you communicate with (what is your relationship with) the HC staff and Feedback Committee (when having difficulties; giving feedback, meeting, participate with different HC activities.....?)  If dystocia, refer to health center.  If there is any doubt about health related problem, check with HC staff.  Monthly report about newborn.  Ask for more Vit A if needed. 3. When the Mother and Child health project of WV stop in 12 months, what do you think about you continuing relationship with HC and community?  Relationship and collaboration remain the same. 4. Do you have any request or suggest?  Want to learn how to resuscitate a new born with asphyxia.  Suction device  Gloves.  Cord tie  Bandage.  Cotton wool (to clean infant eyes). Recommendation:  Refreshment training  Invite TBA to participate monthly meeting with VHV.  Motivation and response to their requests or needed. Commune Council Leaders 1. Do you know what services are provided at the HC?  General Consultation - DHF, diarrhea, fever, respiratory infection, - Other diseases outbreak.  Birth spacing, antenatal care.  Immunization against 6 diseases, TT for mothers.  Health education and patient referrals. 2. From whom do people seek help when anyone in the family gets sick –Private practitioner or health center?  Majority will go to HC 3. If they don’t go to the health center, why?  Many private practitioners at Prek Tunloap.  People who can afford use private practitioner.  Private practitioner can be invited home. World Vision 84 Kean Svay Extension Midterm Evaluation  There are not enough drugs at the health center.  Private practitioner gets enough drugs.  Distance from the health center.  HC lack of information dissemination.  Villagers are not well understood about fee for services.  Some HC staff is not well communicated with villagers. 4. Since the project started activities 12 months ago are there any changes in the Health Center Services? If Yes. What are the changes?  Better than before.  More patients visit.  There is always staff at the health center.  People are more confident with HC staff.  Enough drug  Some HC staff changes their behavior. 5. Have the quality of examination been improved?  Better than previous 6. Is the cost of the service at the health center expensive or not?  Cheap (some people are in doubt about the charge for health services.) 7. What do you think about the fees at the HC?  Comparing to Private price it is cheaper.  Villagers can afford to pay.  There is exemption for the poor. 8. Have you ever reported to HC or project about health problems.  If there is outbreak in the village. We always report to the HC for intervention. 9. When the Mother and Child health project of WV stop in 12 months time- what do you think you can support the health center?  In the future if the commune can diversify income, we will allocate a sum of money to the health center.  CC will replace current VHVs who are members of the council 10. Do you have any request or suggest?  HC to collect data from private clinic in term of DHF, Diarrhea.  HC staff to take good care patients.  The project to continue. Recommendation There are many private clinics in Prek Tunloap commune, in order to convince the people to use PHF the Health center staff:  There is staff at HC all the time. World Vision 85 Kean Svay Extension Midterm Evaluation  Be punctual.  HC staff strengthens the technical capacity.  Internal solidarity.  Broadcasts about the services provide at the health center.  Clearly define and follow the price of HC services.  Support and care for partners (VHVs, Health center management committee) Operation District staff 1. In the last 12 months, what do you think have been the major accomplishments of the KSCSP (Phase 2)? What has the project done well?  More knowledge about disease and prevention.  Good immunization coverage.  Establishment of Feedback Committees 2. What factors do you think have contributed to this success?  Good collaboration, skill development, planning, and community participation. 3. Have you conducted supervision activities with the project?  Yes, Three supervisions team. 4. What has been your experience with this supervision?  Use of checklist, clear planning, feedback to staff, problem-solving strategies. 5. What kind of support has the project provided for the supervision?  Financial, checklist, transportation and human resources. 6. How many supervisory visits has the OD made in the last three months?  Follow-up after training session one time. Regular supervision, three times. 7. Is the Supervisory Checklist useful for supervision? If yes, describe how it is useful.  Yes, capacity of HC staff, evaluation. Activities of HC, est. progress, and problem solving . Actions plans. 8. So far, how has the project helped you in achieving the tasks of the OD?  Capacity building, financial support (equipment), sharing experience (participation and study tours) 9. What kind of additional support does the OD need from the project in order to improve the function of the OD?  Team building, short courses for HC, more support from VHV, extend project, annual review. 10. In what areas do you think the OD and project need Technical Assistance? World Vision 86 Kean Svay Extension Midterm Evaluation  No needs identified. 11. Do you have any difficulties in collaborating with the project?  12. What ideas/ suggestions do you have to strengthen collaboration between the OD and the project?  Continue the project. 13. After you have received training, have you applied this knowledge? If yes, what activities have you done where you have applied this knowledge?  Transfer of knowledge to others for use in supervision, planning, and management. 14. After you receive information from the HC, what do you use this information for?  Share with technical health committee, planning for epidemics, annual reports, statistical reports. Leuk Dek ADP staff 1. In the last 12 months, what do you think have been the major accomplishments of the KSCSP (Phase 2)? What has the project done well?  Immunization against the 6 diseases.  VHVs & VDC training.  Surveys  Provision of transport to VHVs.  Participate in LQAS.  Birth Spacing  Integrate with ADP.  Formation of Feedback Committee. What the project has done well.  Provide training as planned.  Formation of FBC, Health Center Management committee  Survey.  Management Information system  Good participation (National Level, PHD, OD, and community.  Good outreach activities.  Enough budget. 2. What factors do you think have contributed to this success?  Enough budget.  Good participation from partners.  High commitment.  Respond to the needs of community. World Vision 87 Kean Svay Extension Midterm Evaluation  Implementation according to plan. 3. What is your understanding of the plans for integration of the KSCSP with the Leuk Dek ADP?  Convenience in communication between project, ADP,and Community and share information on time.  Maintain the same activities.  Learning from each others.  Show one WV identity.  Accelerate the process of community development. 4. What are the difficulty of integration of ADP and KSCSP.  Lack of experience regarding integration.  Lack of health related skill and community development.  Add more work for ADP.  Cannot respond to the need of community because of limited resource. 5. In what ways that you think can help the integration of ADP and KSCSP going well.?  Regular meeting  Participate in planning.  More involvement with health activities.  Regular sharing report with each other.  Share information about the integration to partners (VHVs) 6. What are the additional resources does ADP need to have the HC running well?  Need more staff.  Need more funding  Commodities. 7. In what areas do you think the project needs Technical Assistance?  Technical knowledge related to health especial mother and child health. Recommendation: In Order to get a good integration:  Regular meeting between APD and KSCSP.  Participate in planning.  The contact person (representative of ADP and KSCSP) should be involved in implementation of major activities of both programs.  Strengthen the capacity of partners  Share report regularly.  Hand over the responsibility of activities planned.  Standard of allowance given to partners? World Vision 88 Kean Svay Extension Midterm Evaluation 2. Results from Focus Group Discussion with Health Center staff in Kampong Thom 1. Changes in Case Management Practice of Under-5 Children with ARI and diarrhea.  Assess, classify and treatment of the two diseases improved according to the protocols  Provided education and counseling after treatment  Have documents from the training for referring while there were issues or doubt 2. How HC Staff have Use Data reported by VHVs and Community  It is important information that can help us to know the event in the community.  We used the information for responding to the events, for instant outbreak of measles after receiving the information we investigated (sometimes with ADP health staff) then responded to the outbreak and reported to the OD for supporting 3. Nature of Supervisory Visits from OD Team  The OD team supervisor came to supervise us on management, EPI, B/S, Drug, Laboratory for TB, Malaria.  They conducted visits every 3 months 4. Problems in referring sick children  Low socio-economic status  Bad reputation from the referral hospital: expensive, staff behavior and too more died after referral to the RH.  Hard to find the transport at night time 5. Types of records kept by HC Staff  Consultation's register, in patients (former district hospital has inpatient) register, ANC and B/S's register, Visitor's book, Financial book, Drug register, TB and HIV register, laboratory register, minute of the meetings 6. HC needs before project phase-out and HC plans to sustain its functions after project phase-out.  Strengthen the capacity of VHV  Provide refresher training courses on ARI/CDD/DHF, B/S and EPI  Generators for lighting at night time  Well  Ambulance  Strengthen the capacity of laboratory technician  Provide health center management 7. Types of HC support for VHVs after project phase-out  HC provided refreshments for the meeting and travel allowance for the meeting  Provide training to them  Exempt the cost recovery for members of the VHV's family World Vision 89 Kean Svay Extension Midterm Evaluation 3. Summative results from Focus Group Discussions in Kampong Thom Results compiled in terms of strengths and weaknesses by the Kampong Thom MTE Core Group and on the basis of the following Focus Group Discussions: 1). P. Sambo AD/ADP One FGD, observation and exit interview at one HC One FGD with mother group One FGD with father group One FGD with VHV/VDC 2). P. Ballang AD/ADP Two FGDs, observation and exit interview at two HCs One FGD with mother group One FGD with father group One FGD with VHV/VDC One FGD with Commune council 3). One FGD with World Vision Kampong Thom staff The Kampong Thom MTE Core Group also prepared an Action Plan on the basis of their findings (Not presented here) Strengths  Consultation and treatment at HC have been done quite well  Health education, and counseling to the mother have been provided after consultation.  Number of out and in- patients has increased  Health information collected and provided by VHV was useful and being used  Supervision from OD has done in the last 3 months  Mothers new health services provided by HC in their community  Mothers knew the VHV in their village and VHVs activities  Mothers went to HC when the members of the family were sick.  Health services provided by HC are better than before.  Cost recovery was introduced to HC, but not yet official.  Good cooperation between WV projects and partners (team work)  WV projects have received support from projects' partners.  WV projects and partners have shared lessons learned and experiences one to the other.  Learn and understand more about the methodology and the effective follow up on standard case management Weaknesses  HC did not keep the HC and VHV minute meeting (just send to WV's ADP) World Vision 90 Kean Svay Extension Midterm Evaluation  HC did not check the understanding of the mothers after providing the health education messages or any instructions.  HC did not tell mother about the causes of disease and how to prevent the disease.  Classification of diseases by HC staff was limited  There is discrimination between the poor and wealthy patients  Quality of health services of HC was limited  Health education activities in the community was limited.  Utilization of HC services was limited.  Supervision from OD to HC was limited (usually one every 3 months, but sometimes 4-5 months).  Sometimes HC closed at the working hours.  Cost recovery was not yet well function  HC support HC was limited  The general knowledge, health education skills of VHV was limited  Plan and schedule between OD/PHD, Kg. Thom and Kean Svay Project was not clear  Annual planning should be done in the beginning of the year  Job responsibility of each partner should be clearly defined before starting working together World Vision 91 Kean Svay Extension Midterm Evaluation 4. Health Center Assessments in Leuk Dek and Kompong Thom Observations of Practices PRACTICE INDICATOR Leuk Dek 5 cases Kampong Thom 6 cases # % # % ARI CASE MANAGEMENT Sick under-5 children with ARI who were checked for danger signs 4 80 4 83.3 Sick under-5 children with ARI who had their respiratory rate checked and recorded 4 80 6 100 Sick under-5 children with ARI who had their vaccination status checked; 3 60 3 50 Sick under-5 children diagnosed with pneumonia by CHC staff and who were given antibiotics 5 100 6 100 Sick under-5 children the upper respiratory tract infections (cough or cold) who were given an antibiotic inappropriately; 0 0 0 0 Sick under-5 children with pneumonia needing referral and who were referred appropriately NA NA NA NA Mothers/caretakers of sick children with pneumonia who were given advice on giving more fluids, food or breastmilk at home. 5 100 5 83.3 Mothers/caretakers of sick children who were advised to return if danger signs of pneumonia emerged 4 100 4 66.6 PRACTICE INDICATOR Leuk Dek 2 cases Kampong Thom 6 cases # % # % DIARRHEA CASE MANAGEMENT Sick under-5 children with diarrhea who were checked for danger signs; 2 100 6 100 Sick under-5 children with diarrhea who had their skin turgor Checked 2 100 4 66.6 Sick under-5 children diagnosed with diarrhea by CHC staff and who were given ORS 2 2 5 83.3 Sick under-5 children diagnosed with simple diarrhea who were given an antibiotic or antimotility agent inappropriately 0 0 1 16.6 Sick under-5 children with diarrhea needing referral and who were referred appropriately. NA NA NA NA Mothers/caretakers of sick children with diarrhea who were given advice on giving more fluids, food or breastmilk at home. 2 100 5 83.3 Mothers/caretakers of sick children with diarrhea who were given advice to return if danger signs emerged 2 100 5 83.3 Mothers/caretakers of sick children with diarrhea who were given Information about the cause and prevention of diarrhea 0 0 4 66.6 World Vision 92 Kean Svay Extension Midterm Evaluation Exit Interviews With Mothers/Care-givers Lek Dek 5 mothers Kampong Thom 2 mothers # % # % ARI CASE MANAGEMENT Recall of advice on antibiotic treatment of pneumonia Told of correct dosage 5 100 1 50 Told of number of times per day to give antibiotic: 5 100 2 100 Told of duration of treatment: 4 80 2 100 Told to return in two 2 days: 3 60 0 0 Recall of home care advise Return if breathing is fast: 5 100 1 50 Return if breathing is difficult: 5 100 1 50 Breastfeed more frequently: 3 60 1 50 Feed during illness: 5 100 1 50 Give more fluids: 3 60 0 0 DIARRHEA CASE MANAGEMENT Recall of advice on ORT/ORS Told of ORS/ORT 2 100 2 100 Told how to mix 2 100 2 100 Recall of home care advice Return if unable to drink 1 50 2 100 Return if vomiting 2 100 1 50 Return if has convulsion 1 50 0 0 Breastfeed more 1 100 1 50 Feed during illness 1 100 2 100 Give more fluids 1 50 2 100 * Only two mothers who had conducted the exit interview Record Reviews Item investigated Leuk Dek 3 HCs Kampong Thom 2 HCs HCs with stock-out % HCs with stock-out % Number of HC that had a stock-out in the previous 3 months of: Vitamin A 0 0 0 0 Clotrimazoxole: 0 0 0 0 Iron/folate: 0 0 0 0 ORS: 0 0 0 0 Paracetomol: 3 100 2 100 Item investigated Leuk Dek Health Centers Kampong Thom Health Centers Prek Tohrp Prek Dach Kg Phnom P.Sambo Salavisay Among 20 children born 6 months prior to MTE Field Visit: # with Documented DPT1 dose 18 - - 3 20 # with Documented DPT3 dose 17 - - 3 14 # with Documented Measles dose 16 - - 2 14 World Vision 93 Kean Svay Extension Midterm Evaluation Attachment H Key Indicators for Leuk Dek and Kampong Thom No. Indicators Leuk Dek Kampong Thom Baseline Nov 2000 1st Year Jan 2002 Target Sept 2003 HHS Feb 2000 LQAS1 Aug 2002 1. % of children (12-23 months) who have received (card documented) BCG, DPT3, OPV3, and Measles vaccines. 71% (169/237) 96.9% (126/130) 85% 13% 65.7% (25/38) 2. % of mothers of children 0-11months who had received two doses of TT (card documented) prior to the youngest child. 26% (47/180) 87.2% (148/170 75% 36% 74% (28/38) 3. % of children (6-23 months) received a (card-documented) dose of Vit A within the last six months. 26.3% (36/137) 97.4% (221/227) 80% 29.3%2 7.9% (3/38) 4. % of mothers of infants (0-11months) who received Vit A (card￾documented) within 2 months of the last delivery. 1.6% (3/180) 77.3% (231/299) 80% - - 5. % of families with children 0-23 months who report use of salt that is iodized. 22.4% (47/210) 8.7% (23/264) 45% 12.7% 18.4% (7/38) 6. % Children 0-23 months with cough and difficult or rapid breathing in the past two weeks who were brought to a trained provider. 66.1% (39/59) 76.9%3 (130/169) 80% - - 7. % Children 0-23 months with diarrhea in the past two weeks who were treated with ORT (ORS, rice soup, SS, or other home – available fluids) and no antidiarrheal agents. 52.7% (29/55) 92.5% (123/133) 80% - - 8. % Children’s caretakers of children o-23 months who can name at least one danger sing of DHF (cold extremities, bleeding, abdominal pain weakness). 20.9% (44/210) 68.1% 203/298) 60% - - 9. % of mothers of children < 2 who are not pregnant and desire no more children in the next two years (or who are not sure) who are currently using a modern contraceptive method. 6.4% (11/170) 63.6% (166/261) 30% - - 1 Estimates based on un-weighted averages of results from lot quality sampling (n=19) in the catchment areas of two health centers considered to have equal population size. 2 Estimate for children 6-59 based on recall. 3 Denominator may be 148 instead of 169 children 0-23 months with cough and difficult or rapid breathing in the past two weeks. World Vision 94 Kean Svay Extension Midterm Evaluation Attachment I Quality of care indicators for health centers, Kean Svay OD PRACTICE INDICATOR March 2002 N= 69 August 2002 N=73 # % # % ARI case management Sick under-5 children with ARI who were checked for danger signs? Able to drink /breast feed? 60 87% 44 60% Vomit after eating/breast feed? 30 34% 39 53% Convulsion? 51 74% 39 53% Fever (Child < 2 months)? 36 52% 21 21% Sick under-5 children with ARI who had their respiratory rate checked and recorded 63 91% 72 99% Sick under-5 children with ARI who had their vaccination status checked; 40 58% 53 73% Sick under-5 children diagnosed with pneumonia by CHC staff and who were given antibiotics 54 78% 50 68% Sick under-5 children diagnosed with upper respiratory tract infections (cough or cold) who were given an antibiotic inappropriately. Sick under-5 children with pneumonia needing referral and who were referred appropriately 62 90% 67 92% Mothers/caretakers of sick children with pneumonia who were given advice on giving more fluids, food or breast milk at home. 55 80% 59 81% Diarrhea case management Sick under-5 children with diarrhea who were checked for danger signs; Able to drink or breast feed? 63 91% 49 67% Vomit after eating/breast feed? 41 59% 39 53% Convulsion? 35 51% 25 34% Fever ( Child < 2 months)? 25 36% 13 18% Sick under-5 children with diarrhea who had their skin turgor checked 56 81% 61 84% Sick under-5 children diagnosed with diarrhea by CHC staff and who were given ORS 63 91% 61 84% Sick under-5 children diagnosed with simple diarrhea who were given an antibiotic or antimotility agent inappropriately 43 92% 51 70% Sick under-5 children with diarrhea needing referral and who were referred appropriately. 61 88% 65 89% Mothers/caretakers of sick children with diarrhea who were given advice on giving more fluids, food or breastmilk at home. 59 86% 69 95% World Vision 95 Kean Svay Extension Midterm Evaluation Attachment J Key findings and recommendations from FAR and progress to date Key findings Recommendations Progress to August 2002 1. Project Management – problems with travel distance/ time and workload for KSCSP staff 1.1 Revise Workplan to phase activities both geographically and chronologically (done) 1.2 Consider staying overnight in Leuk Dek 1.1 Workplan revised with phased approach to selected activities 2. Project Management – strengthen linkages between HC and VHVs, VDCs and community 2.1 Facilitate formation of Health Centre Management Committees following MoH Guidelines 2.2 Encourage VHVs and VDCs to regularly attend HC staff meetings and participate in planning as well as working 2.3 Develop a package of non-financial incentives for VHVs 2.1 Four HCMC formed but not yet functioning in Leuk Dek –see Capacity Building objectives below 2.2 VHVs regularly attending monthly meeting with HC staff 2.3Package of non-financial incentives developed and practiced– see FAR p15-16 and detailed notes from Leuk Dek integration meetings 3. Capacity Building – Case Management practices of HC staff does not yet meet quality standards 3.1 Provide training or refresher training (see also below on Quality Improvements) 3.1 Refresher training provided. Checklist for Quality of Care revised using IMCI criteria – see MOH Capacity Building objectives below 4. Capacity Building- OD staff need additional training/skills as master trainers of Demo site. 3.1 Support MoH organised training opportunities for OD staff 3.2 Support funding for Health Services Management training through NIPH for OD staff 3.3 Strengthen clinical and management skills and expertise in reproductive and child health to Kean Svay Referral Hospital staff. 3.4 Provide Training of Trainer Skills to OD staff, including presentation skills and establish a training team at OD. 3.5 Conduct COPE exercise for OD staff 3.1 ?? 3.2 Health Services management – OD candidates not accepted by NIPH – will submit names for next training. 3.3?? 3.4ToT training done, Presentation skills training not done. 3.5Details obtained from RACHA re COPE training in September 2002 5. BCC - Lack of IEC materials at HC and Village Level 5.1 Provide three TV/VCR units to share among the four HC 5.2 Obtain IEC materials from HKI (breast feeding), 5.1 Done 5.2 IEC Material obtained from HKI, UNICEF and MOH/NCHP World Vision 96 Kean Svay Extension Midterm Evaluation Key findings Recommendations Progress to August 2002 UNICEF/MoH 5.3 Disseminate 16 Community Practices for IMCI 5.4 Prepare a VHV IEC “kit” with key IEC materials 5.3 16 Community Practices disseminated along with IMCI Mother Card 5.4 ?? 6. Technical Assistance – identify TA needs and resources – externally and in-country See Section 4.4 of FAR BCC POA being implemented CBDDS POA being implemented including Social Autopsies Capacity Building POA being impleneted Oganisation Capacity Assessment done for KS and KgThom Sustainability assessments – not done (?tool) Private Practitioner’s case management assessment – not done LQAS training done and applied HKI approached regarding training in Vit A – not needed RACHA approached re COPE training – being planned CIMCI Working Group for iron/folate – following MOH guidelines Research studies on VHVs (Barbara Main) and Health Seeking Behaviours (Satoko Yanagisawa) obtained 7. Quality Improvements – Case Management practices of HC staff does not yet meet quality standards See 3. above. 7.1 Regularly supervise and monitor through Supervision Team comprising PHD, OD and the project staff (provide limited supplementary funding from the project for supervision activities) 7.2 Work with OD, PHD and central MoH to enhance supervisory checklists to strengthen quality of service delivery 7.3 OD should establish a technical mobile team to assist some health center where the capacity of staff is 7.1 Supervisory Team active – well structured but supervisory techniques and tools need improvement 7.2 Simplified IMCI like checklist for assessment of case management. MOH Case Management forms complicated. MOH Supervisory Checklists also very detailed. 7.3 Done in practice but no formal Technical Team 7.4 Details obtained from RACHA re COPE training World Vision 97 Kean Svay Extension Midterm Evaluation Key findings Recommendations Progress to August 2002 limited 7.4 Conduct COPE exercise for HC staff (see 4. above) in September 2002 8. Quality Improvements - HC staff not always available 24 hours/day or punctual for clinic hours 8.1 Chief of HCs should conduct a meeting focusing on working hours and staff on duty. Staff have to decide and agree among themselves on working hours and staff on duty (Bottom up approach). Chief and vice chief of the OD should participate in the meeting. 8.2 Strengthen the unity between chief of HC and his staff. 8.3 Provide training on leadership and management styles which will encourage teamwork and loyalty in each HC. 8.4 Consider appropriate rewards to staff as well as to HC that are open 24 hours per day and punctual for clinic times - HC, OD and the project will discuss the rewards. 8.5 Conduct missed opportunity survey at HCs 8.1 ?good progress in Lovea Em and Leuk Dek? 8.2 ? 8.3 Not done 8.4 Preliminary discussions only – wider implications for OD and PHD 8.6 Not done – need appropriate tool(s) 9. There is a lack of some Essential Drugs/supplies at HC level 9.1 OD and HC need to demonstrate transparency and accountability in management of drugs and supplies. The project will then be able to effectively support the HC through the provision of buffer stock. 9.2 Begin long term planning to address the issue of insufficient drugs/supplies from MOH in anticipation of the Project phasing out support. 9.3 Strengthen the cost-recovery system (with strong advocacy for free drugs for those clearly defined as poor). Use 50% of these funds to address drug/ supply shortages. 9.4 Advocate for increased support from MoH, World Bank, ADB where needs are clearly identified. 9.1 ? Continuing to provide supplementary drugs to HC in Leuk Dek 9.2 Structural reform in MOH contributing towards solutions 9.3 Some progress – HCMC and FC will be important vehicles for this process 9.4 Advocacy efforts through cIMCI Working Group, Medicam, IEC Working Group and other IO/NGO/MOH Forums World Vision 98 Kean Svay Extension Midterm Evaluation Key findings Recommendations Progress to August 2002 10 Quality Improvements – inadequate trained staff, equipment and supplies at Referral Hospital to support Project interventions 10.1 Explore possibilities for using WVUS GIK matched funds to strengthen reproductive and child health services at the referral hospital, particularly where cases are referred from HC for management. 10.2 Discuss with OD, referral hospital and HC an effective means of documenting referral from HC to the referral hospital (referral letter) 10.1 Some progress made on GIK procurement 10.2 ? No standard Letter of Referral as yet 11 Quality Improvements – no effective system exists to refer patients to the referral hospital 11.1 Provide motorbike-ambulances to each of the four HC to facilitate affordable transport at least in the dry season (done) 11.2 Advocate with the ADP to support infrastructure repairs to key roads, culverts and bridges 11.1 Done 11.2 In ADP POA but only for three communes at this stage – gradual geographical expansion 12 Quality Improvements – no reliable power source to provide lighting for 24 hour service 12.1 Provide generators to all four HC (done) 12.1 Done 13 Specific Interventions – need to keep up-to-date with MoH programs regarding IMCI and Nutrition See 4.6 above – IMCI update and Nutrition update Up-to-date with cIMCI activities. In process of seeking funding with WHO/UNICEF for cIMCI for Kg Chhanag Province Attended key Nutrition Policy Launch and obtained key policy and planning documents 14 Project in process of integration with ADP 14.1 Joint planning - checking for overlapping activities. 14.2 Regular meetings in new ADP project office 14.3 Conduct Lessons Learned/ Best Practice Workshops for direct and indirect impact areas 14.4 Plan for cross-visiting of ADP staff, including Kg Thom 14.5 Based on KSCSP experiences, develop standard health status and service indicators, quality of care indicators and capacity and sustainability indicators 14.1 & 14.2 Continued regular meetings between CS and ADP staff and overall Integration Plan drafted – see Minutes of Meeting 21 March 2002. 14.3 Lessons Learned Workshop conducted – see Minutes of Workshop 14.4 Cross visits to and from Kg Thom done – see Monthly Reports 14.6 Health Status Indicators document prepared and disseminated (Feb 2002) World Vision 99 Kean Svay Extension Midterm Evaluation Key findings Recommendations Progress to August 2002 to be introduced in all ADPs in Cambodia 15 Surveillance- need to strengthen Health Information System Provide training on HIS to HC staff, facilitated by PHD and OD TA – see 4.4. See CBDDS POA below 16 Networks – need to strengthen links with other USAID funded CS projects in Cambodia Plan regular meetings with other CS project staff and seek other opportunities for joint training/ sharing of experiences First meeting of all CS partners in Cambodia held in WV National Office on 29 July 2002 – networking and cross visit plans made. Invitations to other CS partners to attend KSCS activities including MTE World Vision 100 Kean Svay Extension Midterm Evaluation Attachment K Capacity Building Plan from FAR and progress to date Objective Indicators/definitions Activities Plan in FAR WV Cambodia Replicate and integrate essential KSCSP interventions into WVC ADPs 8 out of 23 WVC ADPs are using KSCSP tools and training materials for delivery of “essential” elements of child survival Operation and ADP Health Coordinators, Operation and ADP Managers involved in deciding on core and recommended health indicators and activities for WVC ADPs National Health Program and Operation Health Coordinators FY02 Q2 - ongoing Leuk Dek ADP = direct impact area 3 Kg Thom ADPs = indirect impact areas Phnom Srouch, Samaki Mean Chey and Samlot ADP using modified KPC survey. Kandal Steung ADP NHC using KS tools All ADPS with copy Health Status Indicators Enhanced Health in ADPs (WVUS) project using KS tools Increase competencies of WVC health and other program staff to enable successful integration of CS interventions into ADPs WVC staff Performance and Development Management (PDM) plans for KSCSP staff, WVC National Health Program staff, Operation and ADP Health Coordinators, Operation and ADP Managers document increased competencies in health-specific skills through training programs specific to each individual’s job description. Training Refresher ToT knowledge and skills Introduction to IMCI principles and practices Community Based Death and Disease Surveillance (including Verbal and Social Autopsies) Behavior Change Communication principles and practices NHP/KSCSP NHP and NPH FY02 Q3 Local Consultant FY02 Q2 or Q3 International Consultant (done) NHP, Partners for Development Enhanced Health in ADPs project providing extra funds for training for Operation Health Coordinators and ADP Health Coordinators (2- 6 Sept Training on Health Education methods by NCHP at NPH) All WVC staff involved in project with PDM2002 documents completed with details of personal and professional development. See also CBDDS and BCC POA See also LQAS training and World Vision 101 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR LQAS surveys Improved counseling and interpersonal communication skills FY02 Q2 WVC, UNICEF FY02 Q4 survey report Strengthen knowledge and skills of WVC staff in project design, planning, monitoring and evaluation WVC staff Performance and Development Management (PDM) plans for KSCSP staff, National Health Program staff, Operation and ADP Health Coordinators, Operation and ADP Managers and CDWs document increased competencies in project design, planning, monitoring and evaluation through training programs specific to each individual’s job description. Training in project design and planning Training in M&E Active participation of ADP staff in KSCSP surveys, planning, implementation, monitoring and evaluation and vice versa. NHP, KSCSP, ADP staff Throughout FY02 and FY03 See above Enhance grant management skills, systems and capacities throughout WVC KSCSP Finance/Admin staff, National Health Program Finance staff and Area Accountants from Kandal-Takeo and Kg Thom Operations have measurable increases in grant management skills. A learning network on grant accounting is established and functioning in WVC USAID Grant Management Workshop (WVUS CYOC) Formation of learning network WVC National Health Program FY02 Q2-3 One NHP Finance staff and one WVC Finance staff to attend Regional Grant Management Workshop in Manila in November Grant Management Workshop for WVC staff scheduled for 10 December – see draft program of topics/ resource persons involved Ministry of Health 1. 100 % of Leuk Dek 1. 100% of outreach Training of Health Center Who: MoH and 1. Almost 100% - document what World Vision 102 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR health center outreach activities managed solely by MoH (Operational District/Health Center. activities will be implemented and supported by the Health Center Staff in Leuk Dek Administrative District by end of project. 2. Cost recovery system in place to provide support for outreach activities by the four Health Centers in Leuk Dek Admin. District. staff in outreaches planning with emphasis on financial management and supervision. Training of Health Center staff in implementation of outreach activity. Continue of gradual phase out of outreach activities now implemented by KSCSP (Logistics, Supervision, Planning, and Supplies). KSCSP When: Beginning in FY02, Q2 KS Project is still providing – almost all planning done by HC staff with OD/VHVs 2. Cost recovery system operating but not fully in-line with MOH guidelines – HCMC and FC will help to strengthen proper operation of cost recovery 2. Increased quality of Case Management of Health Center Staff in LD, LE, KS, and KgT. “Good” performance rating in Case Management of HC staff in CDD, ARI, DHF. Training on Case Management of CDD, ARI, DHF for HC Staff Cross supervision using the Skill Check List (DIP Annex E) Training on counseling and interpersonal communication skills. CBDDS Training including verbal and social autopsies. Who: MoH & KSCSP When: FY02, Q2 Every six months. WVC, RACHA, UNICEF, FY02, Q4 Local Consultant, FY02, Q2 or 3. Case management training now following IMCI like approach. Project Manager attended IMCI Training of Trainers course in Siem Reap. Using revised and simplified IMCI like case management checklist – see M&E report on results No training as yet on interpersonal communication skills (but will be some overlap with COPE exercise See CBDDS POA 3. To strengthen the technical and managerial capacity of KS, and KgT OD to conduct quality supervision of Child 80 % of LD’s HCs visited by KS OD and 50% of HCs visited by KgT OD received all five selected Supervision Quality Indicators. Training on supervisory skills to Operational District & HC staff for Case Management FY02, Q2 Need to get data from Od in Kean Svay and Kg Thom World Vision 103 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR Survival Interventions. 4. Health Center Management Committee (HCMC) functioning in two Health Centers in Leuk Dek. (According to MoH Guidelines, see attachment) Two of four HCMC in Leuk Dek are able to perform according to the ten points in the MoH Guidelines (TOR for HCMC). Establish HCMC in four HCs in Leuk Deak Provide orientation on community CMCF in HCs Identify role of HCMC Training on problem identification and solution development. Orientation for HC management staff on how to conduct meetings Provide training on use of Health Information Systems. Equip and continue to support when needed. Who: MoH & KSCSP When: FY02, Q3 FY02, Q4 Start FY02, phase out FY03 Four HCMC and FC formed in Leuk Dek but not yet functioning 5. Increased organizational capacity of Kean Svay Operational District to manage and sustain child health programs. Increase in organizational capacity score (OCA adapted for KS OD) from 1- 2 to 4-5-6 in the KS Operational District Sponsor three senior KS OD staff to attend the six-month Health Services Management training at the National Institute of Public Health. Three trained senior staff to conduct workshop on capacity building for OD staff. Who: OD staff, Public Health Department & KSCSP. When: FY02, Q2-Q3 When: FY03, Q1 See FAR Progress Report – NIPH not able to accept nominations from Kean Svay OD on basis of age/ qualifications – will try again next round Not done World Vision 104 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR LQAS surveys Agreed Plan of Action for phased handover of management responsibilities Training in principles and practices of adult education. Training and site visit to local NGO (CRS) implementing increasing community contributions for health services, and community participation in the management of these services On-going through FY02 & 03 WVC Staff Development Unit, KSCSP. FY 02, Q3. Done POA needs to be documented Done, included with PLA training? Not done, but field visits made to Kg Cham and Kg Thom Community Organization 1. Health Center Management Committee and Feedback Committee functioning in two HCs in Leuk Dek. MoH Guideline description of Feedback Committee and Health Center Management Committee. Establish FBC and HCMC in two HCs in Leuk Dek. Provide orientation of community CMCF to community representatives. Identify role of HCMC and FBC. Equip and continue to support when needed. Orientation for community Who: KSCSP, MoH, HC Chief, and Public Health staff. When: Beginning in FY02, Q2 See above – 4 HCMC and FC formed in 4 HC in Leuk Dek but not yet functioning World Vision 105 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR representatives on how to conduct meetings. Training on problem identification and solutions development. Provide training on use of Health Information Systems. 2. Families with children 0 – 23 months are practicing selected C/IMCI family practices. (Home care, men’s participation and health seeking behavior.) Increased percent of mothers of children 0 – 23 months who continue to feed, provide fluids (including breast milk) to children with diarrhea. Increased percent of mothers who provide appropriate home treatment to children with ARI, CDD, and DHF. Mothers recognize when children need treatment outside the home and seek appropriate care. Increased percent of mothers follow the health worker’s advice about treatment, follow-up, and referral. Men actively participating in childcare and Training of caretakers in homecare, and treatment seeking in the village. Via Mother’s Group Model. Via Family Group Model. Who: VHVs, FBC, HC Staff, KSCSP, and HCMC. When: FY02, Q3 Who: HC staff, and KSCSP When: FY02, Q See KPC results ? additional data from BCC, LQAS and PLA focus groups See BCC POA World Vision 106 Kean Svay Extension Midterm Evaluation Objective Indicators/definitions Activities Plan in FAR reproductive health in the family. (Focus Groups) 3-4 3. Increased capacity of nine of the forty￾five VDCs in Leuk Dek, and the majority in Kompong Thom to design, manage, and implement child health activities. Eight VDCs are enabled to demonstrate skill in planning, designing, implementation, and managing the essential Child survival interventions. Eight VDCs are able to monitor, make decisions using data from the Community Based Death and Disease Surveillance tool. And, to the problems, and develop solutions in the village. Eight VDCs are able to link and access other resources to support community initiatives. Training of VDCs in the design, and implementation of child health activities. Set-up and maintain community owned registers. Training in CBDDS including social and verbal autopsies. Training in principles and practices of adult education. Who: HC Chief, KSCSP, ADP staff. When: FY02, Q 3-4 On-going Local consultant, FY02, Q2 or Q3 WVC staff development unit and KSCSP staff FY02. Q3 Need VDC organizational capacity assessment tool (?use recently published MRD tool) Community registers for births and pregnancies established CBDDS training for VHVs (who are usually members of VDC) Need tool(s) to assess this indicator World Vision 107 Kean Svay Extension Midterm Evaluation Attachment L Sustainability Plan from DIP and progress to date Overall Sustainability Goal: To facilitate a process of active community participation and partnership in responses to health issues affecting children Sustainability Objectives Sustainability Strategies/Activities Sustainability Indicators Time Frame Progress to August 2002 1. Integrate CS interventions into WV ADP design and activities CS approaches adapted to match strategies used by ADP Regular meetings between OD/HC staff, WV CS project staff, WV ADP staff and local communities to discuss and plan activities CS interventions effectively implemented within ADP structure in three communes - measured by comparing baseline results against repeat survey results Documented active participation of all project partners at all stages of project cycle over 2-3 years and long￾term as above See WVC Capacity Building Plan for all ADPs See overall POA for integration of CS interventions into Leuk Dek ADP Assess progress in 3 Kg Thom ADPs Ongoing meetings – documented in monthly reports and special reports 2. Phased transfer of program responsibility to OD staff and community Agreed Plan of Action with SMART objectives for handover of management responsibilities for all project activities and with clearly described roles and responsibilities. Training of OD/HC staff in project planning, management, supervision, monitoring & evaluation. Formation of active CHDMTs and VHVs (see below). Progress measured against agreed Plan of Action Measures of organizational capacity baseline and annual: including job descriptions, training to meet identified needs, plan of action for supervision/ support, resource allocation. over 2-3 years 2 years No formal POA with SMART objectives – need to bring together sustainability strategies into a coherent and realistic POA Training conducted in supervision, M&E but not yet in project planning and project management (not successful in gaining a place for OD staff to attend current 6 month NIPH Health Management training course) Now called HCMC, FC – formed but not yet functioning Successful formation and active World Vision 108 Kean Svay Extension Midterm Evaluation Sustainability Objectives Sustainability Strategies/Activities Sustainability Indicators Time Frame Progress to August 2002 Involvement of Women’s Groups in health issues (see below). involvement in health activities of VHVs without financial incentives Formation of Mother’s Groups - ? assess value/ function 3. Build MoH capacity by equipping and training staff In Direct Impact Area: Training in health-specific interventions consistent with National MoH policy and local needs Training in principles and practices of adult education Training in participatory methodologies to identify community needs and issues Obtain support from Provincial Health Department Schedule cross-visits and appropriate study tours in-country and regionally for selected key MoH staff In Indirect Impact Areas: Training in health-specific interventions consistent with National MoH policy and local needs Obtain support from Provincial Results of pre and post￾training assessment of knowledge, practices and quality of care Direct observation of health education activities conducted by health workers Record of methodologies used, results obtained and feedback to communities Active participation by PHD in health activities – planning, management and supervision Sharing of post-visit reports from participants on lessons learnt. Results of pre and post￾training assessment of knowledge, practices and quality of care 2 years 2 years 2 years 2 years after each visit 2 years 2 years See Project Training Report Not yet done – but health education methodology assessed using IMCI like Supervisory Checklist PLA training conducted and applied PHD have been actively participating in all project planning, implementation, monitoring and evaluation activities Cross visits made to Kg Cham and Kg Thom but not regionally See Training Report See above World Vision 109 Kean Svay Extension Midterm Evaluation Sustainability Objectives Sustainability Strategies/Activities Sustainability Indicators Time Frame Progress to August 2002 Health Department Schedule cross-visits and appropriate study tours in-country and regionally for selected key MoH staff Active participation by PHD in health activities – planning, management and supervision Sharing of post-visit reports from participants on lessons learnt. after each visit Cross visits from Kg Thom to and from Kean Svay are continuing 4. Strengthen community capacity to identify and respond to health needs Community involved in selection and training of active VHVs and CHDMTs Involvement of Women’s Groups in health issues Health center action plans developed through community participation Identify non-financial means to support and motivate VHVs Training in participatory methodologies to identify communinty needs and issues, including monitoring and evaluation Strengthening existing community based health surveillance structure including provision of regular and timely feedback to community – Document number and nature of community initiated recommendations discussed/ implemented at OD and Health Center meetings. as above as above document community initiatives and evaluate response Record of methodologies used, results obtained and feedback to communities Monitoring of surveillance system, formal evaluation report using participatory over 2 years and long￾term as above as above as above as above 2 years CHDMT is now HCMC and FC. Community actively involved in selection of VHVs and HCMC/ FC members Mother’s Group need to assess value and function See comments on HCMC and FC Active participation of VHVs in health activities with demonstrated good outcomes and without financial incentives PLA training done and applied See CBDDS POA World Vision 110 Kean Svay Extension Midterm Evaluation Sustainability Objectives Sustainability Strategies/Activities Sustainability Indicators Time Frame Progress to August 2002 partnership with NIPH methodologies 5. Strengthen equitable cost￾recovery mechanisms and other means of financial support Transparency of cost-recovery system Plan for more efficient use of current resources, including allocation from MoH for PHD and OD Seek income from other activities to fund health services eg. fund raising ceremonies Local communities (CHDMT) to explore opportunities to contribute to resources for health Budget allocation for post-grant sustainability assessment at least two years after USAID funding has ceased Develop, with community participation, indicators to measure quality of care and client satisfaction with services Monitor progress against agreed Plan of Action noted in Strategy 2 above Document successful alternative fund sources and track how this income is utilized Written report 2 years and long￾term 2 years and long￾term 2 years and long￾term Post￾grant Current cost-recovery practice not fully in line with MOH Guidelines – HCMC and FC will help with transparency and effectiveness Need documented POA One possible additional income source may be through MED program expanding through Leuk Dek – decide with MED team on suitable indicators to monitor how income is used Discuss with WVUS and USAID – also consider funding (non-USAID) for 1-2 year extension for northern communes of Leuk Dek where the ADP will not be active for some years – to consolidate integration of health activities 6. Explore opportunities to involve key private sector providers in quality CS services Collect accurate data through a variety of quantitative and qualitative methodologies Review National MoH policy on private providers Obtain from various sources relevant data on utilization of private health providers, including measures of quality care and client satisfaction with private health providers 2 years See Satoko’s study in Leuk Dek and various other studies (mostly urban) Most private providers are also HC staff World Vision 111 Kean Svay Extension Midterm Evaluation Sustainability Objectives Sustainability Strategies/Activities Sustainability Indicators Time Frame Progress to August 2002 Develop appropriate Action Plan after above steps 7. Use organisational and community structures to advocate for healthy public policy Workshop on lessons learnt Select health issues of significant community concern and prepare an advocacy action plan Documented Proceedings of Workshop and dissemination of this document Document response of decision-makers to advocacy action plan 2 years 2 years and long￾term Regional Lessons Learned Workshop planned for Cambodia in FY03. Two or three WVC participants will attend Lessons Learned/ Final Evaluation of Ballia CSP in India in September 2002 ?