World Vision, Inc. Final Evaluation of the Landak Child Survival Project October 1st, 1999 − September 29th, 2004 Cooperative Agreement Number: FAO-A-00-99-00027-00 Submission Date: December 31, 2004 Submitted to: Susan Youll Child Survival and Health Grants Program USAID/GH/HIDN/NUT Room 3.7-74, Ronald Reagan Building 1300 Pennsylvania Avenue NW Washington D.C. 20004-3002 Prepared by: Marc Debay and Alex Papilaya, with other members of the Final Evaluation Team PVO Headquarters Contact: Peggy McLaughlin Maternal & Child Health Specialist Technical Resources Team World Vision US 300 I Street NE Washington DC 20002 Phone: (202) 572-6300 pmclaugh@worldvision.org World Vision i LCSP Final Evaluation Final Evaluation Team members contributing to writing and editing the report: Alex Papilaya, MD, DDTS Final Evaluation Team Leader Public Health Faculty, University of Indonesia Marc Debay, MD, MPH, PhD Final Evaluation co-Team Leader Independent Consultant Mary Wangsaraharja, DDS, DDPH National Health Advisor World Vision Indonesia Untung Sidupa ADP Pontianak Manager World Vision Indonesia Esther Indriani, MPH Project Officer Landak CSP World Vision Indonesia Teresa Zakaria, MD Technical Team Leader CSP World Vision Indonesia Perry Mansfield Program Officer World Vision US Warren Wright Senior Finance Officer, Eurasia Region World Vision US Lyndon Brown, MPH Child Health Advisor World Vision US World Vision ii LCSP Final Evaluation Table of Contents A. SUMMARY 1 B. ASSESSMENT OF RESULTS AND IMPACT OF THE PROGRAM 8 1. Results: Summary Chart 8 2. Results: Technical Approach 12 a. Project Overview 12 b. Progress by intervention area 14 i. Immunization 14 ii. Vitamin A Supplementation 18 iii. Malaria Control 20 iv. Diarrhea Diseases Control 24 v. Acute Respiratory Infections Control 26 c. New tools or approaches 27 i. Development of ITN distribution strategies 27 ii. Chloroquine Efficacy Study 30 iii. Community-Based Health Information System 31 iv. Doer-Nondoer Survey of ITN Use 35 v. Organizational Capacity Assessment 35 3. Results: Cross-cutting Approaches 36 a. Community Mobilization 36 b. Communication for Behavior Change 40 c. Capacity Building 43 i. World Vision 44 ii. Partners Organizations 45 iii. Health Facilities Strengthening 46 iv. Strengthening Health Worker Performance 47 v. Training 49 d. Sustainability 52 i. Achieving Sustainability Goals and Objectives 52 ii. Developing a Sustainability Plan 53 iii. Building Financial Sustainability 54 4. Program Management 57 a. Planning 57 b. Staff Training 58 c. Human Resources Management 61 d. Financial Management 62 e. Logistics 64 f. Information Management 65 g. Technical and Administrative Support 68 h. Management Lessons Learned 69 World Vision iii LCSP Final Evaluation C. CONCLUSIONS AND RECOMMENDATIONS 72 a. Success in meeting objectives 72 b. Main Achievements and Factors Affecting Performance 75 c. Best practices and lessons learned 76 d. Final Evaluation Recommendations 78 e. Dissemination of Best Practices and Lessons Learned 82 f. Potential for Scale-Up 82 ANNEXES 84 Annex 1. Child Survival Grants Program Project Summary 85 Annex 2. Milestones of the Landak Child Survival Project 89 Annex 3. Timeline of Overall Project Implementation, July 2000 – September 2004 95 Annex 4. Maps of Landak District and ADP Pontianak 96 Annex 5. Demographic and Health Services Profile of Project Area 97 Annex 6. LCSP FY 2004 Work Plan and Achievements 98 Annex 7. LCSP FE KPC Results and Targets for 14 LCSP indicators 105 Annex 8. LCSP Progress Towards Targets for 14 LCSP Indicators 108 Annex 9. Terms of Reference for the Final Evaluation 111 Annex 10. FE Assessment Methodology and Schedule 114 Annex 11. FE Field Visits – Health Centers 116 Annex 12. FE Field Visits – Sub Health Centers 122 Annex 13. FE Field Visits – TBA and midwives 124 Annex 14. FE Field Visits – Posyandu Focus Group Discussions 128 Annex 15. FE Field Visits – Posyandu Day 130 Annex 16. FE Field Visits – Posyandu Cadre Assessment 133 Annex 17. FE Field Visits – Village Drug Post 136 Annex 18. FE Field Visits – Self Help Group 138 Annex 19. FE Field Visits – Koperasi Pancur Dangeri 139 Annex 20. FE Field Visits – Village Leaders Feedback Meeting 140 Annex 21. List of Participants in FE Meetings and Presentations 142 Annex 22. FE KPC Results for EPI Antigen-Specific Coverage 144 Annex 23. ITN Distribution Diagram 145 Annex 24. Posyandu Monthly Report Data for Year 2003 146 Annex 25. Posyandu Monthly Report Form 147 Annex 26. Organizational Capacity Assessment Results and Questionnaire 148 Annex 27. BEHAVE Framework for Use of ITNs 154 Annex 28. Training Jointly Facilitated by LCSP 155 Annex 29. Summary Data on Training Jointly Facilitated by LCSP 160 Annex 30. Posyandu Cadre Training Curriculum 161 Annex 31. POD Cadre Training Curriculum 163 Annex 32. Child Survival Sustainability Assessment and Action Plan 164 Annex 33. ITN Distribution Plan – ADP Pontianak 167 Annex 34. Training Events Attended by LCSP Staff 171 Annex 35. Summary Events Attended by LCSP Staff 175 Annex 36. ADP Pontianak – Landak Child Survival Project Organizational Chart 177 Annex 37. Timeline of Human Resources, July 2000 – September 2004 178 Annex 38. Human Resources Management Indicators 179 Annex 39. Timeline of Technical and Administrative Support, July 2000 – September 2004 180 World Vision iv LCSP Final Evaluation LIST OF FIGURES Figure 1 KPC survey results for immunization ......................................................................................................... 15 Figure 2 Measles immunization coverage from services statistics and KPC surveys ............................................... 17 Figure 3 KPC survey results for vitamin A supplementation ................................................................................... 19 Figure 4 KPC survey results for malaria control - treatment seeking for fever ........................................................ 21 Figure 5 KPC survey results for malaria control - ITN ownership and use .............................................................. 22 Figure 6 KPC survey results for diarrheal diseases control ...................................................................................... 24 Figure 7 KPC survey results for acute respiratory infections ................................................................................... 26 Figure 8 Posyandu cadres, shopkeepers and TBAs trained by Fiscal Year .............................................................. 50 Figure 9 WV Indonesia staff trained per LCSP Semester ......................................................................................... 59 Figure 10 LCSP Monthly Grant Financial Cycle ........................................................................................................ 63 LIST OF TABLES Table 1 Development of the ITN distribution strategy in the LCSP area .................................................................. 28 Table 2 ITNS Sales during Mass Campaign and Coverage of Target Group in three Health Center Areas .............. 29 Table 3 Results on epidemiology of malaria from Mass Blood Survey ..................................................................... 31 Table 4 Results on chloroquine efficacy .................................................................................................................... 31 Table 5 Selected Indicators from Posyandu Monthly Reports for Year 2003 in Mandor, Senakin and Pahauman ... 33 Table 6 Number of Posyandus and population covered ............................................................................................ 38 Table 7 BCC materials distributed by LCSP ............................................................................................................ 41 Table 8 Selected results of the FE Posyandu Cadres Assessment ............................................................................. 48 Table 9 Community and Health Center Annual Costs of Providing Health Services in Posyandus ......................... 57 Table 10 Roles and responsibilities of the LCSP Core Team and Health Motivators ............................................... 61 Table 11 Indicative LCSP Financial Statement as of August 2004 ........................................................................... 64 Table 12 Summary of Baseline and Final Evaluation Results from KPC survey ...................................................... 72 World Vision v LCSP Final Evaluation Acronyms ADP Area Development Program ADP/P Area Development Program Pontianak ARI Acute Respiratory Infections BCC Behavior Change Communication CBDDS Community-based Disease and Death Surveillance CBHIS Community Based Health Information Systems CU Credit Union DHO District Health Office DIP Detailed Implementation Plan EPI Expanded Program of Immunization HC Health Center HIS Health Information System HIV Human Immuno-Deficiency Virus IMCI Integrated Management of Childhood Illness ITN Insecticide Treated Nets KPC Knowledge, Practices and Coverage survey KSM Village Self-Help Group (also SHG) LCSP Landak Child Survival Project MCH Maternal Child Health MOH Ministry of Health MTE Midterm Evaluation NGO Non-Governmental Organization OCA Organizational Capacity Assessment ORT Oral Rehydration Therapy PERSUAP Pesticide Evaluation Report and Safe Use Action Plan PHO Provincial Health Office PVO Private Voluntary Organization SHG Village Self-Help Group TBA Traditional Birth Attendant TDI Transformational Development Indicators TOT Training of Trainers USAID United States Agency for International Development WV World Vision WVAPRO World Vision Asia and Pacific Regional Office WV Indonesia World Vision Indonesia Indonesian terms commonly used: Desa Village Dusun Sub village Kelompok Swadaya Masyarakat (KSM) Self-Help Group (SHG) Koperasi Serba Usaha (KSU) General Cooperative Polindes - Bidan Village Birthing Post with a midwife Polindes + Bidan Village Birthing Post without midwife Pos Obat Desa (POD) Village Drug Post Posyandu Village Health Post (or Integrated Service Post) Posyandu Cadre Village Health Volunteer Puskesmas Sub district Health Center Pustu Sub village Health Center World Vision 1 LCSP Final Evaluation A. Summary The goal of the Landak Child Survival Project (LCSP; FY 2000-2004) was to assist the Landak District Health Office in West Kalimantan, Indonesia, in reducing mortality and morbidity among infants, children and mothers through an integrated child survival and rural enterprise project. The LCSP focused on five technical interventions: 1) Immunization 2) Vitamin A supplementation 3) Diarrheal diseases control 4) Malaria control 5) Acute respiratory infections control The main strategies defined in the Detailed Implementation Plan were to: 1. Improve the quality and coverage of existing child survival interventions; 2. Empower communities for prevention and control of common diseases; 3. Initiate micro-enterprise efforts and promote collaboration in health; 4. Develop a child survival intervention model for further replication and scale up. The LCSP covered two sub-districts of the World Vision Indonesia (WV Indonesia) Area Development Program in Pontianak (ADP). The total population of the project area in 2004 was 75,677, with 1,970 infants and 10,225 children under five. The local partners of the LCSP were the Ministry of Health, the Landak District Health Office and three health centers, as well as community-based groups and volunteers in the project area. The LCSP was implemented in a different site from the one in the original grant. USAID approved a change in project site after social unrest and insecurity made the original project site inaccessible. The DIP workshop for the LCSP therefore only took place in October 2000 and while WV Indonesia was still recruiting staff for the new project site. The Technical Team Leader and most of the other project staff were only recruited and able to begin work on-site by April 2001, when the initial training began. The MTE was conducted in August 2002, a year before the end of the original grant period. Given satisfactory progress despite the change in project site and the resulting one-year delay, one of the main recommendations of the MTE was to apply for a two-year no-cost extension. However, WVUS staff decided to seek only a one-year no-cost extension, for FY 2004. This was approved by USAID. However, World Vision's request for a second no-cost extension for FY 2005, submitted on the grounds that a significant proportion of the project funding still remained unspent and that new opportunities to scale up project activities and improve the potential for sustainability had arisen, was not approved. The Final Evaluation Team (FE Team) found that the LCSP has assisted the Landak District Health Office in reducing mortality and morbidity of children under five through an integrated child survival project in Mandor and Sengah Temila sub-districts as follows: World Vision 2 LCSP Final Evaluation Technical interventions Overall, the LCSP achieved significant results in terms of increased coverage with child survival interventions, and improved health behaviors of mothers with children under two years of age. The table below summarizes the main results obtained from baseline and final KPC surveys for 14 key LCSP indicators. # Indicator1 (children 0-23 months or their mothers, except when noted) Baseline KPC Final KPC EOP Target Target Met2 1 Fully immunized children, 12-23 months, card verified 23% 61% 80% N 2 TT2 coverage, card verified 2% 44% 50% Y 3 Vitamin A supplementation, 12-23 months 54% 88% 90% Y 4 Vitamin A supplementation, post-partum 12% 41% 50% N 5 Seek treatment for fever, 12-23 months 26% 54% 75% N 6 Treatment of fever in health facility within 48 hours - 49% 45% Y 7 ITN ownership - 23% 30% N 8 ITN use last night - 21% 20% Y 9 ORS use, 0-23 months with diarrhea 32% 52% 75% N 10 Same amount or more fluids, 0-23 months with diarrhea 63% 78% 75% Y 11 Same amount or more foods, 0-23 months with diarrhea 34% 39% 70% N 12 Antidiarrheal medicine, 0-23 months with diarrhea 32% 50% 10% N 13 Mother’s knowledge of 1 pneumonia symptom 25% 55% 80% N 14 Treatment seeking, 0-23 months with pneumonia signs 59% 72% 75% Y 1 See full definition of indicators in Results: Summary Chart (section B.1). 2 Target met if included in 95% confidence interval around Final Evaluation KPC estimate. Twelve of the 14 key LCSP health and behavior indicators had improved at the end of the LCSP, and 6 met their end-of-project targets. The FE Team determined that some of the unmet targets were probably unrealistic, while others were due to programmatic shortcomings. Immunization: The percentage of card-documented Fully Immunized Children 12-23 months increased from 23% to 61% but the ambitious 80% target was not met. The percentage of mothers of a child under two who received at least two card-documented tetanus toxoid injections during their last pregnancy increased from 2% to meet the 50% target for this indicator. The LCSP contributed to the quality of the cold chain in the three Health Centers, although there are still deficiencies. Vitamin A supplementation: The percentage of children 12-23 months who received vitamin A in the last 6 months met the 90% target. Although the percentage of mothers of children under two who received vitamin A within one month after delivery steadily increased from 12% to 41%, the 50% target for this intervention was not met. Malaria control: The percentage of children under two with fever in the last two weeks who are brought to a facility has increased from 26% to 54% but the high target of 75% was not met. The proportion of children brought to a health facility within 48 hours after the onset of a fever met the 50% target for this indicator. The LCSP also met the 20% target for the use of ITNs by children under two during the night before the survey. Diarrheal diseases control: The percentage of children under two with diarrhea in the last two weeks steadily increased from 32% to 52% but remained short of the high 75% target. The percentage of children with diarrhea who received the same amount or more fluid, however, World Vision 3 LCSP Final Evaluation increased from 63% to meet the 80% target for this indicator. The LCSP did not appear to improve feeding practices, nor was the use of anti-diarrheal medicines reduced for children with diarrhea. These are two shortcomings of particular concern. Acute respiratory infection control: The percentage of mothers with children under two who know at least one symptom of pneumonia increased from 25%, but did not reach the 80% target. The percentage of mothers with children under two who had symptoms of pneumonia (cough and rapid or difficult breathing) who sought appropriate treatment increased from 60% to meet the 75% target. Micro-Enterprise Development: The LCSP did not reach the objective set for the MED intervention. The ADP only began introducing health-related activities in a few community￾based groups in the last years of the LCSP, and no training in health education had been provided to such groups at the time of the Final Evaluation. Capacity building The number of functioning Posyandus in the project area increased from 44 in 2000 to 108 at the end of the project, which corresponds to a number of children under five per Posyandu within the range of that for distributing ITNs recommended by the MOH. The LCSP team estimates that population with acceptable access to a Posyandu reached 95% at the end of the project. In 2003, the average Posyandu attendance rate was 71% for infants, and 39% for children 1-4 years old. Other trained community health workers contribute to health services in their communities. Government health workers regularly provide services in the three Health Centers and in the Posyandus. In 2003, for example, an immunization officer was present at 87% of the Posyandu days. The LCSP provided training to health workers at all levels, but did not provide direct support to the PHO, DHO or Health Centers in terms of planning and management. The LCSP did not have initiate activities with the local NGO for distributing ITNs designated in the DIP until the last year of the project. The ADP staff participated in training and other LCSP activities, and their capacity to implement child survival interventions was strengthened. Sustainability Health workers make nearly monthly visits to Posyandus to provide health services while trained Posyandu cadres organize the services, and provide health education using visual aid and other participatory techniques. The coverage of most health and behavior indicators have increased because of increased community participation and increased numbers of functioning Posyandus. Health Centers and Posyandus have improved their information system, and use registers and reporting forms linked to the MOH information system on a routine basis. Reports from community members and village leaders clearly show that most communities are committed to maintaining their Posyandus, and have in some cases petitioned the government proactively for improvements in the quality and availability of health services in their areas. The LCSP has trained a ADP staff in child survival-related topics. The ADP has been designated a “Child Survival Focus ADP” by WV Indonesia. However, the ADP did not fulfill the project's objective of allocating 20% of its FY 2005 budget for health and child survival. World Vision 4 LCSP Final Evaluation In addition to the LCSP results noted above, the project successfully conducted a series of special studies including: Posyandu Health Information System: The LCSP actively collected close to 90% of the expected MOH Posyandu Monthly Reports in the project area over a period of 12 months (calendar year 2003). Data from these reports were entered and available to the FE Team and WV Indonesia has planned for further analyses to be conducted in cooperation with the MCH Department of the Ministry of Health. Development of ITN distribution strategies: The LCSP introduced Long-Lasting Insecticide￾Treated Nets in the project area and distributed close to 4000 units for use by pregnant women and children under five in about two years. The various social marketing strategies developed and tested during that period have been documented and the ADP will use this experience to expand this intervention after the end of the project and beyond the project area. Chloroquine Efficacy Study: Following up on reports from health workers in the project area that malaria patients were not always successfully treated with chloroquine, the LCSP carried out a study that confirmed high levels of Plasmodium Falciparum resistance to chloroquine in the project area. The LCSP used this evidence to conduct advocacy for a change in the malaria drug policy and for malaria control in general at the DHO and PHO levels. The LCSP was designed to assist the Landak District Health Office in reducing mortality and morbidity of children under-five by achieving the following Intermediate Results: IR 1.1: The use of child and maternal health services by the target population has increased; IR 1.2: The participation and contribution of communities in the prevention and management of diseases has increased; IR 1.3: Communities are investing their limited resources in low cost high impact CS interventions delivered through Posyandus and in ITNs; IR 2.1: Local health care system and communities are equipped and provide child survival interventions at home, at the community level, and in health services; IR 3.1: The DHO and three Health Centers deliver CS interventions and engage with community volunteers to improve maternal and child health; IR 3.2: Most communities in the project areas engage in the prevention and early and complete treatment of common illnesses. The FE Team identified several factors that have impeded project performance:  Landak is a relatively new district, with still limited human and financial resources;  Many villages are located in remote areas, with very difficult access health workers and motivators providing outreach services during the rainy season;  The number of health facilities and health workers is limited, and there is not always staff available to travel to Posyandus, and provide services on the specified days;  Approximately half of the Village Birthing Posts do not have an active midwife;  Some health workers are not performing as expected, supervision is inadequate and there is no system in place to identify and correct these deficiencies; World Vision 5 LCSP Final Evaluation  The availability of vitamin A capsules for post-partum vitamin A supplementation and ORS for case management of diarrhea remains limited at the community level;  The practice of prohibiting post partum mothers from leaving the house is still common and prevents babies from immunization and other services;  ITNs were only available at an affordable price during the project's last few months. Given the findings above, the FE Team recommends: Village Leaders  Take the lead in supporting Posyandus and other child survival and health activities as described in their Declaration of Commitment at the time of the Final Evaluation. Health Centers  Enforce proper cold chain maintenance and waste disposal procedures;  Develop and implement procedures for ORS and vitamin A distribution to Posyandus on a routine basis;  Support monthly meetings with TBAs in all village birthing posts to ensure that all trained TBAs have access to vitamin A capsules for post partum mothers;  Follow up on the Declaration of Commitment by Village Leaders made at the time of the Final Evaluation to ensure it is implemented;  Develop realistic plans and made appropriate budget allocations to continue support to Posyandus; Landak District Health Office  Integrate the most successful and sustainable strategies of the LCSP into the Landak District Strategic Health Plan and related budgets and use of available financial resources and partnerships to support these activities;  Encourage support for Posyandus by Village Leaders and sub-district officials;  Ensure the active presence of a midwife in all Village Birthing Posts n the district;  Develop and implement procedures for routine ORS distribution to Posyandus and vitamin A distribution to trained TBAs;  Maintain and further improve service quality of health centers, sub-centers, Posyandus and TBAs achieved under the LCSP by supporting, monitoring and supervising these activities;  Promote use of ITN by pregnant mothers and children under five through continued partnership with local NGOs; Provincial Health Office  Use the LCSP area as a model for community-based child health programs in West Kalimantan province and use available financial resources and partnerships to support these activities; World Vision 6 LCSP Final Evaluation ADP Pontianak  Develop a more detailed three-year plan for continuing LCSP activities in the two sub￾districts and further expansion to other districts;  Conduct advocacy with DHO, health workers and village leaders to ensure support of Posyandus;  Become a learning site for the replication of the successful child survival interventions and strategies of the LCSP in other ADPs or contexts; World Vision Indonesia  Designate one Child Survival or Health respondent in the NO Health Unit to follow up on the child survival plans and activities of the ADP;  Provide technical and programmatic assistance to ADP to further specify the FY2005 Plan of Action and Budget for child survival; develop a three-year plan for child survival including scale up strategies; integrate health and child survival into MED activities; and become a learning site visited by other ADP’s staff;  Develop training, BCC, HIS and other materials for implementation of child survival interventions in ADPs based on the experience of the LCSP;  Provide technical assistance to integrate child survival into the three ADPs in Kalimantan, then gradually into all 28 WV Indonesia ADPs within the next 5 years;  Document and disseminate success stories, best practices and lessons learned that could be used within WV Indonesia, MOH and other partner organizations;  Conduct trends and other analyses of the data from the three KPC surveys to further assess the determinants of coverage and the impact of LCSP interventions;  Pursue analyses of the Posyandu information system in collaboration with the Department of Maternal and Child Health;  Conduct a fourth KPC survey in the ADP area in two or three years as a post-project sustainability assessment; World Vision US  Provide more intensive technical assistance for strategic planning and specific needs to gain the full benefit of special projects, and build capacity of WV National Offices;  Use programmatic and financial monitoring data from special projects to revisit current plans of action, and make necessary changes including formal amendments to the DIP or agreements, as appropriate and in a timely way;  Consider selection of indicators and targets carefully, ensuring that indicators are measurable and sensitive, and that targets are sufficiently challenging yet achievable. USAID/GH/CSHGP  Ensure that annual reviews and midterm evaluation guidelines specify that CSHGP grantees need to further develop or revise their DIP when changes occur that could affect project results.  Consider including advocacy as a crosscutting approach in the CSHGP technical guidelines, and developing an advocacy module to be included in the Techncial Resource Materials. World Vision 7 LCSP Final Evaluation World Vision 8 LCSP Final Evaluation B. Assessment of Results and Impact of the Program 1. Results: Summary Chart Intermediate Results (as in the DIP) Indicators and end-of-project objectives (as in the DIP and revised in MTE) BL MTE FE Target met Comment 9/00 7/02 9/04 1. TECHNICAL INTERVENTIONS 1. Increased use of integrated child and maternal health services by the target population 2. Increased participation and contribution of communities for the prevention and early / complete management of diseases Immunization  LCSP achieved an impressive increase in the proportion of Fully Immunized Children but did not reach the target of 80%. LCSP achieved the two other targets for immunization.  LCSP contributed to the cold chain, but deficiencies exist.  80% of children 12-23 months completely immunized verified by card 23% 47% 61% NO  50% of mothers with children less than 2 years of age received TT2 before the birth of their youngest child 2% 15% 44% YES  100% of health facilities have cold chain temperature recorded in the expected range and vaccines within expiration date 20% 100% 100% YES Vitamin A Supplementation  The target for vitamin A supplementation for children 12-23 months was achieved, but the target for post-partum supplementation was not.  90% of children 12-23 months received Vitamin A in the past 6 months 54% 60% 88% YES  50% of mothers with children less than 2 years received Vitamin A within one month of their last delivery 12% 26% 41% NO Malaria Control  Health-seeking behavior for fever increased by midterm, but remained stable and below the target later on. Early care seeking from a health facility increased after midterm and reached the target.  The target for ITN use by children under two years of age is achieved, but not that for ITN ownership (nearly all ITNs owned are being used appropriately, per definition).  75% of mothers with a child less than 2 years of age who was ill with fever during the past 2 weeks seek treatment (Modified) 26% 58% 54% NO  45% of children less than 2 years of age with febrile episode that ended during the last 2 weeks were brought to a health facility within 48 hours after fever began (Modified). NA 30% 48% YES  30% of children whose mothers report the presence of insecticide treated bednet in the house (Added during FAR) NA 5% 23% NO  20% of children less than 2 years of age who sleep under an ITN the previous night (Added during FAR) NA 2% 21% YES World Vision 9 LCSP Final Evaluation Intermediate Results (as in the DIP) Indicators and end-of-project objectives (as in the DIP and revised in MTE) BL MTE FE Target met Comment 9/00 7/02 9/04 Diarrheal Diseases Control  LCSP did not reach its target for ORS use, but did reach the one for increase in fluids. LCSP did not emphasize ORS use but promoted use of any kind of home-available fluids.  75% of children less than 2 years of age who had diarrhea in the past two weeks received oral rehydration therapy. 32% 43% 52% NO  75% of children less than 2 years of age who had diarrhea in the past two weeks received the same / amount or more of fluids or breast milk 63% 59% 78% YES  70% of children less than 2 years of age who had diarrhea in the past two weeks received the same / amount or more of food 34% 29% 39% NO  LCSP did not achieve change in the amount of food given to children with diarrhea.  Less than 10% of mothers with child less than 2 years who had diarrhea in the past two weeks report their child received an antidiarrheal medicine. 32% 34% 50% NO  The increase in the use of antidiarrheal for diarrhea for children under two is of concern. Acute Respiratory Infection Control  LCSP improved knowledge of pneumonia symptoms but did not reach this target.  LCSP reached its target for health-seeking behavior for pneumonia.  80% of mothers with a child less than 2 years of age who mention at least 1 symptom of pneumonia (Added) 25% 40% 55% NO  75% of mothers with a child less than 2 years of age who had symptoms of pneumonia seek treatment. (Added) 59% 59% 72% YES 3. Equip communities to invest their limited resources in low cost high impact CS interventions and to strengthen household livelihood. Micro Enterprise Development  The ADP began introducing health-related activities among SHGs in the last year of LCSP and has not started training its members in health education.  20 KSMs (Self-Help Groups) trained to deliver CS health education and conducting monthly community CS health education sessions. NO World Vision 10 LCSP Final Evaluation Intermediate Results (as in the DIP) Indicators and end-of-project objectives (as in the DIP and revised in MTE) BL MTE FE Target met Comment 9/00 7/02 9/04 2. CAPACITY BUILDING 1. Local health care delivery system, NGO partners, and communities equipped and providing CS interventions at all levels (home, community, health services)  Provincial Health Office meeting on bimonthly basis for updates of project process and PHO providing facilitators for key training events.  District Health Management Team supported to further strengthen their newly formed district. ADP and Project Manager conduct monthly meeting for planning and priority setting for upcoming activities as well as review of accomplishments to date.  Strengthening of health center management teams in direct impact areas for routine supervision and use of data for targeting and decision-making. Health center teams have map design demonstrating catchment area including presence of village health volunteers and TBAs and a plan for CS intervention coverage and supervision for their area.  Capacity building of all levels of staff including: health center, sub center, and village post staff for CS interventions.  Support provided to revitalize the Integrated Service Posts, village health posts and the functioning of the village health volunteers.  Capacity building of ADP staff in CS interventions and related skills (BCC, quality improvement, data for decision making, credit with education, etc)  Capacity Building/ Organizational Development of Local NGO, including activity plan and operations in accordance with established MOU  Capacity building of informal health partners for CS initiatives including Traditional Healers, shopkeepers, Self Help Groups, community leaders, households, etc. NO NO NO YES YES YES NO YES  LCSP felt such formal meetings with PHO were not necessary on monthly basis but met with PHO officials as needed for project activities.  Regular ad-hoc meetings between DHO officials and LCSP staff were held for project activities, but not with ADP and not for systematic planning, priority setting and review of accomplishments.  LCSP has not worked directly with health centers’ management teams on routine supervision or use of data for decision making.  Health workers were trained in IMCI, malaria microscopy, nutrition and involved in many LCSP activities.  LCSP created or revitalized 108 Posyandus, training and supporting 703 cadres.  ADP staff was trained in basic Child Survival interventions.  LCSP had limited activities with local NGO except for involvement of Pancur Dangeri in distribution of ITNs in 2004.  Most informal health partners for CS were trained, except for Traditional Healers. World Vision 11 LCSP Final Evaluation Intermediate Results (as in the DIP) Indicators and end-of-project objectives (as in the DIP and revised in MTE) BL MTE FE Target met Comment 9/00 7/02 9/04 3. SUSTAINABILITY 1. District and Provincial Health Offices prioritizing the delivery of CS interventions and engaging with community volunteers for creating an enabling environment for improved maternal and child health. 2. Communities engaging in prevention and early and complete management of common illnesses.  Health center and post staff regularly supervising and encouraging village based volunteers. YES  Health workers visit Posyandu on a nearly monthly basis although usually focus on providing health services.  Health center staff and village-based volunteers utilizing participatory education methodologies and visual health education materials to promote CS interventions. Shop keepers and traditional healers able to state treatment protocols and danger signs indicating need for referral. YES  Posyandu cadres regularly provide health education using visual aids provided by LCSP. Shopkeeper may need to be retrained. LCSP did not train or support traditional healers.  Increased coverage for all CS interventions demonstrating high community participation. YES  Significant increases in coverage for all LCSP interventions, due in large part to increased participation of community and Posyandus.  At least 70% of the villages will establish community based EPI/Pregnancy Register and CBDDS (Community Based Death and Disease Surveillance) NO  LCSP strengthened existing Posyandus HIS (registers and reporting forms) and supported by MOH.  At least 60% of the Self-Help Groups (SHG/KSM) spend profits on items/activities that contribute to improved household health and child survival. Groups will be engaged in the promotion of CS messages at the village level. NO  Most SHG do not have budget allocated for health. None is engaged in promotion of CS messages.  ADP staff trained in key CS interventions and ADP design plans including at least 20 percent of resources being spent on health. NO  ADP staff had been trained in child survival and ADP budgeted 10% of its budget for health in FY05 World Vision 12 LCSP Final Evaluation 2. Results: Technical Approach a. Project Overview The Landak Child Survival Project (LCSP) was implemented in a different site from the one for which USAID originally awarded a grant to World Vision US. The original four-year grant (2000-2003) was awarded for a project based in Poso, Central Sulawesi. USAID approved a change in project site after social unrest and insecurity made the original project site inaccessible. Annex 2 presents a chronological list of the milestones of the LCSP that includes the period before the change in project site. Annex 3 presents a timeline of the overall project implementation of the LCSP in Landak District. The LCSP covers two sub-districts (Mandor and Sengah Temila) of the rural area of Landak District in West Kalimantan Province, which are also two of the four sub district of the Indonesia Pontianak Area Development Program1 (ADP; see maps in Annex 4). The total population of the project area is 75,677, with 1.970 infants and 10,225 children under five. There are three Health Centers (HCs) in the project area: on in Mandor and two in Sengah Temila (Senakin and Pahauman). Annex 5 presents these and other demographic and health services data for the three Health Centers areas and the total project area. The Goal of the LCSP was “to assist the Landak District Health Office to reduce mortality and morbidity and nurture growth and development (thriving) among infants, children and mothers through an integrated child survival and rural enterprise project in Mandor and Sengah Temila sub-districts.” The main strategies defined in the DIP were (in summary): 1. To improve the quality and coverage of existing child survival interventions; 2. To empower communities for prevention and control of common diseases; 3. To initiate micro-enterprise efforts and promote collaboration in health; 4. To develop a child survival intervention model for further replication and scale up The LCSP Detailed Implementation Plan (DIP) focused on four technical interventions: immunization, vitamin A supplementation, control of diarrheal diseases, and control of malaria. Upon recommendations from the DIP review, the LCSP staff added the control of acute respiratory infections (ARI) as fifth intervention. The local partners of the LCSP were the Ministry of Health (mainly the Landak District Health Office (DHO) and the three Health Centers in the project area), the Pancur Kasih Foundation, and the community-based groups and volunteers in the project area. The LCSP staffing comprised 1 Technical Team Leader, 1 Training officer, 1 Monitoring and Evaluation Officer, 1 Financial Officer (hereafter called “the Core Team”), 7 Health Motivators, 1 Driver, 1 Janitor, and 1 Administrative Assistant. The LCSP was otherwise under the overall management responsibility of the Pontianak Area Development Program (ADP) and the 1 An Area Development Area (ADP) is the World Vision unit of operations based on a 10 to 15 year commitment to one area and a gradual multisectoral development approach. World Vision 13 LCSP Final Evaluation technical supervision of the Health Unit of World Vision Indonesia (WV Indonesia) in Jakarta, with 1 Program Officer fulltime on the project working under the guidance of the National Health Advisor. The DIP workshop for the LCSP took place in October 2000 while WV Indonesia was still recruiting staff for the new project site. The Technical Team Leader and most of the other project staff were recruited and on site by April 2001 when the first training event took place. Training of various community health workers began in June 2001 and continued until the end of the project, although most of this training was completed by June 2003. The First Annual Review (FAR) was conducted in September 2001 by a team led by an external consultant, with other members from the MOH (national, provincial and district levels), USAID Jakarta, and World Vision (WV Indonesia and Asia Pacific Regional Office (APRO). The MTE was conducted with similar participation from various stakeholders in August 2002, a little more than a year before the end of the grant period. Given the satisfactory progress despite the change in project site and the resulting one-year delay, one of the main recommendations of the MTE was to apply for a two-year no-cost extension2 to fully achieve the project's scale-up, capacity building and sustainability objectives. However, WVUS decided to seek a one-year no￾cost extension, which was approved by USAID in May 2003. The Third Annual Review (TAR) was conducted in October 2003, and focused on achievements and constraints since the MTE, with recommendations for the last year of the project. The FY 2004 Work Plan was prepared during the TAR, and progress achieved by September 2004 was summarized in Annex 6 of the TAR team's report. In May 2004, a second request for a one-year No-cost Extension was submitted on the grounds that a significant proportion of the project funding remained unspent3 and that new opportunities to scale up project activities to two additional sub-districts and improve the potential for sustainability had arisen. This second no-cost extension request was not approved by USAID and, in June 2004, LCSP began the process of closing down project operations, and preparations for a final KPC survey and the Final Evaluation (FE). Annex 7 presents the results of the KPC survey for the 14 key LCSP indicators as defined in the DIP and in the MTE report, with a mention of whether the DIP target was reached or not. Annex 8 compares these results with those from the midterm and the baseline KPC surveys. These two annexes present these results in a tabulation and graphical format. All these results were available to the FE Team at the beginning of its fieldwork. The terms of reference and team composition of the FE Team are in Annex 9. The FE Team comprised 29 (22 members + 7 motivators) members who worked together from September 20th to 30th. The FE methodology and schedule are presented in Annex 10 and followed by the findings and tools of all the site visits (Annex 11 thru Annex 20) and lists of participants in various meetings organized during the Final Evaluation (Annex 21). 2 Less than one third of the project funds were spent at the end of the second year of the three-year LCSP (see MTE report). 3 One third of USAID grant amount of $999,712 remained unspent at the end of March 2004. Current estimates do show grant and match funds expended by the end of the project, given costs associated with intensified techncial assistance and procurement of 24,000 insecticide-treated nets for ongoing distribution. World Vision 14 LCSP Final Evaluation b. Progress by intervention area The five sections below discuss the LCSP technical interventions starting with a brief description of the main strategies and continuing with a figure and discussion of the KPC surveys results. For consistency purpose, the results of the baseline and midterm KPC surveys are those reported in the MTE report and the results of the final KPC are those for indicators defined at midterm, except when noted otherwise.4 Each section continues with a discussion of the factors of success and constraints, the lessons learned and special outcomes, and the plans for continuation or scale￾up. All recommendations specific to the technical interventions are in the Conclusions and Recommendations (see section C). i. Immunization The LCSP immunization intervention consisted of making services available at the community level by reactivating existing and creating new Posyandus and by providing management and logistic support to the DHO and HCs. The LCSP also and supported behavioral change communication activities for mothers and other community members. 4 See Final Evaluation KPC survey report for further adjustments of these indicators and comparison of three KPC surveys’ results. World Vision 15 LCSP Final Evaluation Figure 1 KPC survey results for immunization 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 1 - Fully immunization # 1 - Target # 2 - TT2 coverage # 2 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 1 Full immunization: % of children 12-23 months completely immunized (verified by card) 23% 47% 61% 80% No 2 TT2 coverage: % of mothers with child less than 2 years of age received TT2 before the birth of their youngest child (verified by card) 2% 15% 44% 50% Yes Note: Target met if included in confidence interval around FE KPC results (see Annex 7) Both immunization coverage indicators show impressive increases over the four-year project implementation period. The percentage of children 12-23 months of age who are fully immunized (as verified by examination of the immunization card5 at the time of the survey) increased from 23% at baseline to 62% at the end of the project, or nearly threefold. The percentage of mothers with a child less than 2 years of age who could show a record of TT2 immunization at the time of the survey increased from 2% to 44%. Overall, the LCSP immunization intervention was successful, but the target of 80% children 12- 23 months fully immunized (verified by card) was not achieved. Annex 21 shows a graph of the percentage of children 12-23 months with an immunization card and the coverage of children 12- 23 months with each antigen and with all antigens (Fully Immunized Children). Access to immunization services as measured by the DPT1 coverage was found to be 82%, and the DPT1/DT3 dropout rate was 10%. Only 5% of mothers reported that their child had received a full series of immunization that had not been fully recorded on a card still in her possession. 5 The child immunization card can be the MCH Booklet (Buku KIA), which is provided to pregnant women and throughout the first five year of the child, or the Growth Monitoring Card (Kartu Menuju Sehat-KMS), which is provided to mothers without MCH Booklet. World Vision 16 LCSP Final Evaluation While the conservative TT2 coverage target of 50% was barely reached,6 the actual coverage is probably much higher than this, since 30% of mothers of a child less than 2 years of age reported having received two or more TT injections during their last pregnancy, but had no card to provide verification. One reason for low card-documented TT2 coverage in this setting is probably the short supply of MCH booklets (which include a form for documenting routine primary immunization and the mother's TT series) at community level. The LCSP team reports that the Landak DHO actually has enough stocks of MCH booklets7 but has difficulty distributing them. On several occasions, the LCSP team has facilitated this process, encouraging DHO or HC staff to distribute more when shortages in Posyandus were encountered. The FE KPC results for these two immunization indicators do not show statistically significant differences between health centers, level of maternal education or age of the mother. One concern during the MTE and the TAR was that the Fully Immunized Children estimates from the baseline and midterm KPC surveys were lower that those provided by the DHO for the same periods. To clarify these discrepancies, the FE Team collected various services statistics reports from the three project Health Centers and the DHO. Only the DHO statistics were suitable to calculate coverage levels and trends for the three LCSP Health Center areas. The DHO uses coverage of measles immunization as indicator of full immunization, assuming that all children receiving that antigen received the others before. Figure 2 shows that the measles immunization coverage estimates from services statistics (82%) and from the KPC surveys (77%) are consistent in 2004 but that their trends over the three previous years differ substantially: the KPC surveys show an increase similar to that shown above for Full Immunization but the services statistics do not show any change.8 These data confirm the past discrepancies between the DHO and KPC estimates but leave open questions about their nature. 6 KPC TT2 coverage estimate is 44%, with a 95% confidence interval from 37% to 51% (see Annex 7. 7 The MCH booklets is co-funded with the Government of Japan. Includes various educational as well as data recording components. It is the preferred MCH data recording tool but when it is not available the Growth Monitoring Card is used. The LCSP provided the DHO with 10,000 MCH booklets at the beginning of the project. 8 The trends in both the number of immunizations and the target used remain stable throughout the period too. World Vision 17 LCSP Final Evaluation Figure 2 Measles immunization coverage from services statistics and KPC surveys 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE Services statistics KPC surveys One LCSP key indicator for the immunization intervention, not measured through the KPC surveys and therefore not presented in Figure 1, refers to the quality of the cold chain (see section 1 - Results: Summary Chart). The LCSP provided cold chain equipment (refrigerators, generators, cold boxes and carriers) and the LCSP health motivators conducted regular assessments of the presence and status of refrigeration of the vaccines at the Posyandu levels using simple observation checklists (see section c.iii). The FE team assessed the quality of vaccine temperature monitoring in the three Health Centers and found clear deficiencies in two Health Centers and evidence of vaccines stored outside of their recommended temperature on the day of their visits (see Annex 11). Each Health Center still has a functioning generator but that is not exclusively dedicated to the vaccine cold chain. The FE Team also observed the immunization practices in one Posyandu located nearby a Health Center (see Annex 15). The ice pack had melted when it reached the Posyandu, and the number of vaccines was complete except for tetanus vaccines. One needle was used for one child, and the vaccines were given to children according to standard operating procedures. Despite these shortcomings, the FE team concluded that the 100% target for this indicator was reached. Following up on findings and recommendations from the MTE and TAR, the FE Team reviewed the waste disposal practices in the three Health Centers and found that only one had practices within acceptable standards. The two other Health Centers did not show progress as compared to similar observations made during the midterm evaluation and third annual review. The increase in immunization achieved by the LCSP is due to the revitalization of the 76 existing Posyandu and the opening of 32 new Posyandus (see section 3.a), the behavior change communication activities (see section 3.b) and the provision of cold chain equipment and two motorcycles to each project health center (see section 4.e). The FE Team identified the following constraints to sustainability or further increase of the current immunization coverage: World Vision 18 LCSP Final Evaluation  Many villages are located in remote areas with very difficult access during the rainy season;  The number of health facilities and health staff is limited yet the health staff is expected to conduct many activities;  There are numerous missed opportunities in remote hamlets where Posyandus have just started;  The availability and use of the MCH booklet is limited;  The traditional practice of prohibiting post partum mothers from leaving the house is still common and affects immunization of newborn babies.  Some health staff do not perform as expected and there is no system in place to rectify such situation;  Landak is a relatively new District with still limited human and financial resources; In addition to these well-known constraints, the LCSP Team learned that:  In villages where the cadres are proactive and highly motivated, they have been able to convince the health staff to visits at home the children who did not attend the Posyandu Day because they were sick;  Providing proper explanation to mothers about vaccination side effects, especially fever, has proven to very effective in reducing resistance from the community to bring children for immunization;  While the support that Posyandus receive from local health authorities improved over the life of the project, it remains limited and may very well decrease drastically at the end of the project. The plans of WV Indonesia and the ADP to expand LCSP activities in two other sub-districts and continuing supporting Posyandus activities in the whole ADP area are discussed in section 3.d. ii. Vitamin A Supplementation The LCSP vitamin A supplementation intervention consisted of educating mothers and other community members on the benefits and timing of vitamin A supplementation; training health volunteers on the benefits, timing and mode of administration of vitamin A supplementation; and supporting the District Health Office and health centers staff for the distribution of vitamin A capsules to children 6 to 59 months old and post-partum mothers. World Vision 19 LCSP Final Evaluation Figure 3 KPC survey results for vitamin A supplementation 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 3 - Vit. A suppl., 12-23 months # 3 - Target # 4 - Vit. A suppl., post-partum # 4 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 3 Vitamin A supplementation, 12-23 months: % of children 12-23 months received Vitamin A in the past 6 months 54% 60% 88% 90% Yes 4 Vitamin A supplementation, post-partum: % of mother received Vitamin A within one month of their last delivery 12% 26% 41% 50% No Note: Target met if included in confidence interval around FE KPC results (see Annex 7) The LCSP reached the 90% target for the key indicator for vitamin A supplementation of children 6 to 59 months old: 88% of children 12-23 months had received vitamin A in the 6 months before the FE KPC survey. This indicator is not “card documented” but vitamin A supplementation is usually recorded on the Growth Monitoring Card (KMS):9 the proportion of children 12-23 months with at least one record of vitamin A supplementation on their immunization card is 68%. Children above 12 months attend Posyandu sessions less regularly than infants do10 and some of this coverage is achieved through home visits during the Vitamin A supplementation day in February and August. Although there has been a steady increase in percentage of mothers of children under two who received vitamin A within one month after delivery, the LCSP did not achieve the 50% target for this indicator. The FE Team identified several reasons for this shortcoming. First, while most pregnant women receive antenatal care from a midwife, a nurse, or a doctor,11 they actually give birth with the assistance of a traditional birth attendant (TBA).12 Only when the TBA is trained and has a good network with the local midwife is the mother likely to receive a vitamin A 9 Kartu Menuju Sehat. 10 See section B.2.c.iii, Table 5. 11 In the KPC survey, 89% had a antenatal care visit by either a nurse or a midwive (87 %), a doctor (3%)or both. 12 In the KPC survey, 74% of mothers were assisted by a trained or untrained TBA (53% by a trained TBA and 21% by an untrained TBA. World Vision 20 LCSP Final Evaluation capsule within one month after delivery. In addition, the supply of vitamin A capsules for postpartum supplementation is often limited, except during the vitamin A months. Only a few Posyandu cadres track newborns and give vitamin A to postpartum mothers. The FE KPC shows statistically significant differences in the coverage of postpartum vitamin A supplementation across Health Center areas. This coverage is 27% in Mandor, 42% in Senakin and 53% in Pahauman. The large difference between Mandor and Pahauman is at least partially explained by the way these two health centers hold monthly meetings with TBAs. These meetings provide opportunities to give vitamin A capsules to TBAs and encourage them to give those capsules to post-partum mothers. In Mandor, the health staff holds these meetings in the Health Center while in Pahauman the health staff holds such meetings in each sub Health Center or Village Birthing Post. The more accessible meetings in Pahauman probably increase the number of TBAs who attend them and who can carry out this intervention. In addition, the Polindes and midwives seem to be more active and have better relationship with TBAs in their areas in Pahauman. iii. Malaria Control The LCSP malaria control intervention primarily consisted of health education for mothers and other community members in early recognition and treatment of malaria and in the use of ITNs by children under five and pregnant women. The LCSP also provided ITNs though various distribution channels and strengthened health centers in their capacity to make early diagnosis and provide effective treatment of malaria cases. World Vision 21 LCSP Final Evaluation Figure 4 KPC survey results for malaria control - treatment seeking for fever 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 5 - Seek treatment for fever # 5 - Target # 6 - Seek treatment for fever w/i 48 hours # 6 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 5 Seek treatment for fever: % of mothers with child less than 2 years of age who was ill with fever during the past two weeks seek treatment for their child 26% 58% 54% 75% No 6 Seek treatment for fever within 48 hours: % of children less than 2 years of age with a febrile episode that ended during the last two weeks were brought to a health facility within 48 hours after the fever began not avail able 30% 48% 45% Yes Note: Target met if included in confidence interval around FE KPC results (see Annex 7) The percentage of mothers who seek treatment when their child under two has fever increased from 26% at baseline to 54% at midterm. It then remained at that level until the end of the project. Although below the 75% target for this intervention, the level achieved for this indicator is probably appropriate given the fact that most cases of fever are caused by common self-limiting infections that do not need special intervention. The percentage of mothers with a child with fever who seek treatment within 48 hours increased from 30% to 48% between midterm and the end of the project, thereby meeting the 45% target for this indicator. This indicator was not assessed at baseline. World Vision 22 LCSP Final Evaluation Figure 5 KPC survey results for malaria control - ITN ownership and use 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 7 - ITN ownership # 7 - Target # 8 - ITN use last night # 8 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 7 ITN ownership: % of mothers with a child less than 2 years of age report that they have an ITN in their house not avail able 5% 23% 30% No 8 ITN use last night: % of children less than 2 years of age slept under an ITN the previous night not avail able 2% 21% 20% Yes Note: Target met if included in confidence interval around FE KPC results (see Annex 7) The two LCSP key indicator of ownership and use of ITNs were adopted and first assessed during the midterm KPC survey. These two indicators increased from a few percent at midterm to 23% (ownership) and 21% (use previous night). The FE KPC shows statistically significant differences for these indicators across Health Center areas: the use of ITNs for children under two was 14% in Mandor, 34% in Senakin and 18% in Pahauman (ITN ownership in the household was 15%, 37% and 19%, respectively). The LCSP achieved the 20% target of ITN use, the expected behavior that ensures protection against malaria. In the FE KPC survey, 64% mothers of a child under two declared that they knew the benefits of ITN, 33% declared that ITNs kill mosquitoes and 35% that ITNs prevent malaria. These results and the data on distribution of ITNs demonstrate that the distribution strategies and mass campaign implemented at the end of the project were successful and that most households with a child under two that own an ITN actually use it. The various efforts to find the best strategies to distribute ITNs and promote their use by pregnant women and mothers of children under two are further discussed in section c.i Since the beginning of the project, the LCSP staff has been actively engaged in informing health and other government officials of the importance of the economic and public health burden of World Vision 23 LCSP Final Evaluation malaria in the area. The information they provided notably included the primary role of illegal gold mining in the difficulty to control malaria transmission. In 2003, for instance, Pancur Kasih in collaboration with ADP organized a seminar on illegal gold mining attended by Government officials from Mandor and a few other sub-districts. At the end of this event, an agreement was issued that everyone present would not to allow any new site for illegal gold mining.13 After having introduced Long Lasting ITNs in the project area, the LCSP encouraged the Provincial and District officials to use their own funds to purchase additional units, which they did in 2002. The DHO also obtained a budget agreement to procure Deltamethrine tablets and will conduct mass campaigns and net dipping in the entire District, especially in those not reached by LCSP, as soon as they get cash disbursement. The LCSP effectively used the results of the Chloroquine Efficacy Study (see section c.ii) to draw attention of provincial and district public health officials on the need to make proper therapy available for malaria. Despite this important progress, malaria is still a public health in problem in Landak and the ADP area. Illegal gold mining is still a major problem and may be increasing, particularly in Mandor. The prevalence and the type of malaria in Landak are better known but not the geographic distribution of the risk. The resistance of falciparum malaria in Landak District is well established and Landak has been selected as a pilot site for testing a new combination therapy but the appropriate drugs may not be widely available any time soon. The use of ITNs by pregnant women and children under five has increased but there is no other source of ITNs than the LCSP. Therefore, the ADP has planned to continue promoting ITNs in its area of intervention in partnership with a local partner organization (see sections c.i and 3.d). With appropriate funding and partners, WV Indonesia is also in a good position to extend malaria control activities beyond social marketing of ITNs in the ADP area. Strategies envisioned with the Malaria Sub￾Directorate14 at the time of the FE include:  Surveillance and mapping: Identify the local areas at higher risk and the source of transmission using a methodology developed by the MOH and already in use in other malaria endemic areas of the country;  Introduction of Artemisinin Combination Therapy: Assist the MOH in introducing a new malaria drug already available for in selected areas in the country.  Screening and treatment of pregnant women: Assist the MOH in introducing this intervention using Rapid Diagnostic Tests at the Posyandus level.  ITN promotion campaigns: Provide discount vouchers to pregnant women and mothers of under two in high transmission villages and to everyone in high transmission hamlets.  Transmission control at the local level: Use of in-door spraying, larvicides and others appropriate methods in well-identified sources of transmission.  Advocacy: Use experience and lessons learned to conduct advocacy in project area as well at the Provincial and National levels 13 This agreement can be obtained from the ADP. 14 The Chief of the Malaria sub Directorate, MOH RI, was a member of the FE Team and has been providing technical assistance and guidance to the LCSP since the beginning of the project. World Vision 24 LCSP Final Evaluation iv. Diarrhea Diseases Control The LCSP diarrheal diseases control strategies consisted of education of mothers and other community members in effective home management of diarrhea and training of health workers and volunteers in standard case management of diarrhea and personal hygiene. Figure 6 KPC survey results for diarrheal diseases control 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 9 - ORS use # 9 - Target # 10 - Same or more fluids # 10 - Target # 11 - Same or more foods # 11 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 9 ORS use: % of children less than 2 years of age who had diarrhea in the last two weeks received ORS 32% 43% 52% 75% No 10 Same amount or more fluids: % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more fluids 63% 59% 78% 75% Yes 11 Same amount or more foods: % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more foods 34% 29% 39% 70% No Note: Target met if included in confidence interval around FE KPC results (see Annex 7) ORS use for children under two with diarrhea more than three times a day in the last two weeks steadily increased from 32% in the baseline to 52% in the FE KPC surveys. This last level of ORS use is below the 75% target for this indicator but is probably appropriate given that the percentage of children with diarrhea who received the same amount or more fluid increased from 63% to 78% during the same period, meeting the target for this indicator. Indeed, the LCSP promoted the use of any kind of home based fluid in case of diarrhea since ORS is not always available in the community. As Posyandu cadres typically receive ORS from health centers during the Posyandu activity and they typically do not have any stock in their homes between monthly Posyandu days. World Vision 25 LCSP Final Evaluation The percentage of children with diarrhea who received the same amount or more food remained around 30% throughout the duration of the project, far below the 75% target for this indicator. This intervention still needs more attention. One difficulty in promoting this behavior is indeed that a sick child, especially with abdominal discomfort, will most likely refuse eating anything or even vomit, thereby aggravating the fear of the mother to continue feeding. In addition, the 75% target might be difficult to achieve for children under two given that most of them are exclusively or partially breastfeeding. The FE Team examined the issue of insufficient availability of ORS at the Posyandu level raised during the MTE and the TAR. In Health Centers (see Annex 11), the quantity of ORS received from the District seems adequate and properly recorded and stocked. In one health center, the FE Team found 10 ORS boxes of 100 tablets about to be discarded because the expiration date (July 2004) was reached while in fact the quality of the product was still good (no lumping). The staff from the three Health Centers seems to regularly bring one box of 100 sachets of ORS each Posyandu day and leave it there at the end of the day. They also seem to provide additional supply upon request by Posyandu cadres but there is no clear mechanism for this to happen. The quantity of ORS given to the Posyandu seems properly recorded. At the Posyandu level (see Annex 15), however, the FE Team noticed regular complaints that their supply of ORS was inadequate. In addition to the interventions above to improve home-based management of diarrhea, the LCSP attempted to decrease the practice of giving antidiarrheal medicines to children with diarrhea. The target for this intervention was to decrease to 10% the percentage of children under two who had diarrhea in the last two weeks and received antidiarrheal medicines. Although the baseline and midterm KPC surveys shows 32% and 34% for this indicator, respectively,15 the FE KPC survey shows an increase to 50%. The LCSP team added one question to the FE KPC questionnaire to find out the type of pills or drugs most likely given. “Dumex,” a brand of tetracycline antibiotic very popular in the area (but contraindicated for children) was frequently mentioned by mothers, who said they had given a pill or syrup to their child during the most recent diarrhea episode, for example. This indicator (#12) may not adequately measure the practice of giving antidiarrheal medicines since the nature of the syrup or pills is not specified. However, the increase in this indicator in the last KPC survey is of concern and calls for further investigation. On several occasions during the FE, discussions of this issue with community members and health providers have suggested that the practice of treating children under two with diarrhea with pills or syrup, and even explicitly with antibiotics, remains indeed very common. Some of the antidiarrheal drugs given to the children are quite expensive, most often cotrimoxazole from various manufacturers. As cotrimoxazole is available but not very popular in the community, the increased utilization of this and other drugs may reflect an increase in care seeking behavior combined with poor prescribing practices by health workers. 15 The M&E Plan in the DIP and therefore the MTE report mention 40% for this indicator at baseline but the Baseline KPC report mentions 32%, the value considered as correct in this FE report. World Vision 26 LCSP Final Evaluation v. Acute Respiratory Infections Control The ARI control intervention was added to those included in the DIP during the FAR in response to a recommendation from the DIP review. There is no explicit (written) strategy for this intervention. The LCSP promoted education of mothers and other community members (Posyandu cadres, TBAs) about the signs of pneumonia and the importance of quick referral to a health center. Figure 7 KPC survey results for acute respiratory infections 0 20 40 60 80 100 Sep-99 Sep-00 Sep-01 Sep-02 Sep-03 Sep-04 PERCENTAGE # 13 - Knowledge of 1 pneumonia symptom # 13 - Target # 14 - Seek treatment for pneumonia # 14 - Target # Indicator BL MTE FE LCSP Target 9/00 7/02 9/04 Target Met 13 Knowledge of 1 pneumonia symptom: % of mother with children less than 2 years of age who mention at least 1 symptom of pneumonia 25% 40% 55% 80% No 14 Seek treatment for pneumonia: % of mother with children less than 2 years of age who had symptoms of pneumonia (cough and rapid or difficult breathing) seek treatment 59% 59% 72% 75% Yes Note: Target met if included in confidence interval around FE KPC results (see Annex 7) The percentage of mothers with children under two who could mention at least one symptom of pneumonia steadily increased from 25% in the baseline to 55% in the FE KPC surveys but remained short of the 80% target for this indicator. The percentage of mothers with children under two who had symptoms of pneumonia (cough and rapid or difficult breathing) who sought treatment for their child remained around 60% in the baseline and midterm KPC surveys but increased to 72% in the FE survey, thereby meeting the 75% target for this indicator. The shortcoming with respect to knowledge of pneumonia probably comes from the late start of this intervention and its initial focus on IMCI at the facility instead of at the community level. The project did promote health education on ARI and pneumonia for cadres, TBAs, shopkeepers and mothers attending Posyandu activities but the demonstration of the danger sign of World Vision 27 LCSP Final Evaluation pneumonia is quite difficult. The threshold of the normal respiratory rate varies with age and chest in drawing and stridor are difficult to demonstrate. The proper term to use for ARI during health education and in the education materials has been a problem since the beginning of the project. There is one local term for this word but still not very suitable. Near the end of the project, the LCSP team decided to use “ISPA” (the acronym for ARI in Bahasa Indonesia) since the pronunciation of this word is the easiest. The increase in treatment seeking at the end of the project reflects the more intensive efforts in 2004 to reach all mothers of children under two with specific messages on early recognition of danger signs and treatment of pneumonia (see section 3.b on the May/June pneumonia campaign). Also at the end of the project, the LCSP team developed a Video CD on the danger signs of ARI but could not show it in all Posyandus before the end of the project. c. New tools or approaches The DIP proposed to introduce three new “methods, strategies or materials:”  Community-owned EPI and Pregnancy Registers  Community-Based Death and Diseases Surveillance System (CBDDS)  Insecticide Treated Bednets for malaria control This section highlights the LCSP achievements in terms of:  Development of an ITN distribution strategy, as proposed in the DIP;  Implementation of a chloroquine efficacy study, a special study;  Community-Based Health Information System, in line with the two first interrelated topics above;  Implementation of a Doer Nondoer survey and the development of a related BEHAVE Framework;  Development and application of an Organizational Capacity Assessment in the context of a child survival sustainability assessment and planning exercise. i. Development of ITN distribution strategies The LCSP introduced Long Lasting Insecticide-Treated Nets16 (ITN; Permanet®) in the project area and explored the most effective and sustainable strategies to distribute them to ensure their use by pregnant women and children under five. Table 1 presents the timeline of the implementation of various strategies and their results in terms of sales of ITNs. 16 In this report, the ITN acronym is used for Long-Lasting ITNs because the LCSP was only involved with the distribution and promotion of this type of nets. World Vision 28 LCSP Final Evaluation Table 1 Development of the ITN distribution strategy in the LCSP area Fiscal Year FY01 FY02 FY03 FY04 ITNs on project site (#) 2000 2000 Distributors Self Help Groups Village Drug Posts Pancur Dangeri Posyandu Social Marketing Vouchers (#) 1000 2000 Mass Campaign Results Total sales (#) 193 562 1958 290 ITN use (KPC survey) 2% 21% Source: LCSP, September 2004 The first batch of 2000 ITNs was purchased in November 2001 but available in the project site in May 2002.17 Initially, the LSCP distributed these ITNs mainly through the Self Help Groups sponsored by the ADP. The nets were stored in the ADP warehouse and the Self Help Group members could obtain them from the ADP staff and sell them as an income generating activity. A total of 31 SHGs were involved in the four sub-districts of the ADP Pontianak: 8 in Mandor, 13 in Sengah Temila, 5 in Toho and 5 in Sungai Pinyuh. At the same time, the LCSP team encouraged the promotion of these ITNs during health education sessions in Posyandus and distributed posters about the danger of malaria and the benefits of ITNs to Posyandus, local community leaders, Village Drug Posts, shopkeepers, and Self Help Groups. By January 2003, 351 ITNs had been sold, including 156 in Sengah Temila and 37 in Mandor (193 in the project site). Another 439 ITNs were given free, leaving 1210 in stock. Realizing that the low rate of sales was due to the lack of marketing skills of the Self Help Group members, the project then decided to involve the Koperasi Pancur Dangeri18 (KPD) as the wholesale distributor and keep the Self Help Groups as last line distributors. The Village Drug Posts were also involved in the sales of ITNs at that time. In May 2003, the project also issued 1000 30%-discount vouchers for pregnant mothers and mothers of children under five (300 vouchers were distributed in Mandor, 300 in Senakin and 400 in Pahauman). By January 2004, the KSU Pancur Dangeri had sold 562 ITNs, leaving 648 in stock. The second batch of 2000 ITNs arrived at the project site at the end of 2003. The nets of this second batch were all XL size because it had become clear at that time that the demand for nets of this size was higher because they can be used by the whole family. The LCSP also replaced 500 L size nets remaining in stock at KPD with XL size nets. 17 The 2000 ITNs arrived in Pontianak port in January 2002 but were not released before April 2002 because of delays to obtain the tax exemption. 18 Koperasi Pancur Dangeri is a cooperative organization affiliate of the originally proposed LCSP partner Pancur Kasi Foundation. World Vision 29 LCSP Final Evaluation Early 2004, given the improved but still slow rate of sales, the LCSP team decided to conduct a Malaria Month Campaign to increase community awareness of the risk of malaria and the benefits ITNs. As reports of discussions with community members had shown that the price was the main constraint to the purchase of ITN, the campaign was purposely conducted in March, after the harvest season and when most people possess some cash money.19 Starting during this campaign, the KSU Pancur Dangeri also added the Posyandu cadres as last line distributors to reach communities up the sub-village level and more effectively target pregnant women and mothers of young children. Posyandu cadres just like any other last line distributors could earn Rp. 10,000 per ITN sold. Annex 23 shows the ITN distribution lines adopted at that time. Also at the time of the campaign, the project issued 2000 45%-discount vouchers with the restriction that one family could only use one voucher and that the purchase of ITN should be done through the Posyandu cadres (the discounted price was only slightly higher than the price of a regular net). Vouchers were distributed by the cadres during the Posyandu day using the Growth Monitoring Cards or MCH booklets to identify the beneficiaries. Considering the limited number of ITNs available at that time (about 2500 XL size), 20 vouchers were initially allocated to each of the 108 Posyandus. This number was revised throughout the campaign month by distributing vouchers from one Posyandu to another depending on the demand. By the end of April 2004, KSU Pancur Dangeri reported that 1958 discounted ITNs had been sold. In August 2004, an additional 290 full-price nets had been sold. Table 2 shows the number of Posyandus that were selling discounted ITNs during the Malaria Month campaign and a few statistics on sales per Posyandu in each Health Center area. The table also presents estimates of the coverage potentially achieved if all these ITNs were used for children under five.20 These coverage estimates are of the same order of magnitude as those of the FE KPC survey. Both these estimates and those from the FE KPC survey show a higher coverage in Senakin, 21 probably because KPD has its main office and a larger number of its branches in this area. Table 2 ITNS Sales during Mass Campaign and Coverage of Target Group in three Health Center Areas Health Center Area Posyandu ITNs Sales Statistics Coverage of Existing Selling Total Min Max Average Under Five MANDOR 32 29 690 3 42 24 3589 19% SENAKIN 31 27 665 4 77 26 2619 25% PAHAUMAN 45 27 603 2 61 22 4017 15% Total 108 83 1958 2 77 24 10225 19% The FE Team interviewed the KPD manager in Senakin to obtain his feedback on the Malaria Month Campaign (see Annex 19). He recognized the roles of the discounted prices and that of the awareness raising activities in the success of the campaign. He found the involvement of 19 More detail on the Malaria Month Campaign are provided in section B.3.b. 20 Number of under five from Annex 5. 21 The FE KPC survey shows statistically significant across Health Center, with the use of ITNs for children under two at 14% in Mandor, 34% in Senakin and 18% in Pahauman (ITN ownership in the household was 15%, 37% and 19%, respectively). World Vision 30 LCSP Final Evaluation Posyandu cadres as last line distributor satisfactory and welcomed their continued involvement in the sale of ITNs through KPD as long as they respect the sales policies. The LCSP learned that not only price but also the availability of cash money in the community are important determinants of a family’s decision to purchase a net. The success of the mass campaign is attributable to the combination of the reduced price, close to that of a regular net, and to its timing during the harvest season when most people possess cash in their home. In terms of sustainability, involving the Posyandu cadres as last line distributors is a promising strategy. However, the community is still dependent on subsidies from the project for the vouchers, and other ways to keep the price down without making the community dependent on external donors must be found. Given the success of the Malaria Month Campaign, and considering the significant amount of grant funds remaining unspent by the last quarter of FY 2004, World Vision sought and received approval from CSHGP to use a portion of these funds to procure a further batch of ITNs for ongoing distribution by the ADP and local partners. With the cooperation of USAID/Indonesia, the required Pesticide Evaluation Report and Safe Use Action Plan was prepared and filed, and LCSP received permission to purchase an additional 24,000 ITNs. Plans for future ITN distribution are discussed in Section 3.d.ii, and ADP Pontianak's ITN Distribution Plan is in Annex 33. ii. Chloroquine Efficacy Study The LCSP includes among its malaria control interventions the (1) Promotion of early recognition and treatment of malaria for children under five and pregnant women and (2) the early diagnosis and effective treatment of cases of malaria by health providers. During the MTE, health workers from Health Centers in the project area reported frequent occurrences of patients with clinical manifestations of malaria that showed unsatisfying results with correct chloroquine treatment, the first line drug for the treatment of uncomplicated malaria according to the Indonesian Ministry of Health’s protocol. However, there was no definite data on the level of resistance of Falciparum Plasmodium to chloroquine in West Kalimantan and therefore no evidence to support a change in the first line malaria drug policy in the project area. Given the importance of the availability of effective malaria treatment for the success of the two project strategies above, the MTE team recommended that the Landak DHO and the LCSP/ADP Pontianak conduct an in-vivo chloroquine efficacy study in the project area. This study was conducted from May to July 2003 in 32 villages of 5 of the 10 Sub-districts in Landak Districts (Mandor, Sengah Temila, Ngabang, Menyuke and Mempawah Hulu). This study aimed at identifying subjects with malaria parasites in their blood and comparing the persistence of these parasites before and up to the 28th day after the administration of chloroquine at a standard dose. The protocol was slightly adapted from the 2001 WHO protocol for such studies and from the protocol adopted by the Ministry of Health in Indonesia. Given the low malaria endemicity in Kalimantan, the study population was selected from mass blood surveys conducted in villages likely to have the highest number of malaria infections. Residents of all ages were screened for falciparum malaria infection through on-site examination of blood smears. All enrollment and follow up procedures were then conducted in the homes of the participants. World Vision 31 LCSP Final Evaluation Although the mass blood survey was conducted to identify cases to enroll in the efficacy study, it provided estimates of the malaria parasitemia prevalence that are most valuable for developing malaria control strategies in the area. Table 3 shows the results of this survey, which included a total sample of 9,868 people among whom 607 were found malaria positive (malaria parasitemia prevalence = 6.2%) and 335 (55% of all malaria positive) were infected with Plasmodium Falciparum alone or in combination with Plasmodium Vivax. WHO classifies areas with parasitemia prevalence below 10% as low transmission or hypo endemic. Table 3 Results on epidemiology of malaria from Mass Blood Survey Blood Sample Examined Positive Results for Malaria Malaria Parasitemia Prevalence Species found Pf Pv Mix Pf (%) 9,868 607 6.2 % 324 272 11 55.2 % Note: Pf: Plasmodium Falciparum; Pv: Plasmodium Vivax; Mix: both Pf. and Pv. found; Pf (%): % malaria infection due to Plasmodium Falciparum alone or in combination with Plasmodium Vivax. A total of 219 cases enrolled in the study out of the 335 found positive for falciparum malaria. Table 4 shows the results of the chloroquine efficacy tests for the entire study population. Overall, there were 11 cases of Early Treatment Failure (ETF—clinical and parasitological criteria for treatment failure are met with three days of the treatment), 90 cases of Late Treatment Failure (LTF-- clinical and parasitological criteria for treatment failure are met with only after 3 days and before the 28th day) and 64 cases of Adequate Clinical and Parasitologic Response (ACPR). Taking into account the 54 cases declared as Drop Out, the Total Treatment Failure (ETF + LTF) rate was 61.2%. Table 4 Results on chloroquine efficacy Total Enrolled ETF LTF ACPR DO TTF % 219 11 90 64 54 61.2 Note: ETF = Early Treatment Failure; LTF = Late Treatment Failure; ACPR= Adequate Clinical Response; DO = Drop out; TTF% =Total Treatment Failure % =100*(ETF + LTF) / (ETF +LTF+ACPR) The Landak chloroquine efficacy study clearly demonstrated the high level of chloroquine resistance in the project area. The final analyses and report have not been issued at the time of the FE. The LCSP team has conducted advocacy activities at the district and provincial levels based on the preliminary findings (see section b.iii). In May 2004, Sengah Temila and Mandor sub-districts were chosen as two pilot areas to test a new malaria Artemisinin-based Combination Therapy in which the LCSP is not involved. iii. Community-Based Health Information System World Vision identified the need to strengthen community based health information systems (CBHIS) in the project area during the preparation of the DIP in October 2000. The DIP proposed to introduce community-owned EPI and pregnancy registers and a Community-Based World Vision 32 LCSP Final Evaluation Death and Disease Surveillance System (CBDDS) in a few pilot-villages by adapting and testing tools that had been successfully implemented in other World Vision Child Survival projects. 22 At midterm, the LCSP had already introduced a revised Posyandu Monthly Report (PMR), the report that the MOH supports to collect data from Posyandu and integrate them in its routine health information system. The LCSP had also introduced a series of new monitoring tools such as the Posyandu Scoring Form, the Exit Interview (EI) and the Observation Checklist (OC) for immunization services, and the Immunization and Pregnancy Tracking Forms (ITF and PTF). The MTE team recommended that the LCSP assist the DHO in the management, analysis and interpretation of the data routinely collected through the PMR and begin monitoring the implementation of the various tools it had introduced. The MTE Team also recommended beginning the implementation of a CBDDS only if the project was extended. In October 2003, the LCSP conducted a Training Workshop on community-based health information systems in cooperation with the Provincial Health Office. A total of 17 participants from health and planning government services and 12 World Vision staff attended this workshop facilitated by 5 LCSP and 2 external facilitators.23 Given the diverse background of the participants, the workshop focused on general principles and tools for the design of HIS and on the importance of community decision-making and participation in the design of CBHIS rather than on the application of specific tools in the LCSP area. At the end of the workshop, the facilitators presented to the participants four options for the development of the CBHIS in the LCSP area and the Landak District:  Assess and strengthen the existing Posyandu-based information system of the DHO, primarily the Posyandu Monthly Reports.  Continue to pilot-test additional tools for monitoring Posyandu activities.  Pilot test a Community-Based Disease and Death Surveillance (CBDDS) system in one or two LCSP areas.  Facilitate the participatory design and implementation of CBHIS in selected communities.24 Although they expressed interest in the other options, the majority of the workshop participants from the government services supported option 1. The LCSP staff then recognized the need for more specific information on the status of the CBHIS in the project area such as its completeness, timeliness, validity and use before planning activities to conduct until the end of the project. A team of four LCSP staff and one external consultant then conducted a series of field visits to further assess the need and opportunities for developing CB-HIS in the project area. They met with the DHO staff, visited a sample of Health Centers, Village Birthing Posts and Posyandus and examined the various tools and data flow in two sub-districts. This 22 Community-Based Disease and Death Surveillance (CBDDS) systems have been developed and implemented in World Vision Child Survival projects in Bangladesh, the Philippines, Cambodia and India. They focus on surveillance of EPI diseases and maternal and neonatal deaths and include participatory case investigations. 23 See CBHIS Training Workshop Report, LCSP, October 2003. 24 This option refers to the methodology presented in the workshop and based on the analysis of community decision-making for health. (http://www.childsurvival.com/documents/CSTS/C-HIS_Final.pdf) World Vision 33 LCSP Final Evaluation assessment led to a series of recommendations for activities to conduct during the last year of the project.25 One recommendation of this assessment was to conduct a comprehensive analysis of the available community-based data for Year 2003 to draw conclusions on Posyandus performance during that year, identify analysis and reporting template supporting decision making at the community level, and identify changes to make in the current information systems before the end of the project. Annex 24 presents three graphs illustrating the data compiled from all the Posyandu Monthly Reports for year 2003 available at the LCSP office.26 Figure 1 shows that the reporting completeness was around 90% on average in the three Health Center areas, with a marked drop in Mandor and Pahauman in the last quarter. The LCSP M&E Officer has achieved these relatively high completeness rates for the purpose of this special “Year 2003 Analysis” by tracking these reports intensively through the LCSP health motivators and visits to the Health Centers. About two thirds of the reports were obtained from the motivators and the remaining was found in the Health Centers. Figure 2 presents the attendance rates of infants (bayi) and 1-4 years old (balista) on Posyandu Days (number of infants attending the Posyandu divided by the target number of infants) and Figure 3 present the percentage of Posyandus where an immunization officer was present. These two indicators reflect a satisfactory performance in the Posyandus that submitted a report. Table 5 presents the annual average of these three and other indicators, and, for reference, the number of infants, 1-4 years old, and Posyandus, in each Health Center and the entire LCSP areas. While the attendance rates are probably quite reliable, the number of diarrhea cases and deaths are far lower than expected in the population. Table 5 Selected Indicators from Posyandu Monthly Reports for Year 2003 in Mandor, Senakin and Pahauman Indicator Mandor Senakin Pahauman LCSP Completeness of Reporting 97% 84% 86% 89% Presence of immunization officer 90% 91% 83% 87% Infant target (#) 690 409 571 1670 Attendance rate of infants (bayi) 60% 77% 80% 71% Attendance rate of 1-4 year old (balita) 32% 38% 47% 39% Number of pregnant women per month 92 97 119 308 Number of diarrhea cases per month 22 19 14 55 Number of deaths under five 28 20 26 74 Total population, infants (bayi) 692 504 774 1970 Total population, 1-4 year old (balita) 2897 2115 3243 8255 Total number of Posyandus 29 30 44 103 Source: See footnote 26. Several aspects of the PMR system have not been documented by the LCSP: use and usefulness of the records and reports at the community level or for the supervisor from the Health Center; 25 See detail on the assessment findings and recommendations in the CBHIS Assessment Report, LCSP, October 2003. 26 See TREND KESEHATAN DAN KEPENDUDUKAN TAHUN 2003 DARI DATA BERBASIS MASYARAKAT, Analysis of Posyandu Monthly Activity Reports from Mandor, Senakin and Pahauman Health Center Areas, Landak District, West Kalimantan, Indonesia . Draft prepared for the Final Evaluation Team, LCSP, World Vision Indonesia, September 2004. World Vision 34 LCSP Final Evaluation potential use of each indicator; quality of the data recorded and reported.27 Answering such questions would require data collection and analyses of a qualitative nature. Although the Posyandu Monthly Reporting system in the project area still has deficiencies, the data compiled by the LCSP for Year 2003 show that it has potentially useful applications at the Health Center and District levels. WV Indonesia has plans to continue these analyses in collaboration with the Department of Maternal and Child Health.28 Such collaboration in the analysis, interpretation and dissemination stage would ensure that findings are directly relevant to the current efforts to develop Posyandu activities at the national level. In the meantime, the DHO and the LCSP have already adopted the last version of the Posyandu Monthly Report and the LCSP provided the DHO with 600 sets of these forms (see Annex 25). In FY 2002, the LCSP developed a Posyandu scoring system that allows comparing and monitoring Posyandus over time and identifying the specific needs for improvement of each Posyandu or group of Posyandu. A scoring sheet of about 40 items was filled in by the motivators and the health center workers and the data was entered and analyzed at the LCSP level.29 This scoring was conducted five times between October 2002 and March 2004. The LCSP team abandoned this system in 2004 to replace it with a simpler one based on the qualitative assessment of five core functions of the Posyandus leading to their classification into one of three classes. This system was developed with the motivators who then used it frequently and found it better reflected the true performance of the Posyandus. All Posyandus were assessed using this tool in March and in June 2004. The FE Team found it was too early to assess whether the new system met the need for a simple and sensitive tool to assess and monitor the performance of Posyandus and how it compares with the earlier Scoring system developed by the LCSP or with the national system that the MOH is developing.30 In October 2001, the LCSP introduced two tools31 to assess the quality of immunization services at the Posyandu level: The Exit Interviews guideline and the Observations Checklist. These tools helped identify weaknesses in the cold chain, in safety of injections and disposal of wastes and in vaccine availability. The motivators have used these tools quite regularly for a while but not systematically. At the time of the FE, these tools were not used by the DHO or Health Center for supervision of immunization services at the Posyandu level. In March 2004, the LCSP team developed another Exit Interview guideline but this time to monitor the impact of the malaria and ARI mass campaigns. This tool was also used by motivators, sometime with the involvement of the cadres, and analyzed at the level of the LCSP office. The LCSP also introduced the EPI and Pregnancy Tracking registers proposed in the DIP. They have first been used by motivators as data collection tools, compiling data from the Posyandu registers and sometimes from the midwives records but not by community members as they are intended to be. They later have been used by midwives in Senakin but without any protocol for 27 The limited data collected by the FE Team on that questions raise concern (see Annex 15 and Annex 16). 28 One representative of this Department was part of the FE Team and took the lead in the assessment of the Posyandus. 29 See description of the system and analyses done as of October 2003 in the the CBHIS Assessment Report, LCSP, October 2003. 30 See “Indikator Tingkat Perkembangan Posyandu.” The LCSP team did not collect the data require for the MOH indicators but could make a rough classification of all the Posyandus in the project area on the basis of their knowledge of the situation in the field. According to this system, most Posyandu in the LCSP area would still be considered at an “Elementary” stage, the lowest of the four-category system. 31 See description and use until October 2003 in CBHIS assessment report. World Vision 35 LCSP Final Evaluation assessing their comparative advantages. The LCSP did not introduce the CBDDS proposed in the DIP beyond the addition to a few indicators to the revised Posyandu Monthly Report at the beginning of the Project in April 2001. These indicators such as the number of infants or 1-4 year olds that got diarrhea, got fever or died were not further defined making them difficult to analyze, interpret and use. The MTE Team recommended not beginning a CBBDS if the project was not extended and the TAR Team renewed this recommendation at the news of the approval of one-year project extension. iv. Doer-Nondoer Survey of ITN Use In April 2004, the LCSP team conducted a doer nondoer survey to determine the key determinants of use of ITNs by pregnant women and mothers of children under two. This study aimed at assessing the actual use of ITNs by the target groups just after the Malaria Month Campaign and at further developing the related behavior change strategy. The methodology consisted of interviewing mothers of children under two who had bought an ITN the previous month regarding the advantages/disadvantages of using ITNs, what makes it easier/more difficult to use ITNs, and who would approve/disapprove them for using ITNs. At least 20 doers (pregnant women or mothers whose children under two slept under an ITN the previous night) and 20 nondoers were to be interviewed in each Health Center area. As the data collection proceeded, it quickly turned out that it was very difficult to find “nondoers” and that therefore most owners of ITN were actually using them. Among the 110 interviews found valid at the analysis stage, 23 (21%) were “nondoers.” As it appeared that there was probably no difference between areas, the project staff analyzed the available data for the entire project area. The findings of the study were used during a follow-on training workshop to develop a BEHAVE framework for use of ITNs in the project area (see section 3.b). v. Organizational Capacity Assessment The LCSP held an Organizational Capacity Assessment (OCA) workshop in Pahauman from March 18 to 20, 2004. The workshop aimed at assessing the LCSP partner organizations prior to developing a project Sustainability Action Plan for the project (see section 3.d). The workshop was attended by 17 participants composed of all LCSP staff (all core team and a few motivators) and several ADP staff. The participants defined the most important element of organizational capacity to assess and the related indicators and scoring system; developed the OCA Questionnaire and tested it on the ADP; and applied the OCA Questionnaire to the three Health Centers and the three Posyandu Cadres Associations of the LCSP area. All participants agreed that the OCA indicators must be flexible enough to be used for all organizations but sensitive enough to be used as an evaluation instrument. The OCA Questionnaire developed during the workshop and the results of its application for the three Health Centers and the three Posyandu Cadres are presented in Annex 26. Although the values of the scores are difficult to interpret without standards to compare them with, they help identify areas of weakness and potential improvement. For instance, the three Cadres Associations show very low scores for their management structure and communication capacity but a very strong score for leadership and ownership. This corresponds well to new organizations created at the initiative of their members and leaders. The three Cadres Associations have otherwise lower scores than the three Health Centers. The three Health Centers have in general very similar scores, with the main weaknesses in the areas of Vision and World Vision 36 LCSP Final Evaluation Mission Statements, Networking and Fundraising, three element that are typically not developed in government services. The ADP scored the higher than all the Health Centers and Cadres Associations, and the lower scores were in the areas of Vision and Mission Statement and that of Fundraising. These assessments are potentially very useful if followed up with action plans and if they are repeated after a year or two to assess improvement in organizational capacity, draw conclusions on the effectiveness of the efforts made so far and develop new objectives and action plans. 3. Results: Cross-cutting Approaches a. Community Mobilization The fundamental strategy of the LCSP has been to revitalize existing and create new Posyandus, thereby supporting this national program in the project area. The LCSP trained various community members involved in health activities and often linked them with health services. The LCSP employed on average seven health motivators for community mobilization and advocacy. All recent graduates in nursing, sanitation or similar background and originally from the project area, the LCSP motivators have been actively mobilizing village leaders and community members to support Posyandus. They have supported Posyandu cadres through training and direct involvement in their activities before, on and after the Posyandu days. They have also been actively involved in obtaining commitment from Health Centers’ staff to provide services in Posyandus. The LCSP provided each motivator with a motorcycle to enable them to reach remote Posyandus and closely supervise their activities. World Vision 37 LCSP Final Evaluation Table 6 shows the progression of the number of and the population with access to Posyandus during the period of the LCSP. At the beginning of the project in 2000, there were 76 Posyandus in the three health centers areas, among which 44 were functioning.32 In Pahauman, the number increased by 20, in Mandor by 11 and Senakin by only by one. At the end of the project there were 108 functioning Posyandu in the project area. This number corresponds to about 92 children under five per Posyandu,33 which is in the range of 50 to 100 recommended by the MOH. Using sub-village level demographic data and information on Posyandu attendance from the LCSP motivators,34 the LCSP team estimates that the population with geographical access to a Posyandu increased from 77% in 2000 to 95% at the end of the project in 2004. The relatively smaller gain in coverage in the last years of the project reflect the fact that Posyandus were last opened in the most remote areas, which are typically small villages, or in sub villages that already had some access to a Posyandu in a nearby village. These satisfactory results in terms of increase in the number of Posyandu and population covered do not adequately represent the probably more important results in terms of improved services. 32 See DIP page 58. 33 Annex 5 shows that there about 10,000 children under five in the project area. 34 For these calculations, the entire population of the sub villages with a Posyandu and a percentage of the population of the sub villages without a Posyandu, as estimated by LCSP motivators based attendance data in neighbouring Posyandu, are counted as being covered. World Vision 38 LCSP Final Evaluation Table 6 Number of Posyandus and population covered Health Center 2000 2001 2002 2003 2004 Mandor 21 22 28 29 32 Senakin 30 28 29 30 31 Pahauman 25 42 43 44 45 Total 76 92 100 103 108 Population covered (%) 77% 89% 93% 94% 95% Source: LCSP, September 2004. As part of the process of revitalization and creation of new Posyandus, the LCSP trained 1345 community members to provide various health services. Below are a few comments on the coverage of community members that the LCSP training program has achieved. Further comments on the training approach and the impact on health workers performance are provided in sections c.iv and c.v  The LCSP trained 703 Posyandu Cadres. Allowing for a some turnover and absenteeism, this number is consistent with the average number of 4 to 5 cadres present on Posyandu days reported in the Posyandu Monthly Reports during FY2003 (see section 2.c.iii). The group discussion held by the FE Team with senior Posyandu cadres from the three Health Center areas suggest that Posyandus are not always run by five cadres, the ideal number recommended by the MOH (see Annex 14).  The LCSP trained 285 TBAs and the FE KPC survey shows that trained TBAs assist 53% and untrained TBA 21% of the approximately 2000 births per year in the project areas. The interview of TBAs held by the FE Team suggests that each trained TBAs assist about 3 to 5 deliveries per year (see Annex 13).  The LCSP trained 320 shopkeepers, which corresponds to about 10 per village and 3 per sub village. The FE field visit in one of the shops and the KPC survey findings suggest that antibiotics may still be widely used for the treatment of children with diarrhea. It is not clear whether this intervention has brought the results initially expected.  The LCSP trained 37 community members to establish 14 Village Drug Posts (Pos Obat Desa--POD) in Mandor and Senakin. The FE Team found that about 5 of these 14 PODs function satisfactorily. The function of the PODs and their status at the end of the LCSP are discussed in section d.iii  In 2004, the LCSP trained 45 village leaders and culture stakeholders to provide support to Posyandus and PODs and to other project related activities.  In 2004, the LCSP trained 20 members of Cadres Associations in organization development and management (see c.ii). Overall, the LCSP was able to reach, train and support adequate numbers of Posyandu cadres. Other LCSP activities to support Posyandus are described in section 2.c.iii (Posyandu information system), section b (development, production and distribution of BCC materials), section c (support to health centers for providing services in Posyandus) and section c.v (training of Posyandu cadres). These community mobilization efforts are probably responsible for most of the increase in coverage of child survival interventions discussed in section 2.b. The LCSP also reach and trained adequate numbers of TBA and shopkeepers but support to their activities after World Vision 39 LCSP Final Evaluation training has been limited. The training of cadres to run Village Drug Posts in remote villages has not led to the expected number of these posts; the limitations of this strategy is discussed in section d.iii The success of the LCSP in creating and revitalizing Posyandus and the related increase in coverage of child survival interventions suggest that this strategy of the MOH is appropriate and promising. The MOH and other development agencies would therefore benefit from a full documentation of the experience of the LCSP to help applying the successful strategies in other areas. One of the strategies that the LCSP has used from the beginning of the project is to employ health motivators from the project area. This has probably ensured their commitment to bring change into their communities and their acceptance by the communities they served. This also increases the chances that they stay and continue to serve their communities at the end of the project. The increase, sometime large, in twelve of the fourteen key indicators measured by the KPC surveys suggests that the demand for the child survival interventions promoted by the LCSP in the project area is strong. In addition, reports from community members and village leaders clearly show that most communities are committed to continue running Posyandus and determined to have health workers offer services like immunization on a regular basis. As a result, community members, Posyandu cadres and village leaders not only effectively run Posyandus in their communities but also proactively ask for immunization services in their areas. The FE Team attended a meeting of Village and sub Village Leaders the LCSP team presented and facilitated discussion on the findings of the FE KPC survey and plans of action beyond the end of the project. At the end of the meeting, the Village Leaders signed a declaration of commitment to continue support to Posyandus in which the roles and responsibilities that the Village Leaders can assume is made explicit (Annex 20). However, various reports to the FE Team suggest that at least some Village Leaders are not as involved as expected. In addition, in one focus group discussion led by the FE Team, cadre leaders mentioned that the role of the Village Leaders was typically limited or inexistent. Therefore, the declaration made by the village leaders should be followed-up by the DHO, health center staff and the ADP so that the declaration will not only be a document but a real commitment to implementation. Although the ADP plans to continue support to Posyandu in the LCSP area and expand this activity to two other sub-districts (see section d.ii), it will have less staff and resources for these activities than the LCSP had and even maintaining the level of activity at the Posyandu level will be a challenge. With the end of the project, for instance, the motivators’ support to the Posyandus will end. The ADP decided to recruit 3 or 4 facilitators after the project ends but their new responsibilities may not let them continue monitoring and follow up Posyandus as intensively as under the LCSP. This is an important issue to address with the District Health Office and Health Centers staff because from previous experiences in Indonesia, monitoring has always been the key factor to sustain community empowerment activities. Health Center staff will certainly have difficulty in taking over the role of the motivators, especially their intensive and regular on-site monitoring of Posyandus. Whether the new Cadre Associations will be able to take up this important function is also not clear. Local governments should consider taking up the support provided by the LCSP for cadre mobilization and training. World Vision 40 LCSP Final Evaluation Focusing activities on posyandu development was the major strength of the LCSP, and the number of cadres trained is an important asset for its sustainability. Continued collaboration and good communication between LCSP motivators and posyandu cadres was one of the success factors of the project. The FE Team found that the DHO and the Health Centers are all committed to continue support of Posyandus. However, these commitments still need to be supported with realistic plans and budget allocations. b. Communication for Behavior Change The DIP characterizes the LCSP behavior change approach as follows:  Emphasis on household and community behaviors  Selection of behaviors to change and development of specific messages with the health center staff and community members  Provision of information and education to all community members  Creation of an enabling environment that supports sustained behavior change  Use of visual and participatory methods to deliver messages The MTE Team recommended that the LCSP staff specify the behaviors to change and further develop the behavior change strategy and action plan of the project. The need for assistance from WV Indonesia and WVUS was highlighted. By the time of the TAR, three LCSP staff members had participated in international training seminars on the BEHAVE framework and a draft framework on malaria had been drafted. The TAR team recommended technical assistance to further develop this framework and the related action plan and communication materials. In March 2004, the WVUS Technical Officer facilitated a workshop during which the protocol of a doer nondoer study was developed (see section 2.c.iii). In July 2004, the LCSP organized a training workshop on the BEHAVE framework facilitated by an international consultant.35 This workshop was the first time this training was provided in Indonesia and the LCSP team prepared all the training materials in Bahasa Indonesia. This four-day training involved all the staff from the LCSP and other organizations and gave the opportunity to some of the participants to develop a BEHAVE framework on ITN use (see Annex 27). One major activities of the LCSP has been the development, production and wide distribution of various BCC materials. These materials, which are typically available in government health services in limited quantities, have played an important role in the revitalization of Posyandus. Table 7 presents the type and topic and the total number of BCC materials distributed in the project area over the life of the project. 35 See consultant’s report: The BEHAVE training, Pontianak, 26 – 29 July 2004. World Vision 41 LCSP Final Evaluation Table 7 BCC materials distributed by LCSP Type and topic Number distributed Posters on diarrhea, malaria/ITNs, vitamin A, Healthy Pregnancy, Child Nutrition 37,000 Stickers on Immunization and Pneumonia 25,000 MCH Booklets (7000 for other districts) 10,000 Growth Monitoring Cards 5,000 T-shirts for Malaria, Breastfeeding, Vitamin A 2,190 Booklets for Cadre 1,000 Umbrellas on Posyandu, Malaria, Vitamin A, Diarrhea 550 Banners for malaria campaign 220 Booklets for Village Drug Post 200 Large-Size Growth Monitoring Charts 150 Bags on “Healthy Children Healthy Nation” 100 Source: LCSP, September 2004 Some of these materials in the table above were replications of ones in use by the MOH while others were designed and developed by the LCSP team. Posters and stickers on ARI and pneumonia, for instance, have been redesigned following recommendations of the MTE to do so. Although the background research conducted to refine these messages and adapt them to the local context are not documented, these stickers and posters have been widely distributed to mothers attending Posyandus during the ARI Month Campaign (see below). The final KPC shows an increase in one of the targeted behaviors (early care seeking when signs of pneumonia). The LCSP also developed three video CDs: one on malaria and illegal gold mining; one on the clinical signs of pneumonia; and one on nutrition. One of the key functions of the Posyandu cadres is to provide health education to mothers who have come for growth monitoring and immunization of their child. The LCSP trained all Posyandu cadres in the messages to deliver to mothers and taught them communication skills. This knowledge and skills are reinforced by the health motivators and health center staff during their regular meetings with the Posyandu cadres. The LCSP also provided all Posyandus with BCC materials to support these health education activities. To further motivate cadres and improve their knowledge and skills, the LCSP staff also organized competition and poster contests. In 2004, the LCSP worked with Health Center staff and Posyandu cadres to plan and implement two disease-focused mass campaigns. 1. The Malaria Month Campaign was conducted in March 2004 to optimize project achievements especially regarding ITN use. The major activities were: - ITN sale and distribution using discount vouchers. The LCSP issued and distributed 2100 45% vouchers through Posyandu cadres. The LCSP began discussing with KSU Pancur Dangeri (KPD) about their involvement as ITN distributor in January and reach an agreement on the mechanism of ITN promotion late February 2004. The LCSP then developed a Standard Operation Procedure and distributed it to the KPD branches in Senakin, the LCSP motivators and the Posyandu cadres. The socialization of this promotion was made in each Health Center area during the regular Posyandu cadres meetings facilitated by health centers staff. World Vision 42 LCSP Final Evaluation - Redesign and printing of 5000 posters including key messages on danger signs, methods of prevention and treatment of malaria. - Development, production and distribution of 1000 T-shirts, 150 Umbrellas, and 220 banners (placed along roads and buildings) - Malaria-focused health education in all Posyandus, with sessions conducted by Posyandu cadres with the assistance of the LCSP motivators. - Development of an exit interview tool focused on malaria. These exit interviews were conducted during the Posyandu sessions by the cadres and with the assistance of the LCSP motivators. - Demonstration of video CD on Malaria and illegal gold mining 2. The ARI Month Campaign was conducted in May and June 2004. The main purpose of this campaign was to increase the community’s knowledge of danger signs of ARI or pneumonia and to promote immediate care seeking behavior for mothers whose children experience one of those signs. The main activities conducted during this campaign are: - Design of new BCC materials in the form of two stickers, and printing and distribution of 9,000 pieces. Between 60 and 80 stickers were allocated to each Posyandu during the regular cadres meeting in each area. These stickers were distributed by the Posyandu cadres to mothers of children under five attending the Posyandu. In the FE KPC survey, 80% of mothers said they had seen the pneumonia sticker when shown to them by the interviewer and 47% could show it to the interviewer. - Health education sessions in each Posyandu focused on ARI signs recognition and the need for immediate referral. The content of these sessions was based on the key messages of the stickers and posters previously distributed. - Billboard competition between Posyandus in each health centers area. The main idea of this competition was to obtain community participation in the development of the billboard to increase the dissemination of the messages. This activity was introduced during the regular Posyandu cadres meeting in each health center area. These billboards had to include all ARI messages included in the stickers but the illustration could vary. The LCSP provided a wooden board, paints and brushes to each Posyandu. The scoring of the billboards was based on the completeness of the messages, the compatibility between the messages and the illustration and the harmony of the whole billboard. The assessment of the result was conducted in the three Health Center areas with a jury composed of one health staff, one local government staff and two motivators from another area. Three winners were selected in each area. The progress achieved for most LCSP indicators and targets for the five technical interventions demonstrate significant behavior change in the community. Twelve of the fourteen key indicators measured by the KPC surveys show an increase and the end-of-project was achieved for six of them (see Annex 8). The two indicators for which no progress was achieved are related to home based management of diarrhea. The LCSP did not improve feeding practices and the use of medicine for children under two with diarrhea. World Vision 43 LCSP Final Evaluation Motivators have gained good understanding and knowledge of the behavior of the community and of the strengths and weaknesses of the health services regarding their support of the Posyandus. Below are examples of the findings that they shared with the FE Team:  In setting where the cadres are proactive and highly motivated, health staff was able to be convinced to do home visits to children who did not attend Posyandu activities due to illnesses.  Proper explanation on vaccination side effects, especially fever, has proven to be very effective in reducing resistance from the community to bring the children to Posyandus for immunization.  A strategy must be found to prevent the traditional practice of prohibiting post-partum women from getting out of their house including for accessing health services for themselves or their newborn babies. The rapid success of the two mass campaigns above, as measured by the sales of ITNs and the KPC surveys, suggests that the use of only one single media to promote malaria messages, as it was done during the first period of the project, is not sufficient; a multimedia mass campaign, combined with substantial discount vouchers in the case of ITNs, appears to have been more effective in achieving behavior change. The LCSP could probably have achieved better results earlier by developing early and specific BCC strategies and adjusting them based on monitoring of the results. Progress with the use of ITNs and health seeking behavior for pneumonia during the second phase of the project are successful examples of such approaches. At the end of the project, LCSP purchased an additional 24,000 LL-ITNs for distribution at Landak district. The further distribution of these LL-ITNs and a moderate ITN promotion campaign by Koperasi Pancur Dangeri and the ADP will most probably sustain the behavior of using ITNs by the community. The ADP will otherwise continue providing support of the Posyandus in the LCSP area and in other sub district (see section d.ii). Most of the measures of the impact of BCC interventions were done the various KPC surveys. For the promotion of ITNs, records of sales provided rapid feedback information on the demand for this product. The LCSP also conducted one doer nondoer analysis to shows that most owners of a new LL-ITN actually use them for pregnant women and mothers of children under two. c. Capacity Building The DIP outlines the capacity building approach of the LCSP as follows:  Revitalize Posyandus and their cadres and build capacity of informal health partners to deliver CS interventions (traditional healers, shopkeepers, traditional birth attendants)  Train Self-Help Groups to deliver CS health education  Assist health center in supervision and use of data for targeting and decision-making.  Train all staff in health center, sub center, and village post staff for CS interventions  Meet regularly with District Health Office and ADP for planning and priority setting.  Meet regularly with Provincial Health Office for project updates and for provision of facilitators for training events. World Vision 44 LCSP Final Evaluation  Build capacity of local NGO, including preparation and use of activity plans and operations in accordance with established MOU.  Build capacity of ADP staff (in the LCSP area and other areas in Indonesia) in CS interventions and related skills such as behavior change communication, quality improvement, data for decision-making, credit with education, etc.  Build capacity of WV Indonesia health staff through technical assistance focused on skills gap and innovations, participation to international training events and exchange visits and project’s growth and performance monitoring. The LCSP results with respect to revitalization of Posyandus are presented and discussed in section a. In this section, the other LCSP results with respect to capacity building are presented successively for World Vision including the ADP; for the partner organizations, including the Landak District and West Kalimantan Health Offices, the Pancur Kasih Foundation, and the Posyandu Cadres Associations; and for the health facilities. The section ends with a presentation and discussion of the training strategy and achievements. i. World Vision During the LCSP, WV Indonesia staff benefited from various national and international training opportunities and from developing experience and commitment to work in child survival and for the poorest communities. The LCSP was also an opportunity to build strong networks with governmental and non-governmental organizations involved in child survival in the project area and at the national level. This constitutes a very valuable investment for WV Indonesia even though the turn over of the staff has been quite high, as usually is the case for such project. WV Indonesia also learned how to integrate child survival activities into its ADPs. In the course of the LCSP, WV Indonesia made the ADP manager in charge of the project to ensure that all activities and major decisions were consistent with those of the ADP. In addition, the ADP staff participated in many of the LCSP activities (such as project reviews, surveys, and other workshops) and gained the experience necessary to conduct at least some of them. At the end of the LCSP, ADP will continue support for Posyandu cadres through advocacy and community activities; ITN promotion and distribution through a partnership with Pancur Dangeri; and ongoing water and sanitation activities. The ADP also decided to recruit three LCSP motivators as ADP motivators. However, the specific activities that ADPs can conduct still need to be clarified: How can support to Posyandus best be included in the ADP? What are the activities that ADP staff without health background can implement? What HIS can be picked up by the ADP? This integration of child survival activities in ADPs remains an important issue for WV Indonesia, which attempts to include health and child survival activities in all its ADPs. This constitutes a promising strategy to scale up and ensure sustainability of the achievements of the LCSP. Based on results achieved in Landak, WV Indonesia plans to introduce similar interventions and activities in some or all its 28 ADPs in the most needy communities in Indonesia. WV Indonesia could achieve through the following strategies:  Provide technical and other support to the ADP to continue LCSP activities at a sustainable level and become a learning site visited by staff from other ADP’s.  Develop training, BCC, HIS and other implementation materials based on the experience of World Vision 45 LCSP Final Evaluation the LCSP and make them available to other ADPs ready to implement child survival activities.  Conduct training and provide technical assistance for the introduction of CS interventions in other ADPs. ii. Partners Organizations Landak District and West Kalimantan Health Offices The various activities of the LCSP to strengthen Health Facilities are discussed in the next section. The Organizational Capacity Assessments done during the Sustainability Action Plan workshop in March 2004 (see section 2.c.ivshowed that the three Health Centers have in general very similar scores, with the main weaknesses in the areas of Vision and Mission Statements, Networking and Fundraising. The LCSP regularly involved the DHO in overall reviews and planning of project activities. The DIP did not include any other specific capacity building activities and there has not been any Organizational Capacity Assessment of the DHO. Pancur Kasih Foundation The DIP designated the Pancur Kasih Foundation as the main local NGO partner of the LCSP but the nature and capacity building strategy of the partnership were not fully defined beyond a MOU. There has not been any Organizational Capacity Assessment of the Pancur Kasih Foundation. The partnership self-assessment done during the DIP workshop was not repeated. Building on past relationships, some ADP staff members formerly employed by the Pancur Kasih Foundation have continued collaboration with this organization but not on LCSP-related activities. One of the roles of the Pancur Kasih Foundation in the ADP area is to provide loans to SHG members for Micro Enterprise Development (see section d.iii). The ADP facilitates this process by networking with the local loan branches and by preparing SHG members for professional activities or businesses. The KPK Human Resource department, which is in charge of training the cooperative members, plans to provide health education to its members but up to now has lacked funding. They hope to find a suitable trainer with health education background willing to cooperate with them for a low fee (see Annex 19). Starting in January 2004, the LCSP began collaboration with Koperasi Pancur Dangeri, which is a daughter company of Pancur Kasih Foundation, to develop a distribution network for ITNs. During the March 2004 Malaria Month Campaign, the LCSP and KPD issued discount vouchers for pregnant women and mothers of children under five and involved Posyandu cadres as last line distributors. Two KPD members joined the BCC training in July 2004. The FE Team interviewed the Senakin KPD manager who demonstrated good understanding of the ITN program and its potential (see Annex 19). At the time of the FE, the ADP had made plans to continue collaborating with KPD on ITN distribution. KPD was not included in the Organizational Capacity Assessments done during the Sustainability Action Plan workshop in March 2004. Posyandu Cadres Associations In 2003, the Posyandu cadres took the initiative to establish a Cadres Association in each Health Center area. The objectives of these associations are to facilitate networking among cadres, World Vision 46 LCSP Final Evaluation provide opportunities to discuss their roles and problems, and to motivate and improve their knowledge and skills. The establishment of these associations was very timely because they maybe used as a vehicle to sustain Posyandu activities after the end of the project. The Organizational Capacity Assessments done during the Sustainability Action Plan workshop in March 2004 (see section 2.c.ivshowed that the three Cadres Associations had low scores for their management structure and communication capacity but a very strong score for leadership and ownership. This corresponds well to new organizations created at the initiative of their members. The three Cadres Association otherwise scored lower than the three Health Centers. In May 2004, the LCSP trained 20 members of Cadres Associations in organization development and management. iii. Health Facilities Strengthening The DIP did not include activities to directly support management and care in the health center in project area. Although the LCSP staff regularly met with Health Center staff to discuss various project activities and issues, they did not specifically “assist in supervision and use of data for targeting and decision-making,” as proposed in the DIP. At its inception, the LCSP provided the three health centers with cold chain equipment and a small generator as standby in case of electricity failure. The FE Team checked the status and related procedures used for this equipment and found that overall, the cold chain was functioning well (see section 2.b.iand Annex 11). Each health center has also been provided motorbikes (2 for Mandor, 1 for Senakin and 2 for Pahauman) to facilitate transportation of health center staff to the Posyandus. After delays in deciding the number to provide to health centers, the DHO only requested one additional microscope to the LCSP to complement 13 microscopes that it in its own budget. The LCSP gave this microscope to the DHO in June 2004. At least two health workers in each HC were trained in microscopic examination, most of this training being offered as part of the chloroquine efficacy study as therefore with a focus on malaria diagnosis. There now are microscopes in all of the three Health Centers. Microscopy diagnosis of malaria is made on a regular basis but these exams remain expensive for most clients. The LCSP provided various training opportunities to health workers. These are presented and discussed in section v. During interviews conducted by the FE Team, Health Center’s staff recognized that the support provided through the project, including training, was valuable for their job. However, several of them doubted that the DHO would be able to provide further training at the end of the LCSP. The DIP mentioned one health facility assessment made as baseline but this has not been repeated later on. The LCSP otherwise introduced various tools to monitor the quality of health services in the Posyandus, which have helped identify problems in individual Posyandus (see section 2.c.iii). These tools have not been used systematically enough to measure change over the life of the project. The Organizational Capacity Assessment conducted prior to the Sustainability Action Plan workshop in March 2004 is described in the previous section and may help to measure change if used every one or two years. World Vision 47 LCSP Final Evaluation One of the main strategies and achievements of the LCSP has been to strengthen the links between the Health Centers and the Posyandus through facilitating communication between the community and the health centers, providing support for transportation of the health worker to the Posyandu, and ensuring that the Health Centers fulfill their commitment with respect to Posyandus. The Posyandu Monthly Reports for Year 2003 show that on average, there was a health worker providing immunization services in 87% of the Posyandu days in the project area (see section 2.c.iii). The LCSP team reports various instances where Village Leaders, Posyandu cadres and community members have contacted health officials to request regular presence of health workers in Posyandus. Clearly, the LCSP’s support to Posyandus, including the facilitation role to involve health workers during Posyandu Days, indirectly strengthened Health Centers’ capacity to increase the coverage of the services they provide. Both HC staff and LCSP motivators have been supervising Posyandus cadres over the life of the project. At the Health Center level, the FE Team established that in principle, there is one designated HC staff member per village (each staff member in charge of several Posyandus), one Posyandu coordinator in each HC (typically the nutritionist) and one Posyandu coordinator at the DHO level. The HC staff in charge of a Posyandu is also responsible for the supervision of the cadres but typically does not do much more during his or her visit besides providing health services, primarily immunization. Plans for joint HC/LCSP supervision of Posyandus have been discussed at the beginning of the project but have not been finalized and implemented. Although the three Health Centers staff and the Posyandus would probably have benefited from such activities, the institutional structure and design of the project were not conducive to the direct involvement of the LCSP staff in their activities. Joint DHO/HC supervision of health centers’ activities in Posyandus probably is one activity that the DHO could adopt to improve the quality of health services at the community level. After the end of the LCSP, the ADP plans to continue supporting Posyandu cadres through their other activities at the community level but not through support of the Health Centers. However, WV Indonesia and the ADP might be able to encourage the Landak DHO to conduct joint DHO/HC supervision of the health services they provide in Posyandus. iv. Strengthening Health Worker Performance The LCSP primarily strengthened Posyandu cadres and other community members as health workers, which is discussed here. Training provided to Government health workers is discussed in the section on training. Training followed with regular motivation and support have been the main approaches to strengthen Posyandu cadres performance, and to smaller extent, that of other community health workers. This approach turns out to be very effective, as demonstrated by the large increase in Posyandu activities including health services and of the coverage related child survival interventions. The group discussions led by the FE Team suggest that, overall, the Posyandus in the LCSP area are run according to the national recommendations of the MOH Manual for Posyandu Cadres (see Annex 14). The analysis of the Posyandu Monthly Reports for calendar year 2003 conducted by LCSP shows that a health worker is present on 80% of Posyandu days and that are on average close to five Posyandu cadres are present (2.c.iii). Although these figures are close to World Vision 48 LCSP Final Evaluation the MOH norms (see section 2.c.iii), a higher attendance of health worker and cadres would most likely result in higher results such as the percentage of Fully Immunized Children by age one. The MTE Team recommended that the LCSP conduct regular assessment of the performance of trained community health workers using explicit standard of performance. Beside the various tools described in section 2.c.iii, which assess the overall performance of Posyandus, there has not been any attempt to measure the performance of the Posyandu cadres at the individual level. The FE Team conducted such an assessment using implicit standards defined from the content of the training materials and a Lot Quality Assurance Sampling methodology. Taking advantage of a quarterly meeting of Posyandu cadres, the FE Team assessed the knowledge and skills of 19 Posyandu cadres chosen randomly in each Health Center area. Table 8 presents selected results of this assessment. Table 8 Selected results of the FE Posyandu Cadres Assessment INDICATOR SENAKIN MANDOR PAHAUMAN <50% 50-80% >80% <50% 50-80% >80% <50% 50-80% >80% Knowledge Purpose of GMC x x x Danger signs of diarrhea x x x Danger signs of ARI x x x Causes of malaria x x x Type of immunization x x x Skill (Health Education on ARI) Rapid Breathing x x x Stridor x x x Chest in-drawing x x x Should not be fasted x x x Immediate referral x x x Skill (GMC recording) Identification x x x Immunization x x x Vitamin A x x x Weighing x x x Knowledge was tested through a simple questionnaire that each selected cadre was asked to fill. More than 80% of the Posyandu cadres were found to have a good knowledge of each of the four project components, except for the “Causes of malaria” in Senakin and Mandor and the “Danger signs of ARI” in Pahauman. This is very encouraging and is one of the successes of the LCSP. To test their health education skills, each cadre was asked to simulate a health education session on ARI. During these role-plays, more than 80% of cadres in each Health Center area spontaneously mentioned rapid breathing as a sign of pneumonia and the importance of immediate referral. Mention of other signs is not as satisfactory except in Pahauman where more than 80% of the cadres mention all items in the table except stridor. Less than 50% of Posyandu cadres are able to correctly record clinical data on the Growth Monitoring Card. Although recording accurate data on the Growth Monitoring Card is an important aspect of the posyandu program, the LCSP probably did not place enough emphasis on this activity during the training of the Posyandu cadres and their supervision. Other results and comments on the limitations of this study conducted for the first time with the LCSP Core Team are presented in Annex 16. World Vision 49 LCSP Final Evaluation Overall, the various tools to assess performance of Posyandus described in section 2.c.iii and the rapid assessment of individual Posyandu cadres’ performance described above do not provide direct measures of the improvement of health worker performance over the life of the project.36 Such measures require explicit definitions of performance standards and objectives, the development of tools including measures of these standards, and the collection of the related data before the performance improvement intervention. Then comparable data need to be collected one or several times during the implementation of the intervention. Ideally, the sensitivity of these measures should be evaluated before attempting to assess health worker performance improvement to avoid inconclusive results. Clearly, there is a need for guidelines on how best measure community health worker performance improvement but the development of the methodology itself is probably beyond the scope of one single child survival project like the LCSP.37 The LCSP used the performance assessment tools discussed in section 2.c.iii in various ways. Probably the most common situation has been a direct response by the motivators or health workers collecting the data who would then try to correct the problem on their own as part of their role of supporting Posyandus. Some of the data collected have also been entered and analyzed at the LCSP office. Typically, some summary statistics would be calculated for a group a Posyandus. Although not always done systematically, conclusions and recommendations made from these analyses would then lead to decisions such as providing refresher training, discussing a topic during the regular Posyandu cadres meeting, or providing specific support to certain Posyandu. v. Training The training strategy outlined in the DIP was to conduct training for all levels: province, district, sub-district, community, household and the LCSP and WV Indonesia staff. The DIP provides a long list of specific topics for each child survival interventions of the project but did not specify any training objectives in terms of types and number of trainees, knowledge or skills to acquire or change, or impact on trainees’ or services performance. The LCSP did not conduct a training needs assessment or develop a training plan that specified such training program. Annex 28 presents the chronologic listing of all training events organized during the LCSP project with the topic of training and number and type of participants. Annex 29 summarizes this data to provide the number of events, total participants and the number of participants by fiscal year by type of training. This section discusses the approach and achievement of the LCSP training for community members, health workers and then for other categories of participants. Community health workers Figure 8 presents distribution of the 1345 Posyandu cadres, shopkeepers, TBAs and POD cadres trained each fiscal year of the LCSP. The coverage of the project area and population achieved by this training program is discussed in section a. Most of the training activities for community 36 Indirect measures such as the increased activities and attendance of Posyandus and the increased in the coverage of related activities are discussed in other sections of the report. 37 This challenging question of the sensitivity of health worker performance improvement measures is proposed in the USAID Final Evaluation guideline: “Were the tools used to assess the results of improving health worker performance sensitive enough to measure change over the life of the program?” World Vision 50 LCSP Final Evaluation health workers took place in FY 2002 but this pattern does not correspond to any explicit training strategy. Figure 8 Posyandu cadres, shopkeepers and TBAs trained by Fiscal Year 0 100 200 300 400 Posyandu Cadre 229 295 87 92 Shopkeeper 11 195 72 42 TBA 185 90 10 POD Cadres 37 2001 2002 2003 2004 Source: See Annex 29. The LCSP team, jointly with Health Centers’ staff, trained a total number of 703 community members as Posyandu cadres. Most of this training was conducted in the first and second year of the project but continued during the third and fourth years as new Posyandus were created and new community members volunteered to become Posyandu cadres. The LCSP also organized refresher training events for 99 Posyandu cadres in Mandor in 2002 and regular meetings in each Health Center area that were attended by a total of 925 and 702 Posyandu Cadres in 2003 and 2004, respectively. After the Sustainability Action Plan workshop (see section d.ii), the LCSP began training Posyandu cadres in Toho and Takong, two sub district covered by the ADP but not by the LCSP so far (214 cadres attended one-day meetings between May and August 2004). The Posyandu cadre training consisted of a three-day curriculum, except for the final year of the LCSP where it was shorten to two days (see training curriculum in Annex 30). LCSP interventions were explained at the beginning of the training. Group discussions also include simulation of health education sessions. After this initial training, the LCSP provided some refresher training during quarterly Posyandu cadres meetings. During the last year of the project, for instance, the LCSP gave refresher training on immunization, malaria and ISPA, facilitated discussion between cadres and Health Center staff and conducted role-plays on specific themes. The LCSP trained 320 shopkeepers through 22 training workshops in the three Health Centers areas. The objective of this training was to expose shopkeepers to common diseases (signs and symptoms), encourage them not to sell harmful drugs and provide their clients with information on common diseases and their treatment. TBAs were trained by the health center midwives in clean delivery techniques using the kits provided to them during the training. At midterm, the LCSP and HC staff had trained 185 TBAs, most of these in Mandor and Pahauman. At the time of the FE, the LCSP had trained 285 TBAs. These trained TBAs assist about half of the deliveries in the project area and on average, assist between 3 and 5 deliveries per year. The FE Team interviewed two TBAs who both knew the information received during their training and had good collaborative relationships with the World Vision 51 LCSP Final Evaluation midwife in their area. In addition to this initial training, the LCSP organized a series of refresher training and supported the monthly meetings that each health center should organize according to a MOH policy. A total of 37 community members were trained as POD cadres in 2002. The training consisted in recognition of signs and symptoms of common diseases and how to treat them and in the management of selected drugs (see training curriculum in Annex 31). Two or three persons were trained in each shop selected as POD to ensure the continued presence of a trained person. In 2003 and 2004, selected POD cadres received refresher training at the same time as villages and sub villages leaders. The expected functions and results of the creation of POD is further discussed in section d.iii Other community members In addition to Posyandu and POD cadres, shopkeepers and TBAs, the LCSP trained various community members. In 2003, 45 village leaders and cultural stakeholders were trained to provide support to Posyandus and PODs and to other project related activities. In July 2004, 222 mothers of underweight children and 49 Posyandu cadres in Senakin were trained in proper feeding practices and nutrition. Senakin was chosen for this activity because of highest malnutrition rates in this area. This training was related to a nutritional supplementation program that the ADP is implementing. Health workers The LCSP provided various training opportunities to the staff of the Health Centers in the project area. In FY 2001, 43 government health staff, primarily from the project area, participated in a one-day training on malaria risk and control. The LCSP supported the training in IMCI for 25 and 13 Health Center staff in FY2002 in FY2004, respectively, most of them coming from the project area. In FY2003, the LCSP also support 20 participants, among which 8 came from the Landak District, for a seven-day Training of Trainers in IMCI workshop. In FY2003, the LCSP also supported 24 Health Center staff in a Training of Trainers workshop on participatory learning organized by the Family Planning Committee. Also in FY2003 and prior to the Chloroquine Efficacy study, about a dozen of Health Center laboratory technicians and nurses received a 12-day training in malaria microscopic examination that included field practice in collecting specimens and on-site preparation and examination of slides (this training was repeated because of delay in the beginning of the Chloroquine Efficacy study). All the training events and participants above are relevant to the LCSP but do not specifically cover the key areas of child survival and related approaches such as behavior change that were proposed in the DIP. There has not been any assessment of the training need of the health workers related to the specific child survival and project activities and no training plans to meet those needs. However, the LCSP training has improved the knowledge and skills of health workers, which would probably not have been possible using DHO resources. In addition, the HC staff provided most of the training of community members and this has increased their commitment to the project and its activities. However, the LCSP had no mechanism in place to assess the training skills and performance of the HC staff. The FE Team did not attempt to assess the performance and skills of Health Center staff. World Vision 52 LCSP Final Evaluation After the end of the LCSP, the ADP will continue supporting training of Posyandu cadres but does not have any plan to support training for health workers. The ADP has the capacity to organize the training of Posyandu cadres using health center staff just as did the LCSP but may need technical assistance to conduct Posyandu cadres training needs assessment and adapt the curriculum as needed. Involving the government health services (Health Centers, DHO and PHO) in the preparation for and cost sharing of training health workers has proven to be very effective as it decreased bureaucracy and increased the sense of ownership of the stakeholders. LCSP has conducted at least two trainings using this approach (ToT on IMCI and regular IMCI training afterward). d. Sustainability i. Achieving Sustainability Goals and Objectives The Results Summary Chart in section 1 presents the achievements of the key sustainability objectives set forth in the LCSP DIP. Intermediate Result 3.1 is achieved although the LCSP staff had more regular interaction with the Landak DHO than with the West Kalimantan PHO. The meeting of the FE Team with the West Kalimantan Governor and Provincial Health Officer clearly showed that World Vision Indonesia, the ADP and the LCSP are well known and appreciated for their contribution to the health and development efforts in the Province. The newly appointed PHO is looking forward to “success stories” and specific recommendations from WV Indonesia that can be included in various projects and programs for which funds are available at the Provincial and District levels. At the District level, the officers who have been most closely involved with the LCSP consistently praise its achievements and contributions. Although regretting the end of the project, the DHO staff showed commitment to continue the activities and maintain or improve the results of the LCSP. Intermediate Results 3.1 and 3.2 are also achieved in so far as the increase in coverage of all the LCSP interventions is largely due to the contribution of the health worker and cadres in the Posyandus. Most village leaders, although not yet all, are aware of the importance of supporting Posyandus and cadres and expressed their willingness to do so (see Commitment of Village Chiefs in Annex 20). Provided with limited support and motivation, Posyandu cadres not only show the ability to continue their activities beyond the LCSP but also have clearly expressed their willingness to do so by organizing themselves in Cadres Associations and in several instances by requesting for or claiming their rights to health services in their village during Posyandu days. Numerous other community members are now aware of the benefits of the LCSP child survival interventions, can apply this knowledge and use the related services. POD cadres, SHG members and TBAs can contribute in providing the related services. Three sustainability objectives included in the DIP are not achieved. The LCSP strengthened existing Posyandus HIS (registers and reporting forms) supported by MOH but did not introduce the new tools and systems initially proposed (see section 2.c.iii). In addition, the LCSP did not develop as many SHGs as expected that would engage in child health and survival activities (see section iii). Most ADP staff acquired substantial CS-related knowledge and skills during the LCSP and several members of the local LCSP staff will keep a position within the ADP; this staff will be able to continue the support still needed by the health centers and Posyandus. World Vision 53 LCSP Final Evaluation However, the ADP has not met the objective of allocating 20% of its FY 2005 budget to health and child survival. ii. Developing a Sustainability Plan The DIP only presents the LCSP sustainability plan in terms of indicators to track progress towards sustainability (see Results Summary Chart in section 1). At midterm, one year before the original end of the project, the MTE Team recommended to accelerate the development of a phase-out strategy. At the time of the TAR, the LCSP and ADP had begun developing plans to transfer LCSP activities to the ADP and with the approval of a one-year no-cost extension, requested the assistance of WVUS to further define a sustainability action plan for the end of the project and beyond. In March 2004, the LCSP conducted a Sustainability Action Plan (SAP) workshop facilitated by the WVUS Technical Officer. The workshop was designed according to the Child Survival Sustainability Assessment framework and methodology and was meant to be the first of a series of workshops at the district level focusing on motivation and ownership of stakeholders at each level. This first workshop done at the ADP level was attended by 9 participants from both LCSP Core team and motivators and the 11 participants from the ADP. The workshop began with the identification of parties and people with the most influence on child survival: everybody agreed that families, the community, Posyandu cadres, ADP and the local health institutions have the most important roles, followed by the government. The next activity was the development of a common vision of the sustainability of each dimension of the framework. The timeframe used was the lifetime of ADP up to year 2009. After developing the vision, the participants conducted a SWOT analysis and developed indicators for each dimension of the framework. They also developed the Plan of Action (POA) for the end of the project. Annex 32 presents the main outputs of the workshop: the vision, SWOT analysis results, indicators and a brief POA for April to September 2004. The POA, which corresponds to a Phase-over Plan, lists several activities from the LCSP Work Plan that were completed at the time of the Final Evaluation. Among others already completed activities are the training of Posyandu cadres in two sub-districts not included in the LCSP area but part of the ADP area. Training SHG members in MED and health is listed but not completed, and so are the Sustainability Action Plan workshops at the District and sub District levels. Building on the results of the SAP workshop, the ADP developed a “Sustainability Strategy for Child Survival in ADP 2005 to 2009” around the four elements below:  Continue to improve local capacity particularly that of cadres and mothers;  Replicate the lessons learned of LCSP to other ADP’s areas using Sengah Temila and Mandor as a model;  Improve health education through local institutions such as Aleat’n,38 SHGs,39 Posyandu Cadres Associations, Koperasi Pancur Dangeri and the Pancur Kasih Foundation;  MED integrated with health education with a focus on Women’s Groups. 38 Aleat’n is a simple community-based organization, originally from Dayak rice field farmers, with which the ADP has been working but not the LCSP. 39 There currently are 109 SHGs in the ADP. World Vision 54 LCSP Final Evaluation Overal, the ADP 2005-2009 strategy seems adequate but need specification of the LCSP activities to continue or expand. While the three first elements of the strategy can build on the experience and achievements of the LCSP, the fourth one still needs explicit formulation since very little was done with this respect during the LCSP. During the Technical Overview presented by the LCSP and ADP staff to the FE Team, the ADP Community Development Coordinator presented the FY 2005 Health Program of the ADP. The FE Team found that this plan is probably not specific enough to best build on achievements of the LCSP at the end of the project. For instance the target group 0-18 year old for diarrhea control covers age groups with very different needs that those of infants and children under five. Activities such as animal fencing and raising awareness about the danger of mercury are certainly valuable from a general development perspective but do not specifically address the main causes of high morbidity and mortality among children under five. Building on the success of the Malaria Month Campaign in raising community awareness of the importance of preventing malaria and in creating demand for ITNs (see section 2.c.i), the LCSP ordered 24,000 ITNs that will be distributed by the ADP in partnership with Pancur Dangeri after the end of the project (see section iii). This program will cover the four sub-districts of the ADP: Mandor, Sengah Temila, Toho and Sungai Pinyuh (16,941 families). At the time of the FE, the ADP had prepared a distribution plan that includes renting a storage building, the continued involvement of the Pancur Dangeri, SHGs and Posyandu cadres, and the use of some of the proceedings of the sales to support micro enterprise development programming (see Annex 33). iii. Building Financial Sustainability The LCSP attempted to build financial sustainability of its results and approaches through the following mechanisms. Micro-Enterprise Development The DIP proposed integrating health in micro credit and Micro-Enterprise Development program through the ADP-sponsored Self Help Groups as a key strategy of the LCSP. The DIP specified that this intervention would be implemented by its designated partner the Pancur Kasih Foundation given their experience in creating and supporting SHGs, but no formal partnership between the ADP and Pancur Kasih was established for this or other activity (see c.ii). The DIP included the budget for a position for a MED Officer starting in FY 2001 but that position was never filled. The ADP currently supports 109 SHGs, which are considered as the pillars of its community mobilization and development strategy. To obtain support from the ADP or credit from Pancur Kasih, SHGs must satisfy certain pre-conditions such as having received training in small business, simple bookkeeping and in some potentially profit making activities (farming, pig breeding, fish pond). Once the SHG receive a loan or other in-kind (rubber-seed) support, the SHG must save part of its profit for its own emergency needs and to support Pancur Kasih. Two types of activities were considered for LCSP MED intervention. The first one is to provide health and child survival education to members of micro credit groups (“credit with education”). At the time of the DIP, WV Indonesia had some experience with such program in urban Jakarta and as a result from collaboration with Freedom from Hunger. This type of activities has not World Vision 55 LCSP Final Evaluation begun during the LCSP. The second activity envisioned under the LCSP is to promote business related to health among SHGs (“MED and Health”). One example of businesses that the LCSP started promoting is the sale of ITNs, but no SHG has been very successful with this business so far. The ADP has otherwise begun MED and Health in five women SHGs including two in the LCSP area. In Melati (Senga Temila), a SHG of 25 women has begun chicken breeding with a seed capital of Rp. 9,000,000 and now has a total asset of Rp. 16,119,600. In Singkut Durian (Senakin), a SHG of 11 women has begun Arisan40 activities with a seed capital of 7,000,000 and now has a total asset of Rp. 3,000,000. The FE Team visited one SHG sponsored by the APD/P to develop a fishpond as additional source of income (see Annex 18). The head of this SHG is also the head of the Posyandu in that village and the link between SHG and Posyandu is expected to make Posyandu activities more sustainable. No fish production activity has started and the SHG has not provided health education for its members yet. The Results Summary Chart include one key indicator and objective related to this intervention (see section 1) in terms of “SHGs spending profits on items/activities that contribute to improved household health and child survival” and “SHGs engaged in the promotion of CS messages at the village level.” None of the two components of this objective is achieved. Village Drug Posts (PODs) In 2002, the LCSP trained 37 community members to create 14 Village Drug Posts (Pos Obat Desa--POD) in the Mandor and Senakin Health Centers areas (see previous mentions of this activity in sections a and c.v). The purpose of PODs is to provide shops in most remote villages with selected drugs. At the end of the training, each POD received a stock of drugs and a special cupboard to store them. A special arrangement was made between POD and health centers whereby health centers would supply the drugs and the POD would sell those drugs in the community. In 2003 and 2004, the LCSP facilitated several meetings between the selected POD cadres, local community leaders and staff of the health centers of Mandor and Senakin. In these meetings, an agreement was reached that the health centers would provide assistance though supervision and included. The FE Team interviewed the Health Center staff about their involvement with PODs (see Annex 11). In Senakin, there are two functioning PODs. Both are located near a Posyandu and the health staff provides routine supervision when they visit those Posyandus. Other informal contacts are reportedly common. There does not seem to be a POD drug request form as such or, if one exists, it is not in regular use by those who interviewed. HC staff expressed concern that POD cadres often succumb to pressure from customers to sell less than a full course of medicine (chloroquine, in particular) but felt that the situation could be improved over time with continued supervision and health education. In Mandor, the health staff reported ten established PODs but the HC In-Charge stated that PODs do not report to the HC nor have any of them submitted requests for replenishment of the initial drug stock. No supervision of the PODs has been provided. The In-Charge stated that the PODs were under the ADP and therefore not under his responsibility. There is no POD in Pahauman. 40 Arisan usually are ladies group activities with the aim to help each other financially. For instance, if 10 women form a group, each one will contribute some amount of money on a monthly or weekly basis and the entire amount will be given to a small number of group members selected randomly. World Vision 56 LCSP Final Evaluation The FE Team also visited two PODs recognized by the LCSP team as functioning well, one in Mandor and the other one in Senakin (see Annex 17). Both POD cadres interviewed seemed to have a clear sense of their roles and functions. Both knew the use of the drugs available in their PODs and said that they always provide explanation to costumers to whom they sold them. Both knew where to procure drugs but had never gone to the health center so far since they still had drugs in stock from the initial supply. Both cadres knew all three danger signs of ARI in a young child and the correct drug and dosage for treatment of malaria for adults and for children. They both knew the correct instructions for managing a child with diarrhea. The registration of clients in both PODs was good, including the names and symptoms of most costumers, the date of visit, price and quantity of drugs given. The drugs dispensed are consistent to the symptoms reported. Records appeared complete and carefully entered in both PODs. The stocks of drugs were well organized in both PODs. Small posters with dosage schedules were posted on the wall next to the cadre’s desk. Among the POD drugs, paracetamol and ORS were most commonly sold. No expired drug was found. The most popular drugs are Paracetamol, Chloroquine, Antacid, cough syrup and Oralit. Overall, the FE Team found that only about 5 out of the 14 PODs created in 2002 are functioning satisfactorily. The other PODs are not functioning well for various reasons such as a lack of appropriate knowledge of the cadres or poor financial records. Possible constraints in their viability include that the drugs that they sell are available almost everywhere (Health Centers, Village Birthing Posts), that people prefer buying them from health staff and that the POD’s price (decided by the communities involved) is close to the market price. Although the number of villagers using them is limited, support of the functioning PODs from the Health Centers is warranted. Establishing PODs is otherwise not a government policy and further assessment of their usefulness is needed. Cost of health services in Posyandus The cost of providing health services in Posyandus is commonly considered as a constraint to this outreach strategy. Table 9 summarizes the discussions and data collected by the FE Team on the direct costs of providing health services in Posyandus from the community and the Health Centers’ perspectives. At the community level (see Annex 15), the cost items include the supplementary food (supposedly available), the snacks of the HC staff and cadres, the registration books and the transport of the cadres. The total annual cost is about 1 million rupiyah. The opportunity costs for the cadres are ignored in this cost accounting because Posyandu cadres typically do this work on a voluntary basis.41 For the Health Centers (See Annex 11), the extra cost of providing health services in Posyandu primarily consist in those associated with the transport of health workers to and from the villages. The other costs such as the salary of the health workers, their registration books and the vaccines that they may provide are assumed available from the MOH upon request and therefore are not counted as costs to the Health Center. The total annual cost for the health center is about 100,000 rupiyah, about ten times less that those born by the community. Given the gain in coverage of immunization resulting from providing these services in Posyandus, ensuring that health workers can visit Posyandus on a regular basis seems a very cost effective strategy, particularly if the financial and timesaving to the mothers is also taken considered. 41 It has otherwise been estimated that the opportunity cost for a cadre is about 10,000 for up to 2 days of work per Posyandy day. World Vision 57 LCSP Final Evaluation Table 9 Community and Health Center Annual Costs of Providing Health Services in Posyandus Supplementary food: Vaccines. Drugs, others: ? Rp 30,000 * 10 Posyandu Days 300,000 Snacks for HC staff and cadres: Daily salary/Per diem: ? Rp 20,000 * 10 Posyandu Days 200,000 Registration books: Registration books: ? Rp 90,000 per year 90,000 Transport for cadres: Transport for health worker: Rp 7,500 * 5 cadres * 10 Posyandu Days 375000 Rp 10,000 * 8 Posyandu visits 80,000 Others? Others? Total annual community cost 965,000 Total annual health center cost 80,000 Community Health Center Note: Posyandus are typically run 10 times a year by a team of 5 cadres and are visited 8 times a year by a teams of 2 health workers using a motorbike. Source: FE Field visits (see Annex 11 and Annex 15) Continuation of selected LCSP activities by the ADP In FY04, the ADP spent USD10.765 out of USD359,814 (3%) on health. This low percentage reflect the fact that the ADP had only begun providing health assistance in one sub district while the others were covered by the LCSP. This amount does not include funds spent on clean water and sanitation activities. For FY05, the ADP budgeted US$19,995 for child survival and health, that is, about 10% of the total budget (1 US$ = Rp. 9000). This percentage is short of the 20% target set as an end-of-project objective for sustainability of the LCSP. In 2003, the LCSP engaged in a partnership with Koperasi Pancur Dangeri, a local institution affiliate of the originally proposed LCSP partner Pancur Kasih Foundation. Pancur Dangeri was to distribute ITNs through its network of cooperatives and other mechanisms including the use of SHGs, Posyandu cadres and PODs as last line distributors (see section 2.c.i). Given the good sales and increased in ITN use for children under two achieved during the Malaria Mass Campaign in March 2004, the ADP decided to continue this collaboration with Pancur Dangeri beyond the end of the LCSP as a promising way to build entrepreneurship and contribute to malaria control in the ADP area. The LCSP purchased 24,000 ITNs for this program at the end of the project (see section ii). 4. Program Management a. Planning The LCSP regularly involved its partners in its annual planning cycle. Starting with the baseline assessments and the DIP and Sustainability workshop in October 2000, WV Indonesia involved various representatives from the national, provincial and district level MOH in the annual project review and planning activities, each one facilitated by an external consultant. Findings of these reviews were systematically presented to various audiences at the district, provincial and national levels. World Vision 58 LCSP Final Evaluation During implementation of the annual work plans, the LCSP staff met with PHO officials as needed and held regular ad-hoc meetings with DHO officials for project activities including training. Health workers in each Health Center staff were directly involved in planning the project-supported activities that they were implementing such as training and provision of services in Posyandus. The LCSP staff has produced regular and complete progress reports including process monitoring data on training, health education sessions, equipment and BCC materials distributed, and a discussion of problems and issues. These reports are themselves based on monthly reports from each motivator. The DIP, overall, served its purpose although important changes at the beginning of the project such the inclusion of the ARI control as a new intervention were not explicitly incorporated to guide project managers and staff with their implementation. No specific objectives, indicators or even strategy for capacity building (including training), sustainability and MED were fully developed and even further specified later in the life of the project. The budget in the DIP was not always appropriate and this for some activities such as those mentioned in section d that remained unspent. Some plans to distribute capital assets for project partners were not carried out, for instance. The plan to distribute bicycles for the best performing Posyandu Cadres was not carried out because this was considered prone to inducing conflicts in the community. In addition, a quite large budget for provision of loans and the salary of a MED officer may have remained unspent because the plans for these activities were unrealistic or incomplete. b. Staff Training The training that the LCSP provided to MOH health workers and community members is discussed in section 3.c.v The LCSP also provided numerous training opportunities to WV Indonesia staff. Annex 33 provides the list of all international and local training events and the respective WV Indonesia trainees between October 2000 and September 2004. Annex 35 provides a summary graph of this data in terms of number of events each WV Indonesia staff attended and a summary table of the topics of these events with the number of WV Indonesia staff that attended them. Figure 9 shows that most of this training occurred at the beginning and at the end of the project. This patterns and the distribution of the training topics (table 2 in Annex 35) does not correspond to a specific training plan or to the timeline of training activities outlined in the DIP work plan, in which training and refresher training in child survival interventions and related crosscutting approaches was to be conducted on a regular basis throughout the duration of the project. World Vision 59 LCSP Final Evaluation Figure 9 WV Indonesia staff trained per LCSP Semester 0 5 10 15 20 25 30 S1 S2 S1 S2 S1 S2 S1 S2 FY01 FY02 FY03 FY04 Local International Source: LCSP, September 2004 (see Annex 33) The health motivators attended on average the same number of training events as the entire staff, which is about four events per staff members. The longer they have been working on the project the more events they have attended. The training events attended by the motivators include one or two-day workshops on project-related issues involving various other LCSP and partners staff but not specifically on child survival or behavior change topics related to their jobs. The 2-day workshop “Training of Trainers for Health Staff” at the beginning of the project was specifically conducted for the seven new motivators before the training of Posyandu cadres and other community members. This training event probably covered the key elements of the child survival interventions selected for the LCSP and to promote at the community level. However, this training was followed by a period of high turnover for the motivators and only two of the seven trainees who attended that training were still on the job by the end FY 2002. During a group interview conducted for the FE, some motivators declared that they found the training that they has received very useful for their work because they all were quite recent graduates with limited work experience but had to work with and supervise older cadres and village leaders. They also found the LCSP support and leadership very useful with this respect. The LCSP Core Team, the ADP and the National Office (NO) Health Unit staff attended on average about four training events per person, a third of which were outside of Indonesia. The training events attended in country consisted for one part in the same events as those attended by the health motivators, that is, various one or two-day workshops on project-related issues and facilitated by external experts, the results of which are discussed in other sections of the report. The other training events were organized by other sections of WV Indonesia, the MOH or other organizations and all relevant to the new staff members with health background but not always much experience with World Vision or rural development. As noted for the motivators, however, very little of this training addressed the specific child survival interventions or other topics directly relevant to the project such as behavior change communication. The events attended internationally are all very relevant to the LCSP and lasting between one and up to five weeks. Except for the KPC survey training, which provided the LCSP Team Leader with the skills necessary to successfully lead the MTE KPC survey, the direct application of skills acquired through the other training events is more difficult to assess. The training in World Vision 60 LCSP Final Evaluation BCC/BEHAVE framework attended by three LCSP staff members probably should have been followed by technical assistance for the timely development of a behavior change strategy and related targeted key messages, tools and training materials. The two senior ADP managers attended one training events each that consisted of the participation as external team member in the evaluation of another CSHGP project implemented by World Vision in Cambodia and in India. All ADP staff (18) received some training on CS interventions,42 sometimes even their family received health education on nutrition, diarrhea and malaria. The Community Development Coordinator, the Monitoring and Evaluation Officer and one motivator participated in the BCC and OCA training in 2003. As these and several other staff members were involved in key LCSP activities such annual reviews, KPC surveys and the sustainability workshop, they also gained valuable experience in implementation of child survival programs. In 2002, the ADP manager and the Community Development Coordinator had the opportunity to participate in the Final Evaluation of the World Vision Child Survival Projects in Cambodia and India, respectively. Through this training and the implementation of the project, the LCSP Core Team, the ADP and the National Office (NO) Health Unit staff members gained expertise in planning, implementation and evaluation of child survival projects. They also had opportunities to test and develop new tools and approaches that will be valuable for other projects: ITN distribution strategies; Posyandu monthly reports, Observation Checklist and Exit Interviews; doer nondoer survey; organizational capacity assessment, and others. However, at the end of the LCSP this increased capacity for child survival only applies to the five staff members remaining employed by WV Indonesia (including two members of the ADP staff) out of the 11 staff members who benefited from training over the life of the project. The DIP allocated 25% of the USAID budget to training and 35% of that budget was unspent at the end of the project in August 2004. Therefore, financial resources have not been a constraint to providing more focused training and follow up to the LCSP staff, particularly the Core Team, the motivators and the ADP staff. The WV Indonesia National Health Advisor and the LCSP Project Officer, both based in Jakarta, have provided regular technical and managerial backstopping of the LCSP and conducted quarterly visits to the project site (see section g). These field visits typically included meetings with project staff to discuss progress on the plan of action, to prepare special activities such as annual project reviews and evaluations, and address any staff or implementation issues. They did not use any systematic staff supervision system to identify needs of the field staff and review their performance. Up until January 2004, the LCSP Core Team was recording individual health motivator’s activities in the monthly report to the NO. As this procedure created concern among the motivators and no follow up was made based on this information, the Core Team decided to delete this section of the monthly report and keep track of the motivators’ activities and provide feedback at their level. There was otherwise no tool or system for supervision of health motivators. As mentioned above, the motivators found the support and leadership from the Core Team very useful in general but felt without enough assistance in situations where they had to face authority figures such as government officials including the heads of Health Centers. 42 Most of this training or education was provided by the LCSP staff during regular LCSP/ADP meetings and is not listed as in Annex 28. World Vision 61 LCSP Final Evaluation The above review of the staff training and development of the LCSP suggests the effective implementation of capacity building projects such as those funded by the USAID/CHSGP requires a staff training needs assessment and plan tailored to the specific objectives of the project; a supervision system to ensure the transfer of learning to the project activities and identify related constraints; and the provision of technical assistance to complement and reinforce the application of the new knowledge and skills. Such staff dev and leadership development plan should take into account the inevitable staff turnover during and at the end of the project. c. Human Resources Management The DIP comprises field positions for 1 Technical Team Leader, 1 Training Coordinator, 1 Monitoring and Evaluation Officer, 1 Micro Enterprise Development Officer, 1 Financial Officer, 7 Health Motivators and 3 support staff (administrative assistant, driver and guard). The recruitment of the Core Team staff was done by the National Office in Jakarta and the recruitment of local staff (health motivator and support staff) was done by the field office. The LCSP staff was integrated to the ADP. Annex 36 presents the ADP and LCSP organizational chart at the end of the project. The MED Officer position is represented in this organizational chart but was never filled throughout the duration of the project. The roles and responsibilities of the members of the Core Team and health motivators are outlined below. Table 10 Roles and responsibilities of the LCSP Core Team and Health Motivators LCSP Position Roles and responsibilities Technical Team Leader Lead project team in day-to-day implementation of LCSP activities Coordinate work of the Monitoring and Evaluation Officer and Training Coordinator Supervise Health Motivators Training Coordinator Organize training for LCSP and government health services staff and for community health workers and other members. Monitoring & Evaluation Officer Establish and maintain LCSP Health Information System and Monitoring and Evaluation activities Micro-Enterprise Development Officer Not filled Finance Officer Ensure financial management of project Health Motivator Ensure field implementation of project activities Work together as one team under the responsibility of the LCSP Technical Team Leader Act as liaison between community members and government health services staff Source: LCSP, September 2004 In addition, the LCSP has provided support for a full time position of Program Officer and the technical and managerial support of the National Health Advisor in the Health Unit of WV Indonesia in Jakarta. Annex 37 provides the timeline of the occupancy and Annex 38 the vacancy, turnover and female occupancy rates of the various positions above from July 2000, when the decision to change the project was made, to September 2004, the end of the LCSP. While some of the staff recruited at the start up of the previous project in Poso was able to continue working on the World Vision 62 LCSP Final Evaluation LCSP, most of the LCSP staff including the Team Leader, the Training Coordinator and all the Motivators only began work in the second quarter of FY2001. Primarily because of this initial delay, the overall vacancy rate for all the positions is 27%. The average vacancy rate is the highest for the motivators’ positions (36%) but primarily because of a series of resignation at the beginning of the project. Overall, the annual turnover rate during the time the positions are filled is 30 percent, that is, one third of the staff is replaced each year. This rate was also the highest for the motivators. Except maybe at the beginning of the project, most staff left for career and family reasons. Annex 38 also shows that overall, female staff occupies about half of the positions and that this percentage was higher for the motivators (61%) and lower for the Core Team (34%). The FE Team confirmed the finding of the MTE that the LCSP enjoyed a very motivated and competent technical and managerial staff. Teamwork has in general been constructive and without major conflicts. WV Indonesia has clear and sound personnel policies in place in all its ADPs and special projects like the LCSP. These policies will remain in the ADP and other ADPs after the end of the LCSP. At the time of the FE, the ADP had decided to retain 3 or 4 motivators as staff members. Although their job description will differ from that on the LCSP, the decision will facilitate the continuation of the fieldwork established by the LCSP, primarily the support provided to the Posyandus. While the first LCSP Technical Team Leader had made plans to work on a similar project but with another organization by the time the one-year extension of the LCSP was approved, the second LCSP Technical Team Leader will remain employed with WV Indonesia and work on other child survival and health projects from the NO Health Unit. All the other Core Team members had made other career and familial plans outside WV Indonesia by the end of the project. d. Financial Management All the financial management of the LCSP in-country activities has been conducted by WV Indonesia, as there has not been any sub grant to any partner. The Financial Officer position was filled by two competent accountants during the entire period of the project, one from the beginning and for more than three years and another one for the last eight months of the project. Both were able to assist the technical staff in budgeting and financial reporting in compliance with the monthly grant financial cycle below. World Vision 63 LCSP Final Evaluation Figure 10 LCSP Monthly Grant Financial Cycle 25 35 15 30 MONTH I Financial Officer sends timesheet by: Accountant sends Sun system report “hold journal” and funding/expenses reconciliation Accountant sends complete financial report NO sends charging expense NO informs rate $ NO sends Grant Financial Report to project MONTH II WV International requires several types of financial audits of grants and special projects: financial; integrated (financial and operational); commodity; and special audits (ex. fraud audit). Financial audit variables are internal control structure; analytical review; financial report / accounting record; implementation of previous audit recommendations; losses & pending litigation; income generating activities; and loan records. In FY 2004, Landak CSP received audit rating of “Limited Risk (I).43” The audit results showed that the project adequately met established NO standards for accountability, reporting and compliance to field systems, and that funds were adequately accounted for. In the auditor’s opinion, the financial and narrative reporting of the project to the NO presented an accurate picture of what was actually taking place at the project. Minimal exceptions/problems were noted. The MTE Team noted the low burn rate of the LCSP and, given the delayed start up resulting from changing project site during the first year of the grant, recommended to apply for a two￾year No-cost Extension to achieve the scale-up, capacity building and sustainability objectives of the project. USAID approved the first request for a one-year No-cost Extension in September 2003 but not the second submitted one-year later. Given the substantial amount of unspent funds at the end of the fifth year of the project, the FE Team briefly reviewed the main reasons for this situation. For this purpose, the LCSP Team provided the indicative financial statement below (Table 11), that shows that as of August 2004, 24% of the USAID grant budget and 33% from the WVUS match was unspent. 43 WV audit ratings are: Limited Risk (I); Acceptable Risk (II); Significant Risk (III); and Unacceptable Risk. World Vision 64 LCSP Final Evaluation Table 11 Indicative LCSP Financial Statement as of August 2004 Total Grant Budget Total Expenses Remaining Funds Total Grant Budget Total Expenses Remaining Funds Salaries and Benefits 192979 175546 17433 508 508 0 Travel 145249 95821 49428 235 235 0 Training 179596 116524 63072 0 0 0 Supplies and equipment 122432 75232 47200 18361 9899 8462 Capital Asset 25874 25874 0 26811 26811 0 Consultant 65985 65874 111 24038 22693 1345 Other direct 273 975 -702 120747 67815 52932 TOTAL DIRECT COST 732388 555846 176542 190700 127962 62738 USAID World Vision US Match The FE Team identified that the reasons for unspent funds are threefold:  The value of the Indonesian rupiyah against the US dollar fell drastically since the grant was awarded in September 1999 and the costs for local salary, purchases and operations have been far lower than budgeted for in US dollars.  The cost of some activities has been reduced from cost sharing from the MOH (IMCI training, microscopes, ITNs, MCH Booklets).  Some activities budgeted for in the DIP were not implemented such as local training in Micro Enterprise Development, Continuing Quality Improvement, Lot Quality Assurance Sampling and others topics; provision of some equipment and supplies such bicycles and “lab in a suitcase”; recruitment of a MED Officer; and provision of loan funds. By the end of the project, the overall budget was reallocated for different items such as an additional 24,000 ITNs, consultant services and capital assets. The LCSP and ADP first received technical assistance for developing a sustainability action plan and exit strategies in March 2004. This assistance did not specifically cover the detailed planning of activities for FY 2005 or the financial planning necessary for implementing these plans. The FE Team found that the ADP Plan of Action and budget for FY 2005 was not completely in line with the LCSP experience. The budget for some activities, like meetings with the DHO, seemed overestimated while the budget for others, like quarterly meetings with Posyandu cadres, seemed underestimated. This suggests that the ADP would benefit from continued technical assistance from the WV Indonesia Health Unit in Jakarta for at least a year for planning and budgeting LCSP routine activities and probably longer if any adaptation or development of new child survival activities are needed. e. Logistics WV Indonesia implements strict purchasing procedures such as the non-involvement of the project manager and accountant; the use of Purchase Requisition and Purchase Order Forms to track and document transactions, request for quote and conduct bid analysis; and the consultation with relevant technical department for equipment such computers, audiovisual, and vehicles. World Vision 65 LCSP Final Evaluation The distribution of equipment and supplies within WV Indonesia has not been a problem and in general, the specific logistic needs of the project in terms of office space, equipment and supplies, computers, and vehicles were adequately met. The ADP also has adequate logistic resources and the activities taken over the LCSP may not require additional resources. The extent to which the ADP will be able to meet the needs of the MOH with respect to its transportation to provide services in the Posyandu is uncertain. f. Information Management The LCSP measured progress towards the technical interventions objectives through a series of three KPC surveys. These surveys were successfully completed at baseline, midterm and the end of the project. The results of the survey were available and used for the DIP workshop, the MTE and the FE. At midterm, findings that several key indicators seemed to lag behind urged project managers to reorient the efforts and related strategies of the project. This is the case for instance for ITN use and ARI knowledge and health seeking behavior: As indicators for these two interventions showed a lack of progress at midterm, the LCSP team decided to organize two mass campaigns in 2004, one on malaria in March and one on ARI on May and June. These two campaigns resulted in significant increases in the related indicators. The progress towards the Capacity Building and Sustainability objectives of the LCSP were not as systematically measured as the health objectives. One reason for this finding is that most indicators, objectives and related strategies were not been fully developed in the DIP or at the beginning of the project (see section 3.c and 3.d.). The LCSP also collected and reported systematically various data on activity at the posyandu level, training events, sales of ITNs, and other project activities that helped program managers following up on the implementation of the work plans. Below are examples of data collected and their at the LCSP level:  The number of active Posyandus and respective coverage helped project managers plan the opening of new posyandu and organize the training and support activities;  The number and type of trainees helped plan the training activities to eventually reach all members of each target groups;  The data on sales of ITN were used to assess the effectiveness of the distribution strategies in meeting the coverage objectives;  The LCSP motivators regularly collected various data from the Posyandu. These data allowed LCSP managers to monitor activities at the Posyandu level and supervise the motivators and also develop and test tools that can be used by Posyandu cadres and Health Centers to monitor Posyandu activities (see section 2.c.iii). The data collection, analysis and use processes above required substantial time of LCSP qualified staff, from the motivators to collect the data, to the M&E Officer to manage and analyze them, and to the Core Team and ADP staff to interpret and make the appropriate decisions. The routine data and information needed to support the activities that the ADP will continue can probably be produced by the ADP, which has staff with appropriate qualifications and experience. No specific provision or plans to ensure the successful transfer and continuation of the necessary information system had been made at the time of the FE. World Vision 66 LCSP Final Evaluation The simple continuation of selected activities of the LCSP project does not necessarily require conducting complete KPC surveys on a regular basis. This would be necessary if the ADP adopts specific knowledge, practice and coverage indicators and objectives and if it is interested in collecting evidence to support a broader analysis of the sustainability of child survival interventions in the context of an ADP. At this time, the ADP does not have the skills needed to conduct such surveys that could be meaningful for these two purposes. The Transformational Development Indicators (TDI) household survey that WV Indonesia plans to conduct in all ADPs in Indonesia includes a few child survival indicators but would those would not satisfy these specific information needs. Section 2.c.iii describes the support that LCSP provided to the MOH in assessing the collection and reporting of the data from the Posyandu Monthly Reports. During year 2003, this system achieved about 90% of completeness of reporting and produced meaningful indicators at the Health Center level (see examples in Table 5). This pilot data collection and analyses efforts were still ongoing at the time of the FE and not shared with the Health Centers and the DHO yet. At the end of the project, the LCSP and the DHO adopted a last revision of the Posyandu Monthly Report and LCSP printed and provided to the DHO with 600 copies of this form (see Annex 25). The staff of Health Centers and DHO will continue implementing this system in the three Health Center areas and in the other sub-districts. The actual and potential uses of the data collected at the Posyandu, Health Center and DHO levels was not fully assessed during the FE. The LCSP was otherwise not involved in other aspects of the HIS in the health centers and DHO. The FE Team visited two Posyandus and examined the completeness and correctness of the immunization registers (see Annex 15). In one posyandu, the report was complete and correct and in the other, the report was complete but incorrect. The assessment of the recording skills of the Posyandu cadres also suggests serious problems in the quality of recording (see sections 3.c.iv Section 2.c describes the use of data and special outcomes of special studies that the LCSP conducted in the area of malaria (chloroquine efficacy study, development of strategies for ITNs distribution and use, and doer nondoer survey of ITN use), community based health information systems (assessing the value of the monthly reports and other tools used at the Posyandu level), and organizational development (use of an Organizational Capacity Assessment tool in the context of a child survival sustainability assessment and planning). The LCSP staff acquired a clear understanding of what the project achieved in terms of health objectives through the series of KPC surveys. The WVUS staff also has a good understanding of the overall progress and achievements and the constraints faced by the project within its national and international context—these were clearly outlined in the May 2004 No-cost Extension submitted to USAID. The FE Team held a one-day Technical Overview meeting at the beginning of the evaluation to present and discuss the FE KPC results and the main achievements of the project before proceeding with the Field Visits. This meeting gave the opportunity to obtain feedback from several representatives from the Health Centers and DHO. Overall, they asked for various clarifications but approved the data presented except those for immunization coverage, as was the case during the MTE. The results of the attempts of the FE Team attempted to clarify the source of the differences between the services statistics and the KPC coverage estimates are presented in section 2.b.i). World Vision 67 LCSP Final Evaluation During the Village Leaders Feedback meeting that the FE Team conducted to present the FE KPC results, the village and sub village leaders present showed interest, understanding and concern for the data and issues discussed. This made it easy to mobilize them to formulate a joint Declaration of Commitment to continue supporting the LCSP achievements and the Posyandus in their village (see Annex 20). Prior to the FE, the LCSP Core Team and one external consultant began documenting several aspects of the project. They conducted this work with the dual perspective of bringing information into the FE process and providing the materials for a series of technical papers. The topics and prospective audience of the papers considered at the time of the FE were:  Posyandu Health Information System: The LCSP has been able to activate the MOH information system for Posyandu and compile the data from 90% of the expected Posyandu Monthly Reports in the project area in calendar year 2003. The LCSP also has collected and compiled various other data from other tools used to assess the performance of Posyandus. The full analysis and dissemination of the results of these analyses will help the Landak DHO and other DHOs interpreting the data from these reports and the national level (MCH Department) assessing and further developing tools for monitoring Posyandus activities and performance.  Chloroquine Efficacy Study: The Landak DHO and the LCSP successfully conducted a study of the resistance of Plasmodium Falciparum to treatment with chloroquine according to the MOH first-line protocol. The study was conducted from May to July 2003 in 32 villages of 5 sub-districts in Landak Districts. The study showed high levels of resistance and the results were promptly made available to support advocacy for a change in the malaria drug policy and for malaria control in general at the DHO and PHO levels. Given the potential value of the study at the national and international levels, a short version of the investigators report is under preparation for wider dissemination.  Promotion of Long-Lasting ITNs. Most of the programmatic information and sales data on the various strategies tested during the course of the LCSP to promote the purchase and use of ITNs had been compiled at the time for FE. Only part of the available information is included in this FE report (see section 2.c.i). Given the commitment of the ADP to continue this intervention and the interest at the national level for such field experience, a more complete description and dissemination of this experience is warranted.  Doer nondoer survey. The LCSP conducted a doer nondoer survey to assess the impact the recent malaria mass campaign on the actual use of ITNs by pregnant women on children under five in households who bought one. The survey helped the development of a BEHAVE Framework for the promotion of ITN use. The full analysis and documentation of this behavior change intervention research would be of interest to other program promoting ITNs in Indonesia and elsewhere and to organizations implementing behavior change interventions in general.  Special analyses of the three, maybe four, LCSP KPC surveys. The relatively large sample size and good quality of the baseline, midterm and final KPC surveys allow for more advanced analyses of the determinants of coverage and behavior and of the impact of the LCSP (trends and other analyses) than those typically expected from a KPC survey. The good documentation of the LCSP activities and the continuation by the ADP of some of these activities including the related information system, also constitute a good opportunity for World Vision 68 LCSP Final Evaluation conducting a fourth KPC survey two or three years after the end of the project. Finding of such study, if successful, would be of interest to the ADP for the last years of their mandate, to WV Indonesia for developing their strategies of integration of child survival into other ADPs in Indonesia, and to WVUS, other PVOs and USAID for the design of future child survival projects. g. Technical and Administrative Support The LCSP received technical and administrative assistance from World Vision's Asia Pacific Regional Office, World Vision US and external consultants, both national and international. Annex 39 shows the timeline of this assistance provided through field visits from October 2000 to September 2004. The World Vision US Program Officers, Technical Officer and Financial Officer primarily assisted the LCSP at the design and startup stages of the project for the preparation and submission of the DIP (October 2000) and for setting up the grant management systems and procedures (March and June 2001). One WVUS technical officer participated in the midterm (August 2003) and the final evaluations (September 2004). In March 2004, the WVUS technical officer assisted the LCSP team and local partners in developing a Sustainability Action Plan using the Child Survival Sustainability Assessment methodology introduced by the CSTS project (see section 3.d.ii). The Regional Health Advisor provided regular technical assistance field visits both to participate in key events such as annual reviews and evaluations, review specific aspects of the projects, introduce new tools or technical updates or facilitate special analyses and development of consensus with local counterparts. Various national consultants provided specific and short-term (a few days) assistance to the LCSP team through training in the area of Malaria, IMCI, use of the MCH Handbook, Nutrition, Monitoring & Evaluation and Community Development (PLA, Peace Building/LCP, Transformational Development). Both national and international external consultants led the First Annual Review, Midterm Evaluation, Third Annual Review and Final Evaluation. The LCSP also used international consultants to provide training and technical assistance in two specific areas. In October 2003, one international and one national consultant led a provincial￾level four-day workshop on community-based information systems, which were followed by a field assessment of the existing systems and potential improvement in the project area (see section 2.c.iii). In July 2004, one international consultant led a four-day training workshop on BCC which gave the LCSP team the opportunity to develop a BEHAVE framework for use of ITNs, among other application of the framework. The assistance provided by an external consultant for the documentation of various aspects of the LCSP is mentioned above. The FE Team identified that more focused and earlier technical assistance could have helped the LCSP achieve additional results in the following area:  Micro enterprise development: No technical assistance has been available for this otherwise important component of the project for which LCSP did not achieve the expected results (see section 3.d.iii).  Behavior Change Communication and Interventions: Three LCSP staff members received international training in the BEHAVE framework (2 in South Africa in 2002 and 1 in World Vision 69 LCSP Final Evaluation Cambodia in 2003) but follow up technical assistance was only made available at the end of the project in July 2004 (see section 3.b).  Sustainability: Technical assistance to define sustainability and capacity building plans and indicators was scheduled during the midterm evaluation and the third annual review but only available in March 2004 (see section 3.d.ii).  CBHIS: One external consultant provided technical assistance on CBHIS in October 2003 but too late to begin adapting and testing the EPI and Pregnancy tracking register and the CBBDS proposed in the DIP (see section 2.c.iii).  Training needs assessment and planning: Training was a major activity proposed in the DIP and to which 25% of the USAID budget was allocated. The development of a training plan based on a complete training needs assessment and including specific objectives and plans specifically linked to project objectives would have helped the program managers and training coordinators to implement a more targeted and intensive training program (see 3.c.v and section b). The FE Team also found that the LCSP could have benefited from a closer backstopping from WVUS during its first three years. This assistance was necessary for specifying the various project components included in the DIP, identifying technical and training needs for the LCSP staff, following up on burn rate of the funding and make the necessary adjustments to the DIP and work plan after the annual reviews and evaluations. h. Management Lessons Learned The fundamental lesson, reflected in findings of the evaluation team and fully consistent with World Vision's experience with other current and former CS grants projects, is that intensive technical assistance and continuous monitoring must be provided throughout the life cycle of these projects. Despite the many positive aspects of Landak CSP, and results that have been achieved, the final evaluation has documented several areas in which management of this project has not been satisfactory. These have been summarized above in Sections 4.a, c, d and g. These sections address Planning, Human Resources Management, Financial Management, Technical and Administrative Support respectively. In each of these areas, there is evidence of significant room for improvement, and specific lessons that should be taken into consideration by World Vision, and any other PVO considering application for new child survival grants: Planning: Initial plans, based on the original proposal, were wholly disrupted by civil strife in Poso District, Central Sulawesi. Responding to this setback, World Vision decided to relocate the project site to Landak District, West Kalimantan. While it was understandable, and probably wise, to limit the scale of the project to two sub-districts during the first year, there was no provision in the Detailed Implementation Plan for expansion to additional sub-districts during the life of the project. With more foresight, or more intensive monitoring combined with flexibility in planning, the project’s slow start and severe underspending could have been corrected with revision of the DIP, adding more staff and covering more sub-districts, from the second year onward. This could have improved project results overall, helping to ensure that grant funds were used optimally; with maximum benefit for the people of Landak District. World Vision 70 LCSP Final Evaluation Human Resources Management: Closely related to planning, good HR management is essential to the success of child survival projects. Even when the need for additional staff is recognized, recruitment of qualified people who are prepared to work very hard in difficult and potentially dangerous conditions, without much assurance of job security given the short lifespan of most projects, is a challenge. In the case of Landak CSP, World Vision was conservative in estimating the number of technical and field staff required to achieve planned results. After the first year, the need for geographically expanded and/or staff-time intensified effort should have been apparent, along with the project’s correspondingly low rate of expenditure. If more local staff had been recruited and trained at that time, to accelerate the pace of activities in the original two sub-districts and extend into other parts of Landak District (with appropriate revision of the DIP), more could have been achieved, and grant funds as well as other available resources could have been used more effectively. As a related HR consideration, if World Vision US fully recognized the intensity of technical assistance and monitoring necessary to adequately meet HQ responsibilities in managing a child survival project, more consistent and focused attention would have been given to Landak CSP, changes might have introduced at a stage they could make a difference. Project results might have been enhanced, and grant funds might have been used more effectively. Although professional staff at headquarters, regional and national office levels are designated to be responsible for the support of child survival projects, there has not been adequate recognition of the staff time required for this purpose, or of the need to limit competing demands and ensure that staff tasked with support of these projects can give due attention to meeting the extraordinarily high standards of performance and reporting set by the HCSGP. Financial Management: World Vision appears to have been successful in using most of the grant amount, with corresponding match, before project ended. However, LCSP has been markedly underspent through most of its four-year cycle, FY 2000-2004. One understandable reason for this underspending is the steady decline in value of the rupiyah against the US dollar during this period of time, and – it can be noted – a remarkable volume of activity was carried out, considering the low rate of expenditure. While seeking maximum value for money expended, and pursuing implementation strategies that have potential for sustainability, are both desireable and commendable, these instincts appear to have constrained World Vision’s freedom of action in correcting an unacceptable situation. In a USAID-funded child survival project, grant and match funds should be used as intended, and cannot fulfill that mandate if they are not being spent according to the very inflexible timeline inherent in all such projects. A one-year no-cost extension was approved, but this was not enough additional time. Analysis of the funding pipeline quarterly through the life of project suggests that a two-year no-cost extension might not have helped much, either. A combination of more flexible planning and more assertive financial management, from the outset, could have made a difference, however, in spite of the real challenges involved. Technical and Administrative Support: Lessons in this realm are closely related to those already noted above. Adequate staffing with qualified professionals, and realistic allocation of their time, are both essential for meeting the support demands of CS grant projects. Evidence available from this evaluation suggests that, despite the considerable support that World Vision World Vision 71 LCSP Final Evaluation headquarters, regional and national offices have provided to Landak CSP, more was needed – that is, more on a consistent basis throughout the life of project, and more on an intensive basis for planning and monitoring, during the earlier years in particular. The decision of World Vision US to refrain from applying for a new CS grant in FY 2005 was prompted by the recognition that current staffing is inadequate for providing support for more than the two currently underway in India and Kenya. The findings of this evaluation tend to confirm that judgement. World Vision should carefully and realistically “count the cost” associated with these projects. Before new grants are sought, ensure the organization is actually prepared to commit the staff time necessary to provide appropriate support from headquarters, regional and national levels. World Vision 72 LCSP Final Evaluation C. Conclusions and recommendations a. Success in meeting objectives Referring to the Summary Chart of Results in section B.1, the LCSP achieved its objectives, intermediate results and goal as follows. Technical interventions The LCSP contributed to Intermediate Results 1.1 and 1.2. in so far as the use of integrated child and maternal health services by the target population and that the participation and contribution of communities for the prevention and early and complete management of diseases have increased. The LCSP contributed to Intermediate Result 1.3 in so far as communities invest their limited resources in low cost high impact CS interventions such as running and attending Posyandus and buying ITNs. However, the LCSP only met the target of 7 of the 16 end-of￾project objectives for the technical interventions. Table 12 summarizes the baseline and final evaluation results for the 14 key LCSP indicators measured through KPC surveys. Table 12 Summary of Baseline and Final Evaluation Results from KPC survey # Indicator1 [children 0-23 months or their mothers, except when noted] Base￾line Final Eval. EOP Target Target met2 1 Fully immunized children, 12-23 months, card verified 23% 61% 80% No 2 TT2 coverage, card verified 2% 44% 50% Yes 3 Vitamin A supplementation, 12-23 months 54% 88% 90% Yes 4 Vitamin A supplementation, post-partum 12% 41% 50% No 5 Seek treatment for fever, 12-23 months 26% 54% 75% No 6 Treatment of fever in health facility within 48 hours - 49% 45% Yes 7 ITN ownership - 23% 30% No 8 ITN use last night - 21% 20% Yes 9 ORS use, 0-23 months with diarrhea 32% 52% 75% No 10 Same amount or more fluids, 0-23 months with diarrhea 63% 78% 75% Yes 11 Same amount or more foods, 0-23 months with diarrhea 34% 39% 70% No 12 Antidiarrheal medicine, 0-23 months with diarrhea 32% 50% 10% No 13 Mother’s knowledge of 1 pneumonia symptom 25% 55% 80% No 14 Treatment seeking, 0-23 months with pneumonia signs 59% 72% 75% Yes 1 See full definition of indicators in Results: Summary Chart (section B.1). 2 Target met if included in 95% confidence interval around Final Evaluation KPC estimate. Immunization The LCSP achieved an impressive increase in the proportion of card-documented Fully Immunized Children, from 23% to 61%. The ambitious target of 80% children 12-23 months fully immunized, verified by card, is not achieved mainly because of a limited access (18%) but also because of a moderate dropout rate (10%) and a lack of immunization card (5%). The LCSP also achieved an impressive increase from 2% to 44% in the proportion of mothers of a child under two who received at least two card-documented tetanus toxoid injections during their last pregnancy. Not only the target of 50% of tetanus toxoid immunization coverage verified by card is achieved but the actual coverage, as measured when the mother’s recall is included, reaches 74%. World Vision 73 LCSP Final Evaluation The LCSP contributed to the quality of the cold chain in the three Health Centers, although there are still deficiencies (not in Table above). Vitamin A supplementation The LCSP achieved the 90% target for vitamin A supplementation, with 88% of children 12-23 months who received vitamin A in the last 6 months. Although there has been a steady increase in the percentage of mothers of children under two who received vitamin A within one month after delivery, from 12% to 41%, the 50% target for this intervention post is not achieved. Malaria control The LCSP improved health-seeking behavior for fever. The proportion of children under two with fever in the last two weeks who are brought to a facility has doubled, from 26% to 54%, although the high target of 75% is not achieved. The proportion of those children who were brought to a health facility within 48 hours of hours after the fever began reached 48%, thereby meeting the 50% target for this indicator. The LCSP also reached the 20% target for ITN use by children under two, increasing from 2% to 21% the proportion of these children who slept under an ITN the night before the survey. This indicator reflects the increase in ownership of ITN and the fact that nearly all ITNs owned are used. The target for household ITN ownership set to 30% to account for possibly large proportion of ITN owned but not used is not achieved. Diarrheal diseases control The LCSP did not reach its target for ORS use but reached the one for increase in fluids in case of diarrhea. ORS use for children under two with diarrhea in the last two weeks steadily increased from 32% to 52%, remaining short of the high 75% target for this indicator. The percentage of children with diarrhea who received the same amount or more fluid increased from 63% to 78% during the same period, thereby meeting the target for this indicator. The LCSP did not improve the feeding practices and the use of antidiarrheal medicines for children with diarrhea. The percentage of children with diarrhea in the last two weeks who received the same amount or more food remained around 30% throughout the duration of the project, far below the 75% target for this indicator. The use of antidiarrheal medicines for diarrhea for children under two in the project area is still a concern. Although the LCSP target was to decrease to 10% the use of antidiarrheal medicines for children under two with diarrhea, the FE KPC survey shows an increase of this indicator to 50%, up from around 30% at midterm and baseline. Acute respiratory infection control The LCSP did not reach the 80% target for the percentage of mothers of children under two who know at least one symptom of pneumonia but reached the 75% target for early treatment seeking. The percentage of mother with children under two who had symptoms of pneumonia (cough and rapid or difficult breathing) who sought treatment increased from 60% at midterm to 72% at the end of the project. Micro-enterprise development The LCSP did not reach the objective set for the Micro-Enterprise Development intervention that 20 KSM would be trained and delivering health education (not in Table above). The ADP only World Vision 74 LCSP Final Evaluation began introducing health related activities in a few SHGs in the last years of the LCSP. No training in health education has been provided to SHG members so far. Capacity building The LCSP contributed to Intermediate Result 2.1 in so far as the local health care system and communities are equipped and provide child survival interventions at home (most of LCSP coverage and behavior indicators increased); at the community level (95% of the population in the project area has access to a functioning Posyandu); and in health services (health workers regularly provide clinical services in the Posyandus and in the three Health Centers). However, the LCSP only achieved 4 of the 8 end-of-project objectives for capacity building. Local health care delivery system  The LCSP and the PHO officials met as needed for project activities but not regularly for updates of project process.  The LCSP and the DHO officials regularly met for project activities but not for systematic planning, priority setting and review of accomplishments.  The LCSP did not work directly with health centers’ management teams on routine supervision or use of data for decision-making.  The LCSP provided training and support to health workers at all levels. Health Center staff was trained in IMCI, malaria microscopy, nutrition and were involved in LCSP activities such as training of Posyandu cadres and other community health volunteers. Posyandu cadres were trained and received regularly support for running Posyandus. Various other community health volunteers (shopkeepers, TBAs, Village Drug Posts cadres) were trained. Communities  The LCSP provided support to create or revitalize 108 Posyandus involving trained 703 cadres. This number corresponds to 92 children under five per Posyandu, which is in the 50 to 100 range recommended by the MOH. The LCSP team estimates that the population with geographical access to a Posyandu reached 95% at the end of the project in 2004. In 2003, the average Posyandu attendance rate was 71% for infants and 39% for children 1-4 years old.  The LCSP trained in CS most informal health partners in the project area (shopkeepers, TBAs, Village Drug Posts cadres), except for Traditional Healers and Self-Help Groups. NGO Partners  LCSP did not have activities with the local NGO designated in the DIP (the Pancur Kasih Foundation) but began working with Koperasi Pancur Dangeri for the distribution of ITNs though Self-Help Groups (in 2003) and Posyandus (in 2004). ADP  The ADP staff members participated in various training and other LCSP activities and had built its capacity to implement child survival interventions World Vision 75 LCSP Final Evaluation Sustainability The LCSP contributed to Intermediate Result 3.1 in so far as the DHO and the three Health Centers deliver CS interventions and engage with community volunteers to improve maternal and child health. The LCSP also contributed to Intermediate Result 3.2 in so far as most communities in the project areas engage in the prevention and early and complete treatment of common illnesses by running and attending Posyandus and adopting improved behaviors. However, the LCSP achieved only 3 of the 6 end-of-project objectives for sustainability.  The LCSP provided logistical support for and facilitated nearly monthly visits of Posyandus by Health workers but they usually focus on providing health services rather than supervising and encouraging community health workers, a function that has largely been fulfilled by the LCSP health motivators so far.  The LCSP provided visual aids and Posyandu cadres regularly use them to provide health education. The LCSP trained shopkeepers but they are not regularly supervised and may need retraining. The LCSP did not to support training of Traditional Healers.  The LCSP achieved significant increases in the coverage of most interventions it supported, most likely largely due to increased community participation and functioning Posyandus. Most communities are committed to continue running Posyandus and sometimes proactively ask for quality and timely government health services in their areas.  The LCSP strengthened existing Posyandus HIS (registers and reporting forms) supported by the MOH but did not introduce community-based EPI/Pregnancy Registers and a Community Based Death and Disease Surveillance system (CBDDS)  The LCSP did not introduce health related activities among some SHGs and did not train their members in health education; at the end of project, most SHGs do not have budget allocated for health and do not engage in the promotion of CS messages.  The LCSP trained the ADP staff in various topics including child survival. Cooperation between government health services, the ADP and communities has been successful and has prepared the ADP to take a leadership role in the health sector. However, the ADP has not met the objective of allocating 20% of its FY 2005 budget to health and child survival. Overall, however, the LCSP only met 14 of the 30 end-of-project objectives specified in the DIP and the MTE report. The FE Team determined that some of these objectives were probably unrealistic while others were due to programmatic shortcomings. b. Main Achievements and Factors Affecting Performance The LCSP has assisted the Landak District Health Office in reducing mortality and morbidity of children under five through an integrated child survival project in Mandor and Sengah Temila sub-districts: IR 1.1: The use of child and maternal health services by the target population has increased; IR 1.2: The participation and contribution of communities in the prevention and management of diseases has increased; World Vision 76 LCSP Final Evaluation IR 1.3: Communities are investing their limited resources in low cost high impact CS interventions delivered through Posyandus and in ITNs; IR 2.1: Local health care system and communities are equipped and provide child survival interventions at home, at the community level, and in health services; IR 3.1: The DHO and three Health Centers deliver CS interventions and engage with community volunteers to improve maternal and child health; IR 3.2: Most communities in the project areas engage in the prevention and early and complete treatment of common illnesses. The FE Team identified several factors that have impeded program performance.  Landak is a relatively new District with still limited human and financial resources;  Many villages are located in remote areas with access by health workers and motivators very difficult during the rainy season;  The number of health facilities and the health staff is limited and there is not always someone available to travel and provide services in Posyandus;  About half of the 31 villages in the LCSP area do not have a Village Birthing Post with an active midwife;  Some health staff do not perform as expected and there is no system in place to identify and rectify such situations;  Viability of PODs is questionable given that the drugs they sell are available almost everywhere at about the same price;  The availability and use of the MCH booklets for pregnant woman and their babies remains limited;  The availability of vitamin A capsules for post-partum vitamin A supplementation and ORS for case management of diarrhea remains limited at the community level;  The traditional practice of prohibiting post partum mothers from leaving their house prevents newborn babies from immunization and other services;  ITNs were only made available at an affordable price through the introduction of discount vouchers at the end of the project;  Some components of the DIP and new project components were not fully detailed in and the LCSP Core Team has occasionally lacked necessary technical assistance. c. Best practices and lessons learned Best practices  The employment of a sufficient number of full time committed motivators from the local community is an effective strategy to revitalize Posyandus and achieve health results; World Vision 77 LCSP Final Evaluation  Proactive and highly motivated cadres can ensure regular health workers services during Posyandus days, including home visit of children who do not attend because of illness;  Involving government health services (Health Centers, DHO and PHO) in the preparation for and cost sharing of training health workers has proven to be very effective to decrease bureaucracy and increase ownership of the stakeholders;  Providing proper explanation to mothers about vaccination side effects, especially fever, is very effective in overcoming resistance from the community to bring children to Posyandus for immunization;  Holding regular meetings with TBAs at the sub Health Center or Village Birthing Post levels is an effective way increase their attendance, give them vitamin A capsules and encourage them to provide those to post-partum mothers;  As price is an important constraint to the purchase of ITNs, purposely conducting mass campaigns in March, after the harvest season and when most people possess some cash money, is an effective way to increase sales and use of ITNs without increasing subsidies;  A multimedia mass campaign to promote behavior change is more effective than a single media to achieving behavior change; Lessons learned  Participation from local health authorities has increased over the life of the project but their assistance to Posyandus may very well decrease drastically at the end of the project;  Involving Posyandu cadres as last line distributors and the use of discount vouchers is an effective strategies to increase sales and use of ITNs but ways to keep the price down without making the community dependent on external donors must be found;  Special studies require follow up with field investigators, ideally on the basis on an agreed upon protocol including data analysis and dissemination plan;  Even specialized staff training and technical assistance events require follow up to ensure prompt transfer of learning or implementation of recommendations to the benefits of project;  Effective behavior change interventions require the development of specific messages and strategies early in the program and regular adjustments based on assessment of progress;  Building capacity of program partners require specific organizational development plans based on needs assessment of each partner;  Support to Posyandus, including facilitating the involvement of health workers during Posyandu Days, indirectly strengthens Health Centers’ capacity to increase coverage of their services;  Given the gain in immunization coverage resulting from providing these services in Posyandus, ensuring that health workers visit Posyandus on a regular basis is a very cost effective strategy;  An explicit, realistic and clear sustainability plan should be developed at the beginning of a project and a project exit plan at the end. World Vision 78 LCSP Final Evaluation d. Final Evaluation Recommendations The FE Team developed the following recommendations to the Village Leaders and Posyandu cadres, the three Health Centers, the Landak District and the Provincial Officials, the ADP, WV Indonesia, WVUS and USAID/GH/CSHGP. For Village Leaders:  Take the lead in supporting Posyandu and other child survival and health activities as described in their Declaration of Commitment. For Health Centers:  Develop realistic plans and made appropriate budget allocations to continue support to Posyandus.  Follow up on the declaration made by Village Leaders during the Feedback meeting to ensure this commitment is implemented.  Store measles vaccine in refrigerator at 4-8 C, according to current WHO recommendations.  Monitor refrigerator temperatures more consistently, twice each day, keeping them within the range of 4-8 C.  Ensure transfer of responsibility for cold chain management, including twice-daily recording of refrigerator temperatures, and prompt action to correct any problems, when vaccinator is on leave or in the field.  Implement proper procedures for safe disposal of syringes and needles. At the minimum, crush used syringes promptly and incinerate waste completely or bury it in a deep pit.  Develop and display Standard Operating Procedures for use of generator to maintain cold chain in the event of power outage, making sure it can be started promptly if power fails. Under no circumstances should the generator, cable or other accessories should be taken from the health center or used for other purposes.  Develop and implement procedures for routine ORS distribution to Posyandus, maintaining adequate stocks in the Health Centers and all the Posyandus on a continuing basis. Continue using ORS packets even beyond their expiration dates if powder is not clumped.  Support monthly meetings with TBAs in all sub health centers and Village Birthing Posts to ensure that all trained TBAs have access to vitamin A capsules to give to post partum mothers.  Continue support to PODs that seem to be doing well but carefully assess the usefulness of those that have not been performing so far. For Landak District Health Office:  Integrate the most successful and sustainable strategies of LCSP into the Landak District Strategic Health Plan and related budgets. World Vision 79 LCSP Final Evaluation  Make use of available financial resources (Deconcentration Fund, District Development Budget, Global Funds for AIDS, Tuberculosis and Malaria, Health Work Force) and partnerships (ADP, WV Indonesia, USAID) to support these activities  Replicate the successes of the LCSP in other sub-districts with the assistance of the ADP.  Encourage support for Posyandus by Village Leaders and sub-district officials  Ensure the active presence of a midwife in a Village Birthing Post of all villages in the District.  Develop and implement procedures for routine ORS distribution to Posyandus by Health Centers.  Develop and implement procedures for routine distribution vitamin A to trained TBAs by midwives and other Health Center staff.  Ensure quality of service provision by health staff in Posyandus by supporting, monitoring and supervising these activities with the Health Centers.  Promote use of ITN by pregnant mothers and children under five through continued partnership with local NGOs  Continue BCC activities for child survival and ITN use in the two sub-districts and extend them to other sub-districts. For West Kalimantan Provincial Health Office:  Use the LCSP area as a model for community-based child health programs in West Kalimantan province.  Make use of available financial resources (Social Safety Net Project, De-concentration Fund, Provincial Development Budget, Global Funds for AIDS, Tuberculosis and Malaria, Health Work Force) and partnerships (ADP, WV Indonesia, USAID) to support these scale up activities For ADP Pontianak:  Develop a detailed three-year plan for continuing LCSP activities in the two sub-districts and further expansion to the two other districts of the ADP.  Clarify the specific LCSP activities to carry on including the type of support to Posyandus; the activities that ADP staff without health background can implement; and the health information and monitoring system.  Conduct advocacy with DHO, health workers and village leaders to ensure support of Posyandus  Give more training, health education and data to Village Leaders in the areas where there is a Posyandu World Vision 80 LCSP Final Evaluation  Prepare organizational development plans with the three Cadres Associations on the basis of the Organizational Capacity Assessments conducted during the Sustainability Action Plan workshop and agree on the specific activities and results that the ADP can support.  Conduct an assessment of the usefulness and viability of PODs before creating new ones or supporting those that are not performing as expected.  Follow up on the Declaration of Commitment of the Village Leaders and ensure that it is implemented. For World Vision Indonesia:  Designate one Child Survival or Health respondent in the NO Health Unit to follow up on the child survival plans and activities of the ADP. This nomination should be decided with the NO ADP manager as part of the WV Indonesia plan to integrate child survival in all ADPs in Indonesia.  Provide technical and programmatic assistance to the ADP to further specify the FY2005 Plan of Action and Budget for child survival and ensure the proper transfer of all relevant LCSP data and expertise to ADP.  Provide technical and programmatic assistance to ADP to develop a three to four year plan for child survival including strategies to scale up to its two other sub-districts.  Provide technical assistance to the ADP on integrating health and child survival into their MED activities.  Compile all valuable training, BCC and HIS materials developed and successfully used during LCSP and prepare a technical description of their purpose, lessons learned and recommendations for further use and adaptation.  Develop training, BCC, HIS and other materials for implementation of child survival interventions in ADPs based on the experience of the LCSP.  Document and disseminate success stories, best practices, lessons learned and technical materials that could be used within WV Indonesia and other partner organizations including the MOH.  Provide technical assistance to integrate child survival into the three ADPs in Kalimantan, then gradually into all 28 ADPs within the next 5 years.  Provide support to the ADP to continue LCSP activities at a sustainable level and become a learning site visited by other ADP’s staff.  Prepare short “success stories,” “best practices” and technical reference documents that could be included in various projects or proposals for which funds are available at the Provincial and District levels.  Develop a full proposal for malaria control in Landak District and potentially other area in West Kalimantan.  Conduct trends and other analyses of the data from the three KPC surveys to further assess the determinants of coverage and the impact of LCSP interventions. World Vision 81 LCSP Final Evaluation  Conduct a fourth KPC survey in the ADP area in two or three years, possibly but not necessarily through the TDI survey, as part of a post-project sustainability assessment.  Pursue analyses of the Posyandu information system in collaboration with the Department of Maternal and Child Health to ensure that findings are directly relevant to the current national level efforts to develop Posyandu activities.  Define explicit partnerships and organizational development plans when involving local organizations in special projects. For World Vision US: In current and future CS grant projects,  Provide more sustained technical assistance for overall strategic planning and for specific area of identified needs to maximize results of special projects and build capacity of WV National Offices.  Use programmatic and financial monitoring data from special projects to revisit current plans of action and make the necessary changes including formal amendments to the DIP or agreements as appropriate.  Plan for WV NOs staff training and development within special projects on the basis of needs assessments and the specific objectives of the projects; include a follow up and technical assistance system to ensure the transfer of learning.  Continue technical assistance to WV Indonesia for the integration of child survival in all ADPs and to the ADP for the detailed planning and implementation of the social marketing of ITNs purchased at the end of the LCSP.  Beware choice of indicators and targets, ensuring that indicators are measurable and sensitive and that targets are challenging yet achievable. For USAID/GH/CSHGP:  Ensure that the annual reviews and midterm evaluation guidelines stipulate that grantees may be required to further develop or revise their DIP in the course of its implementation if changes occur that could affect project results. The level of detail expected in DIP revisions should also be specified (background information; rational for the intervention; results of baseline assessments, specific plan of action, monitoring and evaluation plan).  Consider including advocacy as a crosscutting approach in the CSHGP technical guidelines, and developing an advocacy module to be included in the Techncial Resource Materials. The module could address, in a systematic and practical way, the role PVOs and CSPs can play at district, provincial, national levels, and sometimes regionally or internationally, to influence policy and allocation of resources for child health, and approaches to achieve these ends. World Vision 82 LCSP Final Evaluation e. Dissemination of Best Practices and Lessons Learned At the time of the Final Evaluation, WV Indonesia and WVUS had started or had identified the opportunity to document and disseminate the following aspects of the LCSP:  Chloroquine Efficacy Study: Given the potential value of this study at the national and international levels, a short version of the investigators report is under preparation for wider dissemination.  Posyandu Health Information System: The Posyandu Monthly Reports and other data available at the LCSP level at the end of the project were under review at the time of the FE. This data is of interest for drawing lessons on its potential use by Health Centers and the DHO and for contributing to the review and development of the Posyandu Health Information System at the national level.  Special analyses of three KPC surveys. The three KPC surveys of the LCSP offer the opportunity to analyze the determinants and trends in the coverage and impact of LCSP interventions. Findings of such study are of interest to the DHO and ADP for the continuation of LCSP activities, to WV Indonesia for integrating child survival into other ADPs, and to WVUS, other PVOs and USAID for the design of future child survival projects.  Promotion of Long-Lasting ITNs. Given the commitment of the ADP to continue this intervention and the interest at the national level for such field experience, a more complete description and dissemination of this experience is warranted. The FE Team otherwise recommends that WV Indonesia, with support from WVUS, compile and prepare technical descriptions of the LCSP training, BCC, and HIS materials, and prepare short best practices, lessons learned, and success stories. WV Indonesia, WV Asia and Pacific Region and WV US will be able to use these resources for integrating child survival into ADPs and for developing proposals or implementing other child survival projects. f. Potential for Scale-Up The LCSP successful results have the potential for scale-up and expansion at four levels: At ADP level: In the next 5 years, the ADP will focus on child survival, education and economic development. In terms of child survival, the ADP will continue some of the LCSP activities by supporting Posyandus and promoting the use of ITNs. The ADP has planned to extend these activities to two additional sub-districts to cover the entire ADP area and to become a learning site for other WV Indonesia ADP integrating child survival in their activities. At DHO and PHO levels: With adequate resources to provide training, continuing support and BCC materials to Posyandu cadres and transportation to Health Centers’ staff, the Landak DHO can extend some of the activities of the LCSP to other sub-districts than Mandor and Sengah Temila. With adequate documentation of the methods, tools and results of the LCSP, the Landak DHO can assist the Provincial Health Office in encouraging other Districts in West Kalimantan to learn and implement the best practices of the LCSP. World Vision 83 LCSP Final Evaluation At WV Indonesia level: The NO Health Unit and the ADP staff gained substantial experience in technical and management areas of child survival programs. With adequate support from the NO ADP manager, they can gradually assist all other ADPs in integrating child survival in their activities. At WV Asia and Pacific Region and WV US levels: With adequate documentation of the methods, tools and results of the LCSP, the WV Asia and Pacific Region and WV US levels can integrate the lessons learned and best practices from the LCSP into their other countries' projects and programs and share them with the international development community. World Vision 84 LCSP Final Evaluation ANNEXES World Vision 85 LCSP Final Evaluation Annex 1. Child Survival Grants Program Project Summary Final Evaluation Submission: Dec-30-2004 WV Indonesia Field Contact Information: First Name: Mary Esther Last Name: Wangsaraharja Indriani Address: Jl. Wahid Hasyim 33 Jakarta Pusat 10340 City: Jakarta State/Province: West Kalimantan Zip/Postal Code: Country Indonesia Indonesia Telephone: 011-62-21-31927467 Fax: E-mail: Mary_Wangsaraharja@wvi.org Project Web Site: Project Information: Project Description: The Goal of the LCSP was “to assist the Landak District Health Office to reduce mortality and morbidity and nurture growth and development (thriving) among infants, children and mothers through an integrated child survival and rural enterprise project in Mandor and Sengah Temila sub-districts.” The main strategies defined in the DIP were (in summary): 1. To improve the quality and coverage of existing child survival interventions; 2. To empower communities for prevention and control of common diseases; 3. To initiate micro-enterprise efforts and promote collaboration in health; 4. To develop a child survival intervention model for further replication and scale up. The LCSP Detailed Implementation Plan (DIP) focused on four technical interventions: immunization, vitamin A supplementation, control of diarrheal diseases, and control of malaria. Upon recommendations from the DIP review, the LCSP staff added the control of acute respiratory infections (ARI) as fifth intervention. Partners: Landak District Health Office West Kalimantan Provincial Health Offices The Pancur Kasih Foundation Project Location: Two sub districts (Mandor and Sengah Temila) of the Landak District, West Kalimantan Province Grant Funding Information: World Vision 86 LCSP Final Evaluation USAID Funding:(US $) $999,712 PVO match:(US $) $309,384 Target Beneficiaries: Type Number infants (0-11 months): 1,970 0-59 month old children: 10,225 Women 15-49: 16,645 Estimated Number of Births: 2,650 Beneficiary Residence: Urban/Peri-Urban % Rural % (No Data) 100% General Strategies Planned: Microenterprise Social Marketing Strengthen Decentralized Health System Information System Technologies M&E Assessment Strategies: KPC Survey Health Facility Assessment Organizational Capacity Assessment with Local Partners Participatory Rapid Appraisal Participatory Learning in Action Lot Quality Assurance Sampling Community-based Monitoring Techniques Participatory Evaluation Techniques (for mid-term or final evaluation) Behavior Change & Communication (BCC) Strategies: Social Marketing Support Groups Capacity Building Targets Planned: PVO Non-Govt Partners Other Private Sector Govt Community (None Selected) Local NGO Private Providers Dist. Health System Health Facility Staff Other National Ministry Health CBOs CHWs World Vision 87 LCSP Final Evaluation Interventions: Immunizations 30 % ** IMCI Integration ** CHW Training ** HF Training Vitamin A 10 % ** IMCI Integration ** CHW Training ** HF Training Acute Respiratory Infection 5 % ** IMCI Integration ** CHW Training ** HF Training Control of Diarrheal Diseases 20 % ** IMCI Integration ** CHW Training ** HF Training Malaria 35 % ** IMCI Integration ** CHW Training ** HF Training World Vision 88 LCSP Final Evaluation Indicator Numerator Denominator Estimated Percentage Confidence line Percentage of children age 0-23 months who are underweight (-2 SD from the median weight-for-age, according to the WHO/NCHS reference population) 0 0 0.0 0.0 Percentage of children age 0-23 months who were born at least 24 months after the previous surviving child 58 493 11.8 4.0 Percentage of children age 0-23 months whose births were attended by skilled health personnel 170 493 34.5 5.9 Percentage of mothers of children age 0-23 months who received at least two tetanus toxoid injections before the birth of their youngest child 355 416 85.3 4.8 Percentage of infants age 0-5 months who were exclusively breastfed in the last 24 hours 51 142 35.9 11.2 Percentage of infants age 6-9 months receiving breastmilk and complementary foods 3 90 3.3 5.2 Percentage of children age 12-23 months who are fully vaccinated (against the five vaccine-preventable diseases) before the first birthday 131 214 61.2 9.2 Percentage of children age 12-23 months who received a measles vaccine 164 214 76.6 8.0 Percentage of children age 0-23 months who slept under an insecticide-treated bednet the previous night (in malaria-risk areas only) 104 493 21.1 5.1 Percentage of mothers who know at least two signs of childhood illness that indicate the need for treatment 167 493 33.9 5.9 Percentage of sick children age 0-23 months who received increased fluids and continued feeding during an illness in the past two weeks 46 186 24.7 8.8 Percentage of mothers of children age 0-23 months who cite at least two known ways of reducing the risk of HIV infection 0 0 0.0 0.0 Percentage of mothers of children age 0-23 months who wash their hands with soap/ash before food preparation, before feeding children, after defecation, and after attending to a child who has defecated 21 493 4.3 2.5 World Vision 89 LCSP Final Evaluation Annex 2. Milestones of the Landak Child Survival Project Date Event May 20, 1999  USAID approved REACH (Rural Entrepreneurs and Advocates for Child Health) Project funding June 30, 1999  Dr. Chander (Regional Health Advisor) and Mrs. Mary Lengkong (WVI National Health Advisor) visited REACH project. July 15-20, 1999  REACH Technical Review meeting in Washington, from WVI: Mrs. Mary. Score of 83.1. Oct 4-8, 1999  Mr. Greg Kearns (WVUS PO for Indonesia), Mrs. Mary, Mr. Eddy Sianipar (GM Wahana Visi Indonesia), and Dr. Amsal (REACH Project Team Leader) went to Poso for REACH Project’s preparation Nov 1999  Recruitment of staff (midwives and nurse), Office set up, Qualitative data collection using PLA Dec 1999  Recruitment for Monev Officer (Ms. Elfrida) Jan 24-28, 2000  Dr. Chander and Mrs. Mary went to Poso for project backstopping and preparation for Baseline Survey Feb 14-19, 2000 Feb 20-24, 2000 Feb 25, 2000  KPC Survey Training followed by 40 participants  Mrs. Mary and Ms. Esther Indriani (PO for REACH) visited Poso to facilitate the Baseline Survey  Baseline Survey in 30 dusuns, 418 respondents  Presentation of preliminary baseline survey results to local government units in project areas  Recruitment of 6 midwives/paramedics, Training Officer (Dr. Ariston), and Finance Officer (Albert) March 4-10, 2000 March 28-29, 2000  Regional Quality Improvement Workshop in Dhaka, Bangladesh (attended by Dr. Amsal, Mangapul, Esther, and two local government partners i.e. Imam Subekti, MPH and Dr. Natsir Borman)  DIP workshop in Tentena, North Pamona. Attended by 140 participants, Dr. Fe Garcia, Greg Kearns, Rachel Brumbaugh, Dr. Chander, Mrs. Mary and Ms. Esther went to Poso April 16, 2000 April 30–May 12, 2000  Social riots in Poso. Staff evacuated to Palu.  DIP report writing in process  WVUS gave postponement for the DIP submission before on May 15 to July 15, 2000  Dr. Ariston joined the IMCI training in Makasar, organized by MOH RI. May 2000  Big social riot in Poso. Staff evacuated to Palu. June 2000  Staff evacuated to Jakarta  During the project suspension, the core team worked in other projects  Finishing of the REACH DIP Report. July 5, 2000 July 10, 2000 July 24-28, 2000  Mrs. Mary and Mr. Mangapul (ADP Poso Manager) went to Poso to do security assessment and feasibility to continue the REACH Project.  Management meeting on the situation of Poso. The conclusion was to move the REACH project to another needy area with WVUS ADP.  Mrs. Mary and Ms. Esther went for field assessment to Pontianak District as the proposed new project area.  Informed Greg Kearns and Dr. Chander about the latest situation Aug 2000  Administrative preparation for the change of project site (asking approval from the West Kalimantan Provincial Health Office), reported to Dr. Fe Garcia, Greg Kearns, Dr. Chander and Dr. Sri Durjati (USAID-Jakarta).  Preparation for Baseline Survey  Recruitment process for the new Team Leader, Monev Officer, Training Coordinator, and Health Motivators. All the REACH staff was absorbed to other projects except the Finance Officer. World Vision 90 LCSP Final Evaluation Date Event Sep2000 Sep18-22, 2000 Sep25-27, 2000  Change of project name from REACH to Landak Child Survival Project  Training on Baseline Survey, attended by 24 participants.  Baseline Survey in 30 sub-villages, 252 respondents.  Mrs. Laura Grosso (new WVUS PO for Indonesia), Ms. Anne Henderson, Dr. Chander, came to project site to backstop the DIP & Sustainability Workshop preparation.  Recruited Mr. Hendrik Rupang as Monev Officer.  Recruitment process for the Health Motivators; 8 candidates were involved in the Baseline Survey as recruitment process. Oct 3-4, 2000 Oct 25-28, 2000  DIP & Sustainability Workshop, attended by 120 participants.  Social unrest in Pontianak.  Finishing of the Landak CSP DIP Report. Nov 2000  Approval of Landak CSP DIP Report.  Recruitment process. Dec 8-19, 2000 Dec 13-20, 2000  Esther went to Pontianak/Landak to conduct the PLA and the 2nd HFA.  DIP Review in Washington. From WVI: Mrs. Mary, Mr. Eddy S, Mr. Sisbandi (ADP Pontianak Manager).  Recruitment process. Jan 2001  One staff from Landak District Health Office (the Head of CDC) joined training for microscopist in Jakarta. Feb 2001  Preparation for Cadres Training.  Due to difficulties in recruiting MPH, we recruited Dr. Ronald as Project Team Leader March 9, 2001  Meeting on LCSP Staffing, attended by Mr. Laura Grosso, Mary, Eddy S, Mr. Sisbandi, Dr. Andre Tanoe, MHP, Dr. Ronald Gunawan, Lucy (PO Wahana Visi), and Ms. Esther in National Office Jakarta.  In reference to the proposal/USAID guideline, the Team Leader should be a person with MPH degree, so we appointed Dr. Andre Tanoe as the LCSP Technical Team Leader, and Dr. Ronald with his agreement as the Training Coord.  Recruitment of 8 Health Motivators. April 10-11, 2001 April 16-21, 2001  First LCSP training. Two batches of TOT for Health Staffs conducted simultaneously. Revise the DIP Workplan by inserting ARI component.  Dr. Andre, Ms. Esther, and Mr. Albert joined the ADP Operation in Puncak. May 2001  Malaria Training, facilitated by Dr. Ferdinand Laihad (CDC MOH RI).  The core team had a two-day meeting in Ngabang to discuss DIP and Workplan. ARI Intervention was inserted in the Logframe. June 3-16, 2001 June 25-27, 2001  Inter-country IMCI Training in Semarang, attended by Mrs. Mary.  Mrs. Laura Grosso and Mary visited the project.  Sought information for Deltamethrine and Impregnated Bed Nets. July 22-28, 2001  IMCI Training in Surabaya, attended by Dr. Andre, Dr. Ronald and Ms. Esther.  Printed 10,000 MCH Handbook. Aug 6-10, 2001 Aug 29-31, 2001  Mrs. Mary and Ms. Esther visited the project for backstopping and preparation for FAR in Sep2001.  Dr. Chander and Ms. Esther conducted the second backstopping visit for FAR preparation.  10,000 MCH Handbook was given to West Kalimantan Province, and 3,000 of them were allocated to the Landak District.  Recruitment process for 2 more health motivators and 1 Admin Assistant. World Vision 91 LCSP Final Evaluation Date Event Sep4, 2001 Sep19-24, 2001 Sep25, 2001 Sep26, 2001  Mr. Sisbandi resigned from WVI.  First Annual Review of Landak CSP.  FAR Team Leader was Prof. Ascobat Gani, DrPH.  The FAR Members: Dr. Chander, Dr. Sri Durjati –USAID Jakarta, Dr. Ina Hernawati (Under five Children’s Health – MOH RI), Mrs. Mary, Ms. Esther, LCSP Core Team members.  Presentation at the Bupati office, and handed over 3 generators for all 3 Health Centers in LCSP areas.  Presentation at the Depkes RI Jakarta.  FAR Report writing. Oct 1, 2001 Oct 31, 2001  Dr. Ronald (Training Coord) resigned from Landak CSP, and currently working for the Relief Dept.  Dr. Andre resumed the Training Officer’s tasks.  Submission of FAR Report by Prof. Ascobat.  2 health motivators resigned (Novi wanted to have clinical work, and Titien wanted to get married). Nov 5-23, 2001 Nov 24, 2001  Regional Workshop on Monev in Reproductive Health, Bangkok. Attended by Dr. Mohammad Subuh (West Kalimantan Provincial Health Office), Hendrik and Esther.  Dr. Andre went to Cambodia for the “Regional Collaboration to Build Field Capacity to Conduct KPC Surveys” until 24 Dec 2001.  Purchased 2,000 Permanet ®, assisted by WVUS (Laura Grosso and Bob Wilson).  Recruited 3 more health motivators. Dec 12-16, 2001  Dr. Sri Chander and Ms. Esther Indriani went to Landak to follow up the First Annual Review Recommendations and for project backstopping. Jan 8 –12, 2002 Jan 25, 2002  Mrs. Mary Lengkong and Ms. Esther Indriani backstopped the LCSP and discussed about the MED plan, including plan to distribute ITNs  Collecting data on the Traditional Birth Attendants (TBAs) in the project areas (90 TBAs)  Finalizing the FAR Report.  Arrival of 2,000 ITNs in Pontianak Port Feb 2-8, 2002  Mrs. Mary Lengkong and Mr. Untung Sidupa (ADP Pontianak Manager) joined the Behavior Change Communication (BCC) Workshop in Johannesburg, South Africa, organized by CORE and CSTS – both paid by the LCSP  Arrangement of Tax Exemption Letter from the Ministry of Social Affairs for the 2,000 ITNs.  LCSP and ADP Pontianak conducted the SURPINAS (National Ministry Info System from WVI)  FAR Report was sent to MOH RI Mar 21, 2002  Mrs. Mary Lengkong and Ms. Esther Indriani joined the meeting in USAID with Ms. Lynn Krueger (Deputy Team Leader for USAID Office of Health, Population and Nutrition) and Dr. Sri Durjati (Reproductive and Child Health Programme)  Continued process to release 2,000 ITNs from the Pontianak port. Difficulties occurred during the document arrangement because the Pontianak Tax Dept never managed Tax-Free imported goods before April 4, 2002 April 29-May 4  The project were able to release the 2,000 ITNs from the Pontianak Port  Mrs. Mary Lengkong and Ms. Esther Indriani backstopped the LCSP  Meeting with Dr. Sri Durjati (USAID Jakarta)  Recruitment process for LCSP Training Officer  Purchased 100 TBA Kit and 15 Vaccine Flasks. Produced 1,000 Cadres T-shirt promoting Exclusive Breastfeeding and 200 TBA T-shirt promoting Vitamin A Capsule Distribution May 28, 2002  Launching of the ITNs in Landak District. Attended by the District Health Office Chief, community leaders, and KSM (Self Help Groups) representatives in ADP Pontianak’s area, and the representative of Permanet® Jakarta, Mr. Aidil Zarkasih.  Recruitment and placement of Ms. Marianni Aritonang  Recruitment process for LCSP Project Officer as Ms. Esther Indriani would take her MPH Course in the Netherlands June 2002  Preparation for MTE (TOR, consultant, etc)  Recruitment of Dr. Fransiska Kaligis (PO LCSP) World Vision 92 LCSP Final Evaluation Date Event July 2002  KPC Survey Training (16-20 July ’02) and KPC Survey Data Collection (22-25 July ’02).  MTE of Landak CSP (30 July – 8 Aug ’02) with the External Team Leader of MTE: Dr. Marc Debay, PhD (John Hopkins University), and the external team members: Dr. Fe Garcia (WVUS), Dr. Sri Chander (WV APRO), Dr. Sri Durjati (USAID Jakarta), Dr. Ferdinand Laihad (Malaria sub￾district MOH RI).  Dr. Fransiska resigned due to her desire to take specialist course on medical science August 2002  Presentation of the MTE at the District level (6 Aug’02), Provincial level (7 Aug ’02), and at the National level (8 Aug ’02).  Dr. Lady Love (Doctor of Pahauman Health Center) and Mr. Untung Sidupa (ADP Pontianak Manager) participated in the Kean Svay CSP Final Evaluation (28 Aug – 6 Sept ’02) September 2002  Recruited Ms. Rainy Wijaya, Apt as the LCSP’s Monev Officer . She received orientation meeting at the WVI Natl Office for one week.  Dr Andre Tanoe (Tech Team Leader), Mariani (LCSP Training Coord), and Mr Markus Akim (Community Development Coordinator) participated in the Final Evaluation of the BRICS CSP of World Vision India October 2002  Tech Team Leader joined coordination meeting at MOH RI Jakarta  The Monev officer received one month EPI 2000 Training by Dr Ronald Gunawan (at the National Office) prior to her placement in the field.  Continue recruitment process for LCSP Project Officer  Development of the IMCI Calendar 2003 November 2002  Produced 2,000 pcs of IMCI Calendar for year 2003, sent to and distributed in the Landak CSP areas, as well as 2 pcs for each ADP in Indonesia  On 14 – 16 Nov ’02, Mrs. Mary Lengkong (National Health Advisor) fulfilled this invitation from WV East Timor to be consultant in Child Survival and IMCI in WVET’s CSP. December 2002  On 18 – 21 Nov’02, Mrs. Mary Lengkong backstopped LCSP January 2003  Recruited Project Officer for LCSP, Ms. Rustini Floranita, SKM (13 Jan ’03). She received training on Positive Deviance by Save The Children in Jakarta. February 2003  Started the TBA selection for “TBA Kit”. After the TBAs received 4 times training, they had to undergo test to be eligible to receive the TBA kit.  The project carried out health educations for community members and high school students. March 2003  Mrs. Mary Lengkong and Dr Sri Chander (Regional Health Advisor) backstopped the LCSP on March 4 –8 ’03.  The Project Officer went to LCSP for field orientation on 4 – 27 March ’03. April 2003  Mrs. Mary Lengkong and Dr Ronal S Gunawan provided technical assistance for WVET’s Child Survival Project May 2003  In collaboration with the Provincial and District Health Offices, the project conducted the Chloroquine Efficacy Study (for Malaria) in the Landak District. The study was preceded by training of malaria microscopist analysts. The study was conducted on May 26 – July 22, 2003. June 2003  On 23 June 2003, attended meeting on MCH Handbook at the MOH RI. July 2003  Chloroquine Efficacy Study (for Malaria) in the Landak District. August 2003  20 August ’03, World Vision Indonesia received an award from MOH RI for its contribution in Malaria Prevention through its various projects in Indonesia, including Landak CSP. Mr James L Tumbuan (WVI’s National Director) received the award, and during this ceremony we distributed 9,000 pcs Malaria leaflet. World Vision 93 LCSP Final Evaluation Date Event September 2003  Mrs. Mary Lengkong and Dr Sri Chander backstopped the LCSP on 1 – 5 Sept ‘03  Conducted workshop on the results of Choloroquin Efficacy Study in the Landak District on 5 Sept ’03.  Esther Indriani joined the Landak CSP again after finishing her study, and since 23 Sept ’03 she went to Landak CSP for project backstopping and preparation for the Third Annual Review of Landak CSP.  Conducted the Third Annual Review on 29 Sept ’03 – 6 Oct ’03. The Evaluation external team leader was Dr Marc Debay, and joined by Dr Alex Papilaya, DTPH and staff of WV East Timor’s CSP (Ms. Joanne Chia – CSP Team Leader and Mr. Alberto Araujo – Training Coord). October 2003  Conducted Community Based Health Information System Workshop in Pontianak on 7 – 10 Oct ’03 for health staff from the Landak District and the whole West Kalimantan province.  Esther Indriani backstopped the project until 19 Oct ‘03 November 2003  Per November 2003, Dr Andre Tanoe (Tech Team Leader LCSP) resigned from the project due to family reason.  Esther Indriani was the acting for Tech Team Leader, and she went to Landak CSP on 6 – 15 Nov ’03. December 2003  Esther Indriani participated in the Kean Svay CSP’s Final Evaluation in Cambodia January 2004  Recruited Dr Teresa Zakaria as the new Tech Team Leader on 7 Jan ’04 and since 12 Jan ’04 she was placed in Landak CSP February 2004  Dr Teresa and Esther I participated in the PCI’s Sustainability Action Plan Workshop in Pandeglang on 24 – 26 Feb ’04. The facilitator for this SAP workshop was Dr Michel Pacque of CSTS – USA. March 2004  On 1 March ’04, Dr Teresa and Esther I had a meeting with Dr Ina Hernawati (Chief of the Under five children’s Health subdirectorate) to discuss about the IMCI TOT in West Kalimantan and asked for IMCI Modules and flipchart for training.  Conducted “Malaria and ITN Month” to promote the ITNs in project areas. The project purchased another batch of 2,000 ITNs and distributed discount vouchers for pregnant mothers and mothers of under five children. This promotion was a success because 1,934 ITNs were sold in this month only. 37 more were sold in the month of April.  Mr Lyndon Brown (Child health Advisor from WVUS) visited the project for technical backstopping on 15 – 28 March ’04. During his visit, the project carried out the Sustainability Action Plan Workshop, BCC exercise and the Doer / Non Doer Workshop as practice for ADP Pontianak/LCSP staff.  On 22 – 27 March ’04, Esther I went to project to assist the project in carrying out the Sustainabilty Action Plan workshop and the Behavior Change Communication (BCC exercise). April 2004  Conducted BCC exercise feedback for the Doer /Non Doer Study on ITN use in the project areas. May 2004  Mrs Mary Lengkong and Esther I participated in the Global Health Workshop in Washington DC and the Child Survival Mini University in Johns Hopkins University on 30 May – 12 June ’04.  In May ’04, started scale up activities in Toho and Takong subdistrict of Pontianak District (the other areas of the ADP Pontianak). The project conducted TBA trainings in Toho and Takong as well as cadres meeting/training. June 2004  The project carried out “ARI Billboard Competition” for the Posyandus in the project areas. All of the 108 Posyandus joined the competition. The assessment of the Billboards was done on 15 June ’04 for Mandor health Center’s area, on 22 June ’04 for Pahauman HC’s area, and on 25 June ’04 for Senakin HC’s area. The judges for the competition came from the HC staff, LCSP staff, and the subdistrict government staff. As the winners for Mandor were: 1st Atong, 2nd Kayu Tanam, and 3rd Sebadu. For Senakin: 1st Pak Tinjun, 2nd Serimbang, and 3rd Runut. For Pahauman: 1st Tumahe, 2nd Sanyang, and 3rd Keranji Birah.  Esther I and Dr Sri Chander backstopped the LCSP to prepare the Final Evaluation and planning for 3 months (July – Sept 2004) on 23 June – 3 July ’04. World Vision 94 LCSP Final Evaluation Date Event July 2004  Mr Wynn Flaten (Senior Operations Manager WVI) visited the project on 13 July ’04, and he had the chance to see one of the Posyandu, i.e. in Sebatih.  The project conducted several nutrition trainings for mothers with underweight children in Senakin area.  Esther I backstopped the Landak CSP on 20 – 30 July ’04.  Conducted Behavior Change Communication Training on 26 – 29 July 2004, facilitated by Mr Anton Schneider (Academy for Educational Development USA). As many as 35 participants joined this training, and they came from various backgrounds e.g. 14 LCSP staff, 3 ADP Pontianak staff, 4 staff from other ADPs in West Kalimantan region, 5 Health Center staff, 5 NGO staff, 3 University Lecturers, and 1 Catholic church pastoral staff. Dr. Sri Chander, Mrs. Mary Lengkong and Esther I participated in this training. August 2004  Conducted Outdoor Management Training in Sanggau for ADP Pontianak / LCSP staff on 4 – 8 Aug ’04. The consultant for this training was X-Care Consultainment. The total number of participants was 15 ADP staff and 14 LCSP staff.  Dr Marc Debay (consultant for LCSP) came to the project since 21 Aug ’04 to assist with project documentation and the KPC Survey.  Conducted the KPC Survey Training on 23 – 28 Aug ’04. As many as 30 participants joined the training. We recruited community members and college students as supervisors and interviewers for this survey.  The Final KPC Survey was caried out in two sub-districts of project areas – Mandor and Sengah Temila. As many as 494 respondents were interviewed in 45 clusters. Mrs. Mary Lengkong, Dr Teresa, Esther I and Mariani Aritonang supervised the implementation of the survey.  Esther I bacstopped the LCSP since 3 Aug ’04 up to now. September 2004  Final Evaluation of the Landak CSP (21 – 29 September 2004)  Lessons Learned Workshop of Landak CSP (30 September 2004) World Vision 95 LCSP Final Evaluation Annex 3. Timeline of Overall Project Implementation, July 2000 – September 2004 Management BL / Midterm / Final KPC xx x DIP / FAR / MTE / FE xxx x x Tech/Adm support, field visits WV US x xx x xxx WV Asia Pacific xx x x x x x WV Indonesia x x x x xx x x x x xxx x x xx Consultant x x x xxx Training TOT for health staff x Posyandu cadres xxxx x xxxx x x x x x Shopkeepers x x xx x xx x TBAs xxxxx xx xxx xxx xx Health Center staff (IMCI) xx x x Others ITN distribution: thru KSM xxxxxxxxxxxxxxxxxxxxxxxxxxxxx + Posyandus xxxxxxx BCC campaigns: malaria/ARI x xx CQ efficacy study xxx FY 2003 FY 2004 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q4 FY 2001 FY 2002 Q1 FY00 Q4 Q2 Q3 Q4 Q1 Q2 Q3 See Annex 37 for Human Resources timeline and Annex 39 for Technical Assitance timeline. World Vision 96 LCSP Final Evaluation Annex 4. Maps of Landak District and ADP Pontianak PONTIANAK CITY SUNGAI PINYUH MEMPAWAH Anjungan Sangking Untang Pentek Sekabuk KualaToho Toho Hilir TOHO Pak Utan Sepang Benuang Sambora MANDOR Sebadu Aur Gombang Pak Kumbang Tampi Semenok Pate Pelanjau Bingge Saham Ubah PAHAUMAN Sidas Pak Banying Lintah Kalawit Ke Menjalin Kayu tanam Air Merah Dema Kepayang Ngarak Lanso Kecurit Terap u Air Pongok Pongok Angkabang Kerohok Sekilap Manggang Agak hilir Sumsum Samangkiling Agak hulu Salatiga Pak Laheng Sui Pinggan Satak Bumbutn Amawakng Kr. Paidang Kr. Mancal Paloatn Andeng World Vision 97 LCSP Final Evaluation Annex 5. Demographic and Health Services Profile of Project Area MANDOR SENGAH TEMILA TOTAL Senakin Pahauman Population Total * 25.669 20.366 29.642 75.677 Households * 5.252 3.791 5.492 14.535 Pregnant mothers ** 781 555 851 2.187 Infant (0 - 11 months) ** 692 504 774 1.970 Child (1 - 4 years) ** 2.897 2.115 3.243 8.255 Health workers *** Trained cadres (Posyandu cadres) 213 204 286 703 Trained TBAs 85 110 110 305 Trained Shopkeepers 126 96 87 309 Village Drug Posts: 14 Administrative units * Dusun / Sub village 57 29 45 131 Desa / Village 17 5 9 31 Health Facilities *** Posyandu / Village Integrated Post 32 31 45 108 Polindes : Total 17 5 9 31 - Midwives 10 1 5 16 + Midwives 7 4 4 15 Pustu / Sub health centers 7 3 3 13 Puskesmas / Health centers 1 1 1 3 Sources: * Project field data, mostly from 2004 regular census for Public Election Day called P4B ** Landak DHO, September 2004 *** LCSP data, September 2004 World Vision 98 LCSP Final Evaluation Annex 6. LCSP FY 2004 Work Plan and Achievements Legend: PLANNED: X DONE: ONGOING: CANCELLED: No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Technical Malaria 1 Strategic planning exercise should be conducted to have community actions X X X LCSP and Pontianak ADP conducted several community meetings on malaria management, especially as follow up on chloroquine Efficacy Study 2 Buy Permanet XL size X Another batch of 2000 ITNs were bought in December 2003 3 Develop & distribute Promotion materials for ITN (season) X X Intensive ITN promotion was done during the Malaria Month Campaign 4 Promote ITNs X X X X X X X X X The project gave 45% discount for ITN using discount vouchers 5 Deltamethrine tablets distributed X X X X X X X X X The DHO has budgeted Deltamethrine tablets distribution 6 Malaria & illegal gold mining movie for HE completed X The movie was given to the Pontianak ADP and shown at several occasions, still ongoing 7 Promoting malaria messages to cadres, POD, TBA, communities X X X X Done in February for PODs and during Malaria Month Campaign for community World Vision 99 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 8 Seminar on Malaria treatment for health centers staff (Artemisinin) X Socialization of Artesdiaquine at the Bupati Convention Room for HC staff and District officials 9 Obtain agreement with DHO/PHO on number of microscopes needed X The DHO has budgeted 13 microscopes and the project is requested to provide one more 10 Provide new microscopes X Submitted to DHO during socialization of Artesdiaquine 11 Doer - Non Doer study X X X X Done as follow up to Malaria Month Campaign. Behavior chosen was pregnant women and mothers of under 2 sleeping under ITN every night. 12 Confirmation from dr Laihad about Artemisinin study and implementation X X X X Done by the MOH without the involvement of the project. 13 Advocacy activities on Malaria at the District, Provincial and National level X X X Done through 3 major activities: Malaria Month Campaign, Artesdiaquine Socialization, one seminar on environmental health held by Pontianak ADP and Pancur Kasih in August Immunization 14 Facilitate opening 7 Posyandus X X 5 Posyandus opened: Sidik, Pampang in Sengah Temila; Penawar, Semanyam't, Mianas in Mandor 15 Redesign supervision tools (OC, EI, PS) X Special OC and EI tools were made about Malaria, ARI and Diarrhea. The Posyandu Scoring System was simplified. 16 Develop guide book for TBA training X The project still uses the guidebook adapted from MOH World Vision 100 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 17 HE session to KSMs & CU X X X The project only managed to conduct a TOT on HE for Pontianak ADP Staff. 18 Assessment of universal precautions procedures in Health Centers. X Needle crushers provided by the DHO 19 Actions for universal precautions X Overtaken by the DHO Diarhea 20 Monitor availability of ORS in Posyandu level X X X New Posyandu monthly report form includes ORS stock table 21 Training needs assessment for dietary management and counselling skills X X Considering the limited time, activities concerning diarrhea was performed mainly using the IEC tools available. 22 Train cadre on dietary management for diarrhea and counselling skills X No special topics on diarrhea management 23 IMCI TOT for health staffs X Training done at provincial level on March 8-13, 2004 24 IMCI training for health staffs X 2 classes of 7 participants each, coming from 13 health centers in Landak District and one LCSP staff 25 Promotion of diarrhea prevention messages X X Especially done in this month considering the increased risk during the dry season Pneumonia 26 Assessment of needs & methods to teach TBAs the danger signs of Pneumonia and referral X After the 4 series regular training for TBAs, the TBAs assistance was overtaken by the health centers midwives 27 Develop curriculum for TBAs X Project still uses the original curriculum from the MOH World Vision 101 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 28 Train TBAs X X X No new training conducted, only refresher trainings 29 Develop additional training materials on ARI including IMCI movie and poster for Posyandu and POD cadres X Done during the ARI month in May-June 2004 30 Train Posyandu cadres and POD cadres X X Refresher training for Posyandu Cadres during quarterly meetings Vitamin A 31 Promote vitamin A messages to mobilize people X X Rigorous promotion was done one month before and during Vitamin A months in February and August by cadres, health staff and project motivators 32 Facilitate distribution of Vitamin A to postpartum mothers X X X X X X X X X X X X Done through Posyandu cadres and TBAs monthly meetings Cross-cutting 33 Visioning exercise X X X Activity overtaken by Pontianak ADP 34 Communication & advocacy skills training for LCSP - ADP staff X Conducted in the form of outdoor management training 35 Developed & completed sustainability plan & Exit strategy X X X Sustainability Action Plan Workshop was done in March and a follow up meeting in July 36 BCC exercise for Malaria with LCSP - ADP staffs, health staffs, the community, local NGO X X Done during the doer-nondoer analysis in April discussed in April 29, and during the BEHAVE Training in July World Vision 102 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 37 Developing Malaria BCC strategy with ADP X During BEHAVE training and Environmental Health seminar 38 Implementing BCC strategy for Malaria X Implementation of discount vouchers in March, not as follow up to BEHAVE Training 39 Monthly TBAs meeting X X X X X X X X X X X X Done by HC midwives 40 Quarterly Posyandu Cadres meeting X X X X Done in each health center area, with specific refresher topics 41 Advocate Pancur Kasih to conduct HE at the CU's members regular meeting X X X Initial discussion was done without further follow up yet. 42 Facilitate CUs to conduct HE for their members X X X X 43 Train KSMs to deliver HE for their members X Pontianak ADP intends to form a health section in each KSM. 44 Continue develop HE movies X ARI movie developed 45 Show the HE movies to the community X X X X X ARI movie shown Capacity building 46 Conduct Organizational Capacity Assessment for KSMs, Posyandus, HCs X X Done by the LCSP- ADP staff for 3 Cadre Association and 3 health centers 47 Close supervision for POD X X X X Supervision done for Agak Hilir & Belak PODs in December as tool for February POD meeting 48 Community meeting about POD X X X X Done in Mandor and Senakin 49 Formal meeting with DHO X X X X Done at various occasions, especially concerning trainings/meetings preparation World Vision 103 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 50 Plan & conduct Lessons Learned Workshop with Cambodia X X X Project officer participated in Lessons Learned Workshop in Cambodia in December 2003 51 Develop and implement supervision tools for POD & Posyandu cadres X X X X Not feasible 52 Develop tools to assess training results directly for TBA, POD, Cadre X Not feasible 53 Supervise and assess POD, TBA, Posyandu cadre X X X X Supervision of Posyandu Cadres done by the project's motivators; POD by HC. 54 Assess the knowledge & skills of shopkeepers X X X Not feasible 55 Retrain Shopkeepers on protocol and danger signs of key childhood diseases X X 3 trainings for new shopkeepers done in Pahauman. No refresher training conducted Sustainability 56 Advocacy to DHO & Health centers to provide optimum health services to the community X X X X X X X X X X X X Project participated in several monthly meetings in HCs; Problems in the field also communicated to DHO 57 Devide 'Posyandu - KSM' for MED program to 4 different KSMs X So, far only one KSM in Singkut Durian. Pontianak ADP has plans to establish others. Program Management 58 Establish Cadre Union in Senakin Area X Done on December 11, 2003 59 Provide CS training for Development motivators and vice versa X Done during the outdoor management training World Vision 104 LCSP Final Evaluation No Narrative FY 2004 Work Plan Progress (Sep 04) Comments 2003 2004 Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep 60 Team building activities (Outbound activities) X Done with participation of project and ADP staff 61 Develop logistics forms for Administrative Officer to submitt to FO X X X X The new form is developed but not finalized yet. 62 Provide MCH Booklet for each Cadre X Done during Posyandu Cadres Quarterly meetings 63 Test and Implement selected Community Based HIS Tools X CBHIS tool accepted by DHO is Posyandu monthly report. Revision done and the new form used since April 2004 World Vision 105 LCSP Final Evaluation Annex 7. LCSP FE KPC Results and Targets for 14 LCSP indicators Intervention # Indicator Final Evaluation Results LCSP Target % LCSP Target Numerator Denominator met * LCL % UCL % Estimate % EPI 1 % of children 12-23 months completely immunized (verified by card) 131 214 52.8 69.7 61.2 80 No 2 % of mothers with child less than 2 years of age received TT2 before the birth of their youngest child (verified by card) 217 493 36.7 51.3 44.0 50 Yes Vitamin A 3 % of children 12-23 months received Vitamin A in the past 6 months 186 211 82.9 93.4 88.2 90 Yes 4 % of mother received Vitamin A within one month of their last delivery 204 493 34.5 48.3 41.4 50 No Malaria 5 % of mothers with child less than 2 years of age who was ill with fever during the past two weeks seek treatment for their child 89 165 45.0 62.9 53.9 75 No 6 % of children less than 2 years of age with a febrile episode that ended during the last two weeks were brought to a health facility within 48 hours after the fever began 54 114 37.1 57.7 47.4 45 Yes 7 % of mothers with a child less than 2 years of age report that they have an ITN in their house 111 492 16.5 28.6 22.6 30 No 8 % of children less than 2 years of age slept under an ITN the previous night 104 493 15.3 26.9 21.1 20 Yes World Vision 106 LCSP Final Evaluation Intervention # Indicator Final Evaluation Results LCSP Target % LCSP Target Numerator Denominator met * LCL % UCL % Estimate % Diarrhea 9 % of children less than 2 years of age who had diarrhea in the last two weeks received ORS 91 188 40.4 56.4 48.4 75 No 10 % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more fluids 146 188 71.3 84.1 77.7 75 Yes 11 % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more foods 72 186 32.0 45.4 38.7 70 No 12* % of children less than 2 years of age who had diarrhea in the last two weeks received antidiarrheal medicine (antibiotics, traditional medicines, or injection) 93 188 41.9 57.1 49.5 10 No ARI 13 % of mother with children less than 2 years of age who mention at least 1 symptom of pneumonia 273 493 49.2 61.5 55.4 80 No 14 % of mother with children less than 2 years of age who had symptoms of pneumonia (cough and rapid or difficult breathing) seek treatment 99 138 62.1 81.3 71.7 75 Yes Source: LCSP Final Evaluation KPC survey report Note: A LCSP Target is met (“Yes”) when 95% Confidence Interval around the FE survey Estimate includes target value. LCL: Lower Confidence Limit; UCL: Upper Confidence Limit. World Vision 107 LCSP Final Evaluation LCSP Targets and Final Evaluation Results for 14 Key Indicators 0 20 40 60 80 100 1 2 3 4 5 6 7 8 9 10 11 12* 13 14 EPI Vitamin A Malaria Diarrhea ARI Target FE Estimate (95% Confidence Interval) Source: See Table above. World Vision 108 LCSP Final Evaluation Annex 8. LCSP Progress Towards Targets for 14 LCSP Indicators Intervention # Indicator Baseline MTE FE LCSP Target Sept '00 July '02 Sept '04 Target met* EPI 1 % of children 12-23 months completely immunized (verified by card) 23.0% 46.8% 61.2% 80.0% No 2 % of mothers with child less than 2 years of age received TT2 before the birth of their youngest child (verified by card) 2.0% 14.9% 44.0% 50.0% Yes Vitamin A 3 % of children 12-23 months received Vitamin A in the past 6 months 54.0% 60.2% 88.2% 90.0% Yes 4 % of mother received Vitamin A within one month of their last delivery 12.0% 25.6% 41.4% 50.0% No Malaria 5 % of mothers with child less than 2 years of age who was ill with fever during the past two weeks seek treatment for their child 26.0% 57.8% 53.9% 75.0% No 6 % of children less than 2 years of age with a febrile episode that ended during the last two weeks were brought to a health facility within 48 hours after the fever began not available 29.9% 47.8% 45.0% Yes 7 % of mothers with a child less than 2 years of age report that they have an ITN in their house not available 4.5% 22.6% 30.0% No 8 % of children less than 2 years of age slept under an ITN the previous night not available 1.6% 21.1% 20.0% Yes World Vision 109 LCSP Final Evaluation Intervention # Indicator Baseline MTE FE LCSP Target Sept '00 July '02 Sept '04 Target met* Diarrhea 9 % of children less than 2 years of age who had diarrhea in the last two weeks received ORS 31.5% 42.9% 51.6% 75.0% No 10 % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more fluids 63.0% 58.6% 77.7% 75.0% Yes 11 % of children less than 2 years of age who had diarrhea in the last two weeks received same amount or more foods 34.0% 29.0% 38.7% 70.0% No 12 % of children less than 2 years of age who had diarrhea in the last two weeks received antidiarrheal medicine (antibiotics, traditional medicines, or injection) 40.0% 20.2% 49.5% 10.0% No ARI 13 % of mother with children less than 2 years of age who mention at least 1 symptom of pneumonia 24.6% 39.9% 55.4% 80.0% No 14 % of mother with children less than 2 years of age who had symptoms of pneumonia (cough and rapid or difficult breathing) seek treatment 58.8% 58.7% 71.7% 75.0% Yes Source: Baseline and MTE data from MTE report; FE data from Final Evaluation KPC survey report. Note:  A LCSP Target is met (“Yes”) when 95% Confidence Interval around the FE survey Estimate includes target value (see Annex 7).  # 8: MTE data does not include "the previous night."  # 12: Antidiarrheal medicines only include "pill or syrup." Yes = 6 No = 8 World Vision 110 LCSP Final Evaluation LCSP PROGRESS TOWARDS TARGETS FOR 14 KEY INDICATORS (DRAFT) 0% 20% 40% 60% 80% 100% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 EPI Vit. A Malaria Diarrhea ARI Baseline MTE FE Target Source: See Table above. World Vision 111 LCSP Final Evaluation Annex 9. Terms of Reference for the Final Evaluation General Objective: To conduct a final evaluation activity of the Landak Child Survival Project (LCSP) and make recommendations on future child survival activities and capacity building needs in Landak District. The Purpose of the Review: The purpose of the Final Evaluation is to provide an opportunity for all program stakeholders to assess accomplishments to date and to listen to the beneficiaries at all levels; including mothers, other community members and opinion leaders, health workers, health system administrators, local partners, other organizations and donors. The final evaluation includes comparison of baseline and final data, elaborates on the lessons learned from the model or implementation approach, and identifies promising practices and opportunities for scaling up and/or replicating the approach within a broader context. The final evaluation provides an additional opportunity for the program to benefit from the outside viewpoint of a consultant who acts as facilitator of the evaluation process. Other PVOs and resource persons will be invited to participate in the evaluation process. Specific Objectives: The Final Evaluation is focused on: 1. Assessing if the program met the stated goals and objectives; 2. The effectiveness of the technical approach; 3. Development of the overarching lessons learned from the project; 4. A strategy for use or communication of these lessons both within the organization and to partners. 5. Communicate key review findings, conclusions and recommendations of the review to clients, and document them in the form of a Final Evaluation Report, which should include but is not limited to the followings:  Summary including brief description of the program and its objectives, main accomplishments, highlights from the comparison of the baseline and final evaluation surveys, and list of priority conclusions resulting from this evaluation  Assessment of Results and Impact of the Program, from the view of Technical approach, Crosscutting approaches, Capacity Building approach and Sustainability Strategy.  Program Management, including Program planning, supervision of program staff, human resource and staff management, logistics, information management, technical and administrative support, and Management lessons learned.  Conclusions and Recommendations,  One page results highlight Evaluation Methodology: The Terms of Reference proposes a review strategy that fulfills the criteria established by the USAID Child Survival Final Evaluation Guidelines. The review methodology will include the following: Evaluation Team Leader The external team leader, Dr. Alex Papilaya, DTPH (Public Health Faculty- University of Indonesia) and Dr. Marc Jean-Paul Debay, PhD, MPH as external Co-Team Leader will facilitate the review activities in a participatory manner and ensure that the review process is conducted according to USAID standards. Data Collection and Analysis: The review team leader will be responsible for overall methodology and design of the data collection techniques, facilitating the analysis of the data, and providing an assessment of the quality of project implementation based on this data. The data collection technique may include:  An internal review based on data generated by the Health and Management Information System  Field visits/observation  Focus group discussion; and key informant interviews with stakeholders  Review of project documents World Vision 112 LCSP Final Evaluation  Others as required by the review team Proposed Review Schedule Sept 21 (Tuesday)  Arrival of all review team members in Pontianak  Meeting with the Chief and officers of West Kalimantan Provincial Health Office.  Depart to Sengah Temila sub-district (one of project areas)  Preliminary discussion and tools development  Over night in Ngabang Sept 22(Wednesday)  Meeting the District Head/Bupati and District Health Chief  Technical overview and Programs in brief Sept 23 (Thursday)  Presentation on the Final KPC Survey results to 31 Village Chiefs in Pastoran Pahauman to get their inputs and obtain their interests in the health field. To be delivered in local language and translated into English. Sept 24 – 25 (Friday - Saturday)  FIELD VISITS – District Health Office and Health Centers, Integrated Service Post (Posyandu), Posyandu Cadres Association, community and partner NGOs.  Review project records and files, staff interviews.  Discuss principal findings and recommendations.  Preparing Final Evaluation report draft Sept 26 (Sunday)  Half day of rest and church  Preparing Final Evaluation report draft Sept 27 (Monday)  Preparing Final Evaluation report draft Sept 28 (Tuesday)  Presentation and Debriefing at the LANDAK District Health Office in Ngabang, the district’s capital  Stay over night in Pontianak Sept 29 (Tuesday)  Presentation and Debriefing at the Provincial Health Office in Pontianak, the province’s capital  Final Evaluation Team depart to Jakarta Sept 30 (Thursday)  Lessons Learned Workshop in Jakarta  Wrap up and exit meeting of Final Evaluation Team World Vision 113 LCSP Final Evaluation Team Composition Review Team Leader:  Dr. Alex Papilaya, DTPH (Public Health Faculty- University of Indonesia) (team leader)  Dr. Marc Jean-Paul Debay, PhD, MPH, independent consultant, as co-team leader Coordinators:  Mr. Edi Sianipar (General Manager, Yayasan Wahana Visi Indonesia)  Mrs. Mary Lengkong, DDS, DDPH (National Health Advisor WV Indonesia)  Ms. Esther Indriani, MPH (Project Officer Landak CSP, World Vision Indonesia)  Drs. Untung Sidupa (ADP Manager Pontianak/Landak)  Dr. Teresa Zakaria (Technical Team Leader Landak CSP) Team Members: From USAID Local Mission:  Dr. Sri Durjati Boediharjo, MSc, DPH (Reproductive & Child Health Program, PHN)* From MOH RI:  Dr. Sri Hermianti, MSc (Director of the Family Health Directorate, MOH RI)*  Dr. Ina Hernawati, MPH (Chief of the Infant & Preschool Children’s Health Sub Directorate, MOH RI)  Dr. Rosmini Day, MPH (Director of the CDC Directorate, MOH RI)*  Dr. Ferdinand J. Laihad, DMM, MPHM (Chief of the Malaria Sub directorate, MOH RI)  Dr. Fahani (Malaria Sub directorate, MOH RI). From the West Kalimantan Provincial Health Office:  Drg. Oscar Primadi, MPH (Chief of the West Kalimantan Provincial Health Office-PHO)*  Drg Junardi, MKes (Chief of the Health Services Division, West Kalimantan PHO)*  Dr. Isman Ramadi, MMed.OM (Chief of CDC Division, West Kalimantan PHO)* From the Landak District Health Office:  Dr. Abang Chaerudin (Chief of the Landak District Health Office)  Mrs. Sri Wahyuni, SKM (Chief of the CDC& EH section)  Mrs. Sophia Tjakre (Chief of the MCH section)  Chief/staff of the Health Centers in Mandor, Pahauman and Senakin Areas From World Vision Intl’ / Yayasan Wahana Visi Indonesia:  Mr. Lyndon Brown, MPH (Child Health Advisor, World Vision US)  Mr. Perry Mansfield (Program Officer, World Vision US)*  Dr. Sri Chander (Asia Pacific Regional Health Officer)  Mr. Wynn Flaten (Senior Operations Manager, World Vision Indonesia)  Mr. Keo Sereivuth, Monitoring and Evaluation Officer, World Vision Cambodia  Dr Te Chilay, MOH Cambodia  Team LCSP – ADP Pontianak  and, community representatives and stakeholders from the Mandor and Sengah Temila sub-districts and the Landak District. Note: *) the above named person could not come to the Final Evaluation Expected Outcome Dr. Alex Papilaya, DTPH and Dr. Marc Jean-Paul Debay, PhD, MPH, the external review consultants, will be responsible for preparing the final report which must meet all the requirements outlined in the Final Evaluation guideline. A draft evaluation report will be completed and presented at the conclusion of the review visit at the District, Provincial, and National levels. Following the visit, the consultant will edit and refine the draft document into the final form. It will be the responsibility of the consultant to forward the final draft to the country office (Mr. James L Tumbuan / Mrs. Mary Wangsaraharja), the regional office (Dr. Sri Chander) and the WVUS office (Mr. Lyndon Brown / Mr. Perry Mansfield) for approval and comments. It is essential that the Final Evaluation Report is received by all offices no later than November 30, 2004. World Vision 114 LCSP Final Evaluation Annex 10. FE Assessment Methodology and Schedule The FE Team comprised the LCSP staff members, other ADP staff members, representatives of the MOH RI, World Vision Indonesia, APRO and US, the two external evaluators. Prior to the FE, the LCSP project conducted a Knowledge, Practice and Coverage (KPC) survey of 493 mothers of children under two (data collection was done between August 29 and September 2, 2004) and prepared summary information on project operations and accomplishments using the DIP, FAR, MTE and TAR reports, project monitoring data and the findings of various studies. All this information was compiled in folders and made available to each FE Team members upon their arrival. The fieldwork ran from September 21 through September 29, 2004. The FE Team first gathered in Pontianak to meet with the Provincial Health Officer and the Governor of West Kalimantan Province. The next day in Ngabang, the LCSP Core Team presented the overall achievements of the LCSP including the FE KPC results during a Technical Overview meeting attended by 32 participants including representatives from the DHO and the Health Centers of the project area. This meeting was an opportunity to brief all the FE Team members, discuss the various questions for the FE including the potential questions for the field visits, and obtain feedback from the MOH representatives. On the third day, the FE Team attended as observers the Village Leaders Feedback meeting that the LCSP Team held to provide feedback on the KPC survey and the overall achievements of the project. A summary of this meeting and the resulting Declaration of Commitment to continue the project activities is included in Annex 20. Based on these various meetings and FE Team group discussion, the FE Team developed data collection tools before spending 2 days visiting various sites as prepared by the LCSP Team. The type of sites visited or individual met, the methodology used and a reference to the specific annex that includes the related findings and tools are presented below. Site visited or individual met Evaluation Method Reference Health Center Interview, observation, record review Annex 11 Sub Health Center Interview, observation, record review Annex 12 TBA and midwives Interview, record review Annex 13 Posyandu cadre leaders Focus Group Discussion Annex 14 Posyandu site visit Interview, observation, record review Annex 15 Posyandu cadre Assessment Knowledge test and observation of role play Annex 16 POD Interview, observation, record review Annex 17 MED Interview Annex 18 KPD Interview Annex 19 World Vision 115 LCSP Final Evaluation Below is the composition of the three field visit teams and the respective sites visited in the three Health Center areas. TEAM 1 TEAM 2 TEAM 3 PAHAUMAN SENAKIN MANDOR Team Composition Dr Ina Hernawati Pak Muhidin (HC Pahauman) Ibu Sophia Tjakre Dr Marc Debay * Lyndon Brown * Dr Sri Chander * Dr Te Chilay * Keo Sereivuth * Dr Ferdinand Laihad Bp Abang Suhaimi Dr Alex Papilaya Kapuskes Mandor Pak Rama Ibu Mary Lengkong Bp Markus Akim Pak Manuk (HC Senakin) Elisabeth Erna Esther Indriani Bp Untung Sidupa Dr Teresa Zakaria Mariani Aritonang Rainy Wijaya Fahanni (Depkes RI) Sites visited Puskesmas Pahauman Puskesmas Mandor Pustu Sidas Pustu Ngarak Posyandu Tumahe Posyandu di Air Merah POD di Air Merah TBAs Kranji Mancal TBAs di Air Merah Polindes Kranji Mancal Puskesmas Senakin POD di Singkut Durian MED di Singkut Durian KPD Senakin * Does not speak Indonesian. After the field visits, the entire FE Team met again to prepare the field visits reports, discuss the findings and develop recommendations. The team then made various presentations to the District and Provincial Health Offices and held a one-day Lessons Learned workshop in Jakarta. The list of participants to the Technical Overview meeting, to the Village Leaders Feedback meeting, to the District- and the Province-level presentations and to the Lessons Learned Workshop are included in Annex 21). A separate report on the Lessons Learned Workshop was under preparation at the time of the preparation of this report. World Vision 116 LCSP Final Evaluation Annex 11. FE Field Visits – Health Centers A. Cold Chain Senakin: The vaccinator, who bears day-to-day responsibility for vaccine and cold chain management, was in the field. There is an operational upright electric refrigerator with freezer compartment. Both compartments had appropriately-positioned thermometers, reading –5 and +6 degrees respectively. Measles vaccine was being stored in the freezer. The HC In-Charge believed this was in accordance with national policy [WHO currently recommends storage between +4 and +8 degrees for measles vaccine]. The refrigerator contained anti-venom and other non-EPI vaccines, but no other items. The stock was not well organized. Temperature records appeared to be maintained carefully, and no reading outside the acceptable range was noted. There is an operational generator. The HC In-Charge attempted to start the generator, but was unsuccessful. He stated that another person, who lived nearby, was responsible for maintaining the generator and starting it when necessary. However, he was not available at that time. A member of the evaluation team was subsequently able to start the generator. There is no written SOP, but the HC In-Charge expressed confidence that it would be started if there were a power failure of significant duration. Fortunately, power outages are not very common there. Pahauman: There is an operational refrigerator, with a correctly positioned thermometer. However, the temperature record was not posted on or nearby the refrigerator. When located, it showed acceptable recording for the month of August, apart from one reading of 9 C. No entries had been made for September. At the time of the team’s visit, the temperature was observed to be 12 C, which is 4 higher than the maximum acceptable. Two bottles of water were being stored in the refrigerator, apparently kept there for refreshment rather than temperature stabilization. The vaccinator was present during the visit, but was in fact on leave for a month. He stated that he had asked another member of the HC staff to maintain the temperature record in his absence, but that it had not been done. He was able to state the generator SOP, but no written copy was available. No vaccine stock-out has occurred in the previous 12 months. Mandor: There is a functioning refrigerator, with an appropriately positioned thermometer. The temperature record was kept in a booklet near the refrigerator. Recording of temperatures were complete up until September 21st, but discontinued after that, because the vaccinator was in the field. Apparently, this responsibility is not routinely transferred in his absence. The refrigerator temperature was found to be in the correct range during the team’s visit, but OPV was being stored in the freezer compartment at a temperature of 0 C. A new ice-lined refrigerator/freezer unit had been delivered to the health center two months ago and, although ready for use, remained empty and unused. The older refrigerator was found to be clean, and contained no other items apart from vaccines. Although the World Vision-supplied generator was said to be operational, a cable was missing and the HC In-Charge stated that he had borrowed it for use at home “overnight”. Project staff reports that the generator had also been used at home by the HC In-Charge for an extended period. No evidence of a generator SOP was found. B. Costs of Providing Services Thru Posyandus Senakin: HC staff visits each of 31 posyandus on a once-monthly basis. Because of Christmas and Ramadan, they are able to maintain this schedule 10 months out of the year. During these visits, they provide MCH and other basic curative services and health education, as well as monitoring and supervision of posyandu kaders and POD kaders. The staff travels to and from the posyandus in a HC vehicle. The only travel cost is for fuel, an estimated Rp 9000 per trip. Food for the staff is provided by posyandu kaders at no-cost to the health center (these costs are covered through patient fees of Rp 3000 per visit, of which Rp 2000 goes for operational costs). Pahauman: The HC staff, working in 9 teams, visits each of 46 posyandus 5-8 times per year. Each team is responsible for 8-10 posyandus. They provide routine immunization, antenatal care (TT immunization and iron￾folate supplements), family planning, ORT and health promotion covering ARI, diarrhea, malaria, family planning, breastfeeding and immunization. The teams do not provide any curative care during posyandu visits. Sick children are referred to the nearest sub-HC or village mid-wife. World Vision 117 LCSP Final Evaluation Mandor: HC staff visits each of 32 posyandus once a month. Each HC staff member is responsible for the support of between 1 and 5 posyandus. Services provided through posyandus include immunization, antenatal and basic curative care. The staff travels to and from the posyandus by motorcycle. Fuel costs an estimated Rp 5000 per month for posyandus near to pukesmas, and an additional Rp 20,000 for those that are farther away. These costs are covered by the health center budget. C. EPI Coverage Data The various EPI reports or data collected by the FE Teams in each Health Center were too eclectic to be compiled and analyzed in a meaningful way. The FE Team also requested and received the EPI reports compiled at the District level for August 2004, December 2003, December 2002 and December 2001. These reports include the monthly and yearly cumulative number of each antigen given and the target populations for 14 sub-districts. These data were examined and included in the FE report. D. Waste Disposal Senakin: Needles and syringes are collected in a tray, and reportedly put through a “crusher” which was supplied by the District Health Office. The resulting “safe” remains are then buried. The team noted that at least 20 syringes had accumulated in the tray, still awaiting disposal. The crusher was in another room, and could not be inspected. The team found refuse at ground level, close to one of the HC buildings, apparently because a shallow hole had been completely filled. Several fully recognizable and still-dangerous syringes were noted, on the surface and fully exposed. The HC staff member present said that cleaning staff had apparently dumped un-crushed syringes there, contrary to instructions. She agreed that the situation was unsatisfactory, and said they would at least dig deeper holes to bury the refuse completely, and that she would alert the rest of the staff to this ongoing problem. Pahauman: The HC has no system or equipment for safe disposal of needles and syringes. They are thrown into an open well in the yard, although other recently-used syringes were found lying openly on the ground in an area where trash is occasionally burned. Mandor: Needles and syringes are either burned or buried in a pit (approximately 50% each). No mention was made of a “crusher” for these items, although one was said to have been supplied by the DHO to each health center. The team found current disposal practices to be within acceptable standards. E. ORS Supply Senakin: The HC In-Charge stated that ORS was always available in adequate quantities there, and that they supplied adequate amounts to the posyandus upon request. They provide 1 box of 100 sachets of ORS for each posyandu. Posyandu kaders, during monthly activity can request for more when stock-out. He seemed surprised to hear that kaders in one posyandu complained their supply was inadequate. He acknowledged that there are more diarrhea cases during the dry season, and that at some point a posyandu may require more ORS. In this situation, a kader must travel to the health center to obtain more ORS. The time and transport costs involved represent an additional burden for the kader. ORS supply and distribution is recorded on a drug registration form. The quantity of ORS distributed to each posyandu is also recorded on this form. Pahauman: HC staff said they routinely take one box of ORS packets to each posyandu during their monthly visits, leaving the unused packets with the kaders each time. Requests for additional packets from posyandus was said to be infrequent. The team was unable to check the ORS stock and distribution register, since the HC In-Charge was absent. Mandor: ORS packets are routinely distributed to 32 posyandus. Records showed the following monthly distribution so far in calendar 2004: January 271 May 109 February 365 June 97 March 388 July 89 World Vision 118 LCSP Final Evaluation April 251 August 108 (not final, only 26 posyandus) Currently in stock: 1500 packets (3 boxes x 500 packets per box) Ordered (June 04): 3000 packets requested 2000 packets received F. Support for PODs (Village Drug Posts) Senakin: There are two functioning PODs in the HC catchment area, in Bejambu-Sairi and Maro'o. There does not seem to be a request form as such or, if one exists, it is not in regular use by those who interviewed. Both PODs are located near posyandus, and the health staff provides routine supervision on the same days they visit those posyandus. Other informal contacts are reportedly common. HC staff expressed concern that POD kaders often succumb to pressure from customers to sell less than a full course of medicine (chloroquine, in particular). When asked if they thought the PODs should be closed for this reason, they said emphatically that this would not be feasible, but felt the situation could be improved over time, with continued supervision and health education. Pahauman: There are no PODs in the HC catchment area. Mandor: Ten PODs have been established in the health center area. The HC In-Charge stated that PODs do not report to the HC, nor have any of them submitted requests for replenishment of initial drug stocks. No supervision of the PODs had been provided by HC staff. The In-Charge stated that the PODs were under the ADP’s and therefore not his responsibility. Possible constraints pointed out by the In-Charge include: drugs sold at the PODs are available almost everywhere (pukesmas, polindes) and people prefer to buy them from health staff; the price of drugs sold by PODs (decided by the communities involved) is close to the market price. World Vision 119 LCSP Final Evaluation Recommendations from Health Center Visits Senakin:  Store measles vaccine in refrigerator, at 4-8 C, according to current WHO recommendations  Establish and post SOP for generator, making sure it can be started promptly if power fails.  Crush used syringes promptly; incinerate waste completely, or bury it in a deep pit.  Continue using ORS packets, even beyond their expiry dates, if powder isn’t clumped.  Continue support for PODs in this catchment area, which seem to be doing well. Pahauman:  Monitor refrigerator temperatures more closely, keeping them within the range of 4-8 C.  Ensure transfer of responsibility for cold chain when vaccinator is on leave or in the field.  Establish and post SOP for generator, making sure it can be started promptly if power fails.  Introduce procedures for safe disposal of syringes and needles: incinerate, bury in a deep pit. Mandor:  Activate use of the ice-lined refrigerator (ILR) and always store OPV below -20 C.  Assign an alternate person to record refrigerator temperatures twice a day whenever the vaccinator is away from the health center.  Develop and display SOP for use of generator to maintain cold chain in the event of power outage.  The generator should be kept ready for immediate use in the event of power outage. Under no circumstances should the generator, cable or other accessories be taken from the health center or used for other purposes.  Develop procedure for routine ORS distribution to posyandus, maintaining adequate stocks on a continuing basis. World Vision 120 LCSP Final Evaluation Questions for Health Center A. COLD CHAIN Observation: Yes No  Check whether there is a functioning refrigerator  Check whether there is a thermometer in the fridge  Check whether there is a temperature chart on the fridge  Check whether the chart is correctly filled with twice a day  Check whether the temperature is in the right range? 4-8 degrees for DPT and measles 0 degrees for polio  Check whether the fridge is clean  Check whether there are other items stored in the fridge such as food, drinks  Check whether there is a functioning generator Questions to the health center staff:  Is there an assigned person to check the temperature of the fridge?  Is there an SOP to operate the generator?  Can you see the SOP? World Vision 121 LCSP Final Evaluation B. COSTS OF PROVIDING SERVICES THROUGH POSYANDUS Questions to health center staff:  How is monitoring of posyandus done?  What type of transportation is used for monitoring and what is the cost? Motor cycle Rp………………. Bicycle Rp………………. Public transportation Rp……………….  Are there any other costs involved for monitoring beside transportation and how much? Food Rp………………  How do you manage to cover the cost? By the health center Y/T Use own funds Y/T  In one year, how many times do you do monitoring of one posyandu?  How many posyandus do have to cover? C. EPI COVERAGE DATA Check health center’s EPI register for month August 2003 and August 2004 D. WASTE DISPOSAL Check how needles are disposed and where E. ORS SUPPLY  Check availability of ORS. How many ORS sachet are available?  Check register of ORS stock and distribution during the last six month. F. SUPPORT FOR PODs (VILLAGE DRUG POST)  How many POD are functioning in health center area?  How many times has POD supervision being done for the last six months? World Vision 122 LCSP Final Evaluation Annex 12. FE Field Visits – Sub Health Centers Pustu Sidas (Pahauman): Cold Chain: N/A Cost of Posyandu Support: Services provided include immunization, basic curative care and health education. Travel by motorcycle, with cost of Rp 5,000 for fuel covered from the HP budget. Waste Disposal: Syringes and needles accumulate in a basket, and with occasional incineration. ORS Supply: Three boxes (300 packets) in stock, distributed to posyandus only for dry season. Pustu Ngarak (Mandor): Interview with Nurse-Midwife Martina (See also TBA and midwife report): Costs of Posyandu Support: None for transport or food – the nurse only supports one posyandu, within walking distance. Waste Disposal: Used needles and syringes are kept in a plastic bag, with sharp ends protruding – to be buried. ORS Supply: One box of 100 packets in stock. According to Drug Usage and Request Form, 10 used and 10 remaining, from an initial stock of 20. (Vitamin A capsules are delivered in February and August, for supplementation of under-fives.) World Vision 123 LCSP Final Evaluation Questions for Sub-Health Center A. COSTS OF POSYANDUS SUPPORT Questions to health center staff:  How is monitoring of posyandus done?  What type of transportation is used for monitoring and what is the cost? Motor cycle Rp………………. Bicycle Rp………………. Public transportation Rp……………….  Are there any other costs involved for monitoring beside transportation and how much? Food Rp………………  How do you manage to cover the cost? By the health center Y/T Use own funds Y/T  In one year, how many times do you do monitoring of one posyandu?  How many posyandus do have to cover? B. WASTE DISPOSAL Check how needles are disposed and where C. ORS SUPPLY  Check availibility of ORS. How many ORS sachet are available?  Check register of ORS stock and distribution during the last six month. World Vision 124 LCSP Final Evaluation Annex 13. FE Field Visits – TBA and midwives Interview with Midwife Desi, in Polindes Kranji Mancal (Pahauman): Relationship with TBAs: Joint management for sick babies; pregnant women; during labor if called; postnatal visit; provision of simple supplies (betadine; vitamin A; ORS). Monthly meetings with 32 TBAs in the Pustu’s area (4 villages) IEC materials given (from LCSP and from HC) No systematic monitoring of TBA activities; only through joint activities. Deliveries in past 12 months: Total of 59 (57 alone, two in partnership, two others by midwife alone) Proportion of deliveries assisted by midwife estimated as 50%; could increase this by offering a package during including assistance at delivery during ANC visits. Problems that occurred during deliveries included 1 bleeding (multigravidae); 1 breech presentation; 3 asphyxia (2 prolonged labor); and 1 prematurity. Also 1 antepartum bleeding referred to Pontianak with IV in Health Center’s ambulance. Births and deaths in the past 12 months: One neonatal death in four years. Reporting: Midwife reports births and deaths to HC during monthly meeting. Available registers include IMCI, delivery, family planning, ANC, and LAM. Interview with (name of TBA not recorded) in Polindes Kranji Mancal (Pahauman): Deliveries assisted in past 12 months: 5 (3 alone, 2 with Polindes midwife); no problem except one premature baby for which she called the midwife. Regular contact with midwife since posted in Pustu. Pregnant women referred to HC in past 12 months: 5 TBA recorded all items correctly in delivery register. A weighing scale was provided by LCSP. TBA knew that colostrum should be fed to baby, having learned this during her LCSP training. Vitamin A reportedly given to post-partum mothers promptly after delivery. Mothers told to breastfeed exclusively for 4 months (but TBA answered T/F question correctly). World Vision 125 LCSP Final Evaluation Interview with Nurse-Midwife Martina, in Pustu Ngarak (Mandor): Relationship with TBAs: TBAs call her if labor is difficult. Routine meetings with TBAs at Mandor HC. Technical support only when she helps with deliveries. No logistic support or IEC materials provided – these come from Mandor HC. Monitoring of TBA is the responsibility of HC midwife. Deliveries in past 12 months: Assisted with 20 deliveries so far in 2004, 29 in 2003. Proportion of deliveries she assisted, compared to those by TBAs, is unknown – record at HC. Reporting: Reports deliveries to HC monthly. Records available include: Monthly report of MCH/FP activities Monthly report of Nutrition & MCH Child Health Activity Report (LB3) Interview with TGAs Norma and Maria Clara, at Air Mehra Posyandu (Mandoor): One delivery in the past 12 months, with no problems associated. Two pregnant women referred to the HC. Delivery reported to midwife at time of pukesmas meeting. No register or other record of birth. No deaths in the past 12 months. Gave correct answer to question on feeding colostrum, with two correct reasons for doing so. Gives vitamin A post-partum; good responses to knowledge questions. World Vision 126 LCSP Final Evaluation Questions for Polindes Midwife Area: ___________________ Midwife’s name: _____________________ Relationship with TBA 1. What kind of partnership between Midwive with the TBA in the last 12 months? 2. Is there any routine meeting between the Midwife and the TBAs? 3. What kind of technical support do you give to the TBAs? 4. What kind of logistic support do you give to the TBAs? 5. What kind of IEC materials is given to TBAs? 6. How is the monitoring system of TBAs done by the Midwife? Tools? Timeline/Schedule? How many deliveries in the last 12 months 7. How many deliveries that you assisted for the last 12 months? If possible, in each month within last year? 8. How can you increase the number of deliveries assisted (by the Midwife)? 9. What is the proportion of deliveries assisted by you – and by the TBA (estimation)? 10. What were the problems that occurred during the deliveries? 11. How did you solve these problems? Report (Births, Deaths in the last 12 months) 12. How do you report deliveries to the Health Center? 13. List what are the Midwife’s records available 14. Look at the record of delivery assisted Completeness ( YES / NO ) Correctness ( YES / NO ) 15. Look at the record of child and maternal death? Completeness ( YES / NO ) Correctness ( YES / NO ) 16. How many child and maternal deaths (in the last 12 months)? World Vision 127 LCSP Final Evaluation Questions for TBAs Area: ___________________ TBA’s name: 1. _____________________ 4. _____________________ 2. _____________________ 5. _____________________ 3. _____________________ 6. _____________________ Questions can be asked to group of TBAs or individual How many deliveries in the last 12 months 1. How many deliveries that you assisted for the last 12 months? 2. What were the problems that occurred during these deliveries? 3. How did you solve these problems? 4. Have you ever sought the help / assistance from the midwife during the delivery or after delivery? And if so, for what kind of problems / any examples? 5. How many pregnant women were sent to the Health Center by the TBA? Report (Births, Deaths in the last 12 months) 6. How do you report deliveries to the Midwife? 7. Look at the record of delivery assisted. Did the TBA record the following items? Circle Yes or No Name of child: ( Y / N ) Sex: ( Y / N ) Name of Mother: ( Y / N ) Name of Father: ( Y / N ) Time of Birth: ( Y / N ) Birth Weight (if there is): ( Y / N ) Length of Baby: ( Y / N ) Status (whether baby is born alive / dead): ( Y / N ) 8. Look at the record of child and maternal death. What did the TBA record about child and maternal death? Completeness YES / NO Correctness YES / NO 9. How many child and maternal deaths (in the last 12 months)? Discussion about key health messages (training materials) 10. Should the mother give the first yellowish milk to the newborn or throw it away? (YES / NO ) Why? 11. Do you give Vitamin A to women after delivery? ( YES / NO) 12. When do you give Vitamin A to women after delivery? 13. Vitamin A should be given to mother within one month after delivery or within two months? 14. How many Iron tablet that the pregnant woman take - one per day or one per month? 15. Please explain what you recommend to mothers about breastfeeding? 16. In the first six months of child life, should the mother give breastmilk only or breastmilk and complementary food? 17. When the child is 2 months old, do you know what the mother should do? (For the interviewer: Bring the baby to get immunization, or the baby should not be brought outside the home, or others….) World Vision 128 LCSP Final Evaluation Annex 14. FE Field Visits – Posyandu Focus Group Discussions The FE Team conducted one FGD with Posyandu team leaders in each Health Center area. The total number of respondents was 20 females (64.5%) and 11 males (35.5%). Findings of these FGD are reported here together. Job description of Posyandu cadres: D day minus: - Contact cadres and remind them all to come to the posyandu session. - Divide roles of cadres for the posyandu session. - Prepare equipment and forms for the posyandu session. - Prepare supplementary feeding (from 31 Posyandus 8 (25.8%) did not prepare food supplement). D day: - Provide services according to table 1-4. - Refer under-nourished and sick children to health center. D day plus: - Prepare posyandu monthly report. - Check register and record absent children for follow-up (home visit). - Remind Cadres who did not participate in posyandu session. - Evaluate posyandu activities. Number of active cadres: Not all posyandus have 5 active cadres. If the number is less than 5, then: - Remaining caders will perform multiple tasks (more than one table). - Sometimes mothers attending the posyandu session are asked to help out by doing simple tasks. - Recruit new cadres. - Ask for help from village chief to find new cadres. Budget: - Posyandus do not have a budget. - Even though there is no budget, yet funds are needed for supplementary feeding and stationeries. - Funds raising is done by: o Collecting money (Rp. 200,- to Rp. 500,-) for weighing of children. o “Arisan” among mothers to collect money. o Village funds o Occasionally from health center budget. - Most cadres mentioned that without funds posyandus activities can be implemented, but several cadres mentioned that if funds are not available poyandu activities cannot to be carried out. - Funds expended by the cadres for one posyandu session: o Snack for health center staff and kaders Rp. 20.000,- o Complementary feeding for children Rp. 15.000,- Rp. 30.000,- o Stationeries Rp. 7.500,- o Transportation (ojek) to go to the posyandu Rp. 10.000,- - Rp. 40.000,- o Opportunity cost for attending posyandu session Rp. 15.000,-/day Logistics: - Houses, village community meeting place, schools and one is under a tree. - All posyandus have a weighing scale. - Not all posyandus have tables available. To implement activities tables are borrowed from neighbors. - Recording forms are available in all posyandus. - Posyandu equipment usually stored in cadres house; in one case, in village community meeting place. - There is no need for maintenance of equipment because the posyandu has only one weighing scale. - Oralit is always available. In case of shortage the cadres send a request to the health center. - Mothers needing ITNs are referred to the local cooperate that sells ITNs. World Vision 129 LCSP Final Evaluation Questions for Focus Group Discussion on Posyandu Management Date : Name of Health Center : Name of Facilitators : Name of cadres : 1. 5. 2. 6. 3. 7. 4. 8. I. General explanations and introduction: 10 minutes II. Cadre ( 25 minutes) 1. Job description  D - (eg. community mobilization, cadres meeting, preparation (place, equipment, forms, etc)  D (eg. registration, weighing, recording, health education/promotion 5 programs, referral cases, supplementary feeding  D (eg. home visit-see page 35 of manual guideline of posyandu cadres) 2. Number of active cadres  if number of active cadres less than 5, please discuss what you are going to do? III. Budget ( 15 minutes) 1. Discuss the availability of resources/budget for posyandu monthly activity e.g budget for supplementary feeding, stationary, etc? 2. Please identify how much money for operational cost posyandu (Rp.). IV. Logistics ( 20 minutes ) 1. If there is no scale, discuss what you are going to do ? 2. If there is not enough stock of oralit, what you are going to do ? 3. If there is no LBKP form for next month, discuss what you are going to do ? 4. If mothers/families would like to have ITN, discuss what you are going to do ? 5. Discuss how cadres maintain/keep posyandu instrument and equipment ? V. Method ( 10 minutes ) 1. Discuss what are you going to do in encouraging mothers attend the posyandu sessions? 2. How to get data number of mothers (the new target) VI. Monitoring ( 20 minutes ) 1. Supervision from Village Chief 2. Supervision from pokja IV PKK 3. Supervision HC Staff 4. Monitoring and Follow up SKDN World Vision 130 LCSP Final Evaluation Annex 15. FE Field Visits – Posyandu Day A. Immunization services at Posyandu 1. Ice pack melted when it reached the posyandu 2. Number of vaccines is complete except for tetanus vaccines. 3. One needle is used for one child. 4. Vaccines given to children follow standard operating procedures B. MCH booklet a. Distribution and utilization of MCH handbooks From 33 mothers interviewed at Senakin and Pahauman, 9 (27.2 %) mothers do not have an MCH handbook. From the 24 mothers(72.8%0 who own an MCH handbook, 7 mothers (29.2%) have never read the handbook. The mothers who read the handbook, only 1 from the 17 mothers is not able to explain correctly what they read. b. Completeness of MCH handbook record From the 33 MCH handbooks, 10 handbooks (41.7%) were partially filled in, 14 handbooks (58.3%) were minimally filled-in C. Availability of Oralit At one of the posyandus evaluated, there was no Oralit available since November 2003. In two other poyandus, oralit stock is not available since the last 6 months. In an other posyandu, oralit is available but stored at the kaders house. D. Register and reports In the 2 posyandus evaluated, one posyandu report is complete and correct but in the other posyandu reporting was complete but not correct. This can raise inconsistency in the posyandu information system. E. Cost analysis - Opportunity costs: Opportunity costs are costs earned by caders per day if they attend posyandu session instead of doing their routine work. Caders at several posyandus in Pahauman, who are mostly teachers (government officials), are not able to mention how much their opportunity cost is because posyandu sessions are operated after hours. In other posyandus, caders earn per day Rp. 10.000,- Rp. 15.000,- which they do not get because of attending posyandu sessions. The time spent by cadres to attend posyandu sessions (per month): cadres spent 2 hours for posyandu preparation activities, 3 hours for posyandu session, and 8 hours for follow-up activities. - Operational costs: Supplementary feeding per session: Rp. 30.000,-. Snack for health center staff and cadres: Rp. 20.000,- Cost for stationeries per month: Rp. 10.000,- Transportation cost for cadres: Rp. 5.000,- to Rp. 15.000,- Transportation cost for health center staff: Rp. 5.000,- to Rp. 10.000,- Transportation cost for mothers to come to the Posyandu: Rp. 6.000,- World Vision 131 LCSP Final Evaluation Questions for Posyandu Visit Posyandu Name : A. Immunization Observation Fill-in the Immunization Observation Check List B. MCH booklet Quick individual interview of 14-19 mothers: Do you have a MCH booklet? Yes / No Can you tell what is written in it? Answer: No / Wrong / Adequate Check the booklet: Is it well completed? Minimum / Partial / Complete 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 World Vision 132 LCSP Final Evaluation C. ORS: Interview of one cadre: 1. How many ORS packets are there today? Count: ________ 2. Has there been any stockouts in the last 6 months? Yes __ No __ If yes, how many stockout month ? __ Please explain : D. Register and report: Look at this year’s register and reports. 1. Check consistency between the register and the report. Consistent: Yes __ No __ Explain: 2. Check completeness of the register for last 3 months. Complete: Yes __ No __ Explain: 3. Check correctness of the register for last 3 months. Correctness: Yes __ No __ Explain: 4. Check completeness of the report for last three months. Complete: Yes __ No __ Explain: 5. Check correctness of the report for last three months Complete: Yes __ No __ Explain: E. Cost analysis How much money do you earn per day ? (1) How much money do you earn if there is Posyandu day ? (2) Opportunity cost (1) – (2) 1 2 3 4 5 World Vision 133 LCSP Final Evaluation Annex 16. FE Field Visits – Posyandu Cadre Assessment The FE Team took the opportunity of quarterly meetings of all the Posyandu Cadres in each Health Center area during the FE period to assess their knowledge and skills. Although purposely organized during the period of the FE, these meetings are otherwise a routine activity supported by LCSP during which all the Posyandu Cadres are provided refresher training, various information related to their work and networking opportunities. The FE Posyandu Cadre Assessment aimed at ascertaining whether the percentage of cadres in each Health Center area who know selected items of information and have selected skills critical for providing expected services during Posyandu Days are above is above 80% (a satisfactory finding) or below 50% (a finding of concern). The Cadres meetings are generally attended by large number of Posyandu cadres and all cadres present at each meeting were considered as representantive of all the active Poysandus cadres in their Health Center area. As written tests are routine activities during the quarterly cadres meeting, all the cadres were asked to take the knowledge test. The FE Team then selected randomly 19 answer sheets and called the respondents to participate in the skill assessments. The sample size of 19 was chosen to follow common Lot Quality Assurance Sampling procedures commonly used in CHSGP CSP.44 The items of knowledge and skills included in the assessment were selected by the LCSP Training Coordinator from the materials used to train Posyandu Cadres. The knowledge assessment consisted of a written test with simple open questions and the answers sheets were codified by the Training Coordinator using criteria defined in advance (see question attached). Two different types of skills were assessed using two different exercises. Health Education skills (the table 4 function) were assessed through role-plays where the cadres were asked to simulate a health education session on ARI/Pneumonia under the observation of one FE Team member who filled a simple checklist. Implementation of this part of the assessment slightly differed from one Health Center to the other because of different understanding of the protocol or type of BCC materials used. In Pahauman, the FE Team interviewed each cadres to assess their knowledge and then assessed their communication skills through a simulation using the ARI poster. In Senakin, the cadres were assessed through a simulation but did not have ARI posters. In Mandor, the cadres were assessed through a simulation and had the option of using of ARI poster or not. Recording skills were assessed by reading simple examples of cases to the cadres and asking them to fill a Growth Monitoring Card accordingly. The answers to this exercise were also codified by the Training Coordinator using predetermined criteria. The table below presents the results of the Cadres Assessment for the three Health Center areas and for the entire LCSP area. The average proportion of each satisfactory answer and skill and the related confidence intervals for the entire LCSP area are also provided. The graph present the percent of cadres satisfying the the various indicators of knowledge and skill and the 95% confidence intervals for the entire LCSP area. 44 Joseph J. Valadez, William Weiss, Corey Leburg, Robb Davis. Assessing Community Health Programs A Participant’s Manual and Workbook Using LQAS for Baseline Surveys and Regular Monitoring. Teaching-aids At Low Cost (TALC), St Albans (UK), 2003. World Vision 134 LCSP Final Evaluation Results of the Posyandu Cadres Assessement # < 50% 50-80% > 80% # < 50% 50-80% > 80% # < 50% 50-80% > 80% # Avg. Wt.Avg.* LCL UCL A. KNOWLEDGE 1 GMC use 19 x 17 x 19 x 55 96% 97% 92% 101% 2 Avantage of MCH versus KMS 10 x 14 x 15 x 39 68% 70% 58% 82% 3 Causes of diarrhea 10 x 14 x 13 x 37 65% 65% 53% 78% 4 Danger sign of diarrhea 17 x 14 x 18 x 49 86% 87% 78% 96% 5 Diarrhea management 18 x 19 x 19 x 56 98% 98% 95% 102% 6 ARI cause 2 x 10 x 8 x 20 35% 36% 24% 49% 7 Danger signs of ARI 15 x 17 x 11 x 43 75% 74% 62% 85% 8 The cause of MALARIA 7 x 8 x 19 x 34 60% 64% 52% 77% 9 Benefits of immunization 16 x 17 x 19 x 52 91% 92% 85% 99% 10f Mention at least 1 immunization type 18 x 19 x 19 x 56 98% 98% 95% 102% 10g Mention all immunization type 16 x 14 x 19 x 49 86% 87% 79% 96% B. HEALTH EDUCATION SKILLS, ARI 1 Common cold can develop into ARI 7 x 12 x 19 x 38 67% 71% 59% 82% 2 Rapid breathing 15 x 14 x 15 x 44 77% 77% 66% 88% 3 Stridor 8 x 11 x 11 x 30 53% 53% 40% 66% 4 Chest Indrawing 11 x 9 x 17 x 37 65% 68% 55% 80% 5 No appetite 3 x 3 x 16 x 22 39% 44% 31% 56% 6 Should be fasted 2 x 2 x 13 x 17 30% 34% 22% 46% 7 Immediate Referral 13 x 15 x 19 x 47 82% 84% 75% 94% C. RECORDING SKILLS, KMS 1 Identification 16 x 6 x 18 x 40 70% 73% 61% 84% 2 Immunization 6 x 2 x 7 x 15 26% 27% 16% 39% 3 Vitamin A 6 x 2 x 8 x 16 28% 29% 18% 41% 4 Weighing 5 x 5 x 4 x 14 25% 24% 13% 35% * Average weighted by proportion of Cadres in each Health Center area NO. SELECTED INDICATOR SENAKIN AREA MANDOR AREA PAHAUMAN AREA LCSP area Decision rules: Take 19 cadres and submit to the test. For each item (indicator) tested: (1) If 13 or more cadres respond or perform satisfactorily, then the percent of all cadres in the area who can respond or perform satisfactorily is considered as higher then 80% with 95% confidence; (2) If 7 or less cadres respond or perform satisfactorily, then the percent of all cadres in the area who can respond or perform satisfactorily is considered as lower then 50% with 95% confidence; (3) If more than 7 but less then 13 cadres, then then the percent of all cadres in the area who can respond or perform satisfactorily is considered as between 50% and 80%. World Vision 135 LCSP Final Evaluation Questions and Observation Checklist for Posyandu Cadres Assessment Tanggal: ________ Nama Kader: ________________ Wilayah: ________ Nama Posyandu: _____________ Knowledge Questionnaire 1 What is the use of KMS (Growth Monitoring Card)? 2 What are the advantages of MCH Booklet compared to the Growth Monitoring Card? 3 What is the cause of diarrhea in under five children? 4 What are the danger signs of diarrhea that may cause death in under five children? 5 What should mother do if her under five child has diarrhea? 6 What is the cause of ARI or pneumonia? 7 What are the danger signs of ARI or pneumonia that may cause death in under five children? 8 What is the cause of malaria? 9 What are the benefits of immunization for children? 10 Mention the types of immunization in children! Health Education Skills on ARI/Pneumonia Please demonstrate how to deliver health education by the aid of poster on ARI / Pneumonia Items to check by observer if mentioned by cadres:  Common cold when not properly treated can develop into pneumonia  Rapid breathing  Stridor  Chest in-drawing  No appetite (the child does not want to eat or drink)  The child should not be fasted  Immediate referral Recording skills Please demonstrate how to fill in the Growth Monitoring Card using this story:  Second child of Mr Herkules and Mrs Agnes, name: Monika.  Born in August 2003 with birth weight of 2,8 kg  By the age of 2 months, Monica’s weight was 4,0 kg, received BCG immunuzation  Afterwards, for 3 months Monika was not weighted because Mrs Agnes travelled somewhere  By the age of 6 months, Monica had fever, but still got vitamin A, her weight was 5,4 kg  By the age of 7 months, her weight was 5,9 kg, received DPT 1 and polio 1 immunuzations  By the age of 8 months, her weight was 6,6 kg, received DPT 2 and polio 2 immunuzations  By the age of 9 months, her weight was 7 kg, received DPT 3 and polio 3 immunizations World Vision 136 LCSP Final Evaluation Annex 17. FE Field Visits – Village Drug Post Introduction The project contributed to the opening of 14 Village Drug Posts (PODs) in the two sub-districts and conducted trainings for the POD cadres. During FY 2004, LCSP also facilitated several meetings between the POD cadres, local community leaders and staff of the health centers of Mandor and Senakin. In these meetings, an agreement was reached stating that the health centers will provide assistance and supervision, regular meetings included. For this Final Evaluation, 2 PODs are selected for field visit. The first one is POD Belak in Mandor and the second one is POD Singkut Durian in Senakin. A. Open-Ended Questions Both POD kaders interviewed in Belak-Mandor and Singkut Durian-Senakin seemed to have a clear sense of their role and functions. Both knew the use of the medicine available in their PODs and said that they always provide explanation to costumers coming for medicine. Both knew where to procure medicines but have never gone to the health center so far. They still had drugs in stock from the initial supply provided after the POD training. Mr. Monen from Singkut Durian has an undergraduate degree in education, and is headmaster of the local school. He said that he knew all of his customers and that all of them live within a few kilometres of his shop. This is plausible, since his shop is located on a back road. This factor seems to contribute to his effectiveness, since he has established relations and a basis for trust with these customers. He seemed fully sincere in stating that he routinely provided health education to his customers, asking about symptoms and dispensing medicine or providing referrals appropriately. He seemed to understand the importance of completing a full course of treatment, and said that he routinely informed his customers of the danger of non-compliance. B. Specific Knowledge Questions Both cadres knew all three danger signs of ARI in a young child. They also knew the correct drug and dosage for treatment of malaria for adults as well as for children. They both also knew the correct instructions for managing a child with diarrhea. C. Patient Record Review The registrations of clients in both PODs was good, including the names and symptoms of most costumers, the date of visit, price and quantity of medicine given. The medicine dispensed are consistent to the symptoms reported. D. Drug Management Record Review Records appeared complete and carefully entered in both PODs. E. Observation of drug stocks The stocks of drugs were well organized in both PODs. The POD in Singkut Durian has experienced stock out of Cajuput Oil but did not need to procure it from the health center. Small posters with dosage schedules were posted on the wall next to the kader’s desk. Among the POD drugs, paracetamol and ORS were most commonly sold. Among non-POD drugs, massage oil was reported to be the most popular item. He also mentioned that he considered steroids dangerous and would not sell those drugs (in contrast to many other shopkeepers in the area). The POD cadre in Mandor had a note on drug dosage and administration in his drug cupboard. No expired drug was found. The most popular drugs are Paracetamol, Choroquine, Antacid, cough syrup and Oralit. World Vision 137 LCSP Final Evaluation Questionnaire for Pos Obat Desa (POD - Village Drug Post) POD Name : A. Open-ended questions 1. Can you explain your role and function as a POD cadre? 2. Tell me about the type of drugs that you have and how do you use them? 3. What do you do when a sick person come to the POD? 4. How do you refill your stock of drugs? B. Specific knowledge questions 1. Can you tell the danger signs of ISPA in a young child (balita)? Rapid breathing: ___ Stridor: ___ Chest In-drawing: ___ 2. Can you tell which drug you give when a child has pneumonia? Prompt for dosage: 3. Can you tell the drug that you give for malaria? Chloroquine: ___ Other: ________ 4. Can you tell how much do you give to a child? Dosage: ______ 5. Can you tell what do you do when a child comes with diarrhea? C. Patient record review Overall quality of the records: Good ___ Bad ___ Explain: Disease Number of case Quality of treatment Malaria Diarrhea Pneumonia Worms D. Drug management record review Overall quality of the records: Good ___ Bad ___ Explain: E. Observation of the stock of drugs Overall quality of the cupboard: Good ___ Bad ___ Stock out : No __ Yes __ Which drug(s)? Explain: Dosage schedules adequately displayed on the wall or equivalent: Yes: __ No __ Check on the original list drugs which are those that are mostly sold. World Vision 138 LCSP Final Evaluation Annex 18. FE Field Visits – Self Help Group 1. Introduction During FY 2004, LCSP sponsored one Self Help Group (KSM ROYONG SKDN) under Pontianak ADP with a fish pond as an additional source of income. The head of this KSM is also the head of the Posyandu Singkut Durian. This link between Micro Enterprise Development with health is expected to make the Posyandu activities more sustainable. 2. Fish Food Production In July 2004, LCSP conducted training on Fish Food Production using local commodity to lower the production cost. During the FE field visit, the team asked about whether the KSM members tried to produce their own fish food, but due to the lack of grinder machine they haven’t started to do so. 3. Fish Marketing The KSM members already have channels for marketing. They told us that people from the local community are the main market. 4. Fund Raising for Posyandu Activities The FE Team asked whether the KSM had allocated funding for health purposes such as Posyandu activities. A few members answered that this issue has been discussed and that they plan to raise funding for Posyandu after the fish harvest which will be around December 2004. 5. Health Education for Members So far the leader of the KSM has not provided health education for the members even though she has done it for other women groups. She confessed that she hasn’t found a suitable time and is planning to do so in the future. Questions to Self-Help Group On Micro Enterprise Development NAME OF GROUP/KSM : NAME OF RESPONDENTS : 1. __________________ 2. __________________ 3. __________________ DATE : INTERVIEWER : 1. ____________________ 4. _________________ 2. ____________________ 5. _________________ 3. ____________________ QUESTIONS : 1. What kind of business does your group run at the moment? 2. How is the result of the fish food production training? Have you produced your own fish food? If not, what is the reason? 3. How is your marketing plan? Do you already have your marketing line? 4. From your group’s business, are there any fund allocated for health purpose, for example Posyandu activities? 5. Is there any health education session for your KSM members? If yes, what methods are used and what topics are discussed? If not, what is the reason? World Vision 139 LCSP Final Evaluation Annex 19. FE Field Visits – Koperasi Pancur Dangeri [Interview of Mr. Yulianus Musir, Senakin KPD manager.] A. Feedback for the Malaria Month Campaign (SWOT Analysis)  Strength: the price of the ITN is very affordable and the fact that the community was already aware that malaria is dangerous.  Weakness: none. The team asked about problems that might have occurred due to the involvement of Posyandu cadres under KPD, but the KPD staff explained that there was none since the project has already provided identification card for the cadres.  Opportunity: KPD is still interested in selling ITNs in the future even though there are no more discount vouchers. Mr. Musir feels that the community awareness about ITN has increased and even though the selling will be slower, the market is still available. KPD provides community trainings on entrepreneurship, law, and politics. Usually KPD staff go to the villages three times a week where they also do promotion on ITN to increase the demand for the nets. He also stated that Posyandu cadres may still be involved in the distribution line as long as they agree to follow the KPD policies, such as not to increase the price beyond the amount set by KPD.  Threat: the ITN price is still a problem for many people. B. Knowledge About ITN Mr. Musir is already quite familiar with ITN. In his shop, there are posters on malaria issued by LCSP, including posters on ITN. According to him children are the most vulnerable and should be prioritized to sleep under the ITN. He did not mention pregnant mothers as equally important. C. Health Education for Members The mother organization for KPD is Pancur Kasih, which is also often involved in ADP activities. Pancur Kasih possess a human resource department where trainings are designed and planned. Pancur Kasih has planned to provide health education for its member, including members of KPD but up to now, lack of funding is the main constraint. They hope to find a suitable trainer with health education background willing to cooperate with them for a low fee. QUESTIONS FOR KSU PANCUR DANGERI (KPD) NAME OF KPD : KPD SENAKIN RESPONDENT’S NAME : 1. __________________ 2. __________________ DATE : SEPTEMBER, 24 2004 INTERVIEWER : 1. ____________________ 2. ____________________ 3. ____________________ QUESTIONS : 1. FEEDBACK OF THE ITN PROMOTION MONTH (SWOT Analysis)  What are the main factors that contributed to your achievement in selling  2.000 ITNs in a month?  Were there any constraint in selling the ITNs?  Do you see any opportunity for similar ITN marketing in the future? If yes, please mention.  On the contrary, do you see any threat for similar marketing? 2. Do you know the use of ITN? And who are the priorities to sleep under an ITN? 3. Are there health education sessions for members in KPD? If yes, what methods are used and what topics are discussed? If not, is there any opportunity for KPD to include health education sessions for the members? World Vision 140 LCSP Final Evaluation Annex 20. FE Field Visits – Village Leaders Feedback Meeting The FE Team invited a meeting with the village leaders from the project area. The purpose of the meeting was to:  To communicate the LCSP Final KPC Survey Results to the local community leaders.  To obtain community feedback on these results and other health issues, including health services  To develop a plan of action by local leaders for further improvements regarding the health conditions of their community. This meeting had been prepared by the LCSP Core team and the motivators. The LCSP Team invited the two heads of Mandor and Sengah Temila Sub-districts and all 31 village chiefs. A total of 15 Village Leaders participated in the meeting held in Pahauman on September 23, 2004. After a brief history of the LCSP by the ADP manager, the LCSP motivators presented the findings of the FE KPC survey using the local Dayak language. They presented the trends in the main project indicators by project’s interventions and comparison between health centres area as approriate. The motivators also facilitated discussions about selected issues raised by the results of the FE KPC. Senior LCSP and ADP staff also participated in the discussions and facilitated the development of a consensus by the Village Leaders on their commitment to continue specific activities to sustain the achievement of the LCSP. The FE Team was expected to act as observers throughout the meeting. The Declaration of Commitment of Village Chiefs is presented below. World Vision 141 LCSP Final Evaluation Commitment of Village Chiefs to Posyandus and Child Survival - Pahauman 23 September 2004 - For Posyandu  There shoud be collaboration between the village committee and the ADP in order to continue the Posyandu activities.  All the Village Chiefs should get the schedule of Posyandu day, so that they can help the preparation and facilitation of Posyandu.  The Village Chief should be involved in the community health education, for example use of Oralit for child with diarrhea, use of ITNs.  There shoud be collaboration between the village committee and the Health Center in order to continue the Posyandu activities  Village Chiefs should recruit cadres who are willing to work  The Midwife should be ready in the Posyandu during Posyandu day  Other activities should be held in Posyandu such as Arisan, and the money collected from these activities should fund the supplementary feeding program  The Village Chiefs should present on Posyandu day once in every 3 months (at least)  Ensure Cadres Re-generation by recruiting new cadres For Malaria prevention  Conduct health education on enviromental health  Promote ITN (village chief should be the role model for community)  There should be no more new sites for Dompeng or illegal gold mining For Diarrhea prevention  Conduct health education to promote use of boiled water for drinking  Have stock of Oralit (ORS packages) ready. The village chief will assist in monitoring the stock of Oralit in the Posyandu  Village chief will approve use of PKK funds for Supplementary Feeding in the Posyandu  Village chief will promote practices of handwashing before eating, washing the kitchen utensils, and use of clean water For ARI  Village chief will recommend mothers (of child with ARI) to seek treatment immediately For Immunization program  Every mother of under five children should have the KMS (Growth Monitoring Card) and Buku KIA (MCH Booklet)  The Village chief should give health education or to remind mother to bring KMS/Buku KIA For Vitamin A program  Village Chief will motivate the community to get Vitamin A for their under five children, but the FACILITIES should be available (health staff, supply of vitamin A) World Vision 142 LCSP Final Evaluation Annex 21. List of Participants in FE Meetings and Presentations Technical Overview Meeting 1. Dr. Abang Chaerudin (DHO-Head) 2. Sri Wahyuni (DHO_CDC) 3. Sophia Tjakre (DHO-MCH) 4. Abang Suhaimi (DHO-CDC) 5. Agustinus (DHO-CDC) 6. 2 other staff from DHO in charge of immunization 7. Elisabeth Tsia (PHO-MCH) 8. Manok (Head of Senakin Health Center) 9. Muhidin (Head of Pahauman Health Center) 10. FE Team (World Vision) Village Leaders Feedback Meeting 1. 17 Heads of Villages 2. Manok 3. Muhidin 4. Elisabeth Tsia (PHO) 5. Sophia Tjakre (DHO) 6. Agustinus (DHO) 7. FE Team (World Vision) District-level Meeting 1. Dr. Abang Chaerudin (DHO) 2. Ludis (Head of BAPPEDA – the place where we had the meeting) 3. Dr. Hendra (PHO) 4. One representative from the District Government 5. One representative from PKK-Family Welfare Program 6. One staff from Pancur Kasih 7. Kristiana (KPD) 8. 3 heads of Health Centers 9. KPC interviewers 10. + 10 cadres 11. + 10 staff from DHO (Sri Wahyuni, Marcella, Abang Suhaimi, Agustinus, etc) 12. FE Team (World Vision) Province--level Meeting 1. Dr. Isman (PHO-CDC) 2. Dr. Hendra (PHO-CDC) 3. Dr. Honggo (PHO-Planning) 4. + 10 more staff from the PHO 5. Manok and Muhidin (Health Centers) 6. Sophia Tjakre (DHO) 7. Agustinus (DHO-CDC) 8. One staff from Pancur kasih 9. Kristiana (KPD) 10. Erickson and Yohana (Urban Pontianak ADP) 11. FE Team (World Vision) World Vision 143 LCSP Final Evaluation Lessons Learned Workshop 1. Koesnalilah Mihardja National Family Welfare Program 2. Purjanto National Health Promotion (MOH) 3. Juwarsih Catholic Relief Services 4. Eddy Irwansyah Department of Internal Affairs (moderator) 5. Ina Hernawati MOH 6. Marina MOH 7. Karmelia Head of CDC Department-MOH 8. Agustin Project Concern InternationalI 9. Yullita Evarini CDC-MOH-Diarrhea Subivision 10. Andre Tanoe Catholic Relief Services 11. Arifin Tanaya MOH-Community Nutrition 12. Fatum CDC-MOH-ARI Subdivision 13. Asmaniar MOH-Immunization 14. F.J. Laihad CDC-MOH-Malaria Subdivision 15. Sutedjo MOH 16. Hermiyanti MOH-Family Health 17. Sri Durjati USAID 18. Azrul Azwar General Director of Community Health-MOH 19. drg. Oscar Primadi Head of West Kalimantan PHO 20. Dr. Isman Head of CDC-West Kalimantan PHO 21. Kristiana Koperasi Pancur Dangeri 22. Marc Debay Independent consultant 23. Lyndon Brown WVUS 24. James L. Tumbuan WVI 25. Wynn Flaten WVI 26. Mary Wangsaraharja WVI 27. Paulus WVI 28. Bekti Sulistyaningsih WVI 29. Edi Sianipar WVI 30. Birgita Soraya WVI 31. Esther Indriani WVI 32. Teresa Zakaria WVI 33. Mariani Aritonang WVI 34. Rainy Wijaya WVI 35. Donny Indra WVI 36. Bonar Panjaitan WVI 37. Herkulanus Sumardi WVI 38. Lina Monika WVI World Vision 144 LCSP Final Evaluation Annex 22. FE KPC Results for EPI Antigen-Specific Coverage 0% 20% 40% 60% 80% 100% Polio 1 Polio 2 Polio 3 Polio 4 DPT1 DPT2 DPT3 Card BCG Polio DPT Measles FIC Maternal recall (FIC only) Card documented Note: Non card-documented coverage is only presented for Fully Immunized Children. World Vision 145 LCSP Final Evaluation Annex 23. ITN Distribution Diagram Third line Second line First line Pontianak ADP Vestergaard Frandsen Community KSU Pancur Dangeri KPD Branch Credit Unions KPD Service Centers Shops Posyandu Cadres Other selling t POD KSM Source: Report on LCSP ITN distribution, LCSP, September 2004. World Vision 146 LCSP Final Evaluation Annex 24. Posyandu Monthly Report Data for Year 2003 Figure 1. Completeness of Reporting 0% 20% 40% 60% 80% 100% 120% Jan Feb Mar April Mei Juni Juli Agust Sept Okt Nov Des Mandor Senakin Pahauman Figure 3. Posyandu Attendance Rate for Infants 0% 20% 40% 60% 80% 100% 120% Jan Feb Mar April Mei Juni Juli Agust Sept Okt Nov Des Mandor Senakin Pahauman Figure 3. Posyandu Days with Immunization Services 0% 20% 40% 60% 80% 100% 120% Jan Feb Mar April Mei Juni Juli Agust Sept Okt Nov Des Mandor Senakin Pahauman)= World Vision 147 LCSP Final Evaluation Annex 25. Posyandu Monthly Report Form Infant Under 5 (0- <1 yr) (1- <5 yr) 1 2 3 B 4 L 5 6 7 c 8 B 9 L 10 11 12 13 14 15 16 17a 18 20 21 22 23 24 25 26 27 28 29 30 31 No. Posyandu Stock Early Stock Additional Usage 1 GMC 2 ORS 3 Iron Tablets (Fe) 4 ……………….. Weight scale condition: good/damaged/………… Cadre's Name Signature Presence 1. Y/N 2. Y/N 3. Y/N 4. Y/N ( ……...….......………. ) 5. Y/N a. Health Center b. Head of Village/Dusun c. Posyandu Archieve d. ……….. Original and copy for: WVI & Landak DHO- March 2004 Acknowledged : ………………., on …….………… Health Worker Posyandu Cadre Remark : …………………..….……………………. Late Stock Remark Date of the next Posyandu day YAD …………………… ……………………………………………………. ……. …………………… Number of cross-sectional worker present during Posyandu activity this month PLS …………….. psn Number of cadres present during Posyandu activity this month KDR …………….. psn Number of ORS Sachets distributed by cadres this month ORL …………….. sac Number of health worker present during Posyandu activity this month Nakes …………….. psn Number of pregnant women who died during this last month IHM …………….. psn Number of post-partum mothers who died during this last month INM …………….. psn Number of delivery helped by a health worker (doctor, midwife, nurse) LTK …………….. psn Number of neonatus who died during this last month NM …………….. psn Number of delivery helped by an untrained TBA (familiy, etc…) LDTT …………….. psn Number of delivery helped by a trained TBA LDT …………….. psn Number of post-partum mothers (0-40 days) who have received Vitamin A NVA …….……….. Psn 19 Number of pregnant women and post-partum mothers reffered to the Village Birthing Post/Health Center/Hospital RIH Preg. Women: …….. RIM Post-partum : …….. 17b Number of pregnant women who came to the Posyandu and received iron tablets for the first (Fe1) or third time (Fe3) Fe 1 Fe 1 = ……....psn Fe 3 Fe 3 = ……....psn Number of all pregnant women in this Posyandu area/Target SIH .…………….. psn Number of pregnant women who came to the Posyandu and received iron tablets ITB .…………….. psn Number of infants and under-fives immunized at the Posyandu IM Number of infants and under-fives who received Vitamin A (Feb & Aug) Vit A Number of infants and under-fives who had fever during this last month DMM Number of infants and under-fives referred to the Health Center during this last month R Number of infants and under-fives who had diarrhea during this last month DR Number of infants and under-fives who died during this lst month MGL Number of infants and under-fives whose weight stayed the same/decreased this month T Number of infants and under-fives whose weight is newly under the red line BGM Number of other infants and under-fives whose weight is under the red line Number of infants and under-fives weighed this month but not last month O Number of infants and under-fives whose weight increased this month N Number of all infants and under-fives registered who own a GMC/MCH Booklet K Number of infants and under-fives who came and are wighed for the first time D Number of other infants and under-fives weighed this month No. A. Posyandu Data Code Number of all infants and under-fives living in this Posyandu area/Target S POSYANDU MONTHLY REPORT FORM Posyandu Name : ………….……………….. Desa/ Dusun : ……………….……… Posyandu Supervisor : …………………………... Day/Month/Year : ……………………… World Vision 148 LCSP Final Evaluation Annex 26. Organizational Capacity Assessment Results and Questionnaire OCA Results for the three Health Centers and the three Cadres Associations in Mandor, Senakin and Pahauman and for the ADP Pontianak, March 2004 HEALTH CENTERS Cadres Association ADP Senakin Pahauman Mandor Senakin Pahauman Mandor Pontianak I E I E I E I E I E I E I E A. Vision & Mission I. Vision & Mission 0 0 0 0 0 0 0 0 2 2 0 0 1 1 B . Organization structure, base & legal status. II. Structure 3 2,7 3 2,7 3 2,7 1 0,3 1 0,3 3 1,3 3 III. Constitution 2 2 2 0 0 0 2 2,7 IV. Legal status 3 3 3 0 0 1 3 C. Management V. Planning 2 2 2 1,4 3 1,8 0 0 1 0,2 2 0,8 3 2,2 VI. Monev 3 0 2 0 0 0 2 VII. Problem solving 3 3 2 0 0 2 2 VIII. Finance 1 2 2 0 0 0 2 IX. HRD 1 0 0 0 0 0 2 D. Leadership X. Decision & delegation 3 3 3 3 3 3 2 2 3 3 3 3 3 3 E. Ownership XI. V&M Suitability 3 2,3 3 3 0 1,7 3 3 3 3 3 3 2 XII. Roles 3 3 3 3 3 3 3 2,7 XIII. Sacrifice 1 3 2 3 3 3 3 F. Equality XIV. Race, ethnic & religion 3 3 3 3 3 3 3 3 3 3 3 3 3 3 XV. Openness 3 3 3 3 3 3 3 G. Communication XVI. Meetings 3 2,3 3 2,5 3 1,7 0 0,3 0 0 2 1,3 3 XVII. R&R 3 2 2 0 0 2 2 2,3 XVIII. Info Dissemination 1 0 0 1 0 0 2 H. Networking XIX. Networking 2 2 2 2 2 2 1 1 1 1 2 2 3 3 I. Fundraising XX. Fundraising 1 1 0 0 1 1 0 0 0 0 1 1 2 2 Average score : 2,20 2,03 2,00 1,96 1,95 1,88 1,00 1,07 1,15 1,39 1,65 1,71 2,45 2,43 Note: I: Individual item score; E: Average score per lettered theme. World Vision 149 LCSP Final Evaluation OCA Questionnaire Name of organization : Date of interview : Name of interviewer : Place of interview : Name of respondents : 1. 5. 2. 6. 3. 7. 4. 8. VISION & MISSION STATEMENT Question A.1 : Does your organization possess a vision and mission statement Is the document available? Record vision and mission statement… Vision: Mission: Score I. Vision & Mission Statement Indicator 0 No written vision and mission statement 1 Available written vision and mission statement, but no member present able to mention them 2 Available written vision and mission statement, but only part of the members present able to mention them 3 Available written vision and mission statement; all members able to mention them STRUCTURE, BASE AND LEGALITY OF ORGANIZATION Question B.1 : Does your organization possess a organizational structure with a clear job delegation (Look at organizational structure chart..) Score II. Organizational Structure Indicator 0 No organizational structure nor clear job delegation 1 Structure only composed of head, secretary, treasurer and members without defined roles nor responsibilities. 2 Structure is more complete with a clear lines of command without lines of coordination resulting in overlapping roles and responsibilities. 3 All members are written in the structure with clear lines of command and coordination, and clear defined roles and responsibilities. Question B.2 : Does your organization possess a constitution? Score III. Constitution Indicator 0 No constitution 1 Existing constitution, but not documented 2 Existing and documented constitution but nor socialized to all members 3 Existing and documented constitution, every member possesses a copy World Vision 150 LCSP Final Evaluation Question B.3: Does your organization possess a legal status documented in a formal letter (Memorandum Of Understanding or letter issued by the local government)? Score IV. Legal Status indicator 0 No legal document 1 No written document, but known by the local government 2 Existing legal document, but no or limited coordination with the local government 3 Existing legal document and good coordination MANAGEMENT Question C.1: Does your organization have a good planning? (Look at organization planning document..) Score V. Planning Indicator 0 No document 1 Present, but planning is nor implemented 2 Present and implemented but not routinely 3 Present and implemented routinely (consistently) Question C.2 : Does your organization have a good monitoring and evaluation system? (Look at document..) Score VI. Monitoring and Evaluation Indicator 0 None 1 Present and implemented 2 Present and implemented but not routinely 3 Present and implemented routinely (consistently) Question C.3 : Are problems in your organization solved effectively and efficiently (correctly)? (effective= the goal is reached and the process involves available resources) Score VII. Problem Solving Indicator 0 Organization does not possess the ability for problem solving 1 Problems are solved, but not with the right process 2 There are still unsolved problems, but the right process is implemented 3 Existing problems are solve using the right process Question C.4 : Does your organization use funding effectively and at a low cost? Score VIII. Funding Utilization Indicator 0 Not at all 1 Effectively but at a high cost 2 Effectively at a low cost, but without optimal results 3 Effectively at a low cost, and with optimal results World Vision 151 LCSP Final Evaluation Question C.5 : Does your organization have human resources development (HRD) programs which are beneficial for the organization and suitable with the interest of the staff involved? Score IX. Human Resources Development Indicator 0 No HRD programs 1 Existing programs but not well implemented 2 Existing HRD programs, beneficial for the organization but not suitable with the interest of the staff 3 Existing HRD programs, beneficial for the organization and suitable with the interest of the staff LEADERSHIP Question D.1 : 1. How are decisions made in your organization? 2. Is decision made involving other work units? 3. Is there a job delegation? Score X. Decision Making and Job Delegation Indicator 0 Decision making does not involve other work units. 1 Decision making does not involve other work units, but there is job delegation. 2 Decision making involves other work units, but there is no job delegation. 3 Decision making involves other work units, and there is job delegation. OWNERSHIP Question E.1 : Are the vision and mission of your organization suitable with yours? Score XI. Vision and Mission Suitability Indicator 0 No member present think that the vision and mission are suitable 1 Less than 1/3 of the members present think that the vision and mission are suitable 2 1/3 to 2/3 of the members present think that the vision and mission are suitable 3 More than 2/3 of the members present think that the vision and mission are suitable Question E.2 : Do you feel that the role you play in your organization useful for other people? Score XII. Role Indicator 0 No member present feels the role they play useful for other people 1 Less than 1/3 of members present feel the role they play useful for other people 2 1/3 to 2/3 of members present feel the role they play useful for other people 3 More than 2/3 of members present feel the role they play useful for other people Question E.3 : Have you ever worked overtime even though you know that there is no compensation? Score XIII. Sacrifice Indicator 0 No member ever worked overtime 1 Less than 1/3 of members present ever worked overtime 2 1/3 to 2/3 of members present ever worked overtime 3 More than 2/3 of members present ever worked overtime World Vision 152 LCSP Final Evaluation EQUALITY Question F.1 : Does your organization consider race, ethnic and religion difference in providing services? Score XIV. Race, Ethnic and Religion Indicator 0 Often 1 It does happen 2 Sometimes 3 Never Question F.2 : Is your organization open for advice, criticism, and additional power and material from various sources? Score XV. Openness Indicator 0 Never 1 A little (less than 1/3) 2 Moderate (1/3 to 2/3) 3 A lot (more than 2/3) COMMUNICATION Question G.1 : Are member meetings held regularly? Score XVI. Meetings Indicator 0 No 1 < one in 3 months 2 One every 1-3 months 3 Regularly, at least once a month Question G.2 : Is there an R&R (Recording & Reporting) for every activity? Score XVII. R&R Indicator 0 No 1 Only if required 2 There is R&R, but not routinely 3 There is R&R routinely, the staff know the importance of R&R for every activity Question G.3 : Does your organization possess a way to disseminate information about its activities to outsiders (bulletin, etc) ? Score XVIII. Information Dissemination Indicator 0 No 1 There is, but informal 2 There is, formal, not routinely 3 There is, formal, distributed using various media World Vision 153 LCSP Final Evaluation COLLABORATION NETWORK Question H.1: Does your organization has a collaboration with another one ? If yes, with which organization(s) this collaboration is done? How are the meetings between organizations conducted? Formal or informal? If formal, is there a routine schedule for those meetings? For how long these collaborated programs have been going on? Score XIX. Collaboration Network Indicator 0 No collaborating network 1 Existing collaboration network with at least one other organization The quality f the network is not good yet, for example: - Informal meetings (only by phone) - Duration of the collaboration is less than 6 months 2 Existing collaboration network with at least one other organization The quality f the network is good enough, for example: - Formal meetings but unscheduled - Duration of the collaboration is 6 to 12 months 3 Collaboration network with more than one organization. The quality f the network is good, for example: - Scheduled formal meetings - Duration of the collaboration is more than 1 year FUNDRAISING Question I.1 : Where does your organization’s operational funding comes from? Score XX. Fund Raising Indicator 0 No external donator 1 Contribution from 1 external donator 2 Contribution from 2 external donators 3 Contribution from 3 external donators or more World Vision 154 LCSP Final Evaluation Annex 27. BEHAVE Framework for Use of ITNs PRIORITY GROUP BEHAVIOR KEY FACTORS ACTIVITIES In order to help To We will focus on Through Pregnant women and children under fives Sleep under an ITN every night Increasing knowledge on the dangers of malaria Development and revision of IEC materials on malaria regularly: stickers, posters, leaflets, banners, billboards Increasing knowledge on the cause of malaria Development of booklets on basic health education for Posyandu beneficiaries Increasing knowledge on the benefits of ITNs (to protect people from the nuisance of insects as well as specifically from malaria) Development of funny/attractive advertisements on malaria and ITNs for newspapers, radio, and television Making ITNs available and affordable Organizing mass campaigns at specific times, including performance arts/concerts involving the community with themes and messages on malaria Provision of discount vouchers, with a specific timing (when cash is available in the community) Building a network with local distributor/NGOs and ITN manufactures Development of short movies on malaria and ITNs to be shown at the dusun level/during Posyandu activities This framework was developed on April 29, 2004 as a follow up to the doer-nondoer assessment done through April. World Vision 155 LCSP Final Evaluation Annex 28. Training Jointly Facilitated by LCSP DATE AREA TRAINING TOPIC FACILITATOR PARTICIPANT May 19, 2001 Mandor & Sengah Temila Training on Malaria for Health Staff PHO, DHO, Dr. F. J. Laihad (MOH RI) 43 government health staff 10 LCSP staff June 7-9, 2001 Sengah Temila (Senakin) Posyandu Cadre Training HC Staff 25 Cadres June 12-14, 2001 Mandor Posyandu Cadre Training HC Staff 25 (Cadres, SHGs, Village Chief) June 25, 2001 Sengah Temila (Senakin) Shopkeeper Training HC Staff 11 Shopkeeper September 11-13, 2001 Mandor Posyandu Cadre Training HC Staff 89 (Cadres, SHGs, Village Chief) June 28-30, 2001 Sengah Temila (Senakin) Posyandu Cadre Training HC Staff 28 Cadres July 23-25, 2001 Sengah Temila (Senakin) Posyandu Cadre Training HC Staff 33 Cadres August 10-12, 2001 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 29 Cadres January 17-19, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 30 Cadres January 18, 2002 Mandor Shopkeeper Training HC Staff 20 Shopkeepers January 28, 2002 Mandor Shopkeeper Training HC Staff 11 Shopkeepers January 30, 2002 Sengah Temila (Senakin) Shopkeeper Training HC Staff 16 Shopkeepers February 1, 2002 Sengah Temila (Senakin) Shopkeeper Training HC Staff 7 Shopkeepers February 2, 2002 Sengah Temila (Senakin) Shopkeeper Training HC Staff 25 Shopkeepers February 19, 2002 Mandor Shopkeeper Training HC Staff 13 Shopkeepers February 21, 2002 Mandor Shopkeeper Training HC Staff 12 Shopkeepers February 26, 2002 Mandor TBA Training I Midwives 17 TBAs March 4, 2002 Sengah Temila (Senakin) Shopkeeper Training HC Staff 9 Shopkeepers March 6, 2002 Mandor TBA Training I Midwives 25 TBAs March 7, 2002 Sengah Temila (Senakin) Shopkeeper Training HC Staff 25 Shopkeepers March 11-13, 2002 Sengah Temila (Senakin) Posyandu Cadre Training HC Staff 33 Cadres March 11-13, 2002 Sengah Temila (Senakin) Posyandu Cadre Training HC Staff 34 Cadres March 12-14, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 37 Cadres March 12-14, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 33 Cadres March 20, 2002 Mandor TBA Training I Midwives 26 TBAs March 20, 2002 Mandor TBA Training I Midwives 17 TBAs March 21, 2002 Mandor Shopkeeper Training HC Staff 14 Shopkeepers March 27, 2002 Mandor TBA Training II Midwives 17 TBAs April 2, 2002 Mandor TBA Training II Midwives 40 TBAs April 2-4, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 26 Cadres April 3-5, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 28 Cadres April 18, 2002 Sengah Temila (Senakin) TBA Training I Midwives 27 TBAs April 26, 2002 Mandor TBA Training II Midwives 4 TBAs April 30, 2002 Mandor Shopkeeper Training HC Staff 10 Shopkeepers April 30, 2002 Mandor Shopkeeper Training HC Staff 11 Shopkeepers May 1, 2002 Mandor TBA Training III Midwives 39 TBAs May 2-4, 2002 Sengah Temila (Pahauman) Posyandu Cadre Training HC Staff 40 Cadres World Vision 156 LCSP Final Evaluation DATE AREA TRAINING TOPIC FACILITATOR PARTICIPANT May 3, 2002 Sengah Temila (Senakin) TBA Training I HC Staff 12 TBAs May 17-18, 2002 Mandor Posyandu Cadre Refresh HC Staff 45 Cadres May 31, 2002 Sengah Temila (Senakin) TBA Training II Midwives 27 TBAs June 1, 2002 Mandor TBA Training IV A Midwives 36 TBAs June 3, 2002 Sengah Temila (Senakin) TBA Training I Midwives 17 TBAs June 5, 2002 Sengah Temila (Pahauman) TBA Training I Midwives 26 TBAs June 11-13, 2002 Mandor Posyandu Cadre Training HC Staff 34 Cadres June 17-18, 2002 Mandor POD Cadre Training HC Staff 22 Cadres June 19, 2002 Mandor TBA Training IV Midwives 12 TBAs June 21, 2002 Mandor TBA Training III Midwives 24 TBAs June 21, 2002 Pahauman Shopkeeper Training HC Staff 8 Shopkeepers June 26-27, 2002 Mandor Posyandu Cadre Refresh HC Staff 54 Cadres June 28, 2002 Sengah Temila (Senakin) TBA Training II Midwives 18 TBAs June 29, 2002 Sengah Temila (Senakin) TBA Training I Midwives 18 TBAs June 29, 2002 Sengah Temila (Pahauman) TBA I & II Midwives 24 TBAs July 2, 2002 Mandor TBA Training IV B Midwives 41 TBAs July 8-13, 2002 Pontianak IMCI Training Provincial Health Office 6 Midwives, 1 DHO staff July 16 – 20, 2002 Intern KPC Survey Training for Mid Term Evaluation LCSP Team 40 participants: community member, HC, LCSP/ADP July 3-4, 2002 Sengah Temila (Senakin) POD Training HC Staff 15 Cadres August 21, 2002 Mandor TBA Training IV HC Staff 26 Cadres August 27, 2002 Sengah Temila (Pahauman) TBA Training III Midwives 29 Cadres August 29, 2002 Sengah Temila (Pahauman) Shopkeeper Training HC Staff 14 Shopkeepers September 20, 2002 Sengah Temila (Pahauman) TBA Training Midwives 15 TBAs September 23 – 28, 2002 Pontianak IMCI Training Provincial Health Office 18 HC Staff, 2 LCSP Staff October 8, 2002 Sengah Temila (Pahauman) TBA Training III Midwives 27 TBAs October 8, 2002 Sengah Temila (Senakin) TBA Training III Midwives 27 TBAs October 17, 2002 Sengah Temila (Senakin) TBA III Midwives 22 TBAs October 14-26, 2002 Landak Analyst Training On Malaria Provincial Health Office 13 HC Staff November 1-2, 2002 Pahauman dan Senakin TOT: Participatory Learning Family Planning Committee (Perkumpulan Keluarga Berencana Indonesia) 24 HC Staff November 1, 2002 Mandor TBA Refresher Training HC staff 59 TBAs November 2, 2002 Mandor TBA Selection for dukun kit Midwives 64 TBAs November 5, 2002 Mandor Cadre Meeting HC Staff 72 Cadres November 7, 2002 Sengah Temila (Senakin) TBA Training III Midwives 19 TBAs November 9, 2002 Sengah Temila (Pahauman) TBA Training IV Midwives 45 TBAs November 14-16, 2002 Sengah Temila (Pahauman) Cadre Training HC Staff 33 Cadres November 25, 2002 Sengah Temila (Senakin) TBA Training II Midwives 20 TBAs January 16-18, 2003 Sengah Temila (Pahauman) Cadre Training HC Staff 24 Cadres January 17, 2003 Mandor Shopkeeper Training HC Staff 23 Shopkeepers January 22, 2003 Sengah Temila (Pahauman) Cadre Meeting HC Staff 172 Cadres January 24-25, 2003 Sengah Temila (Senakin) Cadre Training HC Staff 30 Cadres World Vision 157 LCSP Final Evaluation DATE AREA TRAINING TOPIC FACILITATOR PARTICIPANT January 29, 2003 Sengah Temila (Pahauman) TODA/TOMA Meeting HC Staff 20 TODAs/TOMAs January 29, 2003 Mandor TODA/TOMA Meeting HC Staff 25 TODAs/TOMAs January 30, 2003 Sengah Temila (Pahauman) TBA Selection for Dukun Kit Midwives 24 TBAs January 31, 2003 Mandor Shopkeeper Training HC Staff 12 Shopkeepers February 3, 2003 Mandor Develop Village Drug Post (POD) HC Staff POD Cadres Mdr, POD Cdres Snk, Village Chief February 4, 2003 Sengah Temila (Pahauman) TBA Selection for Dukun Kit Midwives 20 TBAs February 7, 2003 Sengah Temila (Pahauman) TBA Selection for Dukun Kit Midwives 17 TBAs February 11, 2003 Sengah Temila (Senakin) TBAs Selection for Dukun kit Midwives 21 TBAs February 14, 2003 Mandor Health Education for community TODA/TOMA, LCSP Staff 30 community members of Pongo Village (most of men) February 15, 2003 Sengah Temila (Pahauman) TBAs Meeting Midwives 62 TBAs February 18, 2003 Sengah Temila (Pahauman) Health Education for High School Student HC Staff, LCSP Staff 69 Senior High Sch. Student 68 Junior High Sch. February 20, 2003 Mandor Health Education for Community TODA/TOMA, LCSP Staff 40 mothers that attend posyandu Sumsum February 21, 2003 Mandor Health Education for Community TODA/TOMA, LCSP Staff 60 mothers that attend posyandu Keramas; 20 mothers that attend posyandu Atong February 22, 2003 Sengah Temila (Senakin) TBAs selection for Dukun Kit Midwives 15 TBAs March 5, 2003 Sengah Temila (Pahauman) Health Education for Senior High HC Staff, LCSP Staff 113 Senior High school Students March 10, 2003 Sengah Temila (Senakin) Health Education for Junior High HC Staff, LCSP Staff 147 Junior High school Students March 15, 2003 Sengah Temila (Pahauman) TBAs Meeting Midwives 55 TBAs March 17, 2003 Sengah Temila (Senakin) TBA Selection for Dukun Kit Midwives 18 TBAs March 22, 2003 Sengah Temila (Senakin) TBAs Meeting Midwives 18 TBAs April 11, 2003 Sengah Temila (Senakin) Shopkeeper Training HC Staff 14 Shopkeepers April 12, 2003 Sengah Temila (Senakin) Community Health Education on Child Health LCSP & HC Staff Community Member of Raba Sekuap Sub Village ( 100 org) April 21 – May 3, 2003 Landak Analyst Training on Malaria Provincial Health Office 10 HC Staff April 29 – May 2, 2003 Landak Nutrition Training Provincial Health Office 28 HC Staff, 5 LCSP Motivator May 28, 2003 Sengah Temila (Senakin) Cadre Meeting HC Staff 113 cadres May 31, 2003 Sengah Temila (Pahauman) Shopkeeper Training HC Staff 23 shopkeepers June 19, 2003 Sengah Temila (Senakin) TBA Training Midwives 16 TBAs June 24, 2003 Sengah Temila (Pahauman) TBA Training Midwives 15 TBAs June 24, 2003 Sengah Temila (Pahauman) Cadre Meeting HC Staffs 187 cadres July 8, 2003 All area (Mdr + Senakin + Phn) Cadre Meeting HC Chief and LCSP Team 160 cadres July 10, 2003 Pahauman Health Education on Malaria LCSP Team ± 500 Catholic Youth members July 17, 2003 Mandor POD Refresher Training HC Chief and LCSP Team 7 POD Cadres, 14 Village/Sub Village Chiefs July 28, 2003 Anjungan and Toho MED Training ADP Pontianak And LCSP Team 20 SHG members July 29, 2003 Sengah Temila (Pahauman) TBA Training HC Staff 30 TBAs July 31 – Aug 1, 2003 Mandor MCH Booklet Training for cadre HC Staff 20 cadres World Vision 158 LCSP Final Evaluation DATE AREA TRAINING TOPIC FACILITATOR PARTICIPANT Aug 1, 2003 Sengah Temila (Senakin) Training for mother of malnourished child HC Staff and LCSP Team 24 mothers Aug 6, 2003 Mandor Cadre Meeting LCSP Staff and Lecturer from UKSW Salatiga 85 cadres Aug 6, 2003 Sengah Temila (Senakin) TBA Selection for Dukun Kit HC Staff 11 TBAs Aug 7, 2003 Sengah Temila (Pahauman) MED Training ADP Pontianak and LCSP Team 30 SHG members Aug 9, 2003 Mandor Cadre Training about TOGA Lecturer from UKSW Salatiga 5 cadres Aug 27, 2003 Sengah Temila (Pahauman) MCH Booklet Training for Cadre HC Staff 33 cadres Aug 28, 2003 Sengah Temila (Pahauman) TBA Training HC Staff 18 TBAs Aug 29, 2003 Sengah Temila (Pahauman) Cadre Meeting HC Staff and LCSP Team 136 cadres Sept 8 – 10, 2003 Intern LQAS Survey Training for Third Annual Review LCSP LCSP – ADP Team 35 community member, 12 LCSP –ADP Team, 1Health Staff Oct 7 – 10, 2003 Provincial CBHIS Training Consultants, MOH, LCSP Core Team 17 Health and planning government services, 12 wv staffs December 11, 2003 Sengah Temila (Senakin) Cadre Meeting HC Staff and LCSP Team 95 cadres December 19, 2003 Mandor Cadre Training HC Staff and LCSP Team 11 cadres January 27, 2004 Sengah Temila (Pahauman) Cadre Meeting HC Staff and LCSP Team 152 cadres February 6, 2004 Mandor POD Meeting HC Staff and LCSP Team 4 POD cadres, 5 village chiefs, 7 sub village chiefs, HC chief, LCSP February 6, 2004 Sengah Temila (Senakin) Cadre Meeting HC Staff and LCSP Team 119 cadres February 19, 2004 Sengah Temila (Senakin) POD Meeting HC Staff and LCSP Team 3 POD Cadres, 3 village chiefs, 4 chief of dusuns February 9, 2004 Sengah Temila (Pahauman) Shopkeeper Training HC Staff and LCSP Team 25 Shop keeper February 13, 2004 Sengah Temila (Pahauman) Shopkeeper Training HC Staff and LCSP Team 12 shop keeper February 17, 2004 Sengah Temila (Pahauman) Shopkeeper Training HC Staff and LCSP Team 5 shop keeper March 17 – 18, 2004 Mandor Cadre Training HC Staff and LCSP Team 54 cadres March 8 – 13, 2004 West Kalimantan IMCI TOT Training Depkes & Provincial IMCI Trainer 8 Landak Health Staffs, 12 Health Staffs from other districts in west Kalimantan, 1 LCSP Staff March 23 – 24, 2004 Intern ADP SAP Workshop Child Health Adv. & LCSP Team 9 LCSP Staffs & 11 ADP Staffs March 25, 2004 Intern ADP Doer – Non Doer Workshop[ Child Health Advisor & PO LCSP 7 LCSP Staffs and 3 ADP Staffs April 29, 2004 Intern LCSP BCC Core Team 11 LCSP Staffs May 7 – 8, 2004 Sengah Temila (Senakin) Cadre Training HC Staff and LCSP Team 27 Posyandu Cadres May 12, 2004 All area Training Organisasi KPD Staf Members of Cadres Associations (6 in Mandor, 3 in Pahauman Union, 11 in Senakin) May 14, 2004 Sengah Temila (Pahauman) Vitameal Training LCSP Team 50 posyandu cadres May 15, 2004 Toho TBA Training HC Staff and ADP – LCSP Team 6 TBAs May 18, 2004 Takong Cadre meeting HC Staff and ADP -- LCSP Team 35 Posyandu Cadres May 25, 2004 Toho Cadre Meeting HC Staff and ADP -- LCSP Team 28 Posyandus Cadres May 25, 2004 Takong TBA Training HC Staff and LCSP Team 15 TBAs May 27, 2004 Mandor Cadre Meeting HC Staff and LCSP Team 86 Posyandu Cadres May 27, 2004 Sengah Temila (Senakin) TBA Refreshing Training HC Staff and LCSP Team 20 TBAs World Vision 159 LCSP Final Evaluation DATE AREA TRAINING TOPIC FACILITATOR PARTICIPANT May 28, 2004 Sengah Temila (Pahauman) Cadre Meeting HC Staff and LCSP Team 94 Posyandu Cadres June 4, 2004 Sengah Temila (Pahauman) Cadre Meeting LCSP Team 35 posyandu Cadres June 5, 2004 Sengah Temila (Senakin) Cadre Meeting LCSP Team 94 Posyandu Cadres June 15, 2004 Toho TBA Training HC Staff and ADP - LCSP Team 8 TBAs June 16, 2004 Takong Cadre Meeting HC Staff and ADP - LCSP Team 23 Posyandu Cadres June 19, 2004 Sengah Temila (Senakin) Cadre Meeting for Vitameal HC Staff and ADP - LCSP Team 27 Posyandu Cadres June 22, 2004 Sengah Temila (Pahauman) TBA Refreshing Training HC Staff and LCSP Team 89 TBAs June 25, 2004 Toho Cadre Meeting HC Staff and ADP -- LCSP Team 27 Posyandu Cadres June 25, 2004 Takong TBA Training HC Staff and ADP -- LCSP Team 15 TBAs July 2 – 4, 2004 Internal LCSP – ADP Pontianak LCSP – ADP Meeting ADP Manager, LCSP Team Leader 27 LCSP/ADP Pontianak staffs July 8, 2004 Sengah Temila (Pahauman) Health Education LCSP Team ±200 Catholic Youth Members July 15, 2004 Toho TBA Training HC Staff and LCSP Team 9 TBAs July 16, 2004 Takong Cadre Training HC Staff and LCSP -- ADP Team 36 Cadres July 15, 2004 Sengah Temila (Senakin) Nutrition Training HC Staff and LCSP Team 52 mothers of underweight children from Aur, 10 posyandu cadres July 17, 2004 Sengah Temila (Senakin) Nutrition Training HC Staff and LCSP Team 56 mothers of underweight children from Andeng, 13 posyandu cadres July 20, 2004 Sengah Temila (Senakin) Nutrition Training HC Staff and LCSP Team 40 mother of under weight children, 12 Posyandu cadres July 23, 2004 Sengah Temila (Senakin) Nutrition Training HC Staff and LCSP Team 49 mothers of underweight children from Tonang, 7 Posyandu Cadres July 24, 2004 Sengah Temila (Senakin) Nutrition Training HC Staff and LCSP Team 25 mother of underweight children from Senakin, 7 Posyandu Cadres July 26 – 29, 2004 Pontianak Behavioral Change Communication Training Consultant: AED 14 LCSP, 3 ADP, 4 staff from other ADPs in West Kalimantan, 5 HC Staff, 5 NGO staff, 3 University Lecturers, 1 parish staff July 26, 2004 Toho Cadre Meeting HC Staff and LCSP Team 27 Posyandu Cadres July 26, 2004 Takong TBA Training HC Staff and LCSP Team 14TBAs Aug 5 – 7, 2004 Sanggau Outdoors Management Training: Communication and Leadership Consultant: X-care Consultainment 15 ADP Staffs, 14 LCSP Staffs Aug 16, 2004 Takong Cadre Meeting HC Staff and ADP -- LCSP Team 38 Posyandu Cadre Aug 19, 2004 Toho TBA Training HC Staff and LCSP Team 10 TBAs Aug 23 – 28, 2004 Intern KPC Survey LCSP Team 27 community members, 3 ADP staffs Aug 25, 2004 Takong TBA Training HC Staff and ADP Team 12 TBAs Sept 6 – 11, 2004 Pontianak IMCI Training Provincial and District Health Care Staff 13 HC Staff, 1 LCSP Staff Source: LCSP, September 2004. World Vision 160 LCSP Final Evaluation Annex 29. Summary Data on Training Jointly Facilitated by LCSP Number of training events, total participants and number of participants per fiscal year by type of training TYPE OF TRAINING Events FISCAL YEAR Total 00 01 02 03 04 Community health workers Posyandu Cadre (3 days) 22 89 140 295 87 92 703 Shopkeeper(1 day) 22 11 195 72 42 320 TBA 11 185 90 10 285 POD Cadre (2 days) 2 37 37 Other community members Village/sub-village chief, culture stakeholder 45 45 Cadres Associations Members (1 day) 1 20 20 Mothers of underweight child (1 day) 5 222 222 Health workers HC Staff (Malaria; 1 day) 1 43 43 HC Staff (IMCI – 7 days) 3 25 13 38 HC Staff (IMCI TOT, 7 days) 1 20 20 HC Staff (Malaria Microscopy, 12 days) 2 23 23 Health & Planning Govt. Services (CBHIS) 1 17 17 8 participants from Landak District and 12 from other Health Centers. World Vision 161 LCSP Final Evaluation Annex 30. Posyandu Cadre Training Curriculum No. Topics Sub-topics Objectives Methods Duration 1 Introduction and schedule Pre-test  Names of participants  Task distribution  Hopes of participants  Objectives and schedule  Person in charge and his/her duties  Ice breaking  Participants can mention objectives  Participants can follow the schedule Explanation, games, brainstorming 60 minutes 2 Introduction to LCSP  World Vision & ADP  5 key interventions  Involvement of Posyandus  Participants know about WVI and Pontianak ADP  Participants know the purpose of LCSP  Participants know the 5 key interventions  Participants know the involvement of Posyandus Explanation, questions & answers 30 minutes 3 Duties of Cadres 5 tables activities  Cadres’ duties on D-, D and D+  Problems often faced by cadres  Minimal service provided by Posyandus  Description of 5 tables  The flow of the 5 tables  Problem solving  Participants can mention the duties of cadres  Participants can mention the minimal service  Participants can mention the 5 tables activities  Participants are able to perform the 5 tables roles Group discussion, Plenary presentation, explanation, role play, brainstorming 60 minutes 4 Filling KMS/GMC and SKDN/ Monthly Report (1)  The use of KMS/GMC  How to fill into the KMS/GMC  How to fill into the SKDN/Monthly Report  Participants can mention the use of KMS/GMC  Participants can fill into the KMS/GMC correctly  Participants can fill into the SKDN/Monthly Report correctly Explanation, practice 120 minutes World Vision 162 LCSP Final Evaluation No. Topics Sub-topics Objectives Methods Duration 5 Read and fill into the MCH booklet (1)  Description and use of MCH booklet  Advantages of MCH booklet  Information and notes in the booklet  Participants can fill into the MCH booklet correctly  Participants can mention the information from the booklet  Participants can understand the information from the booklet and inform the target group Explanation, practice, brainstorming 90 minutes 6 Infectious diseases in infants and under-fives  Diarrhea  ARI  Malaria  Participants can explain the cause of the diseases  Participants know the danger signs of the diseases  Participants know the management of the diseases Explanation, discussion, practice 60 minutes 7 Immunization  Types of immunization  Benefits of each immunization  Immunization schedule  Procedure of immunization  Participants can explain the use of immunization  Participants can mention the types of immunization  Participants can mention the immunization schedule Explanation, discussion 60 minutes 8 Post-test Plan of action Closing  Planning of Posyandu Activities  Participants are able to plan activities for Posyandus for the following 3 months Group discussion 60 minutes World Vision 163 LCSP Final Evaluation Annex 31. POD Cadre Training Curriculum No. Domain Topics Duration 1 Function of POD  Duties and function of POD cadres  Criteria to be a POD cadre  Interest from drug selling  Relation between POD and health institutions 30’ minutes 2 Introduction of medicine  Forms of medicine  Types of medicine provided at PODs  How to store and handle drugs properly  Identify damaged drugs  How to write on the sticker (instruction)  How to deliver the drugs  Drug dosage 90 minutes 3 Facing sick people  Effective history taking  Make diagnosis  Decide on the need to refer  Drug decision  Deliver drugs and explain how to use it  Referral 90 minutes 4 Malaria  Etiology  Signs and symptoms  Management of malaria  Types of anti malaria drugs, their uses and dangers  Prevention of malaria  Insecticide Treated Nets  Relation with anemia 120 minutes 5 Diarrhea  Etiology  Signs and symptoms  Management of diarrhea  Types of anti diarrhea drugs, their uses and dangers  Prevention of diarrhea 90 minutes 6 ARI  Etiology  Signs and symptoms  Management of ARI  Types of drugs for ARI, their uses and dangers  Prevention of ARI  Transmission of TB  Dangers of TB  Period of therapy 90 minutes 7 Ascariasis  Etiology  Signs and symptoms  Management of Ascariasis  Types of drugs for Ascariasis, their uses and dangers  Prevention of ascariasis 60 minutes 8 Recording and reporting  Report needed, its form and use  How to record and report  Submit and sore reports 60 minutes 9 Procurement of drugs  Procurement planning  Purchasing 60 minutes World Vision 164 LCSP Final Evaluation Annex 32. Child Survival Sustainability Assessment and Action Plan Vision Formation Dimension 1: Child Health The increase of community knowledge on health and decrease of contagious diseases with proper health facilities available in all ADP areas. Dimension 2: Organization In year 2009, every organization operating in ADP areas will be able to contribute in improving the quality of mother and child health services. Dimension 3: Community In year 2009, the community will have the awareness and capacity to allocate income for health purposes, including environmental health supported with adequate basic health services from the government. SWOT Analysis Dimension I: CHILD HEALTH A. Strength 1. Availability of trained Posyandu cadres 2. Trained TBAs 3. Trained KSMs 4. Posyandu Cadres Association 5. Puskesmas/health centers 6. District/Provincial Health Office 7. The community 8. Community elders B. Weakness 1. Lack of health facilities 2. Lack of health workers 3. Inappropriate regional laws 4. Lack of commitment from the health staff 5. Limited health budget 6. Weak follow up 7. Lack of health awareness/ knowledge of the community C. Opportunity 1. School: school health unit 2. MED 3. Posyandu Cadres Association 4. Local foundations D. Threat 1. Unsupportive attitude 2. Political interest World Vision 165 LCSP Final Evaluation Dimension II: ORGANIZATION A. Strength 1. Existing institutions 2. Institution groups training 3. Existing committee 4. Existing activities 5. Existing organization vision (some) 6. Facilities available partially 7. Existing organization constitution B. Weakness 1. Lack of funding 2. Lack of human resources 3. Insufficient transfer of skills/knowledge 4. No network among organizations 5. Lack of management skills C. Opportunity 1. Our organization ( WVI – ADP ) is ll known in the community 2. Time available for further assistance 3. The community is open for new things 4. Available support from the government D. Threat 1. Conflict of interests 2. Limited health assistance staff in the community 3. Conflicts Dimension III: COMMUNITY A. Strength 1. The community 2. Posyandu cadres 3. TBAs 4. Shopkeepers 5. POD/Village Drug Posts 6. Transportation facilities available 7. Transformation access 8. Existing health services B. Weakness 1. Lack of human resources (health staff) 2. Low economic level 3. Limited skills 4. Lack of awareness 5. limited mobility, for example: roads and cars/buses 6. Far referral access C. Opportunity 1. Trainers available 2. MED (rubber/ plantation/ cattle) 3. Acknowledgement of Pontianak ADP-LCSP in the community 4. Political and governmental development aftermath D. Threat 1. Conflict potentials 2. Lack of funding 3. Other NGOs 4. Sense of security of the staff World Vision 166 LCSP Final Evaluation Indicators for Sustainability Dimension I 1. Targets set in the DIP and FAR achieved for the 5 technical approaches 2. Vitameal socialized and distributed in all Pontianak ADP areas. Dimension II 3. 80% of Posyandu cadres in Toho and Sungai Pinyuh trained. 4. 80% of KSM committee obtained basic health training. 5. Posyandu cadres able to conduct health education sessions independently. 6. Collaboration network among Posyandu Cadres Associations and KSMs. 7. 40% of Posyandu cadres obtained organizational and leadership training. 8. All development staff obtained basic health training. Dimension III 9. BCC conducted at least once. 10. All Pontianak ADP-LCSP staff obtained MED training. 11. Committee of 50% of KSM obtained MED training. 12. 40 community entrepreneurship groups obtained MED training. Pontianak ADP-LCSP Plan Of Action, April to September 2004 No. PLAN OF ACTIVITIES April May June July Aug Sept 1 Coordination with health centers and community groups in Toho and Takong 2 Posyandu cadres training in the Toho health center area 3 Posyandu cadres training in the Takong health center area 4 Basic health training for KSM members and teachers in S.Pinyuh and Mandor 5 Basic health training for KSM members and teachers in Toho and Takong 6 Basic health training for KSM members and teachers in Senakin and Pahauman 7 Community health promotion on ARI/ISPA 8 Community health promotion on diarrhea 9 Vitameal socialization in Toho and Takong 10 Early Vitameal distribution in Toho and Takong 11 Vitameal socialization in Anjungan 12 Early Vitameal distribution in Anjungan 13 BCC on Malaria 14 MED and health training for Pontianak ADP staff 15 MED training for KSM committees and small community entrepreneurship groups 16 Meeting between Posyandu cadres, KSMs, CUs, and village government for network building in Sidas Village 17 Organizational and leadership training for Posyandu cadres 18 SAP Workshop at the sub district level 19 SAP Workshop at the district level 20 Final Survey 21 Final Evaluation World Vision 167 LCSP Final Evaluation Annex 33. ITN Distribution Plan – ADP Pontianak Background The project design for ADP Pontianak calls for a focus on child survival for the next five years. The ADP plans to continue CS interventions in the following areas: malaria, nutrition, community empowerment and support for health cadres. To address the malaria problem, in May 2004 Sengah Temila and Mandor sub-districts were chosen as pilot areas to introduce a new malaria drug called artemicine. It is particularly useful in combating cerebral malaria. The ADP team will continue to distribute 24,000 ITNs. The purpose of this exercise is to assist both pregnant women and children under 5 by providing access to low cost insect treated mosquito netting (ITN). The total number of families in the target area is approximately 31,860 and includes 16,941 direct beneficiaries. Income from the sale of nets will be used to procure additional ITNs. Additionally, project income will be utilized to support micro-enterprise development (MED) programming that is to be integrated with health programming. Beginning in 2003, ADP Pontianak has implemented MED activities which integrate nutrition and malaria training within five Self-Help Groups (SHG). In addition to the WV ADP and district health authorities, there are several institutions such as self help groups, Cadre Union, and drug stores that address the problem with Malaria. It is anticipated that these local bodies will assume the distribution of the ITN in order to assure project sustainability. Rationale and Goals 1. ITN program is integrated into a larger Malaria strategy for West Kalimantan. The project is non-profit. Though recipients will pay for ITN, it is at a discount and project proceeds will be utilized to procure additional netting. The lowest income groups will receive netting at no cost. Integral to the project will be education stressing the importance of family health. 2. Use of ITN income: a. To procure and distribute additional ITNs, with the planned result that all families in Mandor, Sengah Temila, Toho and Sungai Pinyuh have at least one ITN. b. To provide training in health education. c. To establish an MED project that will provide rubber plant seedlings and other basic income generating projects for low-income families. d. Beneficiaries among the lowest income groups will receive ITNs at no cost. World Vision 168 LCSP Final Evaluation Distribution Targets 1. Direct beneficiaries include 13,553 families or 80% of the 16,941 families in the project area. Total beneficiaries are as follows: a. Mandor (3.595), b. Sengah Temila (7053), c. Toho (4856) d. Sungai Pinyuh (1.437) 2. Indirect beneficiaries in the project area are 14,919 families. 3. In the project timeline, it is anticipated that 1600 ITNs per month can be sold though local commercial institutions with a price of 65,000 rupiyah per ITN. It is further anticipated that Self Help Groups can distribute an additional 50-100 ITNs per month for a combined total of 1650-700 ITNs per month. Distribution Strategy Primary strategy: The main avenue for distribution will be through the Koperasi Pancur Dangeri (KPD); a local institution, which provides basic trading needs to communities in Landak District and throughout West Kalimantan. The KPD is widespread, and thus will have good distribution management. The ADP will hand over ITN distribution to the KPD in 2006. The KPD plans to distribute approximately 1600 ITNs/month through its 15 branches, 15 CU (credit union) and 500 local traders (see below). Secondary strategy: Self-Help Groups and Posyandu Cadre Unions will also distribute ITNs in Mandor, Sengah Temila, Toho and Sungai Pinyuh sub-districts, giving priority to reaching the poorest and most inaccessible families. WV Health staff and the local District Health Office will conduct a campaign of community meetings, local broadcasts and poster distribution detailing the use and need for ITN. ITN PLAN for FY 200545 a. ITN Income (Current status): ITN income in Bank = Rp. 72,443,660 (Bank record October 2004) Additional ITNs = 24,000 b. ITN Budget for FY 2005: 1. Warehouse = Rp. 24,000,000 (Fill-up Year 2004) 2. Staff salary (1) and Admin. Cost = 13,150,000, 3. Re-purchasing ITN = N/A 4. Social Mobilization = 18,140,000 5. MED and Health Education = 36,220,000 Total: = 91.510.000 45 Please note that for calculation purposes, an exchange rate of $1 USD = 8500 rupiyah is utilized throughout the document. World Vision 169 LCSP Final Evaluation c. ITN Distribution Plan Goal Indicator Activity & Time Budget (Rp.) PIC Staff And Rent Warehouse Rent warehouse Contract agreement letter To seek one rent building for 1 year (Oct. 2004 Rp. 24.000.000) (October 2004) N/A ADP Hire one malaria staff person responsible for distribution & reporting Staff person recruited; progress reports each month, and annually. Salary 1 staff Rp. 650.000 x 13 months .= Rp. 8.450.000,-(1 month benefit) Health assistance/insurance (November 2004) ------------------------------- Transport Rp. 200.000 x 12 months = Rp. 2400.000 Tuitions : Rp. 500.000, Communication : Telephone and electricity cost : Rp. 150.000 x 12 = Rp. 1.800.000 13.150.000,- ($1547) PM Social Mobilization Gaining stakeholders commitment (DHO, ADP, KPD, SHGs committee, Cadre Union) Mini workshop plan Approach DHO staff Meal and Transportation for 2 ADP staff = Rp. 250.00,- Conducting 1 mini workshop (November 2004) PM HC Set up a new agreement and strategy with KPD and some selected SHGs New MOU New distribution plan from KPD Discussing The agreement with KPD committee/Manager Transport to Pontianak 100.000 Meal = 5 x 1 x 20.000 = 150.000 (November 15th. 2004) 250.000 ($29) ADP and KPD Share responsibility on Malaria issue Selected 2 SHGs for each FP or at least 20 SHGs Transportation: SHGs 20 x 2 Rp. 20.000 = 400000 ADP cost (transport) = 50.000 Meal 40 x 11.000 = 440000 (November to December 2004) 890.000,- ($104) FP and SHG Increase the families understanding to Malaria disease Poster Offers the training /st6rangthening Distribute poster to family = NA Training sessions 4 times a year In 4 sub district with Posyandu cadres: Meal (snack) 4 x 100 x Rp. 5000 = 2.000.000,- Transport cost for 400 cadres x 20.000 = 8.000.000,- (October to September 2005) 10.400.000 ($1224) FP and Malaria Staff Monitoring and evaluation the ITN distribution and strategy with SHG and Offers 4 times meeting Report and suggestions by KPD and SHGs committees Meeting and discussion 3 times a year 3 x 3 x 20 people x 20.000 = Rp. 3.600.000,- ( January, March, August 2005) 3.600.000 ($424) ADP staff and KPD Purchase of Additional ITNs Data of new target for 24.000 ITNs List and progress report timely Distributing ITNs to main and second line subjects N/A Malaria staff Purchasing of New ITN/re-stock New ITN in the storehouse Rent One ware-house Management and administrated the ITN stock N/A Malaria Staff World Vision 170 LCSP Final Evaluation d. MED and Health Education Plan FY 2005 Goal Indicator Activity and Time Budget (Rp.) PIC Increase knowledge on Malaria and Nutrition for mothers and children under￾five in ADP Area. Mandor = 20% of 1687 mother/ under-five trained Senakin = 20% of 886 mother /under-five trained Pahauman = 20 of 1905 mother/ under five trained Toho = 20% 1833 mother/r under-five trained S. Pinyuh = 20 % of 660 mother/ under-five trained (tot mother in ADP Area only1437) Training 337 mothers in Mandor. (Meal and tuitions a’ Rp. 20.000) = Rp. 6.740.000 Training 177 mother in Senakin (177 x 20.000 = Rp. 3.540.000,-) Training 381 mother in Pahauman (381 x 20.000 = Rp. 7.620.000,) Training 366 mother in Toho ( 366 x 20.000 = Rp. Rp. 7.320.000,) Training 132 mother in Sungai Pinyuh (132 x 20.000 = Rp. 2.640.000,) 25.220.000 ($2968) MED and Malaria staff Increase the awareness of community participation of cadres 500 cadres trained Training once each area (100 cadres) cost per￾person : Meal a’ Rp. 7500 (person) Transportation a’ Rp. 7.500 Tuitions a’ 5000 10.000.000 ($1176) Health staff Increase the understanding of beneficiary on home economic management Training on home management to all SHGs (5 woman groups) At least 50 members have home management training (10 for each SHG) Meal 50 x 15.000 = Rp. 750.000 Tuitions 50 x Rp. 5000 = Rp. 250.000,- 1.000.000,- ($118) MED staff e. Role of Partner Institutions Item Health Center KPD SHG Cadre ADP Training on Health v Treatment v Approach mothers v v v Monitoring & evaluation v v v v v ITN Fund v v v v MED training v v World Vision 171 LCSP Final Evaluation Annex 34. Training Events Attended by LCSP Staff DATE TRAINING / WORKSHOP STAFF October 3-4, 2000 Detailed Implementation Plan & Sustainability Workshop of LCSP Mr. Hendrik Rupang (LCSP Monev Officer) Mr. Albert Silalahi (LCSP Finance Officer) Ms. Esther Indriani (LCSP Project Officer) Ms. Dini Susanti (Candidate for staff – now Health Motivator) Ms. Titien Zurianti (Candidate for staff – now Health Motivator) Ms. Lina Monika (Candidate for staff – now Health Motivator) December 11-13, 2000 PLA (Participatory Learning for Action) Mr. Hendrik Rupang (LCSP Monev Officer) Ms. Esther Indriani (LCSP Project Officer) Ms. Dini Susanti (Health Motivator) Ms. Titien Zurianti (Health Motivator) Ms. Lina Monika (Health Motivator) Ms. Novianti (Health Motivator) Ms. Petronella (Health Motivator) Ms. Dewi (Health Motivator) Mr. Michael Thommy (Health Motivator) February 2001 Grant Accounting Workshop in Bangladesh Mr. Albert Silalahi (Finance Officer) April 10-11, 2001 Training of Trainers (TOT) for Health Staff Ms. Dini Susanti (Health Motivator) Ms. Titien Zurianti (Health Motivator) Ms. Lina Monika (Health Motivator) Ms. Novianti (Health Motivator) Ms. Petronella (Health Motivator) Ms. Dewi (Health Motivator) Mr. Michael Thommy (Health Motivator) World Vision 172 LCSP Final Evaluation DATE TRAINING / WORKSHOP STAFF May 19, 2001 Malaria Training Dr. Andre Tanoe (Technical Team Leader) Dr. Ronald Gunawan (Technical Training Coordinator) Mr. Hendrik Rupang (LCSP Monev Officer) Ms. Esther Indriani (LCSP Project Officer) Ms. Dini Susanti (Health Motivator) Ms. Titien Zurianti (Health Motivator) Ms. Lina Monika (Health Motivator) Ms. Novianti (Health Motivator) Ms. Petronella (Health Motivator) Ms. Dewi (Health Motivator) Mr. Michael Thommy (Health Motivator) June 2001 Training of Trainers in Utilization of MCH Book Dr. Ronald Gunawan (Technical Training Coordinator) June 3-16, 2001 Inter-country IMCI Training Mrs. Mary Lengkong, DDS, DDPH (National Health Advisor) July 22-28, 2001 IMCI Training Dr. Andre Tanoe (Technical Team Leader) Dr. Ronald Gunawan (Technical Training Coordinator) Ms. Esther Indriani (LCSP Project Officer) September 17-19, 2001 Local Capacity for Peace Building - WVI Dr. Ronald Gunawan (Technical Training Coordinator) Ms. Dini Susanti (Health Motivator) Ms. Titien Zurianti (Health Motivator) Ms. Lina Monika (Health Motivator) Ms. Novianti (Health Motivator) November 5-23, 2001 Regional Workshop on Monev in Reproductive Health, Bangkok. Mr. Hendrik Rupang (LCSP Monev Officer) Ms. Esther Indriani (LCSP Project Officer) November 24 – December 24, 2001 Regional Collaboration to Build Field Capacity to Conduct KPC Survey, Cambodia Dr. Andre Tanoe (Technical Team Leader) February 4-8, 2002 BCC (Behavior Change Communication) Workshop in Johannesburg, South Africa Drs. Untung Sidupa (ADP Pontianak Manager) Mrs. Mary Lengkong, DDS, DDPH (National Health Advisor) September 11 – 21, 2002 Final Evaluation Ballia Rural Integrated Child Survival, India Dr. Andre Tanoe (Technical Team Leader) Mr. Markus Akim (Community Development Coordinator) Ms. Mariani (Training Coordinator) September 23 – 28, 2002 IMCI Training, Pontianak Ms. Dewi Helpina (Health Motivator) Ms. Muliawati (Health Motivator) February 3 – 7, 2003 Behave Training, Cambodia Ms. Mariani (Training Coordinator) April 14 – 16, 2003 Local Capacity for Peace (LCP) Training -- WVI Pontianak Ms. Rainy Wijaya (Monev Officer) Ms. Lina Monika (Motivator) World Vision 173 LCSP Final Evaluation DATE TRAINING / WORKSHOP STAFF April 29 – 2 Mei, 2003 Nutrition Training, Pontianak Mr. Bonar Panjaitan (Health Motivator) Mr. Herkulanus Sumadi (Health Motivator) Mr. Edi (Health Motivator) Ms. Dewi Helpina (Health Motivator) Ms. Muliawati (Health Motivator) May 19 – 24, 2003 Transformational Development Indicator Roll Out ADP Sanggau, Sanggau Ms. Rainy Wijaya (Monev Officer) June 24 – 27, 2003 Positive Deviance Inquiry Training, Sungai Asam, Pontianak – conducted by West Kalimantan Rehabilitation Program Ms. Dewi Helpina (Health Motivator) Ms. Muliawati (Health Motivator) Mr. Markusius (Health Motivator) July 14 -- 16, 2003 Local Capacity for Peace (LCP) Training -- WVI, Pontianak Ms. Mariani (Training Coordinator) Dec 8 – 19, 2003 East Asia Regional Workshop on Managing Information for Effective Health Program, Vietnam Ms. Rainy Wijaya (Monev Officer) Dec 5 – 20, 2004 Final Evaluation Kean Svay CSP, Cambodia Ms. Esther Indriani (Project Officer) Feb 24 – 26, 2004 Sustainabity Action Plan Workshop – PCI, Pandeglang, West Java Ms. Esther Indriani (Project Officer) Dr. Teresa Zakaria (Team Leader) March 8 – 13, 2004 Integrated Management Childhood Illness Training For Trainers, Pontianak Dr. Teresa Zakaria (Team Leader) March 23 – 24, 2004 Sustainability Action Plan Workshop (Intern ADP), Pontianak Dr. Teresa Zakaria (Team Leader) Ms. Rainy Wijaya (Monev Officer) Ms. Mariani (Training Coordinator) Mrs. Lina Monica (Health Motivator) Ms. Sri Astuti(Health Motivator) Ms. Muliawati(Health Motivator) Mr. Bonar Panjaitan (Health Motivator) Mr. Herkulanus Sumadi (Health Motivator) Mr. Edi (Health Motivator) Ms. Dewi Helpina (Health Motivator) March 25, 2004 Doer – Non doer Analysis Workshop, ADP Pontianak – Office, Sungai Pinyuh Dr. Teresa Zakaria (Team Leader) Ms. Rainy Wijaya (Monev Officer) Ms. Mariani (Training Coordinator) Ms. Sri Astuti(Health Motivator) Ms. Muliawati(Health Motivator) Mr. Herkulanus Sumadi (Health Motivator) Mr. Edi (Health Motivator) World Vision 174 LCSP Final Evaluation DATE TRAINING / WORKSHOP STAFF May 30 – June 12, 2004 Global Health Workshop, Washington DC. Child Survival Mini University Mrs. Mary Lengkong, DDS, DDPH (National Health Advisor) Ms. Esther Indriani (Project Officer) July 26 – 29, 2004 Behavioral Change Communication Training, Pontianak Ms. Esther Indriani (Project Officer) Dr. Teresa Zakaria (Team Leader) Ms. Rainy Wijaya (Monev Officer) Ms. Mariani (Training Coordinator) Mrs. Lina Monica (Health Motivator) Ms. Sri Astuti(Health Motivator) Ms. Muliawati(Health Motivator) Mr. Bonar Panjaitan (Health Motivator) Mr. Herkulanus Sumadi (Health Motivator) Mr. Edi (Health Motivator) Ms. Dewi Helpina (Health Motivator) September 6 – 11, 2004 IMCI Training, Pontianak Ms. Sri Astuti(Health Motivator) Source: LCSP, September 2004. World Vision 175 LCSP Final Evaluation Annex 35. Summary Events Attended by LCSP Staff 1. Number of training events attended by individual LCSP staff members 0 2 4 6 8 10 Mrs. Lina Monica (Health Motivator) Ms. Dewi Helpina (Health Motivator) Ms. Dini Susanti (Health Motivator) Ms. Titien Zurianti (Health Motivator) Mr. Edi (Health Motivator) Mr. Herkulanus Sumadi (Health Motivator) Ms. Novianti (Health Motivator) Mr. Bonar Panjaitan (Health Motivator) Mr. Michael Thommy (Health Motivator) Ms. Muliawati (Health Motivator) Ms. Petronella (Health Motivator) Ms. Sri Astuti(Health Motivator) Mr. Markusius (Health Motivator) Ms. Mariani (Training Coordinator) Ms. Rainy Wijaya (Monev Officer) Dr. Teresa Zakaria (Team Leader) Dr. Andre Tanoe (Technical Team Leader) Mr. Hendrik Rupang (LCSP Monev Officer) Dr. Ronald Gunawan (Training Coordinator) Mr. Albert Silalahi (LCSP Finance Officer) Drs. Untung Sidupa (ADP Pontianak Manager) Mr. Markus Akim (ADP CDC) Ms. Esther Indriani (LCSP Project Officer) Mrs. Mary Lengkong (National Health Advisor) Health Motivators Core Team ADP NO International In-country World Vision 176 LCSP Final Evaluation 2. Topics of training events and number of trainees FY Training Event Topic Number of Trainees Health Motivators FY01 DIP & Sustainability Workshop (2 days) 3 Participatory Learning for Action (3 days) 7 Training of Trainers for Health Staff (2 days) 7 Malaria Training (1 day) 7 Local Capacity for Peace Building (3 days) 5 FY02 IMCI Training, Pontianak (7 days) 2 FY03 Nutrition Training, Pontianak (3 days) 5 Positive Deviance Inquiry, Pontianak 4 days) 3 FY04 Sustainability Action Plan Workshop (2 days) 7 Doer – Non doer Analysis Workshop (1 day) 4 Behavioral Change Communication (4 days) 7 IMCI Training, Pontianak (7 days) 1 Core Team, ADP and National Office staff In-Country FY01 DIP & Sustainability Workshop (2 days) 3 Participatory Learning for Action (3 days) 2 Malaria Training (1 day) 4 Training of Trainers in Utilization of MCH Book (2 days?) 1 IMCI Training, Pontianak (7 days) 1 Local Capacity for Peace -- WVI, Pontianak (3 days) 3 FY02 - - FY03 Local Capacity for Peace -- WVI, Pontianak (3 days) 3 TDI Roll Out, ADP Sanggau (6 days) 1 Local Capacity for Peace -- WVI, Pontianak (3 days) 3 FY04 Sustainability Action Plan Workshop – PCI, West Java (3 days) 2 IMCI Training For Trainers, Pontianak (6 days) 3 Sustainability Action Plan Workshop (2 days) 3 Doer – Non doer Analysis Workshop (1 day) 3 Behavioral Change Communication (4 days) 4 International FY01 Grant Accounting Workshop in Bangladesh (1 week) 1 Inter-country IMCI Training (2 weeks) 1 FY02 Monitoring and evaluation in Reproductive Health, Bangkok (3 weeks) 2 KPC Survey, Regional Workshop, Cambodia (4 weeks) 1 BCC / BEHAVE, South Africa (1 week) 2 Final Evaluation Ballia Rural Integrated Child Survival, India (2 weeks) 3 FY03 BCC / BEHAVE, Cambodia (1 week) 1 FY04 Final Evaluation Kean Svay CSP, Cambodia (2 weeks) 1 Managing Information for Effective Health Program, Vietnam (2 weeks) 1 Global Health Council/Mini University, Washington DC (2 weeks) 2 World Vision 177 LCSP Final Evaluation Annex 36. ADP Pontianak – Landak Child Survival Project Organizational Chart ADP Manager Mr. Untung Sidupa Finance Officer Mr. Joshua Guard Mr. Alex Jerry Driver Mr. Stefanus Health Motivator Ms. Sri Astuti Health Motivator Ms. Muliawati Health Motivator Mrs. Dewi Health Motivator Mr. Herkulanus Asst. Area Coord. Mrs. Lina Health Motivator Mr.Edi SR Coordinator Mr. Deni Teman Monev Officer Ms. Elisabeth Erna CD Coordinator Mr. Markus Akim Mandor Area Coord … CD Motivator Mr. Malinus CD Motivator Tumbur Manalu CD Motivator Mr. Anto Sengah Temila Area Coord … CD Motivator Mr. Agustian CD Motivator … CD Motivator Ms. Setiana Toho Area Coord Mr. Sutejo Book Keeper Ms. Cecilia Driver Mr. Said Purchasing Mr. Alexander Lini Guard Mr. Hadi Training Coordinator Ms. Mariani Finance Officer Mr. Donny Indra Wardhana Local Capacity for Peace Coord … Yayasan Pancur Kasih Yayasan Karya Kasih ADP PONTIANAK – LANDAK CHILD SURVIVAL PROJECT ORGANIZATION CHART Training Coordinator Mr. Malinus Last Revision: 200904 Monev Officer Ms. Rainy Wijaya Admin Assistant Ms. Paula Erni CSP Tech. Team Leader Dr Teresa Zakaria Asst Area Coord. Mr. Bonar MED Integration & Health Malaria and Nutrition CD Motivator Mr. Amon CD Motivator Mr. Herdi Vitameal Officer Ms. Imelda Banase World Vision 178 LCSP Final Evaluation Annex 37. Timeline of Human Resources, July 2000 – September 2004 Core Team Tech Team Leader Monitoring and Evaluation Officer Training Coordinator Finance Officer Field Staff (Health Motivator) Mandor HM 1 Mandor HM 2 Mandor HM 3 Senakin HM 1 Senakin HM 2 Pahauman HM 1 Pahauman HM 2 Pahauman HM 3 Pahauman HM 4 Support Staff Administrative Staff Driver Guard National Office Staff National Health Advisor Project Officer = Female = Male = Resigned/Replaced = Transferred POSITION FY 2002 FY 2003 FY 2004 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q1 Q2 Q3 Q4 FY00 FY 2001 Q4 Q1 Q2 Q3 Q4 World Vision 179 LCSP Final Evaluation Annex 38. Human Resources Management Indicators Position Vacancy Turn over Female occupancy Core Team Technical Team Leader 20% 29% 22% Monitoring and Evaluation Officer 0% 24% 47% Training Coordinator 27% 32% 78% Finance Officer 0% 24% 0% Average: 12% 27% 34% Health Motivator Mandor HM 1 16% 0% 100% Mandor HM 2 35% 73% 100% Mandor HM 3 41% 80% 67% Senakin HM 1 31% 0% 0% Senakin HM 2 45% 86% 100% Pahauman HM 1 16% 0% 100% Pahauman HM 2 16% 28% 19% Pahauman HM 3 47% 44% 11% Pahauman HM 4 73% 171% 21% Average: 36% 41% 61% Support Staff Administrative Staff 31% 0% 100% Driver 0% 24% 0% Guard 75% 0% 0% Average: 35% 12% 35% National Office Staff National Health Advisor 0% 0% 100% Project Officer 8% 51% 100% Average: 4% 24% 100% Total average: 27% 30% 56% Note: Vacancy : Percentage of the project time during which the position was vacant Turn over : Number of replacements per project year during which the position was filled Gender balance : Percentage of the project time with the position during which the position was filled by a female World Vision 180 LCSP Final Evaluation Annex 39. Timeline of Technical and Administrative Support, July 2000 – September 2004 WV US Program Officer xx Finance Officer x Technical Officer x x xx WV Asia Pacific Regional Health Advisor xx x x x x x WV Indonesia National Health Advisor xx x x x x x x Project Officer x x x xx x x x xxx x x xx Consultant x x x xxx Q2 Q3 Q4 FY 2004 Q1 FY 2003 Q1 Q2 Q3 Q1 Q2 Q3 Q4 FY 2002 Q4 Q3 Q4 Q1 Q2 Q4 FY00 FY 2001 POSITION Source: LCSP, September 2004