EVALUATION COMMUNITY HEALTH SYSTEMS STRENGTHENING IN CIBITOKE PROVINCE, BURUNDI: MABAYI CHILD SURVIVAL PROJECT FINAL EVALUATION REPORT December 2013 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Bonnie Kittle, Independent Consultant. MABAYI CHILD SURVIVAL PROJECT FINAL EVALUATION REPORT STRENGTHENING COMMUNITY HEALTH SYSTEMS IN CIBITOKE PROVINCE, BURUNDI Submitted December 13, 2013 CSHGP Cooperative Agreement Number GHN-A-00-08-00005-00 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. CONTENTS Acronyms ………………………………………………………………………………................i Executive Summary ……………………………………………………………………………..ii Evaluation Purpose and Evaluation Questions ……………………………….. ........... 1 Project Background ……………………………………………………………………………….1 Evaluation Methods and Limitations ………………………………………………………..4 Findings, Conclusions, and Recommendations ………………………………………….6 Annexes Annex I. Program Learning Brief – not required Annex II. List of Publications and Presentations Related to the Project Annex III. Project Management Evaluation Annex IV. Work Plan Table Annex V. Rapid CATCH Table Annex VI. Final Knowledge, Practice, and Coverage Survey Report Annex VII. Community Health Worker Training Matrix Annex VIII. Evaluation Scope of Work Annex IX. Evaluation Methods and Limitations Annex X. Data Collection Instruments Annex XI. Sources of Information Annex XII. Disclosure of Any Conflicts of Interest Annex XIII. Statement of Differences – not provided Annex XIV. Evaluation Team Members, Roles, and Their Titles Annex XV. Final Operations Research Report Annex XVI. Operations Research Brief Annex XVII. Stakeholder Debrief PowerPoint Presentation Annex XVIII. Project Data Form Annex XIX. Optional Annexes – Final Health Facility Assessment Survey Report, Final M&E Table ACRONYMS ACT Artemisinin-based Combination Therapies ARI Acute Respiratory Infections BCC Behavior Change Communication CCM Community Case Management CG Care Group CGV Care Group Volunteer C-HIS Community Health Information System CHW Community Health Worker COGES Comité de Gestion (Management Committee of a COSA) Colline Literally, “hill” in French – an administrative unit of a commune COSA Comité de Santé (Health Committee) CMAM Community-based Management of Acute Malnutrition CU5 Children Under Five DHT District Health Team DIP Detailed Implementation Plan HC Health Center HFA Health Facility Assessment iCCM Integrated Community Case Management IMCI Integrated Management of Childhood Illness IRB Institutional Review Board ITN Insecticide-Treated Net KPC Knowledge, Practice and Coverage MCSP Mabayi Child Survival Project M&E Monitoring & Evaluation MHD Mabayi Health District MOH Ministry of Health MTE Midterm Evaluation MUAC Mid Upper Arm Circumference NGO Non-Governmental Organization OR Operations Research QIVC Quality Improvement Verification Checklist RDT Rapid Diagnostic Test SOW Scope of Work TIPS Trials for Improved Practices Titulaire Head Nurse at Health Centers Titulaire-Adjoint Head Nurse Assistant at Health Centers TPS Technicien de Promotion de la Santé (Health Promotion Technicien) USAID United States Agency for International Development USG United States Government WHO World Health Organization Mabayi Child Survival Project Final Evaluation Report: Executive Summary This project was funded by the U.S. Agency for International Development through the Child Survival and Health Grants Program. December 2013 Evaluation, Purpose, and Evaluation Questions The in-country portion of the Mabayi Child Survival Project (MCSP) final evaluation took place from September 2 – 20, 2013. The purpose of the evaluation, as stated in the Evaluation Scope of Work, was to take stock of accomplishments to date and to listen to the beneficiaries at all levels, including mothers and caregivers, other community members and opinion leaders, health workers, health system administrators, local partners, other organizations, and donors. The three key questions that the evaluation answered were: 1. To what extent did the project achieve the project objectives as stated in the project detailed implementation plan (DIP)? 2. What was the appropriateness and effectiveness of key project interventions, as well as their potential for scale-up and sustainability? 3. What was the extent of collaboration with the Ministry of Health, UNICEF, and other USG-funded partners at the national, provincial, and district levels? The evaluation report will be used by USAID to determine if the funds provided to the grantee were used effectively and if the grantee was able to produce the anticipated results. The recommendations and lessons learned will be used to inform future decisions taken by USAID, Concern, and Burundi’s MOH. The MOH – specifically the Malaria, Nutrition, and Community Health Units – will use the results of the final evaluation to decide which approaches demonstrated by the MCSP are most effective in helping them achieve their own objectives. Specifically, the MOH partners will use these results to decide how to scale-up community case management (CCM) of malaria, Delphin Sula, Project Manager, speaks to Care Group Volunteer Key Findings:  The MCSP achieved or came within a few percentage points of achieving 12 of the 14 project indicators related to improved child health and nutrition practices.  The OR study found that the Integrated Care Group model was as effective as the Traditional Care Group model. This study paves the way for the MOH to replicate and scale up the Integrated Care Group approach in other districts.  The project helped to increase access to quality malaria diagnosis and treatment services, by collaborating with the Department of Malaria Management to implement community case management of malaria. FINAL EVALUATION EXECUTIVE SUMMARY how to pilot test CCM/diarrhea and CCM/pneumonia, if the community management of acute malnutrition (CMAM) approach is effective or not, if training staff in the integrated management of childhood illness (IMCI) is worthwhile, if strengthening the health management committee (Comite de Santé or COSA) improves the management of health centers, and if the Integrated Care Group model should be replicated elsewhere. Concern, the grantee, will use the results of the final evaluation to design future projects aimed at reducing maternal and child morbidity and mortality in Burundi and other under-resourced settings. Project Background The Mabayi Child Survival Project, implemented by Concern Worldwide and the Burundian MOH from October 2008 – September 2013, supported health services in the Mabayi Health District in Cibitoke Province. The overall goal of the MCSP was to sustainably reduce infant and child morbidity and mortality among children under five in Mabayi Health District. To achieve this goal, the project team, composed of Concern personnel and MOH staff, focused on three results: 1) Improved household maternal and child health care and nutrition practices; 2) Improved access to quality child health care services with a balance of provision at the health center and community levels; 3) Strengthened community leadership in health. The MCSP interventions focused on Infant and Young Child Feeding (25%); Vitamin A/Micronutrients (15%); Prevention and Treatment of Malaria (25%); Control of Diarrheal Disease (20%); and Pneumonia Case Management (15%). The interventions follow the national IMCI approach at both the community and health facility levels. The project introduced the five technical themes one at a time through three approaches: Care Groups, district and health center strengthening, COSA capacity building and policy dialogue at the national level. Activities and achievements for each of these are discussed in more detail throughout the document. Evaluation Questions, Design, Methods, and Limitations The final evaluation of the MCSP was implemented in three phases: Preparation Phase (June – August 2013), In-Country Data Collection Phase (September 2 – 20, 2013) and Report Writing Phase (September 23 – October 30, 2013). During the first three days in-country, the external consultant facilitated a team planning meeting, the primary purpose of which was to prepare for the field-based data collection which took place the following week. The evaluation team consisted of six MCSP staff, two district health team members (one of whom later dropped out), the Health Advisor from Concern HQ, a Concern Trainee, and the external evaluator. The field visit schedule was developed allowing the team to visit 10 of the 25 health centers in Mabayi district. A total of 162 respondents were interviewed (primarily individually). Data collection took place over a four day period. The data collected during the individual and group interviews were analyzed by the evaluation team members in a participatory manner. Working in a group, the responses to each question for each type of informant were reviewed and analyzed quantitatively. For findings of consequence, the team was asked to formulate a conclusion and potentially a recommendation and/or a lesson learned. In this way, almost all of the recommendations found in this report came directly from the evaluation team members. In addition to the qualitative data collected through these interviews, the evaluation team also analyzed the quantitative data provided by the final KPC survey, the operations research study draft report, and the final Health Facility Assessment. Findings and Conclusions The most critical achievements of the project are:  The MCSP achieved or came within a few percentage points of achieving 12 of the 14 project indicators related to improved child health and nutrition practices. The project achieved or significantly improved all five indicators related to the strengthened community leadership objective.  The MCSP tested the Integrated Care Group approach that relied more on the MOH and less on the NGO. The OR study found that the Integrated model was as effective as the traditional model. This trial paves the way for the MOH to replicate the Integrated Care Group approach in other districts.  The project helped to increase access to quality malaria diagnosis and treatment services, by collaborating with the Department of Malaria Management to implement CCM of malaria. Through this initiative, 317 CHWs were trained and have provided 27,826 consultations and treated 12,291 children. Head nurses report a reduction in the number of severe malaria cases at the health centers. Key recommendations include the following:  Concern, the Mabayi District Health Team, and the Department of Community Health should write a protocol for the implementation of the Integrated Care Group approach based on the best practices and lessons learned of the MCSP and responding to as many of the lingering questions as possible. This protocol should guide the scale-up of the Integrated Care Group approach.  The Malaria Management Department and Concern should review the lessons and best practices associated with the CCM/malaria initiative and use these to inform the implementation of iCCM.  The MOH/Department of Community Health should go beyond the existing COSA orientation and offer formal training to all COSA member using the MOH-approved curriculum. The Mabayi Child Survival Project in Cibitoke Province, Burundi is supported by the American people through the United States Agency for International Development (USAID) through its Child Survival and Health Grants Program. The [Project name] is managed by Concern Worldwide under Cooperative Agreement No. GHN-A-00-08-0005. The views expressed in this material do not necessarily reflect the views of USAID or the United States Government. For more information about the Mabayi Child Survival Project visit: www.concernusa.org 0 10 20 30 40 50 60 70 80 90 100 Exclusive Breastfeeding IYCF ITN Use Handwashing ARI Care Seeking Selected Results Baseline Endline 1 December 2013 EVALUATION PURPOSE AND EVALUATION QUESTIONS The in-country portion of the Mabayi Child Survival Project (MCSP) final evaluation took place from September 2 – 20, 2013. The purpose of the evaluation, as stated in the Evaluation Scope of Work (SOW) provided in Annex VIII, was to take stock of project accomplishments and to listen to the beneficiaries at all levels, including mothers and caregivers, other community members and opinion leaders, health workers, health system administrators, local partners, other organizations, and donors. The three key questions that the evaluation answers are: 1. To what extent did the project achieve the project objectives as stated in the project detailed implementation plan (DIP)? 2. What was the appropriateness and effectiveness of key project interventions, as well as their potential for scale-up and sustainability? 3. What was the extent of collaboration with the Ministry of Health (MOH), UNICEF, and other United States Government (USG)-funded partners at the national, provincial, and district levels? The evaluation was led by an external consultant with 30 years of experience evaluating community health programs. She was hired with project funds, and USAID approved the external consultant as well as the SOW. The external consultant submitted the draft and final reports directly to USAID at the same time that they were submitted to the grantee. The evaluation report will be used by USAID to determine if the funds provided to the grantee were used effectively and if the grantee was able to produce the anticipated results. The MOH – specifically the Malaria, Nutrition, and Community Health Units – will use the results of the final evaluation to decide which approaches demonstrated by the MCSP are most effective in helping them achieve their own objectives. Specifically, the MOH partners will use these results to decide how to scale-up community case management (CCM) of malaria, how to pilot test CCM of diarrhea and CCM of acute respiratory infections (ARI), if the community-based management of acute malnutrition (CMAM) approach is effective or not, if training staff in integrated management of childhood illness (IMCI) is worthwhile, if strengthening the Comite de Santé (COSA) improves the management of health centers, and if the Integrated Care Group model should be replicated elsewhere. Concern, the grantee, will use the results of the final evaluation to design future projects aimed at reducing maternal and child morbidity and mortality. PROJECT BACKGROUND The MCSP was implemented by Concern Worldwide and the Burundian MOH in the Mabayi Health District (MHD) in Cibitoke Province from October 2008 – September 2013. The MHD is comprised of three communes - Mugina, Bukinanyana, and Mabayi – that are further divided into administrative units known as collines (hills in French). These three communes are all adversely affected by malaria, pneumonia, diarrhea, and malnutrition, diseases that have a significant impact on maternal and child health outcomes. As shown in Table 1, the three communes in the MHD have a total of 25 government health centers, four of which were opened in the last two years of the project. On average, each health center has a team of five staff, headed by the Titulaire (Head Nurse) who is assisted by a deputy, the Titulaire-Adjoint. Each commune has two Health Promotion Technicians (Techniciens de Promotion de la Santé or TPS) whose mandate is to promote public health at the community level. In September 2012, one nurse at each health center (not the Titulaire or Adjoint) was given responsibility to supervise and support community health activities. This nurse is referred to as the Community Health Nurse. 2 December 2013 Table 1: Administrative/MOH Structures and the Key Project Contact at Each Level Administrative and MOH Structures Project Contact Cibitoke Province Provincial Health Officer Mabayi Health District District Health Officer Communes (three in MHD) Commune Administrators and TPS Collines (74 in MHD) Colline Leaders MOH health centers (25 in MHD) Titulaires (Head Nurses) Target Population: The 2013 target population consists of women of reproductive age (WRA) and children under five (CU5). The following table updates the population table used in previous reports: Table 2: Population Table Beneficiary Population Mabayi Mugina Bukinanyana Total Total Population 71,982 77,081 113,341 262,404 Total Children 0-59 months (17.8%) 12,813 13,720 20,175 46,708 Children 0-11 months (3.6%) 2,591 2,775 4,080 9,447 Children 12-23 months (3.6%) 2,591 2,775 4,080 9,447 Children 24-59 months (10.6%) 7,630 8,171 12,014 27,815 Women of Reproductive Age (23%) 16,556 17,729 26,068 60,353 Total WRA and CU5 (40.8%) 29,369 31,449 46,243 107,061 Goal, Strategic Objective, and Expected Results: The overall goal of the MCSP was to sustainably reduce infant and child morbidity and mortality among children under five in Mabayi Health District. To achieve this goal, the project team, composed of Concern personnel and MOH staff, focused on three Results as illustrated in the framework below. See Annex IV for the Work Plan Table which identifies the changes made since the DIP was submitted. Figure 1: Results Framework Strategic Objective: Achieve sustained and equitable uptake of key child health and nutrition services and practices in Mabayi Health District. Result 1: Improved household maternal and child health care and nutrition practices Result 2: Improved access to quality child health care services with a balance of provision at the health center and community levels Result 3: Strengthened community leadership in health Strategy 1.1: Deliver an effective behavior change strategy using multiple channels. Strategy 1.2: Study and document the effectiveness and feasibility of an Integrated Care Group Model (Operations Research). Strategy 2.1: Improve technical and managerial capacity of district and health facility staff. Strategy 2.2: Establish and expand system for CCM Strategy 2.3: Integrate CMAM into district health services. Strategy 2.4: Improve data systems for decision-making and action. Strategy 3.1: Strengthen assessment and planning capacities of Care Groups and COSAs. Strategy 3.2: Improve linkages between community members, COSAs, and health providers. Strategy 3.3: Formalize mechanisms for community voice in the implementation and monitoring of performance contracts. 3 December 2013 Technical Interventions: The MCSP interventions focus on Infant and Young Child Feeding (25%); Vitamin A/Micronutrients (15%); Prevention and Treatment of Malaria (25%); Control of Diarrheal Disease (20%); and Pneumonia Case Management (15%). The interventions follow the national IMCI approach at both the community and health facility levels. Although the project does not have a child spacing indicator, in response to a recommendation made during the midterm evaluation (MTE), activities related to contraceptive use were initiated during the second half of the project. Project Strategy and Cross-cutting Approaches: The project introduced the five technical themes one at a time through three approaches: Care Groups, district and health center strengthening, and policy dialogue at the national level. Activities and achievements for each of these are discussed in more detail throughout the document. Operations Research: The MCSP was in the first batch of projects that were required to conduct an operations research (OR) study. After much discussion, the MCSP agreed to study the process of integrating the Care Group approach into the MOH system. The study was carried out in Bukinanyana Commune. In half of the Commune, the Care Group Volunteers (CGV) were trained and supervised by project staff (Promoters), which is the ‘traditional’ Care Group model. In the other half of the commune, the CGV were trained and supervised by MOH-supported Community Health Workers (CHW). Five functionality1 indicators were used to monitor Care Group activities in each study area. In addition to functionality, the project also sought to measure Care Group sustainability. Sustainability was assessed during the last six months of the project when MCSP staff withdrew all support to Care Group activities in Bukinanyana Commune to see if functionality declined. Concern measured the effectiveness of the approach by measuring changes in 40 knowledge and practice indicators in both study areas. Partnerships and Collaboration: Concern’s principal partner in project planning and implementation was the MOH at the national, provincial, district, and health center levels. At the national level, the Health Advisor and Project Manager were both actively involved in policy discussions and advocacy for community health programming. Following the MOH’s decentralization policy, the Provincial Health Office played an important role in coordination, implementation, and strategic planning. The primary collaborators and beneficiaries, however, have been the Mabayi District Health Team (DHT) and the staff at the 25 health centers in Mabayi District. At the district level, the members of the DHT, especially the District Health Officer and the three District Supervisors, played an essential role in all aspects of the MCSP: planning, implementation, designing behavior change materials, training, supervising, collecting data, and reporting. During key informant interviews conducted by the external consultant, both project staff and members of the DHT commented on the excellent working relationship they enjoy. This was corroborated by observation during the final evaluation. Supporting players in project implementation included local administrators, especially the three Commune Administrators and the 74 colline leaders; community-level structures such as the COSAs, the CHWs, other NGOs involved in community health programming in Burundi; and international organizations such as UNICEF and the World Health Organization (WHO). With its focus on child survival, community health outreach, and reaching vulnerable populations, the MCSP fits in well with the USAID Mission’s overall health plan for Burundi. The Project Manager and the Health Advisor provided regular written reports and verbal updates to the USAID Mission Project Activity Manager. When the USAID Project Activity Manager was interviewed for the MTE, he said that he is well-informed on the project, has made one field visit to the project site, and meets with visitors such as consultants from the Maternal and Child Health Integrated Program (MCHIP). A Mission representative also participated in the DIP workshop and a Mission representative attended the presentation of final evaluation results. 1 This type of indicator measures the extent to which the Care Groups and supporting systems function. 4 December 2013 EVALUATION METHODS AND LIMITATIONS The final evaluation of the MCSP was implemented in September 2013 and sought to respond to the three key evaluation questions as well as the more detailed questions provided in the SOW. The evaluation was implemented in three phases: Preparation Phase (June – August 2013), In-Country Data Collection Phase (Sept 2 – 20, 2013) and Report Writing Phase (Sept. 23 – Oct. 30, 2013). During the first three days in country, the external consultant facilitated a team planning meeting, the primary purpose of which was to prepare for the field-based data collection which took place the following week. The evaluation team consisted of six MCSP staff, two DHT members (one of whom later dropped out), the Health Advisor from Concern US, a Concern Trainee, and the external evaluator. Evaluation team members and their roles are provided in Annex XIV. As part of the team planning meeting, the evaluation team reviewed the evaluation questions and identified the key informant(s) for each (see Annex IX for the outcomes of this). The field visit schedule allowed the team to visit 10 of the 25 health centers supported by the project. The team elected to visit those health centers where the Head Nurse (Titulaire) and Community Health Nurse had been present at least one year and would therefore be able to provide an informed impression of the project. At each health center the Titulaire, the Community Health Nurse, and the COSA members were all interviewed. In addition, from each health center a number of randomly selected CHW and CGV were interviewed. A total of 162 respondents were interviewed (primarily individually) as show in Annex XI. The evaluation team decided not to interview any mothers as they were the focus of the KPC survey and had been interviewed during the MTE. Each interview (excluding MCSP staff) lasted between 40 and 60 minutes. The questionnaires can be found in Annex X. Aside from the interviews conducted with the COSA members, all other interviews were individual in-depth interviews. Though this took a bit longer, it was decided to have only one group interview to limit the amount of bias which occurs naturally when addressing a group. Care was taken to avoid any project staff members or MOH staff from interviewing people they knew either personally or professionally. Data collection took place over a four-day period. Twice during that time, the entire evaluation team gathered to debrief. These two hour-long debrief meetings allowed the various team members, who had been working independently, to share their views about emerging patterns and trends and to verify if other members concurred with those observations. Over the four days, it became apparent that two of the questionnaires (for the Titulaire and Community Health Nurse) had been leaked, allowing interviewees to respond in a very predictable manner. To address this, on the last day, the external evaluator visited three health centers that were not on the original plan to interview the Titulaires and Community Health Nurses. Unfortunately, due to the spontaneous nature of this plan, two of the three Titulaires and Community Health Nurses were not present. In the end, only the open-ended questions for Titulaires and Community Health Nurses were analyzed. The data collected were analyzed by the evaluation team members in a participatory manner. Working in a group, the responses to each question for each type of informant were reviewed and analyzed quantitatively. For findings of consequence, the team was asked to formulate a conclusion and potentially a recommendation and/or a lesson learned. In this way, almost all of the recommendations found in this report came directly from the evaluation team members. In addition to the qualitative data collected through these interviews, the evaluation team also analyzed the quantitative data provided by the final Knowledge, Practice, and Coverage (KPC) survey (the final KPC report is shown in Annex VII), the Health Facility Assessment (HFA) (report in Annex XIX), and the OR study draft report (final report shown in Annex XV). The external consultant also reviewed a myriad of project documents (see Annex IX) including the DIP, MTE report, annual reports, training curricula and visual aids, data collection tools, training reports, and supervision and quality control tools. 5 December 2013 DATA QUALITY AND USE The MCSP has a very capable Monitoring and Evaluation (M&E) Officer whose skills improved over the life of the project. He was present during the entire project and was involved in all of the M&E studies conducted (baseline and endline KPC, OR KPCs, OR process evaluation, HFA, COSA functionality assessment). The end line studies were also supported by a Concern Trainee specializing in M&E. Project Monitoring: On a monthly basis, each of the 25 Community Health Nurses sent a monthly report to the project and to the District Health Information Officer. The information in the monthly report was compiled from monthly reports submitted by the 152 CHW, who in turn collected it from the 3,021 CGV. The CGV data was collected from approximately 30,000 families with pregnant women or children under age five during the monthly home visits. The community-based health information system (C-HIS) collected the following data from each household on a monthly basis: births; deaths of children under five (by age ranges) and perceived cause of death; cases of malnutrition as identified through mid-upper arm circumference (MUAC) screening and edema identified; cases of fever (suspected malaria), diarrhea, and cough (suspected ARI); and the number of children referred to the CHW and health center. CGV also collected the number of pre￾natal consultations each pregnant woman had, where the birth took place, if the mother breastfed within an hour of delivery and if she had a post-natal consultation. This data collection system established by the project and made possible by the establishment of the Care Groups represents the first time that the Mabayi Health District had access to data emanating directly from the community. Both the District Health Officer and a District Supervisor expressed satisfaction with the C-HIS. They told the external evaluator that this information is reviewed regularly by the DHT and used to identify health issues in the district such as unusually high peri-natal deaths and potential epidemics. In addition to the information mentioned above, the reports furnished by the CHW also included information on the functioning of the CGV (number of home visits conducted) and the function of the Care Groups themselves (number of CGV, number of meetings held with CGV, number of supervisory visits, number of reports submitted, etc.). The project team reviewed the data each month and used it to assess CGV and CHW performance. At a point in the project when all of the 23 health lessons had been covered, the project staff noticed that attendance at the second Care Group meeting of the month with had fallen off. After investigating this issue, they created a different meeting protocol that called for the CGV to choose the topic of discussion for the day. In this way, the meetings became more ‘need/interest’ driven and attendance at the second monthly meeting increased. Review of the vital events data also compelled the project team to bring the problem of high peri-natal deaths to the attention of the DHT. Once the CCM activities got underway, data generated by that reporting system was incorporated into the MOH health information system and was also reviewed by project staff. It was a review of this data which compelled the project to organize a review session on how to correctly complete the reporting forms and other paper work. Furthermore, the data was used to inform supervision activities. The external evaluator found the monitoring system well developed and functioning effectively. She reviewed some of the data collection tools with the M&E Officer, the Concern Trainee and HQ Health Advisor and together they identified was to simplify the reporting tools (registers), making them easier to complete and also more likely that the MOH will be able to reproduce the forms should the Care Group model be scaled up. The project conducted a number of different studies to inform various programmatic decisions. A Trials of Improved Practices (TIPS) study was conducted to inform the development of the nutrition modules and a Barrier Analysis was conducted to guide the development of the family planning module. In 2012 6 December 2013 the project conducted an OR process evaluation and also a study to assess the learning needs of the COSAs. The combined use of these qualitative studies and the monthly data collection plus the monthly review of project implementation against the work plan were an appropriate mix of monitoring approaches to ensure effective project implementation. Project Evaluation: The project conducted a baseline KPC survey in 2009 using the standard KPC questionnaire. It then administered the same KPC questionnaire in July 2013 to determine the degree of change over the life of the project. The project also conducted three HFAs which assessed the quality of care provided at the health center. Table 3 compares the baseline and final KPC results related to Objective 1. A review of the activities (Care Groups, CHW training and support, Community Health Nurse training and support, DHT training and support) associated with these outcomes suggests that the project activities very likely resulted in the changes shown below. The entire M&E Table is shown in Annex XIX and the results are discussed in the Findings, Conclusions and Recommendations section. Table 3: Outcome indicators for Objective 1: Improved health and nutrition practices Indicator Target Baseline Final Percentage of children 0-5 months who were exclusively breastfed within the previous 24 hours 80 67 90 Percentage of infants and young children age 6-23 months fed according to a minimum of appropriate feeding practices within the previous 24 hours 50 20 47 Percentage of children 6-23 months who ate solid, semi-solid or soft foods other than liquids the minimum number of times or more in the previous 24 hours 50 30 52 Dietary diversity score of children 6-23 months in the previous 24 hours 4 3.1 3.7 Percentage of children 6-23 months who ate from 4 or more diet diversity categories (of 8 total) in the previous 24 hours 50 37 51 Percentage of children age 0-23 months who slept under an insecticide-treated bed net the previous night 60 38 53 Percentage of children age 0-23 months with a febrile episode during the last two weeks who were treated with an effective anti-malarial drug within 24 hours after the fever began 60 26 11 Percentage of children 0-23 m with a febrile episode during the last two weeks who were taken to the health center within 24 hours 85 54 88 Percentage of children 0-23 months with diarrhea in the last two weeks who received increased fluids/breast milk and maintained normal feeding 50 11 47 Percentage of caretakers of children 0-23 months who washed their hands at least 3 of 4 critical times during the previous 24 hours 30 8 37 Percentage of mothers of children age 0-23 months who have soap at the place for hand washing at home 30 22 34 Percentage of children age 0-23 months with diarrhea in the last two weeks who received therapeutic zinc for 12 days 75 n/a n/a Percentage of children age 0-23 months with diarrhea in the last two weeks who were reportedly fed the same or more than usual 40 24 53 Percentage of children age 0-23 months with chest-related cough and fast and/or difficult breathing in the last two weeks who were taken to an appropriate health provider or antibiotic-equipped CHW 85 78 92 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS MAJOR ACHEIVEMENTS 1. The MCSP achieved or came within a few percentage points of achieving 12 of the 14 project indicators related to improved child health and nutrition practices. The project achieved or 7 December 2013 significantly improved all five indicators related to the strengthened community leadership objective. (The complete M&E table is located in Annex XIX) 2. The MCSP tested the Integrated Care Group implementation approach that relied more on the MOH and less on the NGO. The OR study found that the Integrated Care Group model was as effective as2 the traditional model. This trial paves the way for the MOH to replicate the Integrated Care Group approach in other districts. 3. A total of 302 Care Groups, facilitated by 3,021 CGV, were formed and trained in five technical areas (malaria, diarrhea, nutrition, family planning, and ARI). These CGV conducted home visits to approximately 30,000 households twice per month thus extending the reach of the 152 CHW into nearly every household with pregnant women or children under age five with life-saving behavior change information. 4. The project developed a community-based health information system (C-HIS) including data collection tools and training curricula, that has been integrated into the Mabayi Health District information system. The data generated by the C-HIS is used by district, health center and project leaders to take decisions based on previously unavailable vital statistics. 5. The project helped to increase access to quality malaria diagnosis and treatment services, by collaborating with the Department of Malaria Management to implement CCM of malaria. Through this initiative, 317 CHW were trained and have provided 27,826 consultations and treated 12,291 children. Titulaires report a reduction in the number of severe malaria cases at the health centers. 6. The MCSP developed a COSA capacity building curriculum and trained 25 COSAs to be equal partners with the Titulaire in managing the health center. This training tool can be used by the MOH to strengthen other COSAs throughout the country. FINDINGS To better understand the findings, conclusions and recommendations presented in this section, please refer to Tables 4a, 4b, 4c, and 4d, which show the key inputs, activities, outputs and outcomes of the MCSP for Objectives 1, 2, 3, and the OR, respectively. The narrative under each Objective is organized by the three main evaluation questions in the evaluation SOW. Objective 1: Improved household child health and nutrition practices Table 4a: Summary of Major Project Accomplishments for Objective 1 Project Inputs Activities Outputs Outcomes  Logistical and technical support for community elections of CGVs and CHWs  Human resource and material inputs for production of BCC modules for HC staff, CHWs, and CGVs  Logistical, human resource, and material inputs for completion of TIPS and Barrier Analysis studies  Community elections held for CGVs (Dec 2009-Jul 2010) and CHWs (Jun 2009)  Cascade trainings for HC staff, CHWs, and CGVs on BCC modules for ARI, malaria, family planning, diarrhea, and nutrition  Training and support to CGVs through bimonthly CG meetings (ongoing since Jun 2011)  Peer-to-peer BCC and support through bimonthly  3,021 CGVs elected and delivering BCC interventions through bimonthly home visits  152 CHWs elected and involved in CGV training/supervision  BCC modules on malaria, diarrhea, ARI, family planning, and nutrition developed, printed, and distributed  302 CGs Percentage of children age 0-5 months exclusively breastfed in the last 24 hours from increased from 67% at baseline to 90% at end-line  Percentage of children age 0- 23 months with febrile episode in the last two weeks who were taken to the health center within 24 hours increased from 54% at baseline to 88% at end￾line  Percentage of children age 0- 23 months with chest-related 2 While Concern used a non-inferiority statistical test to assess the relative effectiveness of the two approaches, for east of reading and comprehension the more familiar language, “as effective as” will be used in this report. 8 December 2013 Project Inputs Activities Outputs Outcomes  Logistical, human resource, and material inputs to build district and HC staff capacity in BCC and cascade training approach  Logistical and human resource inputs for the supervision of CG meetings and home visits home visits from CGVs for households with children under five and pregnant women  Creation of Community Health Focal Point position at HCs and training of Focal Points to manage CHW training and supervision (Dec 2012) established and meeting bimonthly to receive training on BCC modules3  Staff from 25 HCs trained in the BCC modules  26 Community Health Focal Points trained in CG approach and CHW management and supervision cough and fast and/or difficult breathing in last two weeks who were taken to an appropriate health provider increased from 78% at baseline to 92% at end-line  82% of CGs had at least 80% attendance in at least one meeting per month (Jun-Jul 2013)  Increase in percentage of households with a child under five and/or pregnant woman that received a CGV visit in the last month from 74% (Jun-Aug 2011) to 91% (Jun-Jul 2013) To what extent did the project achieve the project objectives as stated in the project DIP? The MCSP achieved, or came within a few percentage points of achieving, 12 of the 14 indicators related to Objective 1. This was achieved by implementing the Care Group approach which enabled the 25 MOH Community Health Nurses to train 152 CHW in five technical areas (malaria, diarrhea, nutrition, ARI and family planning), who in turn trained 3,021 CGV. The CGV then visited an estimated 30,000 families at home – one of the most potent behavior change strategies - twice each month. The Contact Intensity Index employed in the endline KPC and showed that 80% of mothers had received at least one CGV visit in the previous month while nearly 60% reported having received two or more CGV visits. The coverage of all “personal contacts” (which includes home visits from CGV as well as from doctors, nurses, CHW, and other health volunteers) was even higher, with 85% having received at least one contact and 75% having received two. Given the level of coverage and intensity of this behavior change strategy it is quite clear that the results of the KPC survey can be attributed with confidence to the Care Group strategy. Establishment of the Care Groups was delayed because the Care Group model was also the subject of the OR and it took more than 1.5 years for the OR study protocol to be approved by USAID. This was due in part to the fact that project staff and Concern HQ were not very experienced with writing OR protocols at that time. It also bears mentioning that USAID was also on a steep learning curve with regards to OR, and struggled to help the grantees fulfill their requirements. Once the protocol was approved in February 2010, another seven month delay was incurred seeking institutional review board (IRB) approval, since there was no ethical review committee in Burundi. Delays were also caused by the long process required to elect CHW who are required to facilitate the Care Groups, and to conduct the census needed to create the Care Groups. Furthermore, during this period all project work was suspended for a period during Presidential elections. Project staff also explained that since they were new to the Care Group approach, they also had to learn about the model as they carried it out; and this took time. All of these delays resulted in postponement of Care Group implementation, the key strategy linked to Objective 1, and this in turn put pressure on the project team to speed up the Care Group establishment process and cut some corners with regard to the development and testing of behavior change communication (BCC) materials. 3 The number of CGs decreased from 346 in Year 4 to 302 due to the reorganization and merging of some groups. 9 December 2013 Once Care Group implementation commenced in May 2011, the Care Group approach was implemented as planned, but in a more accelerated manner, given the delays incurred. Aside from the delays there were a few other Care Group implementation issues worth mentioning. During the MTE it was revealed that many husbands were disgruntled that their CGV wives (and to a lesser extent the female CHW) were spending so much time away from home and not tending to their household chores. As a result, some CGV quit and they had to be replaced. Aware of the issue, project staff convened a series of meetings with CGV husbands to solicit their support. Support for CGV’s work was also sought from the local authorities (colline heads) and the latter were invited to attend the bi-monthly meetings between CHW and CGV so they would better understand the importance of the work. Although some improvement was noted following these efforts, during the final evaluation some key informants still noted continued resistance from the husbands. On the other hand, a few key informants mentioned positive changes in some husbands. As far as other gender issues are concerned, it is worth noting that 100% of the CGV are women and more than half of the 152 CHW in charge of the Care Groups are female (82/152). Furthermore, CGV and CHW mentioned that their work improved their reading and writing and public speaking skills. After the MTE, the MCSP team decided to add a module on family planning because many key informants interviewed expressed interest in learning more about contraceptive use. The team conducted a Barrier Analysis around this topic and developed training modules and visual aids. When word got out to the religious leaders that the CHW and CGV were talking about contraceptive use there was considerable push back, even some threats of ex-communication. Faced with this serious predicament, project staff had to address the issue. They met with religious leaders on a couple of occasions and even asked the colline authorities to help mediate the situation. Through this series of events a couple of important lessons were learned. It is usually not a good idea to veer off the planned program and launch into a new technical area without the time spent to reflect and plan. Adding a new topic midway through the project and especially when the project is already behind schedule is also not recommended. Instead of working on new technical topic, the time would have been better spent strengthening the supervision of the Care Group initiative: introducing the quality improvement verification checklist (QIVC) and supportive supervision as a routine part of the program. While achievement of the project indicators can be confidently attributed to the Care Group approach, the quality of the bi-monthly meetings and home visits could be strengthened. Specifically, training on how to conduct the meetings (between CHW and CGV) should be more detailed and focus on the facilitation methodology, not just the technical content (the message). Furthermore, the power of the behavior change approach would be strengthened if instead of two home visits per month, the CGV conducted one group meeting with their Neighbor Group (all the women they normally visit each month) at the beginning of the month and followed by a home visit later that month. The meeting with all the Neighbor Group members serves to create the ‘enabling environment’ where the mothers hear the same information at the same time and recognize that they will not be the only one trying out this new behavior. The group meeting also allows the women to share their opinions about the new behavior with their neighbors and together discuss the barriers to behavior change and ways to overcome them. Finally, making a commitment to try the behavior with their neighbors as witnesses is a proven potent behavior change strategy. The external evaluator reviewed the materials used by the project to train HC staff, the CHW, and CGV and found that too much focus was placed on communicating ‘messages’ and not enough was being assigned to behavior change. Everyone involved in the Care Group approach should recognize it as a behavior change strategy and employ facilitation methods that focus on helping mothers (and other family members) to adopt illness preventing, life-saving behaviors. This includes discussing current behaviors, identifying barriers to behavior change, ways to remove those obstacles, and finally a 10 December 2013 commitment to change. This approach should be modeled from the top of the learning chain (district to Community Health Nurse) on down to the Neighbor Group mothers. At present the project requires that all CGV be literate since the job includes completing written reports. This is unfortunate since the literacy rate is low, especially among women. This means that women who may be very motivated and effective as behavior change agents are being passed over because of this criterion. Many Care Group programs around the world operate in similar situations but find ways to include illiterate women as CGV. What was the appropriateness and effectiveness of key project interventions, as well as their potential for scale￾up and sustainability? As mentioned above, the key intervention for Objective 1 was Care Groups and it is clear that this behavior change strategy resulted in the changes measured by the baseline and final KPC surveys. It is not news that Care Groups are effective as a behavior change strategy. What is noteworthy, however, is the effectiveness of the Integrated Care Group approach designed and implemented by Concern Worldwide. Other NGOs in Burundi have implemented Care Groups in the traditional way, but Concern is the first NGO in the world to test a way for the MOH to assume greater responsibility for Care Group implementation. Using the CHW to train and supervise the CGV and the Community Health Nurses to support the CHW has been shown to be as effective as the traditional Care Group approach which relies completely on NGO support. Furthermore, the ratio of CHW to Care Group is lower (1:2) than in the typical Care Group approach (1:9) which means the CHW, a volunteer, will not likely be burdened by the workload but rather s/he will consider the Care Group an approach that lightens her work load. The Integrated Care Group approach essentially enables a small number of CHWS (two per colline) to reach into the homes of 80% of all households on a frequent (bi-monthly) and regular basis with life￾saving information. This effectively expands the reach of CHW, increasing their coverage enormously. One of the hallmarks of the Care Group strategy is that the materials used to change behaviors should be informed by formative research. This helps the behavior change process by addressing the barriers identified through the research. The MCSP carried out two types of formative research to inform two of the five modules: nutrition and family planning. The first type of formative research was TIPS, which was used to inform the nutrition module. The first in-country Health Advisor took responsibility for this research which he contracted out to a local research group. Unfortunately, it is clear from the report that neither the contractor nor the in￾country Health Advisor was familiar with this formative research approach. Nevertheless, the project gleaned some useful information from the findings and used them to create a visual aid that showed all the types of foods that mothers could choose from to serve a balanced diet to her child. They did not however use one important finding – that mother also give beer to their children – in the visual aids on nutrition, which would have been a very important message, indeed. After having attended a workshop on qualitative research methods in 2011, the project teams decided to conduct a Barrier Analysis survey on family planning. Family planning was a subject that was added to the list of behaviors promoted by CGV following the MTE, when substantial interest was expressed on the part of some key informants interviewed. The Barrier Analysis revealed that there are many myths about the use of contraceptives. The project addressed this indirectly by explaining the side effects of each method. While this may have addressed the myths, it would have been better had the lesson specifically asked the mothers about myths and then added the correct information regarding side effects. From these two examples, it is evident that guidance on how to use research results to inform visual aid development and lesson planning would have been useful. The Care Group initiative only got underway in May 2011 – more than two years into the project. This left ample time for the Training Officer to plan the modules on the four technical topics – malaria, 11 December 2013 diarrhea, nutrition, and ARI. Unfortunately, this did not happen and instead, when the Care Group approach began to unfold the Training Officer had to quickly develop the required modules. This did not leave time to have the modules reviewed by the HQ Health Advisor or to test the visual aids. As a result, there are some inaccurate messages, some inconsistencies between the different modules (Titulaire, CHW, and CGV) and some of the visuals appear inappropriate (bacteria on a hand being shown under a magnifying glass). The entire idea behind the Integrated Care Group approach was to test the ability of the MOH to support the Care Groups so the approach could be sustained and scaled up. During the final six months of the project Concern staff ceased to support Care Groups. The project found that activities continued well without support from the project. These results suggest that the Integrated Care Groups have a good chance of being sustained with limited support from an NGO. The District Health Officer expressed her intention to continue the approach after the departure of Concern. That said, regarding Care Group scale-up, the Director of the Department of Community Health raised many questions that would have to be answered before the MOH would be willing/able to replicate the approach in other districts. The project recognizes that questions remain and plans to address some of them in an Integrated Care Group protocol. This protocol will inform the scale up process. What was the extent of collaboration with the MOH, UNICEF, and other USG-funded partners at the national, provincial and district levels? Collaboration between the Mabayi District Health Team and the MCSP team was quite strong regarding Care Group implementation despite frequent changes in the District Health Officer position. The District Supervisors helped the project Training Officer to develop the Care Group lesson plans and then together they trained the Titulaire and other health center staff. UNICEF shared visual aids with the project to use in the training modules and visited the project to better understand the approach. At the national level, the Project Manager participated in meetings and helped develop the national Community Health Procedures Manual. He strategically identified a place in the plan where the Care Group approach can be introduced for scale-up consideration. During one of these meetings the Project Manager introduced the Care Group approach along with the Integrated CG approach being tested through the operations research. Objective 2: Improved access to quality basic child health care services with a balance of provision at the health center and community levels. Table 4b: Summary of Major Project Accomplishments Project Inputs Activities Outputs Outcomes  Logistical, human resource, and material inputs for: training of district and HC staff; conducting joint HC supervision visits with DHT; supervision of malaria CCM; the design, piloting, and launch of a C-HIS  Development and printing of CHW tools and registers  Scales, respiratory timers, and pitchers, cups, and spoons (for ORS) provided to  Training of DHT in health planning, information systems, BCC, M&E, budgeting, and drug management (Jul-Sep 2009)  MOH trainings for district and HC staff in IMCI (Oct-Dec 2009) and C-IMCI (May 2012) with support from Concern  Advocating with MOH for the integration of CMAM into National Nutrition Policy (policy finalized in Apr 2010)  Training of HC staff in CMAM policy/protocols (Jul-Aug 2010)  Assisting in the development of the National Community Health Strategy (finalized in Jan  DHT trained in health planning, management, and M&E  Essential supplies (e.g. timers, containers for ORS preparation) provided to 25 HCs  30 HC staff trained in IMCI  29 HC staff trained in C￾IMCI  National Nutrition Policy finalized with CMAM integrated  21 HC staff trained in CMAM protocol  First National  Percentage of five key assessment tasks for child health performed per child consultation in all public HCs increased from 19% (an 2009) to 50% (Jul 2013)  Percentage of CHWs providing correct treatment and referral according to CCM guidelines from 67% (Aug-Sep 2012) to 92% (Jun-Jul 2013)  Percentage of health centers receiving 12 December 2013 Project Inputs Activities Outputs Outcomes HCs  CHW kits, ACT, and RDTs purchased for malaria CCM using cost-share funds  Human resources invested in advocating with national MOH for integration of CMAM and approval of CCM  Human and material resources invested in developing National Community Health Strategy and integrated CCM curriculum 2012)  Training-of-trainers for malaria CCM (Mar 2012)  Election of additional CHWs and training of CHWs in malaria CCM (Apr-May 2012)  Distribution of malaria CCM kits to 22 public HCs (Jul 2012)  Monthly supervision visits for CHWs in malaria CCM provided by MCSP and HC staff (ongoing since Aug 2012)  Design, piloting, and launch of a C-HIS in close collaboration with MOH officials from MHD (data collection began in Jun 2011)  Development of an integrated CCM curriculum on malaria, diarrhea, and ARI with national MOH officials and NGO partners (finalized in Jun 2013) Community Health Strategy developed with Concern input  34 trainers in malaria CCM trained  165 additional CHWs elected and 317 CHWs trained in malaria CCM  Secured cost share funds for 15,600 RDTs and 10,620 ACT blister packs  Monthly supervision plan for malaria CCM implemented by HC and MCSP staff  Functional C-HIS (including CG data) established in MHD  Integrated CCM curriculum for CHWs and HC staff finalized quality supervision from the DHT in the past quarter increased from 19% (Jan 2009) to 75% (Jul 2013)  Percentage of HCs with all six essential medicines to support child health in stock increased from 0% (Jan 2009) to 43% (Jul 2013)  94% of trained CHWs submitting monthly reports on CG Activity Indicators and C-HIS to HCs per month (Jun-Jul 2013)  90% of CGVs reporting on C-HIS data to CHWs or Health Promoters per month (Jun-Jul 2013) To what extent did the project achieve the project objectives as stated in the project DIP? The strategies employed to achieve Objective 2, as shown in Table 4b, include the organization and support of a myriad of training opportunities for members of the Mabayi District Health Team and also for staff in the 25 health centers in the district. Furthermore the MCSP participated in a number of national level working groups to establish policies that contribute to improved quality of care. Among these were CMAM and the Community Health Strategy (or Community Health Procedures Manual). The most important strategy linked to Objective 2 was the implementation of CCM of malaria which got underway in April 2012 once the MOH approved the policy and developed the protocol. Despite the late start, the project was able to recruit and train 317 CHWs to diagnose and treat simple malaria at the community level and to refer serious cases to the health center. Although it was originally planned that the Health Promotion Technicians and/or Titulaire at the health centers would supervise the CHW, once the CCM program got underway it became apparent that neither person had the time to adequately oversee and support the CHW, especially their CCM work which requires monthly supervision. In view of this, the project, in consultation with the DHT decided that another nurse at the health center would assume responsibility for CHW supervision and support and C-HIS data collection/compilation. This person was named the Community Health Nurse. This decision was a critical positive change to the original plan as regular supervision is an essential part of both the CCM and the Care Group strategies. Of the eight indicators used to measure achievement of the second objective, the Health Facility Assessment (HFA), C-HIS, and CCM records showed that three had been fully achieved. Some of the short fall can be explained by high turnover in staff, particularly among the Titulaires who had been trained during the first year of the project. By the fifth year of the project, more than half of the 25 health centers had Titulaires who had not been present during the first year of the project. This dilemma highlights the need to incorporate such courses as IMCI into pre-service training. Another cause of the shortfall is the change in first line drugs between the baseline and final HFA surveys. And finally a few of 13 December 2013 the indicators fell short due to one deficiency among many requirements. For example, percentage of health centers with the six essential medicines would have been 95% had Vitamin A been removed from the essential medicines list. In Burundi, Vitamin A is not supplied to health centers because it is only given out during national campaigns. While the endline KPC results shown in Table 3 related to malaria treatment seem to suggest otherwise, the CCM approach increased coverage and access to malaria diagnosis and treatment substantially. The C-HIS showed that between August 2012 and July 2013, the CCM-trained CHW consulted 27,827 sick children and treated 12,291 of them. The poor results compared to the baseline KPC are likely due to the way the KPC question was worded. Whereas at baseline the mothers were allowed to select all of the drugs that their child had received for malaria, the final questionnaire only recorded the most recent drug that the child had received. This may have acted to skew the results given that the list of potential drugs in the final KPC questionnaire included both ACT and Paracetamol, which are often prescribed together, and thus children who were prescribed both drugs but who were administered the Paracetamol first would have been erroneously classified as not having received ACT. A review of the activities related to Objective 2 indicates that, as a package, both access to and quality of care were improved as a result of these interventions. Titulaires interviewed for the final evaluation were able to mention several skills that they acquired from training courses supported by the project. These include more complete child consultations (IMCI), diagnosis and treatment of malnourished children (CMAM), and activity planning. The CCM approach also improved equitable access to health services. The ratio of CHW to geographical area (colline) was changed by the MOH so from the original 152 CHW recruited to facilitate Care Groups, an additional 93 CHW were recruited and trained for a total of 317 CHW providing CCM services. Of the 317 CHW, 174 are women, once again over half. Community members interviewed for the evaluation mentioned not having to have a birth certificate to seek health care now that CHW have been trained to offer services in the community. The project also introduced the Community-based Health Information System (C-HIS) which was also mentioned by the Titulaire as an achievement of the project. The C-HIS allowed the health center staff to access such data as births, deaths, cases of diarrhea, malaria, ARI, and malnutrition. This data was collected by CGV from the households they visited each month. It was then compiled by the CHW and taken to the Community Health Nurse at the corresponding health center. The Community Health Nurse compiled those reports and shared them with the Titulaire. The C-HIS data was then combined with the health center monthly report and sent to the Mabayi District Health Office and was then included in the district monthly report to the Provincial Health Office. The data was examined at each health center, at the district level and by the project. Although it is not clear if actual action was taken based on the C-HIS, the project did bring to the attention of the district the unusually high number of peri-natal deaths recorded. The project had hoped to also introduce CCM for diarrhea and ARI, but the policy was not passed before the project ended. Likewise the policy on the use of zinc in the treatment of diarrhea was only approved for health center use in the fourth year of the project but not for use at the community level. What was the appropriateness and effectiveness of key project interventions, as well as their potential for scale￾up and sustainability? The key interventions for Objective 2, training, policy support, CCM, and the C-HIS, were all appropriate and to varying degrees effective. Of the four activities, CCM is the most likely to have lasting and wide spread impact. The Malaria Management Department Director told the external evaluator that the MOH is very interested in replicating the CCM approach and expanding it to include diarrhea and ARI case management. To this end the department organized an evaluation of the CCM 14 December 2013 approach in Mabayi Health District, which was taking place at the same time as the MCSP final evaluation. The results of the CCM evaluation will be used to inform scale-up efforts. The training of the CHW in CCM took place over a period of two weeks and included a practicum in the health facility. This not only helped the CHW to gain skills in diagnosis, but strengthened the rapport between the health center staff and the CHW. The training could be improved by spending more time during the training on correct report completion. Also, CHW should be given more guidance during follow-up visits to children who have been treated for malaria. CCM effectiveness was also diminished by stock outs of rapid diagnostic tests (RDT) and in some cases ACT. According to a MSH participant in the final presentation of the final evaluation results, the cause of the nation-wide stock out is apparently linked to higher than expected number of malaria cases nationally. The evaluation team also speculated that in Mabayi District, the CCM approach increased access to services to the extent that supplies could not keep pace with the demand. The support for training provided by the project was appropriate and effective in helping health care providers and administrators gain new skills that improved the quality of care and the quality of support (supervision). These efforts were somewhat undermined, however, when trained staff was transferred outside the district, as mentioned above. Having seen and experienced the benefits of the C-HIS, it is likely that the Mabayi health centers and District Health Team will continue to collect, analyze, and use the C-HIS data to make decisions. However, it is not clear to what extent the C-HIS will be replicated in other districts, since it is dependent on the CGV collecting data each month from their target households. A separate C-HIS currently being considered at the national level seems much less practical since it relies entirely on the CHW to collect and transfer data and the amount of data to be collected is much greater. If the need for a national C-HIS is felt strongly enough, then it might turn out to be the catalyst for scaling up the Care Group approach. What was the extent of collaboration with the MOH, UNICEF, and other USG-funded partners at the national, provincial and district levels? All of the activities related to Objective 2 necessitated a strong collaboration with the MOH, especially the Departments of Nutrition, IMCI, and Malaria Management. The project supported the inclusion of CMAM into the National Nutrition Policy and then supported the training of key health center staff in its use. Likewise, the project supported the Ministry’s goal to roll out IMCI by supporting the training of key health center staff in both IMCI and community-IMCI. And finally long and productive collaboration with the Malaria Management Department was required to finalize the CCM protocol, to develop the CCM training curriculum, and then to train the 317 CHW in the protocol. The project played a key role in developing the materials needed to test the CCM approach and was instrumental in helping the MOH to pilot this strategy. Once initiated, the project also supported CCM at the district level by supporting monthly supervision and collecting and analyzing data related to CCM. Objective 3: Strengthen community leadership in health Table 4c: Summary of Major Project Accomplishments Project Inputs Activities Outputs Outcomes  Logistical, human resource, and material inputs for: CG formation and support; COSA elections  See activities mentioned under Objective 1 for Care Group formation.  COSA elections held (Feb￾Mar 2010, reelections in June 2012) and all COSA members trained in accordance with MOH guidelines  See Outputs under Objective 1 related to CG formation  Functioning COSAs established in all 25 public HCs in MHD  337 COSA members trained in roles and in COSA  Percentage of public HCs with an established COSA increased from 0% at baseline to 100% (Aug 2013)  Percentage of COSAs that held a meeting and produced a meeting report at least two times in previous quarter 15 December 2013 Project Inputs Activities Outputs Outcomes and training; COSA capacity/ functionality; sensitization of local leaders and husbands of CGVs  Staff time dedicated to establishing and participating in PBF Verification Committee  Human resources invested in developing national guidelines for PBF at both facility and community levels  Completion of a COSA capacity and functionality assessment (June 2012)  Awareness raising sessions held with local administrative and religious leaders to discuss CG approach and foster community ownership of CG activities  Participating in supervision of PBF implementation as an official member of the PBF Verification Committee for Cibitoke Province (established in Apr 2010)  Participating in Community Health and PBF Policy meetings to develop national guidelines for PBF and advocate with MOH for inclusion of CHW indicators ToR (Feb-Mar 2010 and Oct 2012)  Final report from the assessment of COSA capacity/functionality  92 administrative and 51 religious leaders sensitized on benefits of CG approach and their role in supporting CG activities  Functioning PBF Verification Committee established in Cibitoke  National guidelines for facility-level PBF finalized (Year 2) with community indicators integrated (Year 5) increased from 67% (May-Jul 2010) to 84% (May-Jul 2013)  Increase in CGV attendance in first CG meetings from 70% the two months before the sensitization of husbands (Jan-Feb 2012) to 81% the two months following the sensitization (Mar-Apr 2012), and in the second CG meetings from 61% before sensitization to 72% after sensitization  94% of trained CHWs submitting monthly reports on CG Activity Indicators and C-HIS to HCs per month (Jun-Jul 2013)  90% of CGVs reporting on C-HIS data to CHWs or Health Promoters per month (Jun-Jul 2013) To what extent did the project achieve the project objectives as stated in the project DIP? Five indicators were chosen by the project to measure achievement of the third objective. Four of the five indicators were achieved and the fifth showed significant improvement since baseline. Three of the indicators are related to the Care Group approach and two are related to the Community Health Committees (COSA). The two most important strategies associated with Objective 3 were the strengthening of the COSA and the implementation of Care Groups. Together, the COSA and the CGV serve two distinct and complementary roles. The CGV serve as leaders in their communities in the prevention and detection of major childhood illnesses. They help create demand for services and refer suspected cases of diarrhea, malaria, malnutrition, and ARI to the health center for treatment. They also play a critical role in the C-HIS. The COSA, and in particular the COGES (Comite de Gestion, or the health committee’s management sub-committee), on the other hand, help ensure the transparent management of the health center. The final evaluation team examined the function of each of these two leaders and concluded that each has contributed to achievement of the outcomes. In fact, the CGV have contributed significantly to outcomes related to all three objectives. The CGV leadership role is significant because together they reach into an estimated 80% of all households and 100% of households with pregnant women and mothers of children under five. The COGES’ leadership role is noteworthy because, thanks to the requirement that they have higher education, they are able to interact with the Titulaire in a more equitable manner. COSA members interviewed for the final evaluation explained that, in the past, some Titulaires acted as though the health center was his/her private domain. Now, with COGES involvement, they are more honest and transparent. COSA members have also been instrumental in making sure that health services are accessible to even the most vulnerable members of society. They do this by helping to create a list of 16 December 2013 ‘unable to pay’ citizens, which is used by health center managers to afford free health care to those deemed too poor to pay. The project was able to support the strengthening of COSA members because in 2012 the new manual on Community Health Procedures was ratified by the MOH and the term of prior COSA members was coming to an end. This meant that new COSA/COGES members would be elected and needed to be trained. Whereas the MOH only provides a brief orientation to new COSA members, the project provided a five-day skills-focused course. The guidance provided in the Community Health Procedures Manual stipulates that half of the COSA members be women. Is also requires all COSA members to be functionally literate. The latter criterion makes it nearly impossible to meet the former requirement because literacy levels among women are very low. Faced with this dilemma, the MOH should consider relaxing the literacy criterion particularly for non-COGES members. What was the appropriateness and effectiveness of key project interventions, as well as their potential for scale up and sustainability? Both the Care Group and COSA interventions were appropriate and effective. The assistance the project provided to the COSA initiative helped strengthen an existing MOH strategy and piloted a means (through the COSA training curriculum) to further reinforce it. The Care Group approach was very appropriate because it provided a way for a small number of CHW to reach into the homes of 30,000 families by working through the CGV. It is also an appropriate leadership initiative because it focuses on prevention and surveillance activities and uses well respected and trusted women as behavior change agents. While the COSA were already established when the project started, in 2012 the Community Health Procedures Manual was ratified, which further clarified the selection criteria, their roles and responsibilities, and new members were elected. To best determine how to prepare the new COSA members for their roles; the project wisely conducted a COSA Capacity and Functionality Assessment in early 2012. This assessment revealed that COSA members were largely unfamiliar with their roles and responsibilities and not playing their intended part in the management of the health center. Based on these findings, the project Training Officer designed a five-day training course for COSA members (three days for all COSA members and two days for the management sub-committee). Using this curriculum, the project supported the training of 337 COSA members, many of whom are women. The assessment conducted ensured that the training addressed the skill-building needs of the new COSA members. It also prepared them to stand up to the Titulaire if they did not agree with proposals s/he makes. During the final evaluation, COSA members were able to cite specific examples of proposals made by the Titulaires with which they expressed disagreement. This suggests a much greater degree of confidence in their role and the ability to communicate with Titulaire in a more equitable manner. Of equal importance, both the COSA and Titulaire were able to mention specify ways that the COSA had assisted the health center. Because it is already a MOH-accepted strategy, the COSA approach will be continued. What is not clear is if the improvement on the strategy made by the project, the more intensive training, will be continued and/or scaled up since to do so would require the MOH to reallocate scarce resources. What was the extent of collaboration with the MOH, UNICEF, and other USG-funded partners at the national, provincial and district levels? The project participated in the development of the Community Health Procedures Manual, which set the stage for both interventions related to the leadership objective. Project staff also kept the Community Health Department informed about the Care Group approach and the OR being conducted. Because the District Health Team was involved in all aspects of Care Group implementation (including module development, training, supervision, and data collection) they are now able to replicate 17 December 2013 and support this initiative with less support than previously. They should also be able to train the COSA members using the new training curriculum. Operations Research Table 4d: Summary of Major Project Accomplishments Project Inputs Activities Outputs Outcomes  Human resource input for development of the OR protocol; data collection tool development and data collection and analysis  Logistical and human resource inputs for the election, training, and supervision of CGVs in Bukinanyana; for baseline and final OR KPC surveys  Development of OR Protocol  Facilitating community election of Traditional and Integrated CGs in Bukinanyana (Dec 2009)  Training and supervision of CGVs by Promoters (Traditional CGs) and CHWs (Integrated CGs)  Completion of fieldwork for baseline (Sep-Nov 2010) and final (May 2013) OR KPC surveys  Completion of Final OR Report (Aug 2013)  Approved OR Protocol  51 Traditional and 45 Integrated CGs formed in Bukinanyana  992 CGVs elected and delivering BCC interventions through bimonthly home visits  Report from Midterm Process Evaluation  Report from baseline OR survey  Final OR Report  The MCSP held national￾level meetings in May and September 2013 to present the OR results and discuss with MOH officials and NGO partners on how the Integrated CG model could be incorporated into the National Community Health Strategy  National MOH officials have indicated their interest in adopting the Integrated CG model on several occasions and conducted a field visit to MHD in May 2013 to observe the implementation of Integrated CGs Information related to the Integrated Care Group approach is included earlier in this chapter related to Objective 1 and the OR Report included in Annex XV. This section will cover other OR-related issues. The MCSP was approved for funding in 2008 and was among the first batch of CSHGP-funded projects which required that an OR study be conducted. While the MCSP eventually agreed to study the Integrated Care Group approach, this was not the original study topic proposed by Concern. This was the topic finally agreed to between Concern and USAID after months of discussion and negotiation. The OR tested two slightly but significantly different ways of implementing Care Groups. The difference between the traditional Care Group model and the Integrated Care Group model is that in the Integrated model, instead of using NGO-paid and supervised staff (Promoters) to train and support the CGV, these tasks were performed by volunteer CHW, recognized and supported by the MOH. In the case of the MCSP, the Promoters were women with low-level nursing degrees, whereas the average CHW has a 6th grade education. In the Integrated Care Group model, project Animators did help with supervision, but otherwise the MOH took primary responsibility for supporting the CHW and the CGV in the Integrated areas. The traditional Care Group model was implemented in half of Bukinanyana Commune, one of the three communes supported by the project, which means that the Integrated model was implemented in 2.5 Communes, or 5/6 of the project area. The project found that the Integrated Care Group model performed comparably to the traditional model with regard to functionality, effectiveness, and sustainability. Despite the delays in initiating the Care Group approach, the OR was implemented as planned. Only one small implementation issue occurred when the project Animators pitched in to help out in the OR study commune during a very short period of time. When the project staff discovered this, they re￾explained the purpose of the study and the importance of NOT helping out where it wasn’t planned. This research is very relevant to the global health community because the traditional Care Group approach has been proven to be effective in reducing child malnutrition rates, but at substantial cost to the NGO, and is currently being employed as a behavior change strategy in more than 20 countries. It 18 December 2013 has generally been assumed that if MOH staff were substituted for any of the key positions in the traditional Care Group model, the results would suffer. The fact that key child health and nutrition behaviors improved in comparable ways in the area where the Integrated approach was implemented as in the traditional area, suggests that the MOH-supported CHW were just as effective in training and supervising their CGV as the NGO-paid Promoters were. This is a significant finding. Implementation of the Integrated Care Groups not only revealed what worked but also exposed the areas that still require investigating before the Department of Community Health would be willing to consider scaling up the Integrated Care Group approach. These questions include:  Who will initiate the approach?  What data should be collected?  How will quality control and formative supervision be provided?  How will the behavior change promotion materials be reproduced?  How will newly assigned MOH staff be integrated into the Care Group program?  What are all the costs involved in all these and how will these be covered? Presumably the reason why USAID included the OR category in the CSHGP was to strengthen the grantees’ abilities to conduct OR and through the research generate irrefutable evidence related to question(s) being studied. It might also be the intention that decision makers at the national level might be more inclined to take note of such study results and be moved to take decisions based on them. The HQ Health Advisor interviewed for the final evaluation agreed that she and the in-country people involved with the OR gained skills related to M&E and implementing surveys. As with all capacity￾building initiatives, however, skill development at the individual level does not necessarily translate into organizational capacity. For this to occur, the organization would need to invest in gaining that capacity on a more widespread and permanent basis. In Concern’s case, the organization’s capacity is being applied in Kenya, Sierra Leone, and Niger where their child survival projects also have OR components. The evaluation team considered the question of whether MOH officials would be more likely to be moved to action by OR study results; in other words to consider scaling up an approach such as the Integrated Care Group, if the OR results proved its effectiveness. It was generally thought that MOH officials are more likely to ‘listen’ to large well-funded projects and that projects that only operate in a small geographical area and have a small budget (such as CSHGP grants) are not likely to be taken as seriously. In fact, during the presentation of the final evaluation results, one (non-project) participant accused the MOH of not showing interest in the Care Group approach despite other organizations proclaiming its effectiveness over the years. It was also thought that given the profile of the average MOH civil servant, study results are not likely to be the motiving factor unless, once again, the study is undertaken by a large donor, as in the case of UNICEF and the PD/Hearth study. Decision-makers are more likely persuaded, it was thought, through field visits and other ‘visual’ experiences. Unfortunately, the USAID Mission in Burundi, which, as a major donor, could serve as a port-parole for the OR results at the MOH table, has only shown cursory interest in the project. With these challenges in mind, the Project Manager has already taken steps to open dialogue with the Department of Community Health regarding the future of the Integrated Care Group model. In May 2013 and again in October 2013, the project was visited by a team of people interested in examining the Integrated Care Group model. This includes the Director of the Community Health Department, a UNICEF representative and other NGOs. Together they will discuss some of the questions listed above and perhaps identify the answers. These will be included in an Integrated Care Group protocol that will serve as a guide for the replication of the Integrated approach. If the Director of Community Health is convinced of the Integrated Care Group model’s effectiveness, he may require all NGOs to support the MOH in its implementation of the Integrated Care Groups, similar to the way NGOs help the MOH to implement EPI programs etc. 19 December 2013 MOST CRITICAL ACHIEVEMENTS 1. The MCSP achieved or came within a few percentage points of achieving 12 of the 14 project indicators related to improved child health and nutrition practices. The project achieved or significantly improved all five indicators related to strengthened community leadership objective. 2. The MCSP tested the Integrated Care Group approach that relied more on the MOH and less on the NGO. The OR study found that the integrated model was as effective as the traditional model. This study paves the way for the MOH to replicate and scale up the Integrated Care Group approach in other districts. 3. The project helped to increase access to quality malaria diagnosis and treatment services, by collaborating with the Department of Malaria Management to implement community case management of malaria. Through this initiative, 317 CHWs were trained and have provided 27,826 consultations and treated 12,291 children. Head nurses report a reduction in the number of severe malaria cases at the health centers. CONCLUSIONS AND RECOMMENDATIONS Table 5 provides the key findings, and their corresponding conclusions and recommendations. Table 5: Recommendations Finding Conclusion Recommendation Who Is Responsible Objective 1: Improved household child health and nutrition practices  The Care Group training materials reference ‘communicating messages’ very frequently.  This focus on ‘messages’ detracts from the purpose of the Care Group approach and the work of the main actors (CHW and CGV) as behavior change agents.  CGV are not asked to track behavior change among their target mothers.  Opportunities are being missed to focus attention on the main purpose of the Care Group approach: behavior change.  CHW and CGV may mistakenly think that communicating messages is the primary purpose of their job.  Concern should review their Care Group training materials and substitute the words ‘behavior change’ for ‘messages’.  CHW and CGV should be referred to as ‘behavior change agents’.  Consider asking CGV to monitor the adoption of healthy behaviors among their Neighbor Group women. Concern/Burundi Future Care Group approach implementers.  The project has required that all CGV be literate because the project wants them to be able to collect data using the written word.  It has been difficult to find enough literate women to be CGV.  Most Care Group programs around the world do not require CGV to be literate.  It’s possible that women who would otherwise make very good CGV (from a behavior change perspective) are not being accepted as CGV due to poor literacy skills.  The project should drop the literacy requirement.  For illiterate CGV, ask her to name a literate person (husband, child) to help with report completion.  Have illiterate CGV report their data orally to the Care Group Responsible (a literate CGV who is already responsible for compiling the data from all Care Group members) or CHW each month. Future Care Group approach implementers 20 December 2013 Finding Conclusion Recommendation Who Is Responsible  CGV are asked to make two home visits per month to each pregnant woman and mother with children under five in their group. This can mean up to 20 home visits per month in addition to the two meetings with the CHW each month.  Some CGV complain that the job takes too much time.  Work load may contribute to attrition among CGV.  There are ways to reduce the work load while at the same time focusing on the most vulnerable populations.  Future Care Group implementers should change the model so that CGV meet with all of their Neighbor Group women together during the first half of the month and then follow up that meeting with a home visit to each household.  Develop a curriculum and teach CGV how to conduct group meetings.  Select a name for the group of women that the CGV are responsible for, such as Neighbor Group.  Instead of including households with children under five, only include households with children under two, as there are fewer children under two per household. Future Care Group approach implementers  Husbands of CGV (and to a lesser extent CHW) complain that the job requires too much time away from home.  Disgruntled husbands push their wives to quit their CGV post, requiring them to be replaced and retrained.  When potential CGV are identified by their neighbors, the CHW should meet with the husband and discuss the requirements of the job and the approximate number of hours it will require each week.  Recruiters should make every effort to gain the concurrence of the husband before the woman can become a CGV.  Opportunities to thank the husband publically for his support of his wife (colline meetings) should be sought. Future Care Group approach implementers  In the current approach, CGV are taught a lesson every two weeks and it takes about 1.5 years to be fully trained.  Concern and MOH are concerned that CGV are not able to give life-saving advice more quickly.  Some children who may require medical attention may not be receiving it in time  Start the CGV training with the lesson on dangers signs, early care seeking and introduce the referral cards within the first month of the training. Anyone implementing the Care Group approach 21 December 2013 Finding Conclusion Recommendation Who Is Responsible  The original project plan called for the TPS or Titulaire to supervise the CHW. Neither had the necessary time to perform this task well, especially when the CCM program began. Another nurse in the health center was named the Community Health Nurse and assigned responsibility for CHW supervision and support.  Appropriate steps were taken to ensure adequate supervision of the CHWs.  With the ‘renaming’ of this nurse, a focus was placed on the growing importance of community health activities  The Community Health Department should consider assigning one nurse in every health center responsible for community health activities. To clarify this nurse’s unique role, s/he should be called the Community Health Nurse. MOH / National Community Health Department  At the outset of the Care Group initiative, supervision of CGV by CHW was not accompanied by any tool or means to control for quality.  The QIVC was only introduced in 2012 as part of the OR Process Evaluation and was not used systematically.  The quality of CGV home visits was not being monitored carefully and opportunities to improve the home visits are being missed.  Develop a lesson on how to conduct an effective meeting and home visits which includes discussing barriers to behavior change and making a commitment. Consider following the FH lesson format found on the Care Group website.  From the outset, teach CHW how to supervise the quality of CGV work – not just the reporting aspects but the behavior change activities as well.  Teach CHW how to use the QIVC to monitor the quality of meetings and home visits.  Standardize the frequency of supervisory visits.  Teach CHW how to provide Supportive Supervision.  Community Health Nurses should review the supervision tools and QIVC on a monthly basis. Future Care Group approach implementers  The training modules and visual aids used in the Care Group approach were developed quickly and were not reviewed by the HQ Health Advisor and the visual aids were not tested.  Some of the training modules and visual aids are less than optimal. Messages are not completely accurate.  These modules and visual aids are not the best examples to be shown to the MOH for scale up purposes.  Plan enough time for training materials development and visual aids testing.  Reference the Technical Reference Materials for message accuracy and have them reviewed by an ‘outside’ technical person such as the HQ Health Advisor. Future Care Group implementers 22 December 2013 Finding Conclusion Recommendation Who Is Responsible  In January 2012 the MOH presented the Manual on Community Health Procedures which stipulated that CHW would no longer work on vertical programs (HIV, TB) but would provide a range of services.  A substantial number of CHW trained by the MCSP express concern regarding the difficulty of fulfilling their responsibilities as CHW while at the same time meeting the needs of their family.  The MOH is planning to introduce iCCM which will further increase the duties of the CHW and the number of hours they are expected to work.  No one has estimated the number of hours each CHW task (Care Groups, CCM, etc.) should require each month of the CHW.  This approach runs the risk of unfairly over burdening the CHW. CHW may stop performing their tasks well. The drop-out rate may increase.  The supporting NGO and the MOH/Community Health Department should keep track of all the tasks assigned to the CHW (iCCM, Care Group), so they are aware of the level of work being required of them.  The time required for each task assigned to the CHW (CCM, Care Group etc.) should be estimated and incorporated into the job description.  When CHW are recruited the number of hours they are expected to work should be clearly explained to the candidate AND to his/her spouse. Female CHW should get concurrence from her spouse to accept the position.  The supervision tools for each task should be combined into one tool so that each time the CHW is supervised all of her/his tasks will be reviewed. The supporting NGO and the MOH / Community Health Department  The MCSP evaluated the performance of the Integrated Care Group approach using five performance indicators and 40 outcome indicators. Behavior change and performance results in the Integrated area were similar to those in the traditional Care Group area.  The OR study conducted by Concern in Mabayi revealed that the Integrated Care Group approach is as effective as the Traditional Care Group approach.  Through the Care Group approach a small number of CHW (152) are able to reach 30,000 household twice per month by working through CGV.  The Care Group model is an exceptionally effective and efficient behavior change approach that can be implemented by the MOH (District and health center levels) and still produce the same results (behavior change).  Answers to some critical questions regarding Integrated Care Group implementation need to be found.  Concern in consultation with the District and the Department of Community Health should write a protocol for the implementation of the Integrated Care Group approach. This protocol should be based on the best practices of the MCSP and provide responses to the following questions: -Who will initiate the approach -What should the data collection tools look like (what data should be collected) -How will quality control and formative supervision be provided -How will the behavior Concern Worldwide with MOH 23 December 2013 Finding Conclusion Recommendation Who Is Responsible  There are still a number of questions that remain to be answered with regard to how the Ministry will take full responsibility for all the aspects of the Care Groups. change promotion materials be reproduced -How will newly assigned MOH staff be integrated into the Care Group program -What are all the costs involved in all these and how will these be covered  The MOH / Department of Community Health should consider scaling up the Integrated Care Group approach following the above mentioned protocol. Objective 2: Improved access to quality of basic child health care services with a balance of provision at the health center and community levels.  USAID expects CSHGP grantees to participant in national level policy dialogue meetings;  Concern expected the in￾country Health Advisor and Project Manager to participate in national level policy dialogue meetings to represent the interests of the organization.  No training was provided to staff in advocacy.  While the current Project Manager performed well without training, the effectiveness of staff engagement in national policy dialogue meetings can vary significantly depending on personal skills and prior training.  USAID should consider developing a course on advocacy which it offers to grantees and other NGOs  Concern should arrange for staff to attend advocacy training to help them be as effective as possible when engaging in national policy dialogue meetings. USAID Concern Worldwide  A substantial number of key informants (CHW, Community Health Nurse, District staff) interviewed for the final evaluation, mentioned stock out of RDT as a major barrier to the effective functioning of the CCM program.  CHW also mentioned difficulties in getting adequate supplies of ACT  The HFA showed that while all health facilities have ACT in stock, usually it is just a few doses.  Stock management challenges at the national, district and health center levels prevent CHW from providing optimal malaria treatment services at the community level.  The HFA tool allows health centers with just one dose of a drug to be considered ‘functional’.  The MOH / Department of Malaria Management should take the necessary steps to ensure an adequate supply of RDT and medicines so that the CHW trained in malaria management can continue to work effectively.  Policy makers providing guidance to MOH decision￾makers regarding procurement estimates should take into consideration the strategies being used such as CCM to generate demand for and increase access to treatment.  The HFA tool should be revised to increase the MOH / Malaria Management Department Burundian Policy Makers USAID/MCHIP 24 December 2013 Finding Conclusion Recommendation Who Is Responsible threshold above which a health center is considered functional.  The CCM protocol requires all CHW who have treated a child with malaria to conduct a follow up visit after two days. The protocol does not instruct the CHW regarding what should happen during this follow up visit.  The CHW may miss ‘teachable moments’ with regard to malaria prevention and sick child feeding (malnutrition prevention).  The National Malaria Management Department should amend the protocol to include more detailed instructions about what should happen during the sick child follow up visit.  During the visit, the CHW should be instructed to give strong advice about how to feed a sick/recovering child in order to prevent malnutrition.  The tool should also provide a means to track completion of the follow￾up visit. MOH / Malaria Management Department  CHW received three weeks of training in CCM. Two weeks of this was spent in the health center focusing primarily on diagnosis and treatment skills. Less training time was spent on report completion skills. The CCM database showed poor performance immediately following the training. In￾depth analysis attributed this to poor report completion skills. Project staff provided additional technical assistance to rectify this issue.  While it was very beneficial for the trainees to spend time in the health center honing their diagnostic skills, the training plan did not include sufficient time and methods to master the report completion skills; as a result, many report completion mistakes were made during the first quarter following training.  The National Malaria Management Department should review the CCM training curriculum and strengthen the report completion component.  This component should include several practical exercises requiring CHW trainees to complete forms themselves when given various case studies.  “Find the error” exercises should also be included.  The health center practicum should also include practice on report completion. MOH / Malaria Management Department Objective 3: Strengthened Community Leadership in Health  Currently one criterion for COSA members is to be literate.  Literacy levels among women are low in Burundi.  The Community Health Procedures Manual recommends that half the COSA members be women.  The literacy requirement makes it difficult to find enough women to serve on the COSA.  Drop the criteria that all COSA members be literate. MOH / National Community Health Dept. 25 December 2013 Finding Conclusion Recommendation Who Is Responsible  In 2012, the MOH redefined and strengthened the role of the COSA (in the Community Health Procedures Manual) to focus more on health center support. The district supervision tool does not currently include any way to monitor COSA performance.  It is difficult to assess the effectiveness of the current COSA protocol.  The district is not able to assess COSA performance.  MOH/ Dept. of Community Health should add COSA performance questions to the monthly health center supervision form. These questions should be introduced at the district level and become part of the monthly supervision report.  Performance questions that should be considered might include: - Did the COSA undertake any activities this month in support of the health center? MOH / National Community Health Dept.  Currently COSA members only receive a brief orientation regarding their roles and responsibilities as COSA/COGES members. In 2012, the project undertook a COSA assessment which revealed that COSA members were not clear on their roles and ill-equipped to play an role in health center management.  The COSA/COGES trained by the project were playing a strong role in the management of the health centers.  COSA members need additional training to fulfill their intended role.  MOH/Dept. of Community Health should design a training module (referencing the course used by Concern) to formally train COSA and COGES members.  MOH/Dept. of Community Health should go beyond the existing COSA orientation and offer formal training to all COSA/COGES using the MOH-approve curriculum. MOH / National Community Health Dept.  The protocol for Community Health Procedures suggests several channels of communication (colline meeting, church services, door to door) that the COSA could use to transmit important information from the health center to the community.  The protocol does not call for the COSA members to use the CHW to help disseminate messages.  Using the present channels of communication, COSA members have a difficult time disseminating critical messages to all of the households they represent.  Create channels of communication between COSA members CHW and, where they exist, CGV, so that COSA members can rapidly disseminate critical information from the health center through them to all households.  Amend the Community Heath Procedure to include mention of CHW and CGV as COSA collaborators. District Health Officers and Titulaires MOH / Nat. Community Health Dept. 26 December 2013 Finding Conclusion Recommendation Who Is Responsible Operations Research  It took 1.5 years for the OR protocol to be approved.  During the first year of the project, Concern did not have a Health Advisor on board (either at HQ or in-country) who was familiar with OR.  The key strategy of the project, Care Groups, could not be initiated until the OR protocol was approved.  Concern Worldwide intends to continue to carry out OR.  The lack of familiarity with how to develop an OR protocol among HQ and in-country staff (and within USAID) caused serious delays to project implementation.  Concern Worldwide should consider hiring an OR specialist to provide the necessary expertise in planning and implementing Operations Research studies. Or  Concern Worldwide could partner with a university to secure the technical assistance they require to work on OR studies. Concern Worldwide decision-makers  USAID requested Concern to select a different OR study topic after the proposal had been funded.  It took two years to decide which alternative topic to study and to get the OR Protocol and IRB approval.  Unlike some OR topics, in this case the OR was the key intervention of the project. These delays impeded CG implementation and seriously jeopardized achievement of the outcomes related to Objective 1.  USAID should select grantees based on the study topic proposed by the NGO (as well as the other merits of the proposal) and not ask them to change the topic after the award is made.  USAID should ensure that all OR protocols are approved within the first year of the project to avoid delays in project implementation. USAID/CSHGP