EVALUATION USAID/Bangladesh: Final Evaluation of the MaMoni Integrated Safe Motherhood, Newborn Care and Family Planning Project JULY 2013 This publication was produced at the request of the United States Agency for International Development Mission in Bangladesh. It was prepared independently by Susan Rae Ross, Dr. Jahir Uddin Ahmed, Iain McLellan, and Wilda Campbell through the Global Health Technical Assistance Bridge III Project. Cover Photo: A grandmother helps attend to her granddaughter's infant in Bangladesh. This adolescent mother's infant was the first child born in the MINIMAT study cohort in Matlab, Bangladesh. Four thousand mothers were followed through their pregnancies with nutritional interventions to prevent low birth weight.© 2003 Ansem Ansari, Courtesy of Photoshare USAID/Bangladesh: Final Evaluation of the MaMoni Integrated Safe Motherhood, Newborn Care and Family Planning Project JULY 2013 Global Health Technical Assistance Bridge III Project (GH Tech) USAID Contract No. AID-OAA-C-13-00032 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 13-03-002) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge III Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Engility-IRG to the United States Agency for International Development under USAID Contract No. AID-OAA-C-13-00032. USAID/BANGLADESH: MAMONI FINAL EVALUATION i ACKNOWLEDGMENTS The team would like to acknowledge and thank the MaMoni managers, particularly Dr. Sabbir Ahmed and Mr. Imteaz Mannan, as well as the field staff of Shimatick and FIVBD for facilitating our trip during a very challenging time in the country. ii USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION iii CONTENTS ACRONYMS ........................................................................................................................................ v EXECUTIVE SUMMARY ................................................................................................................ vii Project Background and Approach ....................................................................................... vii Overall Assessment ................................................................................................................. viii Results ...........................................................................................................................................ix Findings, Lessons Learned, and Challenges ............................................................................ x Recommendations ..................................................................................................................... xi I. INTRODUCTION ....................................................................................................................... 1 II. BACKGROUND .......................................................................................................................... 3 Status of Maternal and Newborn Care in Bangladesh ......................................................... 3 Health, Maternal, and Newborn Policies ............................................................................... 3 MaMoni Project Description ..................................................................................................... 4 III. PURPOSE OF EVALUATION ................................................................................................... 9 IV. EVALUATION METHODOLOGY AND LIMITATIONS ................................................11 Methodology ..............................................................................................................................11 Limitations of Evaluation .........................................................................................................11 V. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ......................................13 Achievements to Date .............................................................................................................13 Effectiveness of Project Strategies ........................................................................................20 Leveraged Activities .................................................................................................................30 Project Management ................................................................................................................30 VI. CHALLENGES AND LESSONS LEARNED .......................................................................33 Challenges ....................................................................................................................................33 Lessons Learned ........................................................................................................................33 VII. RECOMMENDATIONS ...........................................................................................................35 Recommendations for Interventions to be Scaled Up .....................................................35 Recommendations for Interventions to be Strengthened ...............................................35 Recommendations for Future Programming ......................................................................36 Power Point Presentations .....................................................................................................56 Questions for Service Providers/Facilities ............................................................................57 Questions for Women ............................................................................................................57 iv USAID/BANGLADESH: MAMONI FINAL EVALUATION Questions for Private CSBAs ..................................................................................................58 Questions for Policymakers ...................................................................................................58 Questions for Community Groups ........................................................................................59 Questions for UP and Local Community ............................................................................60 Questions for CC and CC Support Group ........................................................................60 Questions for Depot Holder .................................................................................................61 Questions for Micro-planning .................................................................................................61 ANNEXES ANNEX I: SCOPE OF WORK......................................................................................................39 ANNEX II: EVALUATION DESIGN MATRIX ........................................................................53 ANNEX III: DOCUMENTS REVIEWED ....................................................................................55 ANNEX IV: STAKEHOLDER QUESTIONS .............................................................................57 ANNEX V: CALENDAR AND PEOPLE INTERVIEWED .......................................................63 ANNEX VI: FY 2012 OPERATIONAL PLAN AND ACHIEVEMENTS ..............................69 ANNEX VII: TRAINING OVERVIEW ........................................................................................71 ANNEX VIII: STRATEGIES FOR HARD-TO-REACH AREAS .............................................73 ANNEX IX: PHASE-OUT OF MAMONI ....................................................................................75 ANNEX X: STAFF BIOGRAPHIES ............................................................................................77 ANNEX XI. DISCLOSURE STATEMENTS ...............................................................................79 FIGURES FIGURE 1: TRENDS IN ANC, HABIGANJ .................................................................................16 FIGURE 2: MATERNAL COMPLICATIONS REFERRED TO THE HABIGANJ DISTRICT HOSPITAL .....................................................................................................................19 TABLES TABLE 1: DIFFERENCES BETWEEN SYLHET AND HABIGANJ INTERVENTIONS ....... 5 TABLE 2: KNOWLEDGE OF DANGER SIGNS AMONG WOMEN BY TIME PERIOD, HABIGANJ .......................................................................................................................14 TABLE 3: MAMONI KEY ACHIEVEMENTS BY DISTRICT ...................................................15 TABLE 4: MAMONI KEY ACHIEVEMENTS BY DISTRICT, LOWEST QUINTILE ..........16 TABLE 5: DELIVERIES CONDUCTED IN UPGRADED FEMALE WELFARE CENTERS .......................................................................................................................17 TABLE A-1. EXPECTED LIFE-OF -PROJECT RESULTS ..........................................................41 USAID/BANGLADESH: MAMONI FINAL EVALUATION v ACRONYMS ANC Antenatal care AOTR Agreement officer technical representative BCC Behavior change communication CAG Community action group CC Community clinic CHW Community health worker CPR Contraceptive prevalence rate CSBA Community-based skilled birth attendant CV Community volunteer DGFP Directorate General of Family Planning DGHS Directorate General of Health Services DH District hospital EmONC Emergency obstetric and newborn care ETAT Emergency triage, assessment, and treatment FANTA Food and Nutrition Technical Assistance project FP Family planning FIVBD Friends in Village Development Bangladesh FWA Family welfare assistant FWC Family welfare center FWV Female welfare visitor HA Health assistant HFA Health facility assessment HMIS Health management information system HPNSDP Health Population and Nutrition Sector Development Plan ICDDR,B International Centre for Diarrhoeal Disease Research, Bangladesh IR Intermediate result KOICA Korean International Cooperation Agency LAM Lactational amenorrhea method LAPM Long-acting and permanent methods LB Live births MCHIP Maternal and Child Health Integrated Program MCWC Maternal and child welfare center vi USAID/BANGLADESH: MAMONI FINAL EVALUATION MMR Maternal mortality ratio MNH Maternal and newborn health MO Medical officer MOH&FW Ministry of Health and Family Welfare NGO Non-governmental organization NMR Neonatal mortality rate OBGSB Obstetric and Gynecological Society of Bangladesh PNC Postnatal care QOC Quality of care SACMO Sub-assistant community medical officer SBA Skilled birth attendant SBM-R Standards-based management and recognition SOPs Standard operating procedures SWAp Sector-wide approach TBA Traditional birth attendant TFR Total fertility rate UHC Upazila health complex UP Union parishad USAID United States Agency for International Development UN United Nations WRA Women of reproductive age USAID/BANGLADESH: MAMONI FINAL EVALUATION vii EXECUTIVE SUMMARY This report summarizes the performance evaluation of the MaMoni project implemented in Bangladesh by Save the Children and two local non-governmental organizations (NGOs), Shimantik and Friends In Village Development Bangladesh (FIVDB). The evaluation was conducted from May 1–26, 2013, by a four-member external team. The team members were Susan Rae Ross, Wilda Campbell, Iain McLellan and Dr. Jahir Uddin Ahmed. The evaluation had three objectives: 1) assess the effectiveness of the MaMoni project in achieving planned objectives, 2) identify constraints, and 3) provide recommendations for future planning. The team reviewed key documents, conducted field visits to Sylhet and Habiganj, interviewed key stakeholders, and reviewed the two population-based surveys and project monitoring data. PROJECT BACKGROUND AND APPROACH MaMoni is an associate award supported by USAID/Bangladesh (Aug. 2009–Jan. 2014) through the Maternal and Child Health Integrated Program (MCHIP). MaMoni aims to increase the use of high-impact maternal and newborn health (MNH) behaviors, including family planning (FP), and to strengthen the Ministry of Health and Family Welfare (MOH&FW) systems to provide quality MNH/FP services. The project works in Sylhet and Habiganj, two poorly performing districts in Bangladesh whose combined population is about 3.5 million people. MaMoni began working in Sylhet in December 2009, building on the work of the ACCESS project (2006–2009) and Projahmo (2001–2006). ACCESS focused on improving demand for MNH behaviors and services in 7 of the 12 upazilas of Sylhet, largely through NGO-supported community health workers (CHWs) who provided home-based counseling and services and community action groups (CAGs) with linkages to MOH&FW services. ACCESS results showed a rise in demand but limited change in service use, so strengthening systems to improve access to MNH/FP services is a key MaMoni strategy. MaMoni added interventions to increase CAG knowledge about FP and hand washing. The CHWs were also trained on these messages, as well as in counseling and resupplying pills and condoms. As the project progressed, USAID decided that the health systems strengthening activities would have to go beyond traditional support to address critical issues, such as high vacancy rates of health workers. Since many of the efforts in Sylhet had been in place since 2006 and were largely led by the NGOs, USAID began to phase out these activities and focus the health systems strengthening efforts in Habiganj. Activities in Habiganj began in June 2010 with the same two local NGOs utilizing a different approach than previously. From the outset, MaMoni focused on an integrated approach that would address critical health systems issues, mobilize communities, and engage the Union Parishad (UP) in a coordinated way to improve MNH/FP outcomes. The project conducted an extensive mapping exercise and a health facility assessment (HFA) to understand all the issues. While the project design outlined some areas that MaMoni would address, many of the interventions undertaken, such as facility renovation or hiring temporary staff, were not anticipated at the project’s outset. The key elements of the approach in Habiganj are presented below. viii USAID/BANGLADESH: MAMONI FINAL EVALUATION  Community mobilization: Community volunteers (CVs) were trained to form and lead CAGs that undertake action planning to address MNH/FP issues, support referrals, and collect community data for the micro-planning meetings.  Behavior change communication (BCC): Family welfare assistants (FWAs) and female welfare visitors (FWVs) were trained to provide counseling to women, supported with BCC materials. The CAGs provided some information at their monthly meetings, and there was some use of mass and media channels (e.g., videos, phones).  Community services: MaMoni supported FWAs, through training and temporarily filling vacant FWA positions to provide counseling and home-based FP and postnatal care (PNC) services. To improve deliveries with skilled birth attendants (SBAs), the project supported private community-based skilled birth attendants (CSBAs). They also provided FP training to female health assistants (HAs) and directed traditional birth attendants (TBAs) to avoid harmful practices and refer women to health facilities.  Specialized strategies: MaMoni developed customized strategies to reach special populations in the urban slums and tea estates and three geographically hard-to-reach upazilas. This included testing new service delivery models to provide 24/7 delivery services at the union level.  Health systems strengthening: MaMoni prioritized seven key systems issues to improve access to quality MNH/FP services: 1) filling (temporarily) critical gaps among key health provider positions (e.g., FWVs, nurses); 2) renovating facilities and staff quarters; 3) providing essential equipment, medicines, and supplies; 4) creating a referral system that included a team at the district hospital (DH) to facilitate admissions; 5) facilitating supportive supervision; 6) strengthening quality of care (QOC); and 7) improving data collection and accuracy through micro-planning meetings.  Stakeholder coordination mechanisms: MaMoni focused on creating relationships among various stakeholders and mobilizing their resources to address MNH/FP outcomes. The various stakeholders included the Directorate General of Health Services (DGHS); Directorate General of Family Planning (DGFP); Ministry of Local Government, Rural Development and Cooperatives; UP members/health committees, CAGs, and communities.  Innovation: In collaboration with MOH&FW and the International Centre for Diarrheal Disease Research, Bangladesh (ICDDR,B) MaMoni is undertaking four operations research projects on the following: treatment of pre-eclampsia/eclampsia with a loading dose of magnesium sulfate provided by FWVs and government CSBAs at family welfare centers (FWCs), treatment of sick newborns at the upazila level (Sylhet), consolidation of MOH&FW facility registers, and use of the referral system. OVERALL ASSESSMENT MaMoni has been very successful in engaging and gaining commitment from many stakeholders, including the DGHS and DGFP at all levels, district health managers and providers, UP members/health committees, and communities. While final conclusions cannot be drawn until the final survey is conducted in November 2013, project data indicate that MaMoni has achieved significant increases in deliveries with SBAs, referrals for maternal complications, and PNC, as well as modest improvements in FP. These gains were realized through active participation of USAID/BANGLADESH: MAMONI FINAL EVALUATION ix CAGs and Ups; testing new models in hard-to-reach areas (e.g., upgrading FWCs); and an integrated approach to health systems strengthening that included hiring health providers, renovating facilities, ensuring adequate medicines and supplies, and strengthening supervision. The micro-planning meetings improved data quality and local-level planning among the DGHS and DGFP workers and the community. There are several promising practices that have showed modest gains, such as support for private CSBAs and joint supervision, but more information is needed to fully understand the effectiveness of these efforts. Greater attention must be given to 1) improving BCC approaches to enhance knowledge of danger signs and birth planning; 2) developing strategies to provide MNH/FP services to women close to their home but not necessarily in their homes; and 3) strengthening QOC, particularly for maternal and newborn complications. MaMoni embraces a spirit of innovation and has become an important learning lab for key MNH/FP interventions. They tested new approaches in hard-to-reach areas. These strategies included upgrading FWCs to provide 24/7 delivery services, supporting private CSBAs, and developing a strong referral system. They also operationalized results from other research projects that included distribution of misoprostol for women delivering at home and training all the health providers in the district on managing newborn asphyxia. As previously mentioned, they are also collaborating with ICDDR,B on four research projects. Changing health systems is a long-term process. In only three years, MaMoni has made great strides in improving the functionality of the existing health systems, as evidenced by increases in service utilization and support by MOH&FW managers and policymakers. This approach has been very flexible, adapting to the local needs and building on the lessons from Sylhet as well as other MNH programs in Bangladesh. Based on this success, the team believes that many of the interventions are ready to be scaled up in other districts, a key step in USAID/Bangladesh’s future strategy. RESULTS A baseline survey was conducted for both districts in October 2010. A second survey was conducted in May 2012. This was considered a final survey in Sylhet, since activities were being phased down, and a midline survey in Habiganj, as the activities were being phased up. A final survey will be conducted in Habiganj in November 2013. There was very little change between the Habiganj surveys, which is not surprising because many of the key interventions (e.g., facility renovation) had not been completed when the second survey was conducted. The team therefore focused on analysis of the project data to assess if changes had occurred since May 2012. The key aims of the project were to increase use of antenatal care (ANC), deliveries with a SBA, referrals for complications, PNC, and FP—including long-acting and permanent methods (LAPM). The following discusses results to date.  In Habiganj, 37.5% of pregnant women received ANC counseling; of these, only 6.9% received information on danger signs. As a result, knowledge of danger signs was mixed and knowledge of birth planning was low.  Satellite clinics providing ANC increased from 484 a month in 2010 to 537 a month in 2013. Coverage of three or more ANC visits has remained constant at 40% of projections since September 2012. x USAID/BANGLADESH: MAMONI FINAL EVALUATION  MaMoni has worked to improve deliveries with SBAs using several approaches, including upgrading FWCs to provide services 24/7 and supporting private CSBAs. Deliveries in the upgraded FWCs have significantly increased, as evidenced by more than 50% of all the deliveries in the upazila from September 2012–March 2013 taking place at two facilities (Shibpasha and Murakari). In addition, 14 private CSBAs conducted 193 deliveries from May 2012–March 2013, which comprised 38% of all the deliveries in the upazila.  MaMoni developed an extensive system for referral from the village to the DH. As a result, there has been a sixfold increase in maternal referrals to the DH, from 111 (April– December 2011) to 605 (January–December 2012). There were 65 newborn referrals to the DH (February 2012–March 2013).  There has been a modest increase in use of injectables, from 16,000 (January–March 2010) to 20,000 (January–March 2013). LAPM uptake has been sporadic, due to a reliance on camps, but the average monthly case load rose from 553 in 2010 to 750 in 2012. MaMoni was supposed to track lactational amenorrhea method (LAM) use and transition to a modern FP method. This has been difficult because LAM is not an MOH&FW priority and their registers do not capture any postpartum FP methods. These data, combined with key stakeholder discussions, indicate that MaMoni has been very successful, resulting in significant increases in the use of key MNH services and exceeding many projected targets. FINDINGS, LESSONS LEARNED, AND CHALLENGES Findings and Lessons Learned The team tried to identify the key elements that made MaMoni’s approach successful in order to inform replication in other areas. The team believes that the following efforts were essential to MaMoni’s success.  Adaptable approach: Since Habiganj was a new district, this required extensive mapping and assessments to understand the availability of existing resources and identify potential gaps. In addition, many of these interventions had not been tried, requiring MaMoni to be flexible and adapt to challenges as they arose.  Stakeholder coordination and commitment: Health systems strengthening required MaMoni to play more of a facilitator role rather than a direct implementer. For the interventions to be successful, it was essential that there was ownership by the district and upazila managers. Mechanisms needed to be reactivated or created (e.g., collaboration between DGHS and DGFP) to foster coordination within the health system and across sectors.  Integrated approach to health systems strengthening: Effectively addressing the key health systems issues that affected MNH/FP services required an integrated approach that combined increases in staffing level, renovation of health facilities and staff quarters, and sufficient equipment and supplies.  Accurate data for decision-making: It is impossible to effectively manage services without accurate data. The micro-planning meetings have provided a useful mechanism to bring the health and FP workers together with the community to accurately capture USAID/BANGLADESH: MAMONI FINAL EVALUATION xi community information and ensure that consistent data are being reported. This enabled FWAs and CVs to develop more targeted plans to follow up with women and children who needed services.  Community involvement: The backing of the community and UP has been an essential component to solving problems, gaining support for local facilities, and encouraging women to access MNH/FP services. Challenges  BCC: MaMoni relied heavily on counseling by the FWAs and FWVs to disseminate the messages to women. However, these workers did not have sufficient time to provide the large volume of messages (60) to women. As a result, knowledge of danger signs was mixed and knowledge of birth planning was low. The team feels that the CVs could have been used more to provide messages. In addition, a more balanced mix of communication channels with reinforcing messages would have provided more opportunities to expose audiences to the messages.  Home-based counseling and services: Working with the FWAs to expand counseling and home-based services has not yet proven to be effective. Midline data indicate that few women received information or services from FWAs. Project data showed improvements in PNC in the first three days after birth. The final survey data may show improvements in this important outreach effort and should be factored into future strategies for improving the FWA outreach services.  QOC: There is a concern that the increased demand and high vacancies will make it difficult to maintain good quality services, even with the temporary MaMoni staff. The standards-based management and recognition approach (SBM-R), which includes indicators on process observation to ensure service standards, facility record review, and client exit interviews, was introduced in 2011. However, because of local stakeholders’ concerns, SBM￾R has not been fully implemented. MaMoni is working with stakeholders to streamline the tools and verification criteria.  High vacancy rates: Vacancy among key service providers and managers continues to be a major challenge. MaMoni has been successful in temporarily filling some FWV and nurse positions but has not been able to hire any medical officers (MOs), despite many efforts.  Sustainability of efforts by MOH&FW: One of the key challenges is how much of MaMoni’s efforts can be sustained by the MOH&FW as the project transitions and phases out. Of particular concern is whether the MOH&FW will be able to 1) maintain the renovated/constructed facilities, 2) hire/retain staff in hard-to-reach areas, 3) maintain ambulances, 4) conduct supervision, 5) ensure QOC, and 6) continue coordination between health and FP directors when staff changes. RECOMMENDATIONS Recommendations for Interventions to be Scaled Up The team believes that there is enough evidence to indicate that the following activities are working well and can be replicated in other districts with appropriate adaption. Thus, in new districts it is recommended that USAID undertake the following activities: xii USAID/BANGLADESH: MAMONI FINAL EVALUATION  Conduct a comprehensive assessment, including an HFA, human resources inventory, and mapping and identification of any areas/populations needing customized strategies.  Develop collaboration mechanisms/relationships among key stakeholders.  Build on existing MOH&FW system and structures (e.g., offices in district facilities).  Create CVs/CAGs to support community mobilization efforts with strong linkages with UP, FWAs, and HAs.  Liaise with UP members and reactivate health committees to support MNH/FP efforts.  Facilitate micro-planning meetings with follow-up meetings at the union, upazila, and district levels.  Support an integrated approach to health systems strengthening based on the assessment process, which may include some combination of the following efforts: – Critical gap management (e.g., hiring FWVs, nurses and MOs); – Provision of key supplies, equipment, and medicines; – Renovation/construction of selected facilities including staff quarters, waste management, and water sources; – Competency-based skills training supported by supervision and on-the-job training to maintain skills; – Referral system that includes a team at the referral facility to ensure prompt treatment of complications; – Quality improvement mechanisms that regularly review service performance; – Data collection that promotes accuracy/reliability and is used to improve performance; – Continued advocacy for policy and operation changes (e.g., implants by FWVs, epidural block by nurses). Recommendations for Interventions to be Strengthened  Develop comprehensive BCC strategy that 1) prioritizes key messages and disseminates them in a manner that is easy for people to absorb, 2) utilizes a mix of communication channels based on cost-effectiveness, 3) explores greater utilization of CV to undertake systematic BCC dissemination of key messages supported with BCC materials, and 4) explores the use of phones as a communication channel (e.g., dissemination by CSBAs, CVs) supported by free airtime.  Develop a transition strategy for the CAGs/CVs that outlines the level of effort needed to start the CAGs and the declining level of effort as they become independent.  Conduct an assessment of the current private CSBAs to understand 1) the conditions that create the best environment for them to be effective (e.g., links with UP); 2) the situations where they are most useful (e.g., hard-to-reach areas), 3) their performance and skills, 4) the challenges they have experienced, and 5) additional support that may be USAID/BANGLADESH: MAMONI FINAL EVALUATION xiii required (e.g., other services, business training) for them to be successful and have an acceptable income.  Determine, based on endline data, whether to reduce reliance on FWAs. That could include not hiring additional temporary FWAs or reducing their training. MaMoni should look at other approaches (e.g., CVs, CSBAs) for BCC efforts. In addition, the project should explore other channels (e.g., community clinics [CCs], pharmacies, depot holders) to improve access to services and supplies that are close to women but not necessarily in their homes.  Explore options to expand support for FP, which may include assessing the effectiveness of depot holders or exploring the potential of providing short orientations for pharmacists and traditional doctors on FP counseling, side effects, and ensuring supplies are available at CCs. To address LAPM, MaMoni should explore other ways to increase LAPM providers (e.g., Marie Stopes, private providers) and continue to link with and support the current efforts of Mayer Hashi and Smiling Sun.  Strengthen QOC by implementing SBM-R. If consensus on the tools and verification criteria cannot be reached, then MaMoni should explore other quality assurance methodologies. In addition, MaMoni should conduct further review of the referral data from a QOC perspective. Recommendation for Future Programming These recommendations provide some overall guidance to UASID/Bangladesh on replicating the MaMoni model in new districts.  Articulate overall health systems strengthening replication model: MaMoni has tested several approaches in Habiganj. The program has been implemented in an integrated manner, but many of the elements have not been fully articulated into an overall model. For example, it is clear that CAGs are a key component, but it is not clear what type of support they need for how long. This information will be very important when thinking about replicating the model versus implementing the interventions. Sylhet provides some insights, but it has only transitioned the community activities, which were much more mature than those inHabiganj. The team thinks that scaling up these efforts will require a one-year start-up period and at least two or three years of intensive intervention and support, followed by two or three years of facilitation, with significantly reduced level of effort and staffing, to support MOH&FW managers in fully adopting the changes. Efforts in Habiganj are still in the intensive intervention support phase. More thinking about how the facilitation phase will be undertaken and how this will be done in other districts is required.  Clearly articulated phase-out strategy: To ensure sustainability and gain commitments from stakeholders, efforts in new districts need to see that all stakeholders understand that this approach is time bound and that the level of effort (staffing) will be reduced with a clear exit strategy. Annual plans can be developed with key benchmarks that articulate how functions will be transitioned to local parties, but sustainability of these efforts should be a key focus of the project. xiv USAID/BANGLADESH: MAMONI FINAL EVALUATION  Incorporate QOC and health system indicators: This is a health systems strengthening project, but all the indicators focus on changes in beneficiary behaviors and service use. It would be useful to include changes in provider knowledge, attitudes, and practices, since providers are also key beneficiaries and considerable effort was made to improve them. QOC indicators also need to be added. USAID/BANGLADESH: MAMONI FINAL EVALUATION 1 I. INTRODUCTION This report summarizes the performance evaluation for the MaMoni project implemented in Bangladesh by Save the Children and two local non-governmental organizations (NGOs), Shimantik and Friends In Village Development Bangladesh (FIVDB). MaMoni is an associate award supported by the U.S. Agency for International Development (USAID) in Bangladesh (August 3, 2009–January 31, 2014) through the Maternal and Child Health Integrated Program (MCHIP). MaMoni aims to increase the use of high-impact maternal and newborn health/family planning (MNH/FP) behaviors and strengthen the Ministry of Health and Family Welfare’s (MOH&FW) systems. The project works in Sylhet and Habiganj, two low-performing districts in Bangladesh. A four-member external team that included Susan Rae Ross, Wilda Campbell, Iain McLellan, and Dr. Jahir Uddin Ahmed conducted the evaluation from May 1–26, 2013. The evaluation aimed to 1) assess the effectiveness of the MaMoni project in achieving planned objectives, 2) identify constraints, and 3) provide recommendations for future planning (Annex I: Scope of Work). The team reviewed key documents, conducted field visits to Sylhet and Habiganj, interviewed key stakeholders, and reviewed the two population-based surveys and project monitoring data. Key audiences for this report are USAID in Washington and Bangladesh, MCHIP partners, MaMoni partners, MOH&FW policymakers and district managers, and other development partners. 2 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 3 II. BACKGROUND STATUS OF MATERNAL AND NEWBORN CARE IN BANGLADESH The 2010 Bangladesh Maternal Mortality and Health Care Survey found a significant decline in the maternal mortality ratio (MMR) from 322/100,000 live births (LBs) in 2001 to 194/100,000 in 2010, a 40% decline in nine years.1 MMRs and rates of decline vary greatly throughout the country. For example, in Khulna the MMR fell from 464 to 65, while in Sylhet it dropped from 765 to 155 during the same time period.2 The two main causes of maternal death are postpartum hemorrhage (31%) and pre-eclampsia (20%). Together these account for half of all the maternal deaths.3 The 2011 Bangladesh Demographic and Health Survey found low rates of under-5 mortality (53/1,000 LBs) and infant mortality (43/1,000 LBs).4 Sylhet has the highest under-5 mortality at 83/1,000 LBs.5 The neonatal mortality rate (NMR) is 32/1,000 LBs in 2011, down from 37/1,000 LBs in 2007.6 Reducing the NMR is a key component to reducing childhood deaths, since it accounts for 74% of infant and 60% of under-5 deaths.7 Sylhet also has the highest NMR (45/1,000 LBs) in the country. The key causes of newborn deaths include infections (24%), birth asphyxia (21%), pneumonia (13%), and pre-term births (11%).8 In Bangladesh, the current total fertility rate (TFR) for women of reproductive age (WRA) is 2.3,9 and the contraceptive prevalence rate (CPR) among married WRA is 61%. Despite this progress, Sylhet lags behind the national levels with a TFR of 3.1 and CPR of 44.8%. Usage of long-acting and permanent (LAPM) methods has declined from 12% in 1991 to 5% in 2011, while injectables increased from 3% in 1991 to 11% in 2011.10 HEALTH, MATERNAL, AND NEWBORN POLICIES The Health, Population, and Nutrition Sector Development Plan (HPNSDP) follows the sector￾wide approach (SWAp) that was first introduced in 1998. The current HPNSDP (2011–2016) is the third SWAp and links to the government’s sixth five-year plan (2012–2016). The key components of the HPNSDP are improving health services and strengthening health systems. Improving health services aims to improve priority health services11 through the upazila health systems, including the community clinics (CCs). Every union facility will be strengthened to conduct normal deliveries and refer complicated cases in order to reduce maternal and 1 Bangladesh Maternal Mortality and Health Care Survey, 2010. 2 Ibid. 3 Ibid. 4 Bangladesh Demographic and Health Survey, 2011. 5 Ibid. 6 Ibid. 7 Ibid. 8 Ibid. 9 Ibid. 10 Ibid. 11 Priority health services include maternal, neonatal, child, reproductive and adolescent health; population/family planning services; nutrition/food safety; communicable/non-communicable diseases; climate change/health protection; disease surveillance; alternative medical care; and behavior change communication. 4 USAID/BANGLADESH: MAMONI FINAL EVALUATION newborn deaths. Hospital services will be improved by using clinical protocols, ensuring adequate supplies with modern materials and diagnostic facilities, making existing hospitals women friendly, improving emergency obstetric and newborn care (EmONC) services, establishing hospital accreditation, improving supervision of total quality management, strengthening referral systems, and improving effective hospital waste management. HPNSDP prioritizes hard-to-reach populations by motivating and counseling the service providers to give care to the marginalized and socially excluded population. Services will be provided in the difficult-to-reach areas through appropriate arrangements with NGOs. These partnerships overcome shortages of human resources on the basis of comparative advantage. The strengthening health systems component will address governance and mainstream gender into core programs. Other priority areas include: local-level planning, monitoring and evaluation, health sector financing, health information system, human resources for health, in-service training, nurse/midwifery services, quality assurance system, supply chain management, maintenance of physical facilities, intersectoral coordination, and financial management. Bangladesh developed a National Neonatal Health Strategy, including several related standard operating procedures (SOPs), in October 2009. This process was conducted jointly by the MOH&FW, development partners, and NGOs, including Save the Children and key professional organizations. In 2001, the MOH&FW developed a Maternal Health Strategy. In September 2012, the MOH&FW requested support from the development partners—including USAID through MCHIP, UNICEF, and JICA—to help update the Maternal Health Strategy and its relevant SOPs. As a result, a national steering committee, headed by the secretary of MOH&FW, and a national technical committee, headed by the Director General of Health Services (DGHS) has been created. In addition, five technical subcommittees have been created to develop the SOPs, with representation from development partners, professional organizations, and NGOs. The aim was to have the first draft of the SOPs completed by December 2012, but this has been delayed. MAMONI PROJECT DESCRIPTION MCHIP is USAID/Washington’s flagship maternal, newborn, and child health program (2008– 2014). It aims to reduce maternal and child mortality; improve coverage and scale up high￾impact MNH/FP interventions; and develop and disseminate learning, tools, and approaches. In Bangladesh, MCHIP is supporting the MOH&FW to update the 2001 Maternal Health Strategy and its related SOPs. MCHIP is also collaborating with DNet to implement MAMA Bangladesh. MaMoni, an associate award totaling $13,493,991, is supported by USAID/Bangladesh (August 3, 2009–January 31, 2014) through MCHIP and implemented by Save the Children. MaMoni aims to increase the use of high-impact MNH/FP behaviors and strengthen MOH&FW systems to provide quality MNH/FP services. The project works in Sylhet and Habiganj, two low-performing districts in Bangladesh with a combined population of about 2.5 million people. Achievement of the project’s overall objectives depends on the successful attainment of the five intermediate results (IR): USAID/BANGLADESH: MAMONI FINAL EVALUATION 5 IR 1: Improved knowledge of MNH/FP behaviors, services, and service delivery points. IR 2: Increased availability and quality of community- and facility-based MNH/FP services. IR 3: Strengthened MOH&FW/NGO capacity/systems for effective delivery of MNH/FP services. IR 4: Increased community capacity, action, and demand for the practice of MNH/FP behaviors. IR 5: Increased key stakeholder leadership, commitment, and action for MNH/FP interventions. MaMoni is designed to directly support USAID/Bangladesh’s Development Objective 3: Improving People’s Health in Bangladesh, under the Investing in People Objective, Health Project Area of the U.S. Foreign Assistance Framework under USAID/Bangladesh’s Country Development Cooperation Strategy. MaMoni began working in Sylhet in December 2009 through Shimantik and FIVDB, two local NGOs, building on the work of the ACCESS project (2006–2009)12 and Projahnmo (2001– 2006). ACCESS focused on improving demand for MNH behaviors and services in 7 of the 12 upazilas of Sylhet. The NGOs trained and supported community health workers (CHWs) to provide home-based counseling and services, complementing the work of the MOH&FW family welfare assistants (FWAs). The NGOs also formed community action groups (CAGs) with linkages to MOH&FW services. ACCESS results showed a rise in demand but limited change in service use. Strengthening systems to improve access to MNH/FP services is a key MaMoni strategy. They also added interventions to increase knowledge about FP and handwashing for the CAGs. The CHWs were trained on these messages as well as counseling and resupplying pills and condoms. Activities in Habiganj began in June 2010 with a more integrated approach to addressing critical health systems issues, mobilizing communities, and engaging the UP in a coordinated way. Table 1 compares the approaches in Sylhet and Habiganj. In Habiganj, MaMoni conducted an extensive mapping exercise and an assessment of all the facilities to understand the key issues. While the project design outlined some areas that MaMoni would address, many of the interventions undertaken were not anticipated at the project’s outset. The MaMoni approach has been very flexible, adapting to the local needs and building on the lessons from Sylhet as well as from other MNH programs in Bangladesh. Table 1: Differences between Sylhet and Habiganj Interventions Effort Sylhet (7/12 Upazilas) Habiganj (8/8 Upazilas) System None Improved coordination between health and FP managers. None Improved supportive supervision. District None Improved coordination between the health and FP managers. Training None Trained district MOs and nurses on nutrition (IYCF program) and emergency triage assessment and treatment (ETAT). Referral None Referral team at the district hospital (DH) to facilitate incoming referrals. 12 Shimantik and FIVBD were also involved in implementation under ACCESS. 6 USAID/BANGLADESH: MAMONI FINAL EVALUATION Table 1: Differences between Sylhet and Habiganj Interventions Effort Sylhet (7/12 Upazilas) Habiganj (8/8 Upazilas) Drugs, supplies Misoprostol only Procured misoprostol and antenatal care urine and hemoglobin testing supplies for the entire district. Procured key equipment and drugs for emergency obstetric and newborn care (EmONC) for DH and maternal and child welfare center (MCWC). Renovate facilities None Strategically renovate key areas at DH and MCWC. Critical gap management None Hired temporary nurses for key vacancies at DH; hired temporary female welfare visitors (FWVs) at the MCWC. Tried to hire doctors without success. Upazila None Improved coordination between health and FP managers. Critical gap management None Hired temporary nurses for key vacancies at DH; hired temporary FWVs at the MCWC. Tried to hire doctors without success. Drugs None Procured key equipment and drugs for EmONC at upazila health complex (UHC). Renovate facilities None Strategically upgraded key facilities in hard-to-reach areas. Temporary nurses for UHC. Union Some linkages with UP and CAGs Supporting UP and Health Standing Committee to CAGs and health services. Referral system CAGs provide some basic information on where to go. Mapped referral system; clear pick-up points for different types of transportation. Three water ambulances and one road ambulance for emergency. Upgrading facility None Strategically upgraded seven FWCs to provide deliveries in hard-to-reach areas. Will construct seven new FWCs. CSBA None Trained private community-based skilled birth attendants (CSBAs) and linked them to the family welfare centers (FWCs). Critical gap management None Hired temporary FWVs to fill vacant MOH&FW positions in FWCs. Inherited CHWs from ACCESS. In 2011, aligned with vacant FWA positions. Hired temporary FWAs to fill key vacancies, with intent that MOH&FW would fill positions. CAGs Introduced/facilitated by NGO staff, later community resource people led CAGs. Separate male and female groups. Community Volunteers (CVs) identified by MaMoni but selected by UP. CVs led the CAGs from the outset. Mostly mixed (male + female) group. USAID/BANGLADESH: MAMONI FINAL EVALUATION 7 Key interventions in the Habiganj model are described below. Community mobilization: CVs were trained to form and lead the CAGs to undertake action planning to address MNH/FP issues, support referrals, and collect community data for the micro-planning meetings. BCC: FWAs and FWVs were trained to provide counseling to women, supported by BCC materials. The CAGs provided some information at their monthly meetings, and there was some use of mass media (e.g., videos, phones). Community services: MaMoni supported FWAs, through training and temporarily filling vacant positions, to provide counseling and home-based FP and postnatal care (PNC) services. To improve delivery, the project supported private community-based skilled birth attendants (CSBAs). They also trained female HAs on FP and reoriented traditional birth attendants (TBAs) to avoid harmful practices and refer women to health facilities. Health systems strengthening: MaMoni utilized an integrated approach that prioritized seven systems issues to improve access to quality MNH/FP services. The project  Developed customized strategies to address hard-to-reach areas and special populations, including testing new service delivery models;  Filled critical gaps among key health provider positions (critical gap management);  Renovated key facilities and staff quarters;  Provided selected essential equipment, medicines, and supplies;  Strengthened the referral system;  Facilitated supportive supervision;  Improved QOC; and  Strengthened the health management information system (HMIS), including micro-planning meetings. Stakeholder engagement and commitment: MaMoni focused on creating relationships among various stakeholders and mobilizing their resources to address MNH/FP outcomes. These groups of stakeholders included:  The DGHS and DGFP at all levels;  MOH&FW and the Ministry of Local Government, Rural Development, and Cooperatives;  UP members/health committees and the health providers of both the DGHS and DGFP; and  CAGs, UP members, and field workers from both the DGHS and DGFP. Innovation: MaMoni tested new models and operationalized research results from other projects. They are also collaborating with the International Centre for Diarrheal Disease Research, Bangladesh (ICDDR,B) to undertake four operations research projects. 8 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 9 III. PURPOSE OF EVALUATION A four-member team that included Susan Rae Ross, Wilda Campbell, Iain McLellan, and Dr. Jahir Uddin Ahmed conducted the external evaluation from May 1–26, 2013. The evaluation had three objectives: 1) assess the effectiveness of the MaMoni project in achieving planned objectives, 2) identify constraints to achieving results, and 3) provide recommendations for future planning. The following are the key evaluation questions that the team was asked to review.  What are the successes in community engagement?  Have MaMoni interventions been effective in enhancing knowledge on MHN/FP-related issues at the household level?  Are there signs of improvement in the proportion of household visits (for babies born at home) by health workers within three days of deliveries?  Are there signs of increased uptake of MNH/FP services at the facilities?  How effectively does MaMoni's micro-planning system link MOH workers and community volunteers to fill gaps in coverage and information?  Are MaMoni interventions strengthening capacity of MOH&FW, including MIS, QOC, and referral systems?  What successes has MaMoni achieved in engaging local government (UP) in promotion and oversight roles? This report aims to answer these specific evaluation questions and provide insights into the project’s progress and effectiveness. Annex II provides the evaluation framework and data sources used to answer these questions. 10 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 11 IV. EVALUATION METHODOLOGY AND LIMITATIONS METHODOLOGY The evaluation methodology consisted of 1) a review of key documents, 2) key informant interviews, 3) field visits, and 4) analysis of survey and project monitoring data. The team reviewed key USAID, MOH&FW, MCHIP, and MaMoni documents (Annex III: Documents Reviewed). The team developed questions to guide informant interviews with key stakeholder groups, including FP and health program managers, health providers (e.g., FWVs), field workers (e.g., FWAs), CAGs/CVs, UP members, women, and national- and district-level policymakers (Annex IV: List of Stakeholder Questions). Three of the team members met with the MCHIP team in Washington, D.C., on May 1, 2013, to get an overview of the project. On May 4, 2013, the team had a planning meeting in Dhaka with Dr. Umme Salam Jahan Meena from USAID and all the team members. On May 5, 2013, the team met with key members of the USAID/Bangladesh health team and program office staff and received a briefing from Save the Children (Annex V: Calendar and People Interviewed). From May 13–18, 2013, the team met with key stakeholders and MaMoni partners in Dhaka. The team conducted field visits in Habiganj from May 7- 9, 2013 and in Sylhet from May 10–-11, 2013. It should be noted that since Sylhet began to phase out its activities in 2011, MOH&FW health systems strengthening efforts were mainly focused on Habiganj. The team directed most of its attention toward these interventions. ICDDR,B conducted the MaMoni baseline survey (October 2010) in both districts. A second survey was conducted in May 2012. Since Sylhet was phasing down its activities, this study was considered a final survey in that district. In contrast, Habiganj was gearing up its efforts and many of the key interventions had not been completed by May 2012, so this is the second study there was considered a midline survey. Although the final comparison report had not been completed, the team extensively reviewed this data and had several meetings with the ICDDR,B and MaMoni staff about this data. The team also reviewed the project monitoring data. On May 21, 2013, the team presented their findings and recommendations to USAID/ Bangladesh. The information was shared with MaMoni staff on May 22, 2013. LIMITATIONS OF EVALUATION The team faced several key limitations when conducting the evaluation. First, the project evaluation indicators did not have specific targets, so the team was not able to determine if the objectives were met. MaMoni, with USAID/ Bangladesh, does set annual process targets and reports on them in quarterly project reports (Annex VI: Achievement vs. Target for FY 2012). The team reviewed this information to identify trends in progress since the May 2012 survey. Second, the team cut short their field visits by two days due to the political unrest in the country. This limited the team’s ability to see all the planned activities and reduced the time available for speaking with stakeholders. In addition, heavy rains and restrictions on vehicle usage made it very difficult to visit some sites. The team did visit one of the upgraded FWCs in a 12 USAID/BANGLADESH: MAMONI FINAL EVALUATION hard-to-reach upazila, which was a key project strategy. However, they were not able to directly visit a remote area. With that said, the team believes that adequate time was spent in the field and there was enough exposure to MaMoni’s key interventions to gain a good understanding of the project’s efforts, successes, and challenges. There were also several limitations with the data. First, even though the final/midline surveys were conducted in May 2012, the data were still being analyzed. Second, there was little change in the data between the two studies, which made it hard for the team to draw any conclusions regarding the project’s achievements. The team tried to use the MOH&FW project data to supplement the survey data. Even though MaMoni has been trying to strengthen this data, they feel that it is quite unreliable. Lastly, the team was asked to conduct a final evaluation, but the final survey in Habiganj will not be conducted until November 2013. USAID/BANGLADESH: MAMONI FINAL EVALUATION 13 V. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ACHIEVEMENTS TO DATE Community Mobilization A key MaMoni strategy was to form CAGs to mobilize the community and develop action plans to address critical MNH/FP issues. In Sylhet, 54% of CAGs are continuing to function with minimal support, since MaMoni began to phase out their activities in September 2011.13 In Habiganj, 93% of villages had a CAG and 85% had a UP member, FWA, or HA as a CAG member, indicating that CAGs are highly valued. MaMoni engaged and strengthened existing CC support groups (69) to function as CAGs and created new CAG groups for the other 106 CCs. The CAGs were encouraged to contribute labor for supporting the CC. Between July 2012– March 2013, they had provided in-kind labor and a total of 36,700 taka for the CCs. As of December 2012, nearly all the CAGs (99%) had developed emergency transportation systems to support access to services for women and newborns, and 84% had funds to support transportation and services.14 The team found the CVs/CAGs are highly motivated and have a strong sense of community ownership. MaMoni has also worked closely with the UPs to orient them to MNH/FP issues, reactivate the health committees, encourage them to monitor the health facilities, and link them with the CAGs. This has been a very successful strategy in terms of leveraging both funding and support. By December 2012, 90% of the UPs had allocated some of their budget for MNH/FP efforts, and 79% of the committees had completed a bimonthly meeting. Between July 2012 and March 2103, the UP provided 81,406 taka to support the FWCs and satellite clinics and 170,834 taka to support the CCs. BCC MaMoni aimed to improve knowledge of danger signs; knowledge and use of birth plans15; and use of handwashing with soap by recently delivered women after cleaning their babies’ bottoms, after defecating, and before breastfeeding. Counseling by FWAs and FWVs was the key strategy for disseminating BCC messages. However, in 2012, only 38% of pregnant women received ANC counseling by any provider and only 5% of these women were counseled by an FWA. Moreover, of the women counseled, only 7% received information on danger signs, which is probably why the results are mixed. Table 2 shows a significant increase in knowledge of bleeding and a modest rise in knowledge of convulsions and retained placenta. However, based on the low levels of counseling the extent to which MaMoni contributed to these increases is unclear. The decline in knowledge of danger signs for pre-eclampsia and fever indicates more attention is needed in this area. 13 MaMoni Project data. 14 Ibid. 15 Birth-planning components include where to deliver (with misoprostol, if at home), with whom to deliver (SBA), savings, and transportation. 14 USAID/BANGLADESH: MAMONI FINAL EVALUATION Table 2: Knowledge of Danger Signs among Women by Time Period, Habiganj Danger Signs Pregnancy Childbirth Postpartum 2010 2012 2010 2012 2010 2012 Severe headache 43.2 35.8 18.6 10.7 19.6 8.9 Blurred vision 20.6 25.4 8.3 5.9 9.9 7.3 Convulsions/fits 14.7 17.8 23.9 30.1 18.2 22.5 Excessive vaginal bleeding 7.5 11.7 22.5 41.4 38 57.1 Fever 22.9 21.7 14.6 11.6 28.3 24.9 Baby in abnormal position N/A N/A 56.2 57.4 N/A N/A Prolonged labor (>12 hours) N/A N/A 55.7 51.1 N/A N/A Retained placenta N/A N/A 15.6 19.1 13.1 22 Source: ICDDR,B baseline report and midline survey data tables for Habiganj In addition, few pregnant women received information on birth planning through ANC counseling. Of the 38% of women who were counseled, only 1% were told about transportation, 2% were told about savings for emergencies, and 5% were told about delivering in a facility.16 Among women who reported that they had a birth plan during their last pregnancy, 27% had decided where to deliver, 24% knew which provider they would deliver with, 12% had savings (down from 21% in 2010), and 4% had arranged for transportation.17 In terms of the use of soap during hand washing among recently delivered mothers, 50% reported using soap to wash their hands after cleaning their babies’ bottoms and after defecating; 21% reported using soap to wash their hands before feeding their babies; and no data were available regarding using soap to wash hands before breastfeeding.18 Thus, while there was some improvement in handwashing with soap, it is not clear to what degree MaMoni’s activities influenced it. It may be better to track accessibility of soap and water, an issue that has proven to be a primary factor in low rates of handwashing. Community-based MNH/FP Services The key community-based services were primarily provided by FWAs. Services included FP (pills, condoms, injectables) and PNC. FP: In March 2013, 88% of eligible couples were using a short-term FP method against targets. Data are not disaggregated by source, so the team could not determine the proportion provided by FWAs. There was a modest increase in the use of injectables from 16,000 (January– March 2010) to 20,000 (January–March 2012), but the data do not show any increase in provision by FWAs. PNC: There appear to be improvements in PNC, but the data collected differ by time period. The survey data looked at PNC visits within 24 hours by any provider, which rose from 13.3% in 2010 to 17.7% in 2012.19 MaMoni tracked PNC visits by FWAs within 72 hours of birth. This 16 ICDDR,B, Habiganj baseline and midline surveys. 17 Ibid. 18 Ibid. 19 ICDDR,B baseline and midline survey. USAID/BANGLADESH: MAMONI FINAL EVALUATION 15 metric greatly exceeded project targets, increasing from an average of 26% (January–June 2012) to an average of 71% (July–December 2012). MNH/FP Services Table 3 presents the key expected results of MNH/FP services outlined in the project framework, including ANC, deliveries with SBAs, PNC, newborn care, and FP. It is important to understand the timing of the interventions in relation to these results. A baseline survey was conducted in October 2010 in both districts. In Sylhet, activities began in December 2009, following the ACCESS project that established many community-level interventions and some linkages with MNH services. Activities in Habiganj began in June 2010, without any previous efforts. A second survey was conducted in May 2012 and was considered a final survey for Sylhet and a midline survey for Habiganj. A final survey will be conducted in Habiganj in November 2013. In Sylhet, activities began to be phased down in September 2011. It is not surprising that some of the indicators are slightly higher in Sylhet, since many of the activities were established under ACCESS (e.g., trained TBAs). However, it is unclear if the increases in women delivering with an SBA can be attributed to MaMoni, as it was phasing out at the time and did not support FWAs to provide deliveries. Table 3: MaMoni Key Achievements by District Indicator Sylhet Habiganj Oct. 2010 May 2012 Oct. 2010 May 2012 % women who received four ANC visits 13.7% 8.2% 10.2% 8.7% % of pregnant women who received two doses of tetanus toxoid 88.3% 89.5% 92% 93.5% % of women who received iron folate during pregnancy 48.2% 48.2% 36.1% 41.8% % of women who used new blade 13.6% 70% 92.6% 87.5% % of women delivering in facilities 19.1% 23% 12.6% 17.6% % of women delivering at home 80.9% 77% 87.4% 82.3% % of women delivering with an SBA 21.0% 25.8% 15% 19.4% % of women delivering with a trained TBA 39.5% 43.7% 13.3% 30% % of women (home deliveries) who received a postnatal visit within 24 hours of delivery by any provider 19.5% 23% 13.7% 17.7% % of women with a danger sign during delivery and sought care from a skilled provider 55.4% 58.0% 37.7% 47.5% % of newborns with a danger sign and sought care from a skilled provider 51.5% 54.2% 42.0% 46.1% % of women practicing lactational amenorrhea (LAM) correctly 0 NA 0 NA Contraceptive prevalence rate 34% 40.3% 39.1% 40.6% % of women using an LAPM 6.2% 8.1% 7.9% 5.5% Source: ICDDR,B baseline report and final survey data tables for Sylhet; ICDDR,B baseline report and midline survey data tables for Habiganj. Note: ICDDR,B is still analyzing data, so it is not clear if these are statistically significant findings. 16 USAID/BANGLADESH: MAMONI FINAL EVALUATION One positive trend—although it is unclear if this is a direct result of project activities—is that while some indicators have only seen modest gains, there appears to be greater increases among women in the lowest quintile (poor women), as Table 4 presents. Table 4: MaMoni Key Achievements by District, Lowest Quintile Indicator Sylhet Habiganj Oct. 2010 May 2012 Oct. 2010 May 2012 Women delivering in facilities 19.1% 23% 12.6% 17.6% Poor women delivering in facilities 3.8% 10.3% 5.7% 8.6% Sought care for delivery complication from SBA 55.4% 58.0% 37.7% 47.5% Poor women who sought care for delivery complication from SBSBA 26.1% 34.7% 23.0% 32.2% CPR (modern method) 34% 40.3% 39.1% 40.6% CPR among poor women 30.5% 41.9% 38.8% 36.7% Source: ICDDR,B baseline report and final survey data tables for Sylhet; ICDDR,B baseline report and midline survey data tables for Habiganj. Note: ICDDR,B is still analyzing data, so it is not clear if these are statistically significant findings. Since there was little change between the surveys in Habiganj, the team extensively reviewed the monitoring data to determine if changes had occurred since May 2012. This section provides analysis of more recent project data. ANC: Although satellite clinics increased from 484 a month in 2010 to 537 in February 2013, ANC coverage has remained relatively static. Figure 1 shows an initial increase among women receiving ANC, but this has remained relatively constant since September 2012. Further investigation is needed to improve ANC coverage. Figure 1: Trends in ANC, Habiganj 23 35 17 22 21 23 24 25 22 20 29 40 31 39 39 40 42 43 42 37 0 10 20 30 40 50 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-13 ANC 1 ANC 3 Source: MaMoni project MIS data The quality of the ANC was also a concern for MaMoni. A key gap identified in the health facility assessment (HFA) was the lack of testing supplies to identify proteinuria and hemoglobin levels. In March 2013, 91% of pregnant women who received ANC from a FWV received a urine and blood test.20 MaMoni also ensured that women who planned to deliver at home received misoprostol during ANC care or through a follow-up visit by FWA. To date, 23% of women 20 MaMoni Project data. USAID/BANGLADESH: MAMONI FINAL EVALUATION 17 planning to deliver at home received misoprostol.21 The team did not hear of any reports of incorrect use of misoprostol. Deliveries with SBAs: MaMoni used several approaches to improve deliveries with SBAs, including upgrading FWCs to provide 24/7 services, supporting private CSBAs in hard-to-reach areas, and enhancing referrals. Table 5 shows significant increases in deliveries at two upgraded FWCs. Table 5: Deliveries Conducted in Upgraded Female Welfare Centers Name of FWC No. of Deliveries (Sept. 2012–April 2013) Population % of Total deliveries Shibpasha 228 27,161 57.2% Murakari 200 25,680 53.1% Source: MaMoni Project data, May 2013 MaMoni trained 31 CSBAs and placed them in hard-to-reach areas. In one upazila, 14 private CSBAs conducted 193 deliveries (May 2012–March 2013). While this is a relatively low number of deliveries per CSBA, it comprised 38% of all the deliveries in the area.22 Newborn care: Essential newborn care (ENC) was part of the PNC visits provided by FWAs. All providers in both districts received training on ENC and helping babies breathe to manage asphyxia. In addition, providers at the Habiganj DH and MCWC were trained on management of sick newborns. There were 65 referrals to the DH for newborn complications (February 2012– March 2013). The project did not have any data regarding newborns successfully resuscitated. MaMoni planned to use village doctors to treat newborn sepsis in the community, but the MOH&FW changed their policy so MaMoni dropped this effort. The current plan is to have the sub-assistant community medical officer at the FWC be the first referral point for newborn sepsis, but this has not been implemented. FP: The MOH&FW reports a contraceptive acceptance rate (CAR) of around 75%, which is much higher than the CPR in the midline survey (40.6%). The MaMoni team reported that the CAR is not a reliable estimate of FP use. As previously mentioned, there was a modest increase in injectable use. LAPM uptake is sporadic, due to a reliance on camps, but the average monthly case load rose from 553 in 2010 to 750 in 2012. About 30% of these referrals were made by CVs. MaMoni was supposed to track LAM use and transition to a modern method. This has been a challenge, because the MOH&FW’s register does not record any postpartum FP method. If this is an area that USAID wants to continue tracking, MaMoni will have to establish a separate mechanism. Health Systems MaMoni worked to improve seven critical systems issues to improve access to and quality of MNH/FP services. Efforts included 1) developing strategies for hard-to-reach areas, 2) filling critical gaps among health providers, 3) providing essential drugs and equipment, 4) renovating/constructing facilities, 4) strengthening the referral system, 5) facilitating supervision, 21 Ibid. 22 MaMoni Sylhet phase-out presentation. 18 USAID/BANGLADESH: MAMONI FINAL EVALUATION 6) improving QOC, and 7) strengthening the HMIS. Table 5 presents the accomplishments in hard-to-reach areas. Critical gap management: MaMoni identified key positions needed to provide quality MNH/FP services, including MOs, nurses, FWVs, and FWAs. Then they identified the key facilities that needed to be supported with appropriate staff. In Sylhet, MaMoni inherited 286 CHWs from ACCESS. These functioned like FWAs but had smaller catchment areas. As part of the phase-down efforts, MaMoni decided they would only fill sanctioned vacant FWA positions, so in 2011, the temporary FWAs (previously called CHWs) were reduced from 286 to 86 (all vacancies). In 2012, the MOH&FW was able to fill 59 of these positions, and the MaMoni￾supported FWAs were reduced to 27. In Habiganj, MaMoni hired 41 temporary FWAs (64% of vacancies) that are still in place. It should be noted that the district was able to fill these positions because the central government conducted a recruitment for them, after a 12-year absence. MaMoni also hired 36 temporary FWVs, based at the FWCs and the MCWC, (all vacancies) and 10 nurses (14% of vacancies), based at the DH. It is clear that the additional staff have significantly contributed to the increases in service use, particularly for deliveries and referrals. In December 2012, only two of the eight MOs-MCHP and four of the 10 RMOs were in place.23 MaMoni tried to hire MOs—even doubling the MOH&FW salary level—without success. Most MOs do not want to be based in remote areas with poor or non-existent accommodations. MaMoni and several collaborating partners have been advocating with the MOH&FW to fill these positions—particularly in hard-to-reach areas—but they do not have the authority to make these decisions alone. Renovation/construction: The HFA found that most facilities did not have adequate infrastructure or access to water, equipment, or supplies needed to provide quality services. In the hard-to-reach areas there were not enough facilities or staff to support services. MaMoni leveraged funding from the Korean International Cooperation Agency (KOICA) to renovate seven existing FWCs to provide normal deliveries (five are currently operational), two upazila health complexes to provide basic EmONC services, and key areas of the DH/MCWC to provide comprehensive EmONC services. MaMoni leveraged funding from Save the Children UK that will be used to construct seven new FWCs in hard-to-reach areas. Medicines and supplies: As previously mentioned, MaMoni provided ANC testing supplies and misoprostol. The HFA found that supplies (e.g., delivery beds) and equipment were insufficient to ensure quality MNH/FP services, so the project supplied those that were not available in the MOH&FW system. There were no reports of any stockouts or issues with supplies. Referral system: MaMoni worked with the CAGs/CVs, UP members, and health providers to develop an extensive referral system from the village to the DH. As a result, there has been a six fold increase in maternal referrals as presented in Figure 2. 23 Ibid. USAID/BANGLADESH: MAMONI FINAL EVALUATION 19 Figure 2: Maternal Complications Referred to the Habiganj District Hospital 0 50 100 150 200 250 300 350 400 450 April - June 2011 July-Sept 2011 Oct-Dec 2011 Jan-Mar 2012 April - June 2012 July-Sept 2012 Oct-Dec 201 Jan-Mar 2013 26 36 49 95 120 107 283 445 Source: MaMoni Project MIS data Supervision: MaMoni supported the MOH&FW managers to develop supervision checklists and conduct joint supervision visits. Joint supervision visits increased from 61% of planning visits in 2011 to 72% in 2012.24 The key limitation of these visits is the high vacancy rate among the district and upazila managers. HMIS: MaMoni developed a micro-planning process that brings together HAs and FWAs to ensure that they report the same information. The micro-planning meetings are being held regularly; 95% of projected meetings take place in both districts. In Sylhet, the micro-planning meetings were well attended, as evidenced by the participation of HAs (99% of meetings), FWAs (94% of meetings), and CVs (69% of meetings) between October 2012 and April 2013.25 In Habiganj, 84% of the March 2013 meetings were attended by both HAs and FWAs, and 75% of meetings had participation of CVs.26 Based on discussions with district managers, this is the first time that there has ever been consistent reporting, particularly for maternal deaths. Stakeholder Commitment and Ownership MaMoni has done an excellent job of engaging and gaining commitments from stakeholders at various levels. The project has leveraged $2.3 million from other donors and private sources to hire temporary staff and renovate facilities. Nationally, MaMoni—with other NGOs—has successfully advocated for key operational and policy changes (e.g., FWAs providing second dose of injectables at home, helping babies breathe). MaMoni collaborates closely with both the DGHS and DGFP at the union, upazilia, district, division, and national levels. MaMoni’s efforts have been recognized, as evidenced by the Shibpasha FWC winning the 2012 best facility for normal delivery award in the Sylhet division, and third best nationally. In addition, Habiganj district had the highest rate of tetanus vaccination in the country in 2011. Innovation: MaMoni embraces a spirit of innovation and has become an important learning lab for key MNH/FP interventions. MaMoni has tested new models (e.g., 24/7 services in FWCs) and 24 MaMoni Project Data. 25 Ibid. 26 Ibid. 20 USAID/BANGLADESH: MAMONI FINAL EVALUATION operationalized research results from other projects (e.g., misoprostol and helping babies breathe). They are also collaborating with MOH&FW and ICDDR,B to conduct four operations research projects that study treatment of pre-eclampsia/eclampsia with a loading dose of magnesium sulfate by FWVs, treatment of sick newborns at the upazila level (Sylhet), consolidation of MOH&FW facility registers, and use of the referral system. These efforts have only recently started, so the team cannot comment on their progress. EFFECTIVENESS OF PROJECT STRATEGIES Community mobilization In Sylhet, the local NGOs formed the CAGs and supported them in an action-planning process. As the program progressed, community resource people were identified and given the responsibility of running the CAGs. Based on these lessons, local NGOs in Habiganj identified potential CVs27 with the communities, FWAs, HAs, and UP members. Final selection was done by the UP chairman. After the CVs received training in MNH/FP messages, social mapping, and development and implementation of action plans, they helped form and lead the CAGs, with support from the NGOs. It is estimated that CVs spend about 10 hours a week on these activities. The team felt the action plans reviewed were quite specific, and the CVs have been able to implement them. The CVs/CAGs were also very effective in creating and supporting the referral system and participating in micro-planning meetings. MaMoni has a well-documented approach to the CAG formation and action-planning process. The process of phasing out their support of the CAGs is less clear. For example, when is a CAG is considered “graduated” and no longer in need of any follow-up? MaMoni staff reported that 60% of the CAGs can function independently after the third action-planning cycle, but it was unclear how MaMoni staff reduced their support of these CAGs. The team feels that the project is heavily staffed and that the Sylhet phase-out process of the CAGs could have been done much quicker. Conclusion CVs/CAGs are highly motivated and have a strong sense of pride. The relatively low CV turnover rate (19%) is an indication that they feel appreciated by community. Recommendations 1. Develop a transition strategy for the CAGs/CVs that outlines the steps and level of effort needed to start the CAGs, as well as the declining level of effort as the groups become independent. 2. Continue to create CVs/CAGs with strong linkages with UPs and FWAs/HAs. Behavior Change Communication The technical proposal indicated that MaMoni would develop a BCC strategy for Habiganj, but this was not completed. The team estimated that there were 60 messages to be communicated. This made these messages daunting to disseminate and challenging for women to understand. MaMoni primarily relied on counseling by FWAs during home visits and FWVs during ANC visits, supported by BCC materials. Based on informant interviews, there has been an increase in understanding of the messages by FWA and FWVs, but they do not have time to provide these 27 CRPs and CVs have the same function in CAGs. USAID/BANGLADESH: MAMONI FINAL EVALUATION 21 messages to the women. The household coverage by FWAs is very low (11%), and FWVs reported that they often spend only 30 minutes a week doing counseling. As a result, women are getting very little information from these providers. MaMoni reproduced most of the BCC materials that were already approved by the MOH&FW. This expedited their use, but not all the messages could be addressed by the existing materials. As a result, FWAs reported using their training materials to counsel women because they did not have adequate materials. MaMoni developed a flip chart for the FWVs to address some of the gaps; the development and approval process took two years. All the materials had a combination of photographs and illustrations. Based on interviews, the photographs were the easiest for women to understand. Some health messages were provided during the CAG monthly group meetings. “Message of the month” was recently started to more systematically disseminate messages during these meetings. MaMoni was very concerned about overburdening the CVs with additional responsibilities and did not ask them to provide any messages outside the CAG meetings. Based on the team’s discussions with the CAGs and MaMoni staff, it was clear that the CVs/CAG members increased their own knowledge on MNH/FP issues, but this information did not trickle down to the larger community. MaMoni used some mass- and mid-media approaches (e.g., videos, phones), mostly through add￾ons to other partner projects (e.g., MAMA). Some videos were shown by local cable providers and small sign boards were placed near facilities. However, no formative assessments were conducted to help select communication channels and data were not collected on the impact of these approaches. Conclusion The BCC approach relied heavily on counseling by the FWAs and FWVs during their contact with women, but these workers do not have sufficient time to provide the (60) messages to women. The team feels that the CVs could have been used more to provide messages. In addition, a more balanced mix of communication channels (e.g., counseling, videos) would have provided more opportunities for key audiences to be exposed to the messages. Recommendations Develop a comprehensive BCC strategy that:  Prioritizes key messages and groups their dissemination in a way that is easier for people to absorb;  Utilizes a mix of communication channels, based on cost-effectiveness;  Explores greater utilization of CAGs/CVs to undertake systematic dissemination of key messages supported with BCC materials; and  Maximizes the use of phones as a communication channel, supported by free airtime. Community services The primary interventions to provide community-based services included 1) supporting FWAs, 2) training and supporting private CSBAs, 3) training HAs on FP, 4) reorienting TBAs, and 5) establishing depot holders. FWAs: MaMoni trained FWAs to improve their MNH/FP knowledge and skills regarding pills, condoms, injectables, and proper use of misoprostol. In addition, MaMoni hired FWAs to 22 USAID/BANGLADESH: MAMONI FINAL EVALUATION temporarily fill vacant MOH&FW positions, which is further discussed under the health systems strengthening section. In Habiganj, half of the temporary FWAs were placed in hard-to-reach areas. Populations in FWA coverage areas have greatly increased since the areas were designated, resulting in low coverage of FWA home visits. Now FWAs are required to be at the CC three days a week, further limiting their ability to conduct home visits. As previously mentioned, there was little change seen in women receiving ANC counseling and key information (e.g., danger signs) from FWAs. Project data have not shown any increase in the use of injectables provided by FWAs nor in home deliveries by those trained as CSBAs. A key positive note is that PNC visits by FWAs within 72 hours of birth have dramatically increased. Conclusion FWAs are not able to adequately cover their currently assigned catchment areas. Current data do not indicate that MaMoni’s support to FWAs has resulted in improved counseling and or increases in FP services. Data do show increases in PNC within 72 hours of birth. MaMoni will need to look at endline data carefully to determine whether to continue support to FWAs. Recommendations 3. If final data does not show improved FWA provision of information and services at the household level, USAID should seriously consider not hiring more temporary FWAs and instead focus on other strategies to assure the FWA responsibilities are addressed. If the decision is made to reduce reliance on FWAs, then it is recommended that their training only focus on an overview of MaMoni efforts, linkages with CAGs, referrals for complications and LAPM, and provision of injectables and counseling on misoprostol. 4. Explore other options to undertake BCC efforts (e.g., CVs, CSBAs). 5. Explore other delivery channels (e.g., CCs, pharmacies, depot holders, private CSBAs) to improve access to services and commodities that are close to the women but not necessarily in their home. Private CSBAs: In Habiganj, the MOH&FW upgraded 129 FWAs and female HAs to function as (government) CSBAs, but few were conducting deliveries because they were already overloaded with their current job responsibilities.28 Access to SBAs in hard-to-reach areas was a major problem, so MaMoni recruited and trained 31 private CSBAs, using KOICA support. CSBA candidates were selected with community and UP input. After they completed their training, the UP introduced them to the community and helped negotiate their rates with the community. MaMoni provided the CSBA with a kit that included basic equipment and supplies (e.g., oxytocin) and some airtime to report emergencies. CSBAs were trained to provide pills, condoms, and injectables, but MaMoni is still advocating with DGFP to provide them supplies. The CSBAs reported their activities to the FWVs and are linked with local depot holders for replacement supplies. Based on informant interviews, most private CSBAs performed deliveries at the women’s homes, while a few have space in their own home or delivered in the CC. Data indicate that CSBAs conduct between one and five deliveries a month on average.29 28 MaMoni Project update, April 30, 2013. 29 MaMoni Project MIS data. USAID/BANGLADESH: MAMONI FINAL EVALUATION 23 Conclusion This is still a new cadre in Bangladesh and more information is needed to fully understand CSBA’s performance, effectiveness, and best use of their skills. There are still questions regarding how to maintain their skills while doing few deliveries. Recommendation Conduct an assessment of the current CSBAs to understand 1)the conditions that create the best environment for them to be effective (e.g., links with UP), 2) the situations where they can be the most useful (e.g., hard-to-reach areas), 3) their performance and skill level, 4) challenges they have experienced, and 5) additional support that may be required ( e.g., other services, business training) for them to be successful and have an acceptable income. Training HAs: HAs belong to DGHS and their sole responsibility is vaccinations. MaMoni trained them on key MNH/FP messages, including provision of pills, condoms, and injectables. However, they are very busy and there is little, if any, evidence that they provided any message or FP services to women. Conclusion There is no evidence to suggest that HAs provided information or FP services to women as a result of their training. It does not appear that they have time for any of these added activities. Recommendation Do not continue training HAs. Reorienting TBAs: MaMoni trained several FWVs as TBA trainers to create a positive relationship between the facilities and the TBAs. Then the FWVs trained TBAs on clean delivery; avoidance of harmful practices; early recognition of danger signs; use of misoprostol and referral; and essential newborn care, including tactile stimulation. Each TBA had the phone number of the FWV and a CSBA. The relationship between the FWVs and TBAs seems to be working well, based on discussions with several FWVs. The team interviewed four trained TBAs who knew danger signs and indicated they accompanied the mother for referrals and in some cases stayed with her at the hospital. The TBAs also reported that the families often still pay them when they refer them to the facility. MaMoni does not track referrals by TBAs. Conclusion The training has built rapport between the FWVs and TBAs. Recommendations 6. Track referrals by TBAs to different health facilities. 7. Continue to reorient TBAs on these topics. Linked depot holders to supply: MaMoni established a cadre of community-based depot holders to provide basic health products (e.g., pills, condoms) in both districts. Many of the depot holders are also CVs. MaMoni linked the depot holders with BRAC to source the supplies that they sell in the village. They function independently, so MaMoni does not track their sales. As a result, the team cannot comment on the effectiveness of this strategy. Strengthening Health Systems MaMoni worked on seven key systems issues to improve access to and quality of MNH/FP services, including 1) developing strategies for hard-to-reach areas, 2) providing critical gap 24 USAID/BANGLADESH: MAMONI FINAL EVALUATION management for health workers, 3) supplying essential drugs and equipment, 4) renovating and constructing facilities, 4) strengthening the referral system, 5) facilitating supervision, 6) improving QOC, and 7) strengthening the HMIS. Hard-to-reach areas/populations: Based on the district assessment, MaMoni developed specific strategies to work in three hard-to-reach upazilas and identified key facilities to be renovated/constructed, equipped, and appropriately staffed based on their geographic location, the ability of the population to access services, and referrals to higher-level facilities (Annex VIII: Maps of Hard-to-Reach Strategies). A key strategy in hard-to-reach areas is upgrading FWCs to provide 24/7 services. The MOH&FW system places only one FWV at a FWC to conduct deliveries and provide other MNH/FP services. MaMoni tested a model in which FWCs were upgraded to provide 24/7 services, including 1) placing two FWVs at the FWC; 2) renovating the FWC and staff quarters; and 3) providing essential equipment, supplies, and medicines. As previously shown in Table 5, this is working very well in Shibpasha and Murakari, where more than 50% of the deliveries occurred in the FWCs (September 2012–April 2013). MOH&FW managers have visited these facilities and been very impressed with the progress. MaMoni is advocating for this approach, particularly in hard-to-reach areas. MaMoni also addressed two other special populations. In the tea gardens, the project oriented worker volunteers on MNH/FP messages to share with their peers and trained their health workers on the use of misoprostol. In the slum areas, the project oriented existing community groups on MNH/FP messages, negotiated a reduced rate for the community to access Smiling Sun satellite clinics, and improved the referral system to the DH, in collaboration with the Urban Partnership for Poverty Reduction project. Conclusion An integrated health systems strengthening approach that combined increases in staffing levels, renovation of health facilities and staff quarters, and sufficient equipment and supplies was needed to address the existing gaps in the health system. This has been a successful approach, evidenced by the increased use of delivery services. MaMoni crafted strategies to address the specific needs of hard-to-reach areas and special populations. This approach built on existing structures and coordinated local resources, allowing the project to maximize its resources while having the greatest impact. Recommendations 8. Continue to implement an integrated health system approach. 9. Continue to customize strategies for hard-to-reach areas and special populations. 10. Continue to advocate with the MOH&FW to adopt the upgraded FWC 24/7 approach. Critical gap management: To deal with the high vacancy rates among health providers, MaMoni received approval from the MOH&FW to temporarily fill vacant positions. MaMoni hired temporary FWAs in both Sylhet and Habiganj. In Sylhet, the district hired 59 FWAs in 2012, reducing the MaMoni-supported FWAs to 27. The 41 temporary FWAs are still in place in Habiganj. Although the temporary FWAs are functioning well individually, there is little evidence that FWAs overall have improved access to information or services. MaMoni hired 36 temporary FWVs and 10 nurses. It is clear that these additional staff have significantly contributed to increased service use, particularly for deliveries and referrals. Nurses USAID/BANGLADESH: MAMONI FINAL EVALUATION 25 have proven to be a very good strategy, especially for management of sick newborns. MaMoni reported that nurses could play a greater role in the future, but they need training on time management and waste disposal, which the project did not initially provide. MaMoni has been advocating for task shifting to nurses (e.g., NSV and epidural block) among the MOH&FW and professional bodies, but these changes have not been approved yet. A high vacancy rate among MOs continues to be a problem. MaMoni tried to hire MOs, and even doubled the MOH&FW salary level, without success. Compensation is important, but most MOs do not want to be based in remote areas, with poor or non-existent accommodations, lack of schools, and limited opportunities for private practice. This has placed an additional burden on the existing MOs and reduced the availability of LAPM and C-section services. MaMoni has been advocating with the MOH&FW to prioritize replacements based on greatest need (e.g., hard-to-reach areas), but it is not their decision alone. The Ministry of Finance and the Professional Establishment Bureau also play key roles in the recruitment, hiring, and placement process. In addition, recruiting nurses is the responsibility of the Department of Nursing and the Nursing Council, which are not part of the MOH&FW. MaMoni has not established a relationship with the Nursing Department, Nursing Council, or Professional Nursing Association. Such steps would be important if the project plans to work more with nurses. Conclusion The additional FWVs and nurses have clearly increased the availability of MNH/FP services. MaMoni should explore increasing the role and number of nurses and continue advocate task shifting to them. As previously mentioned, hiring FWAs did not appear to be a successful strategy in terms of improving access to household counseling or services. Hiring MOs has been a major challenge that the project still needs to address. Recommendations 11. Continue to advocate with the MOH&FW to fill vacant positions in low-performing areas. 12. Continue to temporarily fill key FWV and nursing positions. 13. Provide additional training for nurses in time management and waste disposal. 14. Develop a relationship with Department of Nursing, Nursing Council, and Professional Nursing Association, and advocate for recruitment and replacement of vacant positions. 15. Continue to explore options to reduce vacancies among MOs. Medicines and supplies: The HFA found that supplies and equipment needed to provide quality services were insufficient or missing. Interestingly, many of these supplies (e.g., generators) were available in the government system, but the district managers did not know they were available or how to access them. MaMoni played an important facilitation role in helping the district managers obtain existing supplies from the central store. Where supplies were not available, MaMoni provided them (e.g., ANC testing supplies, misoprostol) and distributed them through the MOH&FW system. 26 USAID/BANGLADESH: MAMONI FINAL EVALUATION Conclusion The facilitative role played by MaMoni was key to educating district managers on how to access existing supplies within their own system. Ensuring sufficient supplies and medicines were available greatly contributed to increases in the use of MNH/FP services. Recommendations 16. Continue to assess the status of essential medicines, equipment, and supplies. 17. Continue to assist managers in navigating the health system and obtaining existing supplies. Health facilities: MaMoni leveraged funds from KOICA and Save the Children UK for the facility renovation and construction. Based on informant interviews, the community has highly praised the renovations and become much more willing to use the services. Improvements of staff quarters have also been a key component to obtaining and retaining staff. The UP and CAGs have been actively engaged in supporting the renovations, either with funds or in-kind contributions, and there was evidence that the UP health committees are monitoring the FWCs. Conclusion The renovations played an important role in improving service use and somewhat improving quality (e.g., water sources). The UP and CAGs take ownership for these facilities. Improving staff quarters, including security in remote areas, has been key to ensuring that 24/7 services were available. Recommendation Continue to complete the renovations and begin construction of the new FWCs. Referral system: MaMoni involved the CAGs/CVs, UP members, and health providers in mapping a detailed referral system with clear pick-up points that was widely shared with the community. Most of the CAGs had established an emergency transportation system or had funds to support women’s access to services. Based on informant interviews, if a woman or newborn needed to be transferred to the DH, a CV called the MaMoni staff in the field or at the DH. The referral team (at the DH) played a critical role in helping the providers prepare for admissions and creating a sense of urgency once the patient arrived. MaMoni needs to think about how to transition this important function to either MOH&FW or the social welfare department for the efforts to be sustained. The relationship created among the providers through training and supervision, supported by cell phones, has greatly improved the referral system. When providers at the FWC have a problem, they call the providers (their trainers) at the DH/MCWC to discuss what they should do and inform them about the referral. This system is working well and is very likely to continue. MaMoni tracked information about the referrals, but further review is needed to assess QOC. For example, many women were admitted for prolonged/obstructed labor and had a normal delivery. This treatment is not consistent with the diagnosis. MaMoni staff reported that the nurses wrote the diagnosis, but they have not received any training on classifying complications. This appears to be an area where EmONC refresher training, as well as correct documentation of complications, could strengthen QOC. USAID/BANGLADESH: MAMONI FINAL EVALUATION 27 Conclusion MaMoni has developed a very comprehensive referral system that is widely known by the community. The presence of a team at the DH has greatly expedited admissions and management of complications. Recommendations: 18. Consider how to transition the DH referral team function to either the MOH&FW or social welfare department for the efforts to be sustained. 19. Provide EmONC refresher training, including documentation of complications, to nurses and MOs. QOC: MaMoni developed master trainers from district managers to improve provider’s knowledge and skills. They also introduced standards-based management and recognition (SBM￾R) tools in 2011. Standards for ANC, PNC, FP, and infection prevention were developed in consultation with district-level managers and members from the Obstetric and Gynecological Society of Bangladesh (OBGSB) and the Bangladesh Neonatal Forum. However, SBM-R has not been fully implemented; several key stakeholders have expressed concern that there are too many tools. MaMoni is working to further simplify the standards in consultation with relevant service providers. High vacancy rates among providers, combined with the increases in demand, raises a concern over whether QOC can be maintained, even with the additional staff hired by MaMoni. Based on informant interviews, it does not appear that facility-based maternal/newborn death audits or review of complications are being conducted in a systematic manner at the DH. Data regarding referrals were collected but have not been analyzed from a QOC perspective, and the project does not have any quality indicators to track changes in provider practices. Conclusion MaMoni has made several efforts to improve QOC, but less than optimal utilization, high vacancy rates, and lack of clear standards may compromise quality. This is an area that needs further attention. Recommendations: 20. Support district managers in establishing clear standards (either SBR-M or another quality assurance approach). 21. Promote regular reviews of complications/deaths at the DH/MCWC. 22. Incorporate QOC indicators (e.g., newborns successfully resuscitated) to track changes in provider practices and overall care. 23. Conduct further review of the referral data to assess management of complications. Supervision: The pool of trainers, developed from the district managers, has proven to be a useful training and supervision approach. Managers were supervising the trainees, but it is unclear how often they are being followed up on. The trainee checklist reviews key areas of knowledge and skills, but it is not scored, a step that would be useful in identifying gaps for refresher training. High vacancy rates among managers, transfers of several trainers, and inadequate training venues limited the ability of the project to conduct training, particularly for FWVs. 28 USAID/BANGLADESH: MAMONI FINAL EVALUATION MaMoni worked with MOH&FW managers to undertake joint supervision visits, based on the supervision checklists. These visits have increased, but high vacancy rates have been a key limitation. District managers reported that sometimes they call the facilities when supervision visits are not possible. In discussions with the managers they found the checklist useful, but it was not clear how problems identified during a supervision visit were managed. MaMoni recently hired a specialist to provide some additional clinical support in light of the high vacancy rates and challenges hiring MOs. Conclusion The training and supervision approach has been fairly successful, but it has been limited by high vacancy rates of managers/trainers. Recommendation: MaMoni should explore other ways to support clinical supervision (e.g., consultants from OBGYN,B) in light of the high vacancy rates. HMIS: The HMIS relies on data collected by FWAs and HAs from the community. Due to low home visit coverage, data collected by FWAs are incomplete. In addition, HAs and FWAs often report different figures for the same indicator (e.g., maternal deaths), because no effective mechanism exists to reconcile these data. To address these challenges, MaMoni developed a micro-planning meeting that brings together HAs and FWAs to ensure that they are reporting the same information. The CVs collected information (e.g., pregnant women, births, deaths, and eligible couples) from the community and provided it at the meeting. This allowed the FWAs to update their registers, fully capturing the information for their area; reconcile their data with the HA before sending to their supervisors; and develop an action plan with the CVs to follow up with women who need additional support. As a result, for the first time ever, the data reported by the HAs and FWAs were the same—a key achievement. The micro-planning meetings are followed by joint meetings with the health and FP providers at the union, upazila, and district levels. Discussions with district managers revealed that they all believed this was a great accomplishment. In addition, FWAs reported greatly appreciating the CVs’ work and viewed them as helping their performance. Conclusion Micro-planning meetings are happening on a regular basis with high participation from HAs, FWAs, and CVs. This has provided a unique opportunity for FWAs to capture accurate data about their catchment area, reconcile the data with HAs, and develop action plans with CVs. This approach has been well received by the FWAs and HAs, as well as the upazila and district managers. Recommendation Continue to support micro-planning meetings with active participation of FWAs, HAs, and CVs. Stakeholder Engagement and Commitment: MaMoni regularly meets with the DGHS and DGFP to update them on the project efforts, address challenges, and plan future activities. MaMoni, with other NGOs, has been very successful in advocating for key operational and policy changes (e.g., FWAs providing the second dose of injectables at home). MaMoni has also hosted field visits by most of the senior officials at the directorate level, and both the current and former secretaries of the MOH&FW have visited MaMoni field activities. The visits have resulted in key support from the MOH&FW officials. MaMoni also participates in several existing USAID/BANGLADESH: MAMONI FINAL EVALUATION 29 national technical committees (e.g., maternal health) and fora to facilitate ongoing dialogue with the MOH&FW, development partners, NGOs, and other professional bodies. At the district level, MaMoni has created strong relationships through an advisory committee comprised of representatives from the MOH&FW; the Ministry of Local Government, Rural Development and Cooperatives; UN agencies; NGOs; and other MNH/FP players. MaMoni has worked to improve coordination between the health and FP directorates by developing joint plans and conducting regular reviews of progress, identifying bottlenecks and developing strategies to address them and plan future activities. As previously mentioned under the HMIS section, efforts have been established to bring the directorates together at all the levels of the health system—community, union, and upazila, as well as at the district level. This is a key accomplishment of the program. Conclusion MaMoni has done an excellent job of engaging and gaining commitments from stakeholders at various levels. Recommendation: Continue to engage a variety of stakeholders to gain commitment and ownership. Overall Conclusion MaMoni has been very successful in engaging and gaining commitment from many stakeholders, including the DGHS and DGFP at all levels, district health managers and providers, UP members/health committees, and communities to support MNH/FP efforts. While final conclusions cannot be drawn until the last survey is conducted in November 2013, project data indicate that MaMoni has achieved significant increases in deliveries with SBAs, referrals for maternal complications, and postnatal care, as well as modest improvements in FP. These gains were realized through the active participation of CAGs and UPs; the testing of new models in hard-to-reach areas (e.g., upgrading FWCs); and an integrated approach to health systems strengthening that included hiring health providers, renovating facilities, ensuring adequate medicines and supplies, and strengthening supervision. The micro-planning meetings improved data quality and local-level planning among the DGHS and DGFP workers and the community. There are several promising practices that have showed modest gains, such as support for private CSBAs and joint supervision, but more information is needed to fully understand the effectiveness of these efforts. Greater attention must be directed toward 1) improving BCC approaches to enhance knowledge of danger signs and birth planning; 2) developing strategies to provide MNH/FP services near women’s homes, but not necessarily in them; and 3) strengthening QOC, particularly for maternal and newborn complications. MaMoni embraces a spirit of innovation and has become an important learning lab for key MNH/FP interventions. MaMoni tested new approaches in hard-to-reach areas, for example, upgrading FWCs to provide 24/7 services, supporting private CSBAs, and developing a referral system. The project also operationalized results from other research projects, including distribution of misoprostol for women delivering at home and training all health providers on managing newborn asphyxia. MaMoni is also collaborating with ICDDR,B on four other research projects. Changing health systems is a long-term process. In only three years, MaMoni has made great strides in improving the functionality of the existing health systems, as evidenced by increased 30 USAID/BANGLADESH: MAMONI FINAL EVALUATION service utilization and project support from MOH&FW managers and policymakers. This approach has been very flexible, adapting to the local needs and building on the lessons from Sylhet as well as other MNH programs in Bangladesh. Based on this success, the evaluation team believes that many of the interventions are ready to be scaled up in other districts—a key step in USAID/Bangladesh’s future strategy. LEVERAGED ACTIVITIES MaMoni has also been very successful in leveraging about $2.6 million in support from other local and international partners:  $1.05 million from KOICA for facility renovation, hiring FWVs and training private CSBAs  $1.3 million from Save the Children UK for the construction of seven FWCs  $80,000 from the Seoul Broadcasting Service for drugs such as calcium and magnesium sulfate, oxytocin for private CSBAs, and diagnostic kits and waste management of three facilities  $126,000 from Alive and Thrive for infant and young child feeding counseling  $45,000 from Venture Strategy Innovations for misoprostol supply  Soap provided by Unilever As previously mentioned, MaMoni is collaborating with ICDDR,B to undertake four operations research projects. ICDDR,B is also helping MaMoni test a unified MNH register for use in FWCs. MaMoni temporary FWAs were trained by the MAMA initiative to help women subscribe to the service. About 85 women subscribed to the MAMA service, which provided customized audio and text messages to change health-seeking behaviors of pregnant women and new mothers using mobile phone technology. While this has been criticized for being unidirectional, hotline service with doctors is planned to increase interactivity. Sixty percent of women have their own phones and others depend on husbands or neighbors to share phones or pass on messages. MaMoni will begin a collaboration with Food and Nutrition Technical Assistance (FANTA) later in 2013 to distribute postpartum iron tablets through FWAs, provide essential nutrition counseling training, and pilot community management of acute malnutrition. While Save the Children has been very effective in leveraging relationships and resources, one of the main challenges has been managing the different timing and reporting requirements. For example, the KOICA support came as a series of annual projects that required all the activities be completed in that year. Unilever was not able to provide supplies in a timely manner, so the project did not have adequate soap for Global Handwashing Day. Lastly, while the operations research projects will provide useful information for Bangladesh, most of these results will not be available until after MaMoni is completed. PROJECT MANAGEMENT MaMoni maintains a continuous relationship with the USAID agreement officer technical representative (AOTR). Based on the team’s interaction, it seems that this is a good relationship. MaMoni participates in the monthly USAID implementing partners meetings and the USAID/BANGLADESH: MAMONI FINAL EVALUATION 31 national TIG (TRAction) with USAID. MaMoni also invites USAID to its coordination meeting with health and FP directorates. MaMoni produces weekly updates, success stories, and quarterly reports for USAID,as well as semiannual and annual program and financial reports. Annually, MaMoni and USAID develop targets for process indicators, and program reports evaluate progress toward these targets. On many of these indicators, MaMoni has achieved or exceeded its targets. However, as previously mentioned, the evaluation indicators were not quantified in the monitoring and evaluation plan, so the team cannot report whether the project goals have been achieved. While the team was very impressed by the quality of MaMoni senior management, as well as that of the partner NGOs, it was hard not to notice the limited number of female senior managers. The team was also struck by the high number of staff at the various levels of the project, particularly in Sylhet, where phase out began in September 2011 (Annex VIII: Sylhet Phase-Out Process). As discussed in the recommendations section, there should be further thought about the number of staff required at different phases of the project. 32 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 33 VI. CHALLENGES AND LESSONS LEARNED CHALLENGES  BCC: MaMoni relied heavily on counseling by the FWAs and FWVs to disseminate the BCC to women. However, these workers did not have sufficient time to provide the large volume of messages (60) to women. As a result, knowledge of danger signs was mixed and knowledge of birth planning was low. The team feels that the CVs could have been used more to provide messages. In addition, a more balanced mix of communication channels, with reinforcing messages, would have provided more opportunities for audiences to be exposed to the messages.  Home-based counseling and services: Working with the FWAs to expand counseling and home-based services has not proven to be effective. Data indicate that few women have received information or services, except PNC, from FWAs. MaMoni will need to investigate the feasibility of providing information and services at the household level versus the community level.  QOC: There is concern that the increased demand and high vacancies will make it difficult to maintain good quality, even with the temporary MaMoni staff. The SBM-R approach was introduced, but it has not been implemented. This is a key area that needs to be addressed.  High vacancy rates: Vacancy among key service providers and managers continues to be a major challenge. MaMoni has been successful in temporarily filling some FWV and nurse positions but has not been able to hire any MOs, despite many efforts.  Sustainability of efforts by MOH&FW: One of the key challenges is how to sustain the efforts that have been undertaken jointly with the MOH&FW can be sustained as MaMoni transitions and phases out. Of particular concern is whether the MOH&FW will be able to 1) maintain the renovated/constructed facilities, 2) hire and retain staff in hard-to-reach areas, 3) maintain ambulances, 4) conduct supervision, 5) ensure QOC, and 6) continue coordination between health and FP directors when staff changes.  Timeliness/reliability of data: MaMoni, working with the MOH&FW, has worked to improve the HMIS. Data are tracked monthly and used in the review meetings. Very little of the facility-level data are being utilized to monitor QOC. In addition, key pieces of data are being collected through operation research studies (e.g., referral, pre-eclampsia) that will be completed after MaMoni ends. Either the data need to be embedded in the project or additional short-feedback loops need to be created so MaMoni and the MOH&FW can use the data for program improvements. LESSONS LEARNED  Phones improved communication. Almost inadvertently, phones played a pivotal role in the project. Trainers used them to supervise health and community workers after the training. Mobile phones were used to coordinate obstetric emergency transport, get help diagnosing birth difficulties, and prepare facilities for arrival. Mothers were given numbers of 34 USAID/BANGLADESH: MAMONI FINAL EVALUATION health providers to call for follow-up information. Action-planning supervision visits were followed up with phone calls. Informal support groups among CVs were created using mobile phones. Micro-planning meeting participants also used phones to problem solve between meetings. MAMA audio and text messages were delivered to pregnant mothers and recently delivered women. Mobile phones have proven to be a cost-effective channel and should be used to complement home visits and facility-based counseling. They should also be systematically built into future interventions.  Linking UP/FWAs/HAs with CAG improved sustainability. It is clear that if the CAGs are valued by the local community they are more likely to continue their efforts.  Renovations of facilities/quarters were key in staff retention and increased use. Staff indicated that improved living quarters and security at the facility were key to their willingness to stay at these facilities. In addition, many women commented that they could not believe how clean the facilities—especially the toilets—were after the renovation. They were more inclined to go to the facilities because of the cleanliness.  Hiring nurses and advocating for task shifting was effective. Initially, hiring nurses was not a main strategy for the project. However, once MaMoni began to realize the difficulties in hiring MOs and the competencies of nurses, the project began to see that this as an important strategy for dealing with some of the key interventions, such as managing sick newborns. Now, MaMoni considers advocating for shifting tasks (such as epidural blocks) to nurses as a key effort.  Provider relationships improved referrals. MaMoni spent a lot of time strengthening the relationship among providers at different levels, through training and supervision. The evaluation team saw that these relationships were quite strong and that cell phones played a key support function in maintaining these relationships and improving referrals.  Coordination between health and FP directorates has been effective. In most districts there is little coordination between the health and FP directorates at any level. MaMoni worked to facilitate coordination, starting at the union level with the micro-planning meetings and then joint meetings at the upazila and district level. This approach greatly improved data consistency and joint problem solving at the various levels. USAID/BANGLADESH: MAMONI FINAL EVALUATION 35 VII.RECOMMENDATIONS RECOMMENDATIONS FOR INTERVENTIONS TO BE SCALED UP The team believes that the following activities have enough evidence to indicate that they are working well and can be replicated in other districts with appropriate adaption. Thus, in new districts it is recommended that USAID undertake the following activities: Conduct a comprehensive assessment, including an HFA, human resources inventory, mapping, and identification of any areas/populations needing customized strategies.  Develop collaboration mechanisms/relationships among key stakeholders.  Build on existing MOH&FW systems and structures (e.g., offices in district facilities).  Create CVs/CAGs to support community mobilization efforts with strong linkages with UP, FWAs, and HAs.  Liaise with UP members and reactivate health committees to support MNH/FP efforts.  Facilitate micro-planning meetings with follow-up meetings at the union, upazila, and district levels.  Support an integrated approach to health systems strengthening, based on the assessment process, which may include some combination of the following efforts: – Critical gap management (e.g., hiring FWVs, nurses, and MOs); – Provision of key supplies, equipment, and medicines; – Renovation/construction of selected facilities, including staff quarters, waste management, and water sources; – Competency-based skills training supported by supervision and on-the-job training to maintain skills; – Referral system that includes a team at the referral facility to ensure prompt treatment of complications; – Quality improvement mechanisms that regularly review service performance; – Data collection that promotes accuracy/reliability and is used to improve performance; – Continued advocacy for operational and policy changes (e.g., implants by FWVs, epidural block by nurses). RECOMMENDATIONS FOR INTERVENTIONS TO BE STRENGTHENED  Develop comprehensive BCC strategy that 1) prioritizes key messages and groups their dissemination in a manner that is easy for people to absorb; 2) utilizes a mix of communication channels, based on cost-effectiveness; 3) explores greater utilization of CVs to undertake systematic BCC dissemination of key messages supported with BCC materials; 36 USAID/BANGLADESH: MAMONI FINAL EVALUATION and 4) explores the use of use of phones as a communication channel (e.g., dissemination by CSBAs, CVs), supported by free airtime.  Develop a transition strategy for the CAGs/CVs that outlines the level of effort needed to start the CAGs and declining level of effort as they become independent.  Conduct an assessment of the current private CSBAs to understand 1)the conditions that create the best environment for them to be effective (e.g., links with UP), 2) the situations where they are most useful (e.g., hard-to-reach areas), 3) their performance and skills, 4) the challenges they have experienced, and 5) additional support that may be required (e.g., other services, business training) for them to be successful and have an acceptable income.  Reduce reliance on FWAs by seriously considering not hiring any additional FWAs and reducing their training. MaMoni should look at other approaches (e.g., CVs, CSBAs) for BCC efforts. In addition, the project should explore other channels (e.g., CCs, pharmacies, depot holders) to improve access to services and supplies that are close to women but not necessarily in their homes.  Explore options to expand support for FP, such as assessing the effectiveness of depot holders, exploring short orientations for pharmacists and traditional doctors on FP counseling and side effects, and ensuring supplies are available at CCs. To address LAPM, MaMoni should explore other options to having more providers offer LAPM (e.g., Marie Stopes) and link with the current efforts of Mayer Hashi and Smiling Sun.  Strengthen QOC by implementing SBM-R. If consensus on the tools and verification criteria cannot be reached, then MaMoni should explore other quality assurance to be implemented in the district. In addition, MaMoni should conduct further review of the referral data from a QOC perspective. RECOMMENDATIONS FOR FUTURE PROGRAMMING These recommendations provide some overall guidance to UASID/Bangladesh in thinking about replicating the MaMoni model in new districts.  Articulate overall health systems strengthening replication model. MaMoni has tested severalapproaches in Habiganj. The program has been implemented in an integrated manner, but many of the elements have not been fully articulated into an overall model. For example, it is clear that CAGs are a key component of the overall model, but it is not clear what type of support they need for how long. This information will be very important when thinking about replicating the model versus implementing the interventions. Sylhet provide some insights, but it has only transitioned the community activities, which were much more mature than those in Habiganj. The team thinks that scaling up these efforts will require a one-year start-up period and at least two or three years of intensive intervention and support, followed by two to three years of facilitation, with significantly reduced level of effort and staffing, to support MOH&FW managers in fully adopting the changes. USAID/BANGLADESH: MAMONI FINAL EVALUATION 37 Efforts in Habiganj are still in the intensive intervention support phase. More thinking is required about how the facilitation phase will be undertaken and how this will be done in other districts.  Clearly articulated phase-out strategy: To ensure sustainability and gain commitments from stakeholders, efforts in new districts need to ensure that all stakeholders understand that this approach is time bound and levels of effort will be reduced with a clear exit strategy. Annual plans can be developed with key benchmarks that articulate how functions will be transitioned to local parties, but sustainability of these efforts should be a key focus of the project.  Incorporate QOC and health system indicators: This is a health systems strengthening project, but all the indicators focus on changes in beneficiary behaviors and service use. It would be useful to include changes in provider knowledge, attitudes, and practices, since providers are also key beneficiaries. QOC indicators also need to be added. 38 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 39 ANNEX I: SCOPE OF WORK I. B3-002 USAID/BANGLADESH: FINAL EVALUATION OF THE MAMONI: INTEGRATED SAFE MOTHERHOOD, NEWBORN CARE AND FAMILY PLANNING PROJECT Contract: Global Health Technical Assistance Bridge 3 Project (GH Tech) II. PERFORMANCE PERIOD Evaluation preparations should begin as soon as possible depending on the availability of the selected consultants. Work is to be carried out over a period of approximately eight weeks by the team followed by four weeks for editing and formatting, beginning on or about (o/a) April 15, 2013, with final report and close out concluding by August 21, 2013. III. FUNDING SOURCE Mission funded IV. PURPOSE OF ASSIGNMENT This performance evaluation comes toward the end of the MaMoni project. It is a final evaluation whose objectives are to:  Review, analyze and evaluate the effectiveness of, the MaMoni project in achieving program objectives and contributing to USAID/Bangladesh’s efforts to improve maternal/neonatal health and family planning outcomes in the project area.  Evaluate major constraints in achieving expected project results.  Provide specific recommendations and lessons learned on strategies and approaches that the Mission should consider to design the follow-on program. The primary users of the evaluation findings are the Mission, the local implementer (Save the Children and sub-recipients), and the Leader Project MCHIP. The Government of Bangladesh (GOB) is also a secondary user of the findings of the evaluation. MaMoni works closely with the GOB in strengthening the local service delivery structure and management capacity of local-level health managers. The Mission will continue health systems strengthening activities and supporting low-performing districts to improve service delivery systems. This MaMoni evaluation will provide information to the Mission on the lessons learned and challenges faced by the project, which will support future design activities aimed at strengthening health systems. V. PROJECT DESCRIPTION MaMoni Project (Integrated Safe Motherhood, Newborn Care and Family Planning): The MaMoni Project (August 3, 2009–January 31, 2014) in Bangladesh is an associate award (Associate Cooperative Agreement No. 388-A-00-09-0104-00) supported by USAID/Bangladesh, funded through JHPIEGO, and implemented by Save the Children. USAID’s investment in the project is $13,493,991. The project aims to increase and sustain the practice 40 USAID/BANGLADESH: MAMONI FINAL EVALUATION of high-impact maternal and neonatal behaviors and the use of high-impact services during the antenatal, childbirth, and postnatal periods, including increased use of modern family planning (FP) methods. VI. BACKGROUND Status of maternal and neonatal mortality, and contraceptive use in Bangladesh: Recently released data from the Bangladesh Maternal Mortality and Health Care Survey (BMMHS) 2010 demonstrate significant reductions in maternal mortality. Maternal mortality declined from 322 in 2001 to 194 in 2010, a 40% decline in nine years. Building on this success, the GOB aims to further decrease maternal mortality by addressing two main causes of maternal death—postpartum hemorrhage and eclampsia. The results of the BMMHS 2010 demonstrated that hemorrhage (31%) and eclampsia (20%) are the dominant direct obstetric causes of death, together responsible for more than half of the Maternal Mortality Rate (MMR). According to the Bangladesh Demographic and Health Survey (BDHS) 2011, the under-5 mortality and infant mortality rates are 53 and 43 deaths per 1,000 LBs, respectively. Deaths in the neonatal period account for 60% of all under-5 deaths. Under-5 mortality is the highest is Sylhet, with 83 per 1,000 births. Following an impressive decline in fertility in the late 1970s and 1980s from 6.3 to 3.4 births per woman, reductions in fertility in Bangladesh began to plateau, causing concern among policymakers. Multiple sources of data show that the TFR stalled at 3.3 during the 1990s and then resumed its decline during the early 2000s. According to the 2011 BDHS, the TFR for women age 15–49 is 2.3, and 61% of currently married women in Bangladesh are currently using a contraceptive method. MaMoni Project: The goal of the MaMoni project is to improve maternal and neonatal outcomes in Sylhet and Habiganj. The project was designed to directly support USAID/Bangladesh’s Development Objective 3: Improving People’s Health in Bangladesh, and under the Investing in People Objective, Health Project Area of the U.S. Foreign Assistance Framework. The MaMoni project contributes to intermediate result (IR) 3.1: Increased Use of Effective Family Planning and Reproductive Health Services under USAID/Bangladesh’s Country Development Cooperation Strategy. The project seeks to increase the practice of healthy maternal and neonatal behaviors, including FP, in a sustainable and scalable manner. Healthy timing and spacing of pregnancy have been shown to reduce both maternal and neonatal mortality. While there is no explicit FP Millennium Development Goal (MDG), the World Bank agreed to support Bangladesh’s efforts to reduce maternal and infant mortality by increasing the number of contraceptive users. MaMoni will contribute to that goal by increasing knowledge of and access to modern methods of contraception to married women of reproductive age (MWRA) in Sylhet and Habiganj. MaMoni builds on the four-year ACCESS project (2006–2009), which focused on maternal and newborn health in seven upazilas of Sylhet. MaMoni introduces FP, hand washing, and newborn sepsis management into the intervention package, and expands geographic reach to cover the Habiganj district. The project strategy shifts toward strengthening public sector systems with a facilitative role of the NGOs at the district level and below. The overall objective of MaMoni is “increased and sustained practice of high impact maternal and neonatal behaviors and use of high-impact services during the antenatal, childbirth, and postnatal periods, including increased use of modern family planning methods.” USAID/BANGLADESH: MAMONI FINAL EVALUATION 41 Specifically, MaMoni seeks to: 1. Increase knowledge, skills, and practices of healthy maternal and neonatal behaviors in the home. 2. Increase appropriate and timely utilization of home- and facility-based essential maternal and neonatal health (MNH) and FP services. 3. Increase acceptance of FP methods and advance understanding of FP as a preventive health intervention for mothers and newborns. 4. Improve key systems for effective service delivery, community mobilization, and advocacy. 5. Mobilize community action, support, and demand for the practice of healthy MNH behaviors. 6. Increase key stakeholder leadership, commitment, and action for these MNH approaches. Achievement of the project’s overall objective depends on the successful attainment of the following intermediate results: IR1: Improved knowledge of MNH/FP behaviors, services, and service delivery points IR 2: Increased availability and quality of high-impact facility-based and community-based MNH/FP services IR 3: Strengthened GOB and NGO partner capacity and systems for effective delivery of high￾impact MNH-FP services IR 4: Increased community capacity, action, and demand for the practice of high-impact MNH/FP behaviors IR 5: Increased key stakeholder leadership, commitment, and action at all levels for high-impact MNH/FP interventions Table A-1. Expected Life-of -Project Results Expected Life-of-Project Results An increased number of recent mothers who: 1. Received four ANC visits during pregnancy; two doses of tetanus toxoid/ iron folate during pregnancy 2. Sought care for obstetric and/or newborn complications 3. Used clean delivery practices 4. Used a skilled birth attendant/Used a trained birth attendant 5. Received early postnatal visits within 24 hours of childbirth 6. Practiced essential newborn care 7. Practiced LAM correctly/Transitioned from LAM to another FP method 8. Increased CYP IR1: Improved knowledge of MNH/FP behaviors, services, and service delivery points 1. Improved knowledge of danger signs in pregnancy, childbirth, postpartum periods 2. Improved knowledge of appropriate health care facilities to go to for management of complications during pregnancy, childbirth, and the postpartum periods 42 USAID/BANGLADESH: MAMONI FINAL EVALUATION Table A-1. Expected Life-of -Project Results Expected Life-of-Project Results 3. Increased use of birth preparedness plans including gathering materials, saving funds, arranging for emergency transport, and birth attendants for mother and baby 4. Increased # of recent mothers who know to use soap for handwashing prior to breastfeeding, after wiping the baby’s bottom, and after defecating IR 2: Increased availability and quality of high-impact facility-based and community-based MNH/FP services 1. Increased capacity of public health facility staff to provide MNH/FP services 2. Increased capacity of existing NGO/GOB counselors to provide counseling on FP, handwashing, and basic hygiene 3. Increased availability of quality management of newborn sepsis management by village doctors 4. Increased # of pregnant women counseled about birth spacing and postpartum FP, including LAM and fertility return 5. Increased # of pregnant women who received information about early PNC 6. Increased # of postpartum women who received counseling on healthy timing/spacing and return to fertility IR 3: Strengthened GOB and NGO partner capacity and systems for effective delivery of high-impact MNH/FP services 1. Decrease # of target facilities reporting stockouts of key MNH/FP supplies/drugs 2. Increased # of pregnant women and eligible couples identified through MNH/FP home visits 3. Increased # of obstetric/newborn complications recognized at an early stage and referred by household counselors 4. Increased # of postpartum women/newborns who received a home visit from a counselor within 24 hours of birth IR 4: Increased community capacity, action, and demand for the practice of high-impact MNH/FP behaviors 1. Increased #of community action groups with emergency transport and/or emergency finance plans in place 2. Increased # of recent mothers who experienced a pregnancy-related complication, or whose newborns experienced a complication, who used the emergency transport and/or emergency finance schemes IR 5: Increased key stakeholder leadership, commitment, and action at all levels for high-impact MNH/FP interventions 1. National policies and strategies revised to promote high-impact MNH/FP interventions 2. National- and district-level MNH/FP advocacy strategies implemented VII. EVALUATION QUESTIONS The evaluation questions are provided below. The evaluation team should review, analyze, and evaluate the MaMoni program along the following criteria, and, where applicable, identify opportunities and recommendations for improvement. In answering these questions, the evaluation team should assess the performance of its implementing partner(s). USAID/BANGLADESH: MAMONI FINAL EVALUATION 43 Task 1: Evaluate the effectiveness of the MaMoni project in achieving planned objectives. (Estimated level of effort: 70%) Community engagement: What are the successes in community engagement in MaMoni project areas (e.g., male and female participation, engagement of community volunteers)? Service delivery: Have MaMoni interventions been effective in enhancing knowledge on MNH/FP-related issues at the household level? Are there signs of improvement in the proportion of household visits (for babies born at home) by health workers within three days of deliveries? Are there signs of increased uptake of MNH/FP (ANC, postnatal care, deliveries, FP) services at the facilities? Systems strengthening: How effectively does MaMoni's micro-planning system link MOH workers and community volunteers to fill gaps in coverage and information? Are MaMoni interventions strengthening capacity of the MOH including MIS, QA, and referral systems? GOB relationship: What successes has MaMoni achieved in engaging local government authorities (UP) in promotion and oversight roles? Task 2: Identify constraints to achieving results. (Estimated level of effort: 20%) 1. Which components have been the most difficult to make progress on and not possible to replicate in a scaled-up situation? 2. What have been some unforeseen challenges to the project’s success and how does MaMoni plan to address them? Task 3: Recommendations for future planning. (Estimated level of effort: 10%) 1. Which of the program components can be used for nationwide scale-up under the health systems strengthening model? 2. In addition, the evaluation should identify any project management issues that adversely impact performance of the project. VIII. EVALUATION DESIGN AND METHODOLOGY The evaluation team will use a variety of methods for collecting information. These methods, to the maximum extent possible, will ensure that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions. Findings should be specific, concise, and supported by quantitative and qualitative information.30 The evaluation team should decide on specific methodologies during the Washington-based team planning meeting before traveling to Bangladesh, and finalize the selection of methodologies during the first evaluation team meetings in-country. The evaluation team should also meet with the MCHIP HQ team (or the Bangladesh country support team at MCHIP HQ) before coming to Bangladesh. The following essential elements should be included in the methodology as well as any additional methods proposed by the evaluation team: 30 These methodological principles reflect the requirements set forth within the USAID Evaluation Policy. 44 USAID/BANGLADESH: MAMONI FINAL EVALUATION Review of background documentation: USAID/Bangladesh OPHNE will provide the evaluation team leader with a core list and/or copies of the agreement, reports of recent relevant evaluations, and other key documentation before the evaluation begins. The evaluation team leader will be responsible for expanding this background documentation as appropriate, and for reviewing, prioritizing, and distributing it to other evaluation team members for their review. All evaluation team members will review relevant documentation before their initial team meetings. Key informant interviews: The evaluation team will conduct qualitative, in-depth interviews with key stakeholders and partners. Whenever possible, the evaluation team should conduct face-to-face interviews with informants. When it is not possible to meet with stakeholders in person, telephone interviews should be conducted. The evaluation team will have interviews with the following (not exhaustive):  Relevant USAID offices and other U.S. Government offices in Bangladesh  MaMoni implementing partners at both HQ and field level  Other donors who are making financial contribution to the project  USAID partners who have collaboration with MaMoni (EngenderHealth, SSFP, and SMC)  MCHIP partners, such as HFS, MAMA, WRA  Stakeholders (beneficiaries, community members, etc.)  Key GOB representatives at both national and local levels  Donors and project managers of UN MNH projects  Staff from relevant implementing organizations  Other key stakeholders, e.g. professional associations and universities Site visits: Evaluation team members, as appropriate, will visit selected project sites in two districts (Sylhet and Habiganj). The evaluation team is expected to review all available data (quantitative and qualitative) at the field level. From existing documents and interviews they will determine the effectiveness of the program. Data must be disaggregated—by sex, age, geographical region, education level, etc. IX. EXISTING DATA Document review: The evaluation team should consult a broad range of background documents in addition to project documents provided by USAID/Bangladesh. USAID and the MaMoni project will provide the evaluation team with a package of briefing materials, including the 1) program description for the MaMoni project; 2) M&E plan of MaMoni; 3) project quarterly and annual reports, workplans, and management reviews developed as part of routine monitoring; 4) MaMoni project benchmark survey report; 5) facility assessment report of Habiganj; 6) HNPSP (revised PIP) and HPNSDP; 7) MOH&FW HR Strategy; 8) National Neonatal Health Strategies and Guidelines (NNHS); 9) project intervention documents, process documents, internal review report of 2010, and training materials and guidelines; 10) BDHS 2007 and 2011 reports; 11) Bangladesh Maternal Mortality and Health Care Survey 2010; 12) USAID/BANGLADESH: MAMONI FINAL EVALUATION 45 Addressing Gaps in the Existing USAID/Bangladesh Health Program 2008 (THIS DOCUMENT IS INTERNAL TO USAID); 13) The GHI Bangladesh Strategy; 14) USAID/Bangladesh Country Development Cooperation Strategy 2011–16 (draft as cleared by the front office); 15) BEST Action Plan for Bangladesh; and 16) USAID Evaluation Policy. X. TEAM COMPOSITION, SKILLS, AND LEVEL OF EFFORT (LOE) USAID/Bangladesh would like to engage the services of a five person evaluation team. The evaluation team should include three international consultants, one local consultant and one logistics coordinator. The former should include specialists with the following areas of expertise: family planning and maternal health, conducting evaluations, behavior change communication, sustainability, and health systems. The one local consultant should have an excellent understanding of the Bangladesh public health system and be fluent in Bangla. The evaluation team leader should be an evaluation expert. The evaluation team leader should be an independent consultant, but one of the technical specialists could be USAID/Washington Global Health staff. Team leader/technical specialist: Should be an independent consultant and have an MPH or related postgraduate degree in public health. S/he should have at least 10 years senior-level experience working in health systems programs in a developing country. S/he should have extensive experience in conducting evaluations. Excellent oral and written communication skills are required. The evaluation team leader should also have experience in leading evaluation teams and preparing high-quality documents. This specialist should have wide experience or familiarity with USAID-funded maternal and reproductive health programs and should have a good understanding of health systems in South Asia, preferably in Bangladesh. S/he should also have a good understanding of project administration, financing, and management. The evaluation team leader will take specific responsibility for assessing and analyzing the project’s progress toward quantitative targets, performance, and benefits/impact of the strategies. The team leader will also look at the potential sustainability of MaMoni project approaches and activities. The evaluation team leader will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables in consultation with the other members of the team. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Bangladesh team and key partners. The evaluation team leader will submit the draft report, present the report, and—after incorporating USAID/Bangladesh staff comments—submit the final draft report to USAID/Bangladesh within the prescribed timeline. Maternal and neonatal health program expert: The MNH program expert will have at least 7–10 years of experience in management of, or consulting on, MNH programs. S/he should have a proven background and experience in maternal and neonatal health and a strong understanding of the challenges Bangladesh faces in the MNH sector. S/he should also have a good understanding of the relevant national programs in MNCH and RH, including the public and private sector. The MNH expert will be responsible for assessing the ability of the project to achieve outcomes according to the project objectives of maternal health, neonatal health, and family planning, and provide technical leadership in those areas. The consultant will participate in 46 USAID/BANGLADESH: MAMONI FINAL EVALUATION evaluation team meetings, key informant interviews, group meetings, site visits, and draft the sections of the report relevant to his/her expertise and role on the evaluation team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or other stakeholders. S/he will submit his/her contributions to the evaluation team leader within the prescribed timeline. Community mobilization and behavior change communication specialist: This specialist should have wide experience in implementation of behavior change communication and community mobilization programs in the areas of maternal health, neonatal health, and family planning. S/he should have a postgraduate degree in health promotion sciences or a related field with a minimum of 5–10 years’ experience working with USAID-supported behavior change and community mobilization programs in developing countries. S/he will analyze MaMoni project behavior change and community mobilization interventions and assess the effectiveness and appropriateness of the approaches adopted by the project to improve knowledge, health-seeking behavior, and health outcomes. S/he will also assess the technical focus of BCC activities and community action group (CAG) activities, and whether they are the appropriate mix and topics for intervention communities. The BCC specialist will participate in evaluation team meetings, key informant interviews, group meetings, and site visits, as well as draft the sections of the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/ Bangladesh or other stakeholders. S/he will submit his/her contributions to the evaluation team leader within the prescribed timeline. Host country national health expert: The host country national health expert will serve under the evaluation team leader. S/he should have at least 10 years of experience working in the field of maternal & neonatal health and family planning and have through knowledge of the national health and population program. Duties will be determined in consultation with the evaluation team leader. The host country national will participate in team meetings, key informant interviews, group meetings, and site visits, as well as contribute in drafting the notes for the report relevant to his/her expertise and role on the evaluation team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or other stakeholders. S/he will communicate with the evaluation team leader and other consultants to produce written notes to incorporate in the report as required, addressing comments and feedbacks from USAID. S/he is required to make his/her contributions to the evaluation team leader within the timeline. Local logistics coordinator: GH Tech will also hire a logistics coordinator who will serve under the team leader. Duties will be determined in consultation with the team leader, but are likely to include providing translation services as necessary for team leader; arranging logistics for the team, and assisting team leader as directed in all aspects of completing evaluation deliverables. An illustrative table of the LOE is found below. Dates may be modified based on availability of consultants and key stakeholders, and amount of time needed for field work. USAID/BANGLADESH: MAMONI FINAL EVALUATION 47 Task Deliverable Task LOE Individual LOE Team Leader Technical Specialist Local Consultant Logistics Coordinator Planning for team arrival. 6 6 Review background documents and remote preparation work. 4 4 3 3 1 Travel to Washington, D.C. 1 1 1 Evaluation team planning –D.C. 1 1 1 Meeting with MCHIP HQ team. 1 1 1 Travel to Bangladesh. 2 2 2 Evaluation team planning – Dhaka. 1 1 1 1 1 Meeting with USAID/Bangladesh and prepare for field work. 1 1 1 1 Information and data collection. 12 12 12 12 10 Discussion, analysis, and draft evaluation report in-country, including USAID. 5 5 5 5 Debrief meetings with USAID and the Mission, implementing partner, key stakeholders. 1 1 1 1 Depart Bangladesh/travel home. 2 2 2 USAID and partners provide comments on draft report. 8 Evaluation team revises draft report and submits to USAID (remotely). 4 4 2 1 USAID completes review and provides feedback or approval. 5 Evaluation team revises draft report and submits final to USAID. 2 2 1 1 USAID completes review and signs off on final draft. 2 GH Tech edits/formats report and submits final report to USAID. 30 Total Estimated LOE 37 days 33 days (2 people) 25 days 18 days A six-day work week is approved while in-country. XI. CONFLICT OF INTEREST All evaluation team members will provide a signed statement attesting to a lack of conflicts of interest, or describing an existing conflict of interest relative to the project being evaluated. GH Tech will provide the conflict of interest forms. XII. LOGISTICS GH Tech will be responsible for all international travel and consultant logistics. 48 USAID/BANGLADESH: MAMONI FINAL EVALUATION Funding and logistical support: The proposed evaluation will be funded through GH Tech using Mission field support funds. GH Tech will provide technical and administrative support, including identification and fielding appropriate consultants. GH Tech will be responsible for all off-shore and in-country logistical support. The logistics coordinator will assist the evaluation team in making all logistical arrangements, including the vehicle arrangements for travel within and outside Dhaka. The evaluation team should also make their own arrangements for space for team meetings, and equipment support for producing the report. There will be no logistic support from the Mission. Scheduling: The evaluation is planned to commence between April 1 and August 21, 2013. The evaluation team will submit a workplan as part of the evaluation methodology proposal with a detailed timeline. Pre-departure arrangements should include travel approval, airline tickets, visa, lodging, work facility and vehicle transport arrangements, dates for meetings with USAID/Bangladesh OPHNE staff and key contacts pre-arranged; in-country travel agenda and accommodations. The following represents a rough timeline. XIII. DELIVERABLES AND PRODUCTS The evaluation team shall be responsible for the following deliverables: Team planning meetings: The evaluation team leader and available team members will have a one-day team planning meeting in Washington, D.C. They will also meet with appropriate USAID Global Health officials in Washington, D.C, as suggested by OPHNE/Bangladesh and should begin planning the evaluation methodology during these initial meetings. The full evaluation team will meet upon arrival in Bangladesh (one day) to finalize plans. The team planning meetings are essential in organizing the team’s efforts. During the meetings, the evaluation team should review and discuss the SOW in its entirety, develop the methodology, clarify evaluation team members’ roles and responsibilities, draft the workplan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments and to prepare for the in-brief with USAID/Bangladesh. The outcome of this meeting will be a detailed workplan for the evaluation, including milestones and deliverables with due dates clearly established. Initial team briefing meetings with OPHNE: The evaluation team will have an initial meeting with OPHNE officials in Bangladesh to go over the SOW and discuss elements of the evaluation. During this meeting they will share an outline and explanation of the design and elements of the evaluation, and receive feedback from OPHNE. The full team and/or members will have follow-up meetings with specific OPHNE staff at the outset, and will remain available for consultation throughout the whole process as appropriate. Workplan: During the team planning meeting, the evaluation team will prepare a detailed workplan, which will include the methodologies to be used in the evaluation. The workplan will be submitted to the AOTR at USAID/Bangladesh for approval no later than the third day of work starting from the arrival date of the team in the country. Methodology plan: A written methodology plan (evaluation design/operational workplan) will be prepared by the evaluation team and submitted to USAID along with the workplan and within the same timeframe (third day). After receiving the workplan and methodology, the AOTR will concur/suggest modifications, and the team will start working accordingly. USAID/BANGLADESH: MAMONI FINAL EVALUATION 49 Data collection instruments: Development and submission of data collection instruments to USAID/Bangladesh during the design phase. Regular updates: The evaluation team leader (or his/her delegate) will brief the AOTR MaMoni and GH Tech on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or complications during the evaluation must be quickly communicated to USAID/Bangladesh as early as possible to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities for the evaluation should also be discussed with USAID/Bangladesh. Debriefings: The evaluation team will debrief OPHNE on their findings, conclusions, and recommendations before leaving Bangladesh using a PowerPoint presentation and any briefing materials required. OPHNE will provide feedback during the briefing meeting. The evaluation team will debrief other USAID offices, the U.S. Government, implementing partners, and/or GOB officials as directed by USAID/Bangladesh (see Schedule/LOE Table). Draft evaluation report: The evaluation team will provide OPHNE with a draft report that includes all the components of the evaluation report prior to their departure from the country. OPHNE will provide comments on the draft report to the evaluation team within eight working days of receiving the draft report. Final report: The evaluation team will submit a final report that incorporates the Mission’s comments and suggestions no later than four days after USAID/Bangladesh provides written comments on the evaluation team’s draft report. The final report should meet the following criteria to ensure the quality of the report: 1. The evaluation report should represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not, and why. 2. Evaluation reports shall address all evaluation questions included in the scope of work. 3. The evaluation report should include the scope of work as an annex. All modifications to the scope of work—whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the technical officer. 4. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides, will be included in an annex in the final report. 5. Evaluation findings will assess outcomes and impact on males and females. 6. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). 7. Evaluation findings should be presented as analyzed facts, evidence, and data, and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. 8. Sources of information need to be properly identified and listed in an annex. 50 USAID/BANGLADESH: MAMONI FINAL EVALUATION 9. Recommendations need to be supported by a specific set of findings. 10. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. The format of the final evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, lessons learned, and recommendations. The report should include, in the annex, any dissenting views by any evaluation team member or by USAID on any of the findings or recommendations. The report should not exceed 30 pages, excluding annexes. The report will be submitted in English, electronically. The report will be disseminated within USAID. A second version of this report, excluding any potentially procurement-sensitive information, will be edited, formatted, and finalized by GH Tech before being submitted (also electronically, in English) to Development Experience Clearinghouse (DEC) for dissemination among implementing partners and stakeholders. Finalizing the report takes about 30 days from receiving sign-off by the Mission on the final draft. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format, (2) organized and fully documented for use by those not fully familiar with the project or the evaluation, and (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data could be provided to the AOR. The total pages excluding annexes should not be more than 30 pages. The report should be structured as follows: Table of contents Acronyms Executive summary—Concisely state the most salient findings and recommendations. (2pp) Introduction—Purpose, audience, synopsis of the task. (1pp) Background—Brief overview of MaMoni project in Bangladesh, USAID program strategy and activities implemented in response to the problem, brief description of MCHIP and Save the Children, purpose of the evaluation. (2–3pp) Methodology—Describe evaluation methods, including constraints and gaps. (2pp) Findings and conclusions—Describe and analyze findings for each objective area using graphs and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcome. (12–15pp) Recommendations—Prioritized for each objective area; should be separate from conclusions and be supported by clearly defined set of findings and conclusions. (3–4pp) Lessons learned—Provide a brief of key technical and/or administrative lessons that could be used for future project or relevant program designs. (2–3pp) Future directions—(1–2pp) Annexes—To include statement of work, documents reviewed, bibliographical documentation, evaluation methods, data generated from the evaluation, tools used, interview lists, meetings, USAID/BANGLADESH: MAMONI FINAL EVALUATION 51 focus group discussions, surveys, and tables. Annexes should be succinct, pertinent, and readable. Should also include, if necessary, a statement of differences regarding significant unresolved differences of opinion by funders, implementers, or members of the evaluation team on any of the findings or recommendations. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins one inch top/bottom and left/right. XIII. Relationships and Responsibilities GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: recruit and hire the evaluation team and make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID/Bangladesh will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before In-Country Work  SOW. Respond to queries about the SOW and/or the assignment at large.  Consultant conflict of interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment.  Local consultants. Assist with identification of potential local consultants, including contact information.  Site visit preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. Assist with invitations for the stakeholder presentation.  Lodgings and travel. Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) and, if necessary, identify a person to assist with logistics (i.e., visa letters of invitation, etc.). During In-Country Work  Mission point of contact. Throughout the in-country work, ensure constant availability of the point of contact and provide technical leadership and direction for the team’s work.  Meeting space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space if available, or other known office/hotel meeting space).  Meeting arrangements. Assist the team in arranging and coordinating meetings with stakeholders. 52 USAID/BANGLADESH: MAMONI FINAL EVALUATION  Facilitate contact with implementing partners. Introduce the evaluation team to implementing partners and other stakeholders and, where applicable and appropriate, prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After In-Country Work Timely reviews. Provide timely review of draft/final reports and approval of deliverables. USAID/BANGLADESH: MAMONI FINAL EVALUATION 53 ANNEX II: EVALUATION DESIGN MATRIX Evaluation Question Data Source (Organization/Individual) Data Collection Methods What are the successes in community engagement?  Quarterly and semi-annual reports  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents Have MaMoni interventions been effective in enhancing knowledge on MNH/FP-related issues at the household level?  Quarterly and semi-annual reports  Baseline/final/midline survey  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents Are there signs of improvement in the proportion of household visits (for babies born at home) by health workers within three days of deliveries?  Quarterly and semi-annual reports  Baseline/final/midline survey  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents Are there signs of increased uptake of MNH-FP services at the facilities?  Quarterly and semi-annual Reports  Baseline/Final. Midline Survey  Project MIS  Field Visit to Habiganj and Sylhet  Interviews with Key Stakeholders  Review of Project documents Are MaMoni interventions strengthening capacity of MOH&FW, including MIS, QOC, and referral system?  Quarterly and semi-annual reports  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents How effectively does MaMoni's microplanning system link MOH workers and community volunteers to fill gaps in coverage and information?  Quarterly and semi-annual reports  Baseline survey  Final/midline survey  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents What successes has MaMoni achieved in engaging local government (UP) in promotion and oversight roles?  Quarterly and semi-annual reports  Baseline survey  Final/midline survey  Project MIS  Field visits to Habiganj and Sylhet  Interviews with key stakeholders  Review of project documents 54 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 55 ANNEX III: DOCUMENTS REVIEWED Alam, M.M., Ubaidur Rob, Md. Noorunnabi Talukder, Farhana Akter. “Performance-based Incentive for Improving Quality Maternal Health Care Services in Bangladesh.” New York: Population Council, 2013. Bhattacharjeem, Anuradha, et al. Addressing Gaps in the Existing USAID/Bangladesh Health Program. Washington, DC: Global Health Technical Assistance Project, December 2008. Bangladesh Demographic and Health Survey. 2011 Bangladesh Maternal Mortality and Health Care Survey. 2010 DNet. MAMA Formative Research Report Government of Bangladesh. Health Population and Nutrition Sector Plan, PIP and HR Strategy (2011–2016). Government of Bangladesh. National Communication Strategy. November 2008. Government of Bangladesh. National Neonatal Health Strategies and Guidelines. 2009. Government of Bangladesh. National Strategy for Maternal Health. October 2001. Government of Bangladesh, JICA, and WHO. Narsingdi Model in Bangladesh, Saving lives of Mothers and Children through Partnership and Capacity Development. Safe Motherhood Promotion Project, 2011. Government of Bangladesh and U.S. Agency for International Development. Mayer Hashi Improving uptake of LAPM in the FP program, Bangladesh National Strategy. 2011–2016. ICDDR,B. Habiganj Baseline Report. 2010. ICDDR,B. Sylhet Baseline Report. 2010. ICDDR,B. Sylhet Preliminary Data Tables Endline Survey. 2012. Luoma, Marc, Jobayda Fathema, Jamil H. Chowdhury, and Hong Wang. Incentives to Improve Retention and Performance of Public Sector Doctors and Nurses in Bangladesh. Bethesda: Abt Associates, December 2009. MaMoni. Community Mobilization Strategy. Save the Children. MaMoni. Health Facility Assessment Report of Habiganj. 2011. MaMoni. Project Benchmark Survey Report. MaMoni. FY12 Annual Report. Save the Children. April 30, 2012. MaMoni. FY12 Quarterly Report: April–June 2012. Save the Children. MaMoni. FY12 Quarterly Report: October–December 2012. Save the Children. MaMoni. FY12 Semi-Annual Report. Save the Children. October 31, 2012. 56 USAID/BANGLADESH: MAMONI FINAL EVALUATION MaMoni. FY11 Annual Report: April 30, 2012. Save the Children. MaMoni. FY11 Quarterly Report: April–June 2011. Save the Children. MaMoni. FY11 Quarterly Report: October–December 2011. Save the Children. MaMoni. FY11 Semi-Annual Report: October 31, 2012. Save the Children. MaMoni. FY10 Annual Report: April 30, 2012. Save the Children. MaMoni. FY10 Quarterly Report: April–June 2011. Save the Children. MaMoni. FY10 Quarterly Report: October–December 2011. Save the Children. MaMoni. FY10 Semi-Annual Report: October 31, 2012. Save the Children. Smith, Jeffrey et al. “Misoprostol for Postpartum Hemorrhage Prevention at Home Birth: An Integrative Review of Global Implementation Experience to Date.” BMC Pregnancy and Childbirth 2013, 13:44. U.S. Agency for International Development. Addressing Gaps in the Existing Health Programs Program 2008 (Internal Document). U.S. Agency for International Development. BEST Action Plan for Bangladesh. U.S. Agency for International Development. Checklist for Assessing Evaluation. U.S. Agency for International Development. Country Development Strategy (2011–2016). U.S. Agency for International Development. Evaluation Policy: Learning From Experience. Washington, DC: U.S. Agency for International Development, January 2011. U.S. Agency for International Development. Policy on Gender-Equality and Female Empowerment. Washington, DC: U.S. Agency for International Development, March 2012. World Health Organization. “Optimizing Health Worker Roles for Maternal and Newborn Health.” Available at http://apps.who.int/iris/bitstream/10665/77764/1/9789241504843_eng.pdf. POWER POINT PRESENTATIONS “MCHIP Overview.” Presented by Koki Agrawal. May 1, 2012. “MaMoni Overview.” Presented by Koki Agrawal. May 1, 2012. “MaMoni Overview.” Presented by Dr. Ishtiaq Mannan. May 5, 2012. “MaMoni Community Mobilization Strategy.” Presented by Dr. Sabbir Ahmed. May 7, 2012. “MaMoni Family Planning Presentation.” Presented by Dr. Sabbir Ahmed. May 7, 2012. “MaMoni Community Micro-planning Meeting Strategy.” Presented by Imteaz Mannan. May 8, 2012. “MaMoni Sylhet Phase-out Strategy.” Presented by Imteaz Mannan. May 8, 2012. “MaMoni Add-on Activities.” Presented by Imteaz Mannan. May 8, 2012. USAID/BANGLADESH: MAMONI FINAL EVALUATION 57 ANNEX IV: STAKEHOLDER QUESTIONS QUESTIONS FOR SERVICE PROVIDERS/FACILITIES 1. Have you received any training by the MaMoni project? In what topics? Has this helped you perform your job better? 2. Do you think there are enough health providers here to provide quality services? 3. Are you willing to stay in this facility and continue to provide services? 4. What services does this facility provide? 5. Did you receive any equipment or supplies? Are you missing any equipment, drugs, or supplies? 6. Have you seen any changes in women/newborns coming for services? 7. If women or newborns have any complications, where do you refer them? Do you have transportation that they can access? QUESTIONS FOR WOMEN 1. How did you first hear about ways you could get help after becoming pregnant? 2. Who helped you when you were pregnant, during delivery, and during postpartum? 3. How did they help you when you were pregnant during delivery and during postpartum? 4. What services (# of ANC visits) and medicine ( IFA, TT, Mistropol) did you get during your pregnancy? 5. Where did you deliver (facility, home), and who helped you (TBA, CSBA, FWV, MD)? 6. Who helped you with your newborn (e.g., breastfeeding, delayed bathing, use of clean blade)? 7. What do you know about the importance of handwashing? When should you wash your hands? Where did you learn about handwashing? 8. What do you know about CAGs/CVs? 9. Describe posters, billboards, slip charts, or other support materials you might have seen on pregnancy, childbirth, family planning, and care of the newborn? 10. What danger signals do you know of potential problems with a pregnancy and newborn? 11. Where would you go if you or your newborn had a complication? How could you access these services? 12. What do you know about getting access to FP in your community/facility? 13. What have you learnt about breastfeeding and introducing weaning foods? 14. What are mothers supposed to do right after their baby is born? 58 USAID/BANGLADESH: MAMONI FINAL EVALUATION 15. If someone came to meet you in your house during or after your pregnancy, who were they and what did they have to say? 16. Have you ever heard of mothers getting messages about health and pregnancy through their mobile phones? If so, what were those messages about? 17. What is the best way to communicate messages about pregnancy, newborn care, and FP to women and men? 18. What mass media is best for communicating with women and men? QUESTIONS FOR PRIVATE CSBAS 1. What is the education level required for CSBA? Was it hard to find women from these districts with that educational level? 2. How often are the CSBAs supervised during their three-month field practicum? 3. How will you introduce yourself to the community when you return? 4. Do you know what other providers exist in your area (e.g., village doctors, pharmacists)? 5. How will you determine how much to charge women? 6. What skills have you found the most challenging in your training? 7. What type of support would you like after your training? QUESTIONS FOR POLICYMAKERS 1. How long have you worked in this post? During this time, what type of involvement have you had with the MaMoni project? 2. Have you worked with other NGO programs like MaMoni that support MNH/FP services in your district? What/where? 3. How has this community mobilization helped or hindered your work? 4. When the project began there were a number of staff vacancies at the community, facility, and supervisory level. What is the vacancy situation now? 5. Have you worked with NGOs to help hire temporary workers or to use their own staff to fill the gaps? How has that worked out? 6. What other ways has the project supported the effort to improve community-level visits? 7. How have the MNH/FP counseling skills of the community-level workers been improved? What other ways could their skills be improved? 8. Has the micro-planning process improved data collection? Relationships with the community? Ability to know what the health providers and community workers are doing? Other benefits or challenges? 9. How do the FWAs follow up recently delivered women to provide PNC visits and help them understand LAM or to help them choose a postpartum FP method? Any way to make this effort more effective? USAID/BANGLADESH: MAMONI FINAL EVALUATION 59 10. How do your staff who supervise the community-level workers know if the FWVs/FWAs are using their new skills or making the postpartum visits? 11. Has the MaMoni’s support for the joint supervision approach helped supervision of the facility/community-level staff? About how often each year are these staff visited by their supervisor or by your office? Are there supervision checklists? 12. Did you or your staff participate in the initial mapping of the district to determine nearest point of service delivery? Have the data been used in community outreach or BCC? 13. Has the CC helped the community-level staff reach women with MNH/FP services? 14. How is information about service use tracked? 15. Has BCC increased in the district since MaMoni began? What types of BCC initiatives have there been? Which do you think are most helpful in encouraging people to adopt MNH/FP? 16. Which BCC initiatives have encouraged husbands to accept MNH/FP either for themselves or their wives? 17. How has the district worked to improve use of LAPM? Other FP methods? 18. Have the facilities experienced stockouts of key drugs (e.g., IFA, FP methods, oxytocin)? 19. What changes and improvements in record-keeping has the MaMoni program supported? 20. Are you able to attend the MaMoni quarterly meetings? Are these meetings helpful in your work? 21. What are the best practices or lessons learned from this project that might be useful to take to other areas in the country? 22. Has your own career been influenced by this project? In what ways? QUESTIONS FOR COMMUNITY GROUPS 1. How did it come about that you formed a community action group? 2. What sort of people are members of the group (age, gender, education level, etc.)? 3. How were those members selected to be in the group? 4. What specific activities is your group involved in (mapping, training, collecting data, etc.)? 5. What was the most important accomplishment of the group? 6. What is the biggest challenge the group faces? 7. How often does the group meet and when was the last time it met? 8. What motivates members to keep active? 9. What are the advantages of linking community leaders with health services? 10. What is micro-planning and what are the advantages of doing it for a community? 60 USAID/BANGLADESH: MAMONI FINAL EVALUATION 11. What is the best way to communicate messages about pregnancy, newborn care, and FP to parents? 12. What mass media is best for communicating with parents? QUESTIONS FOR UP AND LOCAL COMMUNITY 1. How does the MaMoni project meet the community need of MNH/FP services? 2. How effective has the MaMoni project been in increasing and sustaining 1) use of CSBA service for delivery, 2) accepting LAPM as FP choice, 3) demonstrating proper child care behavior for newborns, and 4) increasing CYP for couples? 3. To what extent is the MaMoni project strengthening the capacity of local-level providers like CSBA, FWA, HA, and community volunteers? 4. Has the UP linked to MaMoni project/report sharing and/or community-level monitoring? 5. Does the UP give input for gap identification (services, manpower, and resources) and participating with responsibility for fulfilment; explain the project integration process 6. What are the barriers/challenges of UP to assist the MNH/FP providers and facilities at the community level? 7. What are some new activities/agenda taken into UP action plan as an outcome of MaMoni project involvement? 8. What were the new approaches/actions taken by MaMoni that have been able to mobilize resources available at UP and/or any other LGB partners? 9. To what extent has MaMoni been able to ensure participation of UP leaders and make them accountable for MNH/FP services? 10. Should USAID continue funding this activity at the same level, at a reduced level, or at an enhanced level? If USAID withdrew its funding support, how could you sustain and could the community take over in course of time? Any initiatives/examples of activities you can remember that were taken by UP following the project theme? 11. Can you tell us how the CAG CCSG, CV, and GOB field staff work? 12. Do you know anything about MAMA? 13. What is the financial contribution of UP/standing committee for health and education for this financial year? 14. Which are the activity/activities you take part in under MaMoni project/program? QUESTIONS FOR CC AND CC SUPPORT GROUP 1. What is the population of this CC? What are the services provided? 2. How many times have FWVs come for conducting satellite clinics? 3. Who does delivery of the pregnant mothers? 4. Are there any CSBAs? USAID/BANGLADESH: MAMONI FINAL EVALUATION 61 5. Do you get all supplies for maternal, neonatal, and family planning? 6. Is there any delivery conducted in this CC and, if yes, who does that? 7. Do you know about LAM? 8. Do you know danger signs of pregnancy? 9. Where are complicated mothers referred? 10. Do poor people get free services from this CC? 11. How has MaMoni helped/helping you in providing services? 12. How does CCSG help you? 13. Is there any other committee working for health and family planning? 14. Who does ANC? Do you know about ANC and PNC? 15. Do you know how many ANC visits are required by a pregnant woman? 16. How many members are there in the CCSG, and who are they? 17. What support CCSG does give? 18. Is any financial support given by CCSG, especially in transporting complicated pregnant woman? 19. Does the CCSG meets regularly? QUESTIONS FOR DEPOT HOLDER 1. What type of products do you sell? 2. How much margin do you keep on each sale? 3. Where do you get your supplies? 4. Have you seen any changes in the use patterns in the last few years? QUESTIONS FOR MICRO-PLANNING 1. Who usually attends the micro-planning meetings? How often do the meetings happen? 2. How much support is provided by the MaMoni project? 3. Have the micro-planning meetings changed the way you work? If yes, how? 4. Have you seen any changes in the use patterns in the last few years? 62 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 63 ANNEX V: CALENDAR AND PEOPLE INTERVIEWED MEETINGS IN WASHINGTON, DC, AND DHAKA Date Name Organization 5/1/13 Koki Agrawal, Project Director JHPIEGO/MCHIP Dr. Jeffrey Michael Smith, Maternal Health Team Leader Angela Brasington, Community Mobilization Save the Children/MCHIP Pat Daley, Deputy Project Director Jaime Mungia, Senior Program Officer Rebecca Levine, Senior Program Officer Jennifer Shindeldecker, Program Officer Joseph Degraft-Johnston 5/4/13 Dr. Umme Salma Jahan Meena USAID Mr. Greg Adams, Acting Director, OPHNE Dr. Felicia R. Wilson, Acting Deputy OPHNE, Senior Education advisor Ms. Christean Cole, Program Officer Mr. Jeff de Graffenried, Program Officer Ms. Kanta Jamil, M&E Advisor Ms. Morunga Monda, Program Officer Md. Shahidul Islam, Education Team Leader 5/5/13 Dr. Jebun Nessa Rahman, JHPIEGO Save the Children Dr. Muhibbul Abrar, Manager, M&E Mr. Imteaz Mannan, Program Advisor Dr. Sabbir Ahmed, Deputy Director Dr. Ishtiaq Mannan, COP Dr. Jatan Bhowmick, Senior Manager Coordination Mr. Michael Foley, Director, H&N Mr. Michael McGrath, Country Director, SC 5/6/13 Dr. Shams El Arifeen, Director ICDDRB, Center for Child Health Dr. Sanwarul Bari, Deputy Project Coordinator 5/6/13 Rizwana Rashid Auni, Chief Program Officer DNet Kabita Yasmin, Senior Assistant Director, Outreach and Training 64 USAID/BANGLADESH: MAMONI FINAL EVALUATION Date Name Organization 5/7/13 Dr. M.A. Mannan, MO (clinic) MCWC, Habangj Dr. Md Abdur Rob Mollah, MO (anesthesia)A Alouas Sultana, Nursing Nasima Akhter, SSN Khairun Nessa, FMA Attendant Kanakendu Paul Choudhury, Pharmacist Naznin Akhter, FWV Lipi Das, FWV Lutfunnessa Khatun, FWV Mamataz Begum, FWV Shahanara Begum, FWV Farhanaz Rahman Monia, Private CSBA Khala Rani Nath, Private CSBA Shibpasha, Union Kukia Sultana, Private CSBA Nilufar Iasmin, FWV Lubna Akter, FWV Shibpasha UH&FWC Papia Akter, FWV Tafsir Mia, Chairman Shibpasha UP Asad Miah, #5 Ward UP member Motaher Rahman, #7 Ward member Nurul Amin Miah, #5 Ward member Ekhlas Miah, #9 Ward member Saleh Ahmmed, #3 Ward member Aiub Ali, #8 Ward member Mojahid Miah, #4 Ward member Azizur Rahman Secretary, UP 5/8/13 Dr. Shafiqur Rahman, CS Superintendent, Dr. Md. Arshed Ali, Consultant, Lakhai, Hobigonj District Hospital Dr. Das Gupta, MO 5/8/13 Dr. Md. Mushfiq Murtaza, MO, Facilitator hired by OGSB for CSBA training PCSBA Training Institute 18 PCSBA students 5/8/13 Dr. Md. Jasim Uddin Bhiuya DDFP, Habigonj District Mussammat Kiron Akhter, CHCP Shankarpasha Community Clinic, Raziura Union Dr USAID/BANGLADESH: MAMONI FINAL EVALUATION 65 Date Name Organization 5/9/13 Mr. Narzul Islam Urban Partnership for Poverty Reduction project, UNDP # TBAs 9 Community Volunteers Dr. Md. Abdur Rob Mollah, UFPO, Sadar Hobigonj UFPO Lipika Rani Shaha, Assistant UFPO, Sadar Hobigonj AUFPO Sunil Chandra Paul, SAMCO Poil UH&FWC, Sadar Hobigonj Noor Jahan Begum, FWV Dr. Mohi Uddin Choudhury, Advisor, MaMoni Meeting with MaMoni (FIVDB) staff 5/9/13 PC- MaMoni, FIVDB Meeting with MaMoni (FIVDB) staff Rikta Das UC, MaMoni, FIVDB Md. Mohibul Karim, Technical Officer Md. Harunor Rashid, Technical Officer, M&E 5/10/13 Dr. Md. Mohi Uddin Choudhury, advisor Shimartik Omorpur Union, Balagonj Upazilla, Sylhet District Anima Rani Chanda (Arpana), BRAC Depot Holder/ CV Gitasree Paul, Field Facilitator, Shimartik Md. Jalal Uddin, Upazila Team Leader Shimtack Dr. Mohi Uddin, Choudhury, Advisor MaMoni Kazi Humayun Kabir, Project Coordinator Shimantic Nazrul Islam, MAMA Subscriber Shamima Akhter, Pregnant Woman, MAMA Subscriber Sumi Rani, CHW trained by MAMA Dolly Akhter, Mother, MAMA subscriber 2 Community Male Volunteers 5 NGO/MaMoni FWAs Dr. Subol Chandra Bormon, RMO Jaintapur UHC, Sylhet Dr. Hasina Akter, Assistant Surgeon, Neonatal Ward Dr. Kuheruba, SCANU in charge, MO Dr. Ahmrd Sibli, SCANU, MO Aisha, SCANU Staff Nurse Sangita Rani, SCANU Paramedic Dr. Iqbal Hussain Choudhury, Divisional Director Health Directorate, Sylhet Division 5/10/13 Dr. Quamrul Islam, Civil Surgeon DGHS, Sylhet District 5/11/13 Md. Mazed Ahmed, Chairman Shimantik 66 USAID/BANGLADESH: MAMONI FINAL EVALUATION Date Name Organization Kazi Humayun Kabir, Director Moha. Shamim Ahmed, General Secretary Kazi Muksedur Rahman, Executive Director Md. Pervez Alam, Director Ruhel Kabir, Director, IFSP Friends in Village Development Bangladesh (FIVDB) Zahid Hossain, Director, LEP Samik Shaheed Khan, Associate Director S.A. Wahid, Director Finance and Administration Shirin Akhter, Director PEP Zahin Ahmed, Executive Director Nazrul Islam Manzoor, Program Coordinator, CLP Kutub Uddin, Divisionsal Director, Joint Secretary FP Directorate, Sylhet Division Dr. Jesmin Akhter, Deputy Director, FP FP Directorate, Sylhet District 5/12/13 Dr. Loshan Moonesinghe, FP Specialist UNFPA Dr. Zarina Rahman Khan Strengthening Democratic Ms. Jermone Sayre, Chief of P Local Governance Project arty Professor Latifa Samsuddin, President OBGYN Society 5/13/13 Mr. Ganesh Chandra Sarker, Director, IEM DGFP Dr. Mohammad Sharif, Director, MCH DGFP Dr. Mahbubur Rahman, Director, Clinical Contraception DGFP Dr. Tajul Islam, Technical Advisor, SMPP JICA Ms. Yukie Yoshimura, Chief Advisor Professor Shahidullah, Vice Chancellor, President, Bd. Perinatal Forum BBSMU 5/14/13 Ms. Jiwon Seo, Program Advisor SC-K, (KOICA) Dr. Ferdousi B, Country Manager Country Manager, FANTA III Mr. S. Roy, Deputy Country Representative Alive and Thrive Dr. Altaf Hossain, Program Manager, IMCH DGHS Dr. Makhduma Nargis, Project Director, Community Clinic DGHS 5/15/13 Dr. Abu Jafar Musa, Director, Prime Health Care DGHS 5/16/13 Dr. A.J Faisel, Country Representative Dr. Mizanur Rahman, Senior Technical Advisor EngenderHealth Dr. S.M. Nizamul Hoque, Team Leader, Policy/Advocacy 5/16/13 Gazi Md. Rezaul Karim, Coordinator, Fistula Care Team Dr. Md. Saekhul Islam Helal, Team Leader, PPH EngenderHealth Dr. Sanjida Hasan, Project Coordinator, MIH USAID/BANGLADESH: MAMONI FINAL EVALUATION 67 Date Name Organization Md. Liaquat Ali, Mayer Hashi Md. Azmal Hossain, Team Leader, BCC Fatema Shabnab, Team Leader, Service Delivery & Training 5/18/13 Dr. Riad Mahmud UNICEF Dr. Ziaul Matin 68 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 69 ANNEX VI: FY 2012 OPERATIONAL PLAN AND ACHIEVEMENTS Standard Indicator FY 12 Target Project Achievement Comment Maternal and Child Health Estimated # of pregnant women to be identified and registered 132,413 114,988 (87% of target) # postpartum/newborn visits within 3 days by any provider 31,253 (26% of all pregnant women) 28,932 (98% of target) Not disaggregated by provider (e.g., FWA) # ANC visits by SBAs 28,063 (%21 of pregnant women receiving 1 ANC visit w/SBA: 9% of pregnant women receiving 3 ANC visits by SBA) 49,105 (175% of target) # people trained on MNH 2,013 4,224 (211% of target) 2,676 TBA trained # deliveries with SBA 12,025 (9% of pregnant women) 18,948 (158% of target) # newborns receiving essential newborn care 22,213 (17% of pregnant women) 22,938 (103% of target) Not disaggregated by provider # women reached with handwashing messages 94,850 (71% of pregnant women) 105,575 (111% of target) Family Planning CYP 210,763 287,767 (137% of target) Counseling visits for FP/RH 1,154,413 1,485,176 ( 129% of target) Not disaggregated by provider (FWA, HA, FWV) People trained on FP 2,013 4,244 2,676 TBA trained 70 USAID/BANGLADESH: MAMONI FINAL EVALUATION USAID/BANGLADESH: MAMONI FINAL EVALUATION 71 ANNEX VII: TRAINING OVERVIEW Type Participant Time Level Trainer Content District When MNH/FP FWA, HA, AHI, FPI, project staff 5 days Upazila MT Pregnancy care, newborn care, FP, project intervention Sylhet, Habiganj 2010 and 2012 MNH/FP FWV, SACMO, paramedic 5 days Upazila MT Pregnancy care, newborn care, FP, project intervention Habiganj 2010–11 and 2012–13 ANC FWV, SACMO, CSBA 2 days Upazila MT ANC content, counseling, logistic use, documentation Habiganj 2011–12 Injectables, DMPA CHW 1 day theory + 10 injections District FP district trainer Injectable contraceptive Habiganj, Sylhet 2010 Orient TBA TBA-30/union 5 days Union FWV, CSBA Safe delivery, harmful practice of delivery, danger sign, referral Sylhet, Habiganj 2010 ETAT and sick newborn care Doctor, nurses, FWV and paramedic 5 days National National Sick newborn care Habiganj 2011–12 IUD Paramedics 12 days National National IUD and other method Habiganj 2013 CSBA Private CSBA 6 months Training center District trainer Pregnancy care, newborn care, FP basic Habiganj 2011–13 72 USAID/BANGLADESH: MAMONI FINAL EVALUATION Type Participant Time Level Trainer Content District When HBB All SBA 5 days District and upazila District trainer ENC, resuscitation Habiganj, Sylhet 2012–13 Basic facilitator skill Master trainer (UH&FPO, UFPO, RMO, MOMCH) 5 days District Outsourcin g Basic skill of facilitator Habiganj, Sylhet 2010 Supportive supervision AHI, FPI, HI, and project supervisor 2 days Upazila Basic supervisory skill, current supervision system, supportive supervision – what and how Habiganj 2012 Micro-planning HA, FWA, AHI, FPI, and project staff 2 days Upazila Out sourcing Concept, strategy, organization, management of CMP Habiganj, Sylhet 2012 UP orientation All members 2 days Union Out sourcing Structure, authority and responsibility of UP, community resource mapping, areas of providing support to MNHFP, action plan development Habiganj, Sylhet 2013 (Health, Education & FP standing committee) Members of H,E & FP standing committee Orient Depo Holder 1/village (approx) 1 day Union Project staff Role of DH, logistics information, supply chain, USAID compliance Sylhet 2010 and 2013 Habiganj 2012–13 USAID/BANGLADESH: MAMONI FINAL EVALUATION 73 ANNEX VIII: STRATEGIES FOR HARD-TO￾REACH AREAS Intervention 5 . In Baniach ang .. J UHC j UH&FWC • Private CSBA o Upgraded fWC provide 24/7 to o Newly constructed FWC Interventi ons in Lakhai.. Interve "tions • in Ajm,· . nlilnj .. ..J UHC j UH&FWC • Pnva, .. r.sR.l. o U~e«IF'''' Cto pnwid~ un D rj.wIy confO/«U_"""" .-......"' .. !IIt ......... o(""'_ J. """"'''',.,....._'''~'''''''9'0 _~_ ... ~J) ..... _0IcW ... .....,..." ..... !IIt_ .....", Of"""""" _ o(!IIt-... .. .......,"'--_ .......... _,...." ... """"'--"'11_ -~J)_.~_"""" .... -"' .... ~ I. - """",.,...._ _____ .. ---"" ____ .... ~ ....... w "'II ,..,- ...-.....I>J_.~! ... """" -., , I from I , ,/ /.-~ , 'S.I. , ~I/ , " , 80 USAID/BANGLADESH: MAMONI FINAL EVALUATION , MaMoni: Integrated Safe Motherhood, Newborn Care and Family Planning Project ~, j)Il''''' ---.J .... , .. ".!. ~ •• 0 . ..... f ..... ,"'''·!f.U .. l .<1, ...... , ~r< ..... -''''''.., (~ .... L It ' .. 1- ."." '" "",,.~ ... v ", '0"'"' '', d~"<, ., ..... .. , tho. + Of'''' I •• ,"", '' "'1'(11)< ~ .. . :~;;;' '··~··'·'· ::~~ 7f t ~ USAID/BANGLADESH: MAMONI FINAL EVALUATION 81 Disclosure of Conflict of Interest for USAID!GH Consultants "m. Organization ~~ , ,~'" , "~O, , il , "''''',' , ""' """"" on I"" '.""""~ _ '0, "' ''''''''''''-. ~ av., .. ,...., __ ' .... ~ ... <_10< of II'< ""'II> _"" .... "'"""9"'9 "'" ",*,,1>1 .. "'0<"'''''''''''1 ;",:,""" .. """'''''' ') ""'"'" proj.ct('1 ..... - art_ , -:-__ ""'''''''''.mct'''''',...r ....... _'g~_",