USAID Pakistan June 2012 This report was produced for review by the United States Agency for International Development Mission in Bangladesh. It was prepared by Pinar Senlet, Mary Jo Lazear, and Ahmad Neaz, through the Global Health Technical Assistance Project. EVALUATION Midterm Performance Evaluation of The Mayer Hashi Project : Effective Delivery of Long-Acting and Permanent Family Planning Methods and Maternal Health Services in Bangladesh MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT EFFECTIVE DELIVERY OF LONG-ACTING AND PERMANENT FAMILY PLANNING METHODS AND MATERNAL HEALTH SERVICES IN BANGLADESH JUNE 2012 This report was produced for review by the United States Agency for International Development Mission in Bangladesh. It was prepared by Pinar Senlet, Mary Jo Lazear, and Ahmad Neaz, through the Global Health Technical Assistance Project. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT EFFECTIVE DELIVERY OF LONG-ACTING AND PERMANENT FAMILY PLANNING METHODS AND MATERNAL HEALTH SERVICES IN BANGLADESH DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-006) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00004. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT i ACKNOWLEDGMENTS The Mayer Hashi evaluation team is grateful to the many individuals who gave so generously of their time and whose perspectives and opinions are reflected in this report. The names of those with whom we met are listed in Annex B. Specifically, the team would like to acknowledge the support received from the staffs of USAID/Bangladesh, the Mayer Hashi project, and the Directorates of Family Planning and Health Services of the Ministry of Health and Family Welfare (MOH&FW). The team is especially grateful to Khadijat Mojidi, Sukumer Sarker, and Thibaut Williams of the USAID Office of Population, Health, Nutrition and Education (OPHNE) for their guidance and participation in the evaluation effort. Our special thanks go to Abu Jamil Faisel, project director, and Ellen Themmen, technical director of the Mayer Hashi project, for making themselves, their staff, and project data and information so readily available. Finally, the team wishes to extend appreciation to the governmental officials and non￾governmental organization (NGO) representatives with whom we met in Cox’s Bazar during our informative field trip. ii MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT iii CONTENTS ACRONYMS ...................................................................................................................................... vii EXECUTIVE SUMMARY ..................................................................................................................ix Summary of Findings, Conclusions, and Lessons Learned ..............................................ix Summary of Recommendations for the Remaining Life of the Project ....................... xi Summary of Recommendations for USAID’s Future Programming ............................ xii I. INTRODUCTION ................................................................................................................... 1 II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE...................................... 3 Brief Overview of Mayer Hashi ............................................................................................. 3 III. PURPOSE OF THE EVALUATION ...................................................................................... 7 IV. EVALUATION METHODOLOGY ...................................................................................... 9 Information and Data Sources ............................................................................................... 9 Limitations of the Evaluation Methodology ......................................................................10 V. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ....................................11 Project Achievements to Date ............................................................................................11 Effectiveness of Project Strategies and Approaches .......................................................18 Contributions to Strengthening Health Systems .............................................................24 Advocating for Policy Change ..............................................................................................25 Project Management ..............................................................................................................27 Coordination ...........................................................................................................................28 Compliance Issues ..................................................................................................................28 Gender Isssues ........................................................................................................................29 VI. LESSONS LEARNED .............................................................................................................31 Overarching Lessons Learned on Project Focus and Strategies ..................................31 Technical Conclusions and Lessons Learned ...................................................................31 VII. SUMMARY OF RECOMMENDATIONS FOR THE REMAINING LIFE OF THE PROJECT .................................................................................................................33 Overarching Strategic Recommendations .........................................................................33 Technical Recommendations ...............................................................................................33 iv MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT ANNEXES ANNEX A. SCOPE OF WORK ....................................................................................................35 ANNEX B. PERSONS CONTACTED .........................................................................................49 ANNEX C. REFERENCE LIST .......................................................................................................53 ANNEX D. EVALUATION FRAMEWORK ...............................................................................57 ANNEX E. LIST OF SITES VISITED IN COX’S BAZAR DISTRICT.....................................59 ANNEX F. USAID’S COORDINATION WITH OTHER DONORS ..................................61 FIGURES Figure 1: Comparison of CYP of LA/PM Acceptors of 30 Months Before the Project Began and During the 30 Months of Project Implementation in 21 MH Districts .........................................................................................................11 Figure 2: Trends in LA/PM Acceptors in 21 MH Districts by Quarter (July 2009– December 2011) ............................................................................................................12 Figure 3: Total Number of Individuals Who Received Clinical or Non-clinical Orientation/Training in LA/PM Activities .................................................................13 Figure 4: Total Number of Individuals Who Received Clinical or Non-clinical Orientation/Training by Type of Organization .......................................................13 Figure 5: Number of Postpartum Women Who Accepted Tubectomy or IUD at MH-Supported Maternity Facilities before They were Discharged (July 2009–December 2011) .......................................................................................14 Figure 6: Number of Immediate Postpartum Family Planning Acceptors in 21 MH Districts (July 2009–December 2011) ......................................................................14 Figure 7: Number of Long-acting Family Planning Method Users by Age Groups in 21 MH Districts (October 2009–September 2010) ....................................................15 Figure 8: Number of Voluntary Surgical Contraception Users by Age Group in 21 MH Districts (October 2009–September 2010) ....................................................15 Figure 9: Number of Voluntary Surgical Contraception Users by Parity in 21 MH Districts (October 2009–September 2010) ............................................................16 Figure 10: Number of Long-acting Method Users by Parity in 21 MH Project (October 2009–September 2010) .............................................................................16 Figure 11: Trends in Distribution and Use of Misoprostol Tablets in Five Districts During PY2 and PY3. ...................................................................................17 Figure 12: Number of Service Providers Trained in AMTSL (September 2009–December 2011) ..........................................................................18 Figure 13: Number of Individuals Oriented/Trained in Distribution of Misoprostol (September 2009–December 2011) ..........................................................................18 Figure A1. Results Framework of Mayer Hashi Project ............................................................38 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT v TABLES Table 1: Demographic Health Survey Data of Bangladesh ........................................................ 2 MAPS Map 1: Mayer Hashi Geographical Coverage .......................................................................... 4 vi MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT vii ACRONYMS AMTSL Active management of the third stage of labor ACQUIRE Access, Quality and Use in Reproductive Health BCC Behavior change communication BCCP Bangladesh Center for Communication Programs BDHS Bangladesh Demographic and Health Survey BPMPA Bangladesh Private Medical Practitioners Association CCSDP Clinical Contraceptive Service Delivery Program CPR Contraceptive prevalence rate CYP Couple years of protection Dfid UK Department for International Development DGFP Directorate General of Family Planning DGHS Directorate General of Health Services DQA Data quality assessment EDL Essential drug list FWA Field Worker Assistant FWV Family Welfare Visitor GHI Global Health Initiative (USAID/Bangladesh) HSS Health systems strengthening ICDDRB International Center for Diarrheal Disease and Research in Bangladesh IEM Information, Education and Motivation Unit of the Directorate General of Family Planning IUD Intrauterine device JHU/CCP Johns Hopkins University/Center for Communications Programs LA/PM Long-acting and permanent methods MCH Maternal and child health MCHIP Maternal and Child Health Integrated Project MH Mayer Hashi MIS Management Information System MOH&FW Ministry of Health and Family Welfare NGO Non-governmental organization NIPORT National Institute of Population Research and Training NSV No-scalpel vasectomy NTC National Technical Committee viii MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT OGSB Obstetrical and Gynecological Society of Bangladesh OPHNE Office of Population, Health, Nutrition and Education OPRH Office of Population and Reproductive Health (USAID, Washington) PAC Post-abortion care PY Project year PPH Postpartum hemorrhage RESPOND Responding to the Need for Family Planning through Expanded Contraceptive Choices and Program Services SDA Supply, Demand, Advocacy (RESPOND model) SHOPS Strengthening Health Outcomes through the Private Sector SIAPS Systems for Improved Access to Pharmaceuticals and Services Program, (Management Sciences for Health) SMC Social Marketing Company SOW Scope of work SRHR Sexual and reproductive health and rights SSFP Smiling Sun Franchise Program TFR Total fertility rate TOT Training of trainers UNFPA United Nations Population Fund USAID United States Agency for International Development WHO World Health Organization YMC Young married couple MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT ix EXECUTIVE SUMMARY A team of three consultants conducted a midterm performance evaluation of the Mayer Hashi (MH) (“Smiling Mother”) project, an Associate Award supported by the U.S. Agency for International Development (USAID)/Bangladesh under the Global Responding to the Need for Family Planning through Expanded Contraceptive Choices and Program Services (RESPOND) project. The evaluation was conducted in April–May 2012. The $12 million award applies from May 2009–September 2013, and the evaluation covered the project period from June 2009– December 2011. MH is being implemented in 21 districts in three low-performing divisions of Bangladesh (Chittagong, Barisal, and Sylhet) and is addressing two important areas in family planning and maternal health that have not received adequate attention and have not shown significant results in the recent past: 1) the need for family planning through the expansion of access, quality, and use of long-acting and permanent methods of contraception (LA/PM) and 2) selective maternal health services to prevent postpartum hemorrhage (PPH), including active management of the third stage of labor (AMTSL) and the community-based provision of misoprostol. The key purposes of this performance evaluation were to: 1) review, analyze, and evaluate the effectiveness of the MH project in achieving program objectives to date; 2) identify major constraints in achieving expected project results; and 3) provide specific recommendations and lessons learned on strategies and approaches the program should pursue over the last two years of implementation and in future program planning. The evaluation methodology consisted of: 1) analyzing data and information gleaned from a variety of background documentation and project reports, the project’s monitoring and evaluation (M&E) plan, and other surveys; 2) extensive interviews with a variety of partners, stakeholders, and beneficiaries; and 3) field visits to project sites to gain an in-depth understanding of project achievements and challenges. SUMMARY OF FINDINGS, CONCLUSIONS, AND LESSONS LEARNED Project achievements to date: In examining the project’s achievements to date against expected results in LA/PM and PPH, the team found that MH had effectively addressed both areas. It appeared that concerted efforts on particular areas of family planning and maternal health could significantly elevate their importance with policy makers, managers, and service providers. The project had achieved and exceeded most of its objectives. For example: The team compared couple years of protection (CYP) of LA/PM acceptors for the 30 months before the program began with the CYP achieved during the 30 months of intervention in project districts. They found that there was more than a 30% increase in the CYP for LA/PM acceptors during the project implementation period, and the number of LA/PM acceptors exceeded the objective. The number of LA/PM acceptors in project districts showed a gradual, but consistent, upward trend for all methods during the project implementation period. The project provided comprehensive technical assistance in training and orientation of decisionmakers, managers, service providers, field workers, and community leaders. Project x MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT data showed that, in total, over 22,500 individuals were trained in the use of LA/PM, exceeding the project’s projection of 17,000. Of the trainees, 54% were women. The number of postpartum tubectomy and IUD (intrauterine device) acceptors in project￾supported sites since the inception of the project also exceeded the projection. A further analysis revealed that the number of postpartum tubectomy acceptors significantly surpassed the number of postpartum IUD acceptors during the project implementation period. A breakdown of LA/PM users by age and parity in the project area was consistent with the findings of the 2011 Bangladesh Demographic and Health Survey (BDHS). The highest numbers of LA/PM users were within the 30–39 age group for those who had three or more children. The distribution and use of misoprostol tablets increased significantly during the project implementation period. However, in the last quarter of the implementation period, there appeared to be a discrepancy between the number of tablets distributed and the number of pregnant women who used the tablets. This could be due to overuse or misuse of the tablets and/or the time between when the tablets were distributed and actually taken. As of December 2011, the number of providers trained in AMTSL and the number of program managers, field workers, and community members who received orientation training in the community distribution of misoprostol both exceeded the project’s projections. Effectiveness of project strategies and approaches: The team examined several successful approaches and strategies implemented by the project. Community-level distribution of misoprostol tablets: This appeared to be a very effective life-saving approach. Using national maternal mortality data, it was estimated that, during the pilot interventions, 19 PPH-related maternal deaths were averted in the two districts. While the project is assisting the MOH&FW in national scale-up of the intervention, several challenges remain. Increasing male involvement in family planning: An analysis of the Directorate General of Family Planning (DGFP) Management Information System (MIS) showed a recent upward trend in acceptance of male methods, particularly vasectomy, in the project districts. This was likely due to using a “satisfied vasectomy clients” approach to promote the method. Additional increases in LA/PM prevalence can be achieved by expansion of this intervention. Introduction of postpartum family planning services into the Directorate General of Health Services (DGHS) maternal health services and select private sector facilities where family planning services were not previously provided: The team observed that the support for training in postpartum family planning seemed to be valued by local program managers as well as service providers. However, providers trained in postpartum IUD insertions were not practicing the skills because the demand for postpartum IUDs was very low. Increasing demand for LA/PM: The actual effectiveness of the Behavior Change Communication (BCC) campaign will not be known until an endline survey is conducted. However, the team’s interviews with key informants and observations in the field revealed some impediments related to the BCC campaign’s potential progress: 1) the launch of the MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT xi campaign was delayed due to a lengthy design and approval process, 2) implementation of the campaign was also delayed and not well synchronized with other program activities, and 3) the campaign was perceived as “old-fashioned” because it has not used modern technologies. In addition, the campaign messages did not adequately address the widespread myths regarding LA/PM in Bangladesh. Several innovative and promising approaches for the project, including promoting the increased use of long-term family planning methods among young married couples (YMC), using a “bottom-up” contraceptive projection approach through client segmentation, and working with the private sector in LA/PM provision: These approaches were recently started, and therefore their effectiveness could not be evaluated fully. Contributions to strengthening the health systems: MH’s project description does not have a clear mandate or an objective on strengthening health systems of the country, although several health systems strengthening (HSS)-related interventions were among the project’s focus activities, including national and local level planning, human resource capacity building, quality improvement, logistics management, and strengthening MIS. Removing policy barriers: MH has shown strong leadership in addressing LA/PM and PPH and has established a well-researched, well-defined, and well-monitored system for addressing vital policy issues that relate to their programmatic achievements. Their work in the policy arena includes reviewing family planning guidelines and strategies and medical eligibility criteria. The team noted that well-planned and action-oriented advocacy efforts can lead to policy changes in a short period of time. Project management and coordination: MH has a well-defined internal program management structure, with team leaders in five areas and a strong project M&E plan. The project works well within a bifurcated system between the two directorates general of family planning and health systems. In addition, MH is working in tandem with several partners in the private and NGO sectors. The team also observed that USAID provides strong management direction to the project. Compliance issues: The recent incident in which the DGFP was about to launch a lottery related to sterilization procedures was stopped as a result of actions undertaken by MH and USAID. As MH gathered information about this lottery, however, they waited approximately three weeks before informing USAID (i.e., lottery observed on February 27th and reported to USAID on March 22nd). SUMMARY OF RECOMMENDATIONS FOR THE REMAINING LIFE OF THE PROJECT The team recommended that, for the remaining life of the project, MH should build program sustainability, ensure timely handover to local partners, continue expanding and replicating successful models, and document lessons learned. Specific technical recommendations include: Continue community-level distribution of misoprostol tablets: MH should investigate the reasons for the discrepancy between the number of misoprostol tablets distributed and used during the last quarter of 2011. The project should continue to assist the local partners with a carefully planned and phased roll-out and closely monitor field activities. xii MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Institutionalize AMTSL: MH should work with the DGFP and DGHS to ensure that the newly recruited service providers receive on-the-job training by the trained providers, and that monitoring and supervision mechanisms are in place and functioning. An audit of service data should be conducted to ensure that AMTSL is actually being implemented in these facilities. Increase male involvement in family planning: MH should assess the results of the efforts so far and provide lessons learned for further expansion of the initiative. Integrate postpartum/post-abortion family planning services into maternal and child health (MCH): MH should discontinue postpartum IUD training and assess the results achieved so far to document the lessons learned. The project should also work to strengthen postpartum counseling. Increase demand for LA/PM through BCC: MH should continue to collect information so that the impact and results of BCC efforts can be documented in the end-of-project assessment. MH needs to examine whether its current approaches are effective in reaching a large number of individuals. Expand other innovative approaches: MH should continue to expand its promising innovative approaches such as YMC, bottom-up contraceptive projections, and working with the private sector in LA/PM provision. Contribute to strengthening health systems: MH should continue its work on strengthening health systems through national- and local-level planning, human resources capacity building, quality improvement, and logistics management. Advocate policy change: MH should build on its strong record of working through the National Technical Committee (NTC) and continue to advocate for policy changes regarding LA/PM and PPH. Address compliance issues: The new Tiahrt poster (produced by MH in conjunction with Johns Hopkins University/CCP JHU/CCP and the DGFP) should be distributed widely with instructions as to where it should be placed where clients can best see it. MH should continue training providers at all levels, as well as government officials, in voluntarism and informed consent. SUMMARY OF RECOMMENDATIONS FOR USAID’S FUTURE PROGRAMMING Project focus: The MH project addressed two important areas in family planning and maternal health that have not received adequate attention. Therefore, the team suggests USAID consider a follow-on project that maintains a focus on the LA/PM and PPH, rather than incorporate these two important issues into a project with a broader mandate in family planning and maternal health. Successful approaches and interventions: MH has introduced and implemented several successful approaches as described in detail in this report. Yet, there has not been enough time for some of these approaches to be tested and fully implemented. Therefore, it is recommended that the follow-on project build on and replicate these approaches. Health systems strengthening: The team recommends that a follow-on project have a more structured mandate for health systems strengthening to be especially implemented at the district MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT xiii level. A “hybrid” model to health systems strengthening is recommended. Field-level HSS activities of the follow-on project should be complemented and supported by national-level HSS support. BCC: In a new follow-on project, BCC efforts should be started almost immediately and should include the use of modernized BCC approaches such as use of cell phones, social media, videos in waiting rooms, hotlines, and so on. The campaign should specifically address widespread myths regarding LA/PM in Bangladesh. Counseling: The follow-on project should emphasize training of providers in interpersonal communication skills, including counseling. The team recommends that training in interpersonal communication, accompanied by a monitoring and evaluation system for BCC activities, be an integral part of a new program design. Capacity-building of service providers: The follow-on project should place emphasis on LA/PM refresher training for providers. It is also recommended that training on postpartum IUDs be discontinued, since there is little use for this skill in the Bangladesh setting. On the other hand, training on postpartum family planning needs to be strengthened by emphasizing that postpartum LA/PM can be implemented during the period immediately after delivery up to one year postpartum. Policy and advocacy: Based on the lessons learned from MH, policy support and advocacy should be integral parts of a follow-on project. xiv MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 1 I. INTRODUCTION A team of two international consultants and a local consultant were invited by the U.S. Agency for International Development (USAID)/Bangladesh (through the GH Tech Bridge Project) to conduct a midterm performance evaluation of the Mayer Hashi (“Smiling Mother”) project, an Associate Award supported by USAID/Bangladesh under the global RESPOND project. The team’s scope of work (SOW) is located in the annexes of this document. EngenderHealth manages MH in collaboration with the Johns Hopkins University/Center for Communications (JHU/CCP) and the Population Council. The award period is from May 2009–September 2013, and the USAID investment is $12 million. MH is a follow-on to previous USAID investments in reproductive and maternal health, most notably the Access, Quality and Use in Reproductive Health (ACQUIRE) project. Bangladesh’s demographic profile underlines the need for strong family planning programs. Bangladesh is one of the most densely populated countries in the world. The legal age for marriage is 18, but many women marry at a much younger age, which correlates with early childbearing and high fertility. Rapid urbanization and migration to cities have intensified population density, especially in urban slum areas. Currently, 26% of women and 28% of men live in urban areas. More than 40% of the population lives below the poverty line. Family planning programs have received a great deal of donor support since Bangladesh’s independence, but much work remains to be done. According to the 2011 BDHS, 11.7% of the need for family planning is unmet, down from 17.6% in the 2007 BDHS. The family planning program in the country must emphasize expanding access to and use of LA/PM. The table below shows the low prevalence of LA/PM and the total contraceptive prevalence rate (CPR) for any method according to the past two surveys. It also shows that the total fertility rate (TFR) has declined on a national level as well as in the divisions in which MH is working. CPR also has risen, both nationally and in the project locations. 2 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Table 1: Demographic Health Survey Data of Bangladesh Level Variable/Indicator TFR CPR (Any Method) LA/PM National BDHS 2007 2.7 55.8% 7.3% BDHS 2011 2.3 61.2% 8.0% MH Project Divisions Barisal BDHS 2007 2.8 56.8% 7.0% BDHS 2011 2.3 64.7% 6.2% Chittagong BDHS 2007 3.2 43.9% 5.7% BDHS 2011 2.8 51.4% 6.9% Sylhet BDHS 2007 3.7 31.5% 4.1% BDHS 2011 3.1 44.8% 6.7% Data source: BDHS 2007 and BDHS 2011 Maternal mortality rates in Bangladesh have started to decline, but there are still 194 maternal deaths per 100,000 live births. About 31% of these deaths are due to hemorrhage, the majority of which occur in the third stage of labor. By practicing AMTSL, an injection of oxytocin can be used during this period to stimulate uterine contractions, thus managing the hemorrhaging. It should be noted, however, that the use of oxytocin requires a cold chain to be in place. Also, it cannot be injected in a home setting (where over 75% of births occur). Misoprostol taken orally immediately following the birth of the baby also reduces the occurrence of PPH. MH lists among its project-related innovations the community-based distribution of misoprostol for PPH prevention. The team’s findings and recommendations, as well as relevant graphs, are contained in the body of this report. The report also examines some of the major constraints and challenges that the project faces in achieving its projected results. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 3 II. THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE USAID’s program strategy is influenced by several global initiatives, and MH works within the context of these initiatives, which are described below: In line with the U.S. president’s Global Health Initiative (GHI), USAID/Bangladesh developed Bangladesh’s Interagency Program Strategy 2011–2015. This initiative expressly states that “over the next five years, the USG will help Bangladesh to adopt and scale-up proven interventions to increase long-term family planning methods and reduce maternal mortality.” MH’s work reflects GHI principles by implementing a woman- and girl-centered approach; implementing a YMC intervention in the Patuakhali District; strengthening and leveraging key multilateral organizations, global health partnerships, and private sector engagement; and building sustainability through health systems strengthening. The MH workplans delineate specific activities in each of these areas. The two targets of Millennium Development Goal (MDG) 5 to improve maternal health are as follows: Target 5A: reduce by three quarters, between 1990–2015, the maternal mortality ratio; and Target 5B: achieve, by 2015, universal access to reproductive health. Development Objective 3 of the USAID Bangladesh Country Development Cooperation Strategy for FY 2011–2016 contains the “flagship” development objective for the “whole of government strategy approach” under the GHI. It stresses improved efficiencies and innovative approaches while building on successful past and ongoing population, health, and nutrition programs. It states that the goals for Bangladesh are to reduce the fertility rate to two children per woman by 2016, revitalize the family planning program, and adopt a multisectoral approach to addressing population issues. Gender, youth, and innovation are the cross-cutting themes in achieving these objectives. BRIEF OVERVIEW OF MAYER HASHI MH is addressing two important areas—PPH and LA/PM—in maternal health and family planning that have not received adequate attention and have not shown significant results in the recent past. It appears that no other organization is exclusively addressing these two areas. The Government of Bangladesh’s new national health sector plan identifies LA/PM as a priority issue, and USAID is supporting a number of activities to assist in the revitalization of the use of LA/PM. MH is being implemented in 21 low-performing districts located in three divisions of Bangladesh (Chittagong, Barisal, and Sylhet). Map 1 shows the geographical coverage of the project. The development hypothesis that guides MH is that USAID’s support for increased use of LA/PM and enhanced clinical and community maternal approaches will further reduce the fertility rate and considerably lower maternal mortality. It is assumed that MH’s interventions will help influence the government’s policies and thereby improve government performances in other areas as well. 4 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Map 1: Mayer Hashi Geographical Coverage MH identified the following critical assumptions that will underpin the success of the project: Ministry of Health and Family Welfare’s policies are supportive to implement best practices, models, and new approaches. National Institute of Population Research and Training (NIPORT) will take the lead in supporting the institutionalization of LA/PM training. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 5 The availability of family planning commodities will be ensured by the DGFP. The Directorates of Health and Family Planning will be supportive by including training on AMTSL and the use of a partograph (a simple chart for recording the progress of labor and the condition of a woman and her baby during labor which, according to the WHO, is key to the appropriate prevention and treatment of prolonged labor and its complications) in the government’s operational plan, and adding misoprostol to the essential drug list (EDL). The theory of change underlying MH is that there is a demographic imperative for continuing, refining, and expanding family planning in Bangladesh due to the cohorts of youth entering their reproductive years coupled with a population growth rate of 1.5% and a projected population size of 220 million in 40 years. MH is working to promote change through increasing access to and information about the effectiveness of LA/PM, improving awareness and demand for LA/PM, strengthening the policy initiatives, and strengthening the enabling environment and health systems to ensure LA/PM services. Also, since childbirth remains a serious risk for women in the country, and maternal mortality surveys show that the maternal mortality ratio has declined from 322/100,000 in 2001 to 194/100,000 in 2010, MH is working to build on this success. MH is helping the Government of Bangladesh decrease maternal mortality by addressing PPH (one of the main causes of maternal death), by providing community-based distribution of misoprostol, and by training providers in the use of AMTSL. MH’s programs are helping Bangladesh achieve MDG 5. MH has two primary goals: 1) addressing the need for family planning through the expansion of access, quality, and use of LA/PM and 2) addressing selective maternal health services to prevent PPH through clinical and community approaches, including AMTSL and the community-based provision of misoprostol by utilizing the supply, demand, and advocacy (SDA) service delivery model. The MH project has adopted EngenderHealth’s SEED programming model for improved sexual and reproductive health by delivering quality services (supply), creating an enabling environment through policy work, and creating demand through awareness raising and behavior change communication. The project utilizes holistic and evidence-based approaches in its programmatic design. The MH results framework contains two strategic objectives: 1) increase LA/PM use and 2) promote PPH prevention practices in MH working areas. The intermediate results (IRs) under each topic address clinical services and evidence based approaches (IR1), demand creation (IR2), and policy support and creation of an enabling environment through knowledge generation, dissemination, and use (IR3). MH works effectively in both the public and private sectors. In the public sector, MH has a direct relationship with both the DGFP and the DGHS within the MOH&FW. While these two directorates have made attempts to unite programmatically in recent years, the current system remains bifurcated, a situation that could present a barrier to moving forward. Cooperation continues to be a politically sensitive challenge; nonetheless, MH is working—to the extent possible—to encourage collaborative efforts between the two directorates. 6 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT In the private sector, MH is working in tandem with several partners on LA/PM service delivery and marketing. These partners include the Smiling Sun Franchise Program (SSFP), the Social Marketing Company (SMC), Strengthening Health Outcomes through the Private Sector (SHOPS), the Obstetrical and Gynecological Society of Bangladesh (OGSB), and the Bangladesh Private Medical Practitioners Association (BPMPA). In maternal health, MH is working closely with USAID’s Ma Moni project—Integrated Safe Motherhood, Newborn Care, and Family Planning. Through several innovative approaches, MH has responded to other issues that have not adequately been addressed by family planning and maternal health care in Bangladesh. These include a gender-based approach to the use of permanent methods by using satisfied no-scalpel vasectomy (NSV) clients to increase male involvement in family planning, integrating postpartum family planning into maternal health care, increasing the use of long-term family planning methods among YMC; using a bottom-up contraceptive projection through client segmentation (to counter centrally set performance benchmarks for districts and upazilas); and working with the private sector in LA/PM provision. MH works through the National Technical Committee (NTC) to foster an enabling environment for LA/PM by removing policy barriers, developing new strategies, reviewing family planning guidelines, and establishing medical eligibility criteria. MH utilizes a mechanism known as the Reality Check family planning tool, which consists of an Excel workbook that allows individuals and organizations to assess past trends and test future scenarios in the CPR for the geographic areas in which the project works. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 7 III. PURPOSE OF THE EVALUATION This external review of MH was a midterm performance evaluation of the project covering the period from June 2009–December 2011. The key purposes of the evaluation were to: Review, analyze, and evaluate the effectiveness of the MH project in achieving program objectives and contributing to USAID/Bangladesh’s efforts to reduce maternal mortality by preventing PPH and increasing use of LA/PM. Evaluate major constraints in achieving expected project results. Provide specific recommendations and lessons learned on strategies and approaches the program should pursue over the next years of implementation and for future program planning. USAID/Bangladesh will use the evaluation’s findings and recommendations to improve implementation of the ongoing project as well as influence the design of a follow-on project or other relevant Population, Health Nutrition and Education (PHNE) projects/programs. USAID will disseminate the report widely with the public health stakeholders such as government and NGO program managers, USAID implementing partners, donors, and professional health associations. Upon clearance of procurement-sensitive information, USAID will share the document with government agencies, donors, implementing partners, and other NGOs through mail correspondence and seminars/workshops. USAID expects the evaluation report will benefit the implementing partners, host government, and other donors by enhancing their understanding of the program and by improving the design of future interventions. The evaluation addresses the following key questions: To what extent has the project achieved its objectives against expected results on LA/PM and PPH? What components of the current MH strategy (both LA/PM and PPH) have been most/least effective and what can be done to improve the project performance on LA/PM and PPH? How effectively has MH coordinated with the Government of Bangladesh, other donors, NGOs, and the private sector to achieve its LA/PM and PPH objectives? How has the project contributed to strengthening the health systems of the country in delivering LA/PM and PPH services? What recommendations or actions should USAID take to support future Government of Bangladesh efforts to expand access to LA/PM and PPH services in Bangladesh? What are the project management issues that positively/adversely impact performance of the project? This report is organized around these specific evaluation questions. Sections V through VIII flow directly from, and attempt to fully respond to, the evaluation questions. The evaluation team’s scope of work (SOW) is found in Annex A. 8 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 9 IV. EVALUATION METHODOLOGY The evaluation methodology consisted of: Analyzing data and information gleaned from a variety of background documentation and project reports, the project’s PMP, and surveys. Conducting extensive interviews with a variety of partners, stakeholders, and beneficiaries. Making field visits to project sites to gain an in-depth understanding of project achievements and challenges. The methodology combined a review of quantitative data and application of qualitative evaluation techniques to obtain information, opinions, and data from counterparts, contractors, partners, clients, beneficiaries, Government of Bangladesh entities, and other donors. By using a mixed approach, the evaluation team gained insights on the effectiveness of MH project activities (mostly from quantitative data collected by the project and others) and the processes (mostly qualitative information provided by the project staff and key informants) that led to those outcomes. The team followed a participatory approach, working closely with USAID/Bangladesh and MH project staff, and used the evaluation SOW and deliverables from the MH program description and M&E plan as its basic analytic framework. Evaluation questions from the evaluation SOW and sources of information and data are found in the attached evaluation framework (Annex D). Expected outcomes corresponding to the two components of the project (LA/PM and PPH) were evaluated against the results achieved by December 2011. INFORMATION AND DATA SOURCES Review of background documentation: The evaluation team conducted a broad review of background documents provided by USAID/Bangladesh and the MH project, as well as national data such as the national surveys and strategies on family planning, maternal health, and population (Annex C). Key informant interviews: The team interviewed a wide range of managers, implementers, and other stakeholders both in Dhaka and in the field, as well as community leaders, service providers, and actual and potential clients during site visits. The team organized data from key informant interviews to include information on certain indicators or to validate data obtained from other reports. The list of persons contacted is found in Annex B. Site visits: The team visited Cox’s Bazar district in Chittagong to observe MH project operations and interview field-level informants. The site was chosen—in consultation with USAID and the MH project leadership—because all project activities are ongoing in this district. The team relied heavily on this visit to address evaluation SOW questions, gain insights into project achievements and challenges, and verify different opinions presented by various informants (Annex E). 10 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT LIMITATIONS OF THE EVALUATION METHODOLOGY The evaluation team was not engaged in primary data collection from any statistically designed sample of beneficiaries or providers. Rather, it depended on the secondary data available from the routine management information system records and the reports of other surveys and assessments conducted by MH or other programs. Since key informant interviews were a major source for validation of information available from MH, the evaluation team selected interviewees with guidance from USAID/Bangladesh to minimize the possibility of bias. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 11 V. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS PROJECT ACHIEVEMENTS TO DATE Increased Use of Long-acting and Permanent Methods of Contraception The first strategic objective of MH is to increase use of LA/PM. To evaluate project accomplishments toward this objective, the evaluation team compared the CYP of LA/PM acceptors for the 30 months before the program began with the CYP achieved over 30 months of intervention in the 21 project districts. The pre-intervention period from December 2006– May 2009 was compared with the 30 month period from the launch of the project in July 2009 until December 2011 (figure 1). The comparison shows that there is more than a 30% increase in the CYP achieved by LA/PM acceptors during the project implementation period compared to the CYP achieved in the same districts before the project was launched. At the time of the evaluation, the number of LA/PM acceptors had exceeded the projected number. Figure 1: Comparison of CYP of LA/PM Acceptors of 30 Months Before the Project Began and During the 30 Months of Project Implementation in 21 MH Districts 3,002,256 4,055,795 3,654,029 0 500000 1000000 1500000 2000000 2500000 3000000 3500000 4000000 4500000 Actual performance Pre￾Intervention Period (Dec. 2006 - May 2009) Actual achievement Intervention period (Jul. 2009 - Dec. 2011) Life of the Project Target (Jul. 2009 - Sep. 2013) Source: DGFP MIS A further analysis of MH MIS data on the numbers of LA/PM acceptors in the project districts shows a gradual but consistent upward trend between September 2009–December 2011. The upward trend is slightly sharper for long-term methods compared with voluntary sterilization. However, the number of voluntary sterilization acceptors is still much higher than those of long-term methods (figure 2). 12 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Figure 2: Trends in LA/PM Acceptors in 21 MH Districts by Quarter (July 2009– December 2011) Source: DGFP MIS MH worked to create capacity within local organizations to improve the access to and quality and use of LA/PM services. This required the provision of comprehensive technical assistance in the training and orientation of decisionmakers, managers, service providers, field workers, and community leaders. Training activities included awareness-raising for vast numbers of individuals who could potentially be involved in demand creation for and provision of LA/PM information and services. In addition, MH supported capacity-building of several cadres of health providers in clinical training for LA/PM. MH/MIS shows that, in total, over 22,000 people were trained to increase the use of LA/PM, exceeding the project’s projection of 17,000. Figure 3 depicts the total number of individuals who received clinical, non-clinical, or orientation training with support from MH. Women accounted for 54% of the trained individuals. The majority of the individuals trained in MH-supported LA/PM programs were staff of the DGFP and DGHS. MH also assisted in training large numbers of NGO staff, community leaders and workers, and private providers (figure 4). - 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 Jul-Sep 2009 Oct-Dec 2009 Jan-Mar 2010 Apr-Jun 2010 Jul-Sep 2010 Oct-Dec 2010 Jan-Mar 2011 Apr-Jun 2011 Jul-Sep 2011 Oct-Dec 2011 No. of acceptors Total VSC Long-Acting Method MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 13 Figure 3: Total Number of Individuals Who Received Clinical or Non-clinical Orientation/Training in LA/PM Activities 924 1,294 16,239 21,169 Achievement Clinical training Non-clinical training/orientation Source: MH MIS Figure 4: Total Number of Individuals Who Received Clinical or Non-clinical Orientation/Training by Type of Organization 10,841 6,148 1,772 1,361 585 1,756 DGFP DGHS NGO Community Private Others Organizations Source: MH MIS MH focuses on increasing the use of LA/PM to address the family planning needs of the postpartum women. To this end, the project worked with DGHS maternity clinics and selected private providers to integrate provision of tubectomy and IUD services immediately after the delivery or prior to the discharge of the postpartum woman from a clinic. Figure 5 depicts the number of postpartum tubectomy and IUD acceptors in project-supported sites since the inception of the project. The total number of acceptors has significantly exceeded projections (figure 5). - 5,000 10,000 15,000 20,000 25,000 Target No. of trainees - 2,000 4,000 6,000 8,000 10,000 12,000 No. of persons Figure 5: Number of Postpartum Women Who Accepted Tubectomy or IUD at MH-Supported Maternity Facilities before They Were Discharged (July 2009–December 2011) 14 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT No. of acceptors Source: MH MIS A further analysis of tubectomy and IUD acceptors during the immediate postpartum period reveals that, during the first year of the project, the postpartum acceptance of both tubectomies and IUDs remained very low. Demand for postpartum tubectomy began to increase gradually during the second year of the project and rose sharply in late 2011. The number of postpartum tubectomy acceptors during the last quarter of 2011 was 35 times higher than the number achieved in the last quarter of 2009. During the same time frame, the acceptance of postpartum IUDs also increased somewhat, but not as significantly as tubectomy acceptance (figure 6). The upward trend in postpartum tubectomies is due to the involvement of an increasing number of project sites and trained providers serving in these facilities. The MH staff told the team that the project would strongly emphasize postpartum family planning training during the third year of the project, and these efforts are reflected in the increased number of acceptors, particularly of postpartum tubectomy. Figure 6: Number of Immediate Postpartum Family Planning Acceptors in 21 MH Districts (July 2009–December 2011) 0 200 400 600 800 1000 1200 1400 Jul-Sep 2009 Oct-Dec 2009 Jan-Mar 2010 Apr-Jun 2010 Jul-Sep 2010 Oct-Dec 2010 Jan-Mar 2011 Apr-Jun 2011 Jul-Sep 2011 Oct-Dec 2011 PPBTL PPIUD Source: MH MIS 2,020 3,190 - 500 1,000 1,500 2,000 2,500 3,000 3,500 Target PY1-PY4 Achievement PY1-PY4 No. of acceptors MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 15 Source: DGFP MIS As required by the evaluation SOW, the team analyzed the breakdown of long-term and permanent method users by age and parity. DGFP MIS data for FY 2010 (October 2009– September 2010) was used for the analysis. The highest number of users of both long-acting (IUDs and implants) and permanent (vasectomy and tubectomy) methods were within the 30– 39 age group. The 20–29 cohort was the second largest group using a long-term family planning method. Men and women over the age of 40 were included in the third largest group (figures 7 and 8). Figure 7: Number of Long-acting Family Planning Method Users by Age Groups in 21 MH Districts (October 2009–September 2010) 15,545 90,002 93,159 40,179 5,300 34,829 36,860 13,789 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 <20 20-29 30-39 40+ No. of users Age group of users IUD Implant Source: DGFP MIS Figure 8: Number of Voluntary Surgical Contraception Users by Age Group in 21 MH Districts (October 2009–September 2010) 1,258 16,548 45,790 40,412 3,585 60,042 187,965 178,769 0 20000 40000 60000 80000 100000 120000 140000 160000 180000 200000 <20 20-29 30-39 40+ No. of users Age group of users NSV Tubectomy 16 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT A similar analysis of the use of long-term and permanent methods by parity shows that the highest number of both long-acting and voluntary sterilization users has three or more children (figures 9 and 10). Figure 9: Number of Voluntary Surgical Contraception Users by Parity in 21 MH Districts (October 2009–September 2010) Source: DGFP MIS Figure 10: Number of Long-acting Method Users by Parity in 21 MH Project (October 2009–September 2010) 271 2,063 24,249 77,425 204 4,520 81,911 343,726 - 50,000 100,000 150,000 200,000 250,000 300,000 350,000 400,000 0 1 2 3+ No. of users Parity NSV Tubectomy 181 38,118 90,555 110,031 100 11,007 33,466 46,205 0 20000 40000 60000 80000 100000 120000 0 1 2 3+ Implant No. of users Parity IUD Source: DGFP MIS The results of the analyses by age group and parity are consistent with the findings in the 2011 BDHS. It is expected that by the end of the project life, the majority of LA/PM users will shift toward younger age groups with fewer children. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 17 Prevention of Postpartum Hemorrhage The second strategic objective of MH is to promote PPH prevention practices at all facility￾based and home-based deliveries in the project districts. To this end, MH worked to strengthen the delivery and sustained use of AMTSL at facilities and misoprostol in communities (a more detailed description of project interventions for preventing PPH is found in the next section of this report). Figure 11 shows the trend in the distribution and use of misoprostol tablets in the project districts during PY2 and PY3. Distribution and usage of tablets increased significantly from PY2 to PY3, and the number of tablets distributed was consistent with the number of pregnant women who delivered and used them.1 However, an analysis of distribution and use during the first quarter of PY4 pointed to a discrepancy between the number of tablets distributed and the number of pregnant women who used the tablets. This is probably due to the time lag between the distribution of tablets and the actual usage. The tablets are distributed to women during the 32nd week of pregnancy, but they are supposed to be used at the time of the delivery. Misuse or overuse of the misoprostol is also a possibility. Figure 11: Trends in Distribution and Use of Misoprostol Tablets in Five Districts During PY2 and PY3. Source: MH MIS To promote PPH prevention practices, MH worked to build the capacity of local institutions to train and update the knowledge of their health personnel in AMTSL. As a result of these efforts, large numbers of public, private, and NGO service providers were trained in provision of AMTSL. By December 2011, the number of providers trained exceeded the target set for PY1– PY4. Figure 12 depicts the target and the actual number of service providers trained in AMTSL. Of the individuals trained in AMTSL, 88% were women. At the time of this evaluation, the project had reached and trained all service providers in target facilities in AMTSL. However, it 1 The recommended regimen was to use three tablets immediately after the delivery during the first two years of the intervention. In the third year, it was reduced to two tablets. - 10,000 20,000 30,000 40,000 50,000 60,000 70,000 PY2 (Oct09-Sep10) PY3 (Oct10-Sep11) Service Statistics # of Misoprotol tablets distributed - # of women delivered at home and used Misoprostol tablets - Figure 12: Number of Service Providers Trained in AMTSL (September 2009– December 2011) 18 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT PY4. Figure 12 depicts the target and the actual number of service providers trained in AMTSL. Of the individuals trained in AMTSL, 88% were women. At the time of this evaluation, the project had reached and trained all service providers in target facilities in AMTSL. However, it was recognized that due to staff reassignments and newly recruited service providers the projections for AMTSL trainings might change over time. 2,606 2,731 Target PY1-PY4 Achievement PY1-PY4 2,000 2,200 2,400 2,600 2,800 3,000 No. of people Source: MH MIS Similarly, the number of program managers, field workers, and community members who received orientation or training in community distribution of misoprostol exceeded projections during the same time period (figure 13). Women represented 47% of all individuals who benefitted from misoprostol trainings. Figure 13: Number of Individuals Oriented/Trained in Distribution of Misoprostol (September 2009–December 2011) 3,581 5,935 - 2,000 4,000 6,000 8,000 Target PY1-PY4 Achievement PY1-PY4 Conclusions: The analysis of MH achievements up to December 2011, evaluated against expected outcomes corresponding to the two components of the project (LA/PM and PPH), indicated that MH has been successful in achieving its objectives to date. MH has reached or exceeded the benchmarks set for selected indicators. Capacity-building of local partners through training has been one of the strengths of MH; these efforts are reflected in increased utilization of LA/PM of family planning. Recommendations: For the remaining life of the project, MH should focus on 1) program sustainability, 2) ensuring timely handover to local partners, 3) continuing to expand and replicate successful models, and 4) documenting lessons learned. EFFECTIVENESS OF PROJECT STRATEGIES AND APPROACHES The team analyzed several approaches and strategies integrated into the MH project design to increase the use of LA/PM and to prevent PPH. The team found adequate quantitative and qualitative data to evaluate the effectiveness of many of these approaches. Many interventions were initiated in Bangladesh prior to the launch of MH with the support of EngenderHealth Source: MH MIS No. of people MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 19 that the team was not able to evaluate fully, because either it was too early to objectively assess outcomes or the data were not yet available. Following is a discussion on effectiveness of various strategies and approaches implemented by MH. Community-level Distribution of Misoprostol Tablets Prior to the launch of MH, EngenderHealth had supported a pilot intervention in Tangail to distribute misoprostol tablets to pregnant women to prevent PPH. An evaluation of the pilot project showed that field workers can safely distribute misoprostol at the community level when they are well trained and supervised. MH conducted the second phase of the pilot in Cox’s Bazar from November 2009–September 2010. The team’s evaluation of the pilots showed that the women who used misoprostol were generally pleased, as misoprostol reduced excessive bleeding and therefore was widely accepted after delivery. Using national maternal mortality data, it is estimated that during the pilots, 19 PPH-related maternal deaths were averted in the two districts.2 Following the completion of the pilots, MH assisted the MOH&FW in developing a national scale-up plan, and the 2011 operational plans and implementation budgets of both DGHS and DGFP included community-based distribution of misoprostol. MH is now assisting with the misoprostol initiative in four additional project districts. Conclusions: Key informant interviews both in Dhaka and the field revealed that community￾level distribution of misoprostol is a highly effective life-saving approach. This opinion was shared by the decisionmakers and program managers at all levels, as well as field workers. Statistics on maternal deaths support these anecdotal perceptions. There is potential for scaling up the intervention throughout Bangladesh, but several challenges need to be addressed in order to do so, including raising community awareness at a national level, training a large number of staff, monitoring and supervising the program, and taking measures to avoid misuse. Moving from district-level pilots to the national scale will take time and require the concerted efforts of many players. Recommendations: MH should investigate the reasons for the discrepancy between the number of misoprostol tablets distributed and used during the last quarter of 2011. MH should assist the local partners with a carefully planned and phased roll-out and closely monitor field activities. Institutionalization of Active Management of the Third Stage of Labor The second approach utilized by MH to reduce PPH is to strengthen the use of AMTSL at facilities by building capacity for and institutionalizing AMTSL training. At the time of MH launch, AMTSL was just being introduced in Bangladesh. MH focused on implementing best practices via training of trainers, strengthening training capacity, and developing curricula and supervision materials. At the time of this evaluation, MH had completed AMTSL Training of Trainers (TOT) and training for facility-based maternity service providers and orientation managers in all 21 target districts. However, due to staff reassignments and newly recruited providers, MH is continuing training activities. 2 “Community-based Misoprostol Distribution to Prevent Postpartum Hemorrhage in Bangladesh.” USAID Program Brief. January 2012. 20 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Conclusions: AMTSL is an easy and powerful practice to prevent PPH; therefore, MH worked to create and expand local capacity to use AMTSL. Decisionmakers as well as program managers and service providers interviewed by the team stated they valued MH’s support with AMTSL. The service providers interviewed were satisfied by the quality of on-the-job training they received. Recommendations: Over the remaining life of the project, MH should work with the DGFP and DGHS to ensure that the newly recruited service providers receive on-the-job training by the trained providers, and that monitoring and supportive supervision mechanisms are in place and functioning. The team also recommends an audit of service data supplemented by direct observation to ensure that AMTSL is actually being implemented in these facilities. Increasing Male Involvement in Family Planning By 2007, vasectomy’s share in the family planning program method mix had significantly declined to 0.7% from 1.5% in 1985 (BDHS 2007). At that time, EngenderHealth had been working with the DGFP to increase the use of LA/PM. The national MIS data showed an upward trend in vasectomy clients since 2007, with some centers and districts performing particularly well. In 2010, MH conducted a rapid assessment to identify factors that influence successful vasectomy performances in those districts. The assessment revealed a range of factors that contributed to the increased performance, with the strong endorsement of satisfied vasectomy clients as one of the key factors for success. Thus far over 3,000 satisfied vasectomy clients have participated in this approach. MH continues to assist the DGFP in collecting and analyzing data to follow up on results of the approach. In addition, MH conducted a performance review among 30 randomly selected satisfied vasectomy clients. Analysis of the information collected thus far shows that the vasectomy acceptance rate in the intervention districts has been increasing gradually. District￾level family planning monitoring data shows that the satisfied clients refer an average of two or three clients per month for vasectomies. The team’s field interviews with service providers and district managers verify the upward trend in vasectomy acceptance. In Cox’s Bazar, DGFP service statistics from 2010 and 2011 show an eightfold increase in vasectomy acceptance. Acceptance of tubectomy also increased during the same timeframe, though not so dramatically (50% increase). The satisfied clients contribute to increasing the overall acceptability of vasectomy by communicating to others that they should not fear the procedure. This client-centered approach also appears to have strengthened male involvement in the Bangladesh family planning program. The official recognition and respect the satisfied vasectomy clients receive encourages some of them to become regular referrers and “champions” in the family planning program. The Government of Bangladesh is committed to further expanding this program and has incorporated it into its current DGFP operational plan. Another successful way to increase male involvement in family planning has been to work through imams (religious leaders). MH continues to support the training program among imams to engage these highly influential religious leaders who can then help generate community demand for family planning services. The initiative aims to help the imams understand their role in improving the health of families by providing information in a manner that is sensitive to religious tradition. The educated involvement of the imams is critical to increasing access and MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 21 use of permanent methods of family planning, particularly among men. The team conducted a small focus group interview with four imams in Cox’s Bazar and observed that the information and training program was well received by them. Their feedback indicated that there has been a positive change in their perceptions of family planning. They were very enthusiastic about their involvement and expressed their comfort with speaking out to stress the beneficial impact that family planning can have in the lives of men and women in their communities. Conclusions: The interventions to increase male involvement in LA/PM appear to be working, and there is potential for expansion and replication. Additional increases in the CPR, particularly through the use of vasectomy, can be achieved by further expansion of the satisfied clients program. Recommendations: During the remaining life of the project, MH should continue to expand and replicate these successful interventions to increase male involvement in LA/PM. MH should also assess the results of the efforts so far and provide lessons learned for further expansion of the initiative. Integrating Postpartum Family Planning Services into Maternal and Child Health Although the immediate postpartum period is an opportune time for initiating contraceptive use within the maternal health services context, the family planning needs of postpartum women were not addressed until recently. EngenderHealth, through the previous ACQUIRE project, introduced postpartum family planning services into select facilities. MH continued the efforts and introduced such services into the DGHS maternal health services and selected private sector facilities where family planning services were not already provided. MH also introduced postpartum family planning in the DGFP facilities where other family planning (not postpartum) services were available. MH interventions included orientation for stakeholders and all facility- and community-level providers, clinical training of service providers, BCC activities for demand creation, and provision of medical equipment. At the time of this evaluation, MH had introduced postpartum family planning services in 44 facilities. DGFP service statistics show that the demand for postpartum tubectomy has begun to increase sharply since 2011. The acceptance of postpartum IUDs, however, remains low. During the field trip to Cox’s Bazar, the team visited two DGHS facilities (a health complex and a general hospital) where delivery services were provided. The team interviewed five service providers who were trained in MH-supported postpartum family planning training. All five stated that they were satisfied with the quality of training they had received, specifically postpartum IUD training. However, they were discouraged because they were not able to practice the skills they had acquired, due to low demand for postpartum IUDs. There had only been two postpartum IUDs inserted at these two facilities since the training of service providers. The team also interviewed five postpartum women who had delivered in these facilities either that day or on the previous day. None of the women appeared to have received adequate family planning counseling, and their knowledge of contraception and family planning options was limited. The service providers then told the team that postpartum women would be counseled prior to their discharge from the clinic. 22 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Postpartum family planning training plans for additional service providers from the general hospital were canceled due to staffing shortages. The nurses were not able to leave their posts for an extended period because there weren’t enough other nurses to provide services in their absence. The team learned that, unfortunately, provision of on-the-job training is not possible because of the low demand for postpartum family planning in these facilities. Conclusions: Training of service providers in postpartum tubectomy appears to be effective, as reflected in upward trends in the method’s acceptance. Immediate postpartum IUDs could also be a good option for many women who deliver at facilities; however, the demand for postpartum IUDs is very low. Clinical training for postpartum IUD insertions has to take place in clinical settings and requires high client caseloads. In addition, if the trainees do not practice the acquired clinical skills, those skills will be lost. Recommendations: MH should discontinue postpartum IUD training and assess the results achieved so far to document the lessons learned. There is also a need to strengthen postpartum counseling skills of service providers in the DGHS facilities. Increasing Demand for Long-acting and Permanent Methods The activities designed to increase the demand for LA/PM are being implemented by the Bangladesh Center for Communication Programs (BCCP), an implementing partner of MH. MH is assisting the DGFP to conduct a multimedia, multilevel (national and local) BCC campaign to create awareness of LA/PM services and generate greater use of these methods. At the national level, the campaign plans includes TV commercials, reality shows, posters, and billboards. Local￾level BCC activities include street dramas; a video documentary; and informational materials for the clients such as leaflets, a flip chart, and a guidebook for the providers. Nonetheless, the launch and implementation of the campaign was delayed due to a lengthy design, planning, and approval process, and the challenges of synchronizing of national and local activities. As an example, national TV shows were aired after the local campaign activities were completed. As part of the project design, an endline survey is planned for the final year of the project. Thus, the team lacked the data to fully evaluate the effectiveness of the BCC campaign. The team relied on insights gained from key informants, project documents, and observations from the field to evaluate demand creation efforts. Some demand creation interventions targeting specific groups such as satisfied NSV clients and religious leaders appeared to be promising, but more structured assessments are needed to measure the impact. During the field visit to Cox’s Bazar, the team observed that the BCC materials produced with MH assistance (posters, brochures, handouts, and flip charts) were available. However, feedback the team received from several interviewees revealed that the BCC materials produced for clients, as well as street dramas and dated videos, were seen as old-fashioned and did not have significant reach. It was also noted that the demand creation interventions did not utilize modern technology, such as cell phones, hotlines, and social media.3 These approaches are being 3 According to the research conducted by MH on media preferences in 2010, short message service (SMS) was identified as a preferred media, but was not utilized by the project. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 23 widely used by other health programs in Bangladesh as well as by family planning programs in other countries.4 The materials were basically promoting use of LA/PM but did not address the negative impressions and misinformation about these methods or attempt to correct them. Myths that IUDs may migrate into the upper body, that vasectomy causes long-term inability to work because it “weakens the body,” that sterilization is a poor person’s method of choice, and that the methods go against religious practices are widespread in Bangladesh. These misperceptions are crucial barriers to increased use of LA/PM. Conclusions: The team believes that a more modernized BCC approach and improved messaging will overcome negative perceptions and incorrect beliefs about LA/PM and will, in effect, promote their use. A more timely start of a BCC (demand creation) campaign would enhance the “supply” side of service provision. Recommendations: To the extent possible in the remaining life of the project, MH needs to examine whether the BCC approaches are effective in reaching a large number of individuals and if a more modernized approach and different messages are needed to overcome negative perceptions and incorrect beliefs about LA/PM. Other Innovative Approaches MH has introduced several innovative approaches to family planning, including promoting increased use of long-term family planning methods among YMC, using a bottom-up contraceptive projection approach through client segmentation (to counter centrally derived projections), and working with the private sector in LA/PM provision. These approaches were recently started and therefore their effectiveness could not be evaluated, although they appeared to be promising. YMC is a model intervention focusing on young married couples (the wife is 20 years old or younger) with at least one child. Young married men and women are provided with information on all family planning methods, while the behavior change objective would focus on long-acting methods. The main channel for BCC is the couple’s peers. The bottom-up contraceptive projection approach works through client segmentation to counter centrally set performance benchmarks for each district and upazila. In the bottom-up approach the community must be the driving force rather than the centrally derived targets. The intervention has been piloted in two districts, and national scale-up began recently. In the third year of the project, MH started working with the private sector to increase its involvement in LA/PM in the MH project divisions. At present, private providers have an insignificant role in service delivery, even though they have a large presence throughout the country and an increasing share of the provision of short-term contraceptives. MH has partnered with the SMC and SHOPS projects to expand the capacity of the private sector to deliver LA/PM. As part of that effort, each partner is responsible for leading specific activities. MH aims to create capacity within the private sector via training private providers in LA/PM, including clinical and counseling skills. SHOPS will lead in the selection of participating facilities 4 The team learned from the JHU representative who works in the DGFP that she recently met with MH to discuss the need to connect to YouTube and Facebook. She also emphasized the fact that cell phone use is widespread throughout Bangladesh. 24 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT and providers, the design of support and agreements with each provider, the provision of support for improved business skills, and the clarification of policy and regulatory issues. SMC will lead the handling and distribution of long-acting method commodities and the design and implementation of behavior change and marketing strategies. Thus far, 249 providers have been trained and services have been introduced in 26 private sector facilities. Conclusions: The aforementioned approaches of the project appear to be promising. However, since they were recently started, the team could not fully evaluate their effectiveness. Recommendations: Mayer Hashi should continue to expand these promising interventions and, by the end of the project, should assess the results and document the lessons learned. CONTRIBUTIONS TO STRENGTHENING HEALTH SYSTEMS The MH project description does not have a clear mandate or an objective on strengthening health systems of the country, although several HSS-related interventions were among the project’s focus activities. Over the course of the project MH intervened in many areas of health systems strengthening related to family planning, particularly in improving the quality of LA/PM services. Through national and local level planning, MH contributed to the development of the strategic plan for the Health, Population, and Nutrition Sector Development Program (HPNSDP) (2011–2016) and operational plans to ensure relevant language for LA/PM and PPH prevention are included in these documents. At the district level, MH assisted in local planning and coordination meetings with all stakeholders to discuss the activities planned to ensure their commitment and ownership. Regarding human resource capacity-building, one of the key strategies of MH has been to strengthen the training capacities of local institutions, including the DGFP, DGHS, professional organizations, NGOs, and the private sector at various levels. Capacity-building efforts included developing curricula, training manuals, and guidelines, and training large numbers of managers, trainers, and service providers as described in detail in earlier sections of this report. MH’s contribution to quality improvement is the assistance provided for the development of curricula related to improving service quality. MH supported the development of the national LA/PM curriculum and the postpartum family planning training curriculum, and these training materials are already in use in project districts. Currently, MH is assisting implementing partners in the development of a curriculum on facilitative supervision for quality improvement. Concerning logistics management, MH remains in contact with the Systems for Improved Access to Pharmaceuticals and Services (SIAPS) project and the DGFP to stay informed on the contraceptive commodity security situation. As SIAPS has the mandate to work in this area at the central level, MH primarily addresses commodity security issues related to LA/PM at the district level. In 2010, the DGFP introduced local-level projections for family planning methods based on client segmentation. MH developed all tools and systems and then field tested and revised them in close collaboration with the DGFP. The DGFP initiated national scale-up in 2011 with MH technical assistance. This included training for the DGFP and statistical officers for all 64 districts. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 25 Relating to management information systems, MH continues to work with the MIS unit of the DGFP to strengthen the MIS and monitoring of the national family planning program as a follow-up on EngenderHealth’s efforts since 2006. This work includes the introduction and adoption of the data quality assessment (DQA) methodology, which is used as a data quality control mechanism. Conclusions: Although not part of its official mandate, MH successfully contributed to strengthening health systems of the country in delivering LA/PM and PPH services in several areas. Recommendations: MH should continue its work on strengthening health systems through national- and local-level planning, human resource capacity-building, quality improvement, and logistics management. ADVOCATING FOR POLICY CHANGE At the beginning of the project, MH staff reviewed all the policies related to their mandate to see how they corresponded with World Health Organization (WHO) norms and guidelines, and they determined which ones were the major barriers to the acceptance of LA/PM interventions. The project conducted an in-depth policy analysis and created a policy framework that listed feasibilities, priorities, and timelines regarding where program efforts would be placed. They also noted which policies were politically sensitive and whether they could get support from the government if they advocated for change. MH’s work in the policy arena includes reviewing family planning guidelines, strategies, and medical eligibility criteria. MH works on policy change through the NTC. MH staff prepare policy briefs and circulate them to members of the NTC before each meeting. The briefs contain evidence-based background information regarding the proposed policy change. After the briefs are circulated, the MH policy team leader follows up with each member of the NTC to make sure the issues are clear and to answer any questions the committee’s members might have. Realizing that the change is ultimately owned by the DGFP, MH does the preparation for the meeting but remains in the background while the line director of the clinical contraceptive program of the DGFP presents the proposed changes. Within this process, the Futures Institute staff provides advocacy training to committee members. The boxes below illustrate MH’s notable achievements in policy and advocacy in the first half of the project, and the issues they are working on so far in the second half of the project. Conclusions: MH has established an excellent process for advocating for policy change and for removing policy barriers related to LA/PM and PPH. The project has a well-researched, well￾defined, and well-monitored system for addressing vital policy issues that relate to their programmatic achievements. Recommendations: MH should build on its strong record of working through the NTC and continue to advocate for policy changes regarding LA/PM. 26 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Summary of Policy Changes Introduced by MH 1. Waiver of the 2-year age requirement of the youngest child for couples with two children who would like to adopt a permanent method: If a couple desires a permanent method (either a tubectomy or a vasectomy), they can have it immediately after the delivery of the second child. They still need two children to receive a permanent method of family planning, but the age requirement is no longer. This was approved in May 2010, and the circular was sent out shortly thereafter. 2. Staff nurses of the DGHS hospitals and clinics of other organizations will be allowed to insert IUDs: This was also approved in May 2010. This was a very important policy change as it meant a large increase in the available number of IUD service providers. 3. Introduction of Sino-Implant (II) in the Bangladesh family planning program: MH provided technical assistance, and the Population Council drafted the research proposal for the acceptability trial of this implant. An acceptability trial, which is required in order for a new contraceptive to be introduced, was conducted by the Government of Bangladesh. June 2012 will mark one year of data collection, and the NTC will be advised about whether Sino is ready to be made available to Bangladeshi women. Jadelle has already gone through the trial, but may never be used since Sino is so much less expensive. 4. Acceptance of the implant as a long-acting family planning method by newly married women and married women with no children is now permitted: This received governmental approval during the NTC meeting that took place on January 9, 2011. A circular was sent out shortly after. There are some delays in the implementation of this policy, as not all service providers are convinced that implant is in fact a suitable method for nulliparous women, despite the circular. Also, as the implant is very popular, some managers prefer to provide the method to women with more children rather than to nulliparous women. As a reaction to these implementation challenges, MH is addressing these issues in ongoing training programs and planning to provide additional information to both service providers and managers. 5. Private clinics/hospitals registered under the DGHS do not need to obtain registration from DGFP: Private facilities used to have to register under DGFP and DGHS. This lengthy process made it difficult for organizations like the SSFP and private sector facilities. The approval of this policy was a big achievement. Even an NGO providing postpartum family planning had to register with the DGFP. A hospital or health facility registered with the DGHS that would like to provide postpartum family planning services can now get the necessary commodities through an imprest fund. There is no need to register twice. It is important to note that both the DGFP and the DGHS signed this policy, which is a good example of cooperation between two directorates that often operate separately. This was approved in the middle of 2011. 6. Introduction of progestin-only pill (minipill) in the family planning program for postpartum contraceptives: This was approved in February 2012 and is already in effect. This is an important policy, as including the progestin-only pill (minipill) in the public sector family planning program will ensure access for all women, including the poorest. 7. The NTC meeting in May 2010 approved the distribution of misoprostol tablets by DGFP structures and field workers for community-based prevention of PPH. This was a key decision that enabled scaling up the misoprostol program through the DGFP structures and field workers. 8. Misoprostol national training guidelines have been approved by the Curriculum Technical Review Committee of DGHS. This approval was important as it enabled the misoprostol national training guidelines to be included in the national scale-up of the community-based misoprostol program for the DGHS. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 27 PROJECT MANAGEMENT MH has a well-defined internal program management structure, with team leaders in five areas: service delivery, increased demand, policy support and enabling environment, PPH prevention, and M&E. The teams regularly meet individually and jointly. This enables the project to maintain distinct areas of responsibility, thereby providing a strong mechanism for overseeing their management and monitoring systems. The team reviewed MH’s M&E plan and found it to be well-defined with appropriate indicators and clear benchmarks. MH’s Policy-related Plans for the Remaining Life of the Project 1. Including comprehensive family planning in the midwifery curriculum: Several meetings have been held thus far with the Registrar, the Nursing Council, and with representatives of the WHO and UNFPA involved in developing the curriculum. As a result of the policy and advocacy team’s consistent efforts, a comprehensive family planning component is now included. The policy and advocacy team is now working to develop the family planning syllabus and handouts. 2. Including a comprehensive family planning component in the nursing course curriculum: The policy and advocacy team had several meetings with the Registrar and the Nursing Council. The Registrar has provided assurance that the nursing curriculum will be revised in 2012, and they will include MH as a member of the revision committee. 3. Scaling up bottom-up contraceptive projection through client segmentation and local-level planning: As a result of continuous advocacy by Team 3, the DGFP introduced last year local-level projections for family planning methods based on client segmentation. By October 2011, 27 districts submitted their projections to the Clinical Contraceptive Service Delivery Program (CCSDP). The CCSDP included this program in their operating plan, and MH continues to support the process. The policy and advocacy team assists the CCSDP in conducting contraceptive projection planning orientations for 2012/13 with upazila and district officials and statistical assistants. 4. Increasing private sector involvement in LA/PM services: During PY3, MH started working with the private sector to increase their involvement in LA/PM service delivery. In PY4, MH expanded its work in the private sector through a joint initiative with the SMC and SHOPS, in collaboration with a broad range of stakeholders, including the DGFP, the OGSB, the JHU/CCP-Knowledge for Health project, and SIAPS. MH will take a coordinating role among the three key project partners to advance the policy and enabling environment issues for LA/PM in the private sector. It will continue working on key private sector policy issues using the strategic advantage of its close contacts with the government, particularly the DGFP, to ensure information exchange and DGFP involvement, and to advance the policy debate on increasing LA/PM service delivery through the private sector. This includes the establishment of a technical committee for family planning in the private sector with the DGFP as the chair and the aforementioned organizations as members. 5. Provision of first dose injectable contraceptive by Family Welfare Assistants (FWAs): Currently, FWAs are allowed to provide injectables only from the second dose onward, while FWVs and doctors are providing the first doses. The first dose can now be given by these trained FWAs. MH has not yet started working on this issue, but will start building support for this through its advocacy efforts and then bring it to the NTC meeting. 28 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MH has retained close ties with both EngenderHealth in Bangladesh and the EngenderHealth headquarters in New York City. The team was told lessons learned would be disseminated by the headquarters after the project is completed. COORDINATION MH coordinates their programs and strategies with stakeholders in both the public and private sectors. MH has a direct relationship with both the DGFP and the DGHS within the MOHFW. While these two directorates have made attempts to unite programmatically in recent years, the current system remains bifurcated, and cooperation remains politically sensitive. The functional and structural bifurcation could present a barrier to progress on objectives. Nonetheless, the team believes MH is working effectively, to the extent possible, to encourage cooperative efforts between the two directorates. In the private and NGO sectors, MH is working in tandem with several partners on service delivery and marketing in relation to LA/PM. These partners include the SSFP, the SMC, SHOPS, the OGSB, and the BPMPA. In maternal health, MH is working closely with USAID’s Ma Moni project—Integrated Safe Motherhood, Newborn Care and Family Planning—and the USAID global MCHIP. Although coordination with other donors occurs primarily at the USAID level, MH might have occasion to participate on task forces and in forums with these donors. At the request of the health team at USAID/Bangladesh, the team met with three donors: UNFPA, Dfid, and the Netherlands. The purpose was to gain information about their collaboration with USAID and to learn what they are doing in their overall family planning/maternal health projects. The summary of these meetings can be found in Annex F. However, the team learned that: Although there was acknowledgement of USAID’s active and important role in health, donors want to know more about USAID programs. The team noted that knowledge of the MH project, for example is very limited at best. There is interest among the donor community for expanded and improved coordination and collaboration. USAID health programs have been fairly insular but, in the last two years, USAID has come “out of its shell.” (Quote from an informant: “The USAID health office has aligned with the MOH and reignited the relationship.”) USAID helped the government develop a national strategy on LA/PM, but this was not widely disseminated or widely distributed. COMPLIANCE ISSUES In February 2012, MH discovered, during an NSV satisfied clients’ orientation, that the DGFP had initiated a permanent method lottery program to motivate clients and had issued a circular without informing MH. As MH gathered information about this circular, USAID was not informed until approximately three weeks after the lottery was observed. In reaction, the project assembled three teams to conduct a compliance issues assessment in six districts and nine upazilas in three randomly selected divisions. MH worked with USAID and the USAID advisor to the IEM Unit of the DGFP to ensure that the family planning advisory committee was educated regarding compliance. MH also agreed to organize a family planning compliance training MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 29 for DGFP staff and for central-level staff of relevant USAID partners. MH will insert a chapter on voluntarism and informed choice in the DGFP family planning manual in line with the wording in the national population policy. The team did not see the compliance report, but was asked to observe whether Tiahrt posters were in evidence at the sites visited in Cox’s Bazar. The team discovered that, although the posters were in evidence at all of the clinic sites, they were not well placed for patients seeking family planning or postpartum services. The Tiahrt posters were also old and worn, often placed in the administrative offices or on walls not visible to clients. MH has designed a new Tiahrt poster which is much more attractive; however, these were only posted in a few sites. Conclusions: The team was not given the opportunity to observe any counseling sessions for clients, so it could not be sure that adequate counseling was provided. The team did ask what kind of counseling was being provided and was shown a kit containing samples of all contraceptive methods along with other educational materials that were distributed to clients. Nonetheless, due to lack of direct observation, the team could not verify that showing clients this kit was actually accompanied by interpersonal counseling. This could be due to a shortage of staff who do not have the time to properly counsel clients. Recommendations: The new Tiahrt poster (produced by MH in conjunction with JHU and the DGFP) should be distributed widely in all of the government facilities where MH trained providers are working with instructions as to where it should be placed. MH, in conjunction with EngenderHealth, should continue training providers at all levels of government, including government officials, in voluntarism and informed consent. MH should monitor to ensure that providers who have received clinical and non-clinical training, as well as those in supervisory positions, are presenting contraceptive options to clients before they select a method. GENDER ISSSUES Mainstreaming gender is one of the overarching project strategies of the MH project. MH has incorporated gender-related considerations into its programming, and gender equity is integrated into its training, community orientation, and outreach activities. MH reported that its project staff aims to ensure that at least equal numbers of women and men are reached through its programs. In line with GHI principle #2, which is to implement a woman- and girl-centered approach, MH concentrates on family planning and PPH—both of which focus on women and girls. MH also has a strong male involvement project that relies on “satisfied users” to promote LA/PM. MH also has a program for young married couples, which involves both men and women. MH has set up a system of disaggregating data for reporting and evaluation purposes. MH has constructed charts to determine the population breakdown of women and men in the three divisions where they are working (Barisal, Chittagong, and Sylhet). MH has also developed an indicator-tracking table as part of its M&E system that tracks male/female statistics in a variety of areas. For example, 86,099 women and 165,921 men have received LA/PM information through U.S.-funded activities to date. Similarly, the number of counseling visits for family 30 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT planning/reproductive health was 412,414 for women and 99,122 for men. Data disaggregated by gender on training activities are presented in different sections of this report. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 31 VI. LESSONS LEARNED Although the evaluators of the MH project only looked at the first half of the project’s interventions, the team determined that there are already some valuable lessons that can be applied to immediate or future programming. OVERARCHING LESSONS LEARNED ON PROJECT FOCUS AND STRATEGIES MH has addressed two important areas in family planning (LA/PM) and maternal health (PPH) that have not received adequate attention. It appears that no other organization is exclusively addressing these two areas. The team believes that a project that maintains a focus on LA/PM and PPH, rather than incorporating these two important issues into a broader mandate, has been more effective. The lesson learned is that concerted efforts in a particular area of family planning can not only elevate its importance to policymakers and providers but, by increasing the method mix, can also positively affect the overall health of women and men in hard-to-reach communities. The analysis of project achievements indicates that MH has been successful in achieving its objectives; the project has reached or exceeded all benchmarks set for selected indicators. Capacity-building of local partners through training has been a strength of MH; these efforts are reflected in increased LA/PM use. TECHNICAL CONCLUSIONS AND LESSONS LEARNED 1. Community-level distribution of misoprostol tablets: Community-level distribution of misoprostol is a highly effective life-saving approach supported by statistics on maternal deaths. There is potential for scaling up the intervention throughout Bangladesh, but several challenges need to be addressed in order to do so. Scaling up from district-level pilots to the national scale might take time and require the concerted efforts of many players. 2. Institutionalization of AMTSL: AMTSL is an easy and powerful practice to prevent PPH. Decisionmakers, program managers, and service providers interviewed highly valued MH’s support with AMTSL. However, it is important to assess whether the service providers are successfully practicing the AMTSL skills they have learned. 3. Increasing male involvement in family planning: The interventions to increase male involvement in LA/PM appear to be working, and there is potential for expansion and replication. Additional increases in the CPR, particularly the use of vasectomy, can be achieved by further expanding the satisfied clients program. 4. Integrating postpartum family planning services into MCH: Training service providers in postpartum tubectomy appears to be effective, as reflected in upward trend in the method’s acceptance. Immediate postpartum IUDs could also be a good option for many women who deliver at facilities; however, the demand for postpartum IUDs is very low. Clinical training for postpartum IUD insertions has to take place in clinical settings and requires high client caseloads. In addition, if the trainees do not practice the acquired clinical skills, those skills will be lost. The team concluded that the MH project should discontinue its emphasis on postpartum IUDs, since there is little use for this skill in the Bangladesh setting. The lesson learned is that, in spite of good intentions, programs need to be monitored and adjusted during the life of a project. 32 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 5. Increasing demand for LA/PM through BCC: It was clear to the team that complete lessons learned will not be known until the end of the project, and that the end-of-project assessment should reveal how effective the currently implemented BCC mechanisms are. The team concluded, based on input received during interviews with other stakeholders, that the weak link in MH is its BCC efforts. The BCC campaign was delayed, did not use up￾to-date technologies, and did not address widespread misperceptions regarding LA/PM. The lesson learned is that if the BCC efforts did not have these weaknesses, demand creation may have been enhanced. 6. Other innovative approaches: The team observed that the MH has incorporated several new and innovative approaches into its programming. These include the YMC initiative, the bottom-up contraception projection based on client segmentation, and involving the private sector in LA/PM provision. Although there has not been enough time for these approaches to be tested and fully implemented, the team believed that they could provide some interesting lessons learned on how to address new and innovative approaches for future programming. 7. Contributions to HSS: Although the MH project did not include an objective on systems strengthening and had a narrow technical focus, the project by its very nature had to incorporate several HSS-related interventions and successfully contributed to strengthening the health systems. With a more structured mandate for HSS, projects like MH can be highly effective, especially at the district level. 8. Advocating for policy change: Policy support and advocacy were integral parts of the MH project design right from the beginning, and the project has had some significant accomplishments in this area. Especially in areas such as LA/PM and PPH which have thus far received such little attention, the lessons learned are that a well-planned, well executed, and action-oriented policy and advocacy campaign can lead to policy changes in a very short period of time. 9. Compliance issues: The recent incident in which the DGFP was about to launch a lottery related to sterilization procedures was stopped as a result of actions undertaken by MH and USAID. As MH gathered information about this lottery, however, there was an approximate three-week delay. 10. Gender issues: MH has incorporated gender-related considerations into its programming, and gender equity is integrated into its training, community orientation, and outreach activities. MH has also set up a system of disaggregating data for reporting and evaluation purposes. The lesson learned is that programs that are purposely designed to respond to gender considerations can have a definite effect on the use of permanent methods. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 33 VII. SUMMARY OF RECOMMENDATIONS FOR THE REMAINING LIFE OF THE PROJECT OVERARCHING STRATEGIC RECOMMENDATIONS The MH project is effectively addressing LA/PM and PPH, areas of family planning and reproductive health that have not received adequate attention in Bangladesh to date. The team therefore recommends that, for the remaining life of the project, MH should: Build on program sustainability. Ensure timely handover to local partners. Continue expanding and replicating successful models. Document lessons learned. TECHNICAL RECOMMENDATIONS 1. Continue community-level distribution of misoprostol tablets – MH should investigate the reasons for the discrepancy between the number of misoprostol tablets distributed and used during the last quarter of 2011. – MH should recognize in its program planning that scaling up from district-level pilots to the national scale will take time and require the concerted efforts of many players. – MH should continue to assist the local partners for a carefully planned and phased roll￾out and closely monitor field activities. 2. Institutionalize AMTSL – Over the remaining life of the project, MH should work with the DGFP and DGHS to ensure that the newly recruited service providers receive on-the-job training by the trained providers, and that monitoring and supervision mechanisms are in place and functioning. – An audit of service data, supplemented by direct observation if possible, should be conducted to ensure that AMTSL is actually being implemented in these facilities. 3. Increase male involvement in family planning – During the remaining life of the project, MH should assess the results of the efforts so far and provide lessons learned for further expansion of the initiative. 4. Integrate postpartum family planning services into MCH – MH should discontinue postpartum IUD training and assess the results achieved so far to document the lessons learned. – MH should work to strengthen the postpartum counseling of service providers in the DGHS facilities. 34 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 5. Increase demand for LA/PM through BCC – MH should continue to collect information so that the impact and results of BCC efforts can be documented in the end-of-project assessment. – To the extent possible in the remaining life of the project, MH needs to examine whether some of its current approaches are effectively reaching a large audience or if a more modernized BCC approach and different messages are needed to overcome negative perceptions and incorrect beliefs about LA/PM. 6. Expand other innovative approaches – MH should continue to expand its promising innovative approaches (YMC, bottom-up contraceptive projections, and working with the private sector in LA/PM provision). – By the end of the project, MH should assess the results and document the lessons learned of these approaches. 7. Contribute to strengthening health systems – Although not part of its official mandate, MH should continue its work on strengthening health systems through national- and local-level planning, human resources capacity￾building, quality improvement, and logistics management. 8. Advocate for policy change – MH should build on its strong record of working through the NTC and continue to advocate for policy changes regarding LA/PM. 9. Address compliance issues – The new Tiahrt poster (produced by MH in conjunction with JHU and the DGFP) should be distributed widely with instructions on where to place it so clients can best see it. – MH, in conjunction with EngenderHealth, should continue training providers at all levels of government—including government officials—in voluntarism and informed consent. 10. Address gender issues – MH should continue implementing its women-centered approaches. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 35 ANNEX A. SCOPE OF WORK TITLE: USAID/Bangladesh: Midterm Evaluation of the Mayer Hashi Project [Effective Delivery of Long-acting and Permanent Methods (LA/PM) of Family Planning and Maternal Health Services in Bangladesh] Contract: Global Health Technical Assistance Bridge Project (GH Tech) I. PERFORMANCE PERIOD Work is to begin depending on the availability of the selected consultants, with work beginning on or about (o/a) early March 24, 2012, with field work completed in the end of April 2012 and final unedited report and close out concluding by mid-May 2012. II. FUNDING SOURCE Mission field support funds III. PURPOSE OF ASSIGNMENT This external evaluation comes at the chronological midpoint of the project. It is a midterm formative evaluation whose objectives are to determine: 1. Review, analyze, and evaluate the effectiveness of the Mayer Hashi Prevention Program in achieving program objectives and contributing to USAID/Bangladesh’s efforts to reduce maternal mortality and increase use of long-acting and permanent contraceptive methods (LA/PM). 2. Evaluate major constraints in achieving expected project results. 3. Provide specific recommendations and lessons learned on strategies and approaches the program should pursue over the next years of implementation and for future program panning. The evaluation should cover the project period from June 2009 to December 2011. However, this project is a follow-on program to previous USAID investments in this area and therefore the Mayer Hashi project activities need to be examined in the overall context of family planning and maternal health program in the country. The findings and recommendations of the evaluation will be used to improve implementation of the ongoing project and will also be used in the design of future follow-on project or in the design of other relevant PHNE projects/programs. With the exclusion of procurement sensitive sections USAID intends to disseminate the report widely with the public health stakeholders such as government and NGO program managers, USAID implementing partners, donors, and health professional associations. Upon clearance on procurement sensitivity, USAID will actively share the document with government agencies, donors, implementing partners, and other NGOs through mail correspondences and seminars/workshops. USAID expects the evaluation report will benefit the implementing partners, host government, and other donors in improving their understanding on the program and in designing interventions for future programs. 36 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT IV. BACKGROUND Mayer Hashi Project (Effective Delivery of Long-acting and Permanent Methods of Family Planning (LA/PM) and Maternal Health Services in Bangladesh) The Mayer Hashi project (May 20, 2009–September 30, 2013) in Bangladesh is an Associate Award (Associate Cooperative Agreement No. 388-00-09-00078) supported by USAID/Bangladesh under the Global RESPOND Project. It is managed by EngenderHealth in partnership with Johns Hopkins Bloomberg School of Public Health Centre for Communication Programs and the Population Council. The project works on two main objectives: 1) address the need for family planning through expanding contraceptive choices with an emphasis on LA/PM and 2) prevent postpartum hemorrhage (PPH) using clinical and community approaches, such as active management of third stage of labor (AMTSL) and community-based provision of misoprostol. Over the life of the project, Mayer Hashi will work in 21 districts in three low performing divisions—Sylhet, Chittagong, and Barisal. Status of LAPM and Maternal Mortality in Bangladesh Bangladesh, with support from USAID, the private sector, and NGOs has managed to reduce the country’s total fertility rate (TFR) over the last three decades from 6.3 lifetime births per woman in 1975 to the current 2.5 births per woman. However, the demographic imperative for continuing, refining, and expanding family planning still remains due to the large youth cohorts entering their reproductive years, coupled with the population growth rate of 1.5% and a projected population size of 220 million over the next four decades. Beyond demographics, family planning has saved lives in Bangladesh. Family planning is helping support Bangladesh’s quest to achieve the Millennium Development Goals for maternal health (MH), water/sanitation, and education (USAID 2007). Despite nearly a decade of investment, use of LA/PM in the contraceptive method mix remains stagnant in Bangladesh. According to the Bangladesh Demographic and Health Survey (BDHS) 2007, the share of LA/PM is less than 10% of the total contraceptive methods used in Bangladesh. Improving access to LA/PM was identified as a priority issue under the U.S. Global Health Initiative (GHI) and the Government of Bangladesh’s new national health sector plan. USAID/Bangladesh supports a number of activities intended to support the efforts of the Government of Bangladesh to revitalize use of LA/PM. While maternal mortality has declined in Bangladesh, childbirth remains a serious health risk for women. The Maternal Mortality Survey 2010 demonstrates a 40% decline in maternal mortality ratio from 322/100,000 live births in 2001 to 194/100,000 live births in 2010. Building on this success, the Government of Bangladesh aims to further decrease maternal mortality by addressing one of the main causes of maternal death—postpartum hemorrhage. In the Bangladesh Maternal Mortality and Health Care Survey (BMMHS) 2010, hemorrhage (31%) and eclampsia (20%) are identified as the dominant direct obstetric causes of death, together responsible for more than half of the maternal mortality rate. Mayer Hashi Project The Mayer Hashi project is a follow-on program to the ACQUIRE project (October 2003– December 2008), which worked to advance and support the availability, quality, and effective use of facility-based reproductive health and family planning services at every level of the health care system and to strengthen links between facilities and communities. The ACQUIRE project MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 37 focused on working with the government at the national, district, upazila, and union level to strengthen LA/PM services. The project provided technical assistance to improve planning, supply, and demand of services, as well as advocacy for policy changes. The Mayer Hashi project applies the Supply-Enabling Environment-Demand (SEED) model previously developed by a prior USAID-supported project. Mayer Hashi uses five overarching strategies to address and overcome barriers to LA/PM’s availability and use, and to support the prevention of PPH: (1) accelerating the transfer of capacity through replication, scale-up, and making sustainable changes in quality of and access to LA/PM services in the government, private sector, and NGOs; (2) coordination and linkage building between the family planning and maternal health communities; (3) integrating family planning and maternal health concepts into systems strengthening (training and supervision), community events, male involvement, and advocacy for all reproductive health issues, with special attention to integrating family planning into postpartum, post-abortion care (PAC, and fistula care services; (4) addressing reproductive health intentions by “increasing” LA/PM; and (5) mainstreaming gender. The program description of the project cooperative agreement and the annual workplans detail the strategies and interventions. Mayer Hashi Results Framework The overall goal of Mayer Hashi is effective delivery of LA/PM and maternal health services in Bangladesh. To achieve this goal, the project pursues the following two objectives: (1) increase the use of long-acting and permanent methods of family planning and (2) improve postpartum hemorrhage prevention practices in the project working area. Development Hypothesis The hypothesis is that USAID support for increased use of LA/PM and enhanced clinical and community maternal approaches will further reduce the fertility rate and considerably lower maternal mortality, respectively. It is assumed that the project’s interventions will influence the government’s policies and thereby improve government performance in other areas as well. Critical Assumptions: The Mayer Hashi project identified the following critical assumptions that underpin the success of the project: Ministry of Health and Family Welfare’s policies are supportive to implement best practices, models, and new approaches. National Institute of Population Research and Training (NIPORT) will come forward to take the lead to support institutionalizing LA/PM training. The availability of family planning commodities will be ensured by DGFP. The Directorates of Health and Family Planning will be supportive, to include training on AMTSL and use of partograph in the government operational plan, and include misoprostol in the Essential Drug List (EDL) and DDS Kit Box, respectively. 38 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Figure A1. Results Framework of Mayer Hashi Project Project Objective: Increase use of long-acting and permanent methods of contraception IR 1: Strengthened effective delivery and increased/sustained performance of LA/PM services through holistic and evidence￾based approaches IR 1.1: Program needs identified; models developed and tested to integrate LA/PM services with other health services IR 1.2: Critical service delivery processes identified and key improvements made IR 3: Strengthened commitment, policy support, and enabling environment for LA/PM services through knowledge generation dissemination and use IR 3.1: Policy barriers identified and advocacy by leadership for policy changes IR 3.2: State-of-the-art global lessons on LA/PM applied IR 2: Increased demand and sustained use of LA/PM through evidence-based and innovative approaches IR 2.1: Social and behavioral change for LA/PM designed and tested IR 2.2: Demand created for LA/PM Project Objective: Promote PPH prevention practices in Mayer Hashi working areas IR 3: Strengthened commitment, policy support, and enabling environment for prevention of PPH through knowledge generation, dissemination, and use IR 3.1: Support AMTSL policy and advocacy IR 3.2: Support advocates/ champions actively involved in changing policies and facilitating the use of misoprostol IR 1: Strengthened effective delivery and increased/sustained use of AMTSL at facility, misoprostol at community IR 1.1: Ensure adoption of best practices for AMTSL clinical activities IR 1.2: Ensure adoption of best practices for Misoprostol use at community level IR 1.3: Strengthen monitoring and supervision systems IR 2: Increased demand and sustained use of drugs for prevention of PPH through evidence-based and innovative approaches IR 2.1: Increased community awareness on importance of facility delivery for PPH prevention IR 2.2: Increase community awareness on importance of proper use of misoprostol for PPH prevention MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 39 V. SCOPE OF WORK The evaluation should address the following questions: 1. To what extent has the project achieved its objectives against expected results on LA/PM and PPH? 2. What components of the current Mayer Hashi strategy (both LA/PM and PPH) have been most/least effective and what can be done to improve the project LA/PM performance? 3. How effectively has the project coordinated with the Government of Bangladesh, other donors, NGOs, and the private sector to achieve its LA/PM and PPH objectives? 4. How has the project contributed to strengthening the health systems of the country in delivering LA/PM and PPH services? 5. What recommendations or actions should USAID take to support future Government of Bangladesh efforts in expanding access to LA/PM and PPH services in Bangladesh? 6. What are the project management issues that adversely impact performance of the project? VI. METHODOLOGY The evaluation team will work in close consultation with USAID/Bangladesh and EngenderHealth. The key issues to be addressed by the evaluation team should be developed in consultation with the Office of Population, Health, Nutrition and Education (OPHNE) team during the evaluation team’s first meeting with the Mission. The evaluation team should start its work with a paper review of all the documents cited in the “Document Review” section. It is recommended that the evaluation team consider a mixed-method evaluation approach with a focus on current clients and potential clients. The methodology should combine a review of quantitative data and application of qualitative evaluation techniques to obtain information, opinions, and data from counterparts, contractors, partners, clients, beneficiaries, government entities, and other donors. The approach should be participatory and should involve the use of questionnaires as appropriate. By using a mixed approach, the evaluation team will gain insight on the impact of Mayer Hashi project activities (mostly from quantitative data collected by the project and others) and the processes (mostly qualitative information provided by the project staff and key informants) that lead to those impacts. Sequential and iterative approaches should be used to integrate the mixture of methods at various stages of the evaluation. In consultation with USAID/Bangladesh OPHNE staff, the team will draft an assessment methodology/design for USAID approval. The team will conduct a two-day in-country team planning meeting upon arrival in Bangladesh and before starting the in-country portion of the assessment. The planning meeting outcomes will be shared with USAID/Bangladesh and the health team will participate in sections of the planning meeting. The following essential elements should be included in the methodology as well as the additional methods proposed by the team: Review of background documentation: USAID Bangladesh OPHNE will provide the team leader with a core list and/or copies of the agreement, reports of recent relevant assessments, and other key documentation before the assessment begins. The team leader will be responsible 40 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT for expanding this background documentation as appropriate—reviewing, prioritizing and distributing it to other team members for their review. All team members will review relevant documentation before their initial team meetings. The evaluation team should consult a broad range of background documents apart from project documents provided by USAID/Bangladesh. These include documents such as the national survey on family planning and maternal health, as well as national strategies on population, maternal health, and BCC. USAID and the Mayer Hashi project will provide the assessment team with a package of briefing materials, including: The program description for the Mayer Hashi project. PMP of the Mayer Hashi project. Project quarterly and annual reports, workplans, and management reviews developed as part of routine monitoring. Mayer Hashi project benchmark survey report. The draft LA/PM strategy developed by Mayer Hashi project. The Report of the Assessment on the Clinical Contraceptive Service Delivery done on request of MOHFW and USAID in 2000. The Bangladesh AMTSL assessment report, 2008. The Misoprostol Tangail project evaluation report, 2010. BDHS 2007 report. Bangladesh Maternal Mortality and Maternal Health Care Survey 2010. Streatfield, P. K., Karar, Z. A. “Population Challenges for Bangladesh in the Coming Decades.” Journal of Health, Population and Nutrition. 2008, 26(3): 261–272. USAID/Bangladesh midterm assessment on long-term and permanent methods of family planning in Bangladesh, 2007. Addressing Gaps in the Existing USAID/Bangladesh Health Program 2008. (THIS DOCUMENT IS INTERNAL TO USAID.) USAID/Bangladesh Family Planning Assessment Report 2009. National Communication Strategy for Family Planning and Reproductive Health. 2008, Ministry of Health and Family Welfare–Directorate of Family Planning, Bangladesh. Summary of BCC assessment (notes by Elizabeth Fox, GH/USAID/Washington). The GHI Bangladesh Strategy. USAID/Bangladesh Country Development Cooperation Strategy 2011–16 (as cleared by the front office). MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 41 Planning meetings: The full team will meet upon arrival in Bangladesh and will finalize planning during their two-day TPM in-country. The TPMs are essential in organizing the team’s efforts. During the TPMs, the team should: Clarify all team members’ roles and responsibilities, including drafting of the report. Develop and review final assessment questions. Review and finalize the timeline and share this with OPHNE. Develop and finalize data collection methods (disaggregated–by sex, age, geographical region, education level, etc.) and instruments (USAID data standards apply). Review and clarify any logistical and administrative procedures for the assignment. Establish a team atmosphere, share individual working styles, and agree on procedures resolving differences of opinion. Develop a preliminary draft outline of the team’s report. Assign drafting responsibilities for the final report. Initial team briefing meetings with OPHNE: The full team will have an initial meeting with OPHNE officials in Bangladesh. During this meeting they will share an outline and explanation of the design of the assessment, and receive feedback from OPHNE. The full team and/or members will have follow-up meetings with specific OPHNE staff at the outset of the process, and will remain available for consultation throughout the whole process as appropriate. Key informant interviews: The full team or team members as appropriate will have interviews with the following (not inclusive): Various USAID offices and other U.S. Government offices in Bangladesh. Mayer Hashi implementing partners at both HQ and field level. Stakeholders: beneficiaries, professional associations, universities, community members, etc. Key Government of Bangladesh representatives across multiple sectors, including field level staff engaged in family planning and maternal health programs. Major donors involved in population/family planning delivery of long-acting and permanent family planning methods and maternal health services. Staff from other relevant USAID implementing organizations. Data from key informant interviews may be organized to quantifiable information on certain indicators or be used to validate data obtained from other reports. Site visits: Team members, as appropriate, will visit selected project implementation sites in 21 districts in the Sylhet, Chittagong, and Barisal divisions in the eastern and southern part of Bangladesh. Limitation: The evaluation will not be engaged in primary data collection from any statistically designed sample of beneficiaries or providers to measure the effect of the project on defined 42 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT indicators. It will rather depend on the secondary data available from the routine management information system records and the reports of other surveys and assessments conducted by this project or other programs. Since key informant interview will be a major source for validation of information available from the project, chances of bias are likely. The evaluation should carefully decide on the methodology and select interviewees in a way that the possibility of bias is avoided or reduced to a minimum. VII. TEAM COMPOSITION, SKILLS, AND LEVEL OF EFFORT Team Composition We would like to engage the services of a four-person evaluation team. The team should include three international consultants and one local consultant. The former should include specialists with the following areas of expertise: family planning and maternal health, experience in conducting evaluations and assessments, behavior change communication, private sector, sustainability, and health systems. The one local consultant should have an excellent understanding of the Bangladesh public health system and be fluent in Bangla. The team leader should be an evaluation expert. The team leader should be an independent consultant, but one of the technical specialists could be USAID/Washington Global Health staff. The fourth team member will be the local technical consultant. Team Leader (Evaluation Specialist): Should be an independent consultant with a PhD or an MPH and related postgraduate training in public health program evaluation. S/he should have at least 10 years senior-level experience working in health systems programs or health program evaluation in developing countries. S/he should have extensive experience in conducting qualitative and quantitative evaluations. Excellent oral and written communication skills are required. The team leader should also have experience in leading evaluation teams and preparing high-quality documents. This specialist should have wide experience or familiarity of USAID-funded reproductive health programs and should have a good understanding of health systems in South Asia, preferably in Bangladesh. S/he should also have a good understanding of project administration, financing, and management. The team leader will take specific responsibility for assessing and analyzing the project’s progress towards quantitative targets, performance, and benefits/impact of the strategies. The team leader will also look at the potential sustainability of Mayer Hashi project approaches and activities. The team leader will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables in consultation with the other members of the team. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Bangladesh team and key partners. The team leader will submit the draft report, present the report, and, after incorporating USAID Bangladesh staff comments, submit the final draft report to USAID/Bangladesh within the prescribed timeline. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 43 Reproductive Health Program Expert The reproductive health program expert will have at least 7–10 years of experience in management of, or consulting on, reproductive health programs. S/he should have a proven background and experience in family planning and maternal health and a strong understanding of the challenges Bangladesh faces for increasing LA/PM. S/he should also have a good understanding of the relevant national programs in RH/ MCH, including the public and private sector. The reproductive health expert will be responsible for assessing the ability of the project to achieve outcomes in LA/PM and maternal health and provide technical leadership in this area. The consultant will participate in team meetings, key informant interviews, group meetings, and site visits, and draft the sections of the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or other stakeholders. S/he will submit contributions to the team leader within the prescribed timeline. S/he will analyze Mayer Hashi project behavior change interventions in its intervention communities and assess the effectiveness and appropriateness of the approaches adopted by the project to improve knowledge, health-seeking behavior, and health outcomes. S/he will also assess the technical focus of BCC activities, and whether they are the appropriate mix and topics for intervention communities. Host Country National Health Expert: The host country national health expert will serve under the team leader. S/he should have at least 10 years of experience working in the field of family planning and maternal health and have thorough knowledge of the national population and health program. Duties will be determined in consultation with the team leader. The host country national will participate in team meetings, key informant interviews, group meetings, site visits, and contribute in drafting the notes for the report relevant to his/her expertise and role in the team. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or other stakeholders. S/he will communicate with the team leader and other consultants to produce written notes to incorporate in the report as required in addressing comments and feedbacks from USAID. S/he is required to make his/her contributions to the team leader within the timeline. Level of Effort An illustrative table of the level of effort (LOE) is found below. Dates may be modified based on availability of consultants and key stakeholders, and time needed for field work. Task Deliverable Team Leader Technical Specialist(Int’l) Local Consultant and Logistics Logistical preparations for team 4 days Review background documents and offshore preparation work 4 days 3 days Travel to Bangladesh 2 days 2 days Team Planning Meeting /meeting with USAID 2 days 2 days 2 days 44 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Task Deliverable Team Leader Technical Specialist(Int’l) Local Consultant and Logistics Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits 10 days 10 days 10 days Discussion, analysis, and draft evaluation report in country including discussion with OPHNE 5 days 5 days 5 days Debrief meetings with USAID OPHNE and the Mission and Government of Bangladesh (preliminary draft report due to USAID) 1 day 1 day 1 day Submit report/depart Bangladesh/travel to U.S. 2 days 2 days USAID and partners provide comments on draft report (5 days) Team revises draft report and submits final to USAID (out of country) 5 days 3 days 3 days USAID completes final review (3 days) TL incorporates final comments 1 day GH Tech edits/formats report (1 month) Total Estimated LOE 32 days 28 days 25 days A six-day work week is approved while in-country. VIII. LOGISTICS GH Tech will be responsible for all international travel and consultant logistics. IX. DELIVERABLES Workplan: During the TMP, the team will prepare a detailed workplan which will include the methodologies to be used in the evaluation. The workplan will be submitted to AOTR at USAID/Bangladesh for approval no later than the sixth day of work. A written methodology plan (evaluation design/operational workplan) will be prepared during the TPM and discussed with USAID prior to evaluation. Key informant interview questionnaire: Different sets of questionnaires will be prepared for interview with key informants at different levels during the TPM and discussed with USAID prior to evaluation. Debriefings: The full team will debrief OPHNE, the USAID Mission Director’s office, and other USAID/U.S. Government offices on their findings, conclusions and recommendations (using a PowerPoint presentation and any briefing materials required) before leaving Bangladesh. OPHNE will provide feedback during the briefing meeting. The team will also debrief USAID implementing partners and/or Government of Bangladesh officials using PowerPoint presentations and other briefing materials as required. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 45 Draft evaluation report: The evaluation team will provide OPHNE with a full draft report that includes all the components of the final evaluation report prior to their departure from the country. OPHNE will provide comments on the draft report to the evaluation team within five working days of receiving the draft report. The draft evaluation report will include, at a minimum, the following: scope and methodology used, important findings (empirical facts collected by evaluators), conclusions (evaluators’ interpretations and judgments based on the findings), recommendations (proposed actions for management based on the conclusions), and lessons learned (implications for future designs and for others to incorporate into similar programs). Final assessment report: The team will submit a final report to GH Tech and USAID incorporating Mission comments and suggestions no later than five days after USAID/Bangladesh provides written comments on the team’s draft final evaluation report (see above). Only if the final draft is approved by USAID prior to April 16, 2012 will GH Tech provide the edited and formatted final document approximately 30 days after USAID provides final approval of the content. Otherwise, USAID will need to go through another mechanism to finalize the report. The final report will then be edited/formatted by GH Tech by approximately one month after the Mission has reviewed the content and approved the final revised version of the report. The final report should have the following criteria to ensure the quality of the report: The evaluation report should represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not, and why. Evaluation reports shall address all evaluation questions included in the scope of work. The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the technical officer. Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. Evaluation findings will assess outcomes and impact on males and females. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data—not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. Sources of information need to be properly identified and listed in an annex. Recommendations need to be supported by a specific set of findings. Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 46 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT The format of the final evaluation report should strike a balance between depth and length. The report will include a table of contents, table of figures (as appropriate), acronyms, executive summary, introduction, purpose of the evaluation, research design and methodology, findings, conclusions, lessons learned, and recommendations. The report should include, in the annex, any dissenting views by any team member or by USAID on any of the findings or recommendations. The report should not exceed 30 pages, excluding annexes. The report will be submitted in English, electronically. The report will be disseminated within USAID. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) to Development Experience Clearinghouse (DEC) for dissemination among implementing partners and stakeholders. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format, (2) organized and fully documented for use by those not fully familiar with the project or the evaluation, and (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data could be provided to the AOR. Reporting Requirements (to be finalized during the team meeting): The total pages, excluding references and annexes, should not be more than 30 pages. The following content (and suggested length) should be included in the report: 1. Table of Contents 2. Executive Summary—concisely state the project purpose and background, key evaluation questions, methods, most salient findings, and recommendations (2–3 pp.) 3. Introduction—context in which intervention took place, including a summary of any relevant history, demography, socioeconomic status, etc. (1 pp.) 4. The Development Problem and USAID’s Response—brief overview of Mayer Hashi project, USAID program strategy, and activities implemented in response to the problem (1 pp.) 5. Purpose of the Evaluation—purpose, audience, and synopsis of task (1 pp.) 6. Methodology—describe evaluation methods, including strengths, constraints, and gaps (2 pp.) 7. Findings/Conclusions—describe and analyze findings for each objective area using graphs and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcome (12–15 pp.) 8. Recommendations—prioritized for each objective area; should be separate from conclusions and be supported by clearly defined set of findings and conclusions (3–4 pp.) 9. Lessons Learned—provide a brief of key technical and/or administrative lessons that could be used for future project or relevant program designs (2–3 pp.) 10. References—(including bibliographical documentation, meetings, interviews, and focus group discussions) 11. Annexes—to include statement of work, documents reviewed, evaluation methods, data generated from the evaluation, tools used, interview lists, and tables. Annexes should be succinct, pertinent, and readable. Should also include, if necessary, a statement of differences regarding significant unresolved difference of opinion by funders, implementers, or members of the evaluation team on any of the findings or recommendations. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 47 The Mission should receive five hard copies of the final unedited version of the report and an electronic copy of the final report. The report format should be restricted to Microsoft products, and 11-point type font should be used throughout the body of the report, with page margins one inch top/bottom and left/right. X. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment: Recruit and hire the evaluation team. Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID/Bangladesh will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks: Before In-Country Work SOW: Respond to queries about the SOW and/or the assignment at large. Consultant Conflict of Interest (COI): To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates. Documents: Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment. Local Consultants: Assist with identification of potential local consultants, including contact information. Site Visit Preparations: Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs. Lodgings and Travel: Provide guidance on recommended secure hotels and methods of in￾country travel (i.e., car rental companies and other means of transportation) and if necessary, identify a person to assist with logistics (i.e., visa letters of invitation, etc.). During In-Country Work Mission Point of Contact: Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work. Meeting Space: Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e., USAID space if available, or other known office/hotel meeting space). Meeting Arrangements: Assist the team in arranging and coordinating meetings with stakeholders. 48 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Facilitate Contact with Implementing Partners: Introduce the evaluation team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After In-Country Work Timely Reviews: Provide timely review of draft/final reports and approval of deliverables. XI. MISSION CONTACT PERSON Dr. Sarker Senior Clinical Officer/PHN USAID/Bangladesh Tel: 880-2-885 5500 x 2313; Cell: 01713-009878 Email: ssarker@usaid.gov Md. Nasiruzzaman Email: mnasiruzzaman@usaid.gov XII. COST ESTIMATE GH Tech will provide a cost estimate for this activity. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 49 ANNEX B. PERSONS CONTACTED U.S. AGENCY FOR INTERNATIONAL DEVELOPMENT/BANGLADESH Khadijat Mojidi, Office Director, OPHNE Chris McDermott, Acting Deputy Office Director, OPHNE Marunga Manda, Senior M&E Advisor, Program Office Leslie-Ann Burnette-Badinga, Supervisory Contracting Officer, Office of Acquisition & Assistance Thibaut Williams, Health and Population Officer, OPHNE Sukumar Sarker, Sr. Clinical Officer, OPHNE Sumana Binte Masud, Project Management Specialist/Civil Society Advisor, Democracy & Governance Farheen L. Khurrum, M&E Specialist, Program Office MINISTRY OF HEALTH AND FAMILY WELFARE, DIRECTORATE OF HEALTH Kajal Kanti Barua, Civil Surgeon, Cox’s Bazar Md. Mahbubul Alam, Superintendent, District Sadar Hospital, Cox’s Bazar Munawar Sultana (Lina), Consultant, Gynae, District Sadar Hospital, Cox’s Bazar Ratan Chowdhury, Upazila Health & Family Planning Officer, Ramu Health Complex, Cox’s Bazar Anwar Ahmed, Upazila Health & Family Planning Officer, Chokoria Health Complex, Cox’s Bazar Saber Ahmed, RMO, Chokaoia Health Complex, Cox’s Bazar Syed Md. Osman Ghani, Assistant Health Supervisor, Fashiakhali Union, Chokoria, Cox’s Bazar Jahangir Alam, Health Assistant, Fashiakhali Union, Ward #1, Community Clinic, Chokoria, Cox’s Bazar MINISTRY OF HEALTH AND FAMILY WELFARE, DIRECTORATE GENERAL OF FAMILY PLANNING M. M. Neazuddin, Director General, Family Planning A. K. M. Mahbubur Rahman, Line Director, Clinical Contraception Services Delivery Program Mohammed Sharif. Director, MCH Services & Line Director, MCRH Md. Shamsul Karim, Assistant Director (Services) Kim Rook, Technical Advisor, Knowledge Management and Communication, IEM Unit Farid Uddin Ahmed, Assistant Director (Services), MCH Services Unit Fahmida Sultan, Assistant Director, MCH & Deputy Program Manager, MHS Ishrat Jahan, Program Manager Md. Joynal Haque, Medicine Specialist 50 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT Dipak Talukder, Deputy Director, Cox’s Bazar Md. Alamgir Hossain Sarkar, Medical Officer (Clinic), MCWC, Cox’s Bazar M. K. Kaar, Medical Officer, MCHFP, MCWC, Cox’s Bazar Jashimuddin Md. Yousuf, Upazila Family Planning Officer, Ramu, Cox’s Bazar Archana Rani Paul, Family Welfare Visitor, Sadar Clinic, Ramu, Cox’s Bazar Chowdhury Morshed Alam, Upazila Family Planning Officer, Chokoria, Cox’s Bazar Pravankar Barua, AUFPO, Chokoria, Cox’s Bazar Shahana Begum, Family Welfare Visitor, Chokoria, Cox’s Bazar Rano Ara Yesmin, FWV, Family Welfare Center (FWC), Fashiakhali Union, Chokoria, Cox’s Bazar Kamrul Islam, SACMO, Family Welfare Center (FWC), Fashiakhali Union, Chokoria, Cox’s Bazar Mohammad Shafiqul Islam, Family Planning Inspector, Fashiakhali Union, Chokoria, Cox’s Bazar Shahin Sultana, FWA, Fashiakhali Union,Ward #1, Community Clinic, Chokoria, Cox’s Bazar MAYER HASHI PROJECT (ENGENDERHEALTH) Abu Jamil Faisel, Country Representative Ellen Themmen, Technical Director Mizanur Rahman, Sr. Technical Advisor S. M. Nizamul Haque, Team Leader, Policy and Advocacy Mahboob-E-Alam, ME&R Program Coordinator Sanjida Hasan, Senior Technical Officer Fatema Shabnam, Team Leader, Service Delivery and Training MD. Saikhul Islam Helal, Team Leader, PPHP & MH Zeenat Sultana, Team Leader, Demand Generation Team Liaquat Ali, Senior Program Officer, ME&R Bashir Ahmed, Assistant Program Officer, PPH, Prevention Team SMILING SUN FRANCHISE PROGRAM Dr. Arefin Amal Islam, Health Officer Mohammed Isha, Clinic Manager, FDSR, Cox’s Bazar Samir Das, Senior Medical Officer, FDSR, Cox’s Bazar BANGLADESH CENTER FOR COMMUNICATION PROGRAM Mohammad Shahjahan, Director & CEO Zeenat Sultana, Deputy Director Md. Shahidul Alam, Assistant Director MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 51 POPULATION COUNCIL Sharif Mohammed Ismail Hossain, Associate SOCIAL MARKETING COMPANY Ashfaq Rahman, Managing Director Toslim Uddin Khan, General Manager, Program UNFPA Arthur Erken, Representative Bangladesh MAMONI PROJECT, SAVE THE CHILDREN, AND JHPIEGO Ishtiaq Mannan, Chief of Party DFID Shehlina Ahmed, Health and Population Adviser OGSB Latifa Shamsuddin, President Elect UNIVERSITY RESEARCH CO. (URC) Yogesh Rajkotia, PHNE Technical Advisor KINGDOM OF THE NETHERLANDS Ella de Voogd, First Secretary SRHR, Gender and Education SHOPS Dr. A. S. A. Masud, Country Representative MARIE STOPES/COX’S BAZAR Sujan Barua, Program Officer, Cox’s Bazar LOCAL GOVERNMENT Ghiasuddin Chowdhury, Chairman, Fashiakhali Union Parishad, Chokoria, Cox’s Bazar 52 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 53 ANNEX C. REFERENCE LIST Aladuddin, Mohammed, Gerald Bowers, Gary Lewis, and Betty Ravenholt. USAID/Bangladesh: Population and Family Planning Program Assessment. Washington, DC: Global Health Technical Assistance Project, April 2010. Al-Sabir, Dr. Ahmed, Mahboob E-Alam, Dr. Farhana Salim, Dr. Nazmul Huda, Dr. Nowrozy Kamar Jahan, Moslehuddin Ahmed, and Dr. Abu Jamil Faisel. “Assessment on the Availability and Routine Use of Active Management of the Third Stage of Labor (AMTSL).” E&R Study. Dhaka: EngenderHealth Bangladesh Country Office, 2008. Bangladesh Center for Communication Programs. Annual Report 2009–2010. 2009–2010. Bhattacharjee, Anurdha, Wahiduzzaman Chowdury, Peter Connell, William Jansen, and Judith Justice. Addressing Gaps in the Existing USAID/Bangladesh Health Program (limited internal distribution). Produced for review by USAID. December 2008. Directorate General of Family Planning (Bangladesh). Improving the Uptake of Long-acting and Permanent Methods in the Family Planning Program, Bangladesh National Strategy, 2011– 2016. Directorate General of Family Planning (Bangladesh). Improving the Uptake of Long-acting and Permanent Methods in the Family Planning Program, National LA/PM Strategy, 2010–2015. Dhaka: October 2010 (final draft). Effective Delivery of Long-acting and Permanent Family Planning Methods and Maternal Health Services in Bangladesh. M&E Plan. September 2010. Effective Delivery of Long-acting and Permanent Family Planning Methods and Maternal Health Services in Bangladesh. Workplan. May 20–September 30, 2009. Mayer Hashi Project. Baseline Survey Report. April 2011. Mayer Hashi Project. Community-based Distribution of Misoprostol for the Prevention of Postpartum Hemorrhage: Evaluation of a Pilot Intervention in Tangail District, Bangladesh. Dhaka: EngenderHealth/ Mayer Hashi Project, 2010. Mayer Hashi Project. Indicator Tracking Table. Mayer Hashi Project. Moving Forward for Health and Happy Families. Information packet of program briefs and project brochures. USAID/Bangladesh, EngenderHealth. Ministry of Health and Family Welfare. National Communication Strategy for Family Planning and Reproductive Health. Dhaka: Directorate General of Family Planning, 2008. The Netherlands Embassy in Bangladesh. Multi-annual Strategic Plan 2012–2015. National Institute of Population Research and Training, Mitra and Associates, and Macro International. Bangladesh Demographic and Health Survey 2007. Dhaka, Bangladesh and Calverton, MD: National Institute of Population Research and Training, Mitra and Associates, and Macro International, 2009. 54 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT National Institute of Population Research and Training, Mitra and Associates, MEASURE DHS, and ICF International. Bangladesh Demographic and Health Survey: Preliminary Report. Dhaka, Bangladesh and Calverton, MD: National Institute of Population Research and Training, Mitra and Associates, and Macro International, 2011. National Performance of Long-acting and Permanent Methods. Mayer Hashi project document. Performance of Long-acting and Permanent Methods. Mayer Hashi project document. Policy Changes Initiated by Mayer Hashi Project/EngenderHealth, 2010. Project Year Four Workplan, 1 October 2011–30 September 2012. Submitted to the Population, Health, Nutrition and Education Office, United States Agency for International Development, USAID/Bangladesh by the Mayer Hashi project RESPOND leader with Associate Cooperative Agreement No. GPO-A-00-08-00007-00 and Associate Agreement No. 388-A-00-09-00078-00. Project Year Three, Annual Report, 1 October 2010–30 September 2011. Submitted to the Population, Health, Nutrition and Education Office, United States Agency for International Development, USAID/Bangladesh by the Mayer Hashi project RESPOND leader with Associate Cooperative Agreement No. GPO-A-00-08-00007-00 and Associate Agreement No. 388-A-00-09-00078-00. Project Year Two Workplan: 1 October 2009–30 September 2010. Submitted to the Population, Health, Nutrition and Education Office, United States Agency for International Development, USAID/Bangladesh by the Mayer Hashi project RESPOND leader with Associate Cooperative Agreement No. 388-A-00-09=0078-00, EngenderHealth in partnership with Johns Hopkins University/Bangladesh Center for Communication Programs, the Population Council, The Futures Institute and Cicatelli Associates, Inc., and Family Health International. Rahaim, Stephen, Ahmed Jahiruddin, Meaghan Smith, and Helen Li. Bangladesh Private Sector Assessment of Long-acting and Permanent Family Planning Methods and Injectable Contraceptives. Bethesda, MD: Strengthening Health Outcomes through the Private Sector, Abt Associates Inc., 2011. Shamsuddin, Latifa. Prevention of PE and Eclampsia through Community-level Intervention in Bangladesh. PowerPoint presentation. Social Marketing Company. Information Packet and Fact Sheets. Stover, Carina, William Jansen, William Shamsher, Ali Khan, and Wahiduzziman Swapon Chowdhury. Long-term and Permanent Methods of Family Planning in Bangladesh. Washington, DC: GH Tech, 2007. Streatfield, P. K., and Shams El Arifeen. Bangladesh Maternal Mortality and Health Care Survey 2010, Summary of Key Findings and Implications. USAID, Australian Government AID Program, UNFPA, MEASURE Evaluation ICDDR, B, NIPORT. 2010. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 55 Streatfield, P. K., and Z. A. Karar. “Population Challenges for Bangladesh in the Coming Decades.” Journal of Health, Population and Nutrition. 2008, 26(3): 261–272. Strengthening Health Outcomes through the Private Sector (SHOPS). Information Packet and Fact Sheets. Updated Program Description (Technical Application): Effective Delivery of Long-acting and Permanent Family Planning Methods and Maternal Health Services in Bangladesh. RESPOND/Bangladesh Associate Award #388-A-00-09-00078-00 of USAID/Bangladesh. Implemented by EngenderHealth in partnership with Johns Hopkins University/Bangladesh Center for Communication Programs, The Population Council, The Futures Institute, and Cicatelli Associates, Inc. June 2, 2009. U.S. Agency for International Development. Best Practices at Scale in the Home, Community and Facilities for Improved Family Planning and Maternal and Child Health and Nutrition. Action Plan for Bangladesh. March 14, 2011. U.S. Agency for International Development. USAID/Bangladesh Country Development Cooperation Strategy 2011–2016. September 2011. U.S. Agency for International Development. USAID Evaluation Policy. Washington, DC: USAID, January 2011. U.S. Agency for International Development. U.S. Global Health Initiative Bangladesh: Interagency Program Strategy, 2011–2015. 56 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 57 ANNEX D. EVALUATION FRAMEWORK The evaluation framework consists of six basic questions, as outlined in the team’s SOW, which will be posed to informants and interviewees. The same questions will be used to ascertain information about both long-acting and permanent methods (LA/PM) and prevention of postpartum hemorrhage (PPH). These questions will serve as guidelines for discussion and will be elaborated upon during interviews with informants and focus groups. Under each of the questions, basic sources of information and data are indicated. A comprehensive list of documents that the team is using for reference and background information for the evaluation report is found in the annexes. This list will be expanded as the evaluation progresses. FAMILY PLANNING: LONG-ACTING AND PERMANENT METHODS (LA/PM) 1. To what extent has the project achieved its objectives against expected results on LAPM? Information will be drawn from informant interviews, baseline surveys, annual and quarterly reports, project workplans, policy-related documents, indicator tracking tables, project assessments, and the Mayer Hashi results framework. 2. What components of the current LA/PM strategy have been most/least effective and what can be done to improve the project LA/PM performance? Information will be drawn from the project’s performance monitoring plan and informant interviews. The project’s communication strategy will also serve as a basis to determine expected results. 3. How effectively has the project coordinated with the Government of Bangladesh, other donors, NGOs, and the private sector to achieve its LA/PM objective? Information will be drawn from informant interviews. 4. How far has the project been able to contribute to strengthening the health system of the country in delivering LA/PM services? Information will be obtained from informant interviews, project assessments, and project descriptions. 5. What recommendations or actions should USAID take to support the Government of Bangladesh’s efforts in expanding access to LA/PM services in Bangladesh in future? The team’s findings and analyses will serve as a basis for future recommendations. 6. What are the project management issues that positively or adversely impact performance of the project? The team will identify project-related management and coordination issues that adversely impact or effectively enhance project performance. 58 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MATERNAL HEALTH: POSTPARTUM HEMORRHAGE (PPH) 1. To what extent has the project achieved its objectives against expected results on PPH? Information will be drawn from informant interviews, baseline surveys, annual and quarterly reports, project workplans, policy-related documents, indicator tracking tables, project assessments, and the Mayer Hashi results framework. 2. What components of the current PPH prevention strategy have been most/least effective and what can be done to improve the project performance? Information will be drawn from the project’s performance monitoring plan and informant interviews. The project’s communication strategy will also serve as a basis to determine expected results. 3. How effectively has the project coordinated with the Government of Bangladesh, other donors, NGOs, and the private sector to achieve its maternal health objective? Information will be drawn from informant interviews. The project’s communication strategy will serve as a basis to determine expected results. 4. How far has the project been able to contribute to strengthening the health system of the country in delivering PPH prevention services? Information will be obtained from informant interviews, project assessments, and project descriptions. 5. What recommendations or actions should USAID take to support the Government of Bangladesh’s efforts in expanding access to PPH prevention services in Bangladesh in future? The team’s findings and analyses will serve as a basis for future recommendations. 6. What are the project management issues that positively or adversely impact performance of the project? The team will identify project-related management and coordination issues that adversely impact or effectively enhance project performance. MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 59 ANNEX E. LIST OF SITES VISITED IN COX’S BAZAR DISTRICT Maternal and Child Welfare Center, Cox’s Bazar District Sadar Hospital, Cox’s Bazar Upazila Health Complex, Ramu Upazila Health Complex, Chokoria Family Welfare Center, Fashiakhali Community Clinic, Fashiakhali 60 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT 61 ANNEX F. USAID’S COORDINATION WITH OTHER DONORS At the request of the health team at USAID/Bangladesh, the team met with three donors: UNFPA, Dfid, and the Netherlands. The purpose was to gain information about their collaboration with USAID and to learn what they are doing in their overall family planning/maternal health projects. UNFPA has remained a steadfast partner to the Government of Bangladesh and to USAID. The UNFPA representative with whom the team met praised MH’s pilot in misoprostol. He also noted that MH works well through the public sector, and that LA/PM has become an important part of the national program. UNFPA is involved with contraceptive procurement (working with the World Bank) and maintains strong liaison with all Government of Bangladesh family planning programs. UNFPA plans to increase their training efforts in areas where MH is not working and is addressing the serious issues surrounding the migration of men to urban areas, since the Government of Bangladesh’s family planning programs is focused in rural areas, and urban women are “oblivious” to family planning. The team met with a representative of the Government of the Netherlands, which is undertaking a new strategic focus on sexual and reproductive health and rights (SRHR). The program’s emphasis will be on advocacy to influence policy and improve ties between the government and the private sector, and raise awareness about issues such as family planning, violence against women, and youth sexuality. The program will look for ways to develop evidence-based knowledge related to SRHR, support policy initiatives in SRHR, and share knowledge and expertise of the Dutch in this area. Among the initiatives planned in the private sector is the provision of reproductive health care for employees in the garment industry. There are three main initiatives: “Unite for Body Rights” will work with five NGOs in 10 upazilas to focus on BCC for young people regarding RH. The “Safe” initiative will address violence against women in conjunction with the International Center for Diarrheal Disease and Research in Bangladesh (ICDDRB) in the slums of Dhaka. Cooperation with BRAC University will focus on research and mapping regarding training needs. The team met with a representative of Dfid. Dfid currently provides support to three initiatives related to reproductive health and family planning: 1) The Health, Population and Nutrition Sector Development Program (HPNSDP) functions in remote areas at the community level and aims to improve access to, and utilization of, essential health, population, and nutrition services, particularly for the poor. In the reproductive health arena, Dfid focuses on health policy and management. 2) Dfid’s Urban Primary Health Care Project (ends in June 2012), co-financed by SIDA and UNFPA, worked in six city corporations and five municipalities to support family planning and reproductive health. Dfid also participated in training midwives with the World Bank. Dfid is working with the Asian Development Bank, Smiling Sun Franchise Program and USAID on the design of a new project. 3) Dfid also works in conjunction with BRAC. BRAC’s Manoshi Project for maternal, neonatal, and child health initiative (urban) links slum residents with both traditional birth attendants in slum birthing huts and referral facilities for birth complications. BRAC is also working in rural areas through the Improving Maternal, Neonatal and Child Survival (IMNCS) initiative. 62 MIDTERM PERFORMANCE EVALUATION OF THE MAYER HASHI PROJECT CIDA: The team did not meet with representatives of CIDA due to time restrictions, although the Mayer Hashi project officials suggested that CIDA is an important player. A brief description of what CIDA is doing is therefore warranted. At the G-8 Summit in Muskoka, Ontario in June 2010, Prime Minister Stephen Harper announced Canada’s commitment to help save the lives of mothers, children, and newborns in developing countries. In January 2011, the Harper government announced a new initiative that included Bangladesh. In Bangladesh, the joint Government of Bangladesh-United Nations Maternal and Neonatal Health project was formed to purchase essential drugs and equipment, recruit and train health professionals, upgrade existing treatment centers and pilot innovative ways for delivering services. UNFPA is the implementing partner for this project. In addition, in its Human Resources for Health project, CIDA will work to improve maternal, newborn, and child health by increasing the number of nurses and skilled birth attendants, and will upgrade a large number of public sector nursing institutes and train nurse midwives annually. For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com